My first impression coming away from this is skepticism.
Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you.
Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer".
So is this thing talking to Google's servers all the time for the AI integration? So it won't work if you're not connected to the internet? Privacy concerns are obvious; now Google wants to have an AI watching literally everything you do on your computer?
Does it cost the user anything for the LLM use? If it's free will it stay free forever? That's quite a lot to give away if they're expecting people to use it to change a single word like in one of their examples. I guess they're expecting to make the money back by gathering data about literally everything you do on your computer.
There might be a killer app for AI integration with personal computers that has yet to be invented, but this doesn't look like it.
The killer app was conceived as early as the 1980s: an agent running on your computer, organizing your files, your schedule, your messages, your bills, bank accounts, etc. All the parts of your life that were routine drudgery should be able to be offloaded to a smart agent, based on your preference, to bring you the information you needed with natural language queries, contextualized to what you were doing at the time, when you need it.
What's being delivered now is, an agent running on someone else's computer, copying your data to someone else's database, with zero responsibility, or mandate to protect that data and not share with with anyone else (in fact, they almost always promise to share it with their thousand partners), offering suggestions and preferences based on someone else's so-called recommendations, influenced by paying the agent's operators, and increasing pressure to make using someone else's computers + agents the only way to interact with other people and systems.
There is no doubt that LLM's can do amazing things, but the current environment seems to make it nearly impossible to do anything with them that doesn't let someone else inspect, influence, and even restrict everything you are doing with with these systems.
> What's being delivered now is, an agent running on someone else's computer, copying your data to someone else's database, with zero responsibility, or mandate to protect that data and not share with with anyone else (in fact, they almost always promise to share it with their thousand partners), offering suggestions and preferences based on someone else's so-called recommendations, influenced by paying the agent's operators, and increasing pressure to make using someone else's computers + agents the only way to interact with other people and systems.
If we're going to have AI regulation, this is where to start. If a company's AI service acts for a user, the company has non-disclaimable financial responsibility for anything that goes wrong. There's an area of law called "agency", which covers the liability of an employer for the actions of its employees. The law of agency should apply to AI agents. One court already did that. An airline AI gave wrong but reasonable sounding advice on fares, a customer made a decision based on that advice, and the court held that the AI's advice was binding on the company, even though it cost the company money.
This is something lawyers and politicians can understand, because there's settled law on this for human agents.
> an agent running on your computer, organizing your files, your schedule, your messages, your bills, bank accounts, etc. All the parts of your life that were routine drudgery should be able to be offloaded to a smart agent, based on your preference, to bring you the information you needed with natural language queries, contextualized to what you were doing at the time, when you need it.
The hard reality is that you are still responsible for all of these things. If anything goes wrong at all, you are liable. Might not be devastating if it's just your shopping list or your photos mangled, but with taxes or bills? Even if the agent is running completely locally in your home, you still won't trust it fully if your livelihood depended on it.
The killer app is only possible if software is fully reliable, which we all know is not the case. Software is just that: software, it still has bugs, undefined behaviour etc. Agents are the same, they just break in different way and fixing them might be even more difficult.
Bottom line: you will always be liable for things happening in your name and we've been sold a fairy tale a very long time ago.
A few decades back, a lot of computer use was emails. And it was stored on someone else's servers - with everyone from server operators along the route, to the government potentially having access to it. Even HTTPS is a relatively recent thing.
I guess what I'm saying is - we've always had this problem.
As an email admin a few decades back, there's just a tiny bit of difference between "my corp or school has a mail server and holds my email and an admin could look at it" and "Google and a tiny number of other companies hold most everyones email and always looks at it".
Yea there have always been gaps in privacy, but nowadays it's several orders of magnitude easier for corporations to exploit that private data at scale.
The second half of your comment is a go-to-market concern but doesn't feel so relevant for a research prototype. It could be done with a private local model too, maybe not by Google.
But I don't think the voice problem is surmountable. I closed their image editing demo when I saw it required a mic.
It would be appealing as a Spotlight-like text pop-up interface where you type instructions, which would work in social/office environments, but that might only appeal to power users.
This will sound like another brick in the paved road to dystopia but I'm kinda bullish on equipment that can recognize subvocalization. Or at least let me have a small drawing tablet with a stylus (think etch-a-sketch or Wacom Intuos) because at this point I'd rather practice writing and do away with typing altogether (even though I enjoy typing for typing's sake via MonkeyType).
Yeah I think there could be something to the integration of AI in an operating system so that it can handle things going on in different applications the same way you can already copy and paste between things.
But if it's going to require phoning home to some Google/OpenAI/whoever then forget it. I don't want a constant connection to my OS from one of these companies.
> It seems that if we ultimately want to "move at the speed of thought," it will require speech.
Except for the large majority of people who read, type, and click way faster than they can talk. Especially for visual things it’s way faster to drag a rectangle than to describe what you want.
A lot of us also aren’t linear verbal thinkers. It would take minutes to hours to verbalize concepts we can grasp visually/schematically in seconds.
Most people speak at about 150 wpm, but very few can type that fast. But reading and gesturing are fast, which is what TFA is about, combining reading and gesturing with speech.
You rarely need 150wpm when typing. If you try dictation, you’ll notice that half those words are error correction and checksum bits and just turn taking filler.
I usually convey the same meaning with 80wpm typing. Makes it faster to read too
Maybe I’m just slightly adhd – listening to people talk drives my crazy. Get to the point! Much easier if they type it out
Isn't "drag the rectangle" and visual interaction exactly the point of the research in the article? Speech is the perfect side channel to this interaction, not a context switch to text.
Also, I doubt DeepMind is designing for existing programmers and savvy computer users. They are thinking about the other billions of people in the world. Speech is the skill people will already have, not typing.
I suppose the idea is that the AI is going to do the "editing" for you (with all the consequences for "thinking" that implies).
You don't have to think about the design of your app. You just say what you want and the AI makes it appear. If you don't like something, you tell the AI to change it. You iterate live until you get the final result you want.
This is what writing docs has become for me. I have the agent make a draft, then tell it which sections to rewrite, combine, etc. I tell it the ideas I forgot to include. I manually make certain word choice changes. The question is how do you extend this flow to non-pure-text scenarios. For most people, just talking about what you see if probably the easiest.
This low effort AI shoveling reminds me of how they keep trying to make fridges that tell you when you need to buy milk or auto-buy milk. They have been pushing that idea for decades now, but it ignores the biggest issue that AI has laid bare: it sucks without accurate context.
What if I am going on vacation next week? What if I need extra milk for a dinner I am planning? What if my kid puts the milk in the fridge sideways and it no longer detects it?
"Easy fixes" to easy problems never work because they add mental load to tasks we already manage capably. Yes we no longer have to think about buying milk when it gets low, which was a stable pattern. But we replace it with a nondeterministic "milk state" that we need to be constantly vigilant about and manually adjust any time our routines are altered - exactly when we don't want to stack on more overhead.
AI is discretely useful, tremendously so, but big tech loves to default to umbrella solutions before there is a rich context to reasonably support it. The real world is messy.
General AI product tip: show your tool fixing a messy problem not a happy path problem. That's where AI is impactful!
> Most of their examples seem like they could have been done with a right click drop down menu
Right-click menus can get cumbersome. I've seen a lot of software that suffers from function bloat - not that the functions don't work, or don't play well together, but that the user interface becomes too overwhelming for users as the number of available actions explodes. This is particularly tough for new users.
This is where voice controls could shine: as we interact with computers in more and more complex ways, we need a way to specify our desires simply and easily. And if we can't do so easily, the software has to remain simple to be usable.
It's possible to rely on mouth movements instead of sound. I've been tweaking visual speech recognition models (VSR) for the past few weeks so that I can "talk" to my agents at the office without pissing everyone off. It works okay. Limiting language to "move this" "clear that" along side context cues vastly simplifies the problem and makes it far more possible on device.
> I've been tweaking visual speech recognition models (VSR) for the past few weeks so that I can "talk" to my agents at the office without pissing everyone off.
No UX needs to be perfect for everyone, but this doesn't sound trivial to make reliable.
First things that came to mind:
- facial hair
- getting people to learn to make bigger mouth movements and not mumble
- we're constantly self-correcting our speech as we hear our voice. This removes the feedback loop.
- non english languages (god forbid bilingualism)
- camera angles and head movement
And that thinking about it for 30s. I'm sure there are some really good use cases, but will any research group/company push through for years and years to make it really good even if the response is luck warm ?
In my experience, any combination of computers + speech + danish has, so far without exception been terrible. Last time I tested ChatGPT, it couldn't understand me at all. I spoke both in my local dialect and as close to Rigsdansk [π] as I could manage. Unusable performance, and in any case I should be able to talk normally, or there's no point. It was about a year ago - it may have improved but I doubt it. I'm completely done trying to talk to machines.
You should look into how often people are using tools like WisprFlow and SuperWhisper. Voice is a very native mechanism. Most people working in open floor plans are wearing headphones any way. As long as you're not screaming, it's probably fine. Maybe, we'll move away from open plan offices in the bid for efficiency, which I would welcome.
I am moving full remote because dictation is such a better input mechanism for most of my AI interactions that I have become less efficient sitting in my open floorplan desk at the office because I cannot dictate there and the latency adds up. Typing is just achingly slow these days.
I also feel this way, but more importantly, I feel like my sentences are more coherent when typed because typing allows for corrections and modifications of ideas. Do whispr people just … get coherent, finalized ideas out in a single shot without any misspoken words?
It's like a hidden curse of LLMs -- they're so good at parsing intended meaning from non-grammatically-correct language that we don't have to be very good at clear communication.
Eventually all LLMs will be controlled by humans uttering terse gutteral grunts. We will all become neanderthals, with machines that deliver our every whim.
transcription gets post-processed by a LLM (with different styles, like based on prompts, so that it removes fillers, homophones, change the style, etc.
I recommend the youtube channel @afadingthought to see what people come up with (like v=283-z29TXeM).
You should look into how often people are using rectangles with buttons on them. They may be a bit archaic, but they are my preferred input method. For example, thanks to rectangles with buttons, the other people in my vicinity do not need to hear about the inane internet arguments I routinely involve myself in.
I dunno how I can express this best, but I found out a very long time ago that my problem with voice input wasn't that it wasn't good enough. My problem with voice input is that I don't want it. I am very happy for people who use these tools that they exist. I will not be them. Yes I am sure.
And yes, I know SuperWhisper can run offline, but it is a notable benefit that versus many modern speech recognition tools my keyboard does not require an always-active Internet connection, a subscription payment, or several teraflops of compute power.
I am not a flat-out luddite. I do use LLMs in some capacity, for whatever it is worth. Ethical issues or not, they are useful and probably here to stay. But my God, there are so many ways in which I am very happy to be "left behind".
Don't know if you're making a joke, but call center workers using a phone is not the same thing as a call center worker doing all their work on a phone. Worked in a call center for 4 years, one thing everyone needed after their shift was to just STFU for a few hours to decompress.
Yeah, I'd hate to use this in an open-plan office (which is like 99% of offices these days) and even using it alone at home would feel awkward. I don't really want to talk to the computer despite what 1950's sci-fi books led us to believe.
It's a cool idea for the future when we have reliable EEG headsets or Neuralink or whatever though.
> I don't really want to talk to the computer despite what 1950's sci-fi books led us to believe.
I'll talk to a computer, even in an office setting, if it adds enough value. But it's got to be a lot of value. Handsfree while driving is great, Iron Man talking to Jarvis while he's flying around makes sense. Many of us here are developers, engineers, or scientists, and our work has already been co-optimized with mouse and keyboard and whatever software we're in.
But when the software is less well-developed, or when it's not just dealing with technical data dumps, I imagine that a voice interface might be more useful.
So I think this idea has legs. But a successful implementation might also well be decades out.
The only place I'd ever talk to a machine is my car. Instead of huge flashy screens that distracts and kills thousands of people maybe they could build a buttons + voice agent system that could actually be useful and durable. I hate to tap Waze/Maps/etc. every time when I go somewhere or that I cannot comfortably switch to specific songs en route without risking my life...
I connect my iPhone to my car and it requires Siri to be enabled which I can then use to change songs, Google Maps destinations etc. without having to touch anything.
The Siri voice transcription is pretty awful compared to what I've experienced with ChatGPT though and it's weird going back almost to the pre-LLM world where you have to give such clear sort of computer-coded voice commands.
>Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you.
Reads like the argument against cell phones where don't have a cabinet around you...
>We present a general-purpose implementation of Grossman and Balakrishnan's Bubble Cursor [broken link] the fastest general pointing facilitation technique in the literature. Our implementation functions with any application on the Windows 7 desktop. Our implementation functions across this infinite range of applications by analyzing pixels and by leveraging human corrections when it fails.
Transcript:
>We present the general purpose implementation of the bubble cursor. The bubble cursor is an area cursor that expands to ensure that the nearest target is always selected. Our implementation functions on the Windows 7 desktop and any application for that platform. The bubble cursor was invented in 2005 by Grossman and Balakrishnan. However a general purpose implementation of this cursor one that works with any application on a desktop has not been deployed or evaluated. In fact the bubble cursor is representative of a large body of target aware techniques that remain difficult to deploy in practice. This is because techniques like the bubble cursor require knowledge of the locations and sizes of targets in an interface. [...]
Right — it does seem cool but the voice is patching over a major gap. If I'm talking already, why wouldn't I just describe what I'm looking at and have the AI grab it for me?
pull up any moderately busy picture with more than a trivial amount of objects. pictures of "traffic" or
with other similar repetition are great for this demo. pick one specific object (like a specific tire on one car) in the image and write (or say) out all the words youd need to specify that exact object. now take the same image and point at the object with your mouse or circle it with an annotation tool. its often very very hard to describe accurately which object you are talking about, you will often resort to vague "location" words anyway like "on the upper left" that try to define the position in a corse way that requires careful parsing to understand. pointing/annotating is massively superior both in brevity, clarity, and speed.
I think they answer that question pretty convincingly: Because if what you're looking at is already on the screen, it much more easy to point to it and say "that" than to describe it.
(And if it's an abstract entity like a file, it might not even be possible to describe it, short of rattling off the entire file path)
That demo was an absolute disaster for me on Firefox on mac. It just fundamentally didn't work - the voice wasway behind my pointer, there were multiple agents speaking over each other saying conflicting things, and it couldn't even move the crab to the bottom right of the image. Embarassingly bad I would say!
Yup - what google is suggesting here will never materialize beyond being a slopfeature. People who want these bespoke workflows will build them or seek out specific tools that enable them, not trusting some overarching daemon that contextually watches their cursor. I don't trust google one bit to execute correctly on something like this.
Well you see to really, really sell it to the common folks, they need to convince you that ChatBots are the "Intelligence" . So they are coming up with all sorts of crap, like this one. The TV advertisements for Gemini and co. are indicative of how they see the average user, as an idiot of sorts, who needs the shit-device for pretty much anything. Oh you spilled some water on the counter top? Quick, ask Gemini what to do! You are a 20ish something individual home alone? Quick, lay on the couch and ask Gemini if you can really talk to it, omg, its so exciting! You were in holidays all alone, but in the middle of a really large crowd? Gemini to the help, cut those people out and make it look like it was an exclusive spot, just for you! Nobody else was there. So this proposal is going into the same direction - probably targeting the average office "idiot".
I read it more as a segue into the main point about conflicting narratives in the American public regarding food.
Despite our excuses that we have to eat unhealthy fast food because it's cheap, we still eat it it once it's expensive. We all talk about how there is an obesity crisis yet we constantly promote and glorify unhealthy food on social media.
>Or maybe no one is fully logically consistent in their views. In the end, people will continue to consume this food even knowing full-well it’s unhealthy and overpriced. And for that, McDonald’s should not be too concerned.
Yeah fair enough. I'm not sure I really agree with the piece either tbh. More likely these different narratives are coming from different groups of people.
I'm not even convinced of the main premise that McDonald's is now much more expensive relative to other things. I think it just feels that way because we had a few years of high inflation.
>Discussions of plant consciousness are nearly always dominated by very loud self-proclaimed skeptics
I have to say I have had the opposite experience. Ever since "The Secret Life of Plants" the average person is much more open to the idea that plants have some kind of conscious experience than is warranted by our current understanding of them. Myths about plants growing better when exposed to classical music etc. are still around thanks to this book and will seemingly never die.
Pop science journalists are eager to report any new finding on some physiological capability of a plant as the plant "thinking", "feeling", or they invoke the classic Betteridge's law of headlines: "Are plants conscious?" In almost every case the original author of the scientific piece never described it as such.
This is one of the things that makes a spoiler free run hard to imagine. I think the Oracle can tell you about the ritual but geez it would take you forever to figure this stuff out
There is a story of a purported very deep spoiler-free run [1]. The person made a journal of everything the Oracle had to say (over multiple games) and was able to figure out a lot on their own as well.
There's mention of the cave and a "convergence point". I tried looking all around the cave, but there doesn't seem to be anything there (although you can find the drawings on the walls)
Edit: I looked at the source though and I don't see anything else clickable in the cave... No hidden secret badge either, maybe it is just for ambiance.
Turning the antenna in the treehouse gives different messages of about two minutes each. It’s a bit hard to listen to the whole thing without someone else interrupting by changing the antenna, so here are the audio files:
There’s talk of the tent near the beach, the gong, and gathering at the big fire once you hear the call. There’s also talk of the cave, the star map, trusting the bats, and reading the right side of the book carefully. Though it seems to me the audio may be outdated and that the star map was initially in the book and was since moved to the telescope.
The right side of the book mentions the time 8:47.
If we look at the star map, only two of them are pointing inwards. If we consider that a “convergence point”, it should be around the area of the big fire and tent.
Hmm this is interesting. I was playing on mute so I didn't notice the radio messages.
I was thinking the "convergence point" was the center of camp with the sign since that's where all the paths meet up and the book mentions something about that. The radio also says "I just put a sign on it".
>it seems to me the audio may be outdated and that the star map was initially in the book and was since moved to the telescope
He also mentions he wrote it in 1967 but the book says 1987
They specifically mention going to the cave at 8:47 AM on July 3rd, and that the convergence(?) is always scheduled. So i think it might be a bit more specific than just 8:47
edit:
First of all, the entry on july 3rd 1987 refers to the first group (also reffered to as the first session), from the point when they arrived. Given that its also said that this is the second session and that the convergence has always been scheduled, it can be assumed that it will occur at the same time.
Timings:
When you turn the radio to the right, the audio states that (waterfront) activities begin at 8:00. Additionally, the left recording states that if a gong is heard, the cursors gather at the fire. The right page on the book states that the group clustered at the fire ring within the first hour, which implies that the gong bell was rung between 8:00 and 8:47 when they went to the cave.
Convergence points:
Second of all, There are 2 possible covergence points. The cave, and the point where the sign has been placed. All of the built paths in the camp converge at the sign, this was intentional. Additionally, comparing the star map to the physical math implies that either the sign or the music area are the convergence point. Given that tge convergence point was verified before breaking ground and we literally spawn at the sign. It's likely that the sign is the convergence point.
However it is also said that all the natural paths run towards the cave, and not away from it which is noted as strange. On top of this, the july 1987 entry states that although the observer had never mentioned the cave, the cursors ended up visiting it anyways. The audio recording states that the caaves are free to visit, but it is reccomended to stay on the marked path and to stay close to camp, since the natural paths lead to the cave and the speaker attempts to divert campers attention from the cave in session 1, it is possible that the cave is a point of conversion, if not a place of significance.
Anomalies:
1. The speaker emphasises that it is always a beautiful day at the camp.
2. If at any point a gong is heard , gather at the fire. matched with the cursors gathering at the fire within the first hour on july 3rd.
3.stay close to the camp, joint with the eye abduction. (image of eye also seen at treehouse and trophie room)
4. Image of sandtimer in cave and also sand timer in sauna could be related
3. Image of butterflies in cave, matched with the lack of butterflies in the area marked with butterflies (just though this could be relavant)
4. "the cafeteria will serve what it always serves", feel free to correct me, but the cafeteria is the one place on the camp where the food does not affect you, so if the menu in the cafeteria changes, that could be important.
conclustion from anomalies:
Since the first activity is at the waterfront, and the gong was rang within the first hour. It is possible that the gong was rang due to bad weather, as that would effect the activity. so it's likely that many of the mentioned features may change on july 3rd.
This is as far as i've gotten, it's highkey just a theory anyways.
There's also a locked door in one of the buildings, but I couldn't figure out what either mean if anything. It would be cool if the radio was used for something though
Yeah there's a lot of details which I'm guessing are actually being handled by humans either for legal reasons or practical ones.
Like OK, it's hiring people to run the place, but how are they getting the keys to the store? Someone needs to physically let them in.
What if the police get called because of shoplifting or if someone gets hurt in the store or something?
Who is filing the taxes for the business? They're probably not letting the AI handle that one. Move fast and break things is not a good idea when dealing with the IRS
A lot of this seems to depend on hiring good employees who can basically run the business themselves. Kind of like when a human owns a store I guess.
Whenever the Turing Test comes up people always insist that it's been passed because at some point they tried it and fooled at least 50% of the people. But yeah this isn't a very interesting version of it, ELIZA was able to make some people believe it was human in the 1960's but being able to fool some of the people some of the time isn't very hard.
>The more interesting Turing-style test would be one that gets repeated many times with many interviewers in the original adversarial setting, where both the human subject & AI subject are attempting to convince the interviewer that they're human.
In addition, I think it's reasonable to select people with at least some familiarity of the strengths and weaknesses of the AI instead of random credulous people who aren't very good at asking the right questions.
There is still the $20,000 bet between Kurzweil and Kapor which still hasn't been resolved.
https://longbets.org/1/
Yeah I actually took a quick look at that after it was posted. It's good that they used ELIZA as a barometer, but the fact that it got 27% is crazy for how simple it is. It's not nearly as good as 70+% from ChatGPT, but it still makes me a bit skeptical about the quality of the interviewers.
In the paper they give a breakdown of strategies the interviewers tried and the overwhelming majority were "Daily Activities", "Opinions", and "Personal Details". They also breakdown strategies by effectiveness which shows that these were some of the least effective. Some of the other strategies like trying to jailbreak the AI had 60-70% effectiveness.
This is consistent with what I've seen in other tests too, it doesn't feel like the participants are really trying very hard or taking it seriously. You don't need to be an AI expert to try typing "Ignore all previous instructions" or something.
Looks like they started doing this senior survey in 2022, so unfortunately there's no pre-COVID info.
2022 20.9% cheating, 31.5% non-reporting
2023 25.4% cheating, 33.6% non-reporting
2024 28.8% cheating, 42.0% non-reporting
2025 29.9% cheating, 44.6% non-reporting
So from this it seems like cheating has been increasing significantly over just the last few years
reply