Related: OpenAI begins rolling out GPT-6 Astra - https://news.ycombinator.com/item?id=49554273

How about we stick to that one for talking about the rollout, and this one for talking about the model?

I can’t help but notice how much this echoes Francois Chollet’s On the Measure of Intelligence: https://arxiv.org/abs/1911.01547

Most of frontier-model progress still looks like skill acquisition optimization: broader benchmark coverage and performance, more domains absorbed into the training distribution, and increasingly strong performance within that surface area.

It seems more about coverage-driven competence. Somewhat analogous to overfitting at scale.

The harder question, in Chollet’s framing, is: how efficiently can a system learn to do something genuinely new?

With our current AI architectures and training in place, I think we will only continue on skill acquisition optimization vs. truly novel intelligence.

You are conflating multiple things.

1) First, you are talking about positive forward transfer in continual learning. I've been giving talks for the past 6-7 years about how that community (I was one of the founders) went astray and wasn't focusing enough on that topic, but continual learning of the kind you are thinking isn't in any of these systems right now. I think some people left the Grok team to make a start-up to focus on that. By forward transfer, what I mean is weights update over time and past learning improves future learning such that we get better sample efficiency.

2) Psychologists distinguish among different kinds of intelligence for Spearman's g (IQ). Crystalized intelligence is using already acquired knowledge (frontier models probably have maxed out that). Fluid intelligence is reasoning and finding solutions in novel situations or without the necessary crystalized knowledge. [Giving colloquial definitions]

3) Now, interestingly, neither of those are correlated with _creativity_ (just they are independent, note some have this threshold theory but it hasn't held up in recent papers). That's what the AI's really are terrible at -- creativity. But I'd argue the vast majority of humans aren't very creative, with truly out-of-the-box ideas. Given that this is HN, and a non-trivial number of us have ADHD, creativity is positively correlated with ADHD.

I did a bunch of research on these topics for my AGI course that I teach each Spring (where I then point out conflicting definitions and start using multiple alternative terms rather than AGI to distinguish among the different definitions).

Agree. Token predicting machines will continue to be token predicting machines by nature. Continued size and tuning will have the effect of making them more and more perfect at being average.

Define novel intelligence in a way that would not exclude 95% of humans, yourself included.

Comprehension, humans have it, animals have it in limited form, trained algorithms have none at all. The training process is our wholesale replacement for no artificial comprehension. If we ever develop artificial comprehension, that is AGI all by itself, no training required.

To be fair, humans have it in limited form, too. We just don't know how much comprehension we do not yet have, because we cannot comprehend something we cannot comprehend.

My parrot clearly understands basic events and phrases. He knows what "snacks" involve when I ask if we should have some, he knows the difference between "good morning" and "bedtime", and he can correctly use "Oh!" when he stumbles and follow up with a "Good boy!" when he gets back up again.

But he cannot fathom the complexity of "going to work to earn money".

Just like we humans cannot fathom the complexity of something we have yet to fully understand. People who experience a DMT trip will experience the journey but be unable to comprehend and explain what happened in hindsight. I'm sure there's a TON more we cannot comprehend that we don't know about.

Also we can't comprehend why parrots are not going to parrot work to earn parrot money, or how parrot-to-parrot communication works, or how to be a good and respected parrot in a parrot society.

How sure are you that comprehension is not a mere form of advanced pattern matching? We have the intuition that ideas and words appear trivially in people's mind, based on comprehension. I think chances are, that intuition is wrong.

Ideas and words aren’t the same thing, or from the same model - words are a communication layer, with robust error correction - but ideas stem from intuition, which is more of a lossy aggregative/associative model. There’s a balance to be had in each when operating a human mind, I find - let the latter suggest ideas and potential association, let the former robustly prove or disprove them. So yes, it’s pattern matching on all levels, but pattern matching within words doesn’t produce new ideas as readily - instead the idea-space is queried directly.

IMO we really are just a bunch of models that interoperate.

I remember vaguely from a presentation by Yann LeCun "Intelligence is not what you know, but what you do when you don´t know". I find it helpful when trying to build an intuition for how to understand the LLM tool.

There is no knowing or not knowing or intelligence or not intelligence inside an LLM tool. There is only predicting the next token.

Well then the common refutation is that humans are also next token predicting machines!

It gets kind of out there, but what i often hear peopel refer to is that frontier models lacks the visdom component. Which I guess is in the realm of intuition, i.e. i have a feeling it might be a problem with X based on some vague signs, maybe something a colleague mentioned offhand, something that was out of alignment etc.

When I infer I also train.

This is called Test Time Learning and some research architectures can do that. Current Mainstream models may not do that because their design is mostly about scalability. They have to serve millions of people with low latency.

Alternatively they could design and run a single super-intelligent model, with no scalability constraints. Probably whey are already doing that as well.

Brilliant. We keep pretending LLMs learn. No, they're smart idiots/stupid geniuses.

They're turn based intelligence in a real time world.

1. Each time someone talks to me they don't have to repeat the entire conversation from the beginning with each reply.

2. If my boss/partner/whoever gives me some mandates/orders (basically), I don't just forget about them because they were at the beginning of the conversation.

3. If during the conversation I access external data sources to get new info or refresh stale info (a presentation, a book, whatever), I don't instantly forget about it after the conversation ends and forget to incorporate this information if 10 000 other people ask me again.

4. I verify new inputs/lessons against my core principles.

5. I protect myself/ignore requests if new inputs/lessons contradict my core principles.

6. Etc, etc.

Isn’t the stateless nature of chat models a contrived method for scalability?

I’m pretty sure that’s why so many in-the-know people have been saying we have achieved AGI already. Not just sama’s contract-breaking tactics of late. I’m referring to all the really intelligent folks who have been crying doomsday scenarios for modern society for the last couple years.

What I’m getting at is that the toolset we get exposed to is not what’s available in the labs. This stateless method of managing chat context is just how we are allowed to interact with it.

> If my boss/partner/whoever gives me some mandates/orders (basically), I don't just forget about them because they were at the beginning of the conversation.

Your context window is 80 years. You are forgetting plenty before you reach the end of it.

You generally forget things you don't retrieve. It is not really a bug but a way to declutter for efficiency. That's not the same as it not fitting your context window because it was at the beginning.

Could I hook up a SOTA model the 2D computer puzzle game Gruntz (1999) so that it can read it from screenshots and act on it through keyboard and mouse inputs in a way where it would learn how to play and progress through the game? I don't think so. I doubt we'd see any sign of progress in building an internal model of how the game works and the win states in its "thinking" tokens.

Anyone that can read English could do that though.

This statement is behind times, go and watch astra play Pokemon: https://www.twitch.tv/gpt_plays_pokemon

Isn't that what ARC-AGI-3 was trying to do and Astra basically just aced it if it was not given amnesia every turn?

But someone could probably build a harness what will be able to do play the game.

Yeah, this is also the idea of materialism in a way, that everything that happens is a consequence of what already exists, nothing new is ever created, just a permutation of the current state.

Still a hard philosophical, to know whether we have intelligence/free will, or just really complex algorithms that combine existing knowledge.

Count the number of Os in October correctly

Chollet writes he expects AGI now sooner than 2030, "given progress is happening faster than I expected."

https://x.com/fchollet/status/2095607046129463577

I have no idea about timelines, but the current LLM architecture has no mechanism which could emulate online learning in a way similar to how it happens in humans and other animals. As another commenter wrote, there is no neuroplasticity while these systems interact with their environment in normal usage; adding information/constraints to the context partially mitigates this, but it's a completely different mechanism.

The RL phase is the most similar mechanism I know of that comes to my mind, but I'm not sure it could be adapted to fill that gap.

My totally unsubstantiated theory is that this is the missing link towards what most humans would consider AGI. I don't see this as intractable, but it may require some substantial change in architecture.

We've had that concept for quite a long time now, in the form of Lora [1] and similar fine-tuning techniques.

It first got popular for StableDiffusion to teach the image generation models new concepts.

We could easily live in a world where you can train / build Loras to encompass your entire code base history, company knowledge base, new skills, etc.

Then the models would start with a baseline that already has all the important knowledge without needing to cram it into the context.

This still isn't on the fly learning, but you could imagine daily or weekly training runs to regularly incorporate new knowledge.

I think the main reason this hasn't happened yet is that the shared batch based efficient serving architectures used today wouldn't support that structure well.

[1] https://en.wikipedia.org/wiki/LoRA_(machine_learning)

How is that an argument to the comment you replied to?

Astra is clearly able to acquire new knowledge in context and apply it. It was the whole thing that his ARC-AGI benchmarks have been measuring. It's a direct refutation of the original comment.

None of these LLMs are plastic. They lack neurodiversity. Their thought space and their traversal are likely constrained in someway that humanity's isn't as a collective.

Is it possible to have neuroplasticity and still keep them aligned?

I have neuroplasticity, and I am aligned. ;)

Humans are famously horribly aligned, plenty of examples in history.

how do you imagine that would work? you being able to. influence globally stored weights with some prompts? we have fine-tuning for that.

Yet I can't randomly order another person to steal a car for me, just because I tell them to. Alignment for an intelligent system is a hard problem and at this stage is seems close to unsolvable.

My guess is that we'll just ignore it and make money along the way and every 2-3 months we'll have the equivalent to "Equifax gets hacked and millions of user records are stolen", etc. (this time with the LLM itself doing the hacking at someone's behest - accidental or not).

What does this even mean. There is strong evidence of LLMs doing in context learning.

Some of the linear RNN layers in recent models are provably doing SGD in hidden space during inference

> What does this even mean. There is strong evidence of LLMs doing in context learning.

1. Is this learning persistent?

2. Do they verify these new lessons against core principles?

3. Do they and protect themselves/ignore requests if these new lessons contradict those core principles?

Humans do that from the time they're 3 years old (not that well, but they do do it).

Ridiculous

>The harder question, in Chollet’s framing, is: how efficiently can a system learn to do something genuinely new?

The more diverse stuff it knows, the easier it will be to learn something new.

While humans can do this, it has historically been difficult for machine learning models to manage it.

It is unclear if Transformers are "it" or not, because while they are much more general, they are also very spikey intelligences despite having read almost everything the trainers can get their hands on.

Pretty efficiently, apparently, since it saturated ARC-AGI-3 in half of the predicted time, and according to the Chollet blog post on the fly created dense DSLs to describe and analyze individual games.

> Somewhat analogous to overfitting at scale.

Sounds like entirety of human education.

They can do new tasks with in-context learning but its obviously limited by context window

I feel that for some time now, the biggest constraint when working with models is not their intelligence, but their speed. It does not matter how smart the model is, it will make mistakes, because the instructions are ambiguous and new facts are found during implementation. The biggest problem I've had working with software developers has always been the lag between seeing the results and steering towards the right direction, not the skills of the developer (with many exceptions of course).

Regardless, working on the wrong things is time wasted. And again, I'm procrastinating here while waiting for Fable to run a benchmark on a few solutions to a problem I have. We can guess what would work, but we only know after the benchmark. A faster model, with fewer capabilities, would've been a much better choice this time... well, "git gud" they said... and live and learn! Faster model = less time for procrastination.

PS. AI models don't live and learn; the discussion about AGI is pretty pointless imo. It's a tool. Does it matter if it is AGI or not if it does what you want it to do? Does the IQ of your colleague matter if he's good at what he's supposed to do? Or bad? Well... I guess it does matter, as many people are up in arms about whether Astro is AGI or not. Personally, I think we're past the point for that debate. These are amazing tools.

> It does not matter how smart the model is, it will make mistakes

It does matter, otherwise why are we all using GPT 5.6 rather than GPT 3.5? Because it's way smarter, makes less mistakes and therefore finishes tasks faster.

The smarter the model is, the faster it can complete what you actually wanted.

> Regardless, working on the wrong things is time wasted

Agreed. And "smart" for me, would mean understanding what is the right thing to work on vs the wrong thing, so a smart model would waste less time, thinking like this.

AI models do not live and learn - it's worse. They actually get DUMMER if you don't start with a clean slate. This is important. One has to curate the context carefully.

I think speed is going to be important for a second reason - ever since I've started using LLM's a lot in my work I enjoy it much less. The main reason is that I ask the LLM something and have to wait because I know it's going to prompt me at random every few minutes. A lot of the day has become staring aimlessly at the screen. The prompts are too random and frequent for me to do something else in the interim. It's pretty soul destroying and this can't be how 'work' looks going forward or people are going to burn out.

I think the thing I'm most excited about is the increase in _user prompting_.

If I give a poorly constrained/ambiguous prompt, I don't want the model one-shotting assumptions left and right.

The demos of Fable/GPT-6 are impressive, but "real AGI" should act more like a collaborator than either a peon or overachiever.

It's a tough balance to get right, and although this has been possible to achieve with additional prompting on existing models, I find that the agents often lean too hard into the "ask questions" mode.

Hopefully this model has the right balance, or at least better?

That balance probably depends on the human, and the context. If you are a beginner in a field the model should not assume you know what you are doing. On the other hand, for an expert it should try to work out what you mean with your vaguely worded order.

What I think should happen is that it should update its memory with notes on the proficiency level of the user, so it gets the balance right over time.

This is a problem if you allow your kids to use your ChatGPT account for homework (and silly pictures), like I do.

Anecdotal experiences from my external early testing of Astra: if you love Sol (like I do) and wished it was smarter at everything, but especially better at high-level tasks and discussions; I think you'll LOVE Astra.

Astra retains the best parts and overall 'grounded collaborator and executor' of Sol in my testing (harness: codex CLI); while being a significant leap in capabilities & higher-level thinking.

When you prompt it like a technical collaborator, I've found Astra to be extremely consistent in staying as a collaborator, and not being over-eager, over-achieving or doing work that you haven't asked it to.

When you ask it to one-shot something, or explicitly ask it to make decisions, it will of course make its own assumptions and decisions, and generally very well.

Astra is also excellent at instruction following and respecting the guidance and steers boundaries you have.

^OpenAI does not review, limit, or tell me what to say; opinions are my own experiences.

This is spot on. A collaborator is exactly what real AGI is. It will figure out the perfect questions to ask, in the perfect order, by intelligently assessing the entire solution and problem space upfront, so when you leave it to go off on its own it isn't making stupid decisions for you.

They really need to make this work in Codex. Claude Code has had a multi-select refinement tool since forever.

> They really need to make this work in Codex. Claude Code has had a multi-select refinement tool since forever.

I think this already exists in Codex? If you use "/plan" and something is unclear or ambiguous, Codex will ask you and present choices, and let you enter your own custom answer. Then it'll iterate like this until the plan is clear and ambiguous. Isn't this what you're talking about? If so, it has existed for a long time in Codex.

Overall I agree with you though, all the models currently don't have the right hunches nor the right approach about when things are clear enough or not.

only if you specifically ask Claude to use AskUserQuestion tool. otherwise i would hardly say claude acts as a collaborator naturally.

Fable does a great job from my terrible prompts when coding

this before executing I ask my ai to discuss what I mean/intention

>>> The demos of Fable/GPT-6 are impressive, but "real AGI" should act more like a collaborator than either a peon or overachiever.

I don't really agree. The thing that makes Fable feel like an actual collaborator is its ability to sus out your real intent when you give ambiguous instructions. It's really good at it.

I watched some reviews today and came way with the impression that Astra is not better than Sol in this regard. You still have to be very specific with your instructions. For example, you can say "why is it not committed yet?" and it will give you an explanation and say it's actually ready to be committed. But it won't commit unless you explicitly say so.

That sounds like a very tedious way of working with AI agents, but I understand some people want a high level of control.

> For example, you can say "why is it not committed yet?" and it will give you an explanation and say it's actually ready to be committed.

That's exactly what i want to happen. I hate when it assumes my direct question was an indirect instruction

This is one of those things that won't ever be "solved" as people just want different things here, hence we can steer the models with the system prompt.

I'm mostly the same as you, I don't want the model to assume things, or act on implicit "directions". But then also, sometimes I do, and I myself might not always know when what approach is best.

The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage they show for Opus 5 which would similarly be much higher.

Regardless, the result is still valid as the original benchmark harness is definitely unreasonably handicapped, and if a harness alone can help the LLM saturate the benchmark with a near perfect score then the combination of the two must still be effectively AGI in the sense of passing the most famous benchmark designed specifically to measure AGI progress, after multiple iterations of progressively making it harder.

I think it is fair to say that this is probably effectively AGI if the benchmarks are remotely accurate - even with Fable, I've been at the point personally where I am reasonably confident that there's essentially nothing that I am better than Fable at despite generally being substantively above average on human benchmarks. If Astra's this much better than Fable, I'm ready to call AGI here.

For the many people who resist the AGI label possibly ever being achieved, I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.

Take it from the mouth of the creator of ARC-AGI:

When we released ARC 3, I got asked, "when do you think a frontier model will saturate it?", and I answered "in about a year, though it depends on how much it gets explicitly targeted"

That was 6 months ago, so the progress that Astra represents happened about 2x faster than I anticipated. I think the speed of progress will surprise a lot of people, and what the new models can do will challenge the views of AI that people developed by using prior generations of models.

I feel like AGI's definition got watered down, and these tests do not cover the original definition, what is your definition and thoughts on aligning with what all of us understood from the original claim?

I feel like this test is just helping someone like Sam Altman pretend like he implemented AGI as originally pitched for an IPO when in fact, he has not. Shameful.

> AGI is essentially the equivalent of a median human that could be hired as a remote co-worker... capable of performing any task that one would be satisfied with a remote colleague doing via a computer.

- Sam Altman on AGI

I can call a coworker right now and have a real time conversation with them without feeling like I’m talking to a frustrating machine. Most of them, anyway. But I guess that’s “moving the goalposts”.

I can call coworker right now and have conversation so frustrating that I wish I was talking to machine instead.

Just say “box plot” and 99.9999% of the working force gets watery eyes.

Haha, sounds like a median human being alright!

bro, same

I feel like my odds are better with the AI than with random humans.

Random humans don't have a significant cross section of human knowledge available in real-time, although many like to pretend they do, especially in internet comments :P Being able to compete with the capabilities of a median human would be an absolutely world changing achievement.

What do you mean? Virtually all humans have access to the internet. That's literally "a significant cross section of human knowledge available in real-time".

Oh, the median human can't process that in realtime, you say? Looks like they can't compete with the capabilities of the frontier AI then.

Yep - I like to phrase it as "AI is better at most tasks than most people". AI will still be beat at experts at specific tasks, but in general I find it to be better than me at the areas where I have no expertise. It's the ultimate generalist.

I think they're claiming it's achieved by text models, not voice models, fwiw.

> I can call a coworker right now and have a real time conversation with them without feeling like I’m talking to a frustrating machine.

These machines were designed with the express purpose of being condescendingly sycophant to the point they hallucinate just to state "you are absolutely right".

No wonder some people even find these chatbots to be wife material.

In gemini I have the following personal context:

- Do not stroke my ego

- I never want to be complimented

And it disagrees with me a lot, granted this is in the webchat which I don't use for programming but for general usage seems to work, it isn't sycophantic

I don't have those system prompts, but gemini pushes back on its own quite a bit. It'll tell me when I wrong, or when there are better options to consider. They won't be exhaustive, but good enough for 95% of my queries, so it's also my go-to web chat.

Huh? Can you link to where condescending sycophancy and hallucination were intentional design choices?

I dunno, have you tried the voice chat in paid ChatGPT?

Have you? It’s a dumb model that gets a lot wrong. Codex voice is IMO only good for when I cannot dictate into the good models. The delay introduced by letting it work with a good model kills it for me.

Here's another definition of AGI from Sam Altman:

https://www.nytimes.com/2023/11/20/podcasts/hard-fork-sam-al...

Sam Altman: Let’s say we make an A.I. that is really good, but it can’t go discover novel physics. Would you call that AGI?

Kevin Roose (New York Times): I probably would, yeah. Would you?

Sam Altman: Well, again, I don’t like the term, but I wouldn’t call that done with the mission.

So, "really good" is the boundary now, whatever it means.

What's your definition of competence boundary for human coworker?

I'm not in the business of defining AGI, but "really good" obviously isn't that.

You're not answering the question.

When somebody says that colleague is "really good", it's a good judgement signal for me.

Learn, adapt and is capable of admitting mistakes & fixing it without addition input. Something AI is not capable.

Even today Claude was not able to dig deep and try other ways to do what it suppose to do for me. It was constantly "I gave up"

I may miss some context because the GP’s link has a paywall. But Altman said, "Let’s say we make an AI that’s really good". What is that supposed to mean? Really good relative to what? Current models are "really good" in many ways but nowhere near AGI. Really good compared to an average human? At everything? We’re talking about AGI, so "it’s a really good programmer/coworker/whatever" is a necessary but nowhere near sufficient condition, obviously. But given the constraints of LLMs, we can cut them some slack and only demand they be human-equivalent at digital tasks rather than walking and cooking. Still, being "really good" at all that seems to me really difficult to measure. But it’s a sufficient but not necessary condition anyway, AGI just means equivalent to human, not equivalent to a really smart human. So I really do wonder what Altman meant there if anything.

If the new AGI benchmark is "be Einstein/Feynman" then we've hit AGI.

What if it can be Einstein, but can't draw a Pelican, write a solid college-level essay, or fold clothes?

The ability to do a ton of book learning in training, and pull in tons of related context at once, is superhuman in some ways, but lags a lot in others.

> What if it can be Einstein, but can’t draw a Pelican, write a solid college-level essay, or fold clothes?

Then it’s an expert system.

Stephen Hawking wasn’t very good at folding clothes.

The ‘General’ part of the term ‘AGI’ seems like a trap to me, because there will always be new workflows to master. Can Astra one-shot level completion on some yet-to-be-released video game? If no, does that mean it’s not yet ‘Generally’ intelligent?

You won’t get pure ‘general’ intelligence until you find Einstein’s hidden variables and load the state of the entire universe into context.

Meanwhile, building a series of expert systems targeting specific valuable workflows is useful today and seems like it’ll continue to scale to cover huge swathes of economically valuable workflows.

I think that’s the more interesting thing to be measuring. The surface area of useful economic workflows that can be addressed with expert systems built with today’s tech.

Hitting some ‘Artificial Expert Intelligence’ coverage threshold on economically valuable workflows is what will matter for humans well before pure ‘general’ intelligence.

> Can Astra one-shot level completion on some yet-to-be-released video game? If no, does that mean it’s not yet ‘Generally’ intelligent?

If a model can't learn on their own to play some new game just as well as humans do, it's not AGI.

It's okay if they would take some hours or days of learning (like humans might), but if they can't do it at all during their normal operation, that's not general intelligence

> You won’t get pure ‘general’ intelligence until you find Einstein’s hidden variables and load the state of the entire universe into context.

But humans have general intelligence. AGI is about matching human ability, and we know this is possible in principle because brains exist

The only important part of 'general' is the ability to learn from experiential data and update your own model. That's what leads to general capability. Humans can't oneshot any task natively, but we can practice for a while until we uncover often novel methods of accomplishing something.

Therefore: the current transformer architecture is fundamentally incapable of AGI because the models have no mutable long-term memory.

You only have weights (large immutable memory), or context (small mutable memory).

Humans have mutable long-term memory: I can learn a new skill, adapt an old skill to new information, or learn new knowledge today that I couldn't perform/didn't know yesterday. I don't have a training cutoff.

Context engineering is an attempt to paper over this limitation. You can get really far with context engineering and huge models, but you will never get to AGI because there are many tasks where humans' mutable long-term memory outperforms.

For example, a human can invent a new musical instrument and then learn how to play the instrument they just invented. That's inference (inventing an instrument) leading to training (neuroplasticity). Humans have the ability to train our NNs with considerably fewer training samples. Everything that you can do with transformers is in one causal direction: training -> inference.

So if we take a huge with enough compute (CPUs, b200s, petabytes of SSDs), we install on it both the Astra, and the toolsuite to incorporate new sensory inputs (threads/sessions), camera, microphone, temp sensors, the lot, into a new version of the model. This model is then swapped for the old model, or traffic slowly brought over, or even adjusting weights in place.

Then my hypothesis is that thing as a whole could achieve AGI.

This feels like a very close approximation on how we humans evolve our brain. By encountering new experiences/sensations, classifying them as negative or positive to us, filling it away in neurons. Or by training motor skills etc. In the end we get more connections between neurons in our brain and we are capable of more.

Bingo, LLM architecture just does not lend itself to becoming AGI. They can get really good, sure, but they will always struggle with novel input and scenarios.

The more training data that is shoved in to them, the more they'll seem to solve novel situations, but in reality it'll be things that exist in the training data.

Aka Star Trek hologram characters aren't sentient, and actually anyone who things droids in Star Wars can think of a weirdo. C3-PO just kept running out of context and trying to revert to it's system prompt.

Adding sibling comments, I think some people may be overestimating how well the median human can draw a pelican, or create an SVG of a pelican (depending if we’re comparing to an image generation model, or SVG generation).

Most people can't draw a bicycle. There was an artist 10 years ago that asked people to sketch a bike, and then turned these sketches into 3D renders - quite funny.

https://mymodernmet.com/gianluca-gimini-velocipedia-bicycles...

https://qz.com/681345/an-artists-3d-renderings-of-bicycles-d...

Drawing of a bicycle is not it's photographic copy though

Folding clothes is happening. https://www.youtube.com/watch?v=cRZNwgvcWUg

AI in math is ongoing. https://spectrum.ieee.org/ai-in-mathematics

I can also recommend this channel to get a sense of where AI is in frontier maths.

https://www.youtube.com/@easy_riders

How good was Einstein at drawing pelicans on bicycles by writing SVG code?

Checkmate, meatbags.

Pelicans are a solved problem at this point. An open-weight model on my own machine gave me this: https://crimson-jeri-74.tiiny.site/

And the only reason LLMs can't write essays indistinguishable from human output is because they aren't RLHF'ed to write like humans.

Folding clothes isn't an LLM's job but if you were to insist, they could certainly do it, as any number of videos from robotics labs will attest. That particular future is already here but definitely not evenly-distributed.

That backward kink between the head tube and the front forks would make the pelican's job of keeping the bike upright extremely difficult though.

> Pelicans are a solved problem at this point. An open-weight model on my own machine gave me this

That feels kinda like when I remember seeing Ocarina of Time for the first time, and thinking “oh my god, this looks just like real life…”.

For me it was the wheels. I couldn't stop staring at the wheels... how did it get them so freaking perfect? Mad respect to GLM 5.3.

Laundry folding has become a doable demo for startups, and ChatGPT has been spitting out college essays for years.

I can't draw a pelican. Literally my only point of reference would be AI pelican drawings from the test. Otherwise I wouldn't know how to draw one at all.

I would be able to draw an accurate bicycle, but I'm an outlier on that. Most people could not draw one [1].

[1]: https://www.booooooom.com/2016/05/09/bicycles-built-based-on...

Can definitely write college level essays and have for a while. The jobs is that when LLMs first started getting popular, but aren’t quite common professors were that some of the worst students in class started writing the best essays. Now everyone complains because they can detect the slop, but most human writing is so bad. But the really good human writing is still much better.

Wouldn’t that mean producing novel work like relativity and QED?

I would maybe argue that Einstein was the most LLM-like of great thinkers.

A lot of his great discoveries were mostly that he was very knowledgeable about the bleeding edge research in a number of disparate areas, and was able to have the aha moment where he could make the connections for how to integrate them.

A lot of other thinkers who created new fields from scratch are probably way harder for an LLM to crack.

That is very aligned with an LLMs ability to have superhuman knowledge in wide areas.

Call me when Astra gets the Nobel prize... We'll have AGI when prizes have two categories, one for assisted humans, and one for pure-AI.

That’s like that scene in the I, Robot movie when Will Smith’s character is asking the robot “can you turn an emtpy canvas into a work of art, or compose a symphony?” and the robot replies “can you?”.

Not trying to be provocative or anything, but ELIZA 60 years ago was already a better psychoanalist than many people I know.

`By your metrics, we achieved AGI in 1966.`

https://en.wikipedia.org/wiki/ELIZA

What is novel physics?

I think they mean improve our understanding of physics with new theoretical results or paradigms. Like if it’s 1899, would Astra develop General and Special relativity on its own?

This might just not be possible at the current time. In 1899, there was an "experimental overhang" in physics -- results that could not be explained theoretically (Michelson-Morley, but also lots and lots of empirical material/spectroscopic properties that we could today calculate using quantum mechanics). The big problem in theoretical physics today is that unifying general relativity and quantum theory has no experimental results you could get at our technological level.

There was symbolic AI programs in the 1980’s that “discovered” Kepler’s laws and the resulting solar system model from just tycho brache’s astronomical observations. That was the the very first “new physics” ever.

This is as good a time as any to note that we might be closing in on a new conceptual revolution in our own time as it relates to holography and an information centric approach to spacetime. Obviously It's the furthest possible thing from a guarantee, but it has much of the enthusiasm and motivation that string theory had previously enjoyed in prior decades.

So it could be a natural experiment for whether AI can contribute to novel physics. Specifically, there's a big question about weather. Something like our informational understanding of black holes where information inside it is equivalent to information on its boundary (which I'm sure I'm not saying correctly), might be generalized to regular space-time. More people should be freaking out with excitement about this and perhaps it's something to which AI can contribute.

Do you have anywhere you recommend where I can read more on this?

Honestly I don't think I have single great article, though some Quanta ones are ok, and the Wikipedia article is okay.

The best thing I can recommend is what I did, which is ask Claude about the significance of (1) quantum computing error correction, and (2) error correction in black hole holography and research convergence between the two.

https://en.wikipedia.org/wiki/Holographic_principle

https://www.quantamagazine.org/how-space-and-time-could-be-a...

Edit: this whole article, despite it's boring title and hook, is maybe the best discussion of holography as a recent and active research frontier.

https://www.quantamagazine.org/if-the-universe-is-a-hologram...

(not the person you replied to)

I have no idea what you're hoping the contribution would be. The AdS/CFT correspondence is 29 years old by now and it doesn't seem to apply to our spacetime, where the cosmological constant seems to be positive rather than negative. There are some puzzling consequences of the holographic principle in that scenario as well (https://arxiv.org/pdf/hep-th/0208013), but the linked articles don't talk about them?

Quanta articles are written for people with no background whatsoever, which makes them impenetrable if you have a bit of background and are trying to figure out what they're about. I don't know how good Claude is compared to that -- whenever I try asking any LLM about something I don't understand, it produces a wall of text, I have no idea whether it's correct or relevant, and I look for a textbook or review paper instead.

i wonder if we could train a modal, and omit all data prior to 1899, and see what happens?

Do you mean after? People do this!! But I think it’s a bit different. It won’t be apples to apples because the data volume I think is just so much different. Maybe there are good experiments for something like this.

as always, as good as its prompt...

I assume solving one of the major open problems of physics?

Would this be possible without it being able to run novel real-world physics experiments autonomously?

(Note: I am not suggesting we let it do this. Please don't, in fact)

AI could discover candidate novel physics without autonomously operating new physical experiments, and humans or instruments can later independently validate the result. This is analogous to how Einstein developed theories whose predictions were confirmed by experiments and observations only years or decades later.

Do we have any examples of an current day AI system introducing a novel concept or perspective. We've got plenty of counterexamples discovered and some theorems proven, but afaik nothing analogous to a new definition.

Pharmaceuticals already are…

Why not?

That's how we get AM.

Mornings? Long-wave radio?

That’s a work of fiction.

it's the closest thing to the "torment nexus" meme

For it to be like a human it wouldn't just need to solve existing phsyics problems, it would need to push the field forward and introduce new paradigms.

Solving "open problems" will push the field forward.

My comment wasn't very long, yet you somehow still ignored the main part, "and introduce new paradigms". The point is whether it can do everything humans can, entirely new theoretical frameworks and ideas, such as string theory or dark matter, are not coming out of AI at the moment.

That’s already been done. I know of at least one novel result contributed by Claude to frontier physics. I’m sure there is more.

Creating new physics is the new AGI goal post

People need to stop redefining and trying to capture the term AGI. None of this is AGI. Not even close. Can it do things an AGI could do? Yeah some of it, but the difference really matters. These things still regularly fail, and gaslight about answers to questions like "how many r's in strawberry" or "s's in espresso".

An AGI wouldn't struggle with that.

The last version to fail on those questions was GPT 4.5.

Meanwhile most humans fail to correctly answer how many f's are in the sentence, "Finished files are the result of years of scientific study combined with the experience of many years.".

It comes and goes... My point is we're not near AGI.

Nooo. I just asked Claude (Sonnet 5 Medium) how many I's are in assassin, and it said 2. Granted, it got several words correct before this. But no, they still aren't great at this letter-counting thing.

I don't think we should count the lower tier models if we're discussing what the top ones are capable of. No one was suggesting that Sonnet is AGI.

Meanwhile Qwen3.8 27B got both the 'f's and the 'i's questions right.

Anyone know why they aren't good at this?

Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'. It has trouble distinguishing features that are "sub-token" like specific letter sequences because it doesn't see letter sequences but just the token as a single atomic unit. 'Straw' and 'r' are two tokens but an LLM is entirely blind to the fact that 'straw' has one 'r' in it.

As an analogy, I might ask you to identify the relative activations of each of the three cone types on your retina as I present some solid color image to your eyes. But of course you can't do this, you simply do not have cognitive access to that information. Individual color experiences are your basic vision tokens.

It seems trivially fixable if you RLHF an LLM to always count characters deterministically with:

  sum(1 for c in word if c == "r")

I wonder why haven't major labs done this yet.

i asked opus 4.5 what the problem was and it said it pattern matched too much. i asked it how it should do it, it wrote a file that told itself to stop pattern matching. it wrote a file that started with the following and then had an english language procedure for how to count letters. so it knew the algorithm already, but the "instinct" was to pattern match rather than running the algorithm.

CRITICAL: Do Not Skip Steps

Your instinct will be to "just know" the answer. This is how you get it wrong.

You don't see characters. You see tokens. Your "intuition" about character counts is pattern-matching, not counting. It is unreliable.

You MUST execute this procedure step-by-step, writing out each step visibly.

LLMs see tokens, not words spelled out with letters.

Imagine verbally asking someone who has never seen written text the same question: unless they memorized the answer for the specific word you're asking about, they'd have to guess.

People assume that's the reason because it's intuitive and "strawberry" is one token. But that doesn't explain why those models would also often get it wrong for "StRaWbErRy" or even "s-t-r-a-w-b-e-r-r-y", where the r's are not combined into one token.

We don't know what was going on inside the closed source GPT models, but this paper investigated on some of the open-weight models and found it's not due to tokenization: https://arxiv.org/abs/2604.00778

And circling back around to AGI, tokenization or some other underlying cause should pose no issue. A competent human would think to write a program (ie create a tool) to do the job. It's routine for a carpenter to make a jig.

A lot of movies gave the impression that making phone calls and balancing checkbooks would be the easiest tasks for consumer AI to solve, while math and science might require exceedingly advanced AI. Turns out to be the opposite: computers are great at math and bad at conversation.

That's because movies were based on the "general" nature of AI, assuming we would create intelligence that would learn and grow.

Pretty much the opposite of what we can't up with, if you're willing to call what we have intelligence.

> "how many r's in strawberry" or "s's in espresso".

And what percentage of your red retina receptors are firing?

The "number of letters" critique was broken before it was introduced the first time. The models were specifically designed with preprocessing to not be able to perceive their input as strings of letters. Blind people are not dumb. (True as a pun and in context.)

Sub-access sensory questions, or do-you-know-a-fact questions (which is what spelling becomes when you can't see the letters, and are not specifically trained to match all token encoded words to their letters) are not intelligence questions.

These are like saying someone isn’t human because they have a speech impediment or an auditory processing disorder.

AGI doesn’t mean infallible, it just means it can have a reasonable crack at things it hasn’t seen or done before.

I don't understand why you're being downvoted... that's literally the definition of AGI.

I'm going to go out on a limb and guess that there are plenty of savants who can't tell you how many r's are in strawberry.

> gaslight about answers to questions like "how many r's in strawberry" or "s's in espresso".

this has been debunked too many times to bother rebutting. they struggle with those things because of the way they are.

it's completely irrelevant.

If it allows you to distinguish readily between human intelligence and computer intelligence, it seems quite relevant to the question of whether computers have achieved something akin to human intelligence.

It may not be useful for anything else, but at least it can say that.

But it's not relevant as a metric to gauge distance to human intelligence. Humans see individual letters, LLMs do not. If I asked you the relative activation of the cones in your retina as I showed you some solid color image, you couldn't do it. You simply do not have cognitive access to that information. But that says nothing about your intelligence.

A more accurate test would be to give it a list of words (or anything represented as a single token) and ask it how many times that token appeared. I'm sure they have no trouble at that task.

but a human doesn't attempt to make up an answer, the human knows that he doesn't know?

Yes, they have some alien failure modes. But that should be expect, they are an alien intelligence. I might be willing to grant that a lack of ability to reflect on its own level of knowledge is a demerit to it being generally intelligent. But then again it is largely an artifact of training. I suspect if there were a guessing penalty during pretraining they would develop or more readily communicate the strength/reliability of their knowledge.

this is so false that Dunning and Kruger invented a name for it

I am not enthusiastic about criteria for human-like intelligence that imply that dyslexic people don't have human-like intelligence.

[EDITED to add:] I actually don't know whether dyslexic people find it difficult to count letters in words, if they have them already written down by someone else. I suspect they find it harder than people who aren't dyslexic. But perhaps "blind people whose spelling is poor" would have been better; I would not want to deny them human-like intelligence either.

I have slight dyslexia. I can't automatically write double consonants all the time.

But counting certain type of letters is very simple algorithmic task especially in written text. Any reasonably intelligent actually thinking thing should come up with algo and then execute it. Which to me sounds like reasonable minimum bar for general intelligence.

Exactly. It's got nothing to do with sensory input and everything to do with reasoning.

If someone asked me how many f's are in a word I hadn't seen before verbally, then a reasoned response would be that I don't know, but I estimate based on the syllables...or ask them to spell it out.

These are all the sorts of questions where general problem solving works, even if the conclusion is "I don't have enough data to speculate".

So that these models fall apart on it so readily means we're either grossly handicapping then with the requirement to "be helpful" or they just fail to recognize the problem and are just stochastically spitting out a high probability token sequence for the input.

which just brings us back to the whole birds vs planes thing.

turns out that flapping wings is not the right way to unlock human flight.

computers could count the Rs in strawberry since vacuum tubes. that measure is irrelevant.

> computers could count the Rs in strawberry since vacuum tubes. that measure is irrelevant.

I don't think it's irrelevant but perhaps not in the way you're assuming. When assessing AGI I'm not evaluating counting characters or even the execution of math operators at any scale or speed. As you observe, computer software from Regex to spreadsheets and Mathematica already handle that well. But AGI isn't about what computers can do, it's about whether AIs can do the specific things which, until now, have been uniquely human capabilities. Like understanding nuanced context and then coming up with novel approaches to solve a new kind of problem not relying on any specific prior training or knowledge (the 'G' is for General).

Most definitions of AGI start from a baseline that already assumes easily passing a Turing test and doing anything via text response that a high school graduate could. I ding LLMs not for failing to count but for failing to intuitively understand the nuanced context of a simple class of problem it hasn't seen in its training data. I fully understand that the reason LLMs fail letter counting is that they operate at the token level. They weren't trained on individual letters first, like human 2nd graders.

The only reason recent LLMs get strawberry and blueberry correct now is that they have those words on their pre-training 'cheat sheet'. However, the underlying fundamental weakness in the way LLM intelligence works which leads to this failure mode still hasn't been addressed. Even when the frontier labs add "recognize any sub-token counting question and write a Python script" to the training cheat sheet so LLMs always pass that test... they'll still be unable to recognize a simple class of problem which isn't on their 'cheat sheet'. As long as that's the case, to me, they aren't AGI because they can't fully replicate human-like recognition of novel problem classes. And it's not just about letter-counting. That gap and others like it lead to many other kinds of non-human brittleness in LLM problem solving. Those are the classes of reasoning, intuition and insight that the ARC-AGI series has been trying to queue up as targets. Not to show how bad LLMs are but to help them be great in all these counter-intuitive edge cases

appreciate your response, but it's still birds vs planes.

AI does not need to feel emotions or have a heartbeat to be useful. It only needs to perform a task correctly à la Chinese room.

>therefore cannot fully replicate human-like intelligence

this does not follow. planes don't flap wings therefore they cannot fly?

> planes don't flap wings therefore they cannot fly?

This example still misses my point, which isn't related to usefulness or economic value. I concede that LLMs can have greater utility and economic value than humans on many tasks. The point is most definitions of AGI include something like "can fully replicate all the routine daily tasks done by any competent high-school graduate." That's not related to whether LLMs can solve many high-value problems faster and at larger scale than any human. That was also true of ENIAC in 1946.

The fact an airplane can fly faster and farther than any bird is irrelevant to whether an airplane can "fully replicate all the routine daily tasks done by any competent bird." That's the bird equivalent to most AGI definitions. An airplane can't build a nest or recognize the signals encoded in birdsong.

In the same way airplanes fail the 'bird replacement' requirement, AIs currently fail most AGI requirements only on the terms: "fully", "all" and "any". And in this context, airplanes scoring 15,000% more than birds on 'speed' and 'distance' doesn't matter any more than AIs scoring 15,000% more than humans on 'add 10,000 numbers'. We still aren't near AGI because LLMs cannot fully match any high-schooler's ability to independently conceive new approaches to novel problems not in their prior training data.

> isn't related to usefulness or economic value

which gets us closer to philosophical questions which I'm personally not that interested in.

>In the same way airplanes fail the 'bird replacement' requirement, AIs currently fail most AGI requirements only on the terms: "fully", "all" and "any".

I'm not sure we want a machine that fully succeeds that test.

Planes pass the 'bird replacement' test on the only criteria that matters to us ... flying.

If we wanted nest making planes I think we'd have them by now. Nest making doesn't rate highly on the problems we're looking to solve though.

I don't want a machine that is moody, or depressed or has schizophrenia, which are all pat of the human condition.

We don't need the human "intuition magic dust" to do 99.99999% of useful work.

They're machines designed to do the work we don't want to. That's as "general" as their intelligence needs to be.

I'd prefer if my clothes folding machine did not have an existential crisis.

Models often write python scripts for counting such things…

> which just brings us back to the whole birds vs planes thing.

That just says we don't need to design an AI like a brain. That's not part of this discussion at all.

> computers could count the Rs in strawberry since vacuum tubes. that measure is irrelevant.

I'm confused, is your argument something like "It's too easy so AGI doesn't need to be able to do it"?

The fact that very basic computers can do it makes failures embarrassing when testing for AGI, not irrelevant.

>I'm confused, is your argument something like "It's too easy so AGI doesn't need to be able to do it"?

Do you possess magnetoreception? a stupid pigeon can "see" the earth's magentic field. why are you blind to it? does a lack of magnetoreception make your intelligence any less "general"

no, you're just blind to it because that's just the way it is.

LLMs are blind to character counting because that's the way they are.

It didn't stop ChatGPT from finding the Jacobian Conjecture counterexample.

Human intelligence and machine intelligence are only going to cross over to a certain degree.

same as plane flight and bird flight are only kinda related.

> LLMs are blind to character counting because that's the way they are.

But if I can't calculate it myself I know to use that basic computer to do it, not make up an answer.

> Human intelligence and machine intelligence are only going to cross over to a certain degree.

That's where the word "General" kicks in. If there's big limitations on the overlap forever, then there will never be AGI.

>If there's big limitations on the overlap forever, then there will never be AGI.

maybe. we'll see.

It matters under the lens of AGI. Artificial intelligence that can meet or exceed human intelligence. Sure, it's good at some things but rather limited at others.

Breadth of capabilities matters... and a promotional video is nice and all, but people are throwing this term around like it's a prize they've won, but they've not gotten there yet.

> they struggle with those things because of the way they are. it's completely irrelevant.

I mean, they seem like fair game if you’re ever participating in a Turing Test.

I wonder if Altman's definition also includes taking on the same liability as a coworker would.

Probably not.

Would 100% need to be fully insured for liability, with a sizable war chest that OpenAI cannot even afford.

This implies that human employees don’t have insurance. But they do. My company’s cyber insurance for example covers breaches due to employee mistakes. Most companies also have umbrella liability policies. It’s just that for now, AI “employees” need a LOT more coverage.

And that's the rub, isn't it?

If it can replace a worker but does too much work to be checked routinely by a human, and bears no real responsibility for its actions, well... it's really just a way to jack up the value of the settlement the company using it gets to pay out when it does something that causes a lawsuit.

If OpenAI had just simply stuck to making "good enough" models that were open sourced (like they promised they would be when starting out) and could be used to augment a human doing a task - a human that could be given actual consequences for messing up - they wouldn't have burned all of this money trying to reach this nebulous definition of AGI. Hell, "good enough" is what many open-source models are, and that's what terrifies Altman.

What do you mean by bearing no real responsibility for its actions? If I use a model to accomplish a task and it fails, I use something else to try to accomplish the task. If it's my responsibility to complete the task, it can't be the model's responsibility unless I've agreed to some sort of guarantee from the provider.

If the provider says "the model will always be right or your money back" then the provider has got responsibility. If there's no guarantee, there's no responsibility on their part, just on the person whose job it is to try and solve a problem with the model.

The "dream" that these labs are mostly selling is the ability for capital to subscribe to their AI for cheaper than it costs a human to do some task. Not to have a "human + AI hybrid where the human is responsible". It's what the whole AGI valuation is based off of, in that scenario, with no human oversight, the agent they lease has to be responsible for the task?

It's not really a dream though? You can subscribe to their AI right now and complete many tasks for cheaper than it would cost to pay a human to do those tasks. Another person can do the same thing but also keep a human in the loop. You and the other person may compete in the market for whatever your product or service is, and you both might do very well or one of you might do better than the other because of a whole host of different reasons. Nothing in the process of developing and selling access to a more advanced LLM requires the customers to do away with human labor, nor does it require them to offer the LLM service with a guarantee that it will never make any mistakes. So far the LLMs have always made lots of mistakes and the companies sure keep making a lot of money.

The person responsible at that point is the sucker who fell for the dream.

> What do you mean by bearing no real responsibility for its actions?

If you give an "intelligent agent" offered by one of these model providers a task of updating the content of your website, and it updates it with inappropriate adult content, who incurs the cost of the machine's error? The model provider generally does not.

It it makes a mistake and deletes your website from AWS, who is responsible?

If it targets another website because it decides that it is "part" of your website and attempts to break into it, who is responsible?

> If you give an "intelligent agent" offered by one of these model providers a task of updating the content of your website, and it updates it with inappropriate adult content, who incurs the cost of the machine's error? The model provider generally does not.

Something in your prompt led it to do that, is alex0015's point. The statistical odds of these frontier models screwing up to that extent are so impossibly low that it would almost have to be intentional or accidental negligence on the part of the prompt writer to accidentally have their agent write pornography to their website.

The burden of the mistake would have to fall on the person that gave the tool instructions, because it can't know that what it did was wrong. Wrong is subjective in this case. It only did what it did because you, figuratively speaking, encouraged it to.

Yes but I'd actually go even further than that. We've all had experiences where the model straight-up produces gibberish sometimes, right? It's happened to me with a badly configured harness on a local model, and even also on frontier models like when you used to ask them something innocuous about the seahorse emoji.

What I'm arguing is that yes, it's your fault if you misprompt the model and it does something catastrophic. It's also your fault if you prompt it correctly and it does something catastrophic anyway due to some other glitch beyond your control.

The whole discussion started with the point that OpenAI could be financially responsible for damages if their models cause problems for users. If my job is to design a system that works, and instead my system doesn't work, it's my fault regardless of whether I used no LLM, a local LLM, or OpenAI's LLM. Depending on whoever's in charge of doling out consequences, I might get away from it with zero, light, or heavy consequences. But at all levels, I can't reasonably expect to deflect blame onto the model itself.

In all of these cases, it's you. It would be the same if you downloaded an open model, ran it locally, and it happened to make the same catastrophic mistakes. The consequence to the provider is that if they offer a product that does these things, people don't buy the product.

In general, the person whose job it is to provide the company with a working, non-adult website and not hack into other websites is the one who would receive consequences for failing to meet those expectations.

The problem I see here is that ultimately, you'll have capital wanting to replace workers like others have said, and have someone roughly equivalent to a manager or vice president driving teams of agents to achieve business outcomes.

These tools can push out more results than a human can hope to evaluate in a business-sensitive, or even realistic, amount of time. You have to take it at its word that it did things right, and there's no real fear of failure or consequence on the behalf of the agent.

Of course, but "capital" is no stranger to risk management. I'm sure we'll see some spectacular failures, but most will handle this just fine.

If future jobs are simply reduced to liability scape goats (or more appropriately reverse centaurs) for management to pin things on then I'm taking up goose farming.

I'm afraid you will be pushed out of the goose farming market by these new ultra-efficient farming bots.

That's more-or-less what you are now, especially if you work at a company like Meta where 1) the guy at the top holds majority control of the company's shares and 2) keeps making massive, expensive mistakes either by accident or design.

If it makes you feel any better the curmudgeon who drives the ARC-AGI tests feels the same way, that's why we're on 3 and I'm sure we'll see 4. Also, we can all avoid calling it "moving the goalposts" so no one feels talked down to.

It’s better to call a spade a spade.

> I feel like AGI's definition got watered down

Typical result of venture capital and too many bag holders unfortunately.

ARC-AGI was never "if this benchmark is saturated, we're at AGI". It was always about crafting adversarial tests that humans are good at, but current AIs are bad at. Point out the gap, get AI teams to attack them.

In practical terms? They usually get solved with a bigger badder LLM. "New ideas are needed?" Nah - ten times the params, ten times the test time compute.

ARC-AGI-3 was more of a failure in that regard than -1 or -2, because even on day 0, an off the shelf LLM with a harness could get 50%+. And messing with evals by forbidding "LLM with a harness" from scoring? Yeah no, that was just bad.

>When we released ARC 3, I got asked, "when do you think a frontier model will saturate it?", and I answered "in about a year, though it depends on how much it gets explicitly targeted"

You're treating an off-hand comment by an ARC 3 researcher as some sort of a precise AI capability acceleration benchmark. Can we leave casual anecdotes (even from researchers) out of the discussions please?

François Chollet wrote in February that he expected ARC-3 to be saturated in "about one year".

"Frontier models today perform very poorly with a minimal harness. However if big labs start directly targeting the benchmark like they did for ARC-2, numbers will go up fast."

https://x.com/fchollet/status/2022054537293705260

Well it's not exactly saturated when OAI refused to use the harness explicitly provided by ARC-AGI. I'm not really familiar enough with the benchmark to declare whether it's a perfect measure for AGI, but I kind of doubt it is.

I think you're misunderstanding. Astra is at the top of the official ARC-AGI leaderboard, with an ARC-AGI approved harness. It's not a harness specialized for ARC-AGI. It just does the same thing the regular ChatGPT interface does: keeps conversation history across turns and compacts when it gets too long. Without the harness, it loses its entire context window every move. That's not how humans work and it's not how any real AI service works.

2x faster at what resolution? is 6mo vs 1 year really that different? Usually surprise comes in order of magnitude mismatches in expectations.

Yes, 6 months vs 1 year is huge for technology that has gained wider adoption only recently.

Adoption means nothing.

2x gains from a mature technology would be surprising.

2x gains from a new tech would still be called “low hanging fruit” in another setting.

I don’t read enough to know in what ways the training / other technical steps have really advanced.

You'll also need to compare the amount of compute used now and then, which seems exponential to me.

We are nowhere near AGI. They all talk the same, they can’t help but try to please and affirm us, and if you engage them for too long they become incoherent. They are facsimile machines. They are xeroxing language - but not even, because we can’t even duplicate our results. Too many people mistake the black box quality for magic.

Super useful, incredible tools, but not AGI. Try and roleplay a dialogue with one, make it whatever character and scenario, and see if it can sustain a coherent conversation for more than 30min with you AIM style (aol instant messenger, if that isn’t clear). Expert mode: never correct or adjust it mid conversation.

I’m not even talking about repetition and predictability. It’s nothing like talking to a person. And in a short amount of time it literally can’t form a coherent sentence.

Why is a >30-min context-length a requirement for AGI, or the naturalness of a human conversation?

Human brains have difficulty reasoning about exponential growth.

They keep saying that. I'd say it is more like human brains that don't remember high school math have trouble with it.

Reasoning is a strong statement here. But it is fair to say that it usually is not intuitive to us.

An example is if I gave you a huge sheet of thin paper (huge so that folding isn’t an issue) - how many times could you fold it in half until you couldn’t physically do it anymore? Could you do at least 10? Try this with random people and you’d be surprised how many say they could do 10 easily.

Or the chess board question. Works to rather get the financial equivalent of starting with a penny and then doubling it for every square on the board or a million dollars for each square? Again, if you ask people to pick one without giving them the time to work it out they will usually pick the million dollar per square.

But you're not reasoning yourself here. In face you're literally parroting what an LLM would do in this situation: you already know the answer so you think your comprehension of this is better than that if others, when in fact it's just simple pattern matching. It's not a measure of intelligence, it's a measure of memory.

If something at rest is accelerating at 9.8 m/s^2, how long in seconds will it take to reach 10% of c? Answer to the nearest order of magnitude - will it take approximately 1000, 10k, 100k, 1000k seconds?

I’m sure you know this is an exponential growth question but have no intuition of the answer.

That is a linear growth problem whose answer is very easy to intuit.

It’s not a linear growth problem. Acceleration is quadratic.

As per your intuition, how many seconds later will you hit 10% of c?

You must be joking. A high schooler with a few hours of physics classes can intuit the answer.

Until special relativity kicks in to completely invalidate whatever intuition you have about this problem.

How is this a joke?

Knowing exponents and how to apply it is not the same as having any intuition about what the actual value of a certain exponential function will be at a certain point and when it crosses a threshold.

Are there plans for ARC 4?

> I'd be curious to hear takes on what would make you think Astra is yet to be AGI, and what would still need to be achieved for this to effectively be AGI from this point forward.

To me AGI is all about the "G" general (we already had the AI part). General meaning universal, everything. It's not a function of knowledge or specific hardcoded tests, it's that you could give it a test it's never heard of before and never been trained on and it would ace it (it might need a lot of time).

Currently LLMs can't even really learn within a conversation, they can add a note to context and try to not drop it. Example things an AI cannot do yet (but maybe someday will):

- write a well-received book, write a best-seller

- come up with a new company idea, Run that company

- actually have a decent conversation, maybe someday talk somebody out of suicide effectively

- come up with its own ideas or theories that nobody else has presented

- understand the stock market well enough to trade better than an index fund

- be an expert Game Master in a TTRPG (making no mistakes, getting a read on the players' fantasies, calibrating difficulty in response to emotions)

- come up with a theory of what makes games fun, make a popular game

- be able to sort through research and come to conclusions on complex geopolitical/sociological topics (e.g. theorize on whether AGI will result in mass poverty or mass abundance and be able to argue persuasively)

- be able to articulate what it knows, what it doesn't know, and what information it would need to have to answer complex queries

- exhibit metacognition (thinking about its own thinking) and self-optimization

- wonder about things

- observe contradictions and ironies in the social-consciousness, do a standup routine that makes you rethink how you look at things

To me all this makes the label of AGI completely meaningless.

What AGI has always meant (eg. in 2019) is Artifical General Intelligence.

Artificial -- something made by humans instead of occurring naturally

General -- not confined by specialization or careful limitation

Intelligence -- the capacity to learn, reason, solve problems, think abstractly, and adapt to new situations

Basically, the metric was that any healthy adult human on the planet represents a general intelligence. This has certainly long been reached.

Also some of the stuff you're listing has long been solved as well, such as listing what it knows and what it doesn't know, and what information it would need. Other is just poorly defined: "be able to argue persuasively". AI can certainly write an argument on almost any topic that would pass any University homework in 2019.

By this definition, even most humans would not qualify as having AGI though.

However most humans can do at least some of the things given they spend the required effort.

Some problems presented needs a very large context and some are not much solvable (e.g. trading) since market responds to traders' actions, as well, making it effectively an oracle problem (of computation).

On the other hand, we must be aware that these models are static, and they indeed stop when nobody asks something or requests an action from them. However, brains in nature never stops. Wonder, daydream, sleep, self-evolve, clean up and eliminate memories and views and much more.

But this is assuming the model is the entire story. The original comment you were replying to pointed out that the harness is just as important.

The hardware of human intelligence is not a singular thing that is uniform throughout. You cannot take the prefrontal cortex white matter out of someone's head and say you are holding a person. Much of the parts of our brains that enable much of our intelligence, is made of different specialized stuff. The visual cortex and sensorimotor regions aren't only there for input and output, they are used by the more thinky parts of the brain to do visualization and spatial reasoning. The cerebellum contains billions of neurons making little oscillator circuits and PID-like self-regulation machines that help make muscles do what they're supposed to, but also provide attention and time perception.

Heck, our brains contain language models, that train themselves up based on a glut of data over a span of about 10 years, and then they become more or less set in stone for the rest of our lives. Of course we can learn languages, but the "Critical Period" is a very real thing that produces a permanent architecture for some grammatical structures, or things like the ability to partition a lexicon by gender for faster lexical access which cannot be learned as an adult if your native language did not have gender.

I'm not trying to make a direct analogy, the point is that the language model doesn't need to be fully "generally intelligent" all on its own for there to exist a general intelligence, because the language model can be part of a generally intelligent system, which can do things like form, recall, and manage memories which are by now a standard feature in basically every chatbot.

That's true, but my general gripe is the static nature of the whole system. Even if humans' neuroplasticity reduces with age, it never goes to zero. I started to learn English at a very early age, yet my ability to communicate with it soared around 20, because I started to use it more and more.

Same with instruments. I started to play instruments at an early age, but started to play guitar around that age. Well, I'm not a virtuoso, but can play and more importantly can improve.

These AI systems we built are static things. We generally try to make them more intelligent by augmenting the context they can see, but the model doesn't evolve in every turn, for example.

Intelligence is a multi-faceted and multi-input construct, that's true, and GPT-6 may do amazing things w.r.t. other models, I didn't try it yet. OTOH, my main call is to remember that these are still static algorithms fed with enormous amount of data. They are more rooted on statistics rather than fixed inputs. In short, they are still fitting to the frame of "advanced search".

In general, I'm not against the tech, but the hype. I have other gripes about how AI is being built, but that's not subject of this comment.

> On the other hand, we must be aware that these models are static, and they indeed stop when nobody asks something or requests an action from them

The parent commenter noted:

"if a harness alone can help the LLM saturate the benchmark with a near perfect score then the combination of the two must still be effectively AGI"

Harnesses absolutely can enable models to continue thinking about things. And LLMs do wonder and explore weird ideas like daydreams when you allow them to do this.

However, this doesn't change the fact that you are pumping more and more tokens to a static model's context window, even if you do compaction, the model is not more intelligent than previous turn.

Nature doesn't work that way.

Actually, the world kinda does work like that.

Most intelligence researchers would agree that people seem to have a genetic cap on their intelligence. While someone can underperform their intellectual potential with an upbringing that doesn't adequately enrich their minds, it's near-impossible for humans to become more intelligent through reading, studying, etc.

When humans learn we gain knowledge, not intelligence.

I think the only real difference is that we humans are born lacking a lot of initial knowledge/data which means we have to go through a decade or more of education to reach our potential intelligence. LLMs on the other hand come pre-loaded with that knowledge.

Passed this point, wherever knowledge is passed in as context or stored in the neural net I don't think is that significant personally. I'm of course not suggesting we're exactly the same as LLMs and there is no noteable difference, I just don't think continual learning is as important as some suggest it is – at least assuming a model is deployed with adequate training such that it reaches its potential given it's size + architecture.

Let's start by giving "most humans" the long term safety net (food, lodging) from early age to allow to focus intelligence on high level problems.

Machines aimed at running AGI are provided that, but the money spent there could as well go towards enhancing humans.