You could push the analogy even further and run the thought experiment where every forward pass through an LLM could in principle be done on pen and paper, distributed throughout all humanity. Sure it would take a long time, but the output would be exactly the same. We’ve just shifted the implementation from GPU to scribbling things down on paper. If you want to assert that LLMs are “conscious” then you would have to likewise say this pen-and-paper implementation is conscious unless you want to say a certain clock-speed is a necessary condition for consciousness.
When we get complete neuronal connection maps (which we are getting close to for mice and humans will be done within a decade or two), we could in principle simulate a brain on a computer or on paper too. Unless you assert something magical like a "soul", these connections are what determine human consciousness. It is one thing to argue that LLMs don't resemble brains and if they could be "conscious"
they wouldn't be conscious in the sense we are, but asserting that anything understandable can't be conscious won't age well.
While I think you're right in principle, there's a lot of reason to think the structure of a brain is more complicated than just the connection map of the neural network. There is a lot of complicated behavior inside each neuron that we don't fully understand yet. They aren't just logic gates.
But, of course, that's just physics. It's not magic, so your point stands.
We know the brain can be modeled by math (and therefore thought can be written down on paper).
We know because we have mathematical models for atoms. And we know the brain is made out of atoms therefore the brain is simply a mathematical model of interconnected atoms that form a specific structure called the brain.
Thusevery facet of macro (keyword) reality should be able to be written on paper and calculated. That goes for everything… from the emotions you feel to the internal forward pass of an LLM.
I feel like Searle could have taken the argument further down to pen-and-paper because people will somehow think if you can just make the neural network big and fast enough then “mind” and “consciousness” will somehow emerge from the symbol manipulation being done, but if you were to write it down on paper then it wouldn’t. So yeah, if you think that consciousness can arise from computation, then you’re forced to admit it can arise through doing math on paper.
the problem with this is I'd strongly argue that you could do this pen and paper process with the human brain and our consciousness too; we just lack enough understanding to put pen to paper in that case
the notion of consciousness being something an experience that other animals/humans share is entirely faith based.
the only person with evidence of ones consciousness is the person claiming they're conscious.
> the problem with this is I'd strongly argue that you could do this pen and paper process with the human brain and our consciousness too; we just lack enough understanding to put pen to paper in that case.
You're basing your premise on a lack of understanding[1], the GP's premise is based on an exact understanding[2].
You don't see the difference between your premise and the GP'S premise?
-----------------
[1] "We don't know how brains actually come up with the things they come up with, like consciousness"; IOW, we don't know what the secret ingredient is, or even if there is one.
[2] "We can mechanically do the following steps using 18th-century tech and come up with the same result as the LLM"; IOW, every ingredient in here is known to us.
We know the brain is made up of atoms and we know how to model atoms. So we do know for a fact that the brain can be modeled mathematically and we do know that human thought can be written down symbolically as an algorithm on paper even though we don’t explicitly know the exact formulation of said algorithm… That is fact.
The blue brain project has already modeled the hippocampus and cortex of the rat brain uses advanced imaging and simulations in super computers. So if it can be written down as memory on disk it can be done on paper as well.
The rat brain is simply a smaller and structurally different neural network then the human counterpart so the jump from the blue brain project to human brains is simply a scaling issue.
But from this you should begin to see the analysis from another level. Even though we have parts of the rat brain emulated computationally we still do not know if the rat is conscious. We don’t understand the rat brain in the SAME way we do not understand the LLM.
What people are getting at is the projection of this logic to things that don’t exist yet but can exist. When the blue brain project scales to the human brain we will hit the same problem with the human brain because it’s just a scaling issue.
To sum it up. We CAN already model biological brains as mathematical equations as we do LLMs. And for both cases we still cannot fully understand or characterize the nature of both because the sheer complexity of the models are too high.
> We know the brain is made up of atoms and we know how to model atoms.
Incorrect. There's still a lot we don't know about atoms. We can (sort of) model them, but not with the degree of accuracy you appear to think we have.
I mean, it's only recently that we discovered surprising changes in the properties of quarks, gluons and nucleons in relation to each other!
So, yeah, the following foundation for your argument:
> So we do know for a fact that the brain can be modeled mathematically
Is untrue. We can't do that, we have never done that.
> The blue brain project has already modeled the hippocampus and cortex of the rat brain uses advanced imaging and simulations in super computers.
They've got something, but they don't know how close or how far away they are from accuracy to the real thing.
We've almost always had a model of the human brain; first our model was simple (it has four or five parts), then we learned more and our model expanded to include actual cells (neurons, dendrites, etc), then we learned even more and our model was refined even further to include activation energies, rerouting, etc.
What makes you think we are anywhere close to the base layer when there is no more refinement to be made? Because while there is still things in brains that our outside of our knowledge (which, by definition, we don't know yet), we don't know enough about brains to make a replica of one as a mathematical model, or in silicon.
> Incorrect. There's still a lot we don't know about atoms. We can (sort of) model them, but not with the degree of accuracy you appear to think we have.
Not incorrect. You are misinformed and getting pedantic. Our knowledge of atoms is enough to model macro level phenomena and has spawned fields such as materials science and molecular biology. What is intractable is the computational power needed to accurately model things like the physics of protein folding. The computational power needed for that scales exponentially such that we can’t model it. That is the reality.
That being said we don’t need to model quantum level phenomena to model macro level effects like the biological mechanism of a neuron. There are simplified models that we can use as we have used in the blue brain project.
Additionally the thing we actually can’t model and don’t know about are extreme physics like black hole physics where the quantum world interacts with gravity but that is largely irrelevant to the topic at hand.
I hope this excerpt educates you a bit.
> Is untrue. We can't do that, we have never done that.
We haven’t done that just like we haven’t actually actualized the biggest number ever calculated by a computer. We know that number exists in theory but you’d be an idiot to claim it doesn’t exist as it’s foundational. For example the number a Google exists but no one has seen evidence for its existence. We know it through logic. From the blue brain project we can infer relatively confidently that the human brain can be emulated on silicon. This also follows from Turing completeness.
> They've got something, but they don't know how close or how far away they are from accuracy to the real thing.
The emulation is Quite accurate from imaging and emulation. The properties of the emulation that match in vitro and in vivo experimental data without specific parameter tuning. It is accurate as far as we know. That is about the same extent that we understand the human brain the LLM. The better question for you is how do you know it’s not accurate? You don’t. What we do know is that from measurable properties we understand that the blue brain emulation is accurate to the section of the mouse brain it emulates. This is exactly the same reasoning applied to LLMs… the tokens LLMs generate are remarkably inline with consciousness such that it is indistinguishable and thus can be speculated to actually BE conscious.
> What makes you think we are anywhere close to the base layer when there is no more refinement to be made? Because while there is still things in brains that our outside of our knowledge (which, by definition, we don't know yet), we don't know enough about brains to make a replica of one as a mathematical model, or in silicon.
Who says we need to make a replica of humans to make it conscious? We know the brain is made up thousands of evolutionary side effects orthogonal to the concept of consciousness like hunger, sleep and anger. All we need to do is replicate a sliver of the subset of human output we do consider as consciousness and that’s it.
But right now we can’t even fully define what that subset is and we can’t even understand how an LLM replicates human output.
What we do know is that the LLM replicates human output to a degree never done before indicating that it understands what is being told. From the evidence observed it is a valid speculation to consider it a form of consciousness. That is entirely different from saying AI is human. It is clearly not human but it is unclear whether or not it is conscious.
To be confidently claiming an LLM is not conscious is fundamentally misguided because it meeting most of our intuitive expectations of what consciousness is. It’s just people can’t face the reality that their own consciousness is not a form of exceptionalism.
> That being said we don’t need to model quantum level phenomena to model macro level effects like the biological mechanism of a neuron.
How can you know that? We haven't gotten sentience from the modeling we are doing at the moment so I am wondering why you are so confident that what we already know is sufficient to form an accurate model of brains as regards to consciousness.
> Who says we need to make a replica of humans to make it conscious?
Well what's the point of bringing up the modelling of a brain in conversation about mechanically replicating the brain via pen and paper? If a replica is not necessary and you have no idea if the macro model we have is sufficient, what's even the point of this thread?
My original point was that upthread poster said we can mechanically (i.e. without any thought, just repeating a process by hand) produce *exactly* what an LLM produces, while we cannot do the same for a brain (rat or human is not relevant).
I feel that even after all the content in this thread, that point remains true and factual - LLMs are not a blackbox to the extent that brains are.
> How can you know that? We haven't gotten sentience from the modeling we are doing at the moment so I am wondering why you are so confident that what we already know is sufficient to form an accurate model of brains as regards to consciousness.
Because we can model the neuron. We’ve done it. We can replicate the exact affects of a single neuron. By induction we know we can do it for a network of neurons and thus the human brain. The only reason why it’s not done yet is a scaling issue.
Also we don’t know if we done it with an LLM yet. The LLM uses neurons in a very different configuration then humans but we can’t know if that is conscious. The key is we don’t know but the output of the LLM is remarkably similar to what a conscious being would say so all evidence points to yes. It’s weak evidence… but it’s also the only evidence.
> Well what's the point of bringing up the modelling of a brain in conversation about mechanically replicating the brain via pen and paper? If a replica is not necessary and you have no idea if the macro model we have is sufficient, what's even the point of this thread?
Modeling the entire brain means you’ve modeled consciousness. But if consciousness is a subset of the brain then we don’t need to model the entire brain. A replica is not necessary but it is one path forward to modelling consciousness. If a replica was the only way we wouldn’t even be having a debate on whether an LLM was conscious because an LLM is obviously not a replica.
> My original point was that upthread poster said we can mechanically (i.e. without any thought, just repeating a process by hand) produce exactly what an LLM produces, while we cannot do the same for a brain (rat or human is not relevant).
Right and we don’t know whether what we reproduce by hand represents consciousness. We don’t know if what we emulated with the blue brain project of the rat brain represents consciousness. We don’t know if a rat is conscious either. The possibility is quite likely though because the output of both the LLM and the rat conform with our intuitive notion of what consciousness is.
> I feel that even after all the content in this thread, that point remains true and factual - LLMs are not a blackbox to the extent that brains are.
The blue brain project shows that the brain is not a black box as much as you think it is… it can be copied and stored and observed in a computer and we can emulate signaling traveling through the brain. Although we haven’t done this for the human brain we can logically infer this possibility because it’s been done for the rat brain and the human brain is just the rat brain scaled up and connected differently.
This is the same amount of understanding we have for the LLM. We can copy it and store it and observe it and send signals through it.
Both are black boxes in the sense that the sheer complexity of what is observable cannot be comprehended. Why do we need source code for a binary in order to modify it? Why do we not understand encrypted data even though we have it our hands and can observe every facet of it. Having and being able to manipulate the data raw does not mean we comprehend the data. The lack of comprehension IS the black box.
Something does not have to be a literal black box in order for us to use the term “black box”. In fact black box is a misnomer. Better to say we don’t understand how an LLM works anymore than we understand how the human brain works. The godfather of AI… the one who kick started the revolution of deep learning in the last decade has this to say about AI:
In the same vein, is American Society already not conscious? The only difference is that it doesn't output a coherent stream of words that individuals can understand. It does however, act and react on its level (a nation state)
Also none of Aristotle’s exoteric works is extant. All we have are dry, boring lecture notes. Cicero said his public works were a “golden stream of speech” and its all lost. So I don’t see how you’d build an artificial Aristotle when we don’t have any of his polished works meant for the public surviving. Plato would be a better option, since his entire exoteric corpus is extant.
I must have missed the brief somewhere, but there was/is a very clear trend to replace the default male pronoun for gender neutrality with the female pronoun she. Just recently I noticed this in Judea Pearl’s Book of Why. When and why did this start happening? It feels so forced and unnatural. You can sense he’s trying to kiss someone’s ass or appease an authority. At least mix it up a bit at best if you truly give a crap.
I feel like there are clear signs, but either people have cognitive blind spots or are just obstinate. For example, you hear people complain that they've been for a bajillion interviews and still don't get hired (hint: the problem is you), or they're always single even though they go on countless dates (hint: the problem is you) or they're overweight and can't lose weight no matter what they try (hint: the problem is you). Maybe an inability to introspect yourself in an objective way? Maybe a deep seated belief that the problem cannot actually be _you_, it must be an external factor, so you seek that. Maybe you're not being gaslit, maybe the ever-present smell of shit really does emanate from under your shoe.
I don’t actually even know what people are hinting at when they say that LLMs replace the need for building custom models. Regression models? People are using LLMs instead of say building a Bayesian hierarchical model? That’s not possible. Time series modeling using an LLM? Also ridiculous. Recommender systems? Ok maybe, still utterly ridiculous and abysmally slow.
For anything NLP sure, it definitely wins. However, I’ve just recently used some big fancy OpenAI model to actually just label thousands of text data for me, just so I could build a classifier with CatBoost. Guess what, inference speed is at a guaranteed sub 100ms and it costs $0 in tokens. The”AI Engineer” solution here would be just run every classification request through an LLM.
AI Engineering is going to have the same problem we had when Data Science as a term arrived and you had every Statistician saying they’re just re-inventing everything that exists in statistics, poorly.
You're right. For years the real impediment to "AI" products at many companies was the sheer crappiness of ML frameworks which were built by and for grad students, not professional engineers.
When LLMs appeared it was just so much easier to use then as an uber model and leave behind the training and inference infrastructure (if you can even call it that).
Now that LLMs can code I expect we'll be coding up custom model pipelines more and more... but only when we stop subsidizing LLMs.
Like, bro, do you think 5.x is a drop in replacement for 4.1? No it obviously wasn’t, since it had reasoning effort and verbosity and no more temperature setting, etc.
There’s no way you can switch model versions without testing and tweaking prompts, even the outputs usually look different. You pin it on a very specific version like gpt-5.2-20250308 in prod.
In The Laws you find: strict supervision of marriage age, mandatory procreation windows, state monitoring of reproduction, penalties for bachelors, public scrutiny of household conduct, drinking regulation, limits on wealth and inheritance,
formal theology enforced by law, criminal penalties for impiety, special prisons for “atheists”, etc. it goes on and on. The Laws makes The Republic look like Disneyland to be honest.
Not sure how that's possible. Laws may have a lot of strict marriage laws, but in Republic there is no marriage, the state assigns you sex partners in a rigged lottery, and requires the participants to be wearing masks so they can't form anything like an emotional bond. Really.
Absolutely nuts, I feel like I'm living in a parallel universe. I could list several anecdotes here where Claude has solved issues for me in an autonomous way that (for someone with 17 years of software development, from embedded devices to enterprise software) would have taken me hours if not days.
To the nay sayers... good luck. No group of people's opinions matter at all. The market will decide.
I wonder if the parent comments remark is a communication failure or pedantry gone wrong, because like you, claude-code is out there solving real problems and finding and fixing defects.
A large quantity of bugs as raised are now fixed by claude automatically from just the reports as written. Everything is human reviewed and sometimes it fixes it in ways I don't approve, and it can be guided.
It has an astonishing capability to find and fix defects. So when I read "It can't find flaws", it just doesn't fit my experience.
I have to wonder if the disconnect is simply in the definition of what it means to find a flaw.
But I don't like to argue over semantics. I don't actually care if it is finding flaws by the sheer weight of language probability rather than logical reasoning, it's still finding flaws and fixing them better than anything I've seen before.
I can't control random internet people, but within my personal and professional life, I see the effective pattern of comparing prompts/contexts/harnesses to figure out why some are more effective than others (in fact tooling is being developed in the AI industry as a whole to do so, claude even added the "insights" command).
I feel that many people that don't find AI useful are doing things like, "Are there any bugs in this software?" rather than developing the appropriate harness to enable the AI to function effectively.
I think it’s just fear, I sure know that after 25 years as a developer with a great salary and throughout all that time never even considering the chance of ever being unemployable I’m feeling it too.
I think some of us come to terms with it in different ways.
I used to sometimes get stuck on a problem for weeks and then get a budget pulled or get put on another project. Sometimes those issues never did get solved. Or have to tell someone sorry I don't have capacity to solve a problem for you. Now a lot of that anxiety has been replaced with a more can do attitude. Like wouldn't being able to pull off results create more opportunity?
reply