We Don't Understand Neural Networks At The Algorithmic Level

The largest ongoing debate about AI is “Are Large Language Models (LLMs) intelligent?” That makes sense, at least: the evidence is ambiguous and the stakes are high. Far more confusing is the disagreement on whether we understand LLMs - how can scientists studying a topic not broadly agree on how much we know about it? Yet respected academics have conflicting opinions, and it is worth seeing why, because it underlies the question of intelligence.

On one side, prominent figures, scientists among them, have described AI models thus:

powerful digital minds that no one – not even their creators – can understand

Skeptics are right to scoff at the marketing hype of “powerful digital minds”, but many scientists agree that our understanding of LLMs is very limited. Here is a more subdued example:

Language models have become more capable and more widely deployed, but we do not understand how they work.

The other side of this debate is presented well in a new preprint by Olivia Guest, Nancy Abigail Nuñez Hernández, and Mark Blokpoel, “Understanding Artificial Neural Networks: Mysterianism about Known Mechanism is Mysticism”. Disagreeing with the above two quotes, which they provide as examples, they write:

We do know the mechanistic structure of [artificial neural networks (ANNs)] because we designed and built them. We also do know their functional role (what they are for) as well as the mathematical function they are asked to approximate (map inputs to target outputs).

As they correctly say, there is a lot that we know. But is it enough?

They also write this:

ANNs, as turbo-charged statistical models [..] can only but provide correlations.

This position is, I think, representative of a particular line of AI skepticism, related to the “Stochastic Parrot” and “Blurry JPEG of the Web” metaphors. There is a lot of truth in those views! However, this preprint is an example of how they can be taken too far: we do not fully understand ANNs, and in part that is because they can do more than provide correlations.

Here are the main points I’ll be making in this post:

• Even though we understand ANNs’ goals at the top level, and their math at the bottom, we don’t understand the algorithms in the middle.
• This is so despite the simple math in each artificial neuron. Enough such simple units are provably Turing complete: capable, in principle, of any algorithm. There is a vast number of things that ANNs could be doing internally. Scientists are still figuring out which.
• The preprint, like Ted Chiang’s “Blurry JPEG of the Web” piece, views ANNs as storing data, a compressed version of the training set. Still, we must ask - as Chiang does - how do ANNs compress? And that question brings us back to algorithms.

I am not saying anything new here, but I think it’s worth seeing this all together, and in the context of whether we understand ANNs.

(Note: LLMs are ANNs, that is, large language models are artificial neural networks. I’ll mostly use the term ANN, following the preprint, except where the distinction matters.)

Levels of analysis

The cognitive scientist and neurobiologist David Marr defined three important levels of analysis:

• Computational: What the system aims to do at the highest level, what tasks it is built to solve.
• Algorithmic / Representational: How, in an abstract sense, the system represents information, and how it manipulates those representations to complete its tasks.
• Physical / Implementation: The actual details of how the system is built, at the lowest level.

Tragically, David Marr died at a young age in 1980. That was before computers were ubiquitous, but today, computers are probably the easiest way to explain his levels. Let’s focus on one type of computer program, a chess engine, which plays chess. Today, chess engines often use ANNs, but for now I just mean a “traditional” engine - some ordinary code that people wrote, that plays chess (most engines were like that until a few years ago). Here is what Marr’s levels mean for such a chess engine:

• At the computational level, a chess engine plays chess. Given a board, it picks the next move. Its goal is to pick the best move so as to eventually win the game.
• At the algorithmic level, a chess engine picks the next move using some specific approach.
  • It may have a list of memorized positions and outcomes, which is useful at the start and end of a game. Looking at possible moves, it sees which are memorized, and picks the one with the best outcome.
  • It may use calculation. While it is impossible to calculate all possible moves to any useful depth, heuristics can focus on promising ones (according to some rules, like checking all captures), while estimating the outcomes (according to some other rules, perhaps giving knights 3 points and rooks 5). It then picks the move with the best estimated outcome.
• At the implementation level, a chess engine is written in some programming language like C++, Rust, or Python. It is compiled to binary code that runs on a physical CPU and memory.

Note how many options there are at the algorithmic level. Some are more robust, some less. Some are more sophisticated, some less. Some are more similar to how humans play chess, some less. Some can defeat top human player Magnus Carlsen, some can’t.

Of course, Marr’s three levels are a simplification: even in the bottom level here, we have both human-readable code and the physical CPU that runs a binary version of that code. That is, each level has sub-levels. And sometimes the boundaries between them are not crisp. Still, this is a very helpful way to look at things: the goal of playing chess can be achieved using many algorithms.

The algorithmic level matters

The preprint authors claim that ANNs are not “outside our current expert understanding”. In support of that, here again are the things the authors say we do understand about ANNs:

We do know the mechanistic structure of [ANNs] because we designed and built them. We also do know their functional role (what they are for) as well as the mathematical function they are asked to approximate (map inputs to target outputs).

Indeed, in terms of Marr’s levels, we certainly understand ANNs at the top and the bottom. What remains to consider is the middle level, the algorithms and representations.

Let’s leave ANNs for the moment and return to our chess engine, where things are simpler: it is just some software that people wrote.

Say we want to beat the chess engine, with money riding on the game. We understand it at the highest level - it plays chess. Suppose that we also have access to it at the lowest level: it is running on a computer right in front of us, and we are allowed to use the keyboard and mouse and so forth. Does this help us defeat it? Well, given it is a machine, we can pull the plug if the game goes badly. But if we want to beat it fair and square then we should look at the algorithmic level in the middle.

Say that we know the chess engine only calculates up to a depth of 3 moves. We could strategize around that, because it will not see threats that take 4 moves to materialize. In other words, different algorithms have different Achilles’ heels - and knowing them can make the difference between winning and losing.

Note that even if we were given access to the chess engine’s source code, we would try to infer higher-level things from it. That is, we might scour the code for hints about what depth it searches to. Or, we could test the machine’s observable behavior and measure its average calculation depth that way. Either way, we are aiming for an algorithmic-level understanding.

We are not talking about anything metaphysical or mysterious here.¹ We can measure calculation depth with human players too.² (The average depth Magnus Carlsen can calculate to is far better than my own, for example.)

Calculation depth may seem like a small matter. But far greater differences are possible: if we know that the engine only uses memorized moves - without flexible calculation - then we can defeat it easily, because there are too many chess positions to memorize. All we need to do is play odd-enough moves to wind up in territory it has nothing memorized for, and it will blunder.³ This would be a guaranteed way to win (and perhaps make a lot of money).

There are, of course, situations with far higher stakes. Consider an app that helps people count carbohydrates and vitamins.⁴ This can be very important for diabetes and other medical conditions. The app lets you pick the things you eat from a list and add estimated portion sizes. It then prints out nutritional values.

The app could be doing this in many ways. Does it use an expert-curated database of nutritional values? Does it take into account interactions (some nutrients inhibit the absorption of others)? Unless the app’s creators document such things (and we trust them), we would need to carefully test the app or piece things together from its binary code. These internal details could have medical implications, so I would not want to use such an app without knowing them!

Clearly such algorithmic-level details - in Marr’s middle level - matter. Why do the preprint authors focus on the top and bottom levels instead? For good reasons: we do not need to be aware of everything about a system in order to understand it. Also, nothing changes if the app crashes sometimes: we do not need to be able to predict everything about a system we understand. I agree on both counts: yes, there is a fundamental, important level of understanding that we have just by knowing the top and bottom levels of a system, its goals and its implementation. The details in the middle are just details (which, often, we can’t know all of anyhow).

And the preprint authors argue this for a good reason. AI fans sometimes suggest that there is something mysterious about ANNs that we do not or even cannot understand, something as special as human thought - perhaps even the very same thing itself. We do not understand human consciousness, and if we do not understand ANNs… perhaps they are conscious, too? This is, of course, wildly speculative.

It is fine to speculate about these things. Some of the smartest people in the world have debated the connections between algorithms, understanding, mechanism, consciousness, and meaning - but they arrived at no clear philosophical conclusions. So it is best to separate the metaphysics from the obvious practical reasons to investigate chess engines and medical apps at an algorithmic level.

And, at that practical level, the more algorithmic details we learn about a chess engine or a medical app, the more we understand it, with potential impact to our finances and health. “Understanding” is not a binary, but a matter of degree, in the way the term is commonly used.

And all this applies to ANNs as well. When we build an ANN, we pick the low-level mathematical details, and we provide it examples of what we want - for a given input, we tell it what the right output is. But we don’t tell it how to produce that output. When training an ANN for chess, we don’t tell it whether to use memorization or move calculation. We don’t suggest what depth to calculate to or which Achilles’ heels it should or should not have.⁵ We must investigate the ANN to figure out which algorithms it uses.

But, wait - do ANNs use algorithms? Yes.

ANNs can implement any algorithm

Let me re-quote the preprint authors:

ANNs, as turbo-charged statistical models [..] can only but provide correlations.

This is a strong claim. Let’s consider it.

Philosophically, we can debate whether showing observations to a computer or a human limits them to correlations, but we will not find any clear answer. Instead, let’s focus on the practical and scientific aspects here.

Can a laptop do more than provide correlations? I think so, since we can play Elden Ring on it or watch cat videos. Even if we focus on software that returns an output for an input - like trained ANNs do - then a computer can run a mathematical proof-checking program or calculate chess moves. Surely those go beyond correlations? If so, what are ANNs missing?

Nothing, it turns out, from the standpoint of computer science. This might be surprising! While it is true that ANNs often do poorly at tasks which laptops do with ease, like adding large numbers, mathematical proofs show that in principle ANNs have the same computational power as laptops. Specifically, ANNs are universal approximators, meaning they can compute any function between inputs and outputs, to any precision (if large enough).⁶

ANNs are also Turing complete,⁷ meaning they can compute anything any computer can (if large enough, and when run in a loop⁸). ANNs can calculate chess moves and implement any algorithm, just like laptops.⁹

When Marr talked about algorithms, he did not only mean interesting ones like quicksort and A* that college professors teach on whiteboards. Any way of processing information is relevant here, which also includes very simple things like memorizing examples. But we do have plenty of scientific evidence of ANNs doing interesting things, algorithmically: this is the focus of the field of neural computation, and in recent years, work along those lines that focuses on LLMs has been termed mechanistic interpretability. A great deal has been discovered inside ANNs, things like accurate internal representations¹⁰ and complex analyses¹¹.

Chess is particularly well-studied. ANNs represent the board internally¹² (a “world model”) and can compete at a level comparable to top human players.¹³ Systems using ANNs can learn to play chess at a superhuman level, even without human guidance or data.¹⁴ And, remarkably, the things chess ANNs compute after training¹⁵ often correspond to human concepts around chess, things like piece values and “can the queen be captured?”

That ANNs process information in interesting ways is perhaps not surprising. When an ANN plays chess at a high level, it does so after being trained on far more games than it can memorize, so simple recall is not enough. Likewise, when an LLM speaks so fluently that it passes the Turing Test, then large as it is, it was trained on a far greater volume of text:

To predict general patterns and regularities in that amount of data requires some form of information processing. It may be more or less sophisticated, but there is something to investigate, algorithmically.

To be clear, ANNs can only implement any algorithm in principle. The math in ANNs can implement quicksort and A* and chess calculations to a depth of 5 and anything else. But that does not mean that ANNs do so in practice, since a particular ANN may not be large enough. Furthermore, even given a sufficiently large network, it is an open scientific question whether ANN training actually searches the space of all algorithms. That we do not know the answer to this fundamental question - whether ANNs reach their theoretical potential, or not - is one reason why ANNs remain outside our current expert understanding.¹⁶

We have no reason to think any of this is an unsolvable mystery. Papers are constantly being published that show significant progress, like the examples in this section. It is just that, currently, much remains unknown.

Logistic regression

One of the major arguments the preprint makes is a comparison to logistic regression:

$p(x)=\frac{1}{1+e^{-(x-\mu )/s}}$

We don’t need to get into the actual equation, but this is an important statistical model which uses simple math to predict data based on examples. The authors correctly say that we understand logistic regression. Furthermore, the authors point out that ANNs are composed of many elements - many artificial neurons - each of which does basically the same math as that logistic regression equation, hence ANNs and logistic regression are closely related. They write:

If we do indeed understand how a single unit in an ANN works, what is the difference when there are many hundreds or thousands? Does understanding a single unit not imply mechanistic understanding of more than a single unit? Is ‘mechanistic understanding’ not exactly this form of understanding?

The answer is that researchers from the fields of neural computation and mechanistic interpretability also mean the algorithmic level,¹⁷ and, at that level, there is a fundamental difference between a single unit and a group.

Logistic regression is not Turing complete. It cannot approximate any function like ANNs can. The math in logistic regression is just too simple for that, it turns out.

This is a classic result in neural computation: a single neuron is not a universal approximator. Even multiple neurons in a single layer are not - another (hidden) layer is needed. So, yes, logistic regression is equivalent to a single artificial neuron, but a single neuron has far less power than a group.

And that difference in power is incredibly vast. Being Turing complete means the ability to do anything that the smartest software engineers could possibly make a computer do. And more: there is no reason to limit ourselves to things humans have thought of. The space of all algorithms includes every possible rule, pattern, logical inference, approximation, mathematical proof, computation, all combinations of those, and more.

As mentioned before, this is in theory. But, even if ANNs search only a small region in the space of all algorithms, plenty of evidence shows that area is far, far larger than what logistic regression covers. For example, a single logistic regression can’t even identify legal moves in chess, much less play competently, but ANNs have expert ability there.

I am certain that the preprint authors are aware of ANNs’ Turing completeness and universal approximation and everything else I said above. My point is not that they have made some kind of oversight. I believe the root of the disagreement is in how the term “understanding” is used: for them, understanding a single logistic regression implies the understanding of many of them. Of course, in a way it does: understanding one billiard ball is enough to understand a game of pool or even larger numbers of them as in the figure below, even if we can’t predict every movement because of complexity or because of details like an uneven pool table.

But there is actually more to understand even there. It is uncontroversial in physics to say that statistical mechanics improved our understanding of the world, teaching us things about how large collections of billiard balls behave.¹⁸ Sometimes understanding a single unit does not imply understanding of more than a single unit.

And it is uncontroversial among researchers in the fields of computer science, neural computation, and mechanistic interpretability to say that investigating algorithms improves our understanding of large systems of artificial neurons. Marr stressed the importance of that, and we have seen why it makes sense: logistic regression does one thing, and we understand it. ANNs can do anything, in theory, and in practice it is very hard to find out what.

I don’t think the preprint authors’ definition of “understanding” is wrong. I think there are other valid ways in which scientists use that term.

Science, ANNs, and Marr’s middle level

So far we mostly focused on practical reasons for understanding the algorithms that ANNs use: it helps us find Achilles’ heels, etc. We also briefly mentioned scientific reasons. Let’s look into that in more depth, using this prompt and LLM response:

“Crevyzik” is a bizarre character name (as befits high fantasy). Surely it does not appear in any text the LLM was trained on. But the LLM responds well here, and would do so no matter what random name you give.¹⁹ How?

Do LLMs parse sentences into parts of speech like verbs and nouns? Using that representation, they could infer which words were character names. Since names are arbitrary to a large extent, operations on that representation could then work the same way no matter what the names are - names would be “variables” as in math or programming. This would explain how LLMs are so consistently good at such prompts, and it would also show that they reason properly about character names at least. But is any of this true?

There is some²⁰ evidence²¹ of grammatical ability in LLMs. It does appear that LLMs process language in a general way, beyond superficial correlations.²² However, even if so - and this is still contested! - we don’t know any of the details, because there are many possible ways to parse text into parts of speech and so forth. This is the sort of thing scientists want to understand - and still don’t.²³

This matters for practical reasons. Some text-processing algorithms are more robust, some less. Some are more sophisticated, some less. Some are more similar to how humans reason, some less. Some can summarize (or translate or rewrite) in useful ways, some can’t.

Of course, if you have no interest in using ANNs, this won’t matter to you. And that is fine - ANNs can’t help everyone or every field. But for those considering ANNs, it is of great practical importance to know which algorithms they use. And it is science’s job to figure out how ANNs work - who else is going to do it?

And this also matters scientifically in a direct sense. Scientists have found ANNs useful as tools, in areas like protein folding and math. Humans have also learned new tactics in chess²⁴ from the ANN component of the AlphaZero chess engine. When an ANN solves a problem, it uses some algorithm - potentially one that humans never thought of. The algorithm may be interesting, and its results as well.

I want to stress that there is nothing metaphysical going on here. That we can learn new things from ANNs is not mysterious - we can also do so from monkeys banging on keys, after all, since they will (eventually) print out a description of every possible algorithm. But ANNs do this in a timeframe that is actually useful to us, unlike the monkeys - and no other technique has come close to this achievement.

Data

The preprint authors do accept that there is something we do not know about ANNs:

What we do not understand is the data compressed or stored, lossily or losslessly, inside the model.

The data in the model comes from the world, and there are things we do not understand about the world, after all. To make this point clear, they compare ANNs to databases:

[databases and ANNs] do not embody an understanding of the world, nor the phenomena within it [..] What they do is in fact exclusively house data

I find this view fascinating! The idea that it is the data we need to understand, and not the algorithms the model derived from the data - that is, honestly, a perspective so different from mine that I struggle to follow it. My own, of course, is what I presented so far, the ‘algorithm’ framing which is common among neuroscientists and computer scientists (my own fields).

But even though the ‘algorithm’ and ‘data’ perspectives seem at odds, I believe they are in fact equivalent, for the following reasons.

Note, first, that the same data can lead to different ANNs. For example, training larger ANNs on the same chess games often leads to better performance¹². But how can the same data lead to different ANNs? Because, in the case of large amounts of data, it is not simply stored - it must be compressed, as the preprint authors say.

There are many ways to compress data: compression uses algorithms. And, as we saw, the math in ANNs is Turing-complete, meaning they can use any conceivable algorithm to do so. Even if we see ANNs as databases, they are databases with very complex mechanisms that we haven’t completely figured out yet. Compression is part of ANNs, and we do not understand that compression, so even in the ‘data’ view we do not fully understand them.

Everything I wrote before can be reframed in this ‘data’ view, without anything fundamental changing. For practical reasons, to learn the Achilles’ heels of an ANN, we still need to study it: whether it calculates 2 moves ahead or 4 on average is determined both by the data and how it is compressed. And the compression in ANNs is scientifically interesting because it can use any algorithm, and we don’t know which in advance - the data does not tell the ANN how to compress it, after all.

Why do some ANNs play chess or summarize text better than others? More data can help, a point the preprint authors make. However, architectural differences matter too, by influencing how the data is compressed (e.g., convolutional neural networks vs. transformers).

How can an ANN be better than humans at chess? The ANN may have simply been shown more games than any human ever has.²⁵ However, recall that no amount of memorization is enough to play chess well, so it must also be compressing all those games into a form that handles novel situations properly too. Notably, no other machine learning technique - or any other software, for that matter - handles novelty so well across a broad range of tasks, not only chess but vision and text and more. That ANNs do so while also responding efficiently to queries in a reasonable time is even more remarkable. Something very interesting is going on in the compression here!

While interesting, this is not mysterious. We can make it very concrete with an example: imagine a chess ANN that evaluates a board by counting how many pieces each side has, giving knights 3 points and rooks 5 and so forth. This is a simple algorithm that works well in many cases. We can see this as a form of compression, a very lossy one in fact, as any two boards with the same pieces end up compressed to the same thing, no matter how the pieces are positioned or moved.²⁶ Of course, this is just one algorithm - or one form of compression - that a chess ANN might be using. Even in the ‘data’ perspective, we must consider algorithms.²⁷

On metaphors

In Ted Chiang’s 2023 piece on ChatGPT, he writes:

Think of ChatGPT as a blurry JPEG of all the text on the Web. It retains much of the information on the Web, in the same way that a JPEG retains much of the information of a higher-resolution image

Like the preprint authors, Chiang considers ANNs from the ‘data’ perspective. But there is a very large difference. The authors mention compression but do not consider it in detail. Chiang does, and correctly points out that there are better and worse ways to do so:

To grasp the proposed relationship between compression and understanding, imagine that you have a text file containing a million examples of addition, subtraction, multiplication, and division. Although any compression algorithm could reduce the size of this file, the way to achieve the greatest compression ratio would probably be to derive the principles of arithmetic and then write the code for a calculator program. Using a calculator, you could perfectly reconstruct not just the million examples in the file but any other example of arithmetic that you might encounter in the future.

In other words, when one recognizes the principle behind something, its data can be compressed very well. Compression therefore might indicate some level of understanding, at least in theory. Chiang’s conclusion is that LLMs in 2023 did not show true understanding, but he kept an open mind.²⁸ And that is the right thing to do. LLMs can be seen as databases, but we shouldn’t conclude that they are as unintelligent as ordinary databases are.

Another popular metaphor in this space can be taken the wrong way as well, the Stochastic Parrot:

“Contrary to how it may seem when we observe its output, an [LLM] is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.”

I would quibble with the word “haphazard”,²⁹ but otherwise this is useful and accurate. Imagine we ask an LLM this:

What did you have for breakfast?

If it responds compellingly about having had orange juice and cereal, we still can’t take it seriously, because it did not eat anything. Humans have a tendency to interpret fluent text as meaningful, but LLMs are still only machines that predict words.

And LLMs certainly use probabilistic information about their data in order to produce that fluent text, as the Stochastic Parrot metaphor states. But this can be taken too literally,³⁰ as if LLMs only provide correlations and nothing more. Here this topic is intertwined with the one before, and with the question of intelligence: if LLMs only do statistics, then they are really just databases that use statistics to compress their data. The details are not interesting. (Are statistics ever interesting?) Only the summarized data is.

LLMs do use statistics - during their training and often after - but they might also be doing more. Computer scientists have proven that combining many logistic-regression-like correlations leads to Turing-completeness, which means ANNs could be using any algorithm whatsoever to compress their data.³¹ As of 2026, those algorithms remain an open scientific question, so we do not understand ANNs at Marr’s middle level.

Conclusion

ANNs may be scientifically interesting and sometimes useful, but also have risks and cause harm. I agree with the preprint authors on many things,³² because AI skepticism has a very important role today, both in outlining the dangers and clarifying the misconceptions. But our skepticism must be accurate.

When startup CEOs say “We don’t understand LLMs”, that can be a marketing line, a cynical way to boost the valuation of their companies. And when AI fans say “We don’t understand LLMs”, they might be justifying speculation about those machines being conscious or alive. The preprint authors are correct to criticize both cases. Yes, we do understand LLMs in important ways, despite ridiculous AI hype.

But when scientists say “We don’t understand LLMs”, they are being humble in the face of all that we really do not know about incredibly challenging questions in their field. There is a lot we don’t understand here.

I have suggested that we separate any metaphysical speculation about ANNs from the science. But, of course, an individual might talk about both. Nobel Prize laureate Geoffrey Hinton probably knows more about ANNs than anyone else - and he has also said that he believes AI models might be conscious. Is his metaphysical speculation related to the current gaps in the science? Perhaps yes, but perhaps no - I can’t say what is in his mind.

Putting Hinton aside, there probably are people who make inferences like this: “We don’t know how ANNs work, so they could be conscious.” If so, then an analysis like the preprint authors’ could explain why such a leap is unsound.³³ But it is the leap that is unsound, not the starting point: Hinton and other scientists are absolutely right to say that ANNs are outside our current expert understanding.

Acknowledgements

Thank you to Gian-Carlo Pascutto, Vincent Carchidi, SE Gyges, and Marcia Cohen-Zakai for helpful comments and suggestions on drafts of this post.

Comments

Feel free to respond on Bluesky.

Footnotes