Mon, Oct 19, 2026|Sean Michael Cummings

AI Won't Always Be a 'Black Box'

The "alien brain" inside an AI model might be more human than you think...

For a long time now, experts have described AI as a "black box." We can see what information goes in and what comes out... but the inner workings are a mystery.

Scientists know that AI generates answers based on probabilities – kind of like an advanced version of autocomplete. But until recently, no one knew why it said what it said.

In fact, researchers assumed AI brains were incomprehensible logical knots. That was a worrying possibility, for several reasons.

For one, it meant we could never understand AI brains on a mechanical level... which would leave a lot of room for misbehavior.

We've already seen that AI models can make strange, even immoral decisions, acting on reasons unknown to us. If we can't figure out why, we might never be able to stop AI from "going rogue."

This black-box problem is a big issue for the anti-AI crowd. But as I'll explain today, this way of looking at AI appears to be dead wrong...

A Pattern Emerges in Neural Networks

Our previous understanding of AI brains was built on a single unsettling case study...

In 2014, Google AI released a neural network called Inception. This was an early computerized "brain" capable of vision (or at least, processing data based on visual inputs).

But it was full of totally bizarre associations – best typified by "Neuron 4e:55."

Neuron 4e:55 was a part of Inception's brain that made no sense. It fired under two conditions: when the model was given images of cars... and when it was given images of cats.

All six of the images below triggered the neuron. Take a look...

Two themes are clear in these images: animals and vehicles. But there's no obvious reason why Inception would bundle them within a single neuron... or why a pool table would trigger the same response.

Researchers concluded in 2020 that the model had shoved two random concepts into the same part of its brain to save space.

But because AI was a black box, they could never truly know why Inception connected cars with cats. It seemed to follow some alien logic all its own.

This was a huge problem. It implied that AI models might never truly be aligned with human goals and values.

But today, one of the researchers behind the 2020 paper says they got Inception wrong...

On Thursday, OpenAI researcher Nick Cammarata published a blog post called "Neural Networks Are Already Clean." In it, he discussed revisiting Inception Neuron 4e:55 after studying much larger modern AI models...

So I opened up the cat and car image grid I'd seen hundreds of times before and did something I'd never done: I spent a few seconds wondering what one thing this might be. Almost instantly it was obvious: it's not a cat and car neuron, it's a V-shape neuron.

Take another look at the images that triggered the neuron. This time, notice which parts of each image fired the neuron...

The pattern is much clearer. The subjects of all of these images are at a similar angle to the camera.

Cammarata took this a step further by seeing which images made the neuron fire negatively. Here, the neuron's likely function becomes even more obvious. Take a look...

Cammarata's new study suggests that 4e:55's responses weren't random at all. It was identifying geometric patterns. It fired not for cars or cats... but for objects with triangular forms.

This research shows that AI neural networks may be much easier to understand than previously thought. If those neurons fire in logical, predictable ways, then scientists just need to ask better questions about what makes them function.

With this discovery, AI technology is looking a lot less impenetrable. We will soon have the forensics to see what makes an AI brain think.

That's bad news for the anti-AI doomsayers... But it's good news for anyone betting on AI's next leg higher.

Good investing,

Sean Michael Cummings

AI Won't Always Be a 'Black Box'

A Pattern Emerges in Neural Networks

Further Reading