Analysis shows self-attention stores over a million memories versus 64 in linear layers
A new technical post explains how transformer components function as memory stores, revealing vast capacity differences between architectural layers.
Key points
- Linear layers store roughly 64 memories with high interference
- MLPs increase capacity to approximately 2,700 memories
- Self-attention mechanisms can store over a million memories
A recent post on Lobsters AI argues that large language models function as associative memory stores, with capacity defined by interference-to-signal ratios. The author demonstrates how different neural network components handle recall and storage limits, providing heuristics for understanding model memory.The author explains that AI models remember facts by storing key-value pairs, with different parts of the network holding different amounts of information.The post derives capacity heuristics for linear maps, MLPs and self-attention using interference-to-signal ratios as the defining metric for associative memory performance.
What changed
The post reframes linear maps, MLPs, and self-attention as distinct types of memory storage. It introduces the interference-to-signal ratio as the key metric for determining how many key-value pairs a layer can hold before recall degrades.
How it works
The analysis treats a memory store as a function retrieving values based on key similarity. Linear layers accumulate memories via outer products, resulting in low capacity. MLPs improve both capacity and recall quality. Self-attention offers superior recall and potentially unbounded memory storage.
What the evidence means
The author provides specific capacity estimates for 64-dimensional keys. Linear layers hold about 64 memories. MLPs jump to roughly 2,700 memories. Self-attention mechanisms can store over a million memories, representing a massive increase in stored information density per layer.
Cost and market position
The post does not discuss vendor pricing, market share or implementation costs. It focuses exclusively on theoretical capacity limits and architectural heuristics. Product teams must infer that self-attention’s high capacity may justify its computational overhead, though no cost data is provided in the source.
Why it matters
- Engineers can estimate memory capacity per layer using interference-to-signal ratios
- The analysis quantifies why self-attention is preferred over linear layers for large datasets
- Understanding these limits helps teams evaluate architectural tradeoffs between capacity and recall
- The post provides heuristics for predicting how much information each component can store
Topics: transformers, memory capacity, self-attention, neural networks, AI architecture
Updated
Then / now / next
Models were viewed primarily as pattern recognizers rather than explicit memory stores.
A new analysis frames transformer components as associative memory systems with quantifiable capacity limits.
The post provides heuristics for estimating memory capacity, but does not predict future architectural changes.
Story timeline
- Then Models were viewed primarily as pattern recognizers rather than explicit memory stores.
- 14 Nov 2026 You could have invented transformers from memory
- Now A new analysis frames transformer components as associative memory systems with quantifiable capacity limits.
- Next The post provides heuristics for estimating memory capacity, but does not predict future architectural changes.
The strongest doubt
Counterpoint from the sources
The post acknowledges that storing infinite facts is useless if recall fails, emphasizing interference as the limiting factor.
Did this article give you the evidence you needed?
Your choice stays on this device. No public tally.Questions readers ask
How much memory can a linear layer store?
A linear layer with 64-dimensional keys can store approximately 64 memories before interference grows as large as the signal.
What is the capacity of an MLP?
An MLP can store about 2,700 memories with 64-dimensional keys, offering better capacity and recall than linear layers.
Why is self-attention considered superior for memory?
Self-attention has very good recall and can store a potentially unbounded number of memories, exceeding one million with 64-dimensional keys.
Connected stories
Reading receipt
- Linear layers store roughly 64 memories with high interference
- MLPs increase capacity to approximately 2,700 memories
- Self-attention mechanisms can store over a million memories
Still open: How do these capacity estimates change with different key dimensions?
Still open: Does higher capacity always translate to better real-world model performance?
