Crossover between mainstream deep learning and neuromorphic computing may help find useful applications for brain-inspired models and hardware, Jason Eshraghian, Assistant Professor at the University of California, Santa Cruz, told EE Times.
Spiking neural networks (SNNs) have been a cornerstone of neuromorphic computing schemes, but commercial deployments have been limited to small, ultra-low-power networks, contrasting with today’s LLMs, which continue to grow in size. While SNNs are seen as distinct from mainstream deep learning approaches, which today focus heavily on very large transformer models, there are some areas of intersection where the two fields can borrow from each other.
For example, until recently, SNNs were difficult to train effectively due to spikes’ non-differentiability. Today, this has primarily been addressed by borrowing techniques used in deep learning – specifically, a combination of backpropagation and quantization-aware training. (Backpropagation is widely credited for being the breakthrough that enabled deep learning networks of all kinds to scale.)
“We’ve built on [backpropagation] in the same way the deep learning folks have,” Eshraghian said. “We turned to backprop because it works, which seemed to be the most sensible way to get large-scale spiking neural nets.”
Eshraghian’s group at UCSC developed the generative language model SpikeGPT two years ago – a huge SNN (by neuromorphic standards) at 216M parameters. This model modified self-attention to reduce its complexity and produced competitive results with non-spiking models, while requiring one-twentieth the amount of operations when implemented on hardware designed for sparse, event-driven activations.
However, scaling the model to 1B produced only a marginal performance improvement, which Eshraghian attributes to using biologically accurate binary spikes.
Biologically accurate SNNs use neurons that accumulate incoming spikes until they reach a certain level. At this point, they fire their activation to the next neuron, after which the neuron resets. These spikes carry information in their timing only – their amplitude is fixed to 0 or 1, which Eshraghian points out is a huge amount of activation compression. To compound the problem, backpropagation works less effectively for encoding information into spike timing versus amplitude.
“There is various research that takes incremental and interesting steps [towards a temporal version of backprop], but they haven’t hit the point where we can comfortably say it wins,” he said, noting that techniques like backpropagation through time (BPTT) are nevertheless making progress in a challenging field.
Jason Eshraghian (Source: UCSC)
The answer, used in Intel Loihi 2 and elsewhere, has been to allow spikes to have limited amplitudes.
“I think people need to be comfortable with shaking the [neuromorphic] branding a little bit, being OK with the fact that binary spikes aren’t the perfect representation,” Eshraghian said.
As scale increases, it’s also harder to take advantage of some of SNNs’ key benefits. SNNs can leverage sparsity by avoiding memory accesses where operands are zero; however, this approach becomes challenging to scale once networks become large enough to require external memory.
“The moment you scale up, it gets pretty challenging,” Eshraghian said. “That’s not to say it’s impossible, it’s just challenging, but it does hurt the sparsity argument.”
Nevertheless, borrowing backpropagation from deep learning enabled the group to build bigger spiking networks, which has enabled new research directions.
Other work from the neuromorphic community, including the development of a compiler intermediate representation designed for neuromorphic models, has enabled easier and fairer comparisons between neuromorphic systems and today’s LLMs.
“I think that’s really important, be it for finding that we are potentially better in a specific application domain or it can show us much faster that we are not, so we can divert our resources towards something more productive,” Eshraghian said.
State Space Models
Given the challenges of scaling spiking neural networks, a better option could be to look at ideas that draw upon spiking, but aren’t spiking themselves, Eshraghian said, highlighting state space models as an example.
State space models were developed by Chris Eliasmith’s group at the University of Waterloo in 2018. They combine elements of control theory with recurrent neural networks (RNNs). The models themselves are a type of linear RNN – that is, they vary with time, but all the operations that happen through time are linear, so all states are known from the start. Their linearity means that state space models can be easily parallelized, which is crucial for training large networks on sufficient amounts of data efficiently. (Spiking neural networks are typically not linear through time since they “reset” themselves after a certain pattern of incoming spikes.)
“[Linearity] matters because large models need to train on a ton of data,” Eshraghian said. “When you’re moving into a space where you want to train on 45 trillion tokens to compete with DeepSeek, you can’t afford that sequential mode of processing. You need to parallelize your training loop, so you need to know what every state is going to be, independent of every other time step. That’s the value state space models have.”
Like SNNs, state space models don’t have to work completely separately from mainstream deep learning. The big LLM world has already discovered that elements from state space models can be combined with “big, fancy transformers,” especially during training, Eshraghian said.
“A ton of this was already being done by the spiking neural net community, but in a silo,” he said.
The challenge with state space models is that they don’t have the transformer’s ability to remember context; transformers require significantly more memory as more context comes in, but this ability to remember context is part of what makes them so powerful. This means state space models don’t deal well with long-range memory tasks, which is becoming increasingly important in the era of chain-of-thought and reasoning, Eshraghian said.
“I have a hard time imagining a world where [state space models] will work in isolation from transformers,” he said. “Unless all you need is auto-predict on steroids – if you don’t need a very complex model, they could definitely work for a certain set of applications, perhaps at the edge.”
Matmul-free
Hybrid models may be the way forward. Last year, Eshraghian’s group developed a stripped-down matrix-multiplication-free LLM, eliminating the most computationally demanding part of inference. This model combined elements of state space models with elements of transformers.
First, the LLM’s size was reduced by removing a significant portion (close to 50%) of its attention heads and compensating with additional fine-tuning.
Then, a couple of techniques were used to eliminate matrix multiplication. In general, matrix multiplication is used in two areas of transformers: the channel mixing layers (which are dense layers) and token mixing layers (like attention).
Matrix multiplication in the token mixing layers was eliminated by replacing it with stateful operators from state space machines. For channel mixing layers, switching to a ternary number scheme (-1,0,1) eliminated matrix multiplication, since multiplying by -1 is simply a bit flip, multiplying by zero is always zero and multiplying by 1 has no effect. (This addresses the compute required, but also the memory bandwidth required, Eshraghian said).
Any resulting reduction in performance was addressed partly by keeping channel mixing layers on either side of token mixing layers (this already exists in most transformers), and partly by exploiting forget gates, where information that isn’t seen as important is scaled to zero or close to zero to “forget” that information. In this way, more relevant information is retained for longer.
“Something’s got to give – if we simplify our operations [remove matrix multiplication] then you have to bulk up something else,” Eshraghian said. “So we put a fair bit of attention into forgetting, and forgetting in a way that’s linear through time in order to ensure parallelism, to make sure we can train on a lot of data with a reasonable amount of compute.”
Experiments showed that the model scaled well, and versions up to 2.7B in size have been produced. These models showed performance improvements on a GPU (10x less memory used during inference vs the original model), with more significant acceleration on an FPGA implementation designed for ternary math. The team has also successfully ported it to Intel’s Loihi 2, a multi-chip implementation that used 77x less energy than a server-class GPU. Dedicated hardware, including ternary math accelerators combined with clock gating, for example, could further benefit performance, Eshraghian said, noting that the group is currently investigating fine-grained sparsity for this model.
Brain inspired
Ultimately, the term “neuromorphic” is rather nebulous, encompassing ideas that aim to emulate the brain, as well as those that take even the vaguest of inspiration from it.
Sticking to the broadest definition, most commercial AI models and systems already use neuromorphic ideas, whether they brand them that way or not, Eshraghian pointed out. He cited anything that uses low-precision maths (for example, Nvidia’s latest GPUs introduced FP4), and techniques like mixture of experts (MoE), which one could consider a form of sparsity.
“Maybe it would be nice for [mainstream use of these concepts] to bring a bit of attention and resources our way, but it is what it is – at least it’s useful, and from an academic perspective, I’m happy with that,” Eshraghian said. “You get a lot of people telling you not to work on [neuromorphic computing], because progress will happen by people far better resourced than you, and that innovation will happen in a handful of frontier labs and everyone else will get left behind.”
The industry often adopts, rebrands, and continues to evolve ideas developed in academia that prove useful. Therefore, knowing when to “take the win” and move on to something new is a critical part of research. This is getting harder, Eshraghian said, since in the LLM space this process is happening much more quickly than for other fields of research.
However, as AI begins to mature, his feeling is that innovation will likely be forced to go up the stack.
“We might be coming to a world where low-level optimisations are few and far between, so it becomes harder to contribute at that level of abstraction, and we need to start thinking more behaviourally,” Eshraghian said.
“[That said,] anyone who says they know where AI is going in five years is full of shit,” he added.