Skip to main content
Be Bitwise

Spiking Neural Networks

Neuromorphic Computing · Frontiers track

9 min read

In this module

The Third Generation of Neural Networks

Artificial neural networks have stepped through three generations. First came simple threshold units — perceptrons with binary outputs. The second generation, today's deep learning revolution, uses continuous-valued neurons with differentiable activations (ReLU, sigmoid, softmax) trained by backpropagation. The third is spiking neural networks (SNNs), where neurons communicate with discrete spikes spread out across time.

SNNs are a different computational model. Time is a first-class dimension here, not something the network drags along by accident. That temporal axis unlocks capabilities conventional ANNs simply don't have, and it maps cleanly onto neuromorphic hardware.

The Leaky Integrate-and-Fire Neuron

The most common neuron model in SNNs is the Leaky Integrate-and-Fire (LIF) neuron. Despite its simplicity, it captures the essential behaviour of biological neurons:

  1. The neuron maintains a membrane potential V(t) that represents its internal state
  2. Incoming spikes are multiplied by their synaptic weight and added to V(t)
  3. The membrane potential leaks (decays towards a resting value) over time
  4. When V(t) crosses a threshold V_th, the neuron fires a spike
  5. After firing, V(t) is reset to a resting value (refractory period)
text
V(t)
^
| * ← spike fired (threshold crossed)
| / \
| / \ ← reset after firing
| * / \
| / \ / \___
| / · \___ ← leak (decay)
|/ \___
+─────────────────────────────→ time
↑ ↑
incoming spikes add to membrane potential

The mathematical description:

text
τ_m * dV/dt = -(V(t) - V_rest) + R * I(t)
if V(t) ≥ V_th:
fire spike
V(t) → V_reset

Where τ_m is the membrane time constant (controlling the leak rate), R is the membrane resistance, and I(t) is the input current from synaptic inputs.

Why the Leak Matters

The "leaky" part of the LIF neuron is what gives it a memory with a time horizon. Recent inputs hit harder than older ones, because the old ones have already decayed away. So the neuron picks up on temporal structure naturally — spikes arriving in quick succession are far more likely to push it over threshold than the same count of spikes spread over half a second. The leak time constant sets how long the neuron "remembers", typically 10–20 ms in biology.

Spike Coding Schemes

How do SNNs represent information using binary spikes? Several coding schemes exist, each with different trade-offs:

Rate Coding

The simplest approach: a value is encoded as the firing rate of a neuron over a time window. A value of 0.8 might be represented by a neuron firing 80 times in a 100-timestep window.

Advantages: Easy to convert between ANNs and SNNs. Tolerates noise well (a missed spike has little effect on the average rate).

Disadvantages: Requires many timesteps to transmit a value accurately. Slow and energy-inefficient (many spikes needed). Does not exploit the temporal precision that neuromorphic hardware offers.

Temporal Coding (Time-to-First-Spike)

A value is encoded by when a neuron fires its first spike. A higher value causes an earlier spike; a lower value causes a later spike (or no spike at all).

Advantages: Extremely efficient - only one spike per neuron per input. Fast - the output can be determined as soon as the first spikes arrive, without waiting for a rate to accumulate.

Disadvantages: Sensitive to noise and timing jitter. Harder to train than rate-coded networks.

Phase Coding

Information is encoded in the phase of a spike relative to a background oscillation. This allows multiple values to be multiplexed on the same temporal window.

Burst Coding

Short bursts of rapid spikes encode different information than single spikes at the same average rate. Bursts can indicate high-confidence signals and can trigger different synaptic plasticity rules.

SNN vs. ANN: A Concrete Example

Consider classifying a handwritten digit (the MNIST benchmark). In a conventional ANN:

  1. The 28x28 pixel image is fed as 784 real-valued inputs
  2. Each layer computes weighted sums and applies activation functions
  3. The output layer produces 10 values; the highest one is the classification
  4. Total: one forward pass, deterministic, instantaneous

In an SNN with rate coding:

  1. Each pixel intensity is converted to a spike train (brighter pixels spike more frequently)
  2. Spikes propagate through the network over multiple timesteps
  3. Each neuron integrates incoming spikes, leaks, and potentially fires
  4. After a sufficient number of timesteps, the output neuron with the highest spike count is the classification
  5. Total: many timesteps, probabilistic, temporal

With temporal coding, the SNN can be much faster:

  1. Pixel intensities are converted to spike times (brighter pixels spike earlier)
  2. The first wave of spikes propagates through the network
  3. The first output neuron to fire determines the classification
  4. The network can stop processing after the first output spike, saving energy on remaining computations

This early-exit capability is unique to SNNs: the network naturally processes easy examples faster than hard ones, because easy examples produce clearer, earlier spike patterns.

Event-Driven Efficiency

The energy edge of SNNs comes from two places. One: computation only fires when a spike arrives, so dormant pathways cost nothing. Two: a well-designed code transmits information with very few spikes, especially under temporal coding. A rate-coded SNN may not beat a plain ANN, but a temporally-coded one can reach the same accuracy with 10–100x fewer operations, because it skips redundant work and can decide before all inputs have even arrived.

Training Spiking Neural Networks

Training SNNs is harder than training conventional ANNs, by some margin. The root cause: spikes are discrete binary events, so the spike-generation function isn't differentiable. Standard backprop wants smooth gradients, and spikes don't oblige.

A handful of workarounds have emerged:

ANN-to-SNN Conversion

The simplest approach: train a conventional ANN using standard backpropagation, then convert it to an SNN by replacing continuous activations with spike rates.

How it works: A ReLU activation value of 0.7 is converted to a neuron that fires 70% of the time. The synaptic weights are transferred directly (with some normalisation).

Advantages: Borrows everything from the mature ANN training ecosystem. No new training algorithms needed.

Disadvantages: Requires many timesteps to accurately represent continuous values as rates. Loses the temporal coding advantages of SNNs. The resulting network is typically not as efficient as a natively trained SNN.

Surrogate Gradient Methods

The most successful direct-training trick for SNNs. The idea: keep the real discrete spike function on the forward pass, but on the backward pass swap it for a smooth surrogate — a sigmoid or a piecewise-linear approximation — purely for gradient computation.

text
Forward pass: spike = 1 if V(t) > threshold, else 0 (non-differentiable)
Backward pass: d(spike)/dV ≈ σ'(V - threshold) (smooth surrogate)

Surrogate gradient methods have achieved competitive accuracy on standard benchmarks (MNIST, CIFAR-10, DVS gesture recognition) while producing networks that are genuinely event-driven and temporally efficient.

STDP and Local Learning Rules

Spike-Timing-Dependent Plasticity (STDP) adjusts weights based on the relative timing of pre- and postsynaptic spikes. It's purely local: each synapse updates from its own activity, end of story.

That locality is gold for neuromorphic hardware — no global error signal, no backprop snaking through the network. STDP on its own, though, can't train deep networks on hard tasks. Active research is mixing STDP with other learning signals to push the technique further.

Equilibrium Propagation and Other Methods

Equilibrium propagation is a biologically plausible learning algorithm that approximates backpropagation using only local computations in a recurrent network. It has been shown to work on neuromorphic hardware, but its scaling to large networks remains an open challenge.

SNN Architectures

SNNs can use many of the same architectural patterns as conventional ANNs, adapted for spike-based computation:

Feedforward SNNs

Layers of LIF neurons connected in sequence, similar to a conventional MLP or CNN. Spikes propagate from input to output over multiple timesteps. Used for classification tasks.

Recurrent SNNs

Neurons with feedback connections, similar to RNNs. The recurrence, combined with the LIF neuron's membrane dynamics, provides rich temporal processing capability. Natural fit for time-series tasks, speech recognition, and control.

Convolutional SNNs

Spiking versions of convolutional neural networks, where convolution filters are applied to spike trains. Particularly effective when paired with event cameras (dynamic vision sensors) that produce spike-like output.

Liquid State Machines (LSM)

A reservoir computing approach: input spikes are fed into a random recurrent network (the "liquid"), and a simple readout layer (often trainable) classifies the liquid's state. LSMs are the spiking counterpart of echo state networks. They are easy to implement on neuromorphic hardware because the liquid does not need to be trained.

Temporal Processing: The SNN Advantage

Where SNNs decisively beat conventional ANNs is temporal data processing — anything that arrives as a continuous stream of events over time.

Event cameras (Dynamic Vision Sensors): These cameras emit events — pixel-level brightness changes — instead of frames. Each event is basically a spike: a timestamp plus a location saying "something moved here". Feeding events straight into an SNN is natural and cheap, because the data is already in spike form. Converting events to frames for a regular ANN throws away temporal information and adds latency.

Audio processing: Sound is inherently a time signal. The cochlea turns sound into spike trains on the auditory nerve. An SNN processes those spike trains while preserving the timing relationships that matter for speech recognition, sound localisation, and music.

Robotics and control: Control loops need continuous, low-latency processing. An SNN processes each sensory spike the moment it lands and updates the motor output immediately — no waiting to batch up a frame for a conventional forward pass.

Anomaly detection: Network traffic, industrial sensors, financial data — picking out unusual patterns requires continuous processing of a streaming signal. SNNs naturally carry a running state that adapts as the stream evolves.

In the next lesson, we will explore how these capabilities are being applied in real-world systems and what the future holds for neuromorphic computing.