I like the idea of AI not as a cloud spectacle, but as a little companion you can carry, hack, and understand. So I dropped BitNet’s 1-bit model onto a Raspberry Pi 5 (8 GB), and something magical happened: it talked at human reading speed.
Imagine reducing every weight in a neural net to just +1 or −1, so one single bit. Instead of storing dozens of decimals, you store just a “+” or “−”. That slashes memory by 16× (compared to float16), and simplifies many operations to sign flips. It’s the pixel art or chiptune of neural nets: limited, but expressive in the right hands.
The Raspberry Pi 5 is cheap, accessible, and beloved by hackers. It forces you to work with tight constraints. It’s also just powerful enough to run tiny AI experiments, if you push and trim. If your model can talk on a Pi, it can talk anywhere.
Important: I had to use the 2024-11-19 Raspberry Pi OS arm64 image.
After flashing that image, I started from a clean Pi 5 (8 GB), updated, and installed BitNet cpp.
When I ran the model, these lines appeared:
sampling time = 17.97 ms / 116 runs → 0.15 ms / token → 6,453.77 tokens/sec
load time = 1,191.79 ms
prompt eval time = 73,131.53 ms / 59 tokens → 1,239.52 ms / token → 0.81 tokens/sec
eval time = 41,181.51 ms / 191 runs → 215.61 ms / token → 4.64 tokens/sec
The key number: ~4.6 tokens/sec during generation. That’s roughly human reading speed: a few words per second. The Pi 5 wasn’t lagging, it was keeping pace with my eyes.
The Pi 5 uses shared RAM (no dedicated VRAM). The BitNet model fit comfortably within 8 GB of memory. Under load, the board draws around 10–12 W. You could run this setup off a battery pack, it’s basically a pocketable AI console.
You might call 4.6 tokens/sec “slow.” But I call it perfectly matched to human scale. At that speed, the machine doesn’t race ahead; it keeps pace with your mind. Like 30 fps games or 8-bit sound, the constraint defines the experience.
The future doesn’t have to be bloated models and server farms. It can be a little board, whispering back to you at human pace. BitNet on Pi 5 is my first step toward that: tiny, local, alive.