Language models
Daedalus - A 150M params LLM built for CPU inference.
Originally published on LinkedIn.
Text description of the demonstration
A terminal runs the quantised Daedalus instruct model with llama-cli, using the model-q4_0.gguf file and the ChatML template. Examples include “Give me an inverted binary tree function”, “Which is the capital of France?”, and “Write a poem about love”.
The model generates Python-like tree code, identifies Paris as the capital of France, and produces a poem. Some responses are repetitive, and the generated code is not verified in the demonstration. The terminal displays prompt-processing and generation token rates after responses. This is a demonstration of local text generation; the controlled performance comparisons are described in the article and paper.
My summer project, Daedalus-150M (architecture, corpus, training, evaluation and write-up). Daedalus is a small language model in which two thirds of the attention layers are replaced by short convolutions that keep a fix size state.
Fully open sourced code - open weights and measurements behind every number, plus the things that didn't work: https://huggingface.co/Unseen1980/daedalus-checkpoints and https://github.com/unseen1980/daedalus/tree/main
Preprint paper: https://arxiv.org/abs/2608.20210
I should say up front that the practical usefulness of this model is minimal! At 150M parameters it is not going to be useful for anything you would actually want an assistant to do. It was built for research purposes, and for me to learn how the blocks of a transformer behave when you take one of them out and put something else in its place. That is the whole point of it...
The reason to try this is CPU inference. An attention layer has to reread its cache for every token it generates, and that cache grows with the conversation, so generation gets slower the longer you talk to it. A convolution layer keeps a state that is two "timesteps" wide no matter how long the context is, so it costs the same at token 2000 as at token 2. Per token of context Daedalus reads 6,144 bytes of cache where a 24 layer all attention model of similar size reads 12,288. At 2048 tokens that is 12.6 MB re-read per generated token instead of 25.2 MB.
To check whether the convolutions actually helped, I trained a second model alongside, same size, same data, same schedule, but with normal attention in every layer.
What would count as a win was decided and written down before either model was scored. The two came out tied on quality: 44.68 vs 44.82 on a five task mean, well inside noise, while the hybrid won the preregistered metric, validation bits per byte, by 0.81%.
The difference is at decoding: at 2048 tokens of context the hybrid runs 1.76× faster on CPU, and the gap widens with context, which is the signature the mechanism predicts. At an empty context the advantage is only 1.20×, and that is the point. The hybrid has nothing to gain when there is no cache to skip. The shape of that curve is the result, not the bottom row of it. The final model was trained on 59.9B tokens and scores 47.31, at 0.8685 validation bits per byte over a 645M token held out set.
Don't be confused it is not a quality win and I am not presenting it as one! SmolLM2-135M, trained on 2T tokens, is still ahead of it, and that was conceded in advance. The trade this project makes is speed.
Plenty of it did not work -> Around 48% of the convolution channels are dead, which is 13.6M parameters doing nothing. Quantisation aware training was built and validated against llama.cpp's exact Q4_0 grid, so the 4 bit model costs about 6% perplexity instead of the 2.5% intended. The vocabulary is around twice the size it should be at this scale and eats 23% of the parameters.
Thanks for reading - Christos