The archive · Product Ideas · Technical decision · 2024–2025
WordLlama recycles a 70B LLM's token embeddings into a 16MB CPU word toolkit
D. Lee Miller extracts an LLM's token-embedding codebook, retrains it without context, and ships similarity, dedup and ranking in a 16MB NumPy-only library.
D. Lee Miller (deepsquirrelnet)
What it had to solve
After years of working with LLMs, developer D. Lee Miller wanted a lightweight utility for preparing inputs, locating information and building evaluators — something pip-installable that runs on a CPU without PyTorch or any deep-learning runtime. Transformer embedding models were accurate but heavy; classic word vectors like GloVe were light but old.
How it works
WordLlama starts from a simple observation: most of a language model's word knowledge sits in a table. The token-embedding codebook of a model like Llama 3 70B encodes what its vocabulary means, and you do not need to run the 70 billion parameters to use it. D. Lee Miller's library extracts that codebook, retrains it without any context, and ships the result as a tiny, fast, pip-installable toolkit.
The training recipe keeps the result honest. Miller concatenated embedding codebooks from models that share a tokenizer (Llama 2 70B with Phi-3, after stripping special tokens), added a learnable projection, and trained a small pooled model with multiple-negatives ranking loss plus matryoshka learning so dimensions can be truncated to fit. Once trained, the network and projections are discarded — the entire vocabulary is saved as a NumPy file and inference is a simple average-pooled token lookup, so nothing heavier than NumPy is needed at runtime.
The payoff is size and speed: the default 256-dimensional model is about 16MB and the smallest 64-dimensional, 32k-vocabulary model is 4MB, with binarised versions that compare by Hamming distance. The Show HN of September 2024 drew 370 points and 36 comments, and the project's own MTEB table shows the WordLlama variants ahead of GloVe-300d on every listed task while running on a single CPU core — a different trade from transformer models, and one the author stated plainly in the thread.
Why it lands
- The vocabulary table is the reusable asset: distilling it lets a small task inherit the word knowledge of a much larger model without carrying its compute.
- Discarding the network after training makes inference a lookup: NumPy-only, CPU-only, no PyTorch, no GPU — which is what makes the library pip-installable everywhere.
- Matryoshka training means one model serves many budgets: dimensions can be truncated from 1024 down to 64 without retraining.
- Binarised embeddings swap cosine for Hamming distance, trading a little accuracy for much smaller memory and faster comparison at the edge.
What it did
The Show HN of 15 September 2024 drew 370 points and 36 comments, with the author explaining the trick and comparing honestly with MiniLM in the thread. By September 2026 the repository had grown to roughly 1.4k stars, 48 forks and 323 commits, and the README's own MTEB table shows WordLlama ahead of GloVe-300d on every listed task while running on a single CPU core.
What you can take
A giant model is full of reusable parts: its vocabulary table can be distilled into the exact lightweight component a small task needs, and shipping it dependency-free multiplies its reach.
Since then
WordLlama kept evolving after the launch: semantic text splitting arrived in October 2024, Model2Vec static embeddings support in January 2025, and callable stdlib-style sorting and min/max helpers in February 2025. The MIT-licensed repository grew to roughly 1.4k stars, 48 forks and 323 commits by September 2026, with a citation file describing the project as recycled token embeddings from large language models and a roadmap pointing to DSPy evaluator examples and retrieval-augmented generation pipelines.
Sources
- Show HN: Wordllama – Things you can do with the token embeddings of an LLM
- dleemiller/WordLlama: Things you can do with the token embeddings of an LLM
spotted an error? The archive wants to know.
Your turn
You just read one. Describe the brief you are staring at, and see who has been given the same problem.
Free account · 3 free questions · no card