The archive · Product Ideas · Product decision · 2024
Secret Llama runs Llama 3 in-browser via WebGPU so no chat data ever leaves your machine
A free chatbot downloads Llama 3, Mistral or TinyLlama into the browser and runs it locally over WebGPU — no server, no install, no data leaving the computer.
Secret Llama
What it had to solve
In spring 2024, a capable open-source model meant downloading llama.cpp or similar and managing GPU software, while every easy web chatbot sent prompts to someone else's server. WebGPU was just arriving in mainstream browsers, and MLC AI's WebLLM engine offered in-browser inference that was still rough around the edges.
How it works
In early 2024, the privacy story around AI chatbots was a policy story: companies promised not to misuse your prompts. Secret Llama's author saw a different answer in WebGPU, the browser API that lets web pages use the local graphics card. If the model itself could run in the browser tab, privacy stopped being something a company promised and became something the architecture guaranteed — no conversation data ever leaves the computer.
Secret Llama is that idea packaged as a product anyone could try. The open-source project pairs MLC AI's WebLLM inference engine with a ChatGPT-like interface, offers TinyLlama, Llama 3 8B, Phi and Mistral in quantized sizes from 600MB to 4.3GB, and downloads the chosen model into browser storage only after the first message. Requirements are stated plainly: a WebGPU-capable browser such as Chrome or Edge, with Firefox support behind a flag.
The Show HN on 3 May 2024 drew 547 points and 139 comments. Users were surprised by what their own hardware could do — a Pixel 6 phone ran TinyLlama 'with alarming accuracy', and a desktop owner found local Llama 3 competitive with ChatGPT-4 for many prompts — while the author answered questions about browser storage, model formats and bugs in the thread. The repository later showed 2.7k stars and 169 forks, and the live demo stayed up at secretllama.com.
Why it lands
- Running the model in the tab turned a privacy promise into a technical fact, which no amount of policy wording can match.
- WebGPU let the user's existing GPU do the work, so 'local AI' needed no install, no command line and no driver setup.
- A ChatGPT-grade interface removed the barrier between curious users and the underlying engine, making the demo the pitch.
- Shipping a live site the same day gave the community something to test immediately, and their hardware reports became the proof.
What it did
Posted as Show HN on 3 May 2024, it drew 547 points and 139 comments: a Pixel 6 user ran TinyLlama with alarming accuracy, an RTX 2080 owner said local Llama 3 competed with ChatGPT-4 on many prompts, and the author fixed reported bugs within a day. The repo page later showed 2.7k stars and 169 forks, and the demo remained live.
What you can take
Kill the trust problem by changing the architecture, not the policy: when privacy is the pitch, put the compute where the data already is, and let the working demo prove the claim.
Since then
The launch thread read like a local-LLM gateway drug: first-time users discovered that their phones and laptops could run open models, veterans pointed newcomers down the rabbit hole of WebLLM and quantization, and the author promised chat history and more models while fixing the reported bugs the next day. The repo stayed open and the demo live, and the project joined the wave of tools that made running models on your own hardware feel normal rather than exotic. Its real legacy was rhetorical as much as technical: it gave 'private AI' a demo that did not ask you to trust anyone.
Sources
spotted an error? The archive wants to know.
Your turn
You just read one. Describe the brief you are staring at, and see who has been given the same problem.
Free account · 3 free questions · no card