Status
Why should it matter to you where the model behind your chatbot actually runs? Lena Fuhrimann, Founder of and Cloud Solution Architect at bespinian, gave TechTalk #29 several uncomfortable answers, then a genuinely practical guide to becoming independent from them. Her talk followed Sebastian Nickel’s Ingress story as the second of three at TechTalk #29, opened as always with a short introduction from Thomas Hug, our CEO and founder.
Why It’s Worth Caring
Lena’s first argument was privacy. Google ended up indexing thousands of ChatGPT conversations that users had shared via a link, some containing highly personal content, after a short-lived “discoverable” feature went wrong. Separately, a US court ordered OpenAI to preserve chat logs that users had explicitly asked to have deleted, as part of an ongoing lawsuit, and OpenAI pushed back against that order rather than comply quietly. Neither company set out to be careless, but both examples show how little control you actually retain over your own conversations once they leave your infrastructure.
Her second argument was money. Despite massive investor backing, neither OpenAI nor Anthropic has turned a profit yet, by her own estimates. Eventually, that gap has to close somehow: through advertising, through monetising influence, or through higher prices. None of those outcomes work in the user’s favour.
The Pendulum
Lena’s framing for where this goes next: IT has always swung between centralisation and decentralisation, from mainframes to PCs, from PCs to cloud computing, from cloud computing to the smartphone in your pocket. Right now, AI is swinging hard toward centralisation, with Amazon, Microsoft, Alphabet and Meta pouring enormous sums into data centre capacity.
Her bet is that it swings back. Frontier proprietary models are already being used to train open source models, which has visibly shortened the gap between the two: what used to take open source over a year to catch up on now sometimes takes as little as six months, DeepSeek being the best-known example. At the same time, consumer and workstation hardware keeps getting more capable, Apple’s new M5 Max and Ultra chips reportedly sold out within minutes of release. Put those forces together with rising prices from proprietary providers, and local, open models running on hardware you own start to look like the more likely long-term direction, not just wishful thinking.
Getting Realistic
Lena was careful not to oversell what’s achievable. She sees roughly three tiers of open models: small ones up to about 35 billion parameters that run comfortably on a laptop, mid-sized ones in the hundreds of billions that need a serious GPU or a small cluster of them, and trillion-parameter-plus models that genuinely compete with the frontier and require a full data centre. For hobbyists and small teams, tiers one and two are the realistic target, not tier three.
Building the Stack
The actual constraint, contrary to popular belief, isn’t compute, it’s memory, specifically VRAM. Quantisation, rounding a model’s weights down to a lower precision, is the main lever for reducing that need, at some cost to accuracy (Google’s Gemma models are a good public example of the difference this makes). On the architecture side, mixture-of-experts models route each token to a small subset of relevant parameters via a lightweight router instead of pushing it through every parameter, which speeds up inference without saving memory, unlike quantisation. Together, these two ideas also explain the otherwise cryptic model names you see on Hugging Face: a 4-bit-quantised mixture-of-experts model with 3 billion active parameters out of 35 billion total shows up in its filename exactly like that.
On hardware, her rule of thumb is roughly 1.5 gigabytes of VRAM per billion quantised parameters. A consumer GPU with 12 gigabytes handles basic classification and simple Q&A. A professional card like Nvidia’s RTX Pro 6000, with 96 gigabytes, can run a 120-billion-parameter model for multiple users. An honourable mention goes to Apple’s own unified memory architecture, which lets the M5 Max and Ultra share up to 128 and 512 gigabytes respectively between CPU and GPU. On software, llama.cpp remains the flexible but complex foundation most tools build on, Ollama trades some of that flexibility for ease of use, and vLLM adds real efficiency gains on top, most notably paged attention, which allocates memory for a model’s answer in chunks as it’s generated instead of reserving a full worst-case block upfront. vLLM also integrates well with Kubernetes for GPU sharing across workloads, a nice link back to Sebastian’s talk earlier that evening.
What We Took Away
Lena’s closing advice was to go small deliberately: a smaller, specialised model often handles the task you actually have just as well as a large generalist one, without the memory bill. That’s a useful discipline for us too. Tier two, in Lena’s framing, workloads that need real GPU capacity but not a full data centre, is exactly the space our own GPU servers are built for. Her core point, that control over your data starts with control over your own infrastructure, applies just as directly when that infrastructure happens to sit in Switzerland.
We wrote more broadly about the questions to ask before committing to AI infrastructure in our post on training your own LLM instead of relying on US hyperscalers. Find out more about our GPU servers on our product page. Questions about anything from the talk? Get in touch. We announce the next TechTalk on our channels, including our Meetup group «TechTalk @ Nine».
