Local AI: open models on your own computer

Why do it (and why not)

Good reasons:

• absolute confidentiality: client documents, health data, trade secrets you won’t or can’t send to third parties;
• predictable costs for very high-volume work (classifying millions of documents);
• offline operation and independence from a provider;
• customisation: you can specialise the model on your data.

Bad reasons:

• “It’s free”: hardware costs, electricity too, and the time of whoever manages it even more.
• “It’s the same as ChatGPT”: it isn’t. According to Artificial Analysis, in October 2026 the best open model (MiMo-V2.6-Pro) scores 46 against 58 for Claude Opus 5.5, and models that fit on a laptop are far behind.

Which models exist

The strongest open models of 2026 come mostly from China and a few Western labs. From the Artificial Analysis leaderboard:

Model	Index	Notes
MiMo-V2.6-Pro (Xiaomi)	46	Most capable open model; needs server hardware
GLM-5.3 (Z AI)	45	Strong in reasoning and code
Kimi K3 (Moonshot)	44	Very long context
Qwen3.8 (Alibaba)	up to 40	Wide family, small versions too (27B)
DeepSeek V4.1 Flash	39	Fast and efficient
Gemma 4 (Google), gpt-oss (OpenAI), Mistral Small 4	9-17	Small: run on a good laptop

“Small” models score low on general benchmarks, but for targeted tasks (summarising, classifying, extracting data, translating) they’re often enough.

What hardware you need

A model is made of parameters, i.e. numbers. To run it they must fit in memory (ideally the graphics card’s memory, or the unified memory of Apple-chip Macs). With 4-bit quantisation, which reduces number precision with limited quality loss, the rule of thumb is:

Model size	Indicative memory	Where it runs
4-8 billion parameters	4-6 GB	Almost any recent computer
12-14 billion	8-10 GB	Laptop with 16 GB RAM
27-35 billion	18-24 GB	32 GB Mac, PC with a 24 GB graphics card
70-120 billion	45-80 GB	Workstation or 64-128 GB Mac
Over 300 billion	Hundreds of GB	Multi-GPU server

Many modern models are Mixture of Experts (names like “35B A3B”): they have 35 billion parameters but use 3 per token. They take the memory of a big model but run as fast as a small one.

Long context takes extra memory: if you want to analyse hundreds of pages, plan headroom.

Start in ten minutes

With LM Studio (recommended to start)

1. Download LM Studio from the official site and install it.
2. In the search section, look for a small model (e.g. from the Gemma or Qwen families) in the 4-bit quantised version the app marks as compatible with your computer.
3. Download it and open the chat: it works like ChatGPT, but entirely local.
4. To use it from other programs, turn on the local server: it exposes an OpenAI-compatible API.

With Ollama (for automation)

Ollama runs from the terminal: one command downloads the model, another starts it. It’s perfect as the “engine” for private automations, for example with n8n installed on the same server: documents are read, classified and summarised without ever leaving the office.

A typical setup for a firm

1. An office computer or small server with a 20-35 billion parameter model.
2. Ollama as the engine, n8n for flows (see the automation guide).
3. An index of internal documents for questions (see the RAG and embeddings guide).
4. For hard, non-confidential tasks, a frontier model via API with a business contract.

This is the hybrid approach: local for what’s confidential and repetitive, cloud for what needs maximum intelligence.

In practice

• Try LM Studio with a small model: in ten minutes you’ll see what it can and can’t do on your computer.
• Consider local AI for confidential data and large volumes, not to fully replace the best models.
• Calculate the real cost: hardware, energy, management time.
• Always check the model’s licence.

Sources

• Artificial Analysis — open-weights model leaderboard
• LM Studio
• Ollama
• Devsplainers — Can a Local LLM Actually Replace Claude or Codex for Coding?
• Kai — Every Way to Get 128GB VRAM for Local AI

Informational content, current as of the date shown. Prices, models and rules change often: always check official sources. Written with the support of AI tools and reviewed by the author.