Run AI models locally on your computer with open-source tools and chat apps

Data analyst runs a local AI model on a Surface beside confidential folders

Local AI means running an AI model directly on your own computer, with tools like Ollama or LM Studio, instead of sending your data to cloud servers like OpenAI’s ChatGPT. It works offline once the model is downloaded, but it needs enough RAM or VRAM to load the model.

What you’ll take from this

By the end you will know whether running a model locally suits your work and how to start on your own machine.

  • Estimate the memory a model needs before you download it
  • Pick the right app for your operating system
  • Decide which tasks run locally and which stay in the cloud
  • Run a short checklist before using sensitive data

What is local AI and how does an AI model differ from cloud AI?

Local AI is an AI model that runs on your own hardware, usually a large language model (LLM), which is a generative AI system trained to produce text from instructions. This calculation, called inference, runs on your processor or graphics card (GPU). With cloud AI, your prompt travels to a provider’s server and the answer returns over the internet. A local model skips that trip entirely.

CriterionLocal AICloud AI
Where data is processedOn your computerOn the provider’s servers
ConnectionNo internet after the model downloadsAlways online
Cost per useNo per-token fee, but you pay for hardware and powerSubscription or pay per token
MaintenanceYou handle updatesThe provider handles it

The Certificate Program in Artificial Intelligence and Innovation from Founderz lists Ollama, LM Studio and AnythingLLM among its tools, next to ChatGPT and Microsoft 365 Copilot, so you can compare local and cloud approaches on real tasks.

If you want a free way to try a local model with your own files, Rebost is a desktop app for Mac and Windows built by Frontierz: the AI runs on your computer, your documents stay there, and each answer cites the files it used.

What hardware do you need to run AI locally: Nvidia GPUs or Apple silicon?

To run a model, you need enough memory to hold it, either the VRAM on your graphics card or the unified memory of a Mac. Small LLMs fit on midrange Nvidia graphics cards or a laptop with spare memory. CUDA, Nvidia’s platform for hardware acceleration, speeds up the calculation on its cards. MLX is Apple’s open-source array framework, optimized for Apple silicon.

To estimate memory, multiply the parameters by the bytes per weight. An 8B model needs about 16 GB at 2 bytes per weight. Quantization, which lowers the precision of those weights, cuts that to about 4-5 GB, plus extra memory for the conversation context. That is why a 4-bit 7B model fits on a card with 8 GB of VRAM, while an 8B model leaves little room for context.

  • On Windows, open Task Manager, go to Performance and note the dedicated GPU memory.
  • Note your total RAM, and remember that the operating system and the conversation context also need memory.

Microsoft documents local AI options in its official guide to AI on Windows.

What are the best tools to run AI locally, and how do you install Ollama or LM Studio?

Install a desktop app such as LM Studio (graphical) or Ollama (terminal), download a small model, and chat with it offline. Ollama runs from the terminal: you type a command to download a model and chat with it. Setup takes four steps.

  1. Check your memory using the list above.
  2. Check that the app supports your OS: Windows, macOS or Linux. Then choose the graphical app if you prefer windows, or the terminal tool if you want a local server.
  3. Download a quantized 7-8B model, such as a Llama or Qwen model.
  4. Try a real task, then turn off Wi-Fi to confirm it works offline.

This prompt tests your model on real meeting notes.

You are my meeting assistant. Summarize the notes below into five actions, each with an owner and a due date. If a detail is missing, write "no data". Notes: (paste text here)

Always review the output, because small models often invent details.

Which open models can you run for chat instead of ChatGPT?

Models with downloadable weights, such as Google Gemma, Llama and Qwen, can run on your own hardware through Ollama or LM Studio. Pick by size and task. A 7-8B model handles drafting and summarizing, and a larger one suits you if your memory allows. ChatGPT, Claude from Anthropic and Gemini run on their providers’ servers.

Quantized versions, such as the 4-bit ones that llama.cpp stores in the GGUF format, shrink the memory a model needs. The llama.cpp project is an open-source AI library, so anyone can inspect its code. Before comparing them, it helps to understand what a large language model is.

Some AI tools let you build useful AI agents that run models locally. An AI agent is a program that plans and completes steps toward a goal, such as sorting notes or drafting replies. Agentic tasks, though, need larger models and more memory.

How do AI apps and open-source backends like MLX run AI models through an API?

AI apps connect to a local model through an API, a standard way for one program to send requests to another. Ollama exposes one on your machine. Backends such as llama.cpp or MLX do the calculation behind that API. The app sends your prompt to the backend, the backend runs the model on your hardware, and the answer comes back without leaving your machine.

Can you run AI locally for free, and how expensive is it?

Yes, the software is free, so it is cheap to start and expensive only if you buy hardware. Local costs are mostly one-off, while cloud costs repeat every month or every token. A PC with 16 GB of RAM is enough to start. A card with more VRAM is a real expense, and if you are using AI only a few times a month, a cloud subscription may cost less.

Is local AI more private than ChatGPT, and does it work offline?

In one specific way, yes. A local model also keeps answering without internet once it is downloaded. With ChatGPT your text travels to a provider, while a local model processes it on your device, which reduces exposure of personal data. It does not exempt you from Regulation (EU) 2016/679 (GDPR), applicable since May 25, 2018 according to EUR-Lex. It still binds you if you handle data of people in the EU.

According to Eurostat (2025), 19.95% of EU enterprises used AI technologies in 2025, and 55.03% of large enterprises.

According to EUR-Lex, the AI Act, Regulation (EU) 2024/1689, entered into force on August 1, 2024. Its obligations depend on your role and on how the system is used, so running a model locally does not by itself place you outside the Act. Before pasting sensitive data, check three things.

  • Encrypt your computer’s disk.
  • Confirm with the network disconnected that the model responds.
  • Delete conversation history containing personal data when you finish.

Is local AI better than ChatGPT?

Not overall, because it depends on the task. It wins on privacy, offline use and cost per use for narrow jobs like summarizing, classifying or drafting. Cloud models win on top-level reasoning and very long documents.

SituationChoiceReason
Summarizing client meeting notes dailyLocalData stays on your device
Analyzing a very long reportCloudLarger models and more context
Drafting one email a monthCloudNo hardware to buy or maintain

Frequently asked questions

These are the questions that come up most when you install a model for the first time, each with a direct answer.

Can I use local AI without a GPU?

Yes. Both apps can run small models on the CPU and system RAM alone. Responses come slower, with fewer tokens per second, but it works for summarizing and drafting. Speed depends on the model and your machine, so start with a small quantized model and a short task to measure it.

Does local AI work without internet?

Yes. After you download the model, the chat runs offline, which is the easiest way to verify your data stays put. You only need a connection to download models or updates, and the model cannot search the web unless you add tools that can.

Can I train a model on my own documents?

Training a model from scratch requires clusters of graphics processors beyond any PC. The usual approach is to take a trained model and give it your documents as context in the prompt, or fine-tune it with lightweight techniques if you have technical skills. To start, context is enough.

How much memory does a local model need?

At 4-bit quantization, expect about 0.5 to 0.6 GB per billion parameters, plus overhead for the conversation context and the operating system. With 8 GB of memory you can run models from 1B to 7B parameters. A 16 GB machine opens the door to 7B and 8B models at higher quality.

Further reading

These official sources cover data protection, AI use in European companies and local AI options on Windows.

To learn which AI tool fits each task and which data you can share, the Certificate Program in Artificial Intelligence and Innovation from Founderz is an online programme. It covers cybersecurity challenges and responsible AI across its 12 modules, and you earn the Founderz + Microsoft certification and official Microsoft badges. You can view the syllabus or request information.

This article is for informational and educational purposes only and is not legal advice or a substitute for a professional’s judgment; always consult your lawyer.

Paul Delaney

Paul Delaney has been engineering AI prompts since the GPT-2 era, long before ChatGPT made prompting mainstream. Paul leads SEO, AEO, and GEO strategy at Founderz, improving how the school and its programs are discovered through traditional and AI-powered search. With more than 25 years of experience in education and digital growth, he has used AI daily since 2021 to support his commercial work.