Run large language models locally with Ollama and pick an AI model for your use cases

Professional running an AI model locally with Ollama on a Microsoft Surface Laptop

Ollama is a free, open-source tool that downloads and runs large language models (LLMs) on your own computer, so your prompts stay on your machine. It works on Windows, macOS and Linux, and it lets you run models such as Llama, Gemma, Qwen and DeepSeek from the command line or through a local API.

What you’ll take from this

By the end, you will know whether Ollama fits your work and how to run your first model in an afternoon.

  • Understand what Ollama does and where it stops.
  • Check whether your computer has enough memory for a useful model.
  • Install Ollama and run your first model with one command.
  • Decide between a local model and a cloud service on privacy and cost.

What are the best use cases for Ollama?

Ollama is best for running open models privately on your own machine, whether you draft, summarize or classify text. It downloads, manages and serves models such as Llama, Qwen, Mistral and DeepSeek, and it lets you customize them with a Modelfile, a plain configuration file.

Picture a consultant who writes proposals with client figures. She pastes a draft into a local model to get a summary, and the text never leaves her laptop.

For the wider picture of LLMs, agents and responsible use, the Certificate Program in Artificial Intelligence and Innovation from Founderz lists Ollama among its tools, next to LM Studio, AnythingLLM and n8n.

Who owns Ollama and its models?

No. A company called Ollama develops the tool and publishes its open-source code on GitHub. The companies that publish models such as Llama or Gemma do not own the software that runs them. Each model has its own license, so read it before using a model commercially.

How does Ollama work for running LLMs locally?

A local LLM is a large language model that runs on your own hardware instead of a cloud service. Ollama starts a server on your computer, downloads the weights of the language model you request and answers using your CPU or GPU. By default, the server listens on port 11434 of your own machine and offers a REST API, with partial, experimental compatibility with the OpenAI API.

An AI model ships compressed through quantization, which stores weights at lower precision so they fit in home-computer memory. The smaller the model, the less memory it needs and the less capable it tends to be. Your prompt is split into tokens, and each token is a small chunk of text. The model reads them within its context window, the amount of text it can consider at once.

How much VRAM does an Ollama model need?

Ollama needs enough free memory to load the model, either in RAM or in the VRAM (video memory) of your GPU. A GPU is optional, since without a compatible one Ollama uses the CPU, which works but feels slow. The table shows disk size for models in the Ollama library, where B means billions of parameters.

ModelDisk size
gemma3:4b (4B)3.4 GB
llama3 (8B)4.7 GB
deepseek-r1:8b (8B)5.2 GB
gemma3:12b-it-q4_K_M (12B)8.2 GB

Your free memory should exceed the file size, with extra room for the prompt. Run ollama list to see the models you have installed. On an older laptop with 8 GB of RAM and 4 GB of VRAM, start with the 4B model and close other apps.

How do you install Ollama and run AI models on your computer?

Download the official installer, run it and confirm in the terminal that Ollama responds. Windows and macOS use an installer, while Linux uses a single command copied from the same download page. After that, you use Ollama the same way on every system.

  1. Go to Ollama’s official website and download the installer for Windows or macOS. On Linux, copy the install command shown there.
  2. Open the installer and follow the wizard to the end.
  3. Open PowerShell on Windows, or Terminal on macOS and Linux, and type ollama –version to confirm the installation.
  4. Type ollama run gemma3:4b. The first time, it downloads the model and opens a chat in your terminal.

Which operating system is best for Ollama? The one you already use. All three are supported, so memory and a compatible GPU matter more than the OS.

If you prefer a container, Docker runs Ollama with this command.

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

To judge how much code you will eventually need, read Founderz’s guide on which programming language to learn for AI (in Spanish).

What are the basic Ollama commands?

Five command-line (CLI) commands cover daily use. You run a model, pull one from the registry, list what is installed, see what is loaded and remove what you do not need.

CommandWhat it does
ollama run modelDownloads the model if missing and opens a chat
ollama pull modelDownloads a model from the Ollama registry
ollama listShows installed models
ollama psShows models loaded in memory
ollama rm modelRemoves a model

This example summarizes a meeting note with Gemma 3 4B, a small model.

ollama run gemma3:4b "Summarize this note in three bullet points. On Monday we agreed to move the launch to March 14. Marta closes the budget on Wednesday. The supplier contract still needs review before Friday."

Always review the answer. A small model can omit a date or assign a task to the wrong person.

Which AI model should you pick in Ollama: Gemma, Qwen, DeepSeek or gpt-oss?

No model is best for everyone, so match the model to your task and your memory. Start with a small general model, test it on real text, and move up in size only if the answers need too many corrections. These families cover most needs.

  • Llama and Gemma are generalist families with small versions for laptops.
  • Qwen and Mistral are worth testing if you work in several languages or with code.
  • DeepSeek offers reasoning versions that think step by step and answer more slowly (see our DeepSeek guide).
  • gpt-oss is OpenAI’s open-weight model family.

Stanford HAI’s AI Index 2025 reports that the performance gap between open-weight and closed models narrowed from 8% to 1.7% on some benchmarks in a single year. That figure covers specific benchmarks, and it does not mean models small enough for a laptop reach that level.

Try two or three models on the same real text and keep the one that needs the fewest corrections.

How do you customize a model with a Modelfile?

A Modelfile is a plain text file that names the base model, the system prompt and the parameters Ollama should apply. A system prompt is a standing instruction that shapes every reply, such as tone, length or format. You then reuse it as your own named model.

FROM gemma3:4b
SYSTEM You are a careful business editor. Answer in plain English and flag anything you are unsure about.
PARAMETER temperature 0.3

Save it as Modelfile, run ollama create editor -f Modelfile, then start it with ollama run editor. A lower temperature makes answers steadier and less inventive, which suits editing and summaries.

How do you set up an Ollama integration through the REST API?

Ollama exposes a local HTTP API on port 11434, so any application can send text and receive the model’s reply without leaving your computer. According to Ollama’s documentation, the local API needs no API key or authentication when you run it on your own machine.

curl http://localhost:11434/api/generate -d '{"model":"gemma3:4b","prompt":"Why is the sky blue? Answer in one sentence.","stream":false}'

A few lines of Python do the same job, and the API also produces embeddings for search tools. Community tools can add a chat window on top of the same server, so check Ollama’s GitHub page for current options. Microsoft Learn documents Ollama as a model provider for its Agent Framework, which helps if you build AI agents.

Coding assistants such as OpenCode, Codex and Claude Code can talk to local or compatible endpoints, but support changes quickly. Check each tool’s documentation and the integrations section of Ollama’s documentation first.

Is Ollama free to use?

Yes, for running open models on your own computer. The software is open source, the local API needs no API key, and your real costs are hardware and electricity. Ollama’s website also offers cloud models with their own plans, so check prices before subscribing.

Local is not always cheaper. The same AI Index 2025 reports that the inference cost of a GPT-3.5-level system fell more than 280-fold in about two years. For occasional use, a pay-as-you-go cloud service can cost less than extra memory or a new GPU.

Is it safe to run Ollama locally?

Mostly yes. With local AI, your text is processed on your computer and not sent to external servers. That helps privacy, but it does not replace compliance, and the risks sit in how you set the tool up.

If you process personal data, the GDPR still applies, and it expects you to protect that data with appropriate security measures. In practice, control who can access the computer and its documents. This is general information, not legal advice.

  • If you switch on cloud models, your messages leave your computer, so read their terms before pasting personal data.
  • Do not open port 11434 to the internet, because the local server asks for no credentials by default.

Is Ollama better than ChatGPT?

They are not comparable as a block. Ollama is software that runs open models on your computer, while ChatGPT is a cloud service from OpenAI. Quality depends on the specific model and your hardware. The table shows when each option makes sense.

SituationReasonable option
Document with client dataLocal model with Ollama
Task that needs top qualityCloud service
Work offlineLocal model with Ollama
Occasional use without sensitive dataCloud service

Ollama vs LM Studio vs llama.cpp, which tool for what?

Ollama, LM Studio and llama.cpp are different ways to run open models on your own computer, so the right choice depends on how you like to work. Try each one on the same short text and keep the one that fits your routine. Founderz’s program lists Ollama and LM Studio among its tools, so you can practice with both.

Frequently asked questions

These are the questions readers ask most before installing Ollama.

How good is Ollama AI?

Ollama is not a model, so its quality depends on the model you run. Small models handle summaries, drafts and text classification well, but they fail more often on long reasoning and recent facts. Compare them on your own work before trusting any ranking.

Is Ollama available on Linux and Windows?

Yes. Ollama runs on Linux, Windows and macOS. Linux installs with one command from the official download page, while Windows and macOS use a standard installer. All three give you the same commands, so a tutorial written for one system mostly works on the others.

Where does Ollama store models?

Ollama stores downloaded models in the .ollama folder of your user account. To free up space, quit Ollama first, then set the OLLAMA_MODELS environment variable to a folder on an external drive, or move the .ollama folder and link to it.

Do I need to code to use Ollama?

Not for the basics. Installing the program, downloading a model and chatting only require typing a couple of commands you can copy. You need code only to connect Ollama to other applications through its API, which takes a few lines.

Further reading

These sources cover the report, the regulation and the documentation cited in this guide.

To work with language models with professional judgment, the Certificate Program in Artificial Intelligence and Innovation from Founderz, in collaboration with Microsoft, is a fully online program.

The program page lists 12 modules with real-world cases, and its tools include ChatGPT, Microsoft 365 Copilot, Copilot Studio, Ollama and LM Studio. Separate modules cover private AI solutions, cybersecurity and responsible AI leadership, so you can place local models in a company setting. You earn the Microsoft + Founderz certification and official Microsoft badges. You can request information or enroll on the program page.

Paul Delaney

Paul Delaney has been engineering AI prompts since the GPT-2 era, long before ChatGPT made prompting mainstream. Paul leads SEO, AEO, and GEO strategy at Founderz, improving how the school and its programs are discovered through traditional and AI-powered search. With more than 25 years of experience in education and digital growth, he has used AI daily since 2021 to support his commercial work.