I have spent much of this year writing about what intelligence costs. Paying for Intelligence Twice looked at the bill you pay in money and institutional knowledge when your workflows run on someone else's model. The Instrument and the Ledger built the meter. This post walks the other side of the street: models you can download and run on your own hardware.
They carry costs of their own. For workloads they can handle, though, they remove the vendor's per-token meter and give you another place to route the work.
The trigger is hard to ignore. In April, Google released Gemma 4, an open-weight family under the Apache 2.0 license. Google says it is built from the same research and technology as Gemini 3. In June, Google added a 12B model with benchmark performance nearing its 26B sibling. Ollama's default 4-bit artifact downloads at 7.6GB. A model from a frontier lab's research line now runs on a class of laptop many developers already own.
The important shift is not that open weights have won, or that hosted prices only move in one direction. Hosted inference keeps getting cheaper while the local capability floor keeps rising, and that creates a routing problem rather than a winner-take-all story.
To be clear, treat Gemma 4 12B as a capable mid-size model rather than the closed frontier in miniature. The useful claim is that the local floor has moved far enough to change the economics, the architecture, and the negotiating posture.
What "open weight" actually means
An open-weight model is one whose maker publishes the trained parameters, the billions of numbers that make it work, for others to download. You can run it on your own hardware, fine-tune it on your own data, and build products on top of it. What you usually do not get is the training data, the full training code, or the complete recipe.
That is why the precise term is open weight rather than open source. The distinction matters when you are making commitments to a regulator, a board, or a customer.
The license class matters more than the label. Apache 2.0 and MIT sit at the low-friction end, but neither is obligation-free. Both permit commercial use, modification, and redistribution while preserving notice requirements. Apache 2.0 also includes express patent terms. Community licenses can add other conditions.
Llama 4, for example, uses Meta's custom license. It requires "Built with Llama" attribution for distributed products and a separate license request for organizations above 700 million monthly active users.
Several prominent releases have moved toward permissive licenses. Gemma 4, Qwen3, Mistral 3, and gpt-oss use Apache 2.0. DeepSeek-R1 uses MIT. Gemma 4 is also Google's first Apache 2.0 release in the family. Earlier Gemma generations remain under Google's own terms.
Legal review stays on the checklist; permissive licenses just make the terms easier to compare.
The landscape in one map
The most useful map is deployment shape, not a ranking of model families.
At one end are giant open-weight mixture-of-experts models with hundreds of billions of total parameters. The weights may be downloadable, but the deployment still requires data-center GPUs, private cloud capacity, or an inference partner. These models change where inference runs and who controls it.
Even relatively efficient models can belong on this side of the map. Meta describes Llama 4 Scout as a 109-billion-parameter model with 17 billion active parameters. Its 4-bit deployment fits on a single NVIDIA H100, not an ordinary laptop.
At the other end are models sized for laptops, desktops, and edge devices. Gemma 4 12B belongs here, along with mid-sized Qwen releases and smaller Mistral models. These models change who can experiment, how quickly they can start, and whether every useful token creates a vendor charge.
The two tracks answer different questions. One changes the control plane for production inference. The other changes the local capability floor.
I am deliberately not printing a benchmark table. Releases and rankings churn too quickly, and a public leaderboard does not represent your workload. The durable facts are the two deployment shapes, the increasingly permissive licenses, and the need to evaluate models against work you actually do.
The cost argument is now a routing argument
Why does this matter to the AI ledger? Three reasons hold up under inspection.
First, routing economics. The price gap is not moving in only one direction. Hosted models keep getting cheaper, and a stronger model can cost less once review and rework are counted. But when a local model clears your quality bar on a bounded, high-volume task, the marginal vendor charge falls to zero.
Zero vendor charge still leaves real costs, but it flips the burden of proof. The question becomes which tasks still need the premium model.
Second, data gravity. Self-hosted open weights can bring the model to your data instead of sending your data to a hosted endpoint. I lead engineering in a regulated industry, and that can simplify a data-residency review. It does not settle endpoint policy, software installation, identity, access control, logging, backups, provenance, or software supply chain risk.
Local is a deployment property; approved is a governance outcome. Where the weights came from, how they were packaged, and who can invoke them remain separate reviews.
Third, the learning loop. Satya Nadella's argument, the one I wrote about in July, is that AI workflows generate valuable learning exhaust from your proprietary context and corrections. My edit was to own the learning loop, not necessarily the model. Open weights make the second half of that sentence a genuine choice in a way it was not two years ago.
That brings me to the caveat. Capability still buys real things, and the closed frontier remains ahead on the hardest agentic and long-horizon work. For some workloads, that gap is worth every dollar. Self-hosting is never free either. GPUs, electricity, security reviews, patching, observability, and the people who run all of it land on the other side of the ledger.
An honest comparison prices both columns. The rational posture is a portfolio, and the entry fee for running one is evaluation capacity. That means a suite of your own tasks with clear pass conditions, so a model swap becomes a measurement instead of a guess. I made that argument at book length, and it applies twice when the weights carry no license fee and local inference carries no per-token API charge.
Getting started: Ollama and Gemma 4 12B
The fastest way to calibrate your own opinion is to run one of these models yourself. Production workloads can stay right where they are. This exercise is about feeling the local capability floor firsthand.
The easiest on-ramp I know is Ollama. One command downloads a pre-quantized build, handles the chat template, and includes Google's recommended sampling defaults: temperature 1.0, top_p 0.95, and top_k 64.
Gemma 4 12B is the model I would start with. It uses Apache 2.0, reads images as well as text, and carries a 256K context ceiling. Google reports benchmark performance near the 26B model. The 4-bit build trades some precision for a much more practical footprint, so your own evaluations still matter.
On hardware, I am writing this for Apple Silicon Macs because that is my daily environment. Ollama also ships Windows and Linux installers, and the commands below are the same. Google identifies 16GB of VRAM or unified memory as the minimum for this model. I would treat that as a floor, not a comfort recommendation.
The default Ollama artifact downloads at 7.6GB, but download size is not runtime memory. Context length, the multimodal projector, and inference overhead all consume additional memory. A 32GB system leaves more room for the model, the operating system, and your browser's opinion of itself.
Install Ollama. Download the installer from ollama.com and open it. It installs the ollama command-line tool and runs a small local server in the background.
Pull and run the model. In a terminal:
ollama run gemma4:12b
The first run downloads about 7.6GB, including a pre-quantized Q4_K_M model and its multimodal projector. Subsequent runs load the model from disk. After the download, disconnect Wi-Fi if you want a blunt confirmation that inference is local. For this local model tag, inference stays on your machine unless you deliberately attach external tools or services.
Give it real work. Generic test prompts teach you almost nothing. Give it a few pages from a document in your actual week and ask for the decisions, owners, and unresolved questions. Hand it an unfamiliar function and ask what it does, where it can fail, and which tests are missing. Ask it to rewrite a dense note without changing its meaning.
Use material you are authorized to process on that device. Local inference does not override data-classification, endpoint, or software-installation policy.
To try image input, include a file path in your message:
What is happening in this chart? /Users/you/Desktop/q2-costs.png
A few commands are worth knowing inside the chat:
/set verboseshows token counts and generation speed after each response/clearresets the conversation context/byeexits
From the shell, ollama ls shows downloaded models, ollama ps shows what is loaded, and ollama rm gemma4:12b reclaims the disk space.
Two tuning notes are worth the extra minute.
First, Ollama also offers a 7.2GB quantization-aware-trained build:
ollama run gemma4:12b-it-qat
Google designed that build with 4-bit inference in mind. Treat it as another candidate for your evaluations rather than assuming it will win every task.
Second, the 256K context window is a ceiling, not a guarantee that Ollama allocates all of it. Ollama currently chooses a default based on available VRAM: 4K below 24 GiB, 32K from 24 GiB to under 48 GiB, and 256K at 48 GiB or above. Check the CONTEXT column in ollama ps before handing it a long document.
Inside an interactive session, you can experiment with a larger window:
/set parameter num_ctx 32768
Watch Activity Monitor while you do. Context is where the memory goes.
What to expect, honestly
A 12B model quantized to four bits is not a frontier model, and pretending otherwise would waste your afternoon.
I tested it against the bounded work that fills an ordinary engineering week: summarizing a dense document, rewriting a decision note, explaining unfamiliar code, extracting actions from notes, and interpreting a chart. It was useful on those tasks and sometimes surprisingly good. It became less dependable when I asked it to preserve many constraints across a long sequence, reason about obscure facts, or sustain autonomous work over multiple steps.
That first pass was calibration, not a benchmark. A useful evaluation would score whether the model found every decision, preserved every constraint, grounded its claims in the source, and produced code that passed the tests. "It felt pretty good" is not an evaluation strategy.
The model also does not know what happened this morning. Google's model card lists a January 2025 training-data cutoff. There is no search engine behind the local model unless you wire one up.
On the Apple Silicon machine I tested, it streamed fast enough to feel conversational. Use /set verbose to capture the exact number for your hardware rather than borrowing someone else's benchmark.
That honest split is the point of the exercise. The interesting discovery is how much everyday work falls comfortably inside those limits.
Run the experiment
If you lead an engineering organization, the fifteen-minute version of this post is worth more than the reading time. The point is calibration.
Every vendor renewal now happens in the shadow of a locally run alternative with no per-token API charge that may be good enough for a growing share of bounded work. The size of that share is an empirical question. You cannot answer it without evaluations you own.
Leaders who have felt that floor firsthand negotiate differently than leaders who have only read about it. They also make better platform decisions because they understand what the local option can do, where it fails, and which controls production use still requires.
The local model does not replace the portfolio. It makes the portfolio real.
So run it tonight. Download Ollama, pull Gemma 4 12B, and hand it something real from your week. Then reread your API bill. The meter looks different once you have met the alternative.
Model availability, licenses, and Ollama behavior checked August 3, 2026.
