It's time for the regular person to start testing local AI models
Earlier in the year, I wrote a post arguing that local AI models (that run on your laptop, and not in the cloud) are an absolutely necessary insurance policy as things play out in the AI era, but they were pretty much out of reach for the average user. For this piece, the average user means someone with a modern laptop with 16-32 GB of RAM and basic built-in GPU (if any). That likely represents 99% of the AI user population (aka the silent majority), but the voices that dominate the entire discourse (superusers, tech bros with GPUs, and anyone with financial incentives within entire AI supply chain) comprise the elite 1% only.
Large language models are awesome, and they are useful for all kinds of use cases. But privacy, cost, and the longevity of our own critical thinking skills still matter. The question worth tracking is when the average user can get their hands on AI models they can truly own, without fear of personal data being sent to third parties, with the ability to tune the model if they choose to, and most importantly, without paying anything more on top of the devices they already have.
So the real question is: when are local models good enough for your own use cases, and when can they start replacing the pricey, privacy-invasive models we’re all depending on right now?
TL;DR
AI economics are changing and it’s getting pricey out there. Companies are jacking up subscription and token prices (doubling in some cases), often accompanied by quality and service degradation to manage the surge in demand.
Open source & fully local models from Gemma & Qwen released in April 2026 are surprisingly capable, which is giving me a lot of hope.
It’s the right time to start tracking local AI readiness across several use cases like agentic coding, personal knowledge management, chat & research. Everything should be running on your laptop and not on expensive external GPUs.
As a proof of concept, the research for this post came from locally running Gemma & Qwen models on my laptop. More fully local & private use cases for personal & proactive agent concepts are next.
Next series of posts will cover some of these use cases and setup guides for you to try out these models without getting into the weeds.
AI economics: the status quo is breaking
I’ve had a genuinely fun ride with frontier models like Claude & ChatGPT, especially for agentic coding and building cool projects personally & professionally. But the experience from Nov 2025 - Feb 2026 is not the same anymore at the end of April 2026.
At the start of the new year, a $20 plan from Claude and a $20 plan from ChatGPT (for Codex) was enough to build a lot: my personal website, and some cool gamified data science experiments here and here. In the early days, coding with Claude & ChatGPT meant the old copy-paste approach, but starting with Sonnet 4.5 (Sep 2025), we truly entered an agentic coding era. Copy-paste was no longer needed, and the models were able to build something that works pretty quickly inside of VS Code. It was still very sloppy, but Opus 4.5 (Nov 2025) took it to a new level of high quality and automated code generation that felt magical even to someone who’s been coding for almost 20 years now. At the same time, GPT 5.3 & 5.4 with Codex were pretty awesome too, only marginally behind Claude in the “fun” factor.
But since then, things have changed a LOT.
For me, the returns from using these overpowered tools are starting to diminish. And I don’t think I’m the only one. The dopamine hits from watching terminal tik-tok wear off. Keeping up with every new feature is exhausting, and the amount of back-n-forth needed to get high quality output can reach the point where doing it yourself takes the same amount of time. It’s time to zoom out and take a pause.
In the last couple of months, we’ve seen model degradations, price hikes, and really bad communication from the AI model providers. This information may be too niche for some of my readers, but the latest drama unfolded around Anthropic’s severe quality degradation with the Claude models.
Anthropic’s April 23 postmortem (Source) traced Claude Code quality degradation to three separate changes that shipped between March and April. The deep technical details are worth summarizing:
On March 4, they downgraded the default reasoning effort from
hightomedium, because the high-effort mode was freezing the UI for some users. Users noticed Claude felt dumber, but didn’t know why. Reverted April 7 after a lot of online backlash.On March 26, they shipped a caching optimization with a bug. After an idle session, Claude’s thinking history got wiped on every single turn instead of just once. Claude became forgetful and repetitive. They couldn’t reproduce the bug internally for over a week. Two unrelated experiments masked it. Fixed April 10.
On April 16, they added a system prompt telling Claude to “keep text between tool calls to ≤25 words” to cut down verbosity. That single prompt line hurt coding quality by ~3%. Reverted April 20.
I’m sharing this because it’s a useful peek behind the curtain, and probably a preview of what’s coming next.
Early, innovative tech will have bugs. It doesn’t need to be perfect. But AI providers market their products as if these are god-like magical tools, while the companies consuming them lay off workers saying they’ve been ‘replaced’ by AI. Neither of these narratives matches the reality at the ground level, where the experience has shifted from being user-centric to being company-centric.
And the pricing? A $100/month plan now can feel worse than a $20 plan did in Oct 2025. Other models like GLM, Kimi, Minimax, and of course GPT all either doubled their prices or nerfed the lower tiers, leaving the higher tier as the only option that delivers an equivalent experience. We have officially entered the enshittification stages of AI. And with every new model release, the excitement level is also reaching a diminishing return. Reddit & X posts also seem like they run content on a schedule. This meme below might seem familiar:
The Rise of Small, Capable Models
Earlier this year, when I wrote my first post about local AI models, I tried several small models from Qwen, Gemma, Mistral, Hermes, GLM, and others in the 8–30B parameter range. They all fit in memory for my 48GB M4 Pro Macbook, but they didn’t really “work”. The context windows were too small, there was barely any tool calling, no vision capabilities (you had to use separate models), and the quality was way off compared to Claude or Codex.
That made a “good enough” local setup feel years away, especially since buying NVIDIA GPUs was not on the table. Most of my readers should not fall into that rabbithole either. Leave that for the r/LocalLLama crowd, they’re ahead of the curve, but also deep into optimizing for the tiniest of details and building bigger and bigger GPU rigs to run very large models at home. All that is not for the average person.
Recent releases of some small, open-source models from Gemma (Google’s open-source division) and Qwen (Alibaba) have bridged that gap significantly. These are the first releases where I felt, “okay, this is actually getting close.”
Gemma 4 models (link):
Super small models: 2B and 4B effective parameter models, built for mobile and in-browser deployment. I’m excited to see these run on phones especially.
Medium-sized dense and MoE models: 31B (dense) and 26B (MoE). Both handle agentic coding and daily-driver work. The MoE variants require less memory.
Qwen 3.5 and 3.6 models:
Small: Qwen3.5-9B — runs comfortably on a laptop with 16 GB+ RAM.
Medium: Qwen3.6 26B (dense) and Qwen 3.6 35B A3B (MoE) — excellent coding, reasoning, and tool-calling support.
They all offer reasoning, multimodality (text, image, audio, video), massive context windows (128K–256K), tool calling (critical for agentic coding), and system prompt support for tailoring model behavior. This was the missing piece a few months ago. The earlier models felt like demos, and now they feel closer to Sonnet 4 or Sonnet 4.5 (especially the medium-sized models).
Pair up these open-source models with some open-source coding harnesses like Pi (my fav) or OpenCode (my 2nd fav), and you are living the open-source dream!
Use Cases: Privacy, Trust, and Zero Cost
For the average person, the motivations for moving to local AI models shouldn’t necessarily be about being “contrarian”. It should be about the fundamental value these tools actually provide:
Data Privacy: What companies already learn from your browsing activity and app usage, via cookies and device IDs, has been supercharged by AI-based data collection, because users literally reveal everything about themselves to these tools, willingly. (This is also one of the reasons why I never use Gemini or Meta AI models. I only use Gemma models from Google’s research division because they’re open-source and private.)
Trustworthiness: If you want, local models can be tuned to behave exactly the way you need them to. Hallucinations won’t be solved since they’re an inherent problem of LLM architecture, but you can make their behavior more predictable and won’t be beholden to an AI company’s policy changes.
Cost: Once you have the hardware, i.e., your laptop and phone, the marginal cost of inference is effectively zero.
What can you do with these models today:
Coding Assistance: Like I mentioned earlier, pair these models with coding tools like Pi or OpenCode, and you have the same Claude Code setup you’re used to.
Personal Knowledge Management (PKM): If your notes, files, and documents live on your laptop (I use Obsidian, for example), then all of that can be accessed by your local AI using tool calling now. And none of this sensitive, personal data is sent to 3rd party servers.
Personal Chat & Research Companion: Local models also have web search abilities, so everyday AI conversations can pretty much be replaced (probably the easiest use case).
Proof of Concept: Gemma 4 26B A4B and Qwen 3.6 35B A3B ran on my laptop using LMStudio + OpenCode and helped with some of the research for this post (the writing is entirely my own). They were fast enough, and the reasoning was sufficient for the initial drafting and doing some basic web searching. Would GPT 5.5 or Claude have produced better output? Absolutely. But is this good enough? Yes, it is, and cost me nothing.
What’s Next
This is just the beginning of a series of posts on local AI, now that capabilities are getting better and models are getting small enough to fit on normal laptop RAM. A few things coming next, especially for folks who are interested in trying some of this out on their own too:
Setup Guides: Step-by-step instructions for the “regular person” to get these models running on their laptop without going into the weeds. Coming up in the next post.
Local AI Readiness Benchmark: I’m building a lightweight benchmarking tool that others can run to assess how well current small models handle everyday use cases like coding, extraction, and summarization on typical laptops. This will help understand when YOUR use-cases (as opposed to cryptic benchmarks) are fulfilled by the local AI models.
Tuning for Trust: Researching methods to fine-tune or prompt small models specifically for low sycophancy, higher trust, and high privacy.
The “Proactive Agent” side quest: Moving toward proactive agents that can manage email, calendar, and reminders, and eventually, the fun part, a proactive agent that works off long-term goals and behaviors to assist without being asked. If you’ve heard of OpenClaw, it’s the same idea, but the goal is to build a thin, tunable slice of that, while also exploring whether LLMs can be combined with traditional ML for some of this. This is ambitious, but could be a lot of good learning.
Thanks for reading if you’ve reached all the way down here. Please let me know your thoughts if any of this resonates with you, or if you have any ideas about what I should research or write about next.