Why Local AI Models Exist And When Your Business Actually Needs One

The law firm in this story was sending privileged client documents through a public AI service. Not maliciously. Casually, because the AI tool they subscribed to was the most useful thing anyone in the office had found for document review, and nobody had thought carefully about what happened to a privilege log or a confidential settlement memo once it entered a commercial chat interface.

This is the story of how the firm moved to local AI, what they found when they did, and what the setup actually looks like for a professional services firm that handles sensitive information.

Before: the habit that created unexamined risk

The discovery happened during a staff review meeting. A paralegal mentioned offhand that they had been running deposition summaries through an AI chat tool to create faster briefs. The summaries contained client names, case strategy, and factual assertions that would be material in litigation. Nobody had told them not to. The tool subscription had been added to the office as a productivity resource with no accompanying policy about what to do with confidential information.

The senior partner's reaction was measured but the decision was immediate: stop sending client documents through any service where the prompts leave the building. The firm represented several clients in active litigation where the opposing party would benefit significantly from knowing the firm's internal case framing. The accidental disclosure risk was real.

The alternative was not to stop using AI. It was to use AI that ran locally, with no prompt leaving the firm's own hardware. The search for how to do that practically led to open-weight models and Ollama.

Month one: choosing the model and the hardware

The hardware question came first. The firm had several workstations running the usual combination of practice management software, document review platforms, and Office. None of them had been spec'd for AI inference. The highest-specification machine was a newer desktop with 32 gigabytes of RAM and a capable GPU, purchased for video depositions and trial presentation work.

That machine was sufficient for the 20 billion parameter GPT-OSS model. The download via Ollama took about 35 minutes and required 13 gigabytes of disk space. After the download the model loaded into memory and accepted a first test prompt entirely offline. The disconnect test, sending a prompt with the internet cable physically disconnected, confirmed that no data left the building.

The 120 billion parameter model was not a practical option for the firm's existing hardware. Running it would require dedicated infrastructure the firm was not prepared to invest in. The 20B model's output quality on the tasks the firm needed, document summarization, clause identification, draft comparison, and first-pass brief writing, was adequate for the intended use. Perfect quality was not the requirement. Adequate quality without confidentiality risk was.

The firm installed LM Studio as the interface for the 20B model alongside Ollama, finding that attorneys and paralegals with no technical background were more comfortable with LM Studio's chat interface than Ollama's terminal-oriented workflow. Both ran the same downloaded model. The choice was purely about which interface the team would actually use.

Month two: connecting the firm's documents to the local model

A local model in its default configuration knows nothing about the firm's specific cases, clients, or internal conventions. It can summarize a document you paste into it but cannot cross-reference that document against the firm's prior work or internal knowledge base. The retrieval step is what transforms a local model into a useful legal assistant.

The firm used AnythingLLM, a locally-runnable document management layer that connects to Ollama and Hugging Face models. It accepts document uploads, stores them in a local vector database, and provides a chat interface where the model can search the document collection when answering questions. Everything, the model, the vector database, and the document store, runs on the firm's own hardware with no external dependencies.

The initial document load took a focused day of work. Attorneys contributed a curated set of documents: sample briefs from prior cases stripped of client identification, internal memos on procedural conventions, relevant case law summaries, and annotated deposition outlines. The goal was not to replicate the firm's full archive but to give the model a knowledge foundation specific to the firm's practice area and working style.

The first useful interaction came within the first week. An attorney working on a motion brief asked the system to identify prior firm briefs that had addressed a similar procedural argument. The system returned three relevant documents from the uploaded collection with specific passages cited. The attorney used those passages as a starting point rather than beginning from scratch. The time saving on that single task was approximately two hours.

Sensitive client documents, including active case files with client identification, required a separate policy decision. The firm decided that highly sensitive materials in active matters would not be loaded into the retrieval system, even locally. They would be pasted directly into the local model interface on a case-by-case basis for specific tasks, but not stored in a searchable collection where a staff member could inadvertently surface them while working on a different matter.

Six months in: the usage pattern and the policy that makes it safe

Six months after the initial setup, the pattern of use in the firm had settled into two clear categories. Routine document work, including summarizing long discovery documents, comparing two contract drafts for key differences, and drafting standard correspondence in the firm's voice, ran entirely through the local system. No judgment call required. Local by default for anything that touches client identity or case strategy.

Research and general legal knowledge tasks, where the question is about law rather than about a specific client, continued to run through cloud AI tools. Looking up how courts in a specific circuit have ruled on a procedural issue, getting a plain-language explanation of a regulatory change for a client memo, or drafting a general FAQ for the firm's website, none of those required local processing because they did not involve sensitive information.

The policy document the firm produced covers three things: which task categories require local processing by default, which categories are permitted to use cloud tools, and what to do when a task does not fit clearly into either category. The last section is the most important. Ambiguous cases go to local processing by default. Cloud processing requires a deliberate decision that the content is not sensitive. The default protects against the casual habit that created the original risk.

The version tracking addition came from the senior partner's suggestion during a monthly review. The firm implemented a simple log for any AI-assisted output that goes to a client: date, task description, model version used, and which attorney reviewed and approved the output. The log is a lightweight record that documents the human judgment applied before any AI output reached a client context. It adds about two minutes per task and creates a defensible record without requiring any technical infrastructure beyond a shared spreadsheet.

The overuse problem the local setup also solved

An unexpected benefit emerged six months in. The cloud AI tools the firm had been using were subscription-based, and the firm's usage had been high enough to periodically hit plan limits during busy periods. An associate would be mid-task at 9pm before a filing deadline and find the service at capacity. Switching to a different tool or restarting the approach from scratch at that moment was disruptive and stressful.

The local model has no rate limits, no capacity constraints, and no subscription tier that governs how many requests can be made in a day. The model is available as long as the hardware is on. For a firm where the work does not stop at business hours and where filing deadlines create genuine crunch periods, this availability assurance has practical operational value.

The cost comparison over six months was also favorable. The firm's cloud AI subscriptions totaled several hundred dollars per month for the level of usage the team maintained. The local setup required a one-time hardware cost for a GPU upgrade to the existing workstation, spread over months of use, with no ongoing per-seat cost for the local model. The retrieval system software was also free. The total cost of the local setup over six months was lower than the cloud subscriptions it replaced for the tasks that moved to local processing.

The non-quantifiable return was the confidence that sensitive client information was not in anyone else's system. For a firm that depends on client trust as the core of its business, that confidence had value that did not appear in any expense comparison but influenced every client relationship.

What the setup actually requires from the firm

Honesty about what the local setup requires is important for any firm considering a similar path. The installation and configuration is straightforward but takes about two days of focused attention from someone who can follow written instructions and troubleshoot basic technical issues. The document curation for the retrieval system requires attorney time to decide which documents are appropriate to load and to organize the initial collection. The policy documentation requires a partners-level conversation about where the firm's risk tolerance sits.

None of those requirements are prohibitive for a functioning law firm. All of them require deliberate allocation of time rather than passive adoption. The risk of a passive adoption without a policy is exactly what the firm discovered at the beginning: useful tools used without boundaries create liability exposure that the firm did not knowingly accept.

The model will not solve legal problems. It will not replace attorney judgment on complex questions. It will not catch the kinds of errors that require legal expertise to recognize. What it does is reduce the time attorneys and paralegals spend on the mechanical tasks that precede and follow the work that requires their expertise. Document summarization before a review session. Draft comparison before a negotiation. Standard communication drafting before a client call. These mechanical tasks take real time, and recapturing that time does not require the model to be perfect. It requires the model to be accurate enough that the attorney's review step confirms rather than rebuilds.

For firms that handle sensitive client data and have been reluctant to adopt AI tools because of confidentiality concerns, the local model path removes the primary reason for reluctance without requiring any sacrifice in the core capability. The model is capable. The data stays in the building. The trade that used to look like a binary choice between capability and confidentiality turns out not to be binary at all.