On-Premise Private AI Hardware Installation

The Brief

Chat based AI systems are revolutionizing how work is done. For a lot of ad-hoc use-cases

ChatGPT, Copilot, and Claude are fine and get the job done. There are situations in the enterprise environment that call for a more structured, secure, and customized approach.

Businesses have more complicated requirements than ordinary users. They may not be able to share the proprietary information of their customers with third parties. Sometimes they want to customize things to a particular workflow. They may need to consider the risk of resting critical load-bearing business functions on single third party providers. They may have data that is sensitive or has legal restrictions. They might have workloads that fall outside the typical API usage patterns or incur high cost, such as long running jobs.

The brief here was to build a hardware solution to meet these complex requirements. A dedicated server performing on-premise inference, acting as an in-house utility over the LAN. This meant proprietary information never leaving the building, no reliance on third party providers, full control over inference scaling to meet internal demand, and low ongoing inference costs for long running jobs.

Key Use-Cases of On-Premise AI

This business is using their local inference setup for analysing proprietary financial information, data extraction from documentation, summarisation, and deep search of their internal knowledge bases.

Here are more use-cases where local inference makes sense:

• Confidential document review: Some of the things agents excel at the most, are useful in exactly the settings where privacy is paramount. This is true of the legal and medical industries, for example, when querying and using a local company knowledge base without risk of leakage, similar to the active knowledge stores used in our form-filler agent. Patient and client data carry mandated privacy conditions by law. On-site inference is ideal for a legal or medical team who want to perform data extraction on a set of documents.

• Protecting proprietary code: This is important where the code itself is a high risk asset sought by competitors and other hostile entities. Total security is only guaranteed by making sure the code never leaves the building. Instead of hooking up your coding harness to OpenAI or Anthropic's APIs, you point them directly at the instance running on your local network and no code goes beyond the security boundary.

• Offline RAG search with air-gapped snapshots: Companies almost always have local technical documentation, documented procedures, shared document filesystems, internal wikis and other types of knowledge bases that hold a wealth of proprietary information. You can give your local inference server access to these sources and then use them safely during network outages or in air-gapped environments needing security. Adding local RAG search can mean the difference between an AI that truly understands your businesses and offers advice tailored directly to the context of your operations, or generic nonsense and hallucinations that don't apply. With an on-site system you can fold in your whole internal corpus without risk of leakage or security issues.

• Long running agent workflow cost savings: Some use-cases require indexing or processing of a huge amount of documents on an ongoing basis. These can be nightly data extraction batch jobs, automated receipt and invoice processing, or archival processes. Where you are running hundreds or thousands of queries per time period, for example in batch processing situations, you must be mindful of runaway costs. Local inference can be run at effectively zero on-going cost, especially where renewable energy such as solar is used to power it.

• Customized workflows and training: Sometimes you want to fine-tune your own model for specific use-cases or run customized domain adapters. LoRA fine-tuning of models can yield an AI that is specifically tuned to your business, without incurring the cost of filling context with specific documents. Instead of using a third party API you can own the entire training and inference stack yourself by doing it on-site.

Of course, local inference is not perfect for every situation. It is often a tradeoff between high powered remote systems and lesser workhorse level intelligence in local systems. For top-tier architectural reasoning or complex multi-step creative strategy it can often make sense to use a frontier model from one of the big labs. Local inference capabilities are rapidly advancing though, and you might be surprised at which automations can be taken offline and run locally today.

The Hardware and Software Architecture

Our goal here was the best inference possible per dollar spent within the kind of budget a small business can afford. Rather than blowing $20k on enterprise level graphics cards, we came up with a simple, accessible setup using a cheap second hand card and a machine built around it.

Here's what the final build looks like. We provisioned a machine with an NVIDIA RTX 3090 graphics card as the core execution substrate. We explicitly told the hardware vendor and assembly shop that VRAM is king, RAM is queen, and everything else is secondary. We just wanted a machine built specifically to feed and support that GPU and its VRAM, which is the inverse of what most people want when buying a PC (fast CPU, huge hard drive, etc.). With 24GB VRAM and 64GB RAM this hardware was more than capable and far less expensive than dual Xeons or $15k A100s, and yet it still runs production-ready 4-bit quant speeds on 14B–32B models.

Onto this machine we installed stock Ubuntu Linux running headless in the back of the office. This server becomes a LAN based utility with Ollama or llama.cpp installed and running as a persistent system service exposed to the local network, where anybody can send API requests using the OLLAMA_HOST environment variable. That means people can build and run scripts on their own machines, or plug in open source harnesses, and have the actual queries run on the server itself. We also made sure to configure an extended context setup of 64k tokens so it can comfortably digest whole documents.

For security we set up a lightweight sandboxing environment built on LXC containers. Admins can run a command to spin up cheap sandboxed containers for particular purposes and these are fully isolated from the parent system where Ollama is running inference. That means an autonomous coding agent reading files, browsing the web, and executing scripts is never going to run something it shouldn't or delete anybody's files. Agents can be set up to run inside isolated playpens on the main server with principle-of-least-authority access to exactly what they need and no more.

The Results

This setup has brought real peace of mind and productivity gains to the business. They can confidently perform local analysis and private investigations using proprietary information and customer data without any risk of leakage. There's none of the paranoia that comes with proprietary service NDAs, terms of service changes, data leaks, or accumulating nasty per-token metering surprises.

The machine has found use in collating and analyzing internal work log entries to find patterns, themes, and valuable insights using sensitive internal operations data you would never send off to a 3rd party. It's also been used for deep financial modeling and planning based on actual company numbers, feeding raw historical CSV transaction data and bank transactions in to forecast future budgets. Once again there's great peace of mind to be found in knowing this sensitive data never goes offsite.

Simply put, it's a magic computer brain in a box running on the local network. Pinging an endpoint with an inference API request feels much like sending off a page to a printer or storing a file on a network drive. In the end we get high utility paired with high confidence and security, and that was the goal from the start. Verdict: success!

If you want to explore an on-premise AI setup or discuss what kind of hardware fits your workload, please feel free to get in touch.