Skip to content
All projects
ResearchResearch

Enterprise Local LLM & Tool-Calling Platform

Research into serving open-weight models locally with Ollama and vLLM so enterprise AI does not have to send all data to external APIs.

Overview

An exploration of local LLM infrastructure: open-weight models served on GPU hardware with tool calling and agent workflows.

Problem

Some enterprise data should not leave the organisation, which rules out sending every request to an external LLM API.

Solution

Serve open-weight models (Qwen, DeepSeek) on local GPUs through Ollama and vLLM, and give them tool-calling capabilities for agent workflows.

Architecture

Enterprise Local LLM & Tool-Calling Platform architecture, top to bottom
  1. 01

    Clients

    • Applications
    • Agents
  2. 02

    Serving

    • Ollama
    • vLLM
  3. 03

    Models

    • Qwen
    • DeepSeek
    • Open-weight models
  4. 04

    Hardware

    • RTX 4090
    • Dual-GPU server
  • Hardware experimentation on an RTX 4090 and a dual-GPU server environment.
  • Model serving is abstracted so local and hosted models can be swapped.

Key features

  • Local GPU inference
  • Tool calling with open-weight models
  • Agent workflows

Engineering challenges

  • Matching hosted-model tool-calling reliability with open-weight models.
  • GPU memory and throughput trade-offs when serving larger models.

Results

No measured outcomes are published for this project. I only list results that have been verified.