AI Flow · How a call travels

One call. Every hop accounted for.

  1. APPIts own key
  2. RELAYOn the same box
  3. GATEWAYLiteLLM
  4. MODELLab or cloud

From the app to the model and back: its own key, its own budget, a relay on every box, and a backup path when the gateway is off.

Last 30 days through the gateway

AI calls
10,635
Tokens
62.4M
Median gateway overhead
5 ms

01 · Per-app keys

Every app carries its own key.

Every app gets its own key on the LiteLLM gateway, with its own model list and its own budget. Nobody shares a key, so every token is booked to the app that spent it.

Dmotion

AI video studio
KeyKey hidden
  • Image
  • Video
  • Chat
Own budget
Set per app, booked per app

SparkChat

Private chat
KeyKey hidden
  • Local Qwen models
Own budget
Set per app, booked per app

Conductor

Agent orchestration
KeyKey hidden
  • Multi-model
  • Cloud
  • Lab
Own budget
Set per app, booked per app

Pilot

Browser agent
KeyKey hidden
  • Browser agent models
Own budget
Set per app, booked per app
apps calling the gateway
66
models used
68
of calls succeeded
96.7%
tokens (36%) served by models running in the lab
22.5M

Last 30 days · Prompts and answers are never stored by the monitoring.

02 · The relay

A relay on every box.

Apps never call the gateway across the network themselves. They call the AILogic relay on their own machine. The relay tries the gateway first. If the gateway doesn't answer, it replays the request straight to the provider using its own backup key.

AILogic relay DGX Spark Strix Halo Intel NUC Mac mini M4 Pro Mac mini M6 LiteLLM gateway Key · budget · routing · logs 21 lab models 74 cloud models Backup path · relay's own key
  1. 01

    Local

    The app talks to the relay on the same machine.

  2. 02

    Gateway first

    Key, budget, routing and logging all happen at LiteLLM.

  3. 03

    Fallback

    Gateway down, so the relay replays to the provider with its own backup key. The app's key never leaves the gateway path.

03 · Failure test

We switched the gateway off.

Gateway offline

69s

  1. Step 1 · 00 s

    Gateway off. On purpose.

  2. Step 2 · calls keep coming

    The relays notice and replay each request to the provider. The backup path takes 58 calls.

  3. Step 3 · 69 s

    Gateway back. Every relay switches back by itself, with nobody touching anything.

  4. Step 4 · result

    156 of 156 answered. Nothing failed.

Relay Gateway Provider Backup path · relay's own key
Answered
156/ 156
Via backup
58

Nothing failed.

04 · Routing patterns

Four ways to route. One is the default.

App · Relay · Gateway · Model

Default

Box relay with fallback

Most apps work this way: relay, then gateway, with a backup path to the provider.

App · Gateway · Model

Pattern 02

Gateway direct

Calls go straight to the gateway with a per-app key, for apps that live next to it.

App · Gateway or second URL · Model

Pattern 03

App-level second URL

The app holds a second address of its own to fail over to.

App · Model, gateway skipped

Pattern 04

Direct by design

Some calls skip the gateway on purpose:

  • Speech on the local network, for latency
  • The image and video servers
  • Subscription logins
  • "Off the record" chat that must not be logged
  • The monitor itself, which has to keep working when the gateway is down

05 · The network

Nothing listens on the internet.

Four internet links come in through a dedicated firewall, then a managed gateway, switches and access points. The AI machines sit on their own isolated segment. Every server dials out to Cloudflare through its own outbound-only tunnel. People come in through Cloudflare Access single sign-on.

4 links → firewall → gateway → isolated AI segment → outbound tunnels → Cloudflare Access → you
Link 1 Link 2 Link 3 Link 4 Edge Firewall Gateway Switches · APs 5 machines Isolated AI segment Outbound-only tunnels Cloudflare Access Single sign-on You
apps published to the web
73
ports open to the internet
0
uptime over 30 days
99.993%

No addresses shown here, or anywhere on this site.

Switch it off. It still answers.

Answered, gateway off
156 / 156
Gateway switched off
69 s
Median gateway overhead
5 ms

Designed, built and broken on purpose by DK.