milkinfrastructure.com/docs

Start with Milk.

Milk has two small programs. Parlor sends OpenAI-style requests to a model and saves selected completed requests and answers. Milk Man reads those records to describe real use, write model tests, create training data, and compare smaller models.

Two programs

Milk Parlor

The Rust gateway. It authenticates, forwards, streams, and saves selected complete request + response pairs.

Milk Man

The local command-line tool. It runs the summary, model-test, training, comparison, and serving jobs.

Connect an app

There is no Milk SDK. Use the official OpenAI package.

  1. Install the official OpenAI package.
pip install openai
# or: npm install openai
  1. Point it at Milk Parlor with your operator-issued Milk key.
export OPENAI_BASE_URL=https://parlor.milkinfrastructure.com/v1
export OPENAI_API_KEY='your Milk key'
  1. Keep the application call unchanged.
from openai import OpenAI

client = OpenAI()
answer = client.responses.create(model="your-model", input="Hello")
print(answer.output_text)

Milk supports POST /v1/responses and POST /v1/chat/completions, including streaming. It does not claim the rest of the OpenAI API.

What happens

your app Milk Parlor your model object store Milk Man
1. Forward
Parlor sends the request to the configured model and streams the answer back.
2. Save
After completion, Parlor writes both sides to local or S3-compatible storage without holding up the answer.
3. Process
Milk Man counts and summarizes new conversations before deciding whether it has enough data for another job.
4. Propose
Milk Man may prepare a traffic rule for a new model. A person must approve and sign it.

Develop locally

Clone both repositories. Milk Parlor needs Rust. Milk Man needs Bash, Python 3, Git, and curl. Processing captured records also needs zstd.

git clone https://github.com/milkinfrastructure/milk-parlor.git
git clone https://github.com/milkinfrastructure/milk-man.git

Run the gateway

Set its Milk key list, model URLs and keys, route public key, and storage values. The gateway README contains one complete local setup.

cd milk-parlor
cargo run --locked

open the gateway setup

Run Milk Man

cd milk-man
bin/man dashboard
# open http://127.0.0.1:8765

The dashboard stays local. It shows the current Milk Man session, gateway health, saved-data progress, and which job settings are present. It never displays secret values.

Start one development task below before using its prompt box.

Give Milk Man a development task

export OPENAI_API_KEY='your development key'

bin/man develop \
  --workspace milk-man=/absolute/path/milk-man \
  --workspace milk-parlor=/absolute/path/milk-parlor \
  -- "make one small reviewed change"

Run the data loop once

export MILK_SCOPE_ID=11111111-1111-4111-8111-111111111111
export MILK_STORE_KIND=local
export MILK_STORE_ROOT="$PWD/.milk-objects"

bin/milk status
bin/milk operate --once

operate --once advances ready work and exits. It does not sleep. When nothing changed, it makes no inference or GPU call.

open the complete job setup

Saved data

Milk gives each customer or test run one unique ID. Folder-like object storage, such as Cloudflare R2 or Amazon S3, keeps the full history under that ID:

milk/v2/scopes/<scope-uuid>/
├── c/           completed conversations
├── s/           summary checkpoints
├── readiness/   the next allowed step
├── e/           model-test cases
├── d/           training and evaluation splits
├── m/           model records
├── v/           comparison results
└── p/           unsigned route proposals

Saved records are never changed. A small current.json points to the latest version. Model weights stay outside Git and container images.

Read next