Guides

Deploying models

A deployment is a model you asked Clystria to run. You name a model, Clystria does placement, quantization, and routing. Sources can be the Clystria registry, Hugging Face, or a local path.

Basic deploy

$ clystria deploy llama-3.1-70b

Clystria resolves the model, picks a sane quantization for your available VRAM, plans placement, and returns an endpoint. To pin choices explicitly:

$ clystria deploy mistral-large \
    --quant awq \
    --replicas 2 \
    --max-context 128000 \
    --name chat-prod

Quantization

Clystria supports several formats and will down-quantize automatically if a model won't otherwise fit the pool. Pass --quant to override.

fp16 / bf16 — full quality, most VRAM
fp8 — near-lossless on Hopper/Ada
awq / gptq — 4-bit, great throughput
fp4 — fit enormous models on modest pools

Auto-fit: deploy a 405B model on a pool that can only hold it at 4-bit and Clystria will say so, quantize, and proceed — instead of failing with an out-of-memory error.

Scaling & autoscaling

Add replicas for throughput, or let the pool decide:

$ clystria scale chat-prod --replicas 4
$ clystria scale chat-prod --replicas auto --target-latency 250ms

The model registry

List what's running, inspect a deployment, or browse the curated registry of ready-to-forge models.

$ clystria models
NAME        MODEL              REPLICAS  GPUS  TOK/S  STATUS
chat-prod   mistral-large      2         4     2,310  live
code-01     qwen2.5-coder-32b  1         2     2,760  live

Zero-downtime updates

Rolling a new model version keeps the old replicas serving until the new ones are warm, then shifts traffic over.

$ clystria deploy chat-prod --model mistral-large-2 --strategy rolling

CLI reference — every flag for deploy, scale, and models.