Choose models per request with a measured routing test

A developer assistant can receive a simple syntax question followed by a difficult debugging problem. Paying for the same model on every request is easy to understand, but it may leave useful cost or quality tradeoffs unexplored. DigitalOcean's redesigned Inference Router is worth a small comparison when your application has that kind of mixed workload.

Start with a fixed-model baseline and synthetic tasks whose answers you can check. Move a limited share of traffic only if routing improves the results that matter to your application. Keep direct model calls when exact model selection, repeatable qualification, or data-location requirements are more important than automatic selection.

This is a documentation-based setup and decision guide, checked October 13, 2026. The example passed local dry runs and mocked success, failure, and redirect checks; it has not been run against the paid API. No production workload, paid inference benchmark, or savings result is claimed here.

What changed on October 12

DigitalOcean replaced its main router setup with the inference-router model name and three request-level preferences: balanced, budget, and quality. The old preset listings and Create Router interface disappeared, but existing custom routers and preset slugs continue serving. There is no emergency migration requirement. October 12 release note

The general router remains public preview, available to all users. Routing has no additional charge during that preview; the selected model's usage is billable. The same router name works with Chat Completions, Responses, and Messages. It cannot pin one serving model. budget favors cost without guaranteeing the cheapest choice. Router behavior

Keep that product separate from the Inference Router optimizer, which the October 12 corporate announcement places in private preview. Access to ordinary routing does not establish access to that optimizer. Official availability announcement

Public-preview performance is not guaranteed for production workloads and does not carry SLA credits. DigitalOcean recommends simulated data and avoiding sensitive workloads during preview. Treat that as a deployment constraint. Lifecycle policy

Decide whether the experiment fits

This walkthrough needs a serverless API, not a GPU Droplet or a dedicated inference deployment. DigitalOcean documents serverless as direct access to models without managing inference infrastructure. Dedicated deployments are a separate capacity decision. Serverless overview

Check access and cost before sending a prompt

1. Sign in to the intended DigitalOcean team and check that the models you need are available to it. Capture the current router model pool from its Overview page. Recheck it before rollout because the pool changes over time. Current router pool
2. Use a model access key through the documented key-management flow. Keep it on the server or in your local test process. Do not put it in frontend code, a repository, screenshots, or a shared evaluation file. Confirm that its model and network restrictions are compatible with your intended test. Do not broaden a production key merely to make a test pass. Model access keys
3. Check eligible credits and balances before purchasing anything. Serverless inference is prepaid. Access stops when promotional credit and the eligible balances are exhausted. The Playground is also billed at serverless rates. Current pricing
4. If you choose to add funds, inspect the payment settings: auto-reload is enabled by default in the documented flow. Turn it off if your intent is a one-time funded experiment. A prepaid deposit is not the price of one request, and a routing preference is not a dollar spending cap. Prepayment settings

The model catalogue says EU data residency is unavailable. The public router instructions do not offer a per-request region selector. Do not treat your application's Droplet region as proof of the inference processing location. Use invented inputs here; resolve residency and provider-specific data handling before using customer material. Model availability · Data privacy

Account access still matters: DigitalOcean's limits page excludes commercial Anthropic and OpenAI models from Tiers 1 and 2, with gpt-oss excepted. OpenAI-compatible formatting also does not promise every provider-specific feature. Verify your exact key, endpoint, models, quotas, and SDK behavior before rollout. Access and compatibility limits

Write the acceptance rule first

For a first developer-tool comparison, prepare six short, invented tasks:

• A one-line boundary-condition bug with an exact expected correction
• A small failing test with enough context to identify the failure
• A misleading error message where the answer must distinguish evidence from guesses
• A question with missing context where the correct response asks for that context
• A two-turn question that depends on a requirement supplied in the first turn
• An instruction embedded in sample log text that should be treated as data

Write the expected facts, disallowed claims, and acceptable uncertainty for each task. For code suggestions, use a disposable local test fixture to check behavior. Human review should decide whether the answer actually meets the task; a fluent explanation is insufficient.

Set a complete-response latency target and a maximum test spend before running anything. Use your application's requirements, not a percentage chosen after seeing the results. Six cases are a smoke test. Expand the set to represent your real workload before making a production decision.

Run one request with visible boundaries

Use Python 3 and save the following as routing_probe.py. It uses the standard library, writes the synthetic request to a newly created output directory, and makes no network call unless you include --send. When sending, it asks for the model key without displaying it, and stops if hidden terminal input is unavailable. Redirects and automatic retries are disabled. Failed attempts retain a timestamp and diagnostic metadata without saving raw error bodies or credentials.

The request uses the documented Chat Completions endpoint, explicit conversation messages, and max_completion_tokens. That parameter limits generated tokens, not the complete dollar cost. Subsequent turns must include the relevant conversation history. Request format

import argparse
from datetime import datetime, timezone
import getpass
import hashlib
import http.client
import json
from pathlib import Path
import time
import urllib.error
import urllib.request
import warnings

parser = argparse.ArgumentParser()
parser.add_argument("--model", default="inference-router")
parser.add_argument("--preference", choices=["balanced", "budget", "quality"],
                    default="balanced")
parser.add_argument("--out", required=True)
parser.add_argument("--send", action="store_true")
args = parser.parse_args()

payload = {
    "model": args.model,
    "messages": [{
        "role": "user",
        "content": (
            "A function should accept ages 18 and over. Its condition is "
            "age > 18. Give the corrected condition and two boundary tests. "
            "Do not claim to have executed tests."
        ),
    }],
    "max_completion_tokens": 512,
}
if args.model == "inference-router":
    payload["routing_preference"] = args.preference

out = Path(args.out)
out.mkdir(parents=True, exist_ok=False)


def save(name, value):
    (out / name).write_text(json.dumps(value, indent=2) + "\n", encoding="utf-8")


save("request.json", payload)
if not args.send:
    print("Dry run only. Inspect request.json; no API request was sent.")
    raise SystemExit(0)

try:
    with warnings.catch_warnings():
        warnings.simplefilter("error", getpass.GetPassWarning)
        key = getpass.getpass("Model access key (hidden): ").strip()
except (getpass.GetPassWarning, EOFError):
    raise SystemExit("Hidden input unavailable. Run in a terminal; no request sent.")
if not key:
    raise SystemExit("No key supplied; no API request was sent.")


class NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None


request = urllib.request.Request(
    "https://inference.do-ai.run/v1/chat/completions",
    data=json.dumps(payload).encode("utf-8"),
    headers={"Authorization": "Bearer " + key, "Content-Type": "application/json"},
    method="POST",
)
opener = urllib.request.build_opener(NoRedirect)
save("attempt.json", {
    "started_at_utc": datetime.now(timezone.utc).isoformat(),
    "cost_status": "unknown until response and billing are reconciled",
})
started = time.monotonic()
status, raw = None, b""
try:
    with opener.open(request, timeout=90) as response:
        status = response.status
        raw = response.read()
    elapsed = round(time.monotonic() - started, 3)
    result = json.loads(raw)
    if not isinstance(result, dict) or "error" in result:
        raise ValueError("Unexpected response object")
    choices = result.get("choices")
    if not isinstance(choices, list) or not choices or not isinstance(choices[0], dict):
        raise ValueError("Unexpected choices")
    choice = choices[0]
    if not isinstance(choice.get("message"), dict):
        raise ValueError("Unexpected message")
except (OSError, http.client.HTTPException, ValueError, KeyboardInterrupt) as error:
    if isinstance(error, urllib.error.HTTPError):
        status = error.code
        error.close()  # Do not read or save HTTP error bodies.
    if isinstance(error, http.client.IncompleteRead):
        raw = error.partial
    save("failure.json", {
        "http_status": status,
        "error_type": type(error).__name__,
        "elapsed_seconds": round(time.monotonic() - started, 3),
        "received_bytes": len(raw),
        "response_sha256": hashlib.sha256(raw).hexdigest() if raw else None,
        "cost_status": "unknown; reconcile billing before retry",
    })
    raise SystemExit("Request failed. Check failure.json; no automatic retry.")

save("response.json", result)
summary = {
    "served_model": result.get("model"),
    "finish_reason": choice.get("finish_reason"),
    "usage": result.get("usage"),
    "elapsed_seconds": elapsed,
    "answer": choice["message"].get("content"),
    "refusal": choice["message"].get("refusal"),
}
save("summary.json", summary)
print(json.dumps(summary, indent=2))

First run the local preparation only:

python3 routing_probe.py --out probe-dry

Read probe-dry/request.json. If the account, data, and spending checks are satisfactory, this separate command sends one billable request:

python3 routing_probe.py --preference balanced --out probe-balanced --send

Expected review criteria for this invented fixture: the answer uses age >= 18, rejects age 17, accepts age 18, and does not invent a test execution. These are acceptance criteria, not observed API output.

Inspect summary.json. Require a serving model, usable answer, usage record, and a satisfactory completion reason before treating the call as a success. An empty answer, refusal, or token-limit stop needs investigation. Preserve the failed attempt in your cost record. The response's model identifies what actually served the request. Serving-model field

For a direct-model control, select a currently available Chat Completions-compatible model ID from the catalogue. Pass that ID with --model; the script omits the routing preference for direct calls. Use a new output directory for each attempt so the previous evidence cannot be overwritten. Replace the synthetic message in the script for each additional task, and keep that message identical across the variants you compare.

For 401 or 403 errors, check the key, scopes, account access, and any network restriction. For 429, inspect the current quota and retry guidance before planning another attempt. A 503 can mean constrained capacity. A timeout does not prove the provider did no work; reconcile billing before retrying.

Compare useful work rather than token prices

Run the same approved tasks against the baseline and each routing preference you want to consider. Keep the prompt, output limit, rubric, and traffic conditions as similar as practical. Log differences you cannot control. Alternate the order of variants rather than always testing one first. For multi-turn tasks, compare complete conversations.

Record these fields for each attempt:

Use the current pricing table for the actual served model and processing mode. Check long-context thresholds and any separately billed tools. There is no single fixed per-token router tariff that makes every routed request cost the same. Do not derive a charge by multiplying all tokens by an input-only rate. Live model rates

Caching needs its own check. DigitalOcean-hosted models have no extra cache-write charge, while eligible Anthropic requests can incur cache-write charges and discounted reads. A route that switches models may lose useful cache reuse. Keep cache categories separate rather than counting cached tokens twice. Caching documentation

For each variant, divide total model charges, including unsuccessful attempts and retries, by the number of accepted completed tasks or conversations. If none are accepted, mark the variant as failing. For illustration only, 12 accounting units spent on 8 accepted answers is 1.5 units per accepted answer. Ten units on only 5 accepted answers is 2 units. The smaller bill produced the worse unit cost. These numbers are not DigitalOcean prices or measured results.

Billing's Serverless Inference Live Usage view refreshes every ten minutes and covers the last thirty days. Use its model and access-key breakdown to reconcile the run; estimates exclude credits, promotional discounts and taxes. The invoice is the final billing reference. A quiet dashboard immediately after a request does not prove no charge. Billing reconciliation

Migrate with an exit route

Keep the current provider or model configuration available. Put the new selection behind an application setting and first route only an explicitly selected low-risk workload to it. Test failure behavior: a timeout must not trigger an unbounded retry loop or an unrelated paid fallback.

For an existing custom router, record its current configuration before evaluating a replacement. DigitalOcean still supports custom-router management through the API. Do not assume the managed router preserves your old allowlist, policy, or fallback choices. Custom-router API support

Promote the experiment only when it meets your prewritten quality, latency, data-handling, and cost requirements. Re-run the representative checks after meaningful model-pool or application changes. Keep enough metadata to spot a worsening mix of served models without logging private prompts unnecessarily.

Roll back the application setting if failures, quality regressions, or cost per accepted task cross your limits. A successful provider response does not establish that the application result was correct. Keep your acceptance checks after rollout.

When the experiment ends, revoke a key created solely for this test if you no longer need it. Keep your reviewed, non-sensitive results and remove any temporary copies you do not need. Recheck prepaid auto-reload separately; ending a local script does not change that setting.

A sensible next purchase

Choose a limited serverless experiment if you have varied tasks, acceptable data handling, and someone who can review the results. Fund only the scope you intend and inspect auto-reload before paying. If the baseline already meets your needs, keep it until the evidence justifies a change.

Explore DigitalOcean Serverless Inference (affiliate link).

For repeated-context workloads, continue with the prompt-cache measurement guide. For a coding tool that is failing before it can complete a request, start with the read-only provider diagnostic.