Tutorial: Claude + LiteLLM on Azure — Logging, Log Extraction, and Evaluation Data

Audience: QA / AI-evaluation engineers who need to understand a deployment where a Claude model is served through a LiteLLM proxy hosted in Azure, and who want to extract log data from Azure for analysis and model evaluation.

Companion page: Claude Telemetry Reference — request/response parameters element by element

Last updated: 2026-08-05

Table of Contents

1. The Big Picture — Architecture
2. What is Claude, and What is "Claude 3P" (Third-Party)?
3. What is LiteLLM and How It Works
4. What LiteLLM Itself Can Log
5. Azure Logging — The Landscape
6. What Gets Logged Where (End-to-End Trace)
7. How to Retrieve Log Data from Azure
8. KQL Query Cookbook for LLM Logs
9. What Else You Can (and Should) Log
10. From Logs to Evaluations
11. Privacy, Cost, and Retention Considerations
12. Quick Reference Card

1. The Big Picture — Architecture

A typical "Claude behind LiteLLM on Azure" deployment looks like this:

┌─────────────┐     ┌──────────────────────────────────┐     ┌───────────────────┐
│  Client App  │────▶│   LiteLLM Proxy (in Azure)       │────▶│  Claude Provider   │
│  (your code, │     │   - App Service / AKS /          │     │  - Anthropic API   │
│   tests,     │     │     Container Apps               │     │  - AWS Bedrock     │
│   eval jobs) │     │   - OpenAI-compatible endpoint   │     │  - MS Foundry      │
└─────────────┘     └──────────────┬───────────────────┘     └───────────────────┘
                                    │
                     ┌──────────────┴───────────────────┐
                     │  Logging / Telemetry              │
                     │  - LiteLLM spend logs (DB)        │
                     │  - stdout/stderr container logs   │
                     │  - Azure Monitor / App Insights   │
                     │  - Log Analytics workspace (KQL)  │
                     └───────────────────────────────────┘

Three layers, each with its own logs:

The key insight for evaluation work: the LiteLLM layer is where the richest LLM-specific data lives (prompts, completions, tokens, latency, cost per request). Azure platform logs add infrastructure context (HTTP status codes, container restarts, scaling events).

2. What is Claude, and What is "Claude 3P" (Third-Party)?

Claude is Anthropic's family of large language models, accessed via the Messages API (POST /v1/messages). You send a list of messages plus parameters (model, max_tokens, etc.) and receive a response with content blocks and a usage object (input/output token counts).

"Claude 3P" = Claude via a Third-Party platform

Claude 3P ("third party") means consuming Claude not through Anthropic's own API (1P — first party), but through a cloud provider's platform that hosts/resells Claude. This distinction matters enormously for logging and debugging, because auth, model IDs, error formats, quotas, and feature availability all change with the platform:

Your scenario is the Bedrock column: LiteLLM (in Azure) → Claude 3P via Amazon Bedrock. So when you investigate failures, remember that many errors are AWS-layer errors (IAM, region quotas, model-access grants) that never touched the Claude model at all — the telemetry reference breaks these out layer by layer.

Why teams choose 3P over 1P

• Existing cloud contract/credits and procurement (billing goes through AWS/GCP/Azure)
• Data-residency and compliance boundaries (traffic stays inside the cloud provider)
• IAM-based auth instead of managing Anthropic API keys
• Co-location with the rest of the stack (VPC networking, private endpoints)

The trade-off: feature lag and platform-specific quirks — some Anthropic features arrive later (or never) on 3P platforms, model IDs differ, and each platform imposes its own quota model. Logging must record which platform and region served each request.

One retirement caveat still worth knowing

Model generations also age out: older Claude model snapshots get deprecated per-platform on their own schedules (Anthropic's for 1P, AWS's for Bedrock). A deployment pinned to an old snapshot will one day start returning ValidationException/404 for every request — a classic "everyone broke overnight" root cause, so log the exact model ID served (see section 9).

How a Claude API call works (the thing LiteLLM wraps)

POST https://api.anthropic.com/v1/messages
Headers: x-api-key, anthropic-version: 2023-06-01
Body: {
  "model": "claude-sonnet-5",
  "max_tokens": 1024,
  "messages": [{"role": "user", "content": "..."}]
}
Response: {
  "id": "msg_...",
  "content": [{"type": "text", "text": "..."}],
  "stop_reason": "end_turn",
  "usage": {"input_tokens": 12, "output_tokens": 84, ...}
}

Every response carries a usage block — this is the ground truth for token accounting, and LiteLLM propagates it into its logs. The stop_reason field (end_turn, max_tokens, tool_use, refusal, …) is also worth logging: it tells you why generation stopped, which matters for failure analysis in evals.

Claude can be reached via several providers, and LiteLLM abstracts all of them:

• Anthropic API (first-party, api.anthropic.com)
• AWS Bedrock (model IDs get an anthropic. prefix)
• Google Vertex AI
• Microsoft Foundry (the Azure-native way to consume Claude — relevant if your whole stack is Azure)

3. What is LiteLLM and How It Works

LiteLLM is an open-source LLM gateway (proxy server) that exposes a single OpenAI-compatible API (/chat/completions, /completions, /embeddings) and translates requests to 100+ backend providers — including Anthropic.

Why teams deploy it

The request lifecycle through LiteLLM

1. Client sends an OpenAI-format request to the proxy (/v1/chat/completions) with a virtual key.
2. LiteLLM authenticates the key, checks budgets/rate limits.
3. LiteLLM maps the requested model alias (e.g., claude-sonnet) to the configured backend model + real credentials (from its config.yaml).
4. Request is translated to Anthropic's Messages API format and forwarded.
5. Response comes back; LiteLLM computes cost (tokens × per-model pricing), records a spend log entry, fires any configured callbacks, and returns the OpenAI-format response to the client.

Typical config (config.yaml)

model_list:
  - model_name: claude-sonnet          # alias your clients use
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL   # Postgres → enables spend logs

litellm_settings:
  success_callback: ["langfuse"]          # or otel, s3, azure_storage, ...
  failure_callback: ["langfuse"]

How it's typically hosted in Azure

Plus, usually: Azure Database for PostgreSQL (LiteLLM's spend-log store) and Azure Key Vault (secrets).

4. What LiteLLM Itself Can Log

This is the richest source of evaluation data. LiteLLM has four distinct logging surfaces:

4.1 Spend logs (Postgres database)

When database_url is configured, every request writes a row to the LiteLLM_SpendLogs table:

Retrieval: direct SQL against Postgres, or LiteLLM's REST endpoints:

# Spend per key
curl -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  "https://<your-proxy>/spend/logs?api_key=sk-..."

# Global spend report grouped by team
curl -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  "https://<your-proxy>/global/spend/report?group_by=team"

Note: by default spend logs store metadata, not the message payloads. To store prompts/responses you must enable it (see 4.4) — with the privacy caveats in section 11.

4.2 Callbacks (success/failure hooks)

LiteLLM can push a structured log object to external systems on every request completion or failure. The callback payload includes the full request (messages), response, token usage, cost, latency, and any exception. Commonly used targets:

A minimal custom callback that writes eval-ready JSONL:

# custom_logger.py — referenced from config.yaml: callbacks: ["custom_logger.eval_logger"]
from litellm.integrations.custom_logger import CustomLogger
import json, datetime

class EvalLogger(CustomLogger):
    def log_success_event(self, kwargs, response_obj, start_time, end_time):
        record = {
            "ts": start_time.isoformat(),
            "model": kwargs.get("model"),
            "messages": kwargs.get("messages"),
            "response": response_obj.choices[0].message.content,
            "prompt_tokens": response_obj.usage.prompt_tokens,
            "completion_tokens": response_obj.usage.completion_tokens,
            "latency_s": (end_time - start_time).total_seconds(),
            "cost_usd": kwargs.get("response_cost"),
        }
        with open("logs/requests.jsonl", "a", encoding="utf-8") as f:
            f.write(json.dumps(record) + "\n")

eval_logger = EvalLogger()

4.3 Proxy server logs (stdout)

The LiteLLM process logs request handling, routing decisions, retries, and errors to stdout — with --detailed_debug it includes the raw request sent to Anthropic and the raw response. In Azure, stdout is captured by the container platform and lands in Log Analytics (section 5). Useful for debugging why a request failed (auth, rate limit, timeout, malformed request) rather than for per-request analytics.

4.4 Payload logging (prompts & responses)

Off by default. Options to capture actual message content:

• general_settings: store_prompts_in_spend_logs: true — puts messages/responses into the spend-log metadata.
• A callback target that stores payloads (Langfuse, blob storage, custom logger).

For evaluation work you almost always want payloads captured somewhere — otherwise you have metrics but no way to re-judge quality.

5. Azure Logging — The Landscape

Azure's observability stack has a few named pieces that are easy to confuse:

The one thing people miss

Platform logs are not collected until you create a Diagnostic Setting routing them to a Log Analytics workspace (or Storage Account / Event Hub). If you look in Log Analytics and find nothing, check:

Azure Portal → your resource (Container App / App Service / AKS) → Diagnostic settings → Add diagnostic setting → select log categories → Send to Log Analytics workspace

Which tables to query (by hosting choice)

6. What Gets Logged Where (End-to-End Trace)

Follow one request through the system and note every log artifact it produces:

Correlation strategy: pass a request ID end-to-end. LiteLLM accepts arbitrary metadata / extra_headers per request — put your trace/run ID there so you can join client logs ↔ spend logs ↔ Azure HTTP logs. With OpenTelemetry enabled on both the client and LiteLLM, W3C traceparent propagation does this automatically in App Insights.

7. How to Retrieve Log Data from Azure

Four practical extraction paths, from interactive to fully automated:

7.1 Azure Portal (interactive)

Portal → Log Analytics workspace → Logs → write KQL → Export → CSV (or "Export to Power BI"). Good for exploration; limited for bulk export (query result caps ~30k rows / 64 MB in the portal).

7.2 Azure CLI

az login
az extension add --name log-analytics   # first time only

# Query a workspace with KQL
az monitor log-analytics query \
  --workspace <WORKSPACE_GUID> \
  --analytics-query "ContainerAppConsoleLogs_CL | where TimeGenerated > ago(1d) | take 100" \
  --output json > logs.json

Timespan flag: --timespan P1D (ISO-8601 duration) instead of embedding ago() in the query if you prefer.

7.3 Python SDK (azure-monitor-query) — best for eval pipelines

from datetime import timedelta

from azure.identity import DefaultAzureCredential
from azure.monitor.query import LogsQueryClient, LogsQueryStatus
import pandas as pd

credential = DefaultAzureCredential()  # az login / managed identity / env vars
client = LogsQueryClient(credential)

WORKSPACE_ID = "<workspace-guid>"
KQL = """
AppServiceHTTPLogs
| where CsUriStem == "/v1/chat/completions"
| project TimeGenerated, ScStatus, TimeTaken, CsBytes, ScBytes
"""

response = client.query_workspace(
    workspace_id=WORKSPACE_ID,
    query=KQL,
    timespan=timedelta(days=7),
)

if response.status == LogsQueryStatus.SUCCESS:
    table = response.tables[0]
    df = pd.DataFrame(data=table.rows, columns=table.columns)
    df.to_parquet("results/http_logs.parquet")

Install: pip install azure-monitor-query azure-identity pandas

7.4 Continuous export (for large volumes / archival)

• Diagnostic Settings → Storage Account: cheap archival of raw JSON logs in blob storage (hourly folders). Read back with azure-storage-blob.
• Diagnostic Settings → Event Hub: real-time streaming to downstream consumers (e.g., a Python consumer writing JSONL for your eval harness).
• Log Analytics Data Export rules: export whole tables to Storage/Event Hub continuously.

Don't forget the non-Azure sources

For LLM-specific fields, extracting from LiteLLM's Postgres is usually simpler and richer than KQL:

import pandas as pd
import sqlalchemy

engine = sqlalchemy.create_engine(DATABASE_URL)  # Azure Database for PostgreSQL
df = pd.read_sql("""
    SELECT request_id, model, spend, prompt_tokens, completion_tokens,
           "startTime", "endTime",
           EXTRACT(EPOCH FROM ("endTime" - "startTime")) AS latency_s
    FROM "LiteLLM_SpendLogs"
    WHERE "startTime" > NOW() - INTERVAL '7 days'
""", engine)

8. KQL Query Cookbook for LLM Logs

KQL (Kusto Query Language) is Azure's log query language — pipe-based, like a SQL/pandas hybrid. Queries to keep handy:

Errors in the LiteLLM container (Container Apps)

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(24h)
| where Log_s has_any ("ERROR", "Traceback", "RateLimitError", "AuthenticationError")
| project TimeGenerated, Log_s
| order by TimeGenerated desc

Request volume + error rate over time (App Service HTTP logs)

AppServiceHTTPLogs
| where TimeGenerated > ago(7d)
| summarize
    total = count(),
    errors = countif(ScStatus >= 500),
    p50_ms = percentile(TimeTaken, 50),
    p95_ms = percentile(TimeTaken, 95)
  by bin(TimeGenerated, 1h)
| extend error_rate = round(100.0 * errors / total, 2)
| render timechart

Latency distribution for the completions endpoint

AppServiceHTTPLogs
| where CsUriStem == "/v1/chat/completions"
| summarize p50=percentile(TimeTaken,50), p90=percentile(TimeTaken,90),
            p99=percentile(TimeTaken,99), max=max(TimeTaken)
  by bin(TimeGenerated, 1d)

429s (rate limiting) — upstream vs proxy

AppServiceHTTPLogs
| where ScStatus == 429
| summarize count() by bin(TimeGenerated, 15m)
| render timechart

App Insights: LLM dependency calls (if OTel enabled)

dependencies
| where target has "anthropic" or name has "chat"
| summarize avg(duration), percentile(duration, 95), count() by name, resultCode
| order by count_ desc

Container restarts / OOM kills (a hidden cause of "missing" logs)

ContainerAppSystemLogs_CL
| where Log_s has_any ("restart", "Killing", "OOM", "probe failed")
| project TimeGenerated, Log_s

KQL survival kit: where (filter) → project (select columns) → summarize ... by bin(TimeGenerated, 1h) (group/aggregate) → render timechart (visualize) → take N (limit) → parse_json() (extract fields from JSON-in-string columns).

9. What Else You Can (and Should) Log

Beyond the defaults, from an evaluation-engineering perspective:

High value — add these if missing

Sometimes valuable

• Guardrail/moderation outcomes (if you add LiteLLM guardrails) — refusal and block rates over time.
• User feedback signals (thumbs up/down) — joinable to request IDs → labeled eval data for free.
• Prompt-template version — tag every request with the template hash; behavioral drift often traces to prompt edits, not model changes.
• Azure resource metrics during load tests — CPU/memory of the proxy under load; the proxy itself can be the latency bottleneck, not the model.

10. From Logs to Evaluations

The point of extracting all this: production logs are an evaluation goldmine. A workflow that matches the baseline → experiment → comparison → report pattern:

10.1 Build an eval dataset from logs

1. Extract a representative sample of real prompts (LiteLLM payload logs → JSONL).
2. Deduplicate & stratify — sample across intents, lengths, and time windows, not just the latest day.
3. Label — human labels, user-feedback joins, or LLM-as-judge on the logged responses.
4. Freeze the dataset (datasets/prod-sample-YYYY-MM-DD.jsonl) — it's now a regression suite.

10.2 Metrics you can compute directly from logs (no re-running needed)

These are your observational baselines. Quality metrics (accuracy, faithfulness, helpfulness) require judging the logged payloads — offline, with pytest + an eval framework.

10.3 Replay-based comparison (model or prompt changes)

logs → frozen dataset → replay through LiteLLM against:
    baseline:   current model (e.g., claude-sonnet-5)
    experiment: candidate (new model / new prompt / new params)
→ judge both outputs (LLM-as-judge or rules)
→ compare: quality delta, cost delta, latency delta
→ report (never metrics in isolation)

Because LiteLLM gives every variant the same OpenAI-format interface, replaying the same dataset against different model_name aliases is trivial — and tagging replays with metadata: {"eval_name": ..., "variant": ...} means the comparison data itself lands back in your logs, queryable by the same KQL/SQL.

Suggested tooling (Python): pytest for the harness, deepeval or promptfoo for judged metrics, pandas for log analysis, results stored as JSONL under results/runs/<date>/<model>/.

11. Privacy, Cost, and Retention Considerations

• Payload logging is opt-in for a reason. Prompts may contain PII/PHI. If you enable prompt storage, know your data classification, restrict access (Azure RBAC on the workspace/storage), and consider LiteLLM's redaction hooks or masking in a custom callback.
• Log Analytics ingestion costs money (~per-GB ingested, plus retention beyond the included period). Verbose LiteLLM debug logging on a high-traffic proxy adds up — use Basic Logs tier or blob export for bulk payloads, keep Analytics tier for operational queries.
• Retention: default Log Analytics retention is 30–90 days depending on config. Eval datasets should be exported and frozen — don't rely on the workspace as your dataset store.
• Secrets hygiene: never log API keys; LiteLLM hashes virtual keys in spend logs — keep it that way in custom callbacks.
• Anthropic-side retention is separate from your logging: your Azure logs are yours; the API provider's retention policy is governed by your agreement with them.

12. Quick Reference Card

Minimal setup checklist for a well-instrumented deployment

• Diagnostic Settings enabled on the proxy's host resource → Log Analytics
• LiteLLM database_url set (spend logs on)
• At least one payload-capturing callback (Langfuse / blob / custom) — with PII review
• OTel → Application Insights for distributed tracing (optional but recommended)
• Per-request metadata tags flowing from clients (run IDs, variants)
• Model IDs pinned & logged from the response, not the request alias
• Export/retention plan for eval datasets (don't rely on workspace retention)

Next steps you might want: (a) a KQL workbook/dashboard spec for this stack, (b) a Python extraction script wired to your actual workspace, or (c) an eval-methodology doc (baseline/experiment design, judge prompts, metrics definitions) built on top of this logging foundation.