LLM Observability: Tracing, Logging and Monitoring AI Apps

Why LLM apps need special observability

Classic monitoring answers "is the service up and fast?". For an AI app that is not enough: the service can return HTTP 200 quickly and still give a wrong, made-up or unsafe answer. Outputs are non-deterministic, and one user action may trigger several model calls, searches and tool invocations.

LLM observability means you can answer three questions for any request:

• What exactly did the model receive and return?
• Which steps, tools and documents were involved?
• How much did it cost, how long did it take, and was the answer good?

What to log for every request

A useful minimum per LLM call:

Tracing multi-step agents

An agent may plan, search, call an API, call the model again and only then answer. Logging each call separately makes this impossible to follow. Use traces and spans:

• a trace is one user request end to end;
• a span is one step inside it: an LLM call, a vector search, a tool call;
• spans are nested, so you see the tree of what happened and where time and tokens went.

OpenTelemetry is the common standard for this, and it has emerging semantic conventions for generative AI. Specialized tools (open-source and SaaS) add LLM-focused views on top: prompt playgrounds, cost dashboards, evaluation runs.

from opentelemetry import trace

tracer = trace.get_tracer("support-bot")

with tracer.start_as_current_span("answer_question") as span:
    span.set_attribute("prompt.version", "support-v7")
    docs = search_docs(question)          # child span inside
    span.set_attribute("rag.docs_count", len(docs))
    reply = call_llm(question, docs)      # child span inside
    span.set_attribute("llm.tokens.total", reply.usage.total_tokens)

Watch agents for typical failure patterns: loops (the same tool called again and again), too many steps, tool errors the model ignores, and runaway token use.

Monitoring quality and drift

Answer quality can change even when your code does not: the provider updates the model, user questions shift, the knowledge base gets stale. Track it:

• Offline evals — a fixed test set run on every prompt or model change.
• Online checks — automated scoring of a sample of real traffic (LLM-as-judge, format validation, groundedness against retrieved documents).
• Proxy signals — share of negative feedback, escalations to humans, repeated questions, empty or refused answers.
• Input drift — new topics or languages appearing in requests that your prompts were not designed for.

Set alerts on sudden changes, not only on absolute thresholds.

Privacy-safe logging

Prompts often contain names, phones, addresses, documents. Logging them as is creates a data leak waiting to happen.

• Mask or redact personal data (PII) before writing logs.
• Store full texts only where necessary, with a short retention period.
• Restrict access to raw traces by role and audit who reads them.
• Keep metrics (tokens, latency, cost) separately from content so dashboards do not expose texts.
• Check that your logging vendor's data storage location and terms fit local law and your contracts.

Common mistakes

• Logging only errors, so good-looking but wrong answers stay invisible.
• No prompt versioning — impossible to tell which change broke quality.
• Tracking cost per month but not per feature or per user.
• Sending raw user data to a third-party logging service without review.