# Litefuse Agent Integration Trace Spec

Version: v1.2 (2026-06-10 — added subagent subtrees and the propagation protocol; naming finalized as `tool:` / `generation` / all-lowercase; metadata flattened to the `agent_` prefix; env reads prefer LITEFUSE_*)
Scope: every integration that wires a terminal / IDE / gateway-style agent into Litefuse observability (Hermes Agent, Claude Code, pi, and later Cursor, OpenClaw, etc.).
Design basis: industry principles for "what a good trace looks like" (trace boundaries, naming, I/O, cost fields), the practice of the Hermes Agent and Claude Code integrations, and the endpoint behavior verified while building the pi integration.

Normative keywords: **MUST** / **SHOULD** / **MAY**.

---

## 1. Trace boundaries

1.1 **One user turn = one trace** (MUST). A turn is: from the user sending one message, until the agent finishes all LLM calls and tool executions and produces a final reply. This is the canonical mapping of Litefuse's data-model notion of "one self-contained unit of work."

1.2 **A multi-turn conversation = one session** (MUST). Set `session_id` on every trace; Litefuse's session replay view stitches the whole conversation together in time order.

1.3 **The trace name is `<Agent name> — Turn N`** (MUST). E.g. `Pi Agent — Turn 7`, `Hermes Agent — Turn 3`.

- N increases **monotonically** per session; on resume/continuation it MUST keep counting (implementation hint: count the existing user messages in the session history).
- **Don't** use a snippet of the user prompt as the trace name: the name must be stable, filterable, and aggregatable; the prompt belongs in the trace input.
- Once set, the trace name **must not be renamed mid-flight**.

## 2. Tree structure: a single root parent, flat mounting

2.1 **Every trace has exactly one root span**, with observation type `agent`, the same name as the trace, and a duration equal to the turn's real wall-clock duration (MUST).

2.2 **Observations within the same agent run mount flat under their container span** (MUST). Even when a turn exceeds 50 steps the timeline view stays clean, and Litefuse's agent graph infers the execution flow from the time order automatically. The only thing that introduces depth is the subagent subtree (§2.5) — a real execution hierarchy, not grouping decoration.

2.3 **Two structures explicitly rejected** (recorded for background):

- *Mounting a tool under the generation that requested it*: violates the OTel convention that "a parent span's time interval contains its children" — the generation already ended when the model returned the toolCall, and tool execution comes after it; forcing nesting either makes the child span poke outside the parent interval, or artificially stretches the generation and dirties the LLM-latency stats. Semantically the model only *requests* the call; the executor is the agent runtime. The industry (OTel GenAI semconv, OpenAI Agents SDK, Pydantic AI) doesn't do this either.
- *Adding a step/cycle grouping span per step* (AWS Strands style): the grouping is redundant with the time order (a tool always immediately follows its plan), and it introduces a container observation with no input/output of its own (violating §4.4), deepening the tree and adding noise.

2.4 **The plan↔tool association is expressed in metadata, not in the tree** (MUST): the tool observation's metadata carries `agent_plan_step` (pointing at the sequence number of the LLM call that requested it) and `agent_tool_call_id`; the toolCall block in the plan's output carries the same id, so the association is queryable in both directions (see §6.3).

2.5 **Subagents form a subtree** (MUST, when the agent supports subagents / agent teams). The structure has three layers:

```
Pi Agent — Turn 1                        AGENT (parent, owns the trace header)
├── plan (1 tool) #1                     GENERATION
├── tool (1 subagent) #2                 TOOL (the tool execution, from the parent's view)
│   └── subagent                         AGENT (the container span for the subagent run)
│       ├── plan (1 tool) #1             ← container-local numbering (§3.4)
│       ├── tool: ls (demo-src) #2
│       └── subagent response
└── response
```

**The tool span wrapping the container is deliberate and MUST be kept**, for three reasons:

- It records the **real cost of delegating**: tool-span duration − container duration = the overhead of child-process startup, runtime loading, and result parsing (measured: of a 16.0s delegation the container was only 5.5s); optimizing "why is the subagent slow" looks exactly at this difference.
- In parallel / chain modes a single tool call spawns multiple child processes, and the **tool span is the only batch container** — N child containers mount under it, expressing "this batch belongs to one decision."
- It preserves the §3.1 invariant "each tool execution = one TOOL observation," which is what makes filtering/statistics by `tool_name=subagent` (delegation frequency, delegation-duration distribution) valid.

The delegation tool span is named `tool (n subagents) #N` (n = the number of subagents spawned by this call: single=1, parallel=number of tasks, chain=chain length; the parenthetical semantics match `plan (n tools)`).

Container span: type `agent`, named `subagent` (**with no number**: child processes can't obtain distinct sequence numbers with zero coordination, and rather than leave the ambiguity of parallel-mode children sharing the same number it's better not to number them — the container's identity is expressed by its tree position, and the agent name and task text live in the tool span's input and the container's own input). input = the task text, output = the subagent's final answer, metadata includes `agent_subagent: true` plus the stats of that run (`agent_api_calls` / `agent_tool_calls` / `agent_steps` / `agent_duration_ms`). The subagent's generations carry their own usage, and **cost rolls up into the parent trace's total automatically**. Nesting (a subagent delegating to a grand-subagent) recurses by the same rules.

2.6 **Subagent context-propagation protocol** (MUST, for process-isolated subagents). When a subagent-class tool starts executing, the parent process's collector writes a W3C-format traceparent into the process environment:

```
LITEFUSE_TRACEPARENT=00-<traceId>-<spanId of the subagent tool span>-01
```

The child process (which inherits the parent environment) detects this variable on startup and enters **child mode**: it joins the given traceId, mounts its container under the tool span as parent, **emits no trace-level headers** (the trace belongs to the parent process), and does not emit its own root trace. When the tool execution finishes, the parent restores the env var to its own inherited value (supporting recursive nesting). Known limitation: when multiple subagent tool calls execute concurrently within the same turn, the env var has a race — pi executes tool calls serially so it isn't triggered; other runtimes must confirm.

## 3. Observation types and naming

3.1 Type mapping (MUST):

| Step | observation type | Naming |
|---|---|---|
| Turn container | `agent` (root) | `<Agent> — Turn N` |
| Subagent run (§2.5) | `agent` (container, mounted under the delegation tool span) | `subagent` |
| Each LLM API call | `generation` | named by behavior, see 3.2 |
| Each tool execution | `tool` | `tool: <name> (<key info>) #N` |
| API error / retry / rate-limit & other system events | `event`, level=ERROR | by event content |

3.2 **Name a generation by "what the model did," not by which model** (MUST; the model name is the generation's `model` attribute — putting it in the name breaks every filter and dashboard when you swap models):

| Message content | Name |
|---|---|
| Contains toolCall(s) (with or without thinking/text) | `plan (n tools) #N`, n = number of tool calls emitted this time, singular written `(1 tool)` |
| Has body text, no toolCall (i.e. the final answer that ends the agent loop) | `response`; inside a subagent it's `subagent response` (distinguished from the parent turn's final answer so name-based evaluators don't confuse them) |
| Only thinking | `think #N` |
| Neither | `generation #N` |

- The wording in parentheses is **nominal key info** (`(2 tools)`, not `(call 2 tools)` / `(to call …)` / `(for …)`), consistent with the parenthetical semantics of tool names (§3.3).
- **Don't expand the name by combining accompanying content**: a tool-calling step that carries thinking / transitional text is still called `plan`, not `think & plan & response`. The name describes the step's **primary action**; combinatorial naming would explode the name space into 2^n variants and break name-based aggregation (and reasoning models carry thinking on nearly every step, so the prefix has no discriminating power); mixing the word "response" into a plan's name would also dilute the semantics of the closing `response` observation. Accompanying info goes into metadata (`agent_thinking_chars`, etc.); filter by metadata.
- **Name once, finalized**: create/name the observation only after the message is fully received (so it's known whether it contains a toolCall); **renaming after an initial send is forbidden** (the early pi integration's Chat→Planning→Response rename chain is the anti-pattern).
- `response` **is the last LLM call itself**, carrying its real usage and latency. **Don't** emit an extra closing generation that restates the final answer and has no usage (redundant). `response` carries no `#N` — at most one per trace, with a stable name so an LLM-as-a-judge can locate it by name.

3.3 **Tool naming: `tool: <name> (<key info>) #N`** (MUST). Key-info principle: **useful but short**, uniformly truncated to 24 chars; if nothing can be extracted, omit the parentheses (`tool: <name> #N`). The full arguments always live in the observation input; the name is only for navigation.

| Tool | Key info | Example |
|---|---|---|
| bash / shell | the executable command name (skip `FOO=bar` env assignments, `-flag`s, quoted strings, paths, and shell prefixes like `cd`/`export`/`sudo`/`env`/`nohup`) | `tool: bash (grep) #1` |
| read / write / edit | the file **basename** (the full path is too long) | `tool: read (index.ts) #2` |
| grep / find | the pattern | `tool: grep (TODO) #3` |
| ls | the directory basename | `tool: ls (src) #4` |
| subagent / delegation-class | named `tool (n subagents) #N`, n = number of subagents; the agent name and task live in the input and the subtree | `tool (2 subagents) #3` |
| Custom tools | the most identifying short field in the args (path basename, name, etc.) | — |

3.4 **The `#N` sequence number: a single global step counter unique within each agent container** (MUST). **The numbering space is scoped to the owning agent container**: the turn root is one container, and each subagent container (§2.5) restarts from #1; cross-container hierarchy is expressed by the tree, not encoded into the number. Each container maintains **one** time-increasing step counter, and every LLM call and every tool execution takes the next number — generations and tools **share the same numbering space**, and the `#N` in the name is the observation's `step_index`. `response` carries no `#N` in its name (at most one per container, to keep the name stable), but it still has its own `step_index` in metadata.

> Why not a dotted global path like `#2.1.4`: in parallel/chain modes a single tool call concurrently spawns multiple child processes that inherit the same environment prefix, and they can't obtain distinct prefixes without introducing filesystem-level coordination (atomic number grabbing); also dotted numbers are verbose under deep nesting, and a grab order inconsistent with the task order is actively misleading. "Per-container local counting + tree positioning" is the common practice of all mainstream frameworks for displaying nested agents.

Purpose: unique names, a linear graph view, no ambiguity among same-named tools, and **the number in the name maps directly to metadata** — no parallel numbering schemes. Example:

```
plan (3 tools) #1           agent_step_index=1
tool: read (note.txt) #2    agent_step_index=2, agent_plan_step=1
tool: bash (wc) #3          agent_step_index=3, agent_plan_step=1
tool: bash (ls) #4          agent_step_index=4, agent_plan_step=1
response                    agent_step_index=5
```

3.5 **Naming style: all lowercase** (MUST). Step-level observation names (`plan` / `tool:` / `response` / `subagent` / `context compaction`) are all lowercase — they are essentially **filterable identifiers**, not UI titles (the industry convention is the lowercase `classify-intent` style; evaluator and dashboard name filters are case-sensitive, and all-lowercase is the least error-prone). Exception: the trace name / root container name (`Pi Agent — Turn N`) is a title and proper noun, so it stays capitalized.

## 4. Input / Output

4.1 Trace level (MUST): input = the user prompt (text; when it contains images, attach a block summary into metadata); output = the final answer text.

4.2 Generation: input = the full messages sent to the provider (including the system prompt and history); output = the assistant message content, **preserving the thinking / text / toolCall block structure** (thinking is an important signal for debugging and evaluation — don't drop it if you can get it; the Hermes hook not getting reasoning is a known limitation, not a design choice).

4.3 Tool: input = the tool argument object; output = the tool result text. Structured execution details (diffs, exit codes, etc.) go into metadata (§6.3) as a flat summary per tool type, to avoid bloating the output.

4.4 **An observation with neither input nor output should not exist** (a basic "good trace" principle, MUST follow). This is the basis for v1 removing the "user message EVENT" (§7.2).

4.5 **Truncation** (MUST): when input/output exceeds the threshold (default 1,000,000 chars, configurable via env var), truncate and record `*_truncated: true` and `*_orig_len` in metadata. Fields absent from the source data **do not appear at all** in metadata; no null placeholders (sparse storage).

## 5. Usage and cost

5.1 **usage_details uses Anthropic-style keys** (MUST): `input`, `output`, `cache_read_input_tokens`, `cache_creation_input_tokens`. Litefuse aggregates the token dashboard and computes cost from these automatically.

5.2 **The model name is set on the generation's `model` attribute** (MUST) — this is the lookup key for cost mapping.

5.3 **No double counting** (MUST): if upstream produces multiple records for the same API message (e.g. multiple lines with the same usage in a Claude Code transcript), attach usage_details only to the last one.

5.4 When the agent runtime carries its own cost data, **SHOULD** emit `cost_details` (the same key structure + `total`) to override Litefuse's automatic pricing; otherwise leave it empty and let the server compute by model name.

## 6. Identifiers, tags, and metadata

6.1 Trace-level identifiers (MUST):

| Field | Value |
|---|---|
| `session_id` | the agent's session id (e.g. pi's session UUID, Hermes's `YYYYMMDD_HHMMSS_<hex>`) |
| `user_id` | `$LITEFUSE_USER_ID` → OS username → a fixed fallback |
| `tags` | `<agent identifier>` (e.g. `pi-agent`) + `model:<name>`; tags are immutable — set them at creation; for after-the-fact judgments use a score, not a tag |
| `environment` | `LITEFUSE_TRACING_ENVIRONMENT` (fallback `LANGFUSE_TRACING_ENVIRONMENT`); configured per target when there are multiple (default production; use test/development for test targets to avoid polluting production dashboards) |
| `release` | (optional) the agent version |

6.2 **metadata uses the uniform `agent_` prefix, flat not nested** (MUST): all agent-specific fields go at the **top level** of metadata with the `agent_` prefix (`agent_step_index`, `agent_plan_step`, `agent_duration_ms`…), **not** namespaced by agent name (`pi.*`/`hermes_agent.*` are the old convention, see the appendix). Rationale: one shared set of keys lets cross-agent filters and dashboards reuse a single query (namespacing by agent name would make a condition like `metadata.pi.duration_ms` fail for other agents) — the agent identity is expressed by the trace name and tags; flat scalar values also naturally avoid the serialization risk described below.

6.3 Minimum metadata set per level (SHOULD):

- Trace root: `agent_turn_number`, `agent_session_id`, `agent_cwd`, `agent_model`, `agent_provider`; at turn end add `agent_api_calls`, `agent_tool_calls`, `agent_steps`, `agent_message_count`, `agent_duration_ms`.
- Generation: `agent_turn_number`, `agent_step_index` (the §3.4 sequence number), `agent_provider`, `agent_api`, `agent_stop_reason`, `agent_api_duration_ms`, `agent_tool_call_count`, `agent_thinking_chars`, truncation flags.
- Tool: `agent_tool_name`, `agent_tool_call_id`, `agent_step_index` (the §3.4 sequence number), **`agent_plan_step`** (the step_index of the plan generation that requested this tool, §2.4), `agent_duration_ms`, `agent_is_error`, a structured `agent_details` summary, truncation flags.

All sequence numbers belong to the §3.4 numbering space (scoped per container); the join condition is `tool.agent_plan_step == generation.agent_step_index` (within the same container).

**Type discipline for metadata values** (MUST): put objects or scalars, **never a pre-serialized JSON string** — when the server flattens metadata into a string map it corrupts "string values whose content is JSON" (measured: inner quotes lose their escaping and the whole namespace degrades into an unparseable string). For oversized structured values (e.g. a subagent tool's details embedding the full child message history), **omit them wholesale** and record `agent_details_omitted_len`; don't truncate and stuff them in — truncated JSON is equally unparseable, and the subagent subtree already preserves that information fully.

## 7. Timestamps and send ordering

7.1 **Real wall-clock timestamps** (MUST): each observation's start/end comes from the moment the event actually happened (the hook callback firing, or a timestamp carried in the source data); don't synthesize an evenly-spaced grid. The timeline must reflect real LLM latency, tool durations, and the gaps between calls. Only in offline-parsing scenarios where the source data has just a single timestamp (e.g. a Claude Code transcript), leave 1ms between adjacent same-level spans to keep the graph view from misjudging parallelism.

7.2 **Each span is sent exactly once, when it ends** (MUST). OTel spans are immutable — this spec does not use the "provisional span + resend with the same spanId to upsert" pattern: it breaks the span-immutability contract, relies on merge semantics the protocol doesn't guarantee on the server, and more than doubles the write count. The cost is that **an in-progress observation is not visible on the web** (a streaming LLM call, a long-running tool); this is an accepted trade-off.

7.3 **Trace headers ride on every span** (MUST): `langfuse.trace.*` (name/input/tags) + `session.id` + `user.id` attributes are attached to **every span emitted within the turn** (same values, idempotent). Since the root span isn't sent until turn end, the trace's first appearance in Litefuse is triggered by the **first completed observation** (typically plan #1, which completes within seconds), and the header comes with it — no need to wait for turn end, and no non-standard mechanism. This matches the SDK's attribute propagation (propagateAttributes) behavior.

> Alternative for SDK-limited scenarios (for reference): some SDKs can only export already-ended observations; if you want the header to appear before the first LLM call completes, you can emit a transient `user message` EVENT that carries the trace header and force a flush (the Hermes/Claude Code v1 approach). Note it conflicts with §4.4 (has input, no output) — it's a workaround, not a normative structure.

At turn end, flush all in-flight requests (with a bounded timeout).

**Transient orphan spans are a known and accepted phenomenon**: a subagent's steps are sent as they complete, while their container waits for the child process to end and the subagent tool span waits for the parent process to collect the result — so the live view will transiently show child observations ("the parent hasn't arrived yet, numbering restarting from #1") sitting at the bottom of the list. This is inherent to the single-send model; **after everything finishes the tree view is fully correct**. The only way to eliminate it is to have the child process buffer all spans until the end, but that would give up mid-run visibility for long subagents, which isn't worth it. (Product-side input: Litefuse's live view can give "spans whose parent hasn't arrived" a "running" visual treatment — something the server can solve gracefully and the collector can't.)

7.4 **Collection-surface recommendations** (event-based integrations should cover as much as possible; offline-parsing ones trade off by source-data availability): beyond the core request/response/tool events, **SHOULD** collect — the first-token moment → the generation's `completion_start_time` (Litefuse's TTFT metric); request sampling parameters (temperature/max_tokens/top_p) → `model_parameters`; the HTTP response status and correlated headers (request-id, retry-after) → generation metadata; the thinking tier → metadata; the context watermark at turn end (tokens/window/percent) → trace metadata `context_usage`; context compaction that occurs within the turn → an `event` observation (explaining the sudden input-token drop on the next call).

## 8. Errors and levels

| Situation | Handling |
|---|---|
| Tool execution fails | tool observation `level=ERROR` + `status_message` (a preview of the first 500 chars of the result) |
| LLM call error (stop_reason=error) | generation `level=ERROR` + status_message |
| API error / retry / rate-limit (system event) | an `event` observation, `level=ERROR`, fields preserved as-is |
| Turn ends with no final text answer (aborted mid tool-loop, etc.) | root span `level=WARNING` + status_message |
| Turn interrupted (exit/reload) | close in-flight tool spans with WARNING ("turn ended before tool completed"), close the root with WARNING, and the shutdown hook flushes defensively |

## 9. Reliability and configuration

9.1 **Fail-open** (MUST): any exception in the integration only writes a local log (e.g. `~/.pi/agent/litefuse.log`, `~/.hermes/state/litefuse_plugin.log`) and returns silently, **never blocking the agent's main loop**. Network sends are fire-and-forget + timeout; an unreachable target is simply skipped.

9.2 **Credential and config conventions** (MUST): the primary target reads `LITEFUSE_PUBLIC_KEY` / `LITEFUSE_SECRET_KEY` / `LITEFUSE_BASE_URL` (alias `LITEFUSE_HOST`) / `LITEFUSE_TRACING_ENVIRONMENT`; **`LITEFUSE_*` takes precedence, and the matching `LANGFUSE_*` is the ecosystem-compatible fallback** (to reuse the user's existing env config). Shared options: `LITEFUSE_USER_ID`, `<AGENT>_LITEFUSE_DEBUG`, `<AGENT>_LITEFUSE_MAX_CHARS`.

9.3 **Multiple targets** (optional): support dual-writing the same trace to multiple Litefuse instances (e.g. local self-hosted + cloud), each with its own environment; the target list comes from a config file or the `LITEFUSE_EXTRA_TARGETS` JSON.

## 10. Transport reference: direct OTLP (verified)

A zero-SDK-dependency path, suitable for integrations that can get in-process events (pi v2 is this implementation):

- Endpoint: `POST <host>/api/public/otel/v1/traces`, OTLP/HTTP **JSON**, `Authorization: Basic base64(pk:sk)`.
- Attribute mapping: `langfuse.observation.type|input|output|model.name|usage_details|cost_details|level|status_message|metadata` (usage/cost/metadata are JSON strings); trace level `langfuse.trace.name|input|output|tags|metadata`, `session.id`, `user.id`, `langfuse.environment`.
- **Measured server behavior** (2026-06, litefuse.cloud):
  1. `langfuse.trace.*` attributes placed on a **non-root child span** take effect just the same — §7.3's "headers ride on every span" relies on this behavior;
  2. resending the same spanId is merged by the server as the latest version (upsert). It works in practice, but **this spec does not adopt it**: it violates the OTel span-immutability contract, relies on server semantics the protocol doesn't guarantee, and doubles the writes (see §7.2). Recorded here only for regression reference.

## 11. Verification checklist

Before releasing an integration, verify each item with `litefuse-cli api traces get <id>`:

- [ ] trace name `<Agent> — Turn N`; sessionId / userId / tags / environment correct;
- [ ] **after the first observation completes** (without waiting for turn end), the trace is queryable and the header (name/session/input/tags) is complete;
- [ ] each span is sent only once (no duplicate spanId writes);
- [ ] tree: 1 AGENT root, everything else flattened as its children;
- [ ] one GENERATION per LLM call: `plan (n tools) #N` / `response` named correctly, with the model attribute, usage_details (Anthropic keys), and real latency in place, and Litefuse computed a non-zero cost;
- [ ] one TOOL per tool: `tool: <name> (<info>) #N`, input=args, output=result, metadata carries `agent_plan_step` + `agent_tool_call_id`;
- [ ] when one plan calls multiple tools in parallel: `plan (2 tools)` and the two tools' `agent_plan_step` agree;
- [ ] tool failure → ERROR; a turn with no text ending → root WARNING;
- [ ] (when subagents are supported) subagents form the tool span → container → child-steps three-layer subtree, container numbering restarts from #1, and the subagent's usage rolls into the parent trace's total cost; nested delegation recurses;
- [ ] trace input/output = the user prompt / the final answer;
- [ ] on a resumed conversation the Turn numbers are continuous and grouped under the same session;
- [ ] when the network is down / the target is unreachable, the agent is completely unaffected and errors go to the local log.

## Appendix: differences from existing integrations (suggested as v2 sync items)

This spec updates the old conventions in the Hermes / Claude Code integration docs in the following places:

1. **Generation naming**: old `api: <model> #N` (model name in the name) → new `plan (n tools) #N` / `response` / `think #N` (named by behavior, model name as an attribute only). Consistent with the §3.2 naming principle.
2. **The `user message` EVENT**: the old convention started every turn with it → the new convention omits it (§4.4, §7.2). It may still be kept as a workaround for early header flushing when the SDK is limited, but its nature should be documented.
3. **Naming case**: the Claude Code integration currently uses capitalization (`Thinking (#1)`, `Tool call: Bash`), inconsistent with the §3.5 all-lowercase convention; v2 should unify to lowercase (Hermes is already lowercase).
4. **metadata namespace**: old nested namespaces `metadata.hermes_agent.*` / `metadata.claude_code.*` → the new unified flat `agent_` prefix (§6.2), so cross-agent dashboards can reuse one query.

The pi integration (`~/.pi/agent/extensions/litefuse/index.ts`) is the reference implementation of v1 of this spec.