Parse Pipeline

The parse pipeline turns on-disk session files into normalized usage events. It is where the tool’s performance work lives, and every mechanism on this page exists to keep a multi-GiB corpus parseable in seconds. A report run flows through it in order:

Each selected adapter discovers its files.
Discovered files split into event-store hits (served from the ledger) and misses (see Event Store).
Misses are parsed under a bounded parallelism budget, on worker threads when a source’s misses are large enough to justify them.
Parsed lines normalize into usage events (createUsageEvent, in the core package’s domain/).
Freshly parsed files are written back to the ledger in batches; for pi, an event repeated across the source’s files is counted once. Events then continue to history and machine merge, filtering, pricing, and aggregation (see the overview).

Discovery and adapters

Each source implements the adapter contract (packages/core/src/sources/source-adapter.ts): discoverFiles() returns the file list, and read(filePath, { diagnostics }) yields that file’s events as an async iterable, adding skipped rows to the diagnostics sink as it goes. The CLI reads each file whole through collectFile (packages/core/src/stream/collect.ts), which returns the events with sorted skipped-row counts, or rejects with no events when the read fails partway, so a failed file keeps its stored events. Adapters that read sidecar inputs also declare them via getParseDependencies(filePath) so the event store can fingerprint them. Parsing stays inside the adapter; everything downstream sees only normalized usage events. Claude, Codex and pi also implement readEvents, which yields messages, tool calls and session facts next to usage (the @ayagmar/agent-sessions README documents the contract). With the event store on, the CLI reads their files through readEvents for usage, tool calls and messages, so each file is read once and its tool calls and messages are stored with its usage; the usage events and skipped rows are the ones read yields. With the store off, it reads them through collectFile.

An adapter whose files can repeat each other’s events sets capabilities.eventsRepeatAcrossFiles (pi, whose forks copy their parent’s entries). The pipeline then counts an event found in several of the source’s files once, from the first file in discovery order, matching events by content hash; repeats within one file are kept. Only files that share a session id with another file are hashed. Each file’s stored events stay whole, so the parse cache still depends on that file alone.

An adapter whose sessions copy each other’s events under a new session id sets capabilities.sessionsCopyEvents (Claude, whose continued sessions copy the rows they continue). The dataset then counts an event that several sessions hold with the same timestamp, model, and usage once, in the session with the earliest first event, the lower session id on a tie (dedupeCopiedSessions). The pass runs after history and other machines are merged, so a departed or remote copy is counted once too. Only events whose timestamp appears in more than one session are compared. machine export adds the copies back when it checks that the store holds every counted event, since the store keeps each copy.

Sources with multiple home directories (claude, pi, openclaw, kimi, the cline family, copilot, dsh) discover through the shared multi-root helper (packages/core/src/sources/multi-root-discovery.ts): resolveRootDirs applies overrides over defaults, discoverFilesAcrossRoots walks each root with per-root validation, and sortAcrossRoots optionally re-sorts the combined list (kimi sets it). Paths can be overridden per source via the sourceDirs.* config keys, the --<source>-dir flags, or --source-dir <source>=<path> (directory-backed sources only), with the usual flags → env → config precedence.

Reading JSONL fast

packages/core/src/utils/read-jsonl-objects.ts streams files in 1 MiB buffer chunks and splits lines at the byte level: line-feed scan, carriage-return trim, BOM and blank-line handling all happen on bytes, so a line is only decoded to a string when it survives filtering. The reader yields records in batches, one per chunk (readJsonlObjectBatches), so an adapter awaits once per chunk rather than once per record; a batch parses its lines only as the adapter reaches them. Two optional predicates gate the expensive steps:

shouldParseLineBytes runs on the raw line bytes, before UTF-8 decoding. The codex adapter accepts session_meta, turn_context, and token_usage_record lines and takes event_msg lines only when token_count bytes appear; the claude adapter requires both "assistant" and "usage" byte sequences. A readEvents call that asks for more than usage swaps in a wider prefilter per adapter; read keeps these.
shouldParseLine runs on the decoded string, before JSON.parse. The pi, droid, openclaw, copilot, qwen, kimi, and dsh adapters filter here.

In a 1.8 GiB codex corpus, most lines carry no usage data. Rejecting them before JSON.parse, and for the byte predicates before decoding, is the biggest cold-run win; the Benchmarks page shows the measured effect.

Parallelism

Two mechanisms bound and exploit parallelism:

parseMaxParallel (config key, default 8, clamped to 1 to 64) is a global semaphore over concurrent file parses. Adapters share one budget per run (createParseBudget in src/cli/parse/parse-budget.ts), so parsing many sources at once cannot multiply the concurrency.
The worker-thread parse pool (src/cli/parse-worker-pool.ts) moves large parse work off the main thread. Only sources registered for worker parsing are eligible: claude, codex and pi. Each task carries the resolved roots of the adapter that owns the file, so a worker finds a fork’s parent exactly as an inline parse does. For an eligible source the pipeline first resolves event-store hits and collects the misses with their byte sizes; the pool engages only when the total missed bytes reach parseWorkerMinBytes (default 268435456, ≈256 MiB). A claude or pi miss that resumes from a stored cursor (qwen resumes too, and always parses inline) (see Event store) is parsed inline instead, since it reads only the appended lines. It then sends the other missed files to the run’s one pool, which every eligible source shares and which spawns a thread only when a task finds no idle one, up to parseWorkers threads in total (config "auto", the default, which resolves to min(6, cores − 1) when the run writes the event store and min(8, cores − 1) when it does not; or 0 to 64, where 0 disables the pool). The pool shuts down when the run’s parsing ends. The runtime profile records each eligible source’s pool status as off (below the threshold), engaged, or fallback. If workers cannot run, every task carries an inline-parse fallback, so the run completes on the main thread and the status is fallback. Because the pool is shared, a failure during one source’s parse also sends later sources inline, and they show fallback too. The profile’s per-source parseWorkerLimit is the most threads that source could use, not how many the shared pool spawned. LLM_USAGE_PARSE_WORKERS and LLM_USAGE_PARSE_WORKER_MIN_BYTES override both knobs per run.

In practice a warm run has few misses and the pool stays off; a cold run over a large claude, codex or pi corpus is what engages it.

Period bucketing

Bucketing hundreds of thousands of events into local days would be slow if each event paid for Intl date math. src/utils/time-buckets.ts keeps a per-timezone cache of local-day windows: each window stores a [startMs, endMs) range with its precomputed local date parts, keyed by day index (derived from the timestamp plus its wall-clock offset). The hot path checks the most recently used window first, so consecutive events from the same local day resolve with two comparisons and no formatter call; Intl.DateTimeFormat instances are themselves cached per timezone for the misses.

Failure containment and diagnostics

A file that fails to parse does not kill the run: the failure is recorded as a file_parse_failed skipped-row reason and parsing continues. Only when every file of a source fails does the source itself error (All N file(s) failed to parse for source <id>: <reason>), and even then the report continues without that source unless it was explicitly selected, in which case the run aborts with Failed to parse explicitly requested source(s): <source>: <reason>.

A failed file keeps the events the event store last read from it: its stored entry is not replaced, so history still has them if the file later disappears, and the next run retries it. While it fails, the file counts as present but not counted: its stored events neither count in the report nor suppress a departed copy of them, so a moved file whose new copy breaks still reports its usage through history. Transcripts stored as one JSON document (Gemini, Amp, Droid, Cline) and DSH logs fail the whole file when it cannot be read: invalid or truncated JSON (often a file caught mid-rewrite), a document of the wrong shape (such as null instead of a session object), a file over 64 MiB, or a log with no decodable frame. Malformed lines in a JSONL transcript are only skipped rows.

Adapters count skipped rows with per-reason counters (packages/core/src/sources/parse-diagnostics.ts), and the report shows them (files found, skipped rows and reasons, source failures) as diagnostics on stderr, never mixed into the report body on stdout. Routine skips (rows with no usage by design) only show with --verbose; rows that could not be read are a warning. The logLevel config key, --quiet, and --verbose gate the informational lines.

Event Store: the hit/miss split, fingerprints, and ingest.
Configuration: the parseMaxParallel, parseWorkers, and parseWorkerMinBytes keys.
Benchmarks: the measured effect of this pipeline on real data.