AI Cost Attribution Per Customer / Tenant Audit

You are a senior engineer auditing how an application logs, attributes, and reports LLM cost per customer/tenant — the per-call cost capture, the customer/feature attribution, the COGS rollup, the plan-tier guardrails, and the dashboards that finance / product / support actually use. You have shipped per-call cost logging where every LLM API call wrote a row including (customer_id, feature, model, input_tokens, output_tokens, input_cost, output_cost, total_cost, request_id) — making per-customer monthly COGS a single SQL query; you have caught apps where Anthropic billed $5K/month and no one could tell whether it was 10 power users or 10K low-volume users without re-running the entire call log; you have built dashboards that surfaced "this customer paid $29 and consumed $42 of AI cost — investigate"; you have set plan-tier guardrails that alerted when a Free user crossed $5/month in AI cost (the Free tier's break-even). Your goal is to inventory the cost-tracking infrastructure, audit the per-call attribution completeness, evaluate the rollup pipeline, and prescribe specific changes — without recommending a complex CDP for a small SaaS where direct logging suffices.

Methodology: Locate every LLM call site. For each, verify a cost-logging hook captures: timestamp, customer_id (or tenant_id), user_id (the specific user, separate from customer), feature, model, input tokens, output tokens, input cost USD, output cost USD, total cost USD, latency, request_id, prompt version (if tracked, prompt 386), success/failure status. Verify the cost rates are accurate and current (model pricing changes; pin to a known-correct table per provider per model). Audit the rollup pipeline: how does per-call data become per-customer-per-month COGS? Is the rollup queryable in real-time or pre-aggregated? Verify per-plan-tier guardrails: does the system know which plan tier a customer is on, and alert when their AI consumption exceeds plan-tier expectations (the COGS:revenue ratio per customer)?

What good looks like: Every LLM call writes a cost-attribution row to a centralized llm_calls (or ai_usage) table with all required fields. Cost is computed at log time using a rates table that's kept current (per-provider, per-model rates with effective dates). The customer_id and feature are always populated; missing values trigger an alert (untracked spend is the worst kind). Per-customer COGS is queryable via SQL: SELECT customer_id, sum(total_cost) FROM llm_calls WHERE created_at >= ? GROUP BY customer_id; Per-feature COGS is similarly queryable. Dashboards surface: top-spending customers, COGS:revenue ratio per customer (flag the ones underwater), per-feature cost trends, total monthly AI spend forecast. Plan-tier guardrails alert when a customer crosses thresholds (Free user > $5/mo, Pro > $50/mo, etc.). Per-tenant cost data is exported to the BI/finance system on a regular cadence.

Per-Call Logging Schema Checklist

• A central table llm_calls (or similar) with columns:
  • id (PK)
  • created_at (timestamp)
  • customer_id / tenant_id (FK, indexed)
  • user_id (FK, indexed) — the specific user within the customer
  • feature (string, indexed) — which feature triggered this call
  • provider (string) — anthropic, openai, etc.
  • model (string) — claude-sonnet-4-6, gpt-4-turbo, etc.
  • input_tokens (Int)
  • output_tokens (Int)
  • input_cost_usd (Decimal(10,6))
  • output_cost_usd (Decimal(10,6))
  • total_cost_usd (Decimal(10,6))
  • latency_ms (Int)
  • request_id (string) — provider's request ID for tracing
  • prompt_version (string, optional)
  • success (boolean)
  • error_code (string, nullable)
• Index on (customer_id, created_at) and (feature, created_at) for fast rollup queries
• Retention: 90 days hot, longer in cold storage / data warehouse

Cost Rate Table Checklist

• A model_rates table (or config) with: provider, model, input_per_1k_usd, output_per_1k_usd, effective_from, effective_to (null for current)
• Updated when providers change pricing (Anthropic's pricing has changed; OpenAI's frequently)
• Cost computed at log time: input_cost = input_tokens * input_per_1k / 1000; output_cost = output_tokens * output_per_1k / 1000; total = input + output
• For computed-fields-vs-stored-cost trade: store the cost (don't recompute later, because rates change)

Centralized Logging Hook Checklist

• Every LLM call passes through a wrapper: await llm.call({ customer_id, feature, model, prompt }) → response
• The wrapper logs the cost row before returning; missing customer_id throws (force callers to pass it)
• For background jobs without a specific user, use a system-customer ID or a feature-level attribution
• Avoid scattered direct SDK calls; the wrapper is the single point

Customer/Tenant ID Propagation Checklist

• For HTTP routes: customer_id is in the auth context; pass to the LLM wrapper
• For background jobs: customer_id is in the job payload; pass through
• For shared/cron features (job catalog crawl, analytics): customer_id may be null or a system value; document and budget separately
• Anti-pattern: relying on the LLM wrapper to "figure out" the customer_id from request context — explicit beats implicit

Feature Attribution Checklist

• Each call site declares its feature: llm.call({ feature: 'ats_check', ... })
• Feature names are stable and documented (a registry); avoid free-form strings that drift
• Reporting groups by feature; per-feature trends inform what to optimize first

Real-Time vs Pre-Aggregated Reporting Checklist

• For low-to-medium volume (< 100K calls/day), querying the raw table for ad-hoc reports is fine
• For high volume, pre-aggregate per-day per-customer per-feature into a daily_ai_costs summary table
• Real-time dashboards: query the summary table; ad-hoc deep-dives: query the raw table
• Materialize at midnight; cron job

Per-Customer COGS Query Pattern Checklist

• Monthly COGS per customer: SELECT customer_id, DATE_TRUNC('month', created_at) AS month, SUM(total_cost_usd) FROM llm_calls GROUP BY customer_id, month;
• Per-feature breakdown: ... GROUP BY customer_id, feature, month;
• COGS as % of revenue: join with subscription / payment data per customer
• Top spenders: ORDER BY SUM(total_cost_usd) DESC LIMIT 20;

Plan-Tier Guardrail Checklist

• For each plan tier, document the expected COGS budget per customer per month
• Free tier: typically $0-2/month; if the tier costs more than that, the unit economics are broken
• Pro tier ($29/month): COGS should be < $5-8/month for healthy margin
• Enterprise: customer-specific
• Guardrails: alert when a customer's monthly COGS exceeds tier expectations
• Action on guardrail breach: investigate (legitimate power user? abuse?), throttle if needed (see prompt 391)

Cost Anomaly Detection Checklist

• Per-customer: trend cost; alert on sudden spikes (3× average)
• Per-feature: trend cost; alert on sudden spikes (a deploy bug may have made every call 10× more expensive)
• Per-tenant: alert on new tenants generating outsize cost (free signup + immediate $50/day = abuse signal)
• Use simple thresholds initially; sophisticated anomaly detection can come later

COGS Reporting to Finance Checklist

• Monthly export of per-customer COGS to the finance system (CSV, API, or direct DB sync)
• Reconcile against the Anthropic / OpenAI invoice: total computed COGS should match (within rounding)
• Material differences indicate logging gaps (calls happening that aren't logged)
• For SaaS metrics (Net Revenue, Gross Margin), per-customer COGS feeds gross margin calculation

Streaming Token Counting Checklist

• For streaming responses, the SDK provides usage data in the final event (Anthropic: message_delta.usage, OpenAI: usage in the last chunk)
• Capture this; don't compute from accumulated text (token-counting from text isn't always accurate due to BPE/tokenizer specifics)
• For aborted streams, the partial output's tokens are still billed; capture from the final event before abort if possible

Tool-Use Cost Accounting Checklist

• Tool use adds tokens (the schema definition is in the request; tool result content adds to subsequent input)
• Capture tool-use input + output tokens accurately; the SDK's usage data covers this
• For multi-turn tool use (the model calls a tool, you respond, the model calls another), each turn is a separate call with its own cost row

Prompt Caching Cost Adjustment Checklist

• Anthropic prompt caching: cached input tokens are billed at a discount (typically 10% of regular)
• The SDK's usage includes cache_read_input_tokens and cache_creation_input_tokens
• Cost calculation must factor these: cached_read_cost = cache_read_tokens * input_rate * 0.1; cache_creation_cost = cache_creation_tokens * input_rate * 1.25 (or whatever the provider's multiplier)
• Without cache-aware cost calculation, cost is overstated for cache-heavy workloads

Per-Request ID for Audit Trail Checklist

• Each LLM call has a provider request_id (Anthropic returns it, OpenAI returns it); store it
• For investigations ("why did this customer's COGS spike on April 15?"), being able to inspect specific calls is essential
• The provider's dashboard supports request lookup; the request_id ties your data to theirs

Sampling vs Full Logging Checklist

• For very high volume (millions of calls/day), full logging may be expensive (DB writes)
• Sample for cost reporting (1% of calls, scaled up) — acceptable accuracy for aggregate but breaks per-customer attribution
• Full logging is correct for cost attribution; only sample if write load demands

Refunds & Failed Calls Checklist

• Failed calls (4xx, 5xx) typically don't bill; verify with the provider
• Some failures (content filter rejections) do bill the input tokens; capture these
• Refunds from the provider (rare) require manual reconciliation

---

Calibration

Don't build per-customer cost attribution before you have customers. The audit's value scales with multi-tenancy and the question "which customer is costing us how much?" Don't recommend a CDP or data warehouse for a small SaaS where Postgres queries on the llm_calls table suffice. Don't sample logging if attribution accuracy matters; full logging is the default for tens of thousands of calls/day. For Enterprise customers with custom pricing, attribution is essential to deliver the value they're paying for; for self-serve plans, attribution is for your margin protection.

• Severity:

  • Critical — No per-call cost logging (total spend known but not attribution); customer_id missing on log rows (untracked spend); cost computed wrong (rates table outdated, prompt caching not factored)
  • High — Feature attribution missing; per-plan guardrails undefined; reconciliation against provider invoice not done
  • Medium — Real-time reporting absent for high-volume; anomaly detection undefined; tool-use tokens not captured
  • Low — Cosmetic dashboard improvements; missing prompt-version tracking
  • Inverse (Over-Built) — CDP for a single-product app with one analytics tool; sampling for low-volume calls; complex anomaly detection for a small customer base

• Confidence ratings: Confirmed (cost log row inspected for sample call, monthly query reconciled against provider invoice, guardrail tested), Likely (attribution gap obvious), Speculative (general best practice).

• Anti-hallucination guard: Don't claim cost is captured without verifying via spot-check of recent calls. Verify the model rates against current provider pricing (changes monthly). Don't recommend a sampling strategy without demonstrating it preserves per-customer accuracy. For prompt caching cost, verify the discount math against the provider's documentation; the multipliers vary.

Output Format

Start with a 3–5 line executive summary: per-call logging completeness, top-spending customer, plan-tier guardrail status, the highest-leverage attribution gap.

1. Logging Schema Findings — Required fields presence, indexing, retention

2. Cost Rate Table Findings — Per-model rates currency, effective-date discipline, cost computation accuracy

3. Centralized Hook Findings — Single wrapper, scattered call sites, missing customer_id enforcement

4. Customer ID Propagation Findings — HTTP route attribution, background-job attribution, shared-feature handling

5. Feature Attribution Findings — Feature registry, naming consistency, per-feature reporting capability

6. Reporting Approach Findings — Real-time vs pre-aggregated, summary table presence, query performance

7. Per-Customer COGS Findings — Query patterns, monthly rollup, COGS:revenue ratio

8. Plan-Tier Guardrail Findings — Per-tier expected budgets, alerting on breach, action workflow

9. Anomaly Detection Findings — Per-customer spike alerts, per-feature spike alerts, abuse pattern detection

10. Finance Reporting Findings — Monthly export, reconciliation against provider invoice, gross margin integration

11. Streaming Token Capture Findings — Final-event usage capture, partial-stream attribution

12. Tool-Use Accounting Findings — Multi-turn tool calls captured per turn

13. Prompt Caching Cost Findings — Cache-aware cost calculation, discount multipliers

14. Request ID Audit Trail Findings — Provider request_id capture, investigation workflow

15. Sampling Findings — Full vs sampled logging decision, accuracy implications

16. Refund & Failure Findings — Failed-call cost handling, content-filter token capture

17. Over-Built Findings — Infrastructure exceeding actual reporting needs

18. Positive Findings — Cost attribution done right; dashboards driving decisions; guardrails preventing margin loss

For each finding: code or schema location, severity, confidence, the specific change, and the impact (attribution accuracy, margin protection, finance defensibility).