Audit logs and tamper-evident records

Designing audit trails that answer who did what and when: what to record, append-only storage, immutability and retention, hash chains and Merkle trees for tamper evidence, querying and exporting, privacy in audit data, and the difference between audit logs, application logs and event sourcing.

Reading is half of it. See this used in a real interview: walk through Design a Payment System →

When something goes wrong (money moved unexpectedly, a document was shared outside the company, an admin changed a permission), someone must be able to answer who did what, when, from where, and with what result. Audit logs provide that answer. They are required for financial systems, enterprise products and regulated industries, and interviewers expect them in payment, banking and B2B designs. Done well, they are complete, immutable, searchable and trustworthy even against insiders.

What to record

Each audit event captures:

• Actor: user id, service identity or admin, plus the authentication context (session, API key, impersonation by support staff).
• Action: a stable verb (permission.granted, payment.refunded, export.created).
• Target: the resource and its tenant.
• Time: a UTC timestamp from a trusted source. See time zones and dates.
• Context: IP address, user agent, request id, trace id. See distributed tracing.
• Outcome: success or failure, and for changes, the before and after values (or a reference to them).

Record security-relevant actions: logins and failures, permission and role changes, data exports and downloads, configuration changes, money movements, deletions, and administrative or support access to customer data.

Audit logs are not application logs

Do not rely on application logs as an audit trail; they are sampled, rotated and edited by pipelines.

Writing them reliably

An action and its audit record must not diverge:

• Write the audit event in the same transaction as the change when they share a database, or through an outbox that guarantees the event is published. See change data capture and the outbox pattern.
• Ship events to a dedicated audit service or stream, separate from the systems being audited, so a compromised service cannot erase its own trail.
• For failed or denied attempts, log at the authorization layer too.

Immutability and tamper evidence

Insiders with database access could alter records. Make tampering detectable or impossible:

• Append-only storage with no update or delete permissions for applications; write-once object storage with retention locks.
• Hash chains: each record includes the hash of the previous one, so modifying or removing any record breaks every later hash. Periodically publish or anchor the latest hash somewhere external.
• Merkle trees over batches allow efficient proofs that a record is included and unchanged; certificate transparency logs use this.
• Separate administrative control of the audit store from the systems it records.

See hashing and encoding.

Event sourcing versus audit logs

In an event-sourced system, the event store already records every state change, which covers much of an audit need. It still lacks reads, denied attempts and actor context unless you add them, and it is designed for rebuilding state, not for compliance queries. Many systems keep both. For money, the double-entry ledger is itself an immutable audit of balances. See payments and ledgers.

Retention and privacy

• Retention is set by law and contracts (often years for financial records).
• Audit logs contain personal data (who did what). Minimise it: store user ids rather than names and emails, and keep access to audit data restricted and itself audited.
• Deletion requests conflict with retention duties; legal retention usually wins for audit records, but document the policy, and use pseudonymised ids where possible. See privacy and data deletion.

Querying and exposing

• Store recent audit data in a searchable store (by actor, target, action, time range) and older data in cheap, immutable archives.
• Enterprise products expose audit logs to customers: an admin console view, export APIs and streaming to the customer's security tools, scoped strictly by tenant. See multi-tenancy and Design Slack.
• Alert on suspicious patterns: mass exports, permission escalations, logins from unusual places. See trust and safety.

Scale

Audit volume can be large (every file access in a storage product). Partition by tenant and time, compress archives, and keep write paths asynchronous after the reliable handoff so auditing does not slow user actions. See Design Dropbox.

In the interview

For Design a Payment System: "Every money movement is in the append-only ledger; administrative actions (refunds, limit changes) also produce audit events written through the outbox to a separate audit store with hash chaining and retention locks; support staff access to customer data is logged and reviewed." For an exchange (Design a Stock Exchange), the sequenced order log doubles as an audit trail.

Checklist

• Actor, action, target, time, context and outcome for every relevant action.
• Separate from application logs; complete, not sampled.
• Written atomically with the change or via an outbox.
• Append-only, access-controlled storage with hash chains or Merkle proofs.
• Retention per regulation; minimal personal data; restricted access.
• Searchable recent data, archived history, customer-facing exports and alerts.