System Design FAQ: Top Questions

An Audit Logging System captures user and system actions in a tamper-evident way for security, compliance, and debugging. It must be durable, structured, searchable, and immutable.

📋 Functional Requirements

• Capture who did what, when, and where
• Immutable log writes with schema validation
• Support filtering/searching based on fields (user, action, resource)
• Retain logs for a configurable period

📦 Non-Functional Requirements

• High write throughput and durability
• WORM (Write Once Read Many) storage compliance
• Tamper detection and retention policy support

🏗️ Core Components

• Log Emitter: Client sends structured logs
• Log Ingestor: Queues and batches entries for storage
• Indexer: For search/filter support
• Store: WORM-compatible DB or blob store with index

📄 Sample Log Schema (JSON)

{
  "timestamp": "2025-06-11T10:22:00Z",
  "user_id": "u_456",
  "action": "delete",
  "resource": "invoice#98213",
  "ip": "192.168.2.23",
  "metadata": {
    "client": "web",
    "reason": "manual override"
  }
}
        

🚦 Kafka Ingestion + S3 Archival (Architecture)

• Apps publish log events to Kafka topic (e.g., audit.events)
• Log processor validates schema + writes to S3 (partitioned by date)
• Parallel stream indexes metadata into OpenSearch
• Retention policy auto-deletes logs from OpenSearch but preserves S3

🔐 S3 Bucket Policy for WORM

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WORMRetention",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:DeleteObject",
      "Resource": "arn:aws:s3:::audit-logs/*",
      "Condition": {
        "StringNotEquals": {
          "s3:VersionId": "null"
        }
      }
    }
  ]
}
        

🧪 Log Producer Code (Node.js)

const axios = require("axios");

async function emitAuditLog(log) {
  await axios.post("https://api.example.com/audit", log);
}

emitAuditLog({
  timestamp: new Date().toISOString(),
  user_id: "u_987",
  action: "update",
  resource: "user#34",
  metadata: { browser: "chrome", feature: "profile_edit" }
});
        

📈 Observability

• Audit logs ingestion rate
• Malformed log count
• Search latency and error rate

🧰 Tools/Infra Used

• Ingestion: Kafka / Kinesis
• Storage: AWS S3 (versioned + locked)
• Search: OpenSearch / ElasticSearch
• Schema Validation: JSON Schema + Ajv / Pydantic

📌 Final Insight

Audit logs are critical for accountability and incident investigation. Immutability, searchability, and WORM compliance are the pillars of a robust audit logging system.