ENGINEERING METHODOLOGY

Last updated September 2026

How I Build Reliability into Data Infrastructure

Building streaming data pipelines, transaction reconciliation systems, and platform abstractions requires deliberate tradeoffs. Here is the framework I use to go from raw event webhooks to battle-tested production pipelines.

Production data systems shipped

100M/day

Transaction scale scoped in architecture spec

Zero

Duplicate transactions from webhook retries

38/38

Tests passing, gating every deploy

Core Principles

Rules that guide every system design call, from webhook ingestion to warehouse schema.

01 // ARCHITECTURE

Events Over Nightly Batches

Treat incoming data (like M-Pesa/Daraja webhooks) as reactive real-time events rather than static database rows inspected hours later.

In practice: PesaGuard's Flask backend processes Daraja webhooks as they arrive.

02 // GOVERNANCE

Governance as First-Class Code

Schema validation, access logging, and Kenyan Data Protection Act compliance are automated inline directly in CI/CD pipeline code.

In practice: GitHub Actions gates every push with an automated Pytest suite.

03 // PLATFORM THINKING

Refuse to Solve Plumbing Twice

Extract reusable ingestion components, auth mechanisms, and monitoring dashboards after they appear in two live projects.

In practice: Deduplication logic shared across the ETL and M-Pesa pipelines.

04 // RELIABILITY

Idempotency Is Not Optional

Any system that receives webhooks will receive duplicates. I design the retry path before I design the happy path.

In practice: Idempotency-key deduplication so a retried webhook is never double-counted.

"A pipeline that only works when nothing goes wrong isn't a pipeline. It's a demo."

The assumption behind every retry path, validation rule, and idempotency check I write

Step-by-Step Delivery Process

How a system moves from operational requirements to long-term automated infrastructure.

Domain Analysis & Risk Modeling

Map failure modes, edge cases (e.g., duplicate M-Pesa webhooks, out-of-order execution), and compliance boundaries before writing any pipeline code.

OUTCOMEIdentified webhook idempotency as PesaGuard's highest-priority risk before writing the reconciliation engine.

Schema Contract & Ingestion Design

Strict schema contracts at the entry point catch malformed or unexpected payload fields before they leak into downstream models.

OUTCOMEA config-driven rule engine for the M-Pesa cleaning pipeline, so validation rules never need a code change.

Stream Computing & Anomaly Rules

Event-driven streaming with Kafka consumers performs near-real-time aggregation, with consumer-group design built for horizontal scalability.

OUTCOMEExactly-once semantics in the streaming system, so retried events can never be double-counted downstream.

Observability & Automated Governance

Metrics, dashboards, and access logging are wired up so the system operates cleanly without manual intervention.

OUTCOMEGrafana dashboards connected directly to the PostgreSQL output layer for live operational visibility.

Scope for the Next Order of Magnitude

Once a system is stable at current load, I write down what breaks first at 10x and 100x scale, before that becomes an incident.

OUTCOMEA target-state architecture spec scoping PesaGuard to 100M transactions/day, mapped across 12 subsystems.

Trade-offs I Make Deliberately

Good engineering is choosing the right constraint for the current stage, not the most impressive one.

CHOOSEA managed platform like Render for zero-maintenance deployment

DEFERSelf-hosted Kubernetes, until traffic actually justifies the operational overhead

CHOOSEShipping idempotency correctness before any UI polish

DEFERFeature breadth, until the data itself can be trusted

CHOOSEPostgreSQL as the default store, for its reliability and tooling

DEFERExotic database choices, unless a specific access pattern demands them

CHOOSEWriting the architecture spec for 100M tx/day now, while it's cheap

DEFERBuilding for that scale today, before the system needs it

Production Tech Stack

Technologies and tools selected for durability, speed, and maintainability.

STREAMING & BUS

Apache Kafka, PyFlink

Real-time event streams, stateful processing, and windowed aggregations.

BACKEND & API

Python, Django, FastAPI

Rest APIs, Daraja webhook handlers, and web backend infrastructure.

OBSERVABILITY

Prometheus, Grafana

Consumer lag tracking, throughput monitors, and alerting thresholds.

GOVERNANCE

Git CI/CD, ODPC DPA '19

Schema enforcement, access logging, and automated retention purges.