Grounded in data engineering. Curious about what comes next.
I'm a senior data engineer working on streaming and batch platforms with Google Cloud, Apache Beam, Flink, Airflow, and BigQuery. I care about the decisions behind the implementation: correctness, recovery, cost, and whether the next engineer can understand and operate the system.
The details matter: late-arriving events, changing schemas, replaying data without duplicates, and keeping warehouse costs predictable. Good engineering means understanding those trade-offs, not just choosing a tool.
I'm also exploring how that foundation supports AI-driven products: trustworthy data for retrieval, useful agent workflows, and ways to evaluate whether an AI system is actually helping. My goal is to connect emerging technology with real industry problems, without losing sight of reliability, privacy, or cost.
Correctness before throughput
A fast pipeline that silently drops late events is worse than a slow one that does not. Event-time semantics, watermarks and dead letter queues are not optional extras.
Failure is the normal case
Brokers restart, schemas drift, upstream teams ship breaking changes on a Friday. Designs get judged on what happens during the incident, not the happy path.
Make it runnable in one command
If a new engineer cannot bring the whole stack up locally, the system is harder than it needs to be. Containerised environments pay for themselves within a week.
Cost is a design constraint
Partitioning, clustering and materialisation choices show up on the invoice. Warehouse design is as much an economics problem as an engineering one.
Streaming
↳Event-time semantics, watermarking, windowing and state management — stay correct when data arrives late, out of order, or twice.
Batch & warehousing
↳Partitioned, clustered models in BigQuery with an eye on query cost, plus the orchestration that keeps them fresh and reliably backfillable.
Reliability
↳Checkpointing, replay, idempotent sinks and dead letter queues — plus the alerting that makes a 3am page actionable instead of terrifying.
Platform
↳Reproducible local environments and CI, so a new engineer runs the whole stack on day one rather than week two.
Experience gives me a foundation, not a reason to stop learning. My current interests include data foundations for AI, retrieval-augmented generation, AI agents & workflow automation, evaluation, reliability & cost, real-time data for intelligent applications, emerging tools & industry use cases. I want to understand where new approaches earn their place, what they improve, and where simpler solutions still work better. I share that learning on the writing page.
- github
- @ankitmohanpandey
- /in/ankitmohanpandey
- substack
- ankitmohanpandey