Do you actually need Kafka?

Kafka is the first thing most teams reach for when they see events. More than 80% of the Fortune 100 run it, and plenty of teams add it before they have a reason to. We run it where it earns its cost, and we steer teams to something simpler when that is the better call.

Streaming infrastructure is worth its real operational cost when the stream is the system. For many teams reaching for it, the queue only supports the system, and a simpler tool fits better. The skill is telling the two apart.

We have built both kinds. This is where Kafka earns its keep, where it is overkill, and how we decide.

Key Takeaways

• Kafka is the reflexive default: 80%+ of the Fortune 100 run it (Apache Kafka).
• It earns its keep when the stream is the system. On Vitals, a change-data-capture pipeline into Kafka computes a live score at 20,000 events a second, under three seconds end to end, without touching the source CRM.
• It is the wrong call when the queue only supports the system; a simpler queue like Postgres or SQS is less to run and easier to debug.
• AI makes a Kafka setup trivial to generate, so the judgment about whether you need it matters more.

Reputation. Kafka runs at more than 80% of the Fortune 100 (Apache Kafka), and that ubiquity makes it the safe-sounding default. Reach for Kafka and nobody questions the choice. Reach for something smaller and you have to defend it.

The other reasons are familiar: we might need scale someday, it is what the big companies use, and it looks good on the team. The reflex is understandable. The part that gets skipped is the operating cost: a cluster to run, brokers to keep healthy, consumer groups and offsets to reason about, and a new failure surface that has nothing to do with your product.

That operating cost is worth paying when Kafka is the right tool. The point is to be sure it is, before you take it on.

When the stream is the system. The clearest example in our work is Vitals, a daily performance dashboard Inside Real Estate built for every brokerage on BoldTrail, around 400,000 users. The score had to be near-real-time and consistent across every level of the org, and we could not modify the CRM it sat on top of.

So we stood outside the CRM and listened to it. A Debezium change-data-capture layer reads BoldTrail's transaction log and streams every change into Kafka. Vitals consumes that stream and rebuilds everything it needs, without ever querying the CRM. Decoupling from a source you cannot touch is exactly what streaming is for.

Each of the eight KPIs runs as its own stream of derived events. A microservice joins the events that move a metric and emits a running value, so the number is ready before the dashboard opens. No nightly batch. If a connector fails, Vitals reconciles from BoldTrail and rebuilds the affected KPIs, so streaming carries the steady state and reconciliation covers the hiccups.

Every reason to pay for Kafka was present at once: a source we had to decouple from, high sustained throughput, many derived consumers, and replay when the streaming path hiccuped. The clearest signal it was the right call came afterward, when Inside Real Estate's own teams adopted the same pattern for later work.

The full Vitals build goes deeper on the pipeline.

When the queue only supports the system. Most times a team reaches for Kafka, what they need is a reliable queue: a list of jobs to work through, a way to hand work between services, a buffer in front of a slow consumer. For that, Kafka is a lot of machinery to run.

A Postgres-backed queue, or a managed queue like SQS, does that job with far less to operate. There is no cluster to keep healthy and no new failure surface. A message is a row you can read, a failure is visible, and a stuck job can be fixed with plain SQL. For early-stage products, internal tools, and moderate workloads, that is usually the right answer, and most teams overestimate the scale they will actually reach.

We make that call regularly. When an engagement needs a reliable way to process background work at a moderate rate, we reach for a Postgres-backed queue. It handles the load, stays easy to inspect, and adds nothing extra to operate. The day a workload genuinely outgrows it is the day to revisit, and most workloads never do.

Kafka is excellent at what it is for. The expensive mistake is paying its operating cost for a problem a queue already solves.

It lowers the cost of the wrong reflex. AI will generate a working Kafka setup in minutes, which makes reaching for it easier than it already was. What the generated config does not show is the operating cost you have just signed up for.

So the judgment matters more than ever. The question is the same as it always was, and now you have to ask it on purpose, because the tooling will happily skip it. That is the broader pattern in what AI changes about building software: the work gets faster, the judgment stays yours.

We ask one question first: does the queue support the system, or is the stream the system? If the queue moves work between parts of a system that would be fine without it, we use the simplest reliable queue. If the stream is the product, the way it was for Vitals, we reach for Kafka and pay its cost on purpose.

Then we size that cost honestly, against the real, demonstrated need, rather than the scale we might reach someday. The simplest tool that meets the actual requirement wins, until the requirement genuinely changes.

Knowing when a heavy tool earns its place is most of what senior engineering is. It is the work we do as an ongoing engineering partner: the calls that are cheap to make right and expensive to undo.