Some Things I Picked Up Working with Kafka

I used Kafka for a distributed systems project I was building and also got to work with event-driven patterns during my internship. These are some things I ran into and learned from. Nothing groundbreaking, but maybe useful if you’re getting started with this stuff.

Schemas matter more than I expected

My first setup used plain JSON with no schema registry. It worked great until I renamed a field on the producer side and forgot to update the consumer. The consumer just silently dropped the data because the key didn’t match anymore. Took me a while to figure out what happened.

After that I started treating schemas more like database migrations. Something you version, validate in CI, and don’t change casually. Avro with a schema registry works well for this. I’ve heard Protobuf is even better if you’re okay with the extra tooling, but I haven’t used it enough to say.

Make consumers idempotent

Messages get delivered more than once. That’s just how it works with Kafka. So every consumer needs to handle duplicates gracefully. If it gets the same event twice, it shouldn’t create duplicate records or corrupt state.

The approach that worked for me: check if you’ve already processed an event ID before doing anything, and use upserts at the database level so replaying a message doesn’t break things.

async function handleEvent(event: DomainEvent) {
  const existing = await db.processedEvents.find(event.id);
  if (existing) {
    logger.debug('Duplicate event, skipping', { eventId: event.id });
    return;
  }

  await db.transaction(async (tx) => {
    await tx.processedEvents.insert({
      id: event.id,
      processedAt: new Date(),
    });
    await tx.orders.upsert(event.payload);
  });
}

Monitor consumer lag early

Consumer lag tells you how far behind your consumers are from the latest messages. I didn’t set this up initially and only found out things were broken when the output data looked wrong. Once I added lag monitoring, problems showed up way earlier and were easier to trace.

If you’re using Kafka, set up lag monitoring right at the start. It’s probably the most useful single metric you can track.

Partition keys aren’t trivial

I initially didn’t think much about partition keys and just picked something convenient. But the choice matters a lot. A bad key can send most of your traffic to one partition while others sit idle. A good key spreads load evenly while keeping related events in order.

Some patterns I’ve seen work: partition by customer ID for order processing (keeps per-customer ordering without global ordering overhead), hash of session ID for analytics, entity type + ID for audit logs.

Is it worth the complexity?

Honestly, event-driven architecture adds a lot of moving parts. More things to set up, more things that can go wrong, more things to monitor. For the project I was working on it made sense because we needed services to be decoupled and process events independently. But if you’re building a straightforward CRUD app, it’s probably overkill.

I think the main thing is to pick the architecture based on what the project actually needs, not based on what sounds interesting to build.