2G

During a rolling deployment, old and new consumers/producers run simultaneously. Message format changes must be safe for both versions to process.

This guide walks through safe message evolution patterns such as versioned payloads, additive fields, dual-publishing during queue migrations, and guarded format changes so producers and consumers can upgrade independently.

Quick Definitions

• Producer: the service that publishes a message to a queue or topic
• Consumer: the service that reads and processes a message
• DLQ (Dead Letter Queue): a queue where messages are sent after processing fails
• Schema evolution: changing a message structure over time without breaking existing producers or consumers
• Dual-publish: temporarily sending the same event to both old and new destinations during a migration

The Core Rule: Include a version Field

Add a version field to every message at creation. It makes format evolution explicit and safe:

{ "version": "2", "userId": "123", "name": "Alice", "email": "alice@example.com" }

Consumers route on version:

if ("2".equals(event.getVersion())) {
    processNewLogic(event);
} else {
    processOldLogic(event);  // handles version "1" and unversioned
}

Takeaway: explicit message versioning turns compatibility decisions into code instead of guesswork.

1. Adding New Fields to Messages

Situation: Adding email to UserCreatedEvent, but older consumers don’t expect it.

Producer Strategy

Use a toggle to control when the new field is sent:

public UserCreatedEvent createEvent(User user) {
    UserCreatedEvent event = new UserCreatedEvent();
    // Version 2 means the payload may include new fields such as email.
    event.setVersion(toggle.isEnabled("users.events.new-user-format") ? "2" : "1");
    event.setUserId(user.getId());
    event.setName(user.getName());
    if (toggle.isEnabled("users.events.new-user-format")) {
        // Only newer consumers are expected to read this field.
        event.setEmail(user.getEmail());
    }
    return event;
}

Older consumers process V1 messages; toggled-on producers send V2.

Consumer Compatibility Rule

Tip

Consumer rule: New consumers must handle messages without the new field (old producers may still be sending them). Use Optional or null-check:

String email = event.getEmail() != null ? event.getEmail() : "unknown";

Real-World Scenario

An account service may start publishing email for analytics, while billing consumers still only care about userId and name. If the new field is optional and versioned, billing keeps working while analytics can adopt the richer payload.

Takeaway: Adding fields is usually safe only when older consumers can ignore them and newer consumers can tolerate their absence.

2. Changing Message Processing Logic

Situation: Consumer updates how it handles OrderProcessedEvent, but producers still send the old format.

Consumer Strategy

@RabbitListener(queues = "order.processed")
public void consume(OrderProcessedEvent event) {
    // New logic runs only when both the feature flag and the newer payload version are present.
    if (toggle.isEnabled("orders.processing.new-logic") && "2".equals(event.getVersion())) {
        processNewLogic(event);
    } else {
        processOldLogic(event);  // safe fallback for old-version messages
    }
}

Real-World Scenario

Suppose a new fraud-check flow depends on fields that only exist in version 2 events. Without the version gate, older messages may enter code paths that assume unavailable data and fail unnecessarily.

Takeaway: Behaviour changes should be guarded by both code rollout controls and message-version awareness.

3. Migrating to a New Queue or Topic

Situation: Moving from user.created to user.events.v2.

Dual-Publish Strategy

Dual-publish during migration, toggle stops the old queue publish once all consumers have migrated:

public void sendEvent(User user) {
    // Always publish to new queue
    UserCreatedEvent newEvent = new UserCreatedEvent("2", user.getId(), user.getName(), user.getEmail());
    rabbitTemplate.convertAndSend("user.exchange", "user.events.v2", newEvent);
    // Continue publishing to old queue until all consumers migrated
    if (!toggle.isEnabled("users.events.disable-old-queue")) {
        UserCreatedEvent oldEvent = new UserCreatedEvent("1", user.getId(), user.getName(), null);
        rabbitTemplate.convertAndSend("user.exchange", "user.created", oldEvent);
    }
}

Migration Steps

Migration steps:

1. Producer dual-publishes (old + new queue)
2. Teams migrate consumers to new queue (one team at a time)
3. Monitor old queue consumer count → reaches 0
4. Enable toggle to stop publishing to old queue
5. Delete old queue after 1 week with no activity

Migration Diagram

Phase 1: Producer -> old queue + new queue
         Consumers -> old queue
Phase 2: Producer -> old queue + new queue
         Some consumers -> new queue
         Some consumers -> old queue
Phase 3: Producer -> new queue only
         All consumers -> new queue

Common Pitfall

Deleting the old queue too early can strand a low-traffic consumer that was still depending on it, especially if that consumer only runs nightly or weekly jobs.

Takeaway: queue migrations are coordination problems, not just routing changes. Keep the old path alive until usage is truly zero.

4. Changing Message Format (JSON → Avro)

Situation: Switching to Avro for performance; older consumers expect JSON.

Format Rollout Strategy

public void sendEvent(User user) {
    if (toggle.isEnabled("users.events.use-avro-format")) {
        // Send the compact binary form only after consumers are ready for it.
        AvroUserEvent avroEvent = convertToAvro(user);
        kafkaTemplate.send("user.topic", avroEvent.toByteArray());
    } else {
        // Keep the legacy JSON path available during migration.
        UserCreatedEvent jsonEvent = new UserCreatedEvent("1", user.getId(), user.getName(), null);
        kafkaTemplate.send("user.topic", jsonEvent);
    }
}

Warning

Format migration (JSON → Avro, Avro schema evolution) requires a Schema Registry. Ensure your schema registry is set up before switching serialisation format.

Real-World Scenario

Teams often move from JSON to Avro to reduce payload size and standardize schema management. The risk is not the new format itself, but consumers that still assume plain JSON and fail deserialization immediately when the first binary payload arrives.

Takeaway: Serialization changes are breaking by default. Treat them as staged migrations with explicit readiness checks.

5. Deprecating a Message Type

Situation: Replacing UserCreatedEvent with a broader UserEvent.

Event-Type Migration Strategy

public void sendEvent(User user) {
    // Always publish new event type
    UserEvent newEvent = new UserEvent(user.getId(), user.getName(), user.getEmail());
    rabbitTemplate.convertAndSend("user.exchange", "user.events", newEvent);
    // Continue publishing old type until all consumers migrated
    if (!toggle.isEnabled("users.events.disable-user-created-event")) {
        UserCreatedEvent oldEvent = new UserCreatedEvent("1", user.getId(), user.getName(), null);
        rabbitTemplate.convertAndSend("user.exchange", "user.created", oldEvent);
    }
}

Common Pitfall

Broad replacement events are attractive because they reduce event sprawl, but they can also push too much meaning into one schema. Consumers that only depended on a narrow event type may need extra filtering logic after migration.

Takeaway: when deprecating an event type, keep the old event flowing until consumers have migrated and validated their new assumptions.

Monitoring Queue Health During Migration

Watch these metrics in Datadog throughout a message format migration:

Takeaway: migration success is measured in queue health and consumer behavior, not just whether producers deployed successfully.

DLQ Safety Check

Before replaying Dead Letter Queue messages after a schema/code change, ask:

1. Why did they fail? (deserialization error, validation failure, downstream unavailable?)
2. Will they fail again? (if the consumer bug is fixed, they should process now)
3. Are they compatible with the new schema? (if DB schema changed, will old messages produce bad data?)
4. Test replay on staging first => replay a sample batch before bulk-replaying production DLQ

Danger: ANTI-PATTERN

Changing a message field from optional to required without a migration plan.

If old producers don’t send the field and new consumers require it, processing fails silently (messages go to DLQ). Always treat adding required fields as a breaking change – use Optional or null-checks in consumers.

Summary

• Put a version field on every message so consumers can route safely.
• Add fields and behaviours in a way that tolerates mixed old and new producers and consumers.
• Use dual-publish and delayed cleanup for queue or event-type migrations.
• Treat format changes like JSON to Avro as staged compatibility work, not simple implementation swaps.
• Watch queue depth, DLQ volume, and consumer errors throughout the rollout.

Written by

Balaji G