Schema Evolution: Changing the Contract Without Breaking What Runs

A schema change is usually one of the most difficult types of change for a software system. However, it might look quite small and simple in review. For example, it might be something as simple as a column being renamed, or a new field being added to a particular event, or a response payload dropping a field that was not being used.

To make matters more complicated, the migration goes smoothly and cleanly during the staging phase, but unrelated services and components start failing as soon as the change is deployed to production. On investigation, it is found that nothing was wrong with the migration itself. But it took effect while two versions of the application were still running against the same database, and only one of those versions referenced the modified schema.

This situation is common with schema-related changes. It is also not limited to the deployment window. For example, rows written years ago can get produced by application code that has since been replaced. The messages sitting in a queue were published before the current version of the consumer was written. Mobile app versions from eighteen months back are still installed on real devices and still calling the API. In each case, data written under a particular schema version is read under a different version, resulting in multiple issues.

In this article, we will look at schema evolution and strategies for the same. Here’s what we will cover:

• Why more than one schema version is always in play at the same time

• Backward and forward compatibility

• Which changes break consumers, which do not, and the qualifiers that decide it

• Expand and contract migrations

• Schema registries and their use

• How the same problem differs across databases, APIs, and event streams

• Versioning strategies and deprecation timelines

Version Overlap...