In Kafka, it's generally recommended to map one table to one topic. However, what if the order of events across multiple tables is important for a specific key, such as CompanyId?
For example, consider the following scenario: events for a CompanyId must be processed (consumed) in order, even though they originate from different tables. Partitioning is configured such that each partition corresponds to a specific CompanyId:
Is it possible to ensure that the order of events (transactions) for each CompanyId is reliably preserved across all relevant tables? For instance, achieving something like this:
I use Avro for schema control.
In Kafka, it's generally recommended to map one table to one topic. However, what if the order of events across multiple tables is important for a specific key, such as CompanyId?
For example, consider the following scenario: events for a CompanyId must be processed (consumed) in order, even though they originate from different tables. Partitioning is configured such that each partition corresponds to a specific CompanyId:
Is it possible to ensure that the order of events (transactions) for each CompanyId is reliably preserved across all relevant tables? For instance, achieving something like this:
I use Avro for schema control.
I had to solve this problem so I could maintain atomic apply of messages in Kafka that represented replicated database transactions. My requirements were stricter in that I needed total ordering of the original source sequence of messages for my customers and to maintain the context of where transaction boundaries occurred.
I designed the following technology to do this and it is in use at a large number of our financial industry customers who need to ensure exactly once characteristics and perfect ordering, along with maintaining transaction atomicity.
https://www.ibm.com/docs/en/idr/11.4.0?topic=kafka-transactionally-consistent-consumer
I gave a talk at the Kafka conference describing the internals of the approach… I think Confluent has it posted somewhere….
Ah here…
https://www.confluent.io/kafka-summit-sf18/a-solution-for-leveraging-kafka-to-provide-end-to-end-acid-transactions/
It is essentially a two phase commit approach to the problem utilizing the information in callbacks to record which exact message is the next to return, and injecting into the meta topic information about when commits occurred on the source.
