I'm currently in a project where JPA and Kafka are used. I'm trying to find a set of good practice for combining those operations.
In the existing code, the producer is used in the same transaction as jpa, however, from what I have read, it seems that they don't share a transaction.
@PostMapping @Transactional public XDto createX(@RequestBody XRequest request) { Xdto dto = xService.create(request); kafkaProducer.putToQueue(dto, Type.CREATE); return dto; } where the kafka producer is defined as the following:
public class KafkaProducer { @Autowired private KafkaTemplate<String, Type> template; public void putToQueue(Dto dto, Type eventType) { template.send("event", new Event(dto, eventType)); } } Is this a valid use case for combining jpa and kafka, are the transaction boundaries defined correctly?
3 Answers
Answers 1
this would not work as intended when the transaction fails. kafka interaction is not part of transaction.
You may want to have a look at TransactionalEventListener You may want to write the message to kafka on the AFTER_COMMIT event. even then the kafka publish may fail.
Another option is to write to db using jpa as you are doing. Let debezium read the updated data from your database and push it to kafka. The event will be in a different format but far more richer.
Answers 2
You shouldn't put the sending message to kafka in transaction. If you need the logic when if it fails to send event to kafka, then revert transaction, it will be better to use spring-retry in this case. Just put the code related to the sending event to kafka in @Retryable annotated method, and also add the @Recover annotated method with the logic of reverting changes to DB made before.
Answers 3
By looking at your question, I'm assuming that you are trying to achieve CDC (Change Data Capture) of your OLTP System, i.e. logging every change that is going to the transactional database. There are two ways to approach this.
- Application code does dual writes to transactional DB as well as Kafka. It is inconsistent and hampers the performance. Inconsistent, because when you make the dual write to two independent systems, the data gets screwed when either of the writes fails and pushing data to Kafka in transaction flow adds latency, which you don't want to compromise on.
- Extract changes from DB commit (either database/application-level triggers or transaction log) and send it to Kafka. It is very consistent and doesn't affect your transaction at all. Consistent because the DB commit logs are the reflections of the DB transactions after successful commits. There are a lot of solutions available which leverage this approach like databus, maxwell, debezium etc.
If CDC is your use case, try using any of the already available solutions.
0 comments:
Post a Comment