The KafkaProducer send method both returns a Future and accepts a Callback.
Is there any fundamental difference between using one mechanism over the other to execute an action upon completion of the sending?
4 Answers
Answers 1
The main difference is whether you want to block the calling thread waiting for the acknowledgment.
The following using the Future.get() method would block the current thread until the send is completed before performing some action.
producer.send(record).get() // Do some action When using a Callback to perform some action, the code will execute in the I/O thread so it's non-blocking for the calling thread.
producer.send(record, new Callback() { // Do some action } }); Though the docs says it 'generally' executed in the producer:
Note that callbacks will generally execute in the I/O thread of the producer and so should be reasonably fast or they will delay the sending of messages from other threads. If you want to execute blocking or computationally expensive callbacks it is recommended to use your own Executor in the callback body to parallelize processing.
Answers 2
The asynchronous approach
producer.send(record, new Callback(){ @Override onComplete(RecordMetadata rm, Exception ex){...} }) gives you better throughput comparing to synchronous
RecordMetadata rm = producer.send(record).get(); since you don't wait for acknowledgements in first case.
Another difference could be that in synchronous call in case of exception you can stop sending messages straightaway after the exception occurs, whereas in second case some messages will be sent before you discover that something is wrong and perform some actions.
Also note that in asynchronous approach the number of messages which are "in fligh" is controlled by max.in.flight.requests.per.connection parameter.
Apart from synchronous and asynchronous approaches you can use Fire and Forget approach, which is almost the same as synchronous, but without processing the returned metadata - just send the message and hope that it will reach the broker (knowing that most likely it will happen, and producer will retry in case of recoverable errors), but there is a chance that some messages will be lost:
RecordMetadata rm = producer.send(record); To summarize:
- Fire and Forget - fastest one, but some messages could be lost;
- Synchronous - slowest, use it if you cannot afford to lose messages;
- Asynchronous - something in between.
Answers 3
My observations based on The Kafka Producer documentation:
Futuregives you access to synchronous processingFuturemight not guarantee acknowledgement. My understanding is that aCallbackwill execute after acknowledgementCallbackgives you access to fully non-blocking asynchronous processing.- There are also guarantees on the ordering of execution for a callback on the same partition
Callbacks for records being sent to the same partition are guaranteed to execute in order.
My other opinion that the Future return object and the Callback 'pattern' represents two different programming styles and I think that this is the fundamental difference:
- The
Futurerepresents Java's Concurrency Model Style. - The
Callbackrepresents Java's Lambda Programming Style (because Callback actually satisfies the requirement for a Functional Interface)
You can probably end up coding similar behaviors with both the Future and Callback styles, but in some use cases it looks like one might style be more advantageous than the other.
Answers 4
Looking at the documentation you linked to it looks like the main difference between the Future and the Callback lies in who initiates the "request is finished, what now?" question.
Let's say we have a customer C and a baker B. And C is asking B to make him a nice cookie. Now there are 2 possible ways the baker can return the delicious cookie to the customer.
Future
The baker accepts the request and tells the customer: Ok, when I'm finished I'll place your cookie here on the counter. (This agreement is the Future.)
In this scenario, the customer is responsible for checking the counter (Future) to see if the baker has finished his cookie or not.
blocking The customer stays near the counter and looks at it until the cookie is put there (Future.get()) or the baker puts an apology there instead (Error : Out of cookie dough).
non-blocking The customer does some other work, and once in a while checks if the cookie is waiting for him on the counter (Future.isDone()). If the cookie is ready, the customer takes it (Future.get()).
Callback
In this scenario the customer, after ordering his cookie, tells the baker: When my cookie is ready please give it to my pet robot dog here, he'll know what to do with it (This robot is the Callback).
Now the baker when the cookie is ready gives the cookie to the dog and tells him to run back to it's owner. The baker can continue baking the next cookie for another customer.
The dog runs back to the customer and starts wagging it's artificial tail to make the customer aware that his cookie is ready.
Notice how the customer didn't have any idea when the cookie would be given to him, nor was he actively polling the baker to see if it was ready.
That's the main difference between the 2 scenario's. Who is responsible for initiating the "your cookie is ready, what do you want to do with it?" question. With the Future, the customer is responsible for checking when it's ready, either by actively waiting, or by polling every now and then. In case of the callback, the baker will call back to the provided function.
I hope this answer gives you a better insight in what a Future and Calback actually are. Once you got the general idea, you could try to find out on which thread each specific thing is handled. When a thread is blocked, or in what order everything completes. Writing some simple programs that print statements like: "main client thread: cookie recieved" could be a fun way to experiment with this.
0 comments:
Post a Comment