So I have a scenario in kafka where there are 2 topics : A and B. Topic A's consumers are online (and need to consume data fast) and topic B's consumers are offline. Since topic B's consumers are offline, I don't need to cache any message related to topic B right now and want topic A to get the whole cache. Is it possible to disable caching for some kafka topics?
Edit: What I mean by caching is that the messages are being stored in memory and consumers are also reading from the memory (instead of the disk). In this scenario, I would want messages related to topic A to be cached in memory. And messages related to topic B don't have to be cached in memory. I want topic A to use the space in memory that would have been given to topic B
2 Answers
Answers 1
You can disable caching for particular consumers :
If you would like to disable the caching for Kafka consumers, you can set spark.streaming.kafka.consumer.cache.enabled to false.
It is based on this github repository and i would i suggest you follow it for further clarifications.
Answers 2
Kafka relies heavily on the filesystem for storing and caching messages. The only configuration available in Kafka for caching is the client's streaming buffer size which is used to buffer incoming messages -- cache.max.bytes.buffering.
0 comments:
Post a Comment