Saturday, August 16, 2014

Embedded Kafka and Zookeeper for unit testing

Recently I wanted to setup embedded Kafka cluster for my unit tests, and suprisingly it wasn't that trivial because most of examples I found around were made for some older versions of Kafka/Zookeeper or they didn't work for some other reasons, so it took me some time to find some proper version.

The project I took it from is Camus which is Kafka->Hadoop ETL project, and I just made some slight changes related to newer Zookeeper, as well as some changes configuration-wise.

My embedded Kafka & Zookeeper servers are available at this gist. All the code is tested against Kafka and Zookeeper 3.4.6.

Here is simple example on how to setup Zookeeper at fixed port (2181), and 2 Kafka servers at random available ports:

     EmbeddedZookeeper embeddedZookeeper = new EmbeddedZookeeper(2181);  
     List<Integer> kafkaPorts = new ArrayList<Integer>();  
     // -1 for any available port  
     EmbeddedKafkaCluster embeddedKafkaCluster = new EmbeddedKafkaCluster(embeddedZookeeper.getConnection(), new Properties(), kafkaPorts);  
     System.out.println("### Embedded Zookeeper connection: " + embeddedZookeeper.getConnection());  
     System.out.println("### Embedded Kafka cluster broker list: " + embeddedKafkaCluster.getBrokerList());  


  1. I'm using your example to test reading from Kafka into a Storm topology. How would you setup a topic and populate it with some messages?


  2. Well, you just use standard client APIs to produce a messages (producer API), same as you would do if there was remote broker in play here. Same thing for creating a topic.
    Although creating a topic is something mostly done via CLI command, but here you need to use programmatic way, thus look at Kafka's AdminClient class.

    Something like:
    ZkClient zkClient = new ZkClient(embeddedZookeeper.getConnection());
    AdminUtils.createTopic(zkClient, topic, 1, 1, new Properties());

  3. Any advice on how you coordinate producing messages and then consuming them as part of a test? Particularly where the test logic isn't in the consumer but the test calls the producer and consumer to generate and fetch the data to validate. It is simplest to produce and consume concurrently, but that could be a bit tricky to coordinate. Not sure if one can coordinate it to producing test messages first then consuming after the production serially/sequentially to simplify testing. Your thoughts?

  4. If I understood your needs, you need some kind of "promise" mechanism for fetching the result of async process (in this case consumer callback) in the original thread that produced the message. There are other ways, but if you're using java 8, the most obvious way is CompletableFuture. You just call future.compete(message) from your consumer code, and have producer waiting for it in original thread via future.get();

  5. Hi,

    I just tried the code under 0.8.2 kafka version. I have an issue when I try to push something in the only one Kafka broker.

    [KafkaApi-1] error when handling request Name: TopicMetadataRequest; Version: 0; CorrelationId: 321764; ClientId: producer-1; Topics: offers
    kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0

    From my understanding, it seems that no Kafka server is started ?

  6. I solved it by using

    ZkClient zkClient = new ZkClient(embeddedZookeeper.getConnection(), 10000, 10000, ZKStringSerializer$.MODULE$);
    AdminUtils.createTopic(zkClient, topic, 10, 1, new Properties());

  7. This is the exact information I am been searching for, Thanks for sharing the required infos with the clear update and required points.
    redmi service center in velachery
    redmi note service center in chennai
    redmi mobile service centre in chennai

  8. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.
    date analytics certification training courses
    data science courses training

  9. wow, great, I was wondering how to cure acne naturally. and found your site by google, learned a lot, now i’m a bit clear. I’ve bookmark your site and also add rss. keep us updated.
    Data Science Course in Pune

  10. Wow, what an awesome spot to spend hours and hours! It's beautiful and I'm also surprised that you had it all to yourselves! Kindly visit us @ Best HIV Treatment in India | Top HIV Hospital in India | HIV AIDS Treatment in Mumbai
    HIV Specialist in Bangalore | HIV Positive Treatment in India | Medicine for AIDS in India

  11. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
    Thank you for your post, I look for such article along time, today i find it finally. this post give me lots of advise it is very useful for me.
    It should be noted that whilst ordering papers for sale at paper writing service, you can get unkind attitude. In case you feel that the bureau is trying to cheat you, don't buy term paper from it.
    Data Science Courses

  12. Its as if you had a great grasp on the subject matter, but you forgot to include your readers. Perhaps you should think about this from more than one marketing course in singapore

  13. This is a wonderful article. I really enjoyed reading this article. Thanks for sharing such detailed information.
    Data Science Course
    Data Science Course in Marathahalli
    Data Science Course Training in Bangalore

  14. It’s very informative and you are obviously very knowledgeable in this area. You have opened my eyes to varying views on this topic with interesting and solid content.
    data analytics courses

  15. Actually I read it yesterday but I had some thoughts about it and today I wanted to read it again because it is very well written.
    Data Analyst Course

  16. This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea.
    data science courses

  17. Nice blog, it's so knowledgeable, informative, and good looking site. I appreciate your hard work. Good job. Thank you for this wonderful sharing with us. Keep Sharing.
    DevOps Training in Chennai

    DevOps Course in Chennai

  18. Thank a lot. You have done excellent job. I enjoyed your blog . Nice efforts
    data science course

  19. Here at this site is really a fastidious material collection so that everybody can enjoy a lot.
    business analytics course in hyderabad

  20. I would like to thank you for the efforts you have made in writing this article. I am hoping for the same best work from you in the future as well. In fact your creative writing abilities have inspired me to start my own Blog Engine blog now. Really the blogging is spreading its wings rapidly. Your write up is a fine example of it.
    best data science training in hyderabad