Quick Notes

Things that came on the way

Creating a Single Node Kafka Server Using Chef

If you need to create a single node Kafka server quickly for development or testing and need to be repeatable, the following are the steps. This uses Chef-Solo and community cookbooks.

Pre-req:

  • A linux server machine or VM with just the OS installed

  • Connection to internet

  • git, curl and wget tools available for the user

  • If the tools are not available, as root or using sudo install these tools using apt-get or yum

Installing Chef:

  • As root:
1

curl -L https://www.opscode.com/chef/install.sh | bash

  • cwd to a directory where you want to run chef-solo from

  • Run:

1

wget http://github.com/opscode/chef-repo/tarball/master
  • Do:
1

tar -zxf master
  • Do:
1

mv opscode-chef-repo* chef-repo
  • Do:
1

rm master
  • cd to chef-repo; if you do a ls there will be a list of directories
1

mkdir .chef
  • Do:
1

echo "cookbook_path [ '/path-to/chef-repo/cookbooks' ]" > .chef/knife.rb

  • This completes the installation and configuration of Chef

  • cd into the chef-repo/cookbooks directory

1
2

git clone https://github.com/socrata-cookbooks/java.git
git clone https://github.com/bijugs/kafka-cookbook.git -b kafka_enhancement kafka
  • cd into kafka directory
1

vi attributes/zookeeper.rb

and change the default[:zookeeper][:data_dir] value if required

  • Under chef-solo directory: vi solo.rb and add the following details
1
2

file_cache_path "/where-ever/chef-solo"
cookbook_path "/where-ever/chef-repo/cookbooks"
  • Under chef-solo directory: vi kafka.json and include the following
1
2
3
4
5
6
7
8
9
10
11
12
13
14

{
  "java": {
      "jdk_version" : "7",
      "accept_license_agreement": true,
      "oracle": {
        "accept_oracle_download_terms": true
      }   
   },
   "run_list": [
     "recipe[java::oracle]",
     "recipe[kafka::zookeeper]",
     "recipe[kafka]"
   ]
}
  • Staying under the chef-solo directory run
1

sudo chef-solo -c solo.rb -j kafka.json

If you don’t have the time to go through these steps manually, check out a quick hack which automates these steps https://github.com/bijugs/simple-scripts/blob/master/install_kafka_using_chef_solo.sh

Comments