Once you have a base set of virtual machines that are ready for application provisioning, you need to install and configure the appropriate packages on each node.
How to do it…
1. Create a new project named storm-puppet with the following folder structure:
2. The entry point into the Puppet execution on the provisioned node is site.pp. Create it in the manifests folder:
node 'storm.nimbus' { $cluster = 'storm1' include storm::nimbus include storm::ui }
node /storm.supervisor[1-9]/ { $cluster = 'storm1'
include storm::supervisor }
node /storm.zookeeper[1-9]/ { include storm::zoo
}
3. Next, you need to define the storm module. A module exists in the modules folder and has its own manifests and template folder structure, much as with the structure found at the root level of the Puppet project. Within the storm module, create the required manifests (modules/storm/manifests), starting with the init.pp file:
class storm {
include storm::install include storm::config }
4. The installation of the Storm application is the same on each storm node; only the configurations are adjusted where required, via templating. Next create the install.pp file, which will download the required binaries and install them:
class storm::install {
package { "wget": ensure => latest }
# call fetch for each file exec { "wget_storm":
command => "/usr/bin/wget ${BASE_URL}${STORM_FILE}" } exec {"wget_zmq":
command => "/usr/bin/wget ${BASE_URL}${ZMQ_FILE}" } exec { "wget_jzmq":
command => "/usr/bin/wget ${BASE_URL}${JZMQ_FILE}" }
#call package for each file package { "libzmq0":
provider => dpkg, ensure => installed, source => "${ZMQ_FILE}", require => Exec['wget_zmq']
}
#call package for each file package { "libjzmq":
provider => dpkg, ensure => installed, source => "${JZMQ_FILE}",
require => [Exec['wget_jzmq'],Package['libzmq0']]
}
#call package for each file package { "storm":
provider => dpkg, ensure => installed,
The install manifest here assumes the existence of package, Debian packages, for Ubuntu. These were built using scripts and can be tweaked based on your requirements. The binaries and creation scripts can be found at https://bitbucket.org/qanderson/
storm-deb-packaging.
The installation consists of the following packages:
Storm
ZeroMQ: http://www.zeromq.org/
Java-ZeroMQ
5. The configuration of each node is done through the template-based generation of the configuration files. In the storm manifests, create config.pp:
class storm::config { require storm::install include storm::params
file { '/etc/storm/storm.yaml':
require => Package['storm'],
content => template('storm/storm.yaml.erb'), owner => 'root',
group => 'root', mode => '0644' }
file { '/etc/default/storm':
require => Package['storm'],
content => template('storm/default.erb'), owner => 'root',
group => 'root', mode => '0644' }
}
6. All the storm parameters are defined using Hiera, with the Hiera configuration invoked from params.pp in the storm manifests:
class storm::params { #_ STORM DEFAULTS _#
$java_library_path = hiera_array('java_library_path', ['/usr/local/lib', '/opt/local/lib', '/usr/lib']) }
Due to the sheer number of properties, the file has been concatenated.
For the complete file, please refer to the Git repository at https://
bitbucket.org/qanderson/storm-puppet/src.
7. Each class of node is then specified; here we will specify the nimbus class:
class storm::nimbus { require storm::install include storm::config include storm::params
# Install nimbus /etc/default storm::service { 'nimbus':
start => 'yes',
jvm_memory => $storm::params::nimbus_mem }
}
Specify the supervisor class:
class storm::supervisor { require storm::install include storm::config include storm::params
# Install supervisor /etc/default storm::service { 'supervisor':
start => 'yes',
jvm_memory => $storm::params::supervisor_mem }
}
Specify the ui class:
class storm::ui {
require storm::install include storm::config include storm::params # Install ui /etc/default storm::service { 'ui':
start => 'yes',
jvm_memory => $storm::params::ui_mem }
}
And finally, specify the zoo class (for a zookeeper node):
8. Once all the files have been created, initialize the Git repository and push it to bitbucket.org.
9. In order to actually run the provisioning, navigate to the vagrant-storm-cluster folder and run the following command:
vagrant up
10. If you would like to ssh into any of the nodes, simply specify the following command:
vagrant ssh nimbus
Replace nimbus with your required node name.
How it works…
There are various patterns that can be applied when using Puppet. The simplest one is using a distributed model, whereby nodes provision themselves as opposed to a centralized model using Puppet Master. In the distributed model, updating server configuration simply requires that you update your provisioning manifests and push them to your central Git repository. The various nodes will then pull and apply this configuration. This can either be achieved through cron jobs, triggers, or through the use of a Continuous Delivery tool such as Jenkins, Bamboo, or Go. Provisioning in the development environment is explicitly invoked by Vagrant through the following command:
config.vm.provision :shell, :inline => "puppet apply /tmp/storm-puppet/
manifests/site.pp --verbose --modulepath=/tmp/storm-puppet/modules/
--debug"
The manifest is then applied declaratively by the Puppet. Puppet is declarative, in that each language element specifies the desired state together with methods for getting there.
This means that, when the system is already in the required state, that particular provisioning step will be skipped, together with the adverse effects of duplicate provisioning.
The storm-puppet project is therefore cloned onto the node and then the manifest is applied locally. Each node only applies provisioning for itself, based on the hostname specified in the site.pp manifest, for example:
node 'storm.nimbus' { $cluster = 'storm1' include storm::nimbus include storm::ui }
In this case, the nimbus node will include the Hiera configurations for cluster1, and the installation for the nimbus and ui nodes will be performed. Any combination of classes can be included in the node definition, thus allowing the complete environment to be succinctly defined.