Puppet ~ a beginners concept guide (Part 4) ~ Where is my Data

parts of the "Puppet ~ Beginner's Concept Guide" to read before ~
Part#1: intro to puppet, Part#2: intro to modules, and Part#3: modules much more.

Puppet
beginners concept guide (Part 4)

Where is my Data?

When I started my Puppet-ry, the examples I used to see had all configuration data buried inside the DSL code of manifests, people were trying to use inheritance to push down data. Then got to see a design pattern in puppet manifests keeping out separate parameters manifest for configuration variables. Then came along the External Data lookup via CSV files as a Puppet function. Then with enhancements in puppet and other modules came along more.

Below are few usable to fine ways utilizing separate data sources within your manifests,


Here, we will see usage styles of data for Puppet Manifests, Extlookup CSV, Hiera, Plug-in Facts and PuppetDB.

params-manifest:

It is the very basic way of separating out data from your functionality code, and the preferred way for in-future growing value-set type of data. It will keep it separate from the code since start. Once the requirement is at a level to have varied value to inferred based on environment/domain/fqdn/operatingsystem/[any-facter], it can be extracted to any preferred ways given below and just looked-up here. That would avoid changing the main (sub)module-code.
[ Gist-Set Externalize into Params Manifest: https://gist.github.com/3683955 ]
Say you are providing httpd::git sub-module for httpd module placing a template generated config file using params placed data...

```

File: httpd/manifests/git.pp
it includes the params submodule to access the data

File: httpd/templates/mynode.conf.erb

File: httpd/manifests/params.pp
it actually is just another submodule to only handle data

Use it: run_puppet.sh

```
_

extlookup-csv:

If you think your data would suit to a (key,value) CSV format being extracted to data files.Puppet need to be told the location for CSV files need to be looked up for key, and fetch the value assigned to it in that file.
Names given to these CSV files would matter to Puppet while looking up the values from all present CSV files. Puppet need to be given hierarchy order for these file-names to look for the key and the order could involve variable names.

For E.g. say you have a CSV by name of HOSTNAME, ENVIRONMENT and a common file, with hierarchy specified in respective order too. Then Puppet will first look for the queried Key in CSV by HOSTNAME, if not found looks up in ENVIRONMENT named file and after not finding it there goes looking into common file. If it doesn't find the key in any of those files, it returns the default value if specified in the 'extlookup(key, default_value)' method like this. If there is no default value also, Puppet will raise an exception for no value to return.

[ Gist-Set Externalize into Hierarchical CSV Files: https://gist.github.com/3684010 ]

It's the same example as for params with a flavor of ExtData. Here you'll notice a 'common.csv' external data file providing a default set of values. Then there is also a 'env_testnode.csv' file overriding the only required changed value. Now as in 'site.pp' file, precedence of 'env_%{environment}' file is higher than 'common', the 'httpd::git' would look-up all values first from 'env_testnode.csv' and if not found there would goto 'common.csv'. Hence would end-up overriding 'httpd_git_url' value from 'env_testnode.csv'.

```
```

extlookup() method used here is available as a Puppet Parser Function, you would read more in Part#5 Custom Puppet Plug-Ins on how to create your own functions
_

hiera-and-backends:

Hiera is a pluggable hierarchical data storage for Puppet. It was started to provide a better external data storage support than Ext-lookup feature with data formats other than CSV too.
This brings in the essence of ENC for data retrieval without having to write one.

Data look-up happens in a hierarchy provided by configuration with self scope resolution mechanism.

It enables Puppet to fetch data from varied external data sources using it's different backends (like local files, redis, http protocol) which can be added on to if needed.
The 'http' backend in turn enables support for data store from any service (couchdb, riak, web-app or so) to provide data.

File "hiera.yaml" from Gist below is an example of hiera configuration to be placed in puppet's configuration directory. The highlights of this configuration are ":backends:", backend source and ":hierarchy:". Multiple backend can be used at same time, their order of listing mark their order of look-up. Hierarchy configures the order for data look-up by scope.

Then depending on what backend you have added, you need to add their source/config to look-up data at.
Here we can see configuration for using local "yaml", "json" files. Look-up data from Redis server (it will set-up datasets for redis usage for current example) with authentication in place. And looking up data from any "http" service with hierarchy as the ":paths:" value.
You can even use GPG protected data as backend, but that is a bit messy to use.

Placing ".yaml" and ".json" from Gist at intended provider location.
The running "use_hiera.sh" would make you show the magic from this example on hiera.

```Gist

```
[Gist-Set Using Hiera with Masterless Puppet set-up: https://gist.github.com/abhishekkr/6133012 ]
_

plugin-facts:

Every system has its own set of information facter (http://projects.puppetlabs.com/projects/facter) by default made available to puppet. Puppet also enable DevOps people to set custom facter to be used in modules.
The power of these computed Facters is they can use full ruby-power to use local/remote plain/encrypted data over REST/Database/API/anyway available channel.

These require the power of Puppet Custom Plug-Ins (http://docs.puppetlabs.com/guides/custom_facts.html). The ruby file doing this would go at 'MODULE/lib/puppet/facter' and would get loaded by the 'pluginsync=true' in action.

Way to set a Facter in such Ruby code is just...

my_value = 'all ruby code to compure it'
Facter.add(mykey) do
  setcode do
    my_value
  end
end

.....all the rest of code there need to compute the value to be set, or even key-set.

[Gist-Set Externalize Data receiving as Facter: https://gist.github.com/3684968 ]

Same 'httpd::git' example revamped to use Custom Facter as 

```
```

There is also another way to provide a Facter in Puppet Catalog, that can be done by providing an Environment variable with capitalized Facter name pre-fixed by 'FACTER_' and the value which it's supposed to have.
For E.g. # FACTER_MYKEY=$my_value puppet apply --modulepath=$MODULEPATH -e "include httpd::git"
_

puppetdb:

It's a beautiful addition to Puppet component set. Something that have been missing for long and possibly the thing because of which I delayed this post by half year.
It enables the 'storeconfig' power without the Master, provides a support of trusted DB for infrastructure-related data needs and thus best suited of all.

To set-up 'puppetdb' on a node follow the PuppetLabs has a nice documentation.

To set-up a decent example for master-less puppet mode, follow the given steps

Place the 2 '.conf' and 1 '.yaml' file in Puppet's configuration directory.
The shell script would prepare the node with PuppetDB service for masterless puppet usage scenario.

Puppet config setting storeconfig to 'puppetdb' enables saving of exported resources to it. The 'reports' config their would push the puppet apply reports to the database.
PuppetDB config makes Puppet aware of the host and port to connect database at.
The facts setting on routes.yaml enable PuppetDB to be used in a masterless mode.

```

```

[Gist-Set Using PuppetDB with Masterless Puppet set-up: https://gist.github.com/abhishekkr/6114760 ]

Now running anything say like...
puppet apply -e 'package{"vim": }'
and beautiful to that 'export resources' would work like a charm using PuppetDB.
The puppet.conf accompanied will make reports dumped to PuppetDB as well.

_

There's a fine article on the same by PuppetLabs...

Puppet ~ a beginners concept guide (Part 3) ~ Modules much more

you might prefer first reading guide Part#1(intro to puppet), & Part#2(intro to modules)
the section after this Part#4(Where is my Data?) discussing how to handle configuration data

Puppet
beginners concept guide (Part 3)

Modules with More

here some time on the practices to prefer while writing most of your modules

[] HowTo Write Good Puppet Modules  
(so everyone could use it and you could use it everywhere)

• platform-agnostic
  With change in Operating System distro; module also might require difference in package names, configuration file locations, device port names, system commands and more.
  Obviously, it's not expected to test each and every module against each and every distro and get it full-proof for community usage. But what's expected is to use case $operatingsystem{...} statements for whatever distros you can and let the users get notified in case they gotta add something for their distro by fail(""), and might also contribute back..... like following
  case $operatingsystem {
    centos, redhat: {
      $libxml2_development = 'libxml2-devel'
    }
    ubntu, debian: {
      $libxml2_development = 'libxml2-dev'
    }
    default: {
      fail("Unrecognized libxml2 development header package name for your O.S. $operatingsystem")
    }
  }
  ~
• untangled puppet strings
  You are writing puppet modules. Good chance is you have a client or personal environment to manage for which you had a go at it.
  That means there gonna be your environment specific some client or personal code &/or configuration that is 'for your eyes only'. This will prohibit you from placing any of your module in Community.
  It's wrong on two main fronts. First, you'll end up using so much from OpenSource and give back nothing. Second, your modules will miss on the community update/comment action.
  So, untangle all your modules into atomic service level modules. Further modularize those modules into service puppet-ization requirement. That will be like sub-modules for install, configure, service and whatever more you can extract out. Now these sub-modules can be clubbed together to and we can move bottom-up gradually.
  Now you can just keep your private service modules to yourself, go ahead and use the community trusted and available modules for whatever you can..... try  making minor updates to those and also contribute these updates. Write the others that you don't find out in the wild and contribute those too for community to use, update and improve.
  ~
  
• no data in c~o~d~e
  Now when you are delivering 'configuration as a code', adapt the good coding practices applicable in this domain. One of those is keeping data separate than the code, as in no db-name, db-user-name, db-password, etc. details stored directly in the module's manifest intending to create the db-config file.
  There will be a detailed section later over different external data usage involving separate parameter manifest setting up values when included, extlookup loading values from CSVs, puppetDB, hiera data-store and custom facts file to load up key-values.
  ~
• puppet-lint
  To keep the modules adhere to dsl-syntactic correct and beautiful code writing practice. So the DSL and the community contributors, both find it easy to understand your manifests. It's suggested to have it added to rake default of your project to check all the manifests, ran before every repo-check-in.
  ~
• do-undo-redo
  It's suggested to have a undo-manifest ready for all the changes made by a module. It mainly comes in handy for infrastructures/situations where creating and destroying a node is not under your administrative tasks or consumes hell lot of time.
  Obviously, in case getting new node is easier..... that's the way to go instead of wasting time in undo-ing all the changes (and also relying on that).
  Those are just there for the dry-days when there is no 'cloud'.
  ~

[] More about Modules  (moreover.....)

Where to get new:http://forge.puppetlabs.com/ is the community-popular home for most of the Puppet Modules.

Where to contribute:
Can manage your public module at GitHub or similar online free repository like puppetlabs-kvm.
Then you can push your module to forge.puppetlabs.com.

Puppet ~ a beginners concept guide (Part 1)

Someone asked me where to start with Puppet learning. I pointed them at PuppetLabs online doc for Puppet, which is actually nice for a proper understanding. 
But for someone trying to start with Puppet, that documentation is a bit long to read similar to any book. 
I searched for few blogs, but didn't found any content (short but enough, fundamentals but usable) that I was looking for.
____________________________________________________


Puppet
beginners concept guide (Part 1)

[] What  it  is?  When  is  it  required?  (for all new guys, who came here while just browsing internet)
Puppet is an OpenSource automated configuration management framework (which means a tool that knows how to configure all machines to a deterministic state once you provide it the required set of manifests pulling the correct strings).
It's managed at enterprise level by an organization called PuppetLabs (http://puppetlabs.com/).

It is required#1 when you have a hell-lot of machines required to be configured in a similar form.
It is required#2 when you have a infrastructure requirement of dynamic scale-up and scale-down of machines with a pre-determined (or at least metadata-calculated) configuration.
It is required#3 to have a control over all set of configured machines, so a centralized (master-server or repo-based) change gets propagated to all automatically.
And more scenarios come up as you require it.

_____________________________________


[] Quickie.
Install Ruby, Rubygems on your machine where you aim to test it.

$ gem install puppet --no-ri --no-rdoc

Download installers @Windows  @MacOSX ::&:: Docs to installing.

Checking, if it's installed properly and acting good

Now, 'puppet --version' shall give you the version of installed puppet once succeed.
Executing 'facter' and you shall get a list of System Environment related major information.

Have a quick puppet run, this shall create a directory '/tmp/pup' if absent. Creates a file '/tmp/pup/et' with 'look at me' as its content.
{In case of trying out on platforms without '/tmp' location. Like for Windows, change '/tmp' with 'C:/' or so}

$ puppet apply -e "file{'/tmp/pup':
                               ensure => 'directory'}
                             file{ '/tmp/pup/et':
                               ensure => 'present',
                               content => 'look at me',
                               require => File['/tmp/pup']}
                           "
_____________________________________


[] Dumb  usage  structure.
Create huge manifest for your node with all resources & data mentioned in it. Then directly apply that manifest file instead of '-e "abc{.....xyz}"'.

Say if the example above is your entire huge configuration commandment for the node. Copy all that to a file say 'mynode.pp'.
Then apply it similarly like
$ puppet apply mynode.pp

_____________________________________


[] How  it  evolves?

Now, as any application had pluggable library components to be loaded and shared as and when required. Puppet too have a concept of modules. These modules can have manifests, files-serving and more.

Modules can be created in any design preference. Normally it works by having different modules per system component. To entertain different logical configuration states for any given system component (and also keeping it clean) further re-factoring can be done in the modules' manifest dividing it into different scopes.

Taking example of a module for 'apache httpd'. For a very basic library, you might wanna structure your module like

• a directory base for your module:  <MODULE_PATH>httpd/
• a directory in module to serve static files:   <MODULE_PATH>/httpd/files
• static configuration file for httpd:   <MODULE_PATH>/httpd/files/myhttpd.conf
  AccessFileName .acl
• directory to hold your manifests in module:   <MODULE_PATH>/httpd/manifests/
• a complete solution manifest:   <MODULE_PATH>/httpd/manifests/init.pp
  class httpd{
    include httpd::install
    include httpd::config
    include httpd::service
  }
• a manifest just installing httpd:    <MODULE_PATH>/httpd/manifests/install.pp
  class httpd::install {  package {'httpd': ensure => 'installed'}
  }
• a manifest just configuring httpd:    <MODULE_PATH>/httpd/manifests/config.pp
  class httpd::config{  file {'/etc/httpd/conf.d/httpd.conf':
      ensure => 'present',
      source => 'puppet:///modules/httpd/myhttpd.conf'
    }
  }
• a manifest just handling httpd service:  <MODULE_PATH>/httpd/manifests/service.pp
  class httpd::service{
    service{'httpd': ensure => 'running' }
  }

Now, using it

  $ puppet apply --modulepath=<MODULE_PATH>  -e "include httpd"
would install, custom-configure and start the httpd service.


  $ puppet apply --modulepath=<MODULE_PATH>  -e "include httpd::install"
would just install the httpd service.



________________________________________________________________

Part2: Road to Modules