Testing Chaos with Automated Configuration Management solutions

No noise making.

But let's be real, think of the count of community contributed (or mysterious closed-and-sold 3rd Party) services, frameworks, library and modules put to use for managing your ultra-cool self-healing self-reliant scalable Infrastructure requirements. Now with so many cogs collaborating in the infra-machine, a check on their collaboration seems rather mandatory like any other integration test for your in-house managed service. After all that was key idea behind having automated configuration management itself.

Now the utilities like Puppet/Chef have been out there accepted and used by dev & ops folks for quite some time now.

But the issue with the initially seen amateur testing styles is it evolved from the non-matching frame of 'Product' oriented unit/integration/performance testing. 'Product' oriented testing focus more on what happens inside the coded logic and less on how user gets affected by product.
Most of the initial tools released for testing logic developed in Chef/Puppet were RSpec/Cucumber inspired Product testing pieces. Now for the major part of installing a package, restarting a service or pushing artifacts these tests are almost non-required as the main functionality for per-say installing package_abc is already tested inside the framework being used.
So coding to "ask" to install package_abc and testing if it has been asked seems futile.

That's the shift. The logic developed for Infrastructure acts as a glue to all other applications created in house and 3rd party. Here in Infrastructure feature development there is more to test for the effect it has on the it's users (software/hardware) and less on internal changes (dependencies and dynamic content). Now the stuff in parentheses here means a lot more than seems... let's get into detail of it.

Real usability of Testing is based on keeping sanctity of WHAT needs to be tested WHERE.

Software/Hardware services that collaborate with the help of Automated Infrastructure logic needs major focus of testing. These services can be varying from the

• in-house 'Product', that is the central component you are developing
• 3rd Party services it collaborates with,
• external services it utilizes for what it doesn't host,
• operating system that it supports and Ops-knows what not.

Internal changes mainly revolve around

• Resources/Dependencies getting called in right order and grouped for specific state.
• It also relates to correct generation/purging of dynamic content, that content can itself range as
• non-corrupt configuration files generated of a template
• format of sent configuration data from one Infra-component to another for reflected changes
• dynamically creating/destroying service instances in case of auto-scalable infrastructure

One can decide HOW, on ease and efficiency basis.

Unit Tests work for the major portion of 'Internal Changes' mentioned before using chefspec, rspec-chef, rspec-puppet like libraries are good enough. They can very well test the dependency order and grouping management as well as the different data effect on non-corrupt configuration generation from templates.

Integration Tests in this perspective are a of a bit interesting and evolutionary nature. Here we have to ensure the "glue" functionality we talked about for Software/Hardware service is working properly. These will confirm that every type of required machine role/state can be achieved flawlessly, call them 'State Generation Test'. They also need to confirm the 'Reflected Changes Test' across Infra-component as mentioned in Internal changes.

Now utilities like test-kitchen/docker in collaboration with vagrant, docker, etc. help placing them in your Continuous Integration pipeline. This would even help in testing same service across multiple linux distros if that's the plan to support.

Library 'ServerSpec' is also a little nifty piece to write quick final state check scripts.

Then final set of Integration Testing is implemented in form of Monitoring on your all managed/affecting Infrastructure components. This is the final and ever-running Integration Test.

Performance Tests, yes even they are required for it. Tools like ChaosMonkey enable you to enable your Infra to be self-healing and auto-scalable. Should be load-test noticing dynamic containers count and behavior if auto-scalability is a desired functionality too.

derailed_cuke ~ now test any website using cucumber as an independent testing utility

GitHub Link: https://github.com/abhishekkr/abk-labs.cucumber

[] What is "Cucumber"?
+ http://cukes.info/

Cucumber lets software development teams define the behavior of any application in Business Readable DSL as plain text.
Hence serves as documentation, automated tests and development aid ~ All In One.

[] What is "derailed_cuke"?
+ Project Repo @GitHub


[] How To get "derailed_cuke"? + you could either clone the repo or download compressed project in following ways ~

Clone Repository:
#git clone git://github.com/abhishekkr/abk-labs.cucumber.git

Download As Zip: https://github.com/abhishekkr/abk-labs.cucumber/zipball/master

[] Pre-requisites for "derailed_cuke"?
+ Ruby, RubyGems & Bundler Gem installed


[] How to use "derailed_cuke"?
+ The downloaded project would have a directory "abk-labs.cucumber" with a directory "derailed_cuke" inside it.
~~~~~~~~~~
Step#1 Now all you need to do is open a console, change current working directory to "derailed_cukes".|
Step#2 Run "#bundle install" to install all required gems.
|
Step#3 If your ruby binary file is accessible by any name other than 'ruby' {i.e. say you are jruby and thus access it using 'jruby'}; then edit the file "Fcuke.rb" and replace the value of variable "RUBY_BIN" in line number 2 with the name you access your Ruby Binary.
|
Step#4 Now simple execute "Fcuke.rb" and by default it will check for two (already present for testing) features checking Google.com and Twitter.com
|
Step#5 To add custom web-application behavioral tests ~ user will need to write the custom Cucumber Features and respective Step Definitions {as in general scenario for using Cucumber}. For reference present feature and step definitions can be seen & used.
~~~~~~~~~~

[] Where to learn cucumber?
+ http://wiki.github.com/cucumber/cucumber/tutorials-and-related-blog-posts
In current release of "derailed_cuke", user need to know writing Cuke Features and Step Definitions.

………………………………………………………
[] Planned Goals for the Project involves
+ auto-generation of basic tests by analysis of application,
 then major task left would be to add uncovered custom tests
 and tweak the generated tests (if required).

………………………………………………………

DummyNet - HowTo { an open-source tool to tweak network latency and bandwidth }

Any service allowing to  tweak Network Latency and Bandwidth as per desire
for testing application  performance at different network latency scenarios.

Tools/Technology Used:
DummyNet
{Home}URL: http://info.iet.unipi.it/~luigi/dummynet/

Background:
Normally the difference which comes in  development and  deployment
environment of Web Applications, is of bandwidth and latency.
To test the applications in actual scenario, one needs to tweak the latency as
per deployment scene and then use it.
There are few paid VE Technology based services like "Shunra" for this. But
we required a free, open-source application, if possible for windows.
DummyNet, it's an old Italian university project started for BSD systems,
recently ported for Windows also. It helps in reducing latency of NIC to
desired level.

Execution Method:
[] Install NDIS Driver
1.  Open the configuration panel for the network card in use
  {right click on the icon on the SYSTRAY, or go to 'Control
Panel' > 'Network' to select}
2.  Click on 'Properties' > 'Install' > 'Service' > 'Add'
3.  Click on 'Driver Disk' and select 'netipfw.inf' in the
folder it has been extracted to.
4.  Select 'ipfw+dummynet' which should be the only service
visible.
5.  Click 'Accept' on the warnings for the installation of an
unknown driver.

Create a BAT-File for your Application to be run under test bandwidth &
latency with following content
--------------------------------------File Content Starts  from Next line

@echo off
@set CYGWIN=nodosfilewarning
@ipfw -q flush
@ipfw -q pipe flush
@echo #################
@echo ## Setting up ##
@echo #################
ipfw pipe 3 config delay 1000ms bw 500Kbit/s mask all
ipfw add pipe 3 ip from any to any
ipfw pipe show

@echo ""
@echo "Network Tweaking Done, Start Testing."
@echo ""
@echo "Press Enter when testing is done, to restore original Network Settings."
pause

@echo #################
@echo ## Cleaning up ##
@echo #################
ipfw -q flush
ipfw -q pipe flush
pause

--------------------------------------File Content Ends at Previous line
Here, 1000ms is latency set and can be changed to desired value
500 Kbps is bandwidth set and can be changed to desired value
'delay x' and 'bw x'; both can be used separately also as per need
either place a command running your application to be tested in place of
'your_command_running_your_app'; or simple remove that line 
and when the command gets paused at the first pause, run your appl ication
manually.
 

Installation:
Follow the wlak-through video on Youtube at :
http://www.youtube.com/watch?v=jP-DrxTMXDc

Then, to test if it gets installed properly as a network services. It has a testme.bat file showing different tests, just run it and
check statistics.

I tested it on Windows XP, Vista, 7... and it worked great.