Fabric management redesigned!

People who know me probably know that when it comes to fabric management i'm a fan of Quattor. Quattor is a great tool that can manage nodes from installation part (utilizing pxeboot and kickstart) till fine-tuning service features. While Quattor is a strong tool that could help administering hundreds or even thousands of nodes, it has some weak spots which i'd like to get rid of:

• Very steep learning curve
• Some operations may be quite time consuming (i.e. doing errata updates)
• Use of a custom programming language (PAN) which is usually unknown to even senior admins
• Inventory of the assets is based on what you describe and not on what exists

The above usually lead to misuse of the Quattor which in turn leads to unmaintained or difficult to maintain templates.
After a very long time of using Quattor and also evaluating other fabric management tools (i.e. puppet, chef) my conclusion is that while they can help admins to scale their infrastructure, they work pretty much against the way that administrators are used to. Admins are used to ssh and do their stuff while these management systems don't know how to interpret these changes so they simply skip them.
So lets move one step back and redesign it!
What if you had something monitoring your systems and had the ability to upload your local changes from the node to the central repository of your configuration?
I'm thinking of a solution which will have the following features:

• A simple PXE image to inventory your nodes (that way you only need to know the MAC address of your nodes before importing them to the system).
• PXE and kickstart (or equivalent) configuration to bootstrap nodes
• Ability to take-over a pre-installed node (i.e. cloud VM)
• Components (per feature) that will run on the nodes and will:

1. Identify current configuration
2. Change the configuration
3. Prepare a feedback commit if the admin wants to push the local changes on the node to the configuration system

• A simple tool/daemon that will run on the nodes as a wrapper for the components
• All the configuration will be based on a tagging system so that each node will have its own tag (to override configuration) and then a sorted list of tags to include (i.e. a tag called web-server and a tag called production would result a node that will be part of the production web-servers)
• The configuration server would store its data on a DB and timestamp every change. Changes should be easily reverted in case it is needed.
• The configuration server would provide API interface with read-only access for other tools to consume data (i.e. an dynamic infrastructure monitoring system) and read-write to update the data.

A list of the first components (i.e. before announce alpha release) that i'm thinking of is:

1. package managing
2. file generator
3. daemon handler
4. user management

And a demo could be the installation of a node from scratch with a couple of users and the ssh service up and running.

So in principal this is going to be yet another fabric management tool but with the addition of feedback from the nodes to the server. From the fabric management tools that i've used i found that Chef is closer to what i'm thinking of and basically because of the "ohai" util that feedbacks information to the server. I'll probably depend on this.

PS: This is just an idea for now so please add comments, implementation will start when time allows. The implementation will be done based on RedHat rules (thus compatible with RedHat Enterprise Linux and its clones) but should be modular enough to be extended to other distributions.

Developing the proper way

Recently i wanted to submit some patches to the Lustre Filesystem to allow the compilation of the kernel modules (patchless client) for Fedora 14 kernels. I initially posted my patches to the discuss list but i was pointed to the company that maintains the community based Lustre fork (whamcloud).

There i was amazed by the way that code is maintained as it uses all the current best practices that i could think of for developing. Thus i wanted to share them with my undefined readers.

Git repository:
First the code is publicly available on a git repo. People can clone the repository locally, do their modifications and they push them to Gerrit.

Gerrit:
Gerrit is a code reviewing web tool for git repositories which allows people to submit via "git push" patches. Patches are verified and reviewed. In Lustre's case, code verification is done by Jenkins while review is done by at least 2 code reviewers

Jenkins:
Jenkins monitors execution of repeated jobs. In Lustre's case it is used for automated builds of the current master branch + the committed patchset.

Finally when patches are approved they are merged to the master branch.