availability: July 2010
![]() |
In the last few months, a movement has begun to take shape. It's a movement of people who think it's time for change in the IT industry - time to stop wasting money, time to start delivering great software, and building systems that scale and last. This movement is being called Devops. But what is Devops? Where did it come from? And what can it achieve? |
| Guest post by : Stephen Nelson-Smith @lordcope a Technical Manager and Devop based in Hampshire, UK and author of Agile Sysadmin | |
Let's face it - where we are right now sucks. In the IT industry, or perhaps to be more specific, in the software industry, particularly in the web-enabled sphere, there's a tacit assumption that projects will run late, and when they're delivered (if they’re ever delivered), they will underperform, and not deliver well against investment. It's a wonder any of us have a job at all!
Let’s have a look at some of the common problems we experience in the software world today.
Once an application is delivered, the business tends to be tremendously afraid of change. The suspicion is that the software itself, and the platform upon which it sits, is somewhat brittle and vulnerable. Bureaucratic change-management systems are put in place, and it takes a painfully long time to introduce new features, or fix problems with the application.
Another symptom of the malaise is the concept of the ‘risky deployment’. This is the situation in which no-one is really terribly confident that the software will work in the live environment. Will the code behave as expected? Will it cope with the load? Often we don’t have the answers to these questions - we just push it out at a quiet time, and watch to see if it falls over.
A common situation we experience is one in which a problem manifests itself once the site is live. These problems are typically picked up by systems administrators, or helpdesk / client services people. After investigation (although to be fair frequently without) the problem is reported to the developers. The developers take a cursory look, and retort: “It works on my machine”.
However, this is frequently meaningless. I’ve worked in places where the code is developed on Windows 2000 workstations, and tested with Tomcat on Windows, and then deployed on Redhat Linux, with software load balancers, different Java versions and different Tomcat versions. This is not to mention the entirely different properties files on developer workstations compared to live machines.
On most projects I’ve worked on, the project team is split into developers, testers, release managers and sysadmins working in separate silos. From a process perspective this is dreadfully wasteful. It can also lead to a 'lob it over the wall' philosophy - problems are passed between business analysts, developers, QA specialists and sysadmins. Furthermore, we see a replication of this silo structure within the teams - it’s not uncommon to see dedicated database and network people in the same system team, alongside sysadmins. Often the larger silos aren't in the same office, the same city, or in sometimes not even in the same country. The result is an ‘us and them’ mentality - groups of people who are simultaneously suspicious of and afraid of each other.
The Devops movement is bold enough to believe that there’s a better way - a way of building teams, and building software that can solve these problems.
The Devops movement is built around a group of people who believe that the application of a combination of appropriate technology and attitude can revolutionize the world of software development and delivery.
The demographic seems to be experienced, talented 30-something sysadmin coders with a clear understanding that writing software is about making money and shipping product. More importantly, these people understand the key point - we’re all on the same side! All of us - developers, testers, managers, DBAs, network technicians, and sysadmins - are all trying to achieve the same thing: the delivery of great quality, reliable software that delivers business benefit to those who commissioned it.
Breaking this down a bit - one of the key statements there is ‘sysadmin coders’. There’s been a certain amount of popularity recently around the concept of ‘polyglot programmers’. These are developers who know multiple languages, and multiple approaches to programming, such that when appropriate they can move swiftly between object-oriented Ruby and a functional language such as Erlang or OCaml. This is simply an example of a larger observation - there is no one IT skill that is more useful or more powerful than another. To solve problems well you need all the skills. When you build teams around people who can be developers, testers, and sysadmins, you build remarkable teams.
Beyond this multi-disciplinary approach, the Devops movement is attempting to encourage the development of communication skills, understanding of the domain in which the software is being written, and, crucially, a sensitivity and passion for the underlying business, and for ensuring it succeeds.
It might seem obvious, but communication really is one of the biggest keys. Devops are bridge-builders. The fact of the matter is that building quality software is really hard - it's error prone, it's risky, it's unpredictable. In the last ten years or so, software developers have started to realise this. The development of the Agile Manifesto, and the consequent improvements in process and peopleware, together with some technology advances around testing have had a massive impact.
However, there's still a gaping hole in what Chris Read calls 'the last mile'. The thing is, 'dev complete' is a long long way from 'live, in production, stable, making money'.
The problem is that it's typically the sysadmins who are responsible for getting the software out live - but often they don't know, trust, like, or even work in the same city as the developers. How on earth are we expected to deliver good software that way?
So, the Devops movement is characterized by people with a multidisciplinary skill set - people who are comfortable with infrastructure and configuration, but also happy to roll up their sleeves, write tests, debug, and ship features. These are people who making connections, because they can - because they have feet in multiple camps, they can be ambassadors, peace makers, facilitators and communicators. And the point of the movement is to identify these, currently rare, people and encourage them, compare ideas, and start to identify, train, recruit and popularize this way of doing IT.
This has a tremendous impact on the business. Suddenly the technical team starts trying to pull together as one. An 'all hands on deck' mentality emerges, with all technical people feeling empowered, and capable of helping in all areas. The traditionally problematic areas of deployment and maintenance once live become tractable - and the key battlegrounds of developers ('the sysadmin built an unreliable platform') versus sysadmins ('the developers wrote unreliable code') begins to transform into a cross-disciplinary approach to maximizing reliability in all areas. This, of course, has a positive effect on the bottom line - better reliability and availability, happier clients, faster time to market, and more time to focus the team's energy on core business rather than wasteful administration and firefighting.
Some people have been a little suspicious - the movement really looks a lot like just a bunch of European sysadmins, many of whom know each other. Is this just an elitist club? Some kind of rebranding exercise?
Well, it's true - Devops as a movement is characterized by a number of sysadmins, many of whom know each other - but then this isn't a surprise. We've already explored the problems that we're trying to solve - these are problems centred around the deployment and management of applications that are, from a simplistic position, completed work. Developers have already realized that change was long overdue, and they’ve started taking steps to aleviate the pain. The result of this is that the problem has moved, and it’s in the systems area that the pain is now being felt. Consequently it's only natural that the majority of the people who have realized it's time to change things operate in the systems area.
Yes, there are some senior, experience, and talented people in the movement - but again, that's no surprise - the kind of sysadmin who is going to recognize the problem, care about fixing it, and have the combination of skills in terms of programming, systems and infrastructure and personal and management skills will by the very definition be a highly skilled and capable person. That’s not to be confused with elitist - the movement is friendly, welcoming, and open.
To draw a comparison, when the Agile Manifesto was drawn up, that was a gathering of well known, experienced, figurehead developers. It could easily have been perceived as exclusive, and elitist, but that didn't invalidate the ideas, its value, or prevent the movement from growing and thriving
The fact that the movement has grown up in Northern Europe is simply the coincidence of a concentration of good Agile groups, and capable sysadmin / developers gathering together in a Belgian city for the inaugural DevopsDays unconference.
So if you’re feeling wary of the movement, or worried that you might not be accepted or fit in, don’t worry at all - if you’ve got a desire to change the world, hop on board.
In my view becoming a Devop requires a certain state of mind. It's an attitude which says: I'm going to make a difference, I'm going to cooperate and communicate, and I'm going to understand that in the business of delivering great software, we're all in it together. If you're a sysadmin, this means spending time with the developers. Get to know your way around the code base. Contribute to it. If you don't know how to program, get started - there are good tutorials available for Python and Ruby, which are both excellent, powerful general purpose languages that will serve you well. Start thinking about systems management as a programming task - use Puppet or Chef to manage your machines, and start thinking about how to test your infrastructure.
If you're a developer, go and make friends with your sysadmins. Don't view them as lower life forms, or as people to lob problems to. Go and join in - pair with them. If they're using Puppet or Chef, get involved - start contributing to their codebase. If you're an experienced programmer, especially if you have an understanding of test driven development, consider mentoring your sysadmins - get them confident in programming, and encourage them to start to take ownership of the code base. Start to understand the roles within your (soon to be removed) silos. Try to understand what skills the QA team brings, and how they work - see if there's a way for you to help. Try to cross-pollenate - sysadmins, DBAs, network people, business analysts - you're all on he same team.
If you're a manager, advertise positions as 'Devop' not just ‘Developer’ or 'SysAdmin'. Make it clear you're interested in hiring people with cross-functional skills, and that you'll provide training to help this. Start implementing configuration management systems, get sysadmin code into version control, get the systems team involved in continuous integration and deployment activities. If you need to hire in consultancy to help get you on the right track, there are people available who do this kind of thing every day.
The Devops movement is still in its infancy, but it's gathering pace - there are conferences, mailing lists, irc channels, blogs and people to get to know. I'm convinced this isn't just a fad, and we're on the brink of a revolution in the software industry - a paradigm shift in which developers and sysadmins start to work together, to train each other, and ultimately to blur the boundary - welcome to the world of the Devop.
Stephen Nelson-Smith is a Technical Manager and Devop based in Hampshire, UK. His company, Atalanta Systems, specializes in applying a blend of automation technologies and agile and lean work practices to enable clients to concentrate on their core business, be more effective, and make more money. You can follow him on Twitter, and read his blog at Agile Sysadmin
If this sounds like the kind of thing you already do, or like the kind of thing you want to do, there are plenty of resources available to help.
The following blogs are written from a Devops perspective, and offer high quality, technically rich content. The Build Doctor, Patrick Debois (This One), Agile Sysadmin, RI, Bitfield , Planet Infra, Unix Daemon
The is a low volume Google group called Agile System Administration. Most of the founders of the movement can be found there.
IRC is a superb resource - the following channels on freenode cover the kinds of tools and approcahes that are central to the movement, and many of the current members are the movement can be found in the channels.
##infra-talk , #puppet , #chef
Most of the inaugural Devops people can be found on Twitter. You can also search for the hash tag #Devops. Here are a selection of interesting people to follow:
Stephen Nelson-Smith , Julian Simpson , Matthias Marschall , Patrick Debois , Mike Poutney , Paul Nasrat , Dean Wilson , R.I.Pienaar , Lindsay Holmwood , Chris Read , Branden Faulls , James Turnbull , John Willis, Kris Buytaert , Gildas Lenadan, John Arundel , Andrew Shafer, John Allspaw, Michael Cote ,Damon Edwards , Israel Gat
And finally, here are some of the tools that you're likely to find in the box of a Devop.
Hudson , Cucumber , Cucumber Nagios , Puppet , mcollective , cobbler , Chef , Flapjack , Visage , Collectd
Thanks to the devopsdays conference, the idea of devops seems to live on. While talking with other people about it, I realize that it is difficult to frame it within the current IT landscape. At lot of the ideas are coming from different kinds of emerging technologies (T) and process management (P) approaches.
For me the two most important observations are:
So where can you look for devops ideas ? As you can see on the map , these interactions are all over the place.

You can probably find a lot more or think that some have nothing to do with devops as such. I really like to hear your thoughts on this list. The focus of this post is not by the 100% exactness or completeness. Think of it as work in process waiting for your feedback ;-)
As a freelancer I spent quite some time updating and sending my CV to different parties. Most recruiting companies ask me for a MS Word version of my CV without any markup. I understand why, they put on their own logo and copy and paste the relevant parts for the company they propose me to. To me this feels rather awkward. In these web oriented times, I would expect enterprises to check candidate backgrounds online. And a flat, unformatted list of job experience, is not what they expect online. Sites like Linkedin and VisualCV try to make a difference here, but they are still text based, very similar to the dull paper documents.
You can debate on the usefulness of a visual resume for jobhunters, I think that potential resume evaluators go through a resume in different passes. The right typography will already helps a lot in this process. Also the visual has to be easily understood by the recruiter, not only by the candidate himself.
The most brilliant examples I've seen, come from graphical or art designers. They are showcasing fonts, styles or other visuals. The problem is, I'm not into design jobs, and creating a resume that visual, will not send the right message.
Another trend is using visual slides to convey who you are and what you stand for. These examples work well when presenting in person, but without audio they don't work that well. Also printing these will result in a condensed overview.
Below is a list of the most beautiful examples I found. I'd love to make one myself, who knows.
Well the next thing is probably create my own... Another idea might be to create an application to visualize f.i. LinkedIn profiles with it using the hresume format. If any visual designers are interested in this project, just let me know.

At xpdays Benelux 2009, I attended an interesting session called 'Developing a Sense of Smells' by Kevin RutherFord and Lindsay McEwan.
The exercise we did went as follows: suppose you are asked to do some work on code you never saw before. How would you assess this, go about estimating the effort and explaining that effort to justify the price/number of days. The first round resulted in terms like 'look for design patterns', 'readibility of the code', ...
Then they explained code smells patterns: f.i. a greedy function: one function that does way to much making it difficult to change, hidden secret: internal knowledge such as the real interpretations of a CSV file data format that can not be deduced by looking at the code. The code smells made it easier to express the problems.
The devops in me,thinks that this can be translated to the sysadmin world. Here are some of the 'smells' I've come up.
The sysadmin uses the system as his toy playground, doesn't clean up.
One server that does every function
Things that are needed to keep the system alive when failed
Mainly indicate not much maintenance is done any more
Loadbalancers, Cluster software, Dependencies
This is just the beginning, so if you have your own ideas/names, just leave a comment.
Here are the slides of my presentations at Xpdays Benelux 2009. The presentation is on how we could bring sysadmins and developers closer together by using continuous integration in both worlds.
I would very much like to thank Gildas Le Nadan for helping with the first version of it at Xpdays France 2009. Also I want to thank the organizers for this great conference! And for giving me the chance to speak about this rather niche subject.
Now that we outlined the programming languages for automating shell scripting, virtual machine creation ,network provisioning
and os installation and beyond, I bet you as a devops are eager start writing your infrastructure code.
After some time chances are that you will end up with lots and lots of scripts executing in sequence. And then when you change something in a script the whole sequence will fail and you'll have a hard time looking for what caused the problem. A better approach for writing your code is to practice Test Driven Development.
In short before writing any code, you first write a test for the code you are writing. Then you run your tests and see that the new test fails.(RED). It is only then that you start writing your code or change your existing code (REFACTOR). When you think the code is done, you run your tests again and see them succeed (GREEN). And then you continue to use this cycle to grow your code. It is important that you chance your code in small increments. For more infor see Test first guidelines.
So how can this help you as a sysadmin? Isn't this more of developer thing? And the answers is a big NO:
Yes, but won't this slow me down? Well this is exactly the reaction most developers have to this process. But to me the benefits should be clear, do you rather go for the fingers crossed go live or the verified state. You will have to find the right balance between writing all tests and writing no tests.
If you are working on a new project that tries to get something running as quickly as possible as a one shot, you'll probably be under pressure to deliver. It will take some time to reach the skills to get the automation and the tests in your fingers. Still a good way of convincing management is by explaining them that these tests not only for the project phase but can be used during the whole maintenance period. This definitely increased the ROI of writing these tests.
Part of a test there usually is a setup and a tear-down part. The idea is to create a state under which you start performing your tests. For applications f.i. this would be to put the right stuff in the database, set the right variables. So how does this translate to machines? Most of the virtualization solutions allow you to do easily take snapshots of both your memory state (savestate) and your disks. So you can easily recreate a certain point in time to start your tests by cloning your systems and running some scripts to change the state (f.i. filling a disk, killing a process) . Another approach could be to take snapshots of your disks using filesystems like ZFS, LVM that easily let you take snapshots.
This also helps during the coding of your infrastructure:
In the following example I will describe the setup of a webserver which serves static pages. Note that there is no standard development involved here. It's all about pre-packaged software.
In this step you would define the hardware of your virtual machine: number of CPU's, memory, network interfaces, mac addresses, disks, ... So how do we test this? In the days of physical hardware, the way to verify that systems had all the hardware in it, we booted up a CD and verified using commands that the system contained the correct number of disks.
To avoid writing your own boot CD , you could use something similar to sysrescueCD. It has a feature called autorun that allows you to boot up the disk and execute scripts that are either on a floppy, disk, NFS or Samba share or HTTP server. If you mount this as a virtual CD in your virtual machine, it can boot up the virtual machine and execute test to test if the definition was according to what was specified.
Next step is provisioning the network information for your virtual machine. In order to test this, we can easily use the same boot CD approach to verify this via scripts using dhclient, dig, tftpclient. Some might argue that this test is better done from within the OS. Off course testing is when the OS is installed is a more complete tests. Doing this test separate from the installed OS, lets you better distinguish where the error occurs. Is it a problem of the OS driver that there is no IP address or is is a problem with the definition of the DHCP/DNS.
This is usually done by defining a kickstart/jumpstart template. It would contain the disks partitioning, network configuration, the minimal packages and patches, a set of minimal services enabled (ssh, puppet), selinux enabled , . As you see there is a lot more that can be tested here:
Up until now these tests are simple checks , and probably most people will do similar things in their monitoring. I've talked about testing being more then monitoring. Aside from these simple checks you can complementing your monitoring to run scenarios. Also destructive tests like failing a disk or bringing down an interface are probably not the best thing to do in production monitoring ;-)
So aren't we re-testing the packages here? The problem here is packages are often tested in isolation from each other and they can only test a limited number of setups. That means it still makes sense to repeat some tests so see if YOUR combination of things actually works.
We now have a tested minimal OS running with a configuration mgt system active. Next in line it applying recipes. While discussing these with a number of people , I heard a lot that you don't need tests because these tools work in a declarative mode: if something doesn't work then it's either an error in your recipe or an error in the configuration management software.
I personally disagree, the argument is similar to why we test even if individual packages are tested: you can write a beautiful recipe to install a webserver but maybe the firewall is preventing you access, maybe your SELinux is blocking things. Or you install a bad apache config file so that it uses the wrong directory to serve.
Also you can add scenario testing here :
Programming languages have developed a lot of frameworks over time to help in writing tests. When coding your infrastructure in these languages check what's available. Still there is no test library that is specific to systems testing. The closest are test frameworks for HTTP testing.
Currently this is an emerging field. There are already a lot of examples in the wild. As we adopt more the idea of testable infrastructure, these frameworks will emerge. The most notable is cucumber-nagios written by Lindsay Holmwood. It brings http testing closer to monitoring and into the sysadmin world. Now you can reuse your tests written in cucumber in your monitoring environment.
After a discussion at devopsdays 09 Lindsay is starting on a similar thing that allows to integrated ssh scripting in this framework, or what he calls Behavior driven infrastructure through cucumber And recently he announced on the agile system administration mailing list that he's joining forces with Adam Jacob of Opscode. So definitely more to come!
Up until now, I've described the options to automate shell scripting, virtual machine creation and network provisioning. So now we can actually get started with automating the installation of the Operating System itself. This not surprisingly is were most sysadmins spent most of their time.
The good news is that we are slowly going a way from the custom scripting toward a reusable and shareable language. Very similar to programming design patterns. Over the years the methods have evolved towards configuration management, which is obviously a good thing.
This also helps in leaving the hero culture sysadmins have because they are the guys who now how to manage their systems.
The first way of automating the OS installation came from using Kickstart or Jumpstart. In essence it is a config file that defines the information necessary to do an installation. It typically contains:
When booting the kernel of an OS, there usually is a way to specify this file, allowing automated installation. Over time this file was put on a floppy, then on a CDROM, USB Drive. To automatically start the installation the install media needed to be modified to change the default kernel boot options. The syntax of these files is typically different per OS.
To create your first file, you would do a manual installation of the OS, and at the end it would create a file corresponding to the manual choices you made. After that, the file could be altered to include additional post-installation scripts or extra packages.
Automating the creation of this has never really been important, I was either using an Editor or using a GUI to generate them. Interestly I found Snake as an example to scripting the creation of the template.
So in other to automate the installation of software, sysadmin would encourage developers to create packages for their install, so that the installation could be easily automated.
An example of a Solaris Jumpstart:
# install_type MUST be first install_type initial_install # start with the minimal required number of packages cluster SUNWCXall cluster SUNWCapache delete cluster SUNWCpcmc delete cluster SUNWCpcmcx delete cluster SUNWCthai delete cluster SUNWClp delete cluster SUNWCnis delete cluster SUNWCppp delete # format the entire disk for Solaris fdisk all solaris all # define how the disk is partitioned partitioning explicit filesys rootdisk.s0 6144 / filesys rootdisk.s1 1024 swap filesys rootdisk.s7 free /state/partition1 # install systems as standalone system_type standalone # specify patches to install patch 119281-06 nfs 172.16.64.194:/export/patches # specify packages to install package SPROcc add nfs 172.16.64.194:/export/packages
Over time frameworks emerged to manage all these packages installation and managing :
As System recipes and configuration management beautifully describes, managing systems is more then just managing and installing packages : sysadmins would have tons of custom scripts and HOWTO's to describe how to do additional things f.i. installing a complete Webserver according to company standards. All of this knowledge was usually put in specific custom scripts, without a good way of sharing this, . What we actually need is a formalized description of all the things that need to be present and configured on the machines in your setup.
Enter the world configuration management tools. These tools provide a ways of describing the recipes in a language that makes abstraction of the actual system its running on. F.i. adding a user or adding a package is abstracted in the language.
Examples are:
Puppet seems to have most of the market for now, but the Chef guys are catching up. Also a lot of the system management tools have puppet integration:
Example puppet recipe for ensuring the permission of a sudoers file
# /etc/puppet/manifests/classes/sudo.pp
class sudo {
file { "/etc/sudoers":
owner => "root",
group => "root",
mode => 440,
}
}
Example for mysql node installation
class mysql-server {
$password = "insert_password_here"
package { "MySQL-client": ensure => installed }
package { "MySQL-server": ensure => installed }
package { "MySQL-shared": ensure => installed }
exec { "Set MySQL server root password":
subscribe => [ Package["MySQL-server"], Package["MySQL-client"], Package["MySQL-shared"] ],
refreshonly => true,
unless => "mysqladmin -uroot -p$password status",
path => "/bin:/usr/bin",
command => "mysqladmin -uroot password $password",
}
}
What's fascinating is the new set of tools that starts to emerge here and how it really starts to integrate in real programming languages such as ruby.
For managing configuration files Augeas is doing a really nice job: it parses configuration files in their native formats and transforms them into a tree. Configuration changes are made by manipulating this tree and saving it back into native config files.
Now that sysadmins are spending more time with their configuration management, they slim down their initial kickstart installations by only using a minimal base install of the OS in their kickstart + a configuration management tool. And when this minimal install is done, they continue applying their recipes to the installation.
This minimal install is sometimes called a base install , an installation with the minimal set of packages installed. Some OS vendors are actively working on providing what is called a JeOS which stands for Just enough Operating System. This is an image that contains an already installed minimal install. They are also useful for creating appliances. Some example are:
If you're installing a complete stack f.i. Oracle Application Server, Database Server, changes are that the initial package install does nothing. What you need is to create your own instances, pools and so on. What's important is that some of these include their own automation language. It is essential that they have a way of doing every GUI action from the commandline. Otherwise you end up writing scripts for managing their config files. This is a bad thing because this might change overtime or even worse impossible because they store things in binary format.
Examples of good API's are:
# create an cluster asadmin create-cluster --user admin --passwordfile adminpassword.txt --host hostname -port 4848 cluster1 # create instance 1 asadmin create-instance --user admin --passwordfile adminpassword.txt --host hostname -port 4848 --cluster cluster1 --nodeagent nodeagent1 --systemproperties "JMX_SYSTEM_CONNECTOR_PORT=8687:IIOP_LISTENER_PORT=3330:IIOP_SSL_LISTENER_PORT=4440:IIOP_SSL_ MUTUALAUTH_PORT=5550:HTTP_LISTENER_PORT=1110:HTTP_SSL_LISTENER_PORT=2220" instance1 # create instance 2 asadmin create-instance --user admin --passwordfile adminpassword.txt --host hostname -port 4848 --cluster cluster1 --nodeagent nodeagent1 --systemproperties "JMX_SYSTEM_CONNECTOR_PORT=8688:IIOP_LISTENER_PORT=3331:IIOP_SSL_LISTENER_PORT=4441:IIOP_SSL_ MUTUALAUTH_PORT=5551:HTTP_LISTENER_PORT=1111:HTTP_SSL_LISTENER_PORT=2221" instance2 # start the cluster asadmin start-cluster --user admin --passwordfile adminpassword.txt --host hostname --port 4848 cluster1
Some tools also provide a 'silent install' feature. It will record the installation into a text file that you can later replay to have the same result. Using some sed/awk scripts you can probably automate some of this.
Up until now, I've described the options to automate shell scripting and virtual machine creation. The next step is to prepare the network for the virtual machine to make it boot with the right network settings. In a lot of companies this is often a manual process but we want to automate it.
In order to boot a linux machine, one relies on:
On linux the de-facto standard DHCP Servers is the ISC DHCPD. It relies on configuration files in a specific format. Most of the automation of this, were homebrew scripts that changes these config files. It's a pity there is no scriptable API for managing these files.
The next runner up is the DNSMASQ which dispite it's name can also act as DHCP server. It too relies on configuration files, but I found dnsmasq mysql that extends dnsmasq with a mysql backend. So we could issues SQL statements to control it?
Other things I found were:
Overall I found no good out of the box way of automating subnet creation, client registration, templates. So if you know any let me know. Maybe be this is the reason why the big virtual vendors provide their own DHCP server in their product, configurable with their own API.
Example of Virtualbox API:
VBoxManage dhcpserver add|modify --netname <network_name>
--ifname <hostonly_if_name>
[--ip <p_address>
--netmask <network_mask>
--lowerip <lower_ip>
--upperip <upper_ip>]
[--enable | --disable]
VBoxManage dhcpserver remove --netname <network_name>
--ifname <hostonly_if_name>
The next step is automating the management of DNS files. For DNS servers under Linux we have more choice then with DHCP server. We have ISC Bind , DJBDNS and dnsMasq. All of them rely on configuration files and for Bind and dnsMasq there have been efforts to integrate it with a database backend f.i. mysql-bind . Another effort I found is [dnsadmin](http://freshmeat.net/projects/dnsadmin/)(outdated)for managing djdns/tinydns files.
DNS by itself allows the use of Dynamic Updates (see RFC 2136). This already allows us to change the attributes of a single host using commands such as nsupdate , but does not provide a way to create/update/delete new zones or other settings.
Another project calls dnspython (outdated) http://www.dnspython.org/ provides both high and low level access to DNS. The high level classes perform queries for data of a given name, type, and class, and return an answer set. The low level classes allow direct manipulation of DNS zones, messages, names, and records.
The most promising is the use of Power DNS http://www.powerdns.com which seem to have full scripting capability. I haven't tried if personally but it's the best alternative I found.
For managing a single machine it is important than changes to both your DHCP, DNS, TFTP and PXE boot are in sync. It doesn't come as a surprise that a lot of global machine management solution use the base blocks and add their own sauce. It's a pity, that the management of DHCP and DNS management are not extended in the tools itself but stay in the global management tool. Another thing is that things like the creation of new DNS zones of DHCP templates are not possible because the underlying tools don't have it. (or maybe nobody normally automates that ;-)
Examples of global solutions :
And because most these have a commandline or other API's , one could easily script the updates to DNS of DHCP settings for individual machines.
Amazon started on an API for DNS/DHCP Options for automating their Private Cloud:
domainname="cobblertest.be"
mac_address="aa:aa:bb:bb:cc:01"
name="puppetclient"
ip="192.168.3.150"
kickstart="puppet.ks"
profile="centos53-latest"
distro="centos53-i386"
# Set the distribution of the machine
# The distribution was previously imported by importing an installation DVD
# Cobbler will detect the possible kernels to boot
# And this will also link the TFTP and PXE file necessary to Boot
cobbler profile add --name=#{profile} --distro=centos53-i386
# Add the new machine with an IP and Mac address
# By the IP Address it knows in which reverse DNS
cobbler system add --name=#{name} --ip=#{ip} --mac=#{mac_address}"
# Set the DNS domain of the machine (this determines in which zone file it is create)
cobbler system edit --name=#{name} --profile=#{profile} --dns-name=puppet1.#{domainname}"
# For linux machines, the kernels are provided with an option
# ks=kickstart so that it will start the kickstart installation
cobbler system edit --name=#{name} --kickstart=/var/lib/cobbler/kickstarts/#{kickstart}
cobbler system edit --name=#{name} --name-servers-search='#{domainname}'
# This finally commits all the changes
cobbler sync
I wanted to automate the creation of an MS-DOS floppy under MacOSX. The usual way is very similar to Linux. In this script I avoided using sudo to mount the filesystem and took advantage of the hdiutils to run everything under normal user credentials.
# creating an empty file
dd if=/dev/zero bs=512 count=2880 of=msdos-floppy.img
# associate the file with a device without mounting it
device=`hdid -nomount msdos-floppy.img`
# formatting disk with msdos format
newfs_msdos $device
# detach the file from the associated device
hdiutil detach $device -force
# mounting the image file
device=`hdid msdos-floppy.img|cut -d ' ' -f 1`
# calculate the mountpoint by checking the mount table
path=`mount |grep -w '$device' | cut -d ' ' -f 3- | cut -d '(' -f 1`
# copying file to the mountpoint $path
cp file $path/
# unmount the image
hdiutil detach $device -forceend
In the old days, getting a new machine could take days. It required ordering of hardware and putting everything together. Now in the these virtual/cloud days, creating new machines is a breeze. While a lot of effort is spent on automating the installation of the machine OS and its application, I see that the provisioning of a virtual machine is often still done by the GUI. So why not automate that step too.
Depending on the virtualization platform you choose, different options exist ranging from GUI (HTTP Posts), Command Lines, SOAP, XML-RPC based or language bindings. What follows is a list of ways I found. Again my experience is that most programming oriented XML-RPC, SOPA or Language Bindings are a subset of the commandline interface.
In all cases, the commandline API is updated first and then the rest follows. Again, this strengthens me to say that when automating the creation within Ruby you should actually write a wrapper around the commandline API. Because the target audience is sysadmins, this makes a lot of sense. At the end I provide an example using VirtualBox CommandLine API wrapped in Ruby Language.
Vmware is one of the most used virtualization tools. These are some of the ways it can be scripted:
Command line - Using vmrun to Control Virtual Machines http://www.vmware.com/pdf/vix180_vmrun_command.pdf
GUI Vmware Studio - http://blog.organicelement.com/2008/09/17/vmware-studio-automated-production-of-virtual-machines/
LibVirt tries to be virtual machine neutral: it has an abstraction for: Xen hypervisor on Linux and Solaris hosts, QEMU emulator, KVM Linux hypervisor, LXC Linux container system, OpenVZ Linux container system, User Mode Linux paravirtualized kernel. It also has experimental support for Virtualbox and Vmware ESX and GSX hypervisors but I found these unstable.
At lot of enterprise management tools use libvirt to build on:
Sun take another approach with their zones. They provide an excellent command API to manage their zones. http://www.sun.com/bigadmin/content/zones/. Beautifully for scripting.
Even non virtual machines can be controlled: you can use wakeonlan, ipmitool or some management interface to power up/down the machines. Cobbler has this kind of powermanagement intergface https://fedorahosted.org/cobbler/wiki/PowerManagement to manage bullpap, wti, apc_snmp, ether-wake, ipmilan, drac, ipmitool, ilo, rsa , lpar, bladecenter.
Virtualbox has one of the most excellent command Line available. And they generate their SOAP API from the same source!
require 'soap/wsdlDriver' require 'pp' WSDL_URL="vboxwebService.wsdl" soap = SOAP::WSDLDriverFactory.new(WSDL_URL).create_rpc_driver soap.wiredump_dev=STDERR #soap = SOAP::WSDLDriverFactory.new(WSDL_URL).create_rpc_driver("vboxService", "vboxServicePort") #pp soap.methods vbox=soap.IWebsessionManager_logon({:username => '', :password => ''}) puts "Sessions"+vbox.returnval version=soap.IVirtualBox_getVersion({:_this => vbox.returnval}) puts version.returnval disks=soap.IVirtualBox_getHardDisks({:_this => vbox.returnval}) diskids=disks.returnval diskids.each do |diskid| type=soap.IHardDisk_getType({:_this => diskid }) size=soap.IHardDisk_getLogicalSize({:_this => diskid }) location=soap.IMedium_getLocation({:_this => diskid }) puts diskid+"-"+type.returnval+"-"+size.returnval+location.returnval end
1 2 require "rubygems" 3 require "open4" 4 require "pp" 5 require "systr/commands" 6 7 #if disk has not been specified with fullname then is stores it in the default VBOX Location 8 9 def wait_for_state_vmachine(vmname, state, options={ }) 10 defaults={ :timeout => 1000 , :pollrate => 5 } 11 options=defaults.merge(options) 12 13 begin 14 Timeout::timeout(options[:timeout]) do 15 while true do 16 begin 17 puts "polling state" 18 actualstate=state_vmachine(vmname) 19 if actualstate==state 20 return true 21 else 22 puts "Currentstate: "+actualstate 23 sleep options[:pollrate] 24 end 25 end 26 end 27 end 28 rescue Timeout::Error 29 raise 'timeout waiting for machine to reach state #{state}' 30 end 31 32 end 33 34 def state_vmachine(vmname) 35 result=Command.execute("VBoxManage showvminfo #{vmname} --machinereadable|grep VMState=|cut -d '=' -f 2|cut -d '"+'"'+"' -f 2") 36 state=result.stdout.to_s 37 return state 38 end 39 40 41 def remove_vmachine(vmname,options={}) 42 43 defaults={ :disk => vmname } 44 45 options=defaults.merge(options) 46 47 if (state_vmachine(vmname)!="poweroff") 48 Command.comment("Can't remove a running machine") 49 throw "machine is still running" 50 end 51 52 Command.execute("VBoxManage modifyvm #{vmname} -sataport1 none") 53 #Command.execute("VBoxManage snapshot #{vmname} discardcurrent --all") 54 55 Command.execute("VBoxManage closemedium disk #{vmname}.vdi") 56 57 Command.execute("VBoxManage unregistervm #{vmname} --delete") 58 59 60 #first stop machine 61 62 #then unregister 63 64 #then remove it? 65 end 66 67 def exists_vmachine(vmname) 68 return Command.test("VBoxManage showvminfo #{vmname}") 69 end 70 71 def floppy_kickstart_vmachine (vmname) 72 #http://www.win.tue.nl/~aeb/linux/kbd/scancodes-1.html 73 Command.execute("VBoxManage controlvm #{vmname} keyboardputscancode 26 17 31 16 2d 39 25 1f 0d 21 26 18 19 19 15 1c") 74 75 end 76 77 def add_floppy_vmachine(vmname,floppyfile) 78 Command.execute("VBoxManage modifyvm #{vmname} -floppy #{floppyfile}") 79 end 80 81 def create_vmachine(vmname,options={}) 82 83 defaults={:ostype => 'RedHat', :memory => '384', :disk => vmname, :net => 'pxenet'} 84 85 86 options=defaults.merge(options) 87 88 Command.execute("VBoxManage createvm -name #{vmname} -ostype #{options[:ostype]} -register") 89 # Trying hostonly 90 # Command.execute("VBoxManage modifyvm #{vmname} -nic1 nat -nic2 intnet -intnet2 #{options[:net]}") 91 # http://www.virtualbox.org/manual/UserManual.html#networkingdetails 92 93 94 Command.execute("VBoxManage modifyvm #{vmname} #{options[:network]}") 95 96 #TODO: VBoxManage modifyvm puppet1 -macaddress1 aaaabbbbcc01 97 98 99 Command.execute("VBoxManage modifyvm #{vmname} -memory #{options[:memory]}") 100 Command.execute("VBoxManage modifyvm #{vmname} -sata on -sataport1 #{options[:disk]}.vdi -sataportcount 1") 101 102 unless options[:dvd].nil? 103 Command.execute("VBoxManage modifyvm #{vmname} -dvd #{options[:dvd]}") 104 else 105 Command.execute("VBoxManage modifyvm #{vmname} -dvd none") 106 end 107 108 unless options[:floppy].nil? 109 Command.execute("VBoxManage modifyvm #{vmname} -floppy #{options[:floppy]}") 110 else 111 Command.execute("VBoxManage modifyvm #{vmname} -floppy empty") 112 113 end 114 115 Command.execute("VBoxManage modifyvm #{vmname} --vram 32") 116 Command.execute("VBoxManage modifyvm #{vmname} --bioslogodisplaytime 0") 117 # Command.execute("VBoxManage modifyvm #{vmname} --bioslogoimagepath path-to-256-bmp") 118 119 Command.execute("VBoxManage modifyvm #{vmname} --acpi on") 120 Command.execute("VBoxManage modifyvm #{vmname} --ioapic on") 121 122 Command.execute("VBoxManage modifyvm #{vmname} -boot1 disk") 123 Command.execute("VBoxManage modifyvm #{vmname} -boot2 dvd") 124 Command.execute("VBoxManage modifyvm #{vmname} -boot3 net") 125 126 #suppress interactive messages 127 Command.execute("VBoxManage setextradata global 'GUI/RegistrationData' 'triesLeft=0'") 128 Command.execute("VBoxManage setextradata global 'GUI/UpdateDate' '1 d, 2009-09-20'") 129 Command.execute("VBoxManage setextradata global 'GUI/SuppressMessages' ',confirmInputCapture,remindAboutAutoCapture'") 130 131 end 132 133 def remove_dhcp 134 result=Command.execute("VBoxManage list dhcpservers| grep NetworkName:|cut -d '-' -f 2").stdout 135 result.each do |interface| 136 Command.execute("VBoxManage dhcpserver remove --ifname #{interface}") 137 end 138 139 # ERROR: Assertion failed at '/Users/vbox/tinderbox/3.0-mac-rel/src/VBox/Main/VirtualBoxImpl.cpp' (1706) in virtual nsresult VirtualBox::SetExtraData(const PRUnichar*, const PRUnichar*). 140 # Unexpected exception 'N3xml12EIPRTFailureE' (Runtime error: -250 (Unresolved (unknown) device i/o error.)). 141 # Please contact the product vendor! 142 # Details: code NS_ERROR_FAILURE (0x80004005), component VirtualBox, interface IVirtualBox, callee nsISupports 143 # Context: "EnableStaticIpConfig(Bstr(pIp), Bstr(pNetmask))" at line 267 of file VBoxManageHostonly.cpp 144 145 Command.execute("VBoxManage hostonlyif ipconfig vboxnet0 --ip 192.168.10.1 --netmask 255.255.255.0") 146 147 end 148 149 def start_vmachine(vmname) 150 Command.comment("starting virtual machine #{vmname}") 151 if ENV['SYSTR_HEADLESS'].nil? 152 Command.execute("VBoxManage startvm #{vmname}") 153 else 154 system("VBoxHeadless -s #{vmname} --vrdp off &") 155 sleep 4 156 end 157 end 158 159 def stop_vmachine(vmname) 160 Command.comment("stopping virtual machine #{vmname}") 161 Command.execute("VBoxManage controlvm #{vmname} poweroff") 162 end 163 164 def remove_snapshot_vmachine(vmname,snapname) 165 Command.execute("VBoxManage snapshot #{vmname} discard #{snapname}") 166 167 end 168 169 def create_snapshot_vmachine(vmname,snapname) 170 Command.execute("VBoxManage snapshot #{vmname} take #{snapname}") 171 end 172 173 def ssh_enable_vmachine(vmname, options={}) 174 defaults={:localport => 2222 , :remoteport => 22} 175 options=defaults.merge(options) 176 177 Command.execute("VBoxManage setextradata #{vmname} 'VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/HostPort' #{options[:localport]}") 178 Command.execute("VBoxManage setextradata #{vmname} 'VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort' #{options[:remoteport]}") 179 Command.execute("VBoxManage setextradata #{vmname} 'VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/Protocol' TCP") 180 return options[:port] 181 end 182 183 184