This article originally appeared in the January 2006 issue of System Administrator Magazine.
Humans are not as good as computers at applying changes consistently. If you maintain several computers manually, your machine configurations will drift apart over time. Besides the problems inherent in manual maintenance, there are several benefits to repeatable processes. Repeatable processes help you understand and document your machines' environments. If you can automate system changes, you can repeat your configuration process. You can develop disaster recovery systems that rely on reinstalling the OS and applying changes, rather than recovering your OS from backups. Automating your system configurations also means you work less over time.
Cfengine, written by Mark Burgess, is one of several tools available to automate system configurations and changes. Cfengine lets you assert certain conditions on a server, and over time the server will converge towards meeting those conditions. Depending on how you write your cfengine configuration files, the first time cfengine runs on your mail server it might install your sendmail RPM. The second time it might update /etc/mail from a CVS repository. The third time cfengine runs it might restart your mail daemon if it weren't running. At midnight cfengine might clear out all old files from "/tmp" and run your log rotation scripts.
Cfengine works by reading its configuration files, setting "classes" for the machine on which it runs, and then processing actions in an "actionsequence" of things to do. "Classes" define the conditions under which an action occurs. Your "actionsequence" define the types of actions--like deleting files or mounting NFS shares--that should be taken.
Every machine in your environment can use the same configuration files. These files will be processed differently depending on what classes have been set. For example, the same configuration file could set your Solaris and Red Hat Linux machines to have identical '/etc/resolv.conf' files, but might only install "atop" on your Red Hat Linux machines.
This article will introduce you to cfengine and hopefully help get you started. It cannot provide a thorough account of all cfengine's functions. For more information on cfengine, I recommend the Cfengine Reference Guide and other documentation at http://www.cfengine.org/documentation.phtml , the cfengine wiki at http://cfwiki.org , and the cfengine-help mailing list at http://lists.gnu.org/mailman/listinfo/help-cfengine .
All the files important to cfengine are stored in '/var/cfengine'.
bin/ Binaries inputs/ Configuration files modules/ Third-party cfengine code outputs/ Time-stamped messages and complaints ppkeys/ Public and private cfengine keys state/ Environmental data
Cfengine's binaries include 'cfagent', 'cfexecd', 'cfenvd', and 'cfservd'. cfagent is a command-line tool for processing cfengine configuration files. cfexecd is the cfengine daemon; it will regularly process your cfengine configuration files. cfenvd monitors system environmentals like the load on the system and converts this data into classes set in cfagent and cfexecd. cfenvd helps you say "run my 'decrease_load' shell script whenever the load is two times above its standard deviation." cfservd is the cfengine server, which among other things can serve arbitrary files to your cfengine clients.
Binaries in /var/cfengine/bin are typically symlinked to somewhere like /usr/sbin for ease of use.
Unless you explicitly specify a full path, cfengine assumes all configuration files are in '/var/cfengine/inputs'. Cfengine's two most important configuration files are '/var/cfengine/inputs/update.conf' and '/var/cfengine/inputs/cfagent.conf'. When cfengine runs, it will evaluate both 'update.conf' and 'cfagent.conf'.
'update.conf' is parsed first; it's designed to be your emergency configuration file. 'update.conf' typically sets your system time and copies the rest of your cfengine configuration files from your cfengine server. The system time has to be set for cfengine to authenticate with the cfengine server, similar to how Kerberos authentication works. The cfengine configuration files get copied to the client in 'update.conf' so that, if you hose the rest of your configuration, 'update.conf' can restore your configuration with a new (corrected) copy. 'update.conf' should get your cfengine client back to a "known state." For these reasons, your 'update.conf' should have no dependencies and be very small.
'cfagent.conf' is parsed second. It controls all the work you want done on any of your cfengine clients. Because environments usually have one set of configuration files, and machine-specific actions to take are controlled via classes, your configuration tends to be quite large. One of cfengine's keywords is 'import', which allows you to include other cfengine configuration files. Our site's 'cfagent.conf' includes many 'import' statements, so we can split up our configuration into manageable files.
Each cfengine configuration file is divided into actions, conditions, and declarations.
action:
class1:: # 'class1' is a condition
declaration
declaration
class2|class3:: # 'class2|class3' is a condition
declaration
Actions are always denoted by a single colon. Declaration syntax varies wildly depending on the kind of action. Hash marks denote comments. Cfengine has no rigid whitespace standard.
Conditions, or "compound classes," are always denoted by double colons. Conditions can get fairly complex, but at their simplest level a condition is just a class name. You can use the special class "any" to apply an action statement to all machines.
Conditions can be intersections of classes by using a pipe: "Hr00|Hr12" means execute when it's the midnight hour or noon, for example. Conditions can be unions by using a period: "Hr00.disk_space_needed" means execute when it's the midnight hour and the class "disk_space_needed" is defined. You can negate classes with a bang: "!Hr00" means do this unless it's the midnight hour. As you might hope (or fear), you can use parentheses with conditions: (!Hr00.(disk_space_needed|Tuesday)) means run the action statements if it's not midnight and either disk space is needed or it's Tuesday.
Cfengine defines many 'classes', also known as 'groups', when it runs. You can define your own classes in configuration files, too. On one Red Hat Enterprise Linux 3 machine, many classes are predefined. I will present them here by logical group:
10_1_1 10_1_1_1 ipv4_10 ipv4_10_1 ipv4_10_1_1 ipv4_10_1_1_1 192_168_1 192_168_1_1 ipv4_192 ipv4_192_168 ipv4_192_168_1 ipv4_192_168_1_1 net_iface_eth0 net_iface_eth1 net_iface_lo host1 host1_example_com
The machine has two NICs, one with IP 10.1.1.1 and another with IP 192.168.1.1. Both NICs, and loopback, are defined. The machine's hostname and FQDN are also set.
August Day20 Hr21 Hr21_Q1 Min10 Min10_15 Q1 Yr2005Classes related to the date are defined.
any cfengine_2 cfengine_2_1 cfengine_2_1_10 compiled_on_linux_gnu
The special class "any" is defined, as well as class names corresponding to the cfengine environment.
32_bit i686 linux linux_2_4_21_4_ELsmp linux_i686 linux_i686_2_4_21_4_ELsmp linux_i686_2_4_21_4_ELsmp__1_SMP_Fri_Oct_3_17_52_56_EDT_2003 redhat redhat_as redhat_as_3
Details about the kernel are defined, as well as the distribution.
cfenvd also helps cfengine define more classes, such as "entropy_www_in_low", based on measurements about the current environment. I have omitted these classes from the above list because they make the issue of learning about classes more complicated. You can see all the classes defined on your machine by running 'cfagent -pv' and looking for the line that begins with "Defined Classes."
To define your own classes, you can use the 'classes' action. The 'classes' action will define a new class if one of the classes you list is defined. For example, if you're going to use cfengine to install software packages on different types of machines, the most useful predefined class is the machine's hostname:
classes:
any::
mx_server = ( host1 host3 host11 )
mail_server = ( mx_server host12 )
cfengine_client = ( any )
These rules define a new class, 'mx_server', for any machine where the class 'host1', 'host3', or 'host11' is defined. 'mail_server' would then be defined for any 'mail_server' or for 'host12'. 'cfengine_client' would be defined when 'any' is defined, and as mentioned earlier 'any' is a special class that is always true.
Using conditions, you could rewrite the above section:
classes:
host1|host3|host11::
mx_server = ( any )
mx_server|host12::
mail_server = ( any )
any::
cfengine_client = ( any )
Either method of defining new classes is OK--the two above examples are equivalent--but I think the first is a little more readable.
Of all the actions in your cfengine configuration files, 'control' is the one action that will always get run. The 'control' block controls what cfengine should do, and how it should do it. In the 'control' block, there are several variables you can set. For example, you can how often when 'cfexecd' should run with the special 'schedule' control variable:
control:
any::
schedule = ( Min00_05 )
This schedule, which happens to be the default, means that cfexecd will run within the first five minutes of each hour. You can read about dozens of other specialized control variables in the Cfengine Reference Guide.
The most important control variable is the 'actionsequence', which defines what actions you want to run. With the exception of the special actions 'control', 'import', 'alerts', and 'classes', your actions will not run unless they are listed in the actionsequence.
Below are all the valid actions for your actionsequence, pulled verbatim from section 4.9.2 of the Cfengine Reference Guide:
mountall mountinfo checktimezone netconfig resolve unmount packages shellcommands editfiles addmounts directories links mailcheck mountall required tidy disable files copy processes module:name
These actions are really the most complicated part of cfengine, because each one uses a slightly different declaration syntax. Here are some of the actions that our site uses, along with short descriptions. See this article's long example for a cfengine configuration file that uses all the below actions.
links: This action creates soft links. It can create simple links, or it can create a "forest" of links. This declaration means "link everything in /usr/local/Sophos/bin into /usr/local/bin":
links:
sophos_server::
/usr/local/bin +> /usr/local/Sophos/bin
packages: Once you define the special variable 'DefaultPkgMgr' in your control block, so cfengine knows what package system you're using, you can query the system's installed packages. You can then use the special tag 'elsedefine', part of the 'packages' action, to define classes that you use later to take the appropriate action. To learn about these special tags I always turn to the Cfengine Reference Guide.
shellcommands: The full path of shell commands to run. Some commands can do strange things with standard input or output that can make cfengine hang, which is why some cfengine shellcommands are written like so:
"/path/to/binary < /dev/null > /dev/null 2>&1"
editfiles: cfengine has a large number of special commands for editing files, including "AppendIfNoSuchLine" and "ReplaceLineWith". For example, to ensure that no one has set the 'forceInstall' parameter for up2date in Red Hat Enterprise Linux 3, we use the following declaration:
editfiles:
redhat::
{ /etc/sysconfig/rhn/up2date
LocateLineMatching '^forceInstall=[1-9]'
ReplaceLineWith 'forceInstall=0'
DefineClasses 'up2date_forceinstall' }
Once again, you need to look in the Cfengine Reference Guide to learn about all the options for 'editfiles' declarations.
directories: This simple action creates directories.
tidy: Given a pattern of files to find and a minimum age, tidy deletes files.
disable: disable renames files. The renamed files get the suffix '.cfdisabled' appended to them. Disable can be used to ensure you have no '/etc/hosts.equiv' files lying around on your servers.
files: files checks file permissions, and can optionally maintain a checksum database.
copy: copy copies files from one directory to another, or from your cfengine server to cfengine clients. 'cfservd' must be running on your cfengine server, the server must allow the client to talk to it, your machines must have the same clock time, and the client and server must have each others' public keys in /var/cfengine/ppkeys.
processes: processes checks to see whether processes are running, and can define classes under certain conditions.
Now that you understand classes, the control block, actions, and actionsequence, please look at the attached example.
Cfengine can be deployed in two ways. One, you can put a 'schedule' line in your control section and run 'cfexecd' like any other daemon. Two, you can add a cron entry to call cfexecd. In our environment, we run cfexecd as a daemon, but we could just have easily added the following to cron:
0 * * * * /var/cfengine/bin/cfexecd -F
The '-F' means "don't fork and become a daemon."
Typically, people who deploy cfengine have a version-controlled repository where they check in configuration updates. The cfengine server checks out the most recent configurations. Cfengine clients then copy the configurations from the cfengine server. This version-controlled server-client configuration is a bit of a pain; our environment set it up with help from Luke Kanies's article, "Distributed Cfengine," which you can find at http://www.onlamp.com/pub/a/onlamp/2004/05/13/distributed_cfengine.html.
In our environment, we have a CVS-controlled repository of cfengine-related information. The 'config' directory from our CVS repository is checked out on our cfengine server as '/cfserver/config' . Our configuration has a line that says "if this machine is the cfengine server, run 'cd /cfserver/config; cvs -q upd -d'." When the cfengine server parses this configuration, it checks out the most recent files from CVS. Each client then copies these checked-out files from '/cfserver/config' as instructed in our configuration.
Please see Luke Kanies's article for more information on the cfengine server-client relationship.
Other than the difficulty required to set up a cfengine server/client environment, you may run into a few other problems with cfengine.
As hinted at in the example configuration file, cfengine parses all your imported files before it executes them. As cfengine reads your configuration files, it creates a list of potential class dependencies. If you import two files "A" and then "B", and file "B" defines a class, file "A" will never see that class defined. You must keep track of the order in which files are imported.
Also, your actionsequence order is really important. You will eventually get the temptation to re-order your actionsequence, because your "links" action really needs to use your "shellcommands" action. My advice is to pick an order and stick with it. Cfengine runs multiple passes through your actionsequence to resolve class dependencies, and if you really need to you can use a more advanced technique and run the same action several times with different defined classes. See the Cfengine Reference Guide, section 4.9.2, for details.
Cfengine defines its default classes when it first runs. If you add an IP address to a machine within a cfengine configuration file, cfengine's default classes will not include that IP address until the next time you run cfengine.
'cfagent', the command-line tool to run cfengine, has forty-six options. Here are the six I think are most useful.
'-K' means not to wait on any cfengine locks before running: run now.
'-v' means to be verbose.
'-q' means not to wait the default "splay time" but rather start running immediately. By default, cfengine waits a random amount of "splay time" before running to keep cfengine clients from overwhelming the cfengine server or other communal resources.
'-n' means to perform a dry run. Don't actually change the system.
'-D
For example, when our site is ready to fully initialize a machine, we run "cfagent -Kvq -DInit". That means ignore locks, tell me everything, run right now, and run with the "Init" class, which for us means "it's OK to do things that might break the system."
'-f
control: actionsequence = ( tidy ) tidy: /tmp recurse=inf age=0
I can then process this file by running 'cfagent -Kvq -f test.conf', to isolate my problem.
The other time I use '-f' is when I just want to run 'update.conf'. To update our cfengine server's copy of '/cfserver', the server only has to process 'update.conf'. I can run 'cfagent -Kq -f update.conf' on our server to make new versions of our configurations available for our cfengine clients and save it from having to run through the whole configuration in 'cfagent.conf'.
Our environment defines classes like 'mysql_server' and 'lsi_client' to install the appropriate binaries and utilities, and to copy out our configuration files. Cfengine is a timesaving maintenance tool once you have expended the effort to configure it initially. Cfengine's advantage and complexity comes from being able to automate virtually all parts of server configuration.
John Borwick is a SAGE Level III systems administrator at Wake Forest University, with a background in Computer Science and English. His goal is to make system administrators' lives easier by optimizing common processes.