Comments and questions about rolling out

Relevant classes: any Some edited comments from user discussions. Roll out question In about 4-6 weeks, we will install a 128-node dual-CPU Linux cluster, running RedHat Linux 7.2, and I'd like to install cfengine on all those hosts at the same time.... I guess that I am not the first, so I'd like to hear how others have done a similar thing. One think that I noticed, is that the cfengine install docs advises to first install cfenvd, and let it run for a week, before cfkey is run to generate the keys....this is not going to work for us, as we need to start using cfengine asap. Also, after this initial rollout, we will need to update the kernels on all hosts very early after the initial bootstrap. How do others update a large number of RedHat 7.2 machines with a new kernel? Is it OK to compile on one host, and then just tar up the /lib/modules and /boot directories, and re-run lilo on each host after untarring? Any pointers to additional info, or example cfengine config files and helper scripts would be appreciated. Replies cfenvd is not essential on linux, because it has /dev/random. Well, I don't know Redhat, but with my Debian machine I have the following lines in one of conf. files :- control: linux.!smp:: smp_up = ( up ) kernel_image = ( kernel-image-2.2.20+up ) linux.smp:: smp_up = ( smp ) kernel_image = ( kernel-image-2.2.20+smp ) shellcommands: # Upgrade the kernel if specifically requested linux.upgrade_kernel:: # Make sure grub is updated as well as update-grub # didn't get the kernel order right in an earlier # version "/usr/bin/apt-get -q -q -y install grub $(kernel_image)" Then I run cfagent -Dupgrade_kernel. I don't do this automatically because I don't like automatic kernel upgrades but there would be no problem removing the upgrade_kernel check then apt-get would be run each time (although, of course it doesn't do anything if the latest kernel is installed). The difference maybe that apt is setup to pickup some packages from a local spool on a central server. If I wanted to do automatic upgrades then I could just compile up a new kernel, copy it to the spool on the server and next cfagent run would do the upgrade. Does Redhat have something similar (I understood that somebody had ported apt to support rpm which maybe one way of doing it), Another example from my cf.standard :- # Sync APT archive first so any other routines use # an uptodate archive; only do this once a day linux.Hr06:: # Skip the update til we decide what to upgrade # against (potato to old, testing perhaps to # unstable currently) #"/usr/bin/apt-get -q -q update" linux:: # Install some more useful packages (binutils contains strings) # and upgrade others (want ssh with V2 support) # - make sure sysvinit is latest woody as it has a fix for # switching from S to 2 runlevels # - make sure cfengine is upgraded # - and make sure apt is newer version # - use netcat in place of telnet for testing network # - netpipe is a network performance tool "/usr/bin/apt-get -q -q -y install less emacs21 rsync man-db logrotate sysstat logcheck sash binutils ssh traceroute nictools-pci dnsutils ipgrab sysvinit cfengine apt netcat netpipe-tcp" # And remove any unnecessary ones # - don't encourage telnet use !! # - using emacs21 now "/usr/bin/apt-get -qq --purge remove telnet emacs20" Note the fact the apt-get update is commented out currently. I'm waiting for the the next Debian release to become stable before doing this (currently the machines are based upon woody which is Debian's next release still in testing). Once this becomes stable I'll uncomment this and that means that any new packages (normally only security packages) will be upgraded automatically. The other point here is that purge is rather uneccesary to leave in but apt-get is rather quick on even somewhat older hardware so its not a problem to levae it in; plus the fact that any new machines I setup from my custom install CD, which contains telnet, need to have that deleted. Another reply > In about 4-6 weeks, we will install a 128-node dual-CPU Linux cluster, > running RedHat Linux 7.2, and I'd like to install cfengine on all those > hosts at the same time.... As the other replies suggest, there's more than one way to do this... :) We do software installation on all our hosts (be they standalone or members of a cluster) via cfengine. All the software to be installed is packaged into a common format for all our OSes (we're currently using RPM for the package format) and placed on an internal anonymous ftp server. All the packages are then tracked in a (SQL) database which lists what software (pkg name & version) should be installed on what hosts. To do the actual installation, we then use a custom cfengine module which, when run via cfengine, will query the database to get a list of all the software that should be installed, compare it to what is currently installed, and then download and install or update (or remove) any packages as needed. The module will then define classes for any software which is installed, and these in turn can trigger actions (eg, for editfiles:, processes:, shellcommands:, etc) in the cfengine script to finalize or otherwise configure the software installation. For a little more detailed description of the process, see http://www.usenix.org/publications/library/proceedings/lisa2000/ressman.html Given that background, then, I'll say that we install cfengine on all our hosts using cfengine. Of course, that's a bit of a chicken-or-egg problem, as one needs to have a copy of cfengine installed in order to install cfengine... To get the process going, we install the initial copy (rpm) of cfengine via a postinstall script to the RedHat installation process. We install all our RedHat systems via network kickstart, so in the %post section of the kickstart profile, we create a boot time script which will be run on the initial boot after the kickstart completes. This boot script then simply uses "rpm -i" to install the cfengine rpm from our anonymous ftp server, using an ftp URL as the name of the cfengine rpm to be installed (eg, "rpm -i ftp://server/cfengine.rpm"). It then runs the freshly installed cfengine to install the rest of the software and configure the system as instructed by the actions in the cfengine script(s). Finally the boot script deletes itself and reboots the system (cfengine also installs patches (ie, updated RedHat rpms in the case of RedHat), so given that a large number of these get installed at OS install time, we do a final reboot for good measure). > One think that I noticed, is that the cfengine install docs advises to > first install cfenvd, and let it run for a week, before cfkey is run to > generate the keys....this is not going to work for us, as we need to > start using cfengine asap. As others noted, Linux has a /dev/random, so this is enough for cfkey to get the randomness it needs. We haven't updated our cfengine rpm for 2.0.0 yet (we currently still run 1.6.3 in production), but when we do, we'll probably have the rpm run cfkey on initial install in order to generate the keypair for the client. We would then either run cfagent with "trustkey=true" to grab the server's public key, or else simply distribute the server's public key in our cfengine rpm. > Also, after this initial rollout, we will need to update the kernels on > all hosts very early after the initial bootstrap. How do others update a > large number of RedHat 7.2 machines with a new kernel? Is it OK to > compile on one host, and then just tar up the /lib/modules and /boot > directories, and re-run lilo on each host after untarring? In theory, yes, that should be fine, esp. since all your nodes will be identical (I assume). You could distribute the new kernel via cfengine's copy: action, and then have that define a class which triggers an editfiles: action to update /etc/lilo.conf (or /boot/grub/grub.conf if you're using grub), which in turn defines another class to trigger a shellcommands: which reruns lilo, which finally, if desired, defines yet another class to trigger another shellcommand: to schedule a reboot (eg, via "shutdown -r" with some time specified or maybe using "reboot" with the "at" command). In our case, we use a scheme like the software one described above for installing a new kernel as a "patch". That is, we have a cfengine patch module which takes care of installing updated RedHat rpms on our RedHat hosts. We would simply have the patch module install either one of RedHat's kernel update rpms or one of our own custom kernel rpms. Through a chain of class definitions, we can then have cfengine take care of updating lilo.conf or grub.conf (note that with grub in RH 7.2 at least, RedHat's kernel rpms automatically update grub.conf as well as regenerate an "initrd" if needed), rerun lilo & schedule a reboot, if desired. At least in theory; we're not doing this in production yet as we need to update our module to take into account the different kernel "architectures" we're running (eg, some of our systems are running RedHat's "i686" kernel, others the "athlon" kernel, yet others the "smp" variety, etc). We do do kernel patching and occasional reboots under Solaris however w/o any problems. I don't anticipate any with Linux. > Any pointers to additional info, or example cfengine config files and > helper scripts would be appreciated. The paper referred to above has a URL with a copy of our cfengine modules. Unfortunately, the information and examples there are a bit out of date and incomplete; we should hopefully be posting more up-to-date info and examples RSN. In the meantime, hopefully the above will give you some ideas on one possible way to manage your cluster with cfengine. /pre> Back to documentation