How To Build RPM Packages In A Chroot Environment Using mach

Note: this article was tested using CentOS 5.1, but it should also work with other RPM-based distros (e.g. Fedora, SuSE)

A description of the problem

OK, so you want to build a binary RPM package for deployment on your servers. You have a .spec file or .src.rpm that you got from one of the many repositories such as freshrpms.net or dag.wieers.com, or that you wrote yourself.

Why not just build it using rpmbuild?

There are several problems you may come across.

1. Given a spec file, rpmbuild won't download the source tarball and/or patches. You have to fetch those yourself into the SOURCES directory.
2. rpmbuild will abort if any build-time dependencies are missing, forcing you to stop what you're doing, and go and build and install those packages too.
3. When your package configures itself, it may auto-detect libraries which are available on your build system, but which are not going to be available on the target system. For example, if openldap-devel is present then openldap may be linked into your binaries, but if the RPM doesn't declare openldap as a dependency, then it will fail to run on the target system. This is an insiduous problem, which I call "the curse of autoconf".
4. You can only build packages for the same type of system as your build machine (e.g. CentOS 4 binaries on a CentOS 4 build system)

If you want an example of how bad the problem is, try building the package perl-SOAP-Lite from its spec file. You will quickly find yourself in dependency hell, with 16 other packages needing to be installed or built, all in the correct order.

The solution: mach

Mach (for "make a chroot") solves all these problems. It installs a minimal subset of your target distribution under /var/lib/mach/roots/distro. When it's time to build a package, given a spec file it turns it into a .src.rpm by downloading the necessary sources automatically (or you can just give it a .src.rpm in the first place).

Mach then switches into this cleanroom environment using chroot. It builds the package jailed into this environment, pulling in any dependencies automatically by yum. The finished binary package is left in a local yum repository, in case it is needed by any future package builds. Finally, the environment is cleaned up to its pristine state by removing all surplus packages.

As a bonus, building in a chroot environment ensures that all your "BuildRequires:" dependencies are correctly declared in the spec file, because the package won't build without them.

Installing mach

I will walk through an installation from scratch on a CentOS 5.1 system, but setting it up to build packages for CentOS 4.

Installing mach itself is a one-time bootstrapping operation. Once you've done this, all your future RPM packages can be built within mach itself.

There are some binary packages available on the download site, but I prefer to build it myself from source:

$ wget http://thomas.apestaart.org/download/mach/mach-0.9.3.tar.bz2
$ sudo rpmbuild -tb mach-0.9.3.tar.bz2
$ sudo yum install createrepo
$ sudo rpm -Uvh /usr/src/redhat/RPMS/i386/mach-0.9.3-1.i386.rpm

Aside: if you want to avoid being root when running rpmbuild then you can put the following in ~/.rpmmacros

%_topdir %(echo "$HOME")/redhat

and create the necessary subdirectories:

mkdir ~/redhat ~/redhat/{BUILD,RPMS,SOURCES,SPECS,SRPMS}

However, you still have to be root to install the package of course.

After installation, check that you have the documentation in /usr/share/doc/mach-0.9.3/README and that there is a group called mach in /etc/group.

Optional: point to a local yum mirror

For deterministic results, you might want to choose a local yum mirror explicitly rather than relying on automatic redirects. This is done in /etc/mach/location. For example, if you are in the UK, you might want to change

centos = 'http://mirror.centos.org/centos'

to

centos = 'http://www.mirrorservice.org/sites/mirror.centos.org'

Setting up users

Any user who is going to use the mach environment needs to be a member of the supplementary group mach. You can do this by adding their username to the end of the line in /etc/group:

mach:x:102:user1,user2,user3...

Do this for your own non-root account, then logout and login again.

NOTE: because mach relies on shared group permissions, your umask value must be 0002 (allowing group write on all files and directories you create). Check it like this, and change it if necessary:

$ umask
0002             # this is OK
$ umask
0022             # this is wrong
$ umask 0002
$ umask
0002             # now it's correct

Your umask is reset at the start of each session, so if it doesn't default to 0002 then you'll need to update your .profile or change it manually each time you login.

Finally, each user needs to create a file ~/.machrc like the following:

config['installer'] = 'yum'
config['files'] = {
  '/etc/hosts': '''
127.0.0.1 builder.example.com builder localhost
''',
  '/etc/resolv.conf': '''
nameserver 1.1.1.1
nameserver 2.2.2.2
search example.com
'''
}
config['defaultroot'] = 'centos-4-i386-extras'
config['macros'] = {
    'vendor':   'My Company Name',
}

(Or: set these globally in /etc/mach.conf)

This gives some parameters which will be needed within the chroot environment. In the /etc/hosts line, the first hostname is the one which you want to appear in the package as the build host, and the second one should be the actual output of 'hostname' on this machine (if different)

Check out the files under /etc/mach/dist.d/ for the various choices of chroot environments. The above example builds packages for a CentOS 4 system which has the 'extras' repository available to it, in addition to the basic ones.

See README for more detailed info.

Preparing the build environment

Make sure you have lots of free disk space available under /var (at least 2GB) then proceed as follows:

$ umask

0002 # Confirm this is 0002 !!

$ mach setup build

Preparing root
Installing package set 'minimal' .........................
Installing package set 'base' ............................
... snip lots ...
Retrying installing package set 'base' .............
Installing package set 'build' ...........................
Making snapshot ...

This is slow the first time you run it, because it downloads (using yum) an entire set of packages for the distro you want to build under. However these packages are cached. So in future, even if you blast away the entire chroot environment, it won't download all the packages again when rebuilding it.

Speaking of which, should you ever wish to completely toast the build environment and recreate it from scratch, here's how:

$ mach clean
$ mach setup build

Your new root is at /var/lib/mach/roots/distro/ and you can browse it like this:

$ mach chroot

Entering /var/lib/mach/roots/centos-4-i386-extras, type exit to leave.
ERROR: ld.so: object '/usr/lib/libselinux-mach.so' from
LD_PRELOAD cannot be preloaded: ignored.

-bash-3.1# ls /

bin dev home media opt root selinux sys usr
boot etc lib mnt proc sbin srv tmp var

-bash-3.1# exit

logout
$

Building packages

This is where the fun starts. You simply give one or more .spec files or .src.rpm files to mach build:

$ mach build foo.spec bar.spec ...

or

$ mach rebuild foo.src.rpm bar.src.rpm ...

Mach does all the rest for you automatically, and dumps the shiny finished packages under /var/tmp/mach/distro/.

A worked example: perl-SOAP-Lite

Let's try out the "hard" RPM package I mentioned before.

$ umask

0002 # just checking!

$ wget http://svn.rpmforge.net/svn/trunk/rpms/perl-SOAP-Lite/perl-SOAP-Lite.spec
$ mach build perl-SOAP-Lite.spec

Building .src.rpm from perl-SOAP-Lite.spec
... snip ...
Build of perl-SOAP-Lite-0.710.05-1 succeeded, results in
/var/tmp/mach/centos-4-i386-extras/perl-SOAP-Lite-0.710.05-1
Results: /var/tmp/mach/centos-4-i386-extras/perl-SOAP-Lite-0.710.05-1
Build done.

That's it. The "Results:" line tells you the directory which contains the RPM packages(s) which were built:

$ ls -l /var/tmp/mach/centos-4-i386-extras/perl-SOAP-Lite-0.710.05-1

total 764
-rw-rw-r-- 1 candlerb mach 440039 May 15 16:43 perl-SOAP-Lite-0.710.05-1.noarch.rpm
-rw-rw-r-- 1 candlerb mach 270301 May 15 16:43 perl-SOAP-Lite-0.710.05-1.src.rpm
-rw-rw-r-- 1 candlerb mach 2734 May 15 16:43 perl-SOAP-Lite.spec
-rw-rw-r-- 1 candlerb mach 50507 May 15 16:43 rpm.log

If all goes well, then smile.

Troubleshooting

Missing dependencies

Sometimes a package will fail to build because it depends on another package which isn't available in the vendor's binary repository. For example, with older versions of CentOS, the perl-SOAP-Lite package wouldn't build because it depended on perl-Compress-Zlib which wasn't present.

No problem: just find a .spec file for it, and tell mach to build that package as well.

$ wget http://svn.rpmforge.net/svn/trunk/rpms/perl-Compress-Zlib/perl-Compress-Zlib.spec
$ mach build perl-SOAP-Lite.spec perl-Compress-Zlib.spec

It doesn't even matter which order your specify the .spec files on the command line. Mach sorts them so that the dependencies are built before the packages which depend on them.

Other problems

If you're still having trouble, add the '-d' flag to enable verbose debugging output, like this:

$ mach -d build foo.spec bar.spec

If mach complains that the repository is locked, and you're sure that no other user is using it (e.g. a previous build was forcibly aborted), then you can unlock it using mach unlock.

Occasionally, a package may have a BuildRequires: dependency on a package which isn't in a yum repository, and also can't be built from source (an example would be a package which depends on the oracle-instantclient-devel binary RPM). You can fix this by manually copying the RPM file(s) to /var/lib/mach/roots/distro/usr/src/rpm/RPMS.mach-local/.

Multiple roots

You can have multiple distro chroots installed at once. Just use mach -r distro ... to select the one you're interested in at any one time, or change defaultroot in ~/.machrc

As a result, a single build machine can build totally clean packages for a variety of different target systems. All you need is a bit of disk space and bandwidth.

RPM Links

1. "The Fight"
2. Maximum RPM
3. rpmforge spec file repository
4. yum and yum repository metadata
5. apt RPM repository
6. cpan2spec and cpan2rpm
7. gem2rpm