Gentoo Logo

Gentoo Icecream Documentation

Content:

1.  Introduction

What is icecream?

Icecream is a program created by SUSE and is based on ideas and code by distcc. Similar to distcc it takes compile jobs(both portage and manual compilations) and distributes them to remote machines. The main difference to distcc is that is uses a central server, the scheduler, to distribute the jobs. This pays off mostly for shared computers, as it also takes the load of each machine into account and will therefore only use (mostly) idle machines.

How icecream works

Let's have a look at how compiling a program works. Usually you execute something like this:

Code Listing 1.1: Sample compilation

$ cc -c -o hello_world.o hello_world.c
$ ld -o hello_world hello_world.o

After the installation of icecream your $PATH will be changed in a way that whenever you execute cc or gcc icecream's binary icecc will be execute instead of your regular compiler. Icecc will then send the job (and the source file) to the scheduler which will distribute it to one of the available helper nodes.

To do the job the helper node needs a compiler (who would have thought of that?). But it will not use the compiler of the helper box (if any is installed at all), instead it will request a tarball (the icecc environment) containing the compiler (the binary and all required libraries) from the machine running the compilation(proxied through the scheduler).

Note: Because of this you may need to recompile your toolchain(gcc, glibc, binutils) without processor-specific cflags, because with them there is a greater chance that your compiler won't work on other machines.

After the compilation is finished it will send the .o file back to the scheduler which will forward it to the machine running the compilation. Which will then run the linker (needs to be run locally and will not be replaced by icecream) and create the final binary.

2.  Installation

Requirements

To use icecream you need at least two computers. They don't need to be of the same architecture and they even don't need to run the same operating system, but they do need to run the same version of icecream. However cross-compiling requires some extra work, if you use identical systems (e.g. two Linux/i686 systems) it works out of the box.

There also need to have a fairly fast network connection to each other (>10Mbit). The exact requirement depends on how much parallel jobs you are planning to run... just don't try it using a dial-up connection. ;)

Install icecream

Install icecream on all boxes that you want to add to your compilation cluster.

Code Listing 2.1: Emerge icecream

$ emerge ">=sys-devel/icecream-0.9.1-r1"

Designate one box as scheduler

As mentioned in the introduction icecream uses one central server (the scheduler) to distribute jobs across all nodes. This box needs to be online every time you want to run a compilation on any of the other nodes.

You don't need to dedicated a complete machine for this, the scheduler can also be a helper node and do compilation jobs.

After you made up your mind, edit /etc/conf.d/icecream on that machine and set ICECREAM_RUN_SCHEDULER="yes".

Set up portage to use icecream

As of 08/2008 portage has no explicit support for icecream (such as FEATURES="icecream"). That means if you use icecream with portage, portage won't know you are using icecream, it will just wonder what a super fast machine you have. ;)

On each machine that you want to benefit from your icecream setup do:

Code Listing 2.2: Edit make.conf

$ nano -w /etc/make.conf
(Set N to a suitable number for your particular setup)
(A common strategy is setting N as twice the number of total CPUs available + 1)
MAKEOPTS="-jN"
(Use icecc instead of your normal cc)
PREROOTPATH="/usr/lib/icecc/bin"

Set up manual compilations to use icecream

To make manually compilations (those you do by typing ./configure ; make -jN ; make install) use icecream all you need to do is adding /usr/lib/icecc/bin to the beginning of your PATH.

Code Listing 2.3: Add icecream to PATH

$ nano -w /etc/profile
(Add this to the end of the file)
export PATH="/usr/lib/icecc/bin:$PATH"

/etc/profile is just one possible place to set it, another (on a per-user basis) place to set it would be ~/.bashrc.

Start icecream

Starting icecream is simple, just execute those two commands on all boxes (scheduler and helper boxes):

Code Listing 2.4: Start icecream and add it to the default runlevel

$ /etc/init.d/icecream start
$ rc-update add icecream default

3.  Non-everyday setups

Using ccache which icecream

First, set up portage to use ccache them same way you'd set it up without icecream. Information about setting up ccache can be found in the handbook.

After you've set up ccache, all you need to do is add /usr/lib/ccache/bin to the beginning of your $PATH:

For portage:

Code Listing 3.1: Edit make.conf

$ nano -w /etc/make.conf
(Use icecc with ccache instead of your normal cc)
PREROOTPATH="/usr/lib/ccache/bin:/usr/lib/icecc/bin"

For manual compilations:

Code Listing 3.2: Edit /etc/profile

$ nano -w /etc/profile
(Add this to the end of the file)
export PATH="/usr/lib/ccache/bin:/usr/lib/icecc/bin:$PATH"

Cross-compiling

Did you read the DistCC Cross-compiling Guide? If yes, please note that the way icecream works is different, so most you read there doesn't apply here.

Let's assume you have two machines, one Gentoo/i686 and one Gentoo/SPARC and you want the i686 box to help the sparc box doing compilations.

In that case you need a cross-compiler that runs on i686 and produces code for sparc (HOST=i686, BUILD=i686, TARGET=sparc).

To create one you can use sys-devel/crossdev. After that you need to create a tarball (the icecc environment) containing the compiler. Basically you need these binaries and their dependencies:

  • as
  • gcc
  • g++
  • cc1
  • cc1plus

Luckily there are lots of tools to help you to do that, you just need to execute these three commands on the i686 box:

Code Listing 3.3: Create an icecc environment for cross-compiling

(Install Gentoo's cross-toolchain generator)
$ emerge sys-devel/crossdev
(Create a cross-toolchain for sparc-unknown-linux-gnu)
$ crossdev -t sparc-unknown-linux-gnu
(Create the icecc environment)
$ icecream-create-env sparc-unknown-linux-gnu

Now you have a file with a name similar to df8f4a336bbfc23ed86a296966f50be4.tar.gz, that's your icecc environment.

Move it to the sparc machine. You can put it anywhere you want and give it a more significant name, e.g. /var/icecc/host-i686-target-sparc.tar.gz.

The native environment icecream created automatically is located in /var/cache/icecream/native. If it doesn't exist you can simply create it by running icecream-create-env.

It's a good practice to store all environments in the same directory. Let's assume you copied it to /var/icecc/host-sparc-target-sparc.tar.gz. But remember: Whenever you update binutils or gcc you should recreate them, otherwise they'd still use the old version.

After that you need to tell icecream about it by setting the ICECC_VERSION environment variable. ICECC_VERSION contains a comma separated list of icecc environments for different architectures, the format is:

Code Listing 3.4: ICECC_VERSION format

[ architecure(output of uname -m) ]:[ path to tar.gz icecc environment ]

In our case you'd need to add this to /etc/profile on the sparc box:

Code Listing 3.5: Edit /etc/profile

$ nano -w /etc/profile
(Add this to the end of the file)
export ICECC_VERSION="i686:/var/icecc/host-i686-target-sparc.tar.gz,sparc64:/var/icecc/host-sparc-target-sparc.tar.gz"

Now restart icecream on the sparc box and your're done.

If you also want the sparc box to assist the i686 box you need to create the cross-icecc environment on the sparc box, copy it to the i686 and set ICECC_VERSION.

More cross-compiling

Imagine you have an i686 box with an arm-linux-gnu cross-toolchain installed which you are using to compile programs for your cell phone and you want to use icecream to speed up compilations?

First, create a few symlinks to invoke icecream instead of your normal compiler:

Code Listing 3.6: Create cross-compiler symlinks

$ icecream-config --install-links arm-linux-gnu

After that you need to set ICECC_VERSION to point to the tarball(icecc environment) for arm-linux-gnu. You can create is using icecream-create-env. You probably don't want to set it permanently, just on an as-needed basis.

Code Listing 3.7: Create arm-linux-gnu icecc environment and set ICECC_VERSION

(Install Gentoo's cross-toolchain generator)
$ emerge sys-devel/crossdev
(Create a cross-toolchain for arm-linux-gnu)
$ crossdev -t arm-linux-gnu
(Create the icecc environment)
$ icecream-create-env arm-linux-gnu

4.  Security concerns

Icecream has no authentication at all and the icecc daemon is run as root.

Distcc has a --allow parameter to specify clients that are allowed to connect, icecream has no such thing, it only provides the config option ICECREAM_ALLOW_REMOTE, but no ACL. That means anybody can use your system as a host for compilations and could potentially exploit bugs in gcc to gain control over your box.

Therefore it is best to block icecream's ports in your firewall and only allow connections from trusted hosts. These are the ports icecream uses:

  • TCP/10245 for the icecc daemon (required)
  • TCP/8765 for the scheduler (required)
  • TCP/8766 for the telnet interface to the scheduler (optional)
  • UDP/8765 for broadcast to find the scheduler (optional)


Print

Page updated August 2, 2008

Summary: This document serves as a HOWTO for using Icecream with Gentoo.

Friedrich Oslage
Author

Donate to support our development efforts.

Copyright 2001-2014 Gentoo Foundation, Inc. Questions, Comments? Contact us.