DistccTycoon: distcc_tycoon.txt

This is a how-to for installing distcc to run on a Tycoon cluster.

distcc is a free, distributed C compiler available at distcc.samba.org.

In this guide the "client" is the machine directing the compilation, 
probably your local machine, and the "servers" are the machines to 
which the jobs are distributed. Of course distcc can also use the 
client to do some of the work.

It is assumed that the user has established accounts on several machines and
has downloaded the distcc tarball from distcc.samba.org.

This guide first shows how to install distcc on the client, then on one
server at a time, and finally on multiple servers at once.
After the installation steps is a brief guide to getting distcc up and running.


INSTALLATION ON CLIENT

1. Install gcc on client machine if you don't have it:
   #yum -y install gcc.

2. Install distcc.
   a. Unzip and untar distcc tarball.
   b. Go into the distcc directory and install it with
      $./configure
      $make
      #make install


INSTALLATION ON ONE SERVER AT A TIME

In this section {} represents the place where you type in the IP address of 
the server.

1. Run the following command to open port 3632 on the server:
   $tycoon host bid {} 1000000000 "{ ('TCPv4Port', '3632'): 1}"

   Here {} is the IP address of the machine you will be connecting to.
   This command is necessary to enable distcc to communicate with the server.

2. Install distcc on the server

   a. Copy distcc tarball to server:
      $tycoon_scp distcc-2.18.3.tar.bz2 {}:

      Don't forget the colon at the end of the copy command.

   b. ssh into machine with 
      $tycoon_ssh {}

   c. Follow the steps above (in INSTALLATION TO CLIENT) to install distcc. 

      The root password on tycoon virtual machines is rootroot.
   
3. Start the distcc daemon on the server with
   $distccd --daemon -a {client IP address} --log-file log --log-level info

   The log file name (I used "log" above) is your choice. 
   I find that "info" is a useful log-level; for others see the man page.


INSTALLATION ON MULTIPLE SERVERS AT ONCE

To save time you can use use vxargs to execute commands on multiple 
remote machines at once. vxargs is a python script available online. 
vxargs reads a text file to get the IP address of the machines to connect to. 
This file must have the machine addresses by themselves on separate lines.

In vxargs commands, you type in {} literally to tell vxargs where
to put the remote machine's IP address. This may be confusing in light of 
my previous notation. So: you must literally include the characters "{}" in
a vxargs command where the IP address would normally go.

1. Install gcc on the servers:
   $vxargs -a iplist.txt -o vxargs_log tycoon_ssh root@{} yum -y install gcc

   In this command "iplist.txt" is a file and "vxargs_log" a directory.
   You do not have to use these exact names.
   iplist.txt is the list of addresses of the remote machines (as
   described above).
   vxargs_log is a directory containing the log files where 
   vxargs will write the output of your commands (stdout, stdin, stderr).

   Again, you must literally include the "{}".

2. Open port 3632 on the servers:

   $vxargs -a iplist.txt -o vxargs_log tycoon host bid {} 1000000000 "{ ('TCPv4Port', '3632'): 1}"

3. Copy in the installed distcc directory from your home machine:
   $vxargs -a iplist.txt -o vxargs_log tycoon_scp -r distcc-2.18.8 {}:

   This is far simpler than trying to install distcc directly 
   on each server using vxargs. There is a technical difficulty with the
   command syntax of tycoon_ssh which makes a remote distcc install difficult.
   Using ssh -p 47277 instead of tycoon_ssh in the remote install is possible 
   but you would have to establish all the servers as known hosts before
   running vxargs because you can't input while vxargs is running.

4. Start the distcc daemon on the servers:

   $vxargs -a iplist.txt -o vxargs_log tycoon_ssh {} ./distcc-2.18.8/distccd --daemon -a {client IP address} --log-file log --log-level info

   Because distcc was not directly installed, the path will not be updated
   on the servers and you have to include the path in the command
   to start the daemon.


RUNNING DISTCC

This is a brief guide to how to issue a distcc command. For more information 
see the man page.

Once distcc is installed on the client and servers, you have opened port 
3632 on the servers and started the daemons there, there is one thing left
to do. On the client, do an

$export DISTCC_HOSTS='{server 1 IP address} {server 2 IP address} ...'

This variable tells distcc where to send the jobs. You can include 
"localhost" in this list and jobs will be compiled on the client too. 
The order of the servers in the DISTCC_HOSTS variable determines the 
priority, so put the fastest machines first for best results.

The man pages include a list of other options for this command, as well
as other shell variables distcc can use.


Finally, to compile using distcc, just type

$make -j # CC=distcc {whatever}.

Here # indicates the number of jobs to run at once. This must be
at least as large as the number of servers if you want to use them all.
You can experiment with increasing this number, which may 
improve the performance.

There are other ways to send the make command to distcc; 
for example, you can set distcc to be the default compiler. See man pages.