Setup of Biowolfe at TCD Genetics

Writing currently in progress!

Overview

The original server and the clients both ran SuSE Linux 6.3 which is available at www.SuSE.com (free download!) or on CDs/DVD in bookshops and computer stores (~£35). After the upgrade a server with preinstalled RedHat 7.0 was used.
I won't go into details on how to install Linux on the server - that's a chapter by itself. If you can't do it then you probably shouldn't be building a Linux cluster on your own anyway.
There are some steps necessary though for the booting of the diskless clients which I will explain in the server section.
Also, since the clients are (hard-)diskless, no information can be stored in the longterm and therefore they have to get their basic filesystem from the server. Setting this up will be explained in the client section.
For some crude traffic balancing the cluster network was arranged in an interleaved manner, which is shown in the network section.

Preliminaries

For the communication between server and client each computer needs to have an ip (internet protocol) number assigned. Since the cluster will run on its own network I could give them any numbers I like. But because the server has a connection to the outside world it is strongly advised to use official private ip numbers. This will avoid confusion in case one of the packages from the cluster will leak out into the main network.
For our setup I chose the 168.192.0 address space which provides ip numbers 168.192.0.1 to 168.192.254.254. This is part of the blocks reserved by the Internet Assigned Numbers Authority (IANA) for private use (see RFC1918). It makes sense to assign 168.192.0.254 as the ip for the server and start naming the clients from 168.192.0.1 onwards, so that more clients can be easily added lateron.

From now on I will use dc as an abbreviation for diskless client.

Server Section

Setting up the server consists of the following steps:
  1. setting up the client names
  2. providing a basic filesystem
  3. configuring NFS
  4. configuring TFTP
  5. configuring DHCP
  6. compiling a kernel for the clients
  7. configuring DNS

client names

It can be handy to call the clients by names, rather than by their IP addresses. The file /etc/hosts allows to set up aliases for other computers and you could insert lines like the following: 

# cluster network
192.168.0.254   master.cluster.gen.tcd.ie       master    # server 1
192.168.1.254   master.cluster2.gen.tcd.ie      master2   # server 2
192.168.0.1     dc_001.cluster.gen.tcd.ie       dc_001    # dc's
192.168.1.2     dc_002.cluster2.gen.tcd.ie      dc_002
192.168.0.3     dc_003.cluster.gen.tcd.ie       dc_003
192.168.1.4     dc_004.cluster2.gen.tcd.ie      dc_004
192.168.0.5     dc_005.cluster.gen.tcd.ie       dc_005
192.168.1.6     dc_006.cluster2.gen.tcd.ie      dc_006
192.168.0.7     dc_007.cluster.gen.tcd.ie       dc_007
192.168.1.8     dc_008.cluster2.gen.tcd.ie      dc_008
192.168.0.9     dc_009.cluster.gen.tcd.ie       dc_009
192.168.1.10    dc_010.cluster2.gen.tcd.ie      dc_010
192.168.0.11    dc_011.cluster.gen.tcd.ie       dc_011
192.168.1.12    dc_012.cluster2.gen.tcd.ie      dc_012
192.168.0.13    dc_013.cluster.gen.tcd.ie       dc_013
192.168.1.14    dc_014.cluster2.gen.tcd.ie      dc_014
192.168.0.15    dc_015.cluster.gen.tcd.ie       dc_015
192.168.1.16    dc_016.cluster2.gen.tcd.ie      dc_016
192.168.0.17    dc_017.cluster.gen.tcd.ie       dc_017
192.168.1.18    dc_018.cluster2.gen.tcd.ie      dc_018
192.168.0.19    dc_019.cluster.gen.tcd.ie       dc_019
192.168.0.20    dc_020.cluster2.gen.tcd.ie      dc_020

TFTP

The trivial file transfer protocol allows the clients to retrieve files like a kernel through the network without authorisation. This is necessary because the network card, which initialises the bootup process has only limited capabilities. A tftp daemon needs to run on the server in order to accept requests. It can be started automatically at bootup through the inet or xinet service. For xinet, place the following information into /etc/xinet.d/tftp: 

service tftp
{
        disable = no
        socket_type             = dgram
        wait                    = yes
        user                    = nobody
        log_on_success          += USERID
        log_on_failure          += USERID
        server                  = /usr/sbin/tftpd
        server_args             = /
}

A configuration line in '/etc/inetd.conf' could look like the following: 
tftp dgram udp wait root /usr/sbin/in.tftpd in.tftpd -s /tftpboot

This will enable exportation of kernels located in /tftpboot. You might have to restart (x)inet with /etc/init.d/(x)inetd restart to make changes to the tftp daemon come into effect.

DHCP

The dynamic host configuration protocol provides, amongst others, information for the automatic assignment of IP numbers to networked computers based on the address of their network card. In our case it also tells the diskless clients which kernel to download from the server.

Kernel

The kernel makes up the core of the Linux operating system. For the clients it needs to have several features which allow remote booting and diskless operation. You therefore need to compile a custom kernel, wjch can be done on the server:
  1. download the latest kernel sources
  2. unpack archive and modify the following options
    1. Enable loadable module support -> no
  3. make sure the following options are activated:
    1. IP:kernel level autoconfiguration
    2. IP:DHCP support
    3. the driver for your network card
  4. compile kernel (make bzImage)
  5. tag kernel (mknbi is available from the Etherboot site): mknbi-linux --format=elf --target=linux --rootdir=/tftpboot/dc --ipaddrs=rom arch/i386/boot/bzImage > /tftboot/vmlinux_2.4.14 (the rom option activates IP-Autoconfiguration with Bootp/DHCPD data)

DNS

Everytime you login from the master to one of the client nodes (via telnet/rsh/ssh) the client normally carries out a reverse DNS lookup to check if the ip-number of the connecting host complies with the hostname. To shorten this, run a DNS server for the client networks on the master.
First of all, put the following lines into the file /etc/named: 
options {
        directory "/var/named";
        pid-file "/var/run/named.pid";
};

zone "." IN {
        type hint;
        file "root.hint";
};

zone "0.168.192.in-addr.arpa" IN {
        type master;
        file "192.168.0.zone";
        check-names fail;
        notify no;
        allow-update { none; };
};
zone "cluster" {
        type master;
        notify no;
        file "cluster.zone";
};

zone "1.168.192.in-addr.arpa" IN {
        type master;
        file "192.168.1.zone";
        check-names fail;
        notify no;
        allow-update { none; };
};
zone "cluster2" {
        type master;
        notify no;
        file "cluster2.zone";
};

Then save the following files in the directory /var/named:
And change the first entries in root.hint: 
A.ROOT-SERVERS.NET.      3600000      A     192.168.0.254
B.ROOT-SERVERS.NET.      3600000      A     192.168.1.254

Start the nameserver with rcnamed start.

Client Section

BIOS

If you don't intend to connect a keyboard or a mouse to the clients you should specify in the BIOS to not stop on a keyboard error (STOP ON ALL ERRORS BUT KEYBOARD). This will prevent it from halting the boot procedure despite the lack of a keyboard. In the boot order the floppy drive should be first. If you can boot directly from the network card, activate this option (NETWORK FIRST). The location and availability of these options depends on the type of BIOS installed on the motherboard.

Remote Boot

Since the clients do not contain a harddrive they must receive all necessary information for the boot procedure through the network. This includes the kernel, the ip-number and the file system.
The network card is responsible for sending out requests for information. There are two freely available programs which can be used for this purpose, Netboot and Etherboot. I used the latter to create an image that can be placed on a floppy disk to emulate a bootrom:
  1. get the latest version of the Etherboot source code;
  2. unpack the archive (e.g. tar zxvf etherboot-5.0.3.tar.gz);
  3. compile a ROM image for your network card, e.g. cd etherboot-5.0.3/src; make bin32/3c90x.fd0
    (using the fd0 extension instead of .bin will write the image directly to the floppy drive)
To have the image on a floppy is very handy for trying it out first or in case you don't have the possiblity to burn it onto your network card. To test it, stick it into a client, make sure the BIOS is set to boot from the floppy drive first and start it up. You should see a message on the monitor stating that it detected a network adapter and is looking for a remote server.

Last modified at $Date: 2001/11/16 16:54:12 $
Contact: khokamp@sfu.ca.