Biowolfe at TCD Genetics

Introduction

To run homology searches in-house the department decided to invest in a parallel computer. Instead of going for a very expensive industrial one the choice was made for a self-built pc cluster.
With ever decreasing prices for PC components and parallel computing support from freely available operating systems (like Linux) it is possible to assemble a system that is equally powerful to an industrial parallel computer at a fraction of its cost.
The setup consists of a powerful server who holds filesystems, programs and databases and a number of diskless clients that act as pure memory and processing units. They are connected to the server through a direct, fast network and work in parallel on parts of the overall task (e.g. a homology search).
The server is responsible for

providing the clients with the basic operating system, programs and data
splitting up tasks into equally sized chunks (for each client)
starting the processes
and finally collecting and reassembling solutions calculated by the clients.

A major advantage (beside the low cost) of the cluster over an industrial parallel computer is its flexibility. It is very easy to add more clients to the PC farm (as long as free network ports are available), and parts of the existing PCs can be easily altered (more memory, faster processors, etc.) to increase overall performance.

Specifications

parts	Server	Client
mainboard	Supermicro 2DGE	Matsonic MS7101C
case	Supermicro 760A	standard
CPU	2 x PII Xeon ~~450~~ 500 MHz 512 KB	1 x Celeron 400 MHz
RAM	2 x 256 MB ECC	1 x 256 MB ECC
graphics card	ATI XPERT, 8MB	S3 Trio 4 MB
NIC	1 x 3Com 3905C, 1 x Intel EtherExpress 100 Pro	1 x Intel EtherExpress 100 Pro
controller	1 x Adaptec 2940 U2W	-
harddrive	4 x 50 GB SCSI Seagate Barracuda	-
floppy disk	1 x standard	1 x standard
CD-ROM	1 standard, 40 speed	-
mouse	1 x PS/2	-
keyboard	1 x PS/2	-
monitor	1 x CTX 19"	-
cost	~£13,000	~£900

additional:	16 port 100Mbit Prime switch (~£600)
	Uninterruptible Power Supply unit (UPS) (~£300)
	network cables (~£70)

All prices in Irish Punt, dating from November 1999 (1 £= 1.26974 Euro).

(the cluster has been expanded, see current configuration)

Installation

The files and documentation for setting up a cluster ours are available on from the Setup Page.

Important points

There are some important issues about our setup that I would like to mention:

Slot 2 vs. Slot 1
I chose Xeon CPUs (Slot 2 processors) because they are particularly designed for high performance server needs. I had problems with running the server at full speed of 450 MHz. It was very hard to figure out the cause of the problem (CPU, RAM, mainboard, BIOS settings, etc.) and until I had access to another Xeon PC I had no chance to swap parts for proper trouble shooting.
In the end it turned out that both of the Xeon processors were either fakes or damaged. I got them exchanged for two PII Xeon 500 MHz processors and everything is working fine now!
In the meantime other solutions for multiple processors (which I think is important for the server) came up: PIII and AMD. And although Xeons are supposed to be much better suited (due to their full speed access cache) I'm not sure if their immense price is justified.
Diskless Clients
There are several reasons why I chose to configure the clients without harddisks:
- reduction of heat, noise, cost
- robustness
- ease of setup, administration, and maintenance
It was a bit tricky in the beginning to work out how to boot the clients via network but the trouble I went through was worth it in the end. The diskless clients concept makes the whole system very easy to administrate. All the files are located on the server. All clients boot from one copy of the server's operating system and changes will therefore be propagated to each machine automatically. The clients don't contain any moving parts and are therefore very robust towards physical and electrical impacts. The small room in which the machines are stored gets slightly warmer when the system runs in full operation for days, but so far we didn't need any air-conditioning.
The one concern I had about the network as a bottleneck was unnecessary: with the preliminary tests that I did so far I didn't notice any delay due to the network connection. I hope this will hold tru for the full operation as well.
Networking
I believe that the diskless setup up with one server will work out easily for up to 32 clients. In that case though, I would recommend a gigabit network card (1000 Mbit) for the server and a switch with one gigabit port and the rest supporting at least 100 Mbit (I have heard of one being available from 3Com).
Power Supply
One thing one has to bear in mind when setting up a cluster is the power supply for all the machines. Make sure there are enough sockets available!

Clients

11 diskless clients with switch

Server

the inside of the PII Xeon server

Last modified at $Date: 2001/11/16 16:54:21 $
Contact: khokamp@sfu.ca.