How To Create A Cluster Testbed Using CentOS 5 Virtualization And iSCSI

A. Overview

This guide attempts to provide a Xen based test environment where you can practice setting up a two node cluster (cluster setup itself is not discussed here - I'm merely giving you what you need to set it up).

XEN can host two type of guest systems para-virtualized and fully-virtualized:

• for para-virtualized guests you require the Red Hat Enterprise Linux 5 installation tree available over NFS, FTP or HTTP.
• for fully-virtualized guest installations you will require DVD or CD-ROM distribution media or a bootable .iso file and a network accessible installation tree

For details, please refer to the RHEL5 Virtualization Manual.

I'll be using para-virtualized guests here in my setup. There will be three systems involved here:

• node00 - physical system
  virtual IPs: 192.168.222.1 (public1 vlan)
  192.168.100.1 (private1 vlan)
• node01 - para-virtualized guest 1
  virtual IPs: 192.168.222.10 (public1 vlan)
  192.168.100.10 (private1 vlan)
• node02 - para-virtualized guest 1
  virtual IPs: 192.168.222.20 (public1 vlan)
  192.168.100.20 (private1 vlan)

B. What I used

• an HP Blade bl25p machine with 4G of RAM (this is actually an AMD64 blade machine). A machine with decent amount of RAM and processing speed should do.
• Centos i386 5 update 1 www.centos.org DVD ISO downloaded HTTP, NFS and FTP installation sources were created from this iso. Also, the yum repository that can be used by host and guest systems will be generated from the centos iso image.
• logical volumes hosting the guests and the "virtual luns" via iscsi (you can also use disk partitions - please refer to the virtualization guide for details).

1. My LVM setup

The following is my LVM configuration. The lvLUN0* entries are the ones I used for iSCSI setup and will be shared by the two virtual guest systems.

lvs

  LV       VG          Attr   LSize   Origin Snap%  Move Log Copy%
  lvLUN01  Virtual00VG -wi-ao  50.00G
  lvLUN02  Virtual00VG -wi-ao  50.00G
  lvNODE01 Virtual00VG -wi-ao  30.00G
  lvNODE02 Virtual00VG -wi-ao  30.00G
  lvNODE03 Virtual00VG -wi-ao  15.00G
  lvsys00  vg00        -wi-ao 512.00M
  lvsys01  vg00        -wi-ao   8.00G
  lvsys02  vg00        -wi-ao   8.00G
  lvsys03  vg00        -wi-ao 512.00M
  lvsys04  vg00        -wi-ao 128.00M
  lvsys05  vg00        -wi-ao   1.00G
  lvsys06  vg00        -wi-ao 256.00M

C. Host Preparation

I'm assuming that you know how to install CentOS or other RHEL based distributions and that you are familiar with rpm installation. Since I do a lot of setup for test/dev environments at work, I already have an installation server making it easy to do a network based install via PXE. The kickstart file for node00 is provided below. You can do a local media install (you have the ISO so you can burn it to a DVD) and just refer to the kickstart file for some of the configuration. The list of packages I used is in the %packages section of node00's kickstart file. You can install them manually using yum, like:

# will list centos installation groups

yum grouplist

# will install Virtualization group

yum groupinstall Virtualization

1. ks file and installation

1.a kickstart file I use for the host (node00)

You'll have to modify the following to suit your setup.

## START node00_ks.cfg
#modify for your own settings
install
nfs --server=remote_server --dir=/path/to/CENTOS5U1/i386
lang en_US.UTF-8
keyboard us
skipx
reboot
network --device eth2 --bootproto static --ip a.b.c.1 --netmask 255.255.255.0 --gateway a.b.c.2 --nameserver x.y.z.n --hostname node00.example.com
# grub and root password is a1s2d3f4g5
rootpw --iscrypted $1$3CXK2$CG9WlX2PuPpp7nxYMQGwP0
firewall --disabled
authconfig --enableshadow
selinux --disabled
timezone Asia/Singapore
bootloader --location=mbr --driveorder=cciss/c0d0 --append="rhgb quiet" --md5pass=$1$3CXK2$CG9WlX2PuPpp7nxYMQGwP0
clearpart --all --initlabel --drives=cciss/c0d0
part /boot --fstype ext3 --size=100 --ondisk=cciss/c0d0
part pv.100000 --size=100 --grow --ondisk=cciss/c0d0 --asprimary
volgroup vg00 --pesize=32768 pv.100000
logvol /tmp --fstype ext3 --name=lvsys05 --vgname=vg00 --size=1024
logvol /opt --fstype ext3 --name=lvsys04 --vgname=vg00 --size=128
logvol /var --fstype ext3 --name=lvsys03 --vgname=vg00 --size=512
logvol /usr --fstype ext3 --name=lvsys02 --vgname=vg00 --size=8192
logvol swap --fstype swap --name=lvsys01 --vgname=vg00 --size=8192
logvol /home --fstype ext3 --name=lvsys06 --vgname=vg00 --size=256
logvol / --fstype ext3 --name=lvsys00 --vgname=vg00 --size=512
%packages
@development-libs
@editors
@system-tools
@text-internet
@x-software-development
@virtualization
@dns-server
@core
@base
@ftp-server
@network-server
@legacy-software-development
@base-x
@web-server
@printing
@server-cfg
@sql-server
@admin-tools
@development-tools
lsscsi
createrepo
audit
net-snmp-utils
iptraf
tftp
lynx
mesa-libGLU-devel
kexec-tools
bridge-utils
device-mapper-multipath
vnc-server
xorg-x11-server-Xnest
xorg-x11-server-Xvfb
imake
openmotif
-vim-enhanced
-zisofs-tools
-zsh
-bluez-hcidump
-sysreport
## END of node00_ks.cfg

2. Host configuration

Setting up an HTTP, NFS and FTP installation server:

2.a web server

#/etc/httpd/conf.d/centos5u1.conf
Alias /centos5u1 /var/ftp/pub/centos5u1
<Location /centos5u1>
Options Indexes FollowSymLinks MultiViews
IndexOptions FancyIndexing
Order deny,allow
Deny from all
Allow from 127.0.0.1 ::1 all
<Location>

Then start the httpd service and make sure it does during startup:

service httpd start
chkconfig httpd on

2.b NFS server

Edit /etc/exports and put the following into it:

# /etc/exports
/var/ftp/pub/centos5u1 192.168*(ro)

service nfs start
chkconfig nfs on

2.c FTP server

Since we already have the source in /var/ftp/pub/centos5u1, all that is needed is to start vsftpd:

service vsftpd start
chkconfig vsftpd on

2.d YUM repository

For this setup, I only use a local yum repository from the Centos DVD ISO I downloaded. First, I loopback mount it in /var/ftp/pub/centos5u1/i386/:

cd /var/ftp/pub/centos5u1/
mkdir temp
mount -o loop CentOS-5.1-i386-bin-DVD.iso temp
cp -pr temp i386
umount temp
createrepo -g i386

(i386/repodata/ will then be updated.)

For RHEL5, it's different:

createrepo -g repodata/comps-rhel5-server-core.xml Server

You need to do this inside the i386 directory (after loopback mounting and copying the whole directory structure).

2.d.1 the yum repo configuration:

I renamed the default repo files in /etc/yum.repos.d/ to *-repo (instead of *.repo) to disable them. I then created this file:

#/etc/yum.repos.d/CentOS5.repo
[centos5-Server]
name=CentOS5 Server
baseurl=http://node00/centos5u1/i386
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

node00 needs to be defined in /etc/hosts for the above file to work. Or just replace node00 with its IP address.

2.e VNC server

You won't be needing a vnc connection if you have local console access to the physical machine. All you need to do is switch into gui mode:

telinit 5

and open a gui terminal (like gnome-terminal or kde konsole). But since I do everything remotely, I use vncserver and vncviewer to do gui based stuff.

2.e.1 run vncserver:

This will bringvup a vncserver in node00 that is accessible via "vncviewer" at node00:1 (assuming node00 is resolveable from your vncviewer host).

vncserver

You will require a password to access your desktops.
	  
	  Password:
	  Verify:
	  xauth:  creating new authority file /root/.Xauthority
	 
	  New 'node00.example.com:1 (root)' desktop is node00.example.com:1
	  
	  Creating default startup script /root/.vnc/xstartup
	  Starting applications specified in /root/.vnc/xstartup
	  Log file is /root/.vnc/node00.example.com:1.log