ClusterSSH on steroids: tips and tricks

Today I would like to share a combination of  tips, tricks and gems that are possible with ClusterSSH. Duncan Ferguson, the creator of CSSH, came up with this invaluable tool that helps many system administrators and QA testers manage many systems in parallel.

Introduction

ClusterSSH controls a number of xterm windows via a single graphical console window to allow commands to be interactively run on multiple servers over an SSH connection. ClusterSSH (cssh) is written in PERL and easily allows modifications to the source code.

Demo

 About ClusterSSH

ClusterSSH (cssh) command opens an administration console and an xterm to all specified hosts. Any text typed into the administration console is replicated to all windows. All windows may also be typed into directly. Commands are performed all at once via this tool, which speeds up testing and allows for easy output comparison between tested nodes. Connections are opened via SSH, therefore a correctly installed and configured SSH installation is required. Extra caution should be taken when editing system files such as /etc/inet/hosts as lines may not necessarily be in the same order. Assuming line 5 is the same across all servers and modifying that is dangerous. Better to search for the specific line to be changed and double-check before changes are committed. Extra caution should be taken when executing destructive commands as root on a large number of physical servers.

About DevilsPie

DevilsPie (devilspie) is a program which can detect windows as they are created, and perform actions on them if they match as set of criteria. ClusterSSH is used in conjunction with DevilsPie to control the position of xterm windows, administration console and to hide windows decorations such as title bar, minimize / maximize / close buttons. This in return provides a clean look and a very efficient use of monitor space.

Devil's pie in action

Planning

To ensure the success of using ClusterSSH, please consider the recommendations below.

1. Because ClusterSSH sorts xterm windows in alphabetical order it is recommended that hosts are given appropriate hostnames. This will ensure that sorting will group similar OS hosts together. For instance if hostnames of all Solaris machines start with “s” and RedHat machines start with “r”, once ClusterSSH is launched, all Solaris and RedHat machines will be grouped together. This simplifies the output comparison process across similar operating systems.
2.  It is preferred that testing hosts have permanent IP addresses set and either entered in DNS or local /etc/hosts file on the operating system that will host ClusterSSH. Working with a large number of DHCP hosts can be inconvenient and time consuming because of changing IP addresses.
3. ClusterSSH setup described in this document is installed on Ubuntu 9.04 Desktop OS running on VMware Workstation 6.5.2. ClusterSSH can also be used with other operating systems.

Installation process

1.        Install Virtual BOX

2.        Install Ubuntu 10 Desktop OS

a.        Note: Use bridged VMware network connection.

3.        Apply all latest updates and patches to Ubuntu 10 Desktop OS

a.        To check for updates from the KDE menu, go to:

 System à Administration à Update Manager

4.        Install Virtual BOX Guest Additions 

5.        Install SSH Server

a.        Execute: “sudo apt-get install ssh”

b.        Create public keys, execute: “ssh-keygen –d” and respond with <ENTER> to questions.

6.        Install ClusterSSH

a.        Execute: “sudo apt-get install clusterssh”

7.        Install DevilsPie

a.        Execute: “sudo apt-get install devilspie”

8.        Perl module[1] XML::Simple

a.        To install:

                                                   i.      # perl –MCPAN –e shell

                                                  ii.      cpan[1]> install XML::Simple

9.     Create .cssrc and .csshrc_send_menu in your users home directory

---

[1] You may need additional PERL modules, just follow provided commands to install them

Important files

$HOME/Desktop/Launch	Custom ClusterSSHv4 launch SHELL script	Contains cssh command with a list of target hosts
$HOME/.csshrc	Custom ClusterSSH configuration file	Contains ClusterSSH configuration parameters
/etc/cssh/cssh.ds	Custom DevilsPie configuration file	Contains DevilsPie configuration parameters
$HOME/.csshrc_send_menu	Menu XML configuration file	Contains menu and folders populated with commands

ClusterSSH launch script “Launch”

This script contains list of servers to connect via SSH. Order is irrelevant, since launched CSSH sorts all windows by name in alphabetical order.

#!/bin/bash
CSSHPID="/tmp/cssh.pid" 

if [ -f $CSSHPID ]; then
 for pid in `cat $CSSHPID`
 do
     pkill -P $pid 
     pkill xterm
     pkill devilspie
 done
 rm -f $CSSHPID
fi

echo $$ >> $CSSHPID
devilspie /etc/cssh/cssh.ds& 

cssh -l root yourhost1 yourhost2 yourhost3 yourhost4 

pkill devilspie
rm -f $CSSHPID

ClusterSSH XML configuration file “.csshrc_send_menu“

$HOME/.csshrc_send_menu

 

ClusterSSH configuration file “.csshrc“

.csshrc

Devilspie configuration file “cssh.ds“

cssh.ds

 

Secure Shell (ssh) key-based authentication

ClusterSSH is using SSH protocol as its transport method to connect to remote test systems. In order to simplify the connection process and avoid being prompted for password every time ClusterSSH SHELL scripts are launched, users may setup an SSH key-based public key authentication. If public keys are already created on UBUNTU desktop, simply change directory to your home directory “.ssh”. From .ssh directory issue the following command: “ssh-copy-id root@<system>” where “<system>” is the target system you are planning to connect using ClusterSSH. Type-in root’s password and this command will copy public key from your UBUNTU desktop login to the remote system. Next time ClusterSSH is launched it will be able to connect directly to your test system(s) without prompting for password - saves time!