Cluster Systems Management (CSM) software provides a bunch of tools to manage from a single point of control your AIX environment. There are a number of commands available but the main four which I find quite useful are dsh, dping, dcp and dshbak. In this post, I’ll go over each command and some of its uses.
Before we begin, we need to select a server which is going to be the point of control. While this can be any AIX server on the network, it’s mainly common to have your NIM server as the point of control. In the case of dsh and dcp, we’re going to be using SSH, so keys need to be setup between your point of control server and the rest of your environment.
Distributed Shell (dsh) allows you to concurrently run commands on remote nodes. It requires very little setup, but it’s one of those hidden AIX commands that few administrators know about. As you may have already guessed, dsh comes with the CSM client filesets. Specifically, it comes from the csm.dsh fileset which creates a symbolic link – /usr/bin/dsh -> /opt/csm/bin/dsh
root@AIX / > lslpp -w /usr/bin/dsh File Fileset Type ---------------------------------------------------------------------------- /usr/bin/dsh csm.dsh Symlink
There are a few environmental variables that we need to setup first.
DSH_NODE_RSH=/usr/bin/ssh – By default, dsh uses rsh to execute remote commands. If you don’t know why using rsh is a bad idea, hand in your sysadmin card now. We need to tell dsh to execute all remote commands via ssh.
DCP_NODE_RCP=/usr/bin/scp – As above, dcp uses rcp to send files remotely, so lets change that to use scp
DSH_NODE_OPTS=-q – This passes the -q option to ssh to suppress the banner message from showing. This just makes the output much easier to read when working with a number of nodes.
DSH_NODE_LIST=/.dsh/model – You can feed a list of nodes to dsh a number of ways, but I find using text files the best way to manage it.
Most places have a typical dev/mod/prod setup which can consist of a number of servers within each environment. I have four node text files in my /.dsh directory.
-rw-rw-r-- 1 root system 577 Jan 11 10:19 all -rw-r--r-- 1 root system 71 Feb 12 13:06 dev -rw-r--r-- 1 root system 75 Oct 28 10:38 mod -rw-r--r-- 1 root system 106 Jan 25 13:00 prod
This way I can run commands on all the nodes in a specific environment, or across all nodes. The easiest way to switch between lists is by using aliases.
In my .kshrc file
alias dshmod='export DSH_NODE_LIST=/.dsh/mod' alias dshprod='export DSH_NODE_LIST=/.dsh/prod' alias dshdev='export DSH_NODE_LIST=/.dsh/dev' alias dshall='export DSH_NODE_LIST=/.dsh/all'
The contents of the node text files is each node on a single line.
root@AIX / > cat /.dsh/mod SERVER1 SERVER2 SERVER3
That’s it for the configuration. You can run dsh -q to show the current configured variables. Look at the man page for dsh to see further configurable options and the IBM website.
Let’s see if it works.
root@AIX / > dshmod root@AIX / > dsh date SERVER1: Mon Feb 28 12:59:34 EETDT 2011 SERVER2: Mon Feb 28 12:53:58 EETDT 2011 SERVER3: Mon Feb 28 12:56:51 EETDT 2011
If you only wanted to run the command on a select number of hosts, you can use the -w switch.
root@AIX / > dsh -w SERVER1,SERVER3 "date" SERVER1: Mon Feb 28 13:01:20 EETDT 2011 SERVER3: Mon Feb 28 12:55:43 EETDT 2011
We can make that output look a bit neater by piping through the dshbak command.
root@AIX / > dsh -w SERVER1,SERVER3 "date" | dshbak HOST: SERVER1 ------------- Mon Feb 28 13:17:17 EETDT 2011 HOST: SERVER3 ------------- Mon Feb 28 13:11:41 EETDT 2011
Just like the dsh command, dcp allows you to send files concurrently to multiple hosts. dcp requires the same environmental setup as dsh. Seeing as that’s already setup, we can use it straight away.
root@AIX / > lmktemp test 1M test root@AIX / > dcp test / root@AIX / > dsh ls -l /test SERVER1: -rw-r--r-- 1 root system 1048576 Feb 28 13:32 /test SERVER2: -rw-r--r-- 1 root system 1048576 Feb 28 13:32 /test SERVER3: -rw-r--r-- 1 root system 1048576 Feb 28 13:32 /test
What I did above is created a 1M file called test, used dcp to send it to my hosts, then verified the file was on each node using dsh. You can find more about dcp from the man pages and the IBM website.
The dping command allows you to ping nodes in parallel (with the option of serially). Now, it doesn’t read the DSH_NODE_LIST variable like dsh/dcp, but you can pass the -f option and tell it to.
root@AIX / > dping -f /.dsh/mod SERVER1: ping (alive) SERVER2: ping (alive) SERVER3: ping (alive) SERVER4: noping (unreachable)
If you only want to ping a select few hosts, then you can use the -n switch.
root@AIX / > dping -n SERVER1,SERVER3 SERVER1: ping (alive) SERVER3: ping (alive)
You can find more on dping by reading the man pages and on the IBM website. Like dsh, you can also pipe the output of dping through dshbak.
If you have any questions on the above, use the comment section below.
 – http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.csm16.cmds.doc/am7cm_csmcomm.html
 – http://www.ece.uci.edu/~chou/ssh-key.html
 – http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.csm16.cmds.doc/am7cm_dsh.html
 – http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.csm16.cmds.doc/am7cm_dcp.html
 – http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.csm16.cmds.doc/am7cm_dping.html