kristijan.org

disk_size.sh – Quick look at disk details

Kristijan — Wed, 18 Apr 2012 01:11:05 +0000

I’ve been doing many AIX server migrations lately. Some which involve taking a mksysb and restoring, others which involve presenting LUN’s from a SVC and then doing a migratepv. The latter can result in a large number of disks presented on the host, so I wrote a quick basic script which gives me the details that I need – hdisk name, size, vg, pvid and serial number.

Maybe someone out there will find it useful.

#!/bin/ksh
 
while getopts "g" OPTION
  do
        case ${OPTION} in
          g) BLOCK_SIZE="GB";;
        esac
  done
 
set -A hdisk `lspv | awk '{ print $1 }'`
set -A pvid `lspv | awk '{ print $2 }'`
set -A vg `lspv | awk '{ print $3 }'`
 
count=0
if [ "$BLOCK_SIZE" = "GB" ]; then
 while (( $count < ${#hdisk[*]} )); do
           hdisk_size_mb=`getconf DISK_SIZE /dev/${hdisk[$count]}`
           hdisk_size_gb=`expr $hdisk_size_mb / 1024`
           size[$count]=$hdisk_size_gb
        let count="count + 1"
 done
else
 while (( $count < ${#hdisk[*]} )); do
           size[$count]=`getconf DISK_SIZE /dev/${hdisk[$count]}`
        let count="count + 1"
 done
fi
 
count=0
while (( $count < ${#hdisk[*]} )); do
           serial[$count]=`lscfg -vl ${hdisk[$count]} | grep "Serial Number" | awk -F. '{print $NF}'`
        let count="count + 1"
 done
 
 
count=0
printf "%-10s %-10s %-15s %-20s %s\n" Disk Size VG PVID Serial
printf "%-10s %-10s %-15s %-20s %s\n" ---- ---- -- ---- ------
while (( $count < ${#hdisk[*]} )); do
           printf "%-10s %-10s %-15s %-20s %s\n" ${hdisk[$count]} ${size[$count]} ${vg[$count]} ${pvid[$count]} ${serial[$count]}
        let count="count + 1"
done
exit

root@AIX #> ./disk_size.sh
Disk       Size       VG              PVID                 Serial
----       ----       --              ----                 ------
hdisk0     286102     None            00c67a84bb96c0ac	   JZW4SL9C
hdisk1     286102     None            00c67a84bbc2fec7     JZVLZJ2C
hdisk2     286102     None            00c67a84d41ba824     JZVLK4BC
hdisk3     286102     None            00c67a84d436cdd3     JZVMUG2C
hdisk4     52224      None            00c67a8433f75e56     75LZ7211200
hdisk5     52224      None            00c67a849f0e70ba     75M17911163
hdisk6     52224      None            none                 75LZ7211201
hdisk7     52224      None            00c67a849f0f2453     75LZ7211202
hdisk8     204800     None            00c67a849ee59661     75LZ7211012
hdisk9     307200     rootvg          00c67a84a67c6b56     600507680180867BB00000000000000B
hdisk10    307200     datavg          00c67a84a67c87b3     600507680180867BB00000000000000C
hdisk11    46080      wpar1           00c67a84a67d4860     600507680180867BB00000000000000D
hdisk12    40960      wpar2           00c67a84a67cb55b     600507680180867BB00000000000000E
hdisk13    51200      wpar3           00c67a84a67cccc9     600507680180867CC00000000000000F
hdisk14    51200      wpar4           00c67a84a67ce357     600507680180867CC000000000000010
hdisk15    51200      wpar5           00c67a84a67cf6ae     600507680180867CC000000000000011

You can use the -g switch to make the size output in GB blocks.

Check those attributes!

Kristijan — Thu, 05 Apr 2012 02:12:57 +0000

This is a script I wrote purely from the frustutation of verifying each and every adapter/device attribute which one sets after an AIX installation. What this script does is loop through the list of user defined attributes, and verifies each and every adapter/device against them. The beauty of the script is that it’s very easy to add/modify or remove checks.

For those with any expierence in scripting or coding will understand how array’s work. The script allows you to define your own array elements to add/modify or remove checks against certain adapters/devices.

Currently, the script checks for the following:

Below is the script, and after I’ll show you how easily additional checks can be added.

#!/bin/ksh
#
# Name:
#       attrib_check.sh
#
# Description:
#       This script performs checks against attributes set
#       post AIX installation.
#
# Usage:
#       # ./attrib_check.sh
#
 
#############
# Variables #
#############
BORDER="echo ==================================================="
OK="\033[1;32mOK:\033[m"
FAIL="\033[1;31mFAIL:\033[m"
 
###############################
# Variables - Server Specific #
###############################
# Virtual SCSI attributes
vscsi[0]="vscsi_err_recov,fast_fail"
vscsi[1]="vscsi_path_to,30"
# Virtual FC attributes
fscsi[0]="dyntrk,yes"
fscsi[1]="fc_err_recov,fast_fail"
# hdisk attributes
hdisk[0]="hcheck_interval,60"
hdisk[1]="queue_depth,32"
# VMO
vmo[0]="lru_file_repage,0"
vmo[1]="strict_maxperm,0"
vmo[2]="strict_maxclient,1"
vmo[3]="maxperm%,90"
vmo[4]="minperm%,5"
vmo[5]="maxclient%,90"
 
 
 
########
# Main #
########
 
# Verify disk adapter attributes
$BORDER
echo "=> Verify disk adapter attributes"
$BORDER
 
## Check vscsi adapter attributes
echo "Virtual SCSI Adapter Attributes"
echo "-------------------------------"
lsdev | grep vscsi > /dev/null 2>&1
if [ $? -eq 0 ]; then
  echo "Server contains vscsi adapters with the following attributes:"
    for vscsi_adapter in `lsdev | grep vscsi | awk '{ print $1 }'`; do
     echo "${vscsi_adapter}:"
     count=0
       while (( $count < ${#vscsi[*]} )); do
          VSCSI_VAR1=`echo ${vscsi[$count]} | awk -F , '{ print $1 }'`
          VSCSI_VAR2=`echo ${vscsi[$count]} | awk -F , '{ print $2 }'`
 
          VSCSI_CURRENT=`lsattr -l $vscsi_adapter -a $VSCSI_VAR1 -E | awk '{ print $2 }'`
 
          if [ "$VSCSI_CURRENT" != "$VSCSI_VAR2" ]; then
            echo "   $FAIL $VSCSI_VAR1 must be set to \"$VSCSI_VAR2\""
            let count="count + 1"
          else
            echo "   $OK $VSCSI_VAR1 = $VSCSI_VAR2"
            let count="count + 1"
          fi
       done
    done
else
   echo "Server does not contain any vscsi adapters"
fi
 
 
## Check fc adapter attributes
echo "\n"
echo "Virtual FC Adapter Attributes"
echo "-----------------------------"
lsdev | grep fscsi > /dev/null 2>&1
if [ $? -eq 0 ]; then
  echo "Server contains fscsi adapters with the following attributes:"
    for fscsi_adapter in `lsdev | grep fscsi | awk '{ print $1 }'`; do
     echo "${fscsi_adapter}:"
     count=0
       while (( $count < ${#fscsi[*]} )); do
          FSCSI_VAR1=`echo ${fscsi[$count]} | awk -F , '{ print $1 }'`
          FSCSI_VAR2=`echo ${fscsi[$count]} | awk -F , '{ print $2 }'`
 
          FSCSI_CURRENT=`lsattr -l $fscsi_adapter -a $FSCSI_VAR1 -E | awk '{ print $2 }'`
 
          if [ "$FSCSI_CURRENT" != "$FSCSI_VAR2" ]; then
            echo "   $FAIL $FSCSI_VAR1 must be set to \"$FSCSI_VAR2\""
            let count="count + 1"
          else
            echo "   $OK $FSCSI_VAR1 = $FSCSI_VAR2"
            let count="count + 1"
          fi
       done
    done
else
   echo "Server does not contain any fscsi adapters"
fi
 
 
 
# Verify hdisks
echo "\n\n"
$BORDER
echo "=> Verify hdisk attributes"
$BORDER
 
## Check hdisk attributes
echo "hdisk attributes"
echo "----------------"
    for pv in `lspv | grep hdisk | awk '{ print $1 }'`; do
     echo "${pv}:"
     count=0
       while (( $count < ${#hdisk[*]} )); do
          HDISK_VAR1=`echo ${hdisk[$count]} | awk -F , '{ print $1 }'`
          HDISK_VAR2=`echo ${hdisk[$count]} | awk -F , '{ print $2 }'`
 
          HDISK_CURRENT=`lsattr -l $pv -a $HDISK_VAR1 -E | awk '{ print $2 }'`
 
          if [ "$HDISK_CURRENT" != "$HDISK_VAR2" ]; then
            echo "   $FAIL $HDISK_VAR1 must be set to \"$HDISK_VAR2\""
            let count="count + 1"
          else
            echo "   $OK $HDISK_VAR1 = $HDISK_VAR2"
            let count="count + 1"
          fi
       done
    done
 
 
 
## Check number of path's to disk
echo "\n"
echo "Disk paths"
echo "----------"
for hdisk in `lspv | grep hdisk | awk '{ print $1 }'`; do
      PATHS=`lspath -l $hdisk | wc -l | sed 's,^ *,,'`
 
      if [ $PATHS -lt 2 ]; then
        echo "${hdisk}:"
        echo "   $FAIL There is only $PATHS path to the disk"
      else
        echo "${hdisk}:"
        echo "   $OK Paths to disk = $PATHS"
      fi
done
 
 
# Check VMO settings
echo "\n\n"
$BORDER
echo "=> Check VMO settings"
$BORDER
 
count=0
  while (( $count < ${#vmo[*]} )); do
   VMO_VAR1=`echo ${vmo[$count]} | awk -F , '{ print $1 }'`
   VMO_VAR2=`echo ${vmo[$count]} | awk -F , '{ print $2 }'`
 
   VMO_CURRENT=`vmo -x $VMO_VAR1 | awk -F , '{ print $4 }'`
 
    if [ "$VMO_CURRENT" != "$VMO_VAR2" ]; then
      echo "   $FAIL $VMO_VAR1 must be set to \"$VMO_VAR2\""
      let count="count + 1"
    else
      echo "   $OK $VMO_VAR1 = $VMO_VAR2"
      let count="count + 1"
    fi
  done
 
echo "\n\n\n"
 
exit

So, let’s add another check.
For example, by default, the follow hdisk attributes are checked.

If you wanted to add an aditional check, say the hcheck_mode attribute, all you need to do is add another element into the array.

hdisk[0]="hcheck_interval,60"
hdisk[1]="queue_depth,32"
hdisk[2]="hcheck_mode,nonactive"

Line #3 shows the additional check. Just make sure you increase the array element id by one.

It’s pretty trivial to add other checks, say for ioo or no. You’ll just need to look at the current code blocks and add further ones. Or just leave a comment on this page and I can add them for you.

The following image shows the script running against a host where all attibutes validate.

The following image shows the script running against a host where some failures were detected.

I guess this script could even be modified to actually make the changes to those adapters/devices for you. Quite easily in fact…so if anyone is interested in something like that, let me know and I’ll throw something together.

As always, use/abuse/mock and enjoy.

Automatically reduce image.data to a single PV

Kristijan — Wed, 04 Apr 2012 05:26:32 +0000

I’ve been working with a client who is going through the process of migrating from physical Power 5 servers to a virtualized Power 7 environment with PowerVM. Often referred to as P2V migrations around the IBM office Due to the I/O limitations in the P5 servers, our only method of migration was to take mksysb/savevg’s of the current servers, create NIM resources out of them, and then restore onto the P7 LPAR’s.

The Power 5 rootvg consisted of two internal disks in a LVM mirror, with the other volume groups backed by either internal disk, or locally attached storage. The Power 7 which we were migrating to had it’s storage provided by a shiny DS8800. Given the boot from SAN solution we had, we no longer required two disks to form the rootvg, as all the mirroring and redundancy was being handled by the SVC’s.

To successfuly restore the Power 5 mksysb onto the new Power 7 LPAR, we needed to provide an alternate image.data file during installation which has been modified to break the mirror and reduce the volume group down to a single physical disk. You can do this manually, and I would have if there was a small number of hosts to migrate, but I was dealing with a rather large enviornment.

The below script does the following :

1. Modifies the source_disk_data stanza to a single entry.
2. Modifies the VG_SOURCE_DISK_LIST stanza to the disk set by the -d paramater.
3. Modifies the LV_SOURCE_DISK_LIST stanza to the disk set by the -d paramater.
4. Modifies the COPIES stanza to 1.
5. Modifies the PP stanza to match the LP’s value.

The script takes two values, the resulting hdisk on the other node (In my case, hdisk0 for rootvg, and hdisk1 for datavg), and the location of the image.data or vg.data file.

root@AIX #> ./modify_data.sh
Usage: ./modify_data.sh -d  -f 
        -d      Destination hdisk on P7 LPAR
        -f      image.data or vg.data file to modify
        -h      Script usage

#!/bin/ksh
#
# Name:
#	modify_data.sh
# Description:
#       This script is for use during migrations where mksysb/savevg 
#       consists of multiple physical volumes and is being restored
#       to a single volume. 
#
#       This script modifies the image.data or vg.data file to allow
#       the restore to a single volume.
#
# Usage:
#       # ./modify_data.sh -d  -f 
#               -d      Destination hdisk on P7 LPAR
#               -f      image.data or vg.data file to modify
#               -h      Script usage
#
 
#*********************************************************#
# Functions, Variables, Checks  and Command line options *#
#*********************************************************#
 
# Variables
DATE=`date +%Y%m%d`
PWD=`pwd`
 
# Usage function
_usage() {
  echo "Usage: $0 -d  -f "
  echo "        -d      Destination hdisk on P7 LPAR"
  echo "        -f      image.data or vg.data file to modify"
  echo "        -h      Script usage"
  echo ""
  echo "Example:"
  echo "       $0 -d hdisk0 -f image.data"
}
 
 
# Display usage if no command line arguments
if [ $# -eq 0 ]; then
  _usage
  exit 1
fi
 
 
# Read command line options
while getopts "d:f:h" OPTION
  do
        case ${OPTION} in
          d) HDISK="${OPTARG}";;
          f) FILE="${OPTARG}";;
          h|*) _usage;;
        esac
  done
 
 
# Check if file exists
if [ ! -f $FILE ]; then
  echo "File \"${FILE}\" does not exist"
  exit 2
fi
 
 
#*********#
# * Main *#
#*********#
 
# Make a copy of the current file
cp $FILE ${PWD}/${FILE}.${DATE}
 
# Modify source_disk_data stanza
awk '
/source_disk_data:/{c=8}!(c&&c--)
' ${FILE} > /tmp/${FILE}.tmp1
 
sed '
/source_disk_data;/ a\
\
source_disk_data:\
        PVID= 000107b0e68c43ec\
        PHYSICAL_LOCATION= U787F.001.DQM0XW6-P1-C4-T1-L3-L0\
        CONNECTION= scsi0//3,0\
        LOCATION= 00-08-00-3,0\
        SIZE_MB= 70006\
        HDISKNAME= '"$HDISK"'
' /tmp/${FILE}.tmp1 > /tmp/${FILE}.tmp2
 
# Modify VG_SOURCE_DISK_LIST, LV_SOURCE_DISK_LIST and
# COPIES stanza
sed -e '
/VG_SOURCE_DISK_LIST/ c\
        VG_SOURCE_DISK_LIST= '"$HDISK"'
' -e '
/LV_SOURCE_DISK_LIST/ c\
        LV_SOURCE_DISK_LIST= '"$HDISK"'
' -e '
/COPIES/ c\
        COPIES= 1
' /tmp/${FILE}.tmp2 > /tmp/${FILE}.tmp3
 
# Modify PP stanza
awk '
BEGIN { FS="="; modified=0}
 
/COPIES/,/BB_POLICY/ {
 
   if ($1 ~ /LPs$/) { saved = $2; }
 
   if($1 ~ /PP$/) { var = saved; }
   else { var=$2; }
 
   print $1"="var;
   modified = 1
}
{
  if (modified == 0) { print; }
 
  else { modified = 0; }
}
' /tmp/${FILE}.tmp3 > /tmp/${FILE}.tmp4
 
# Copy modified file to $PWD
cp /tmp/${FILE}.tmp4 ${PWD}/${FILE}.1PV
 
# Cleanup temp files
rm /tmp/${FILE}.tmp1
rm /tmp/${FILE}.tmp2
rm /tmp/${FILE}.tmp3
rm /tmp/${FILE}.tmp4
 
# Output
echo ""
echo "Backup of ${FILE} is located at ${PWD}/${FILE}.${DATE}"
echo ""
echo "Modified file is located at ${PWD}/${FILE}.1PV"
echo ""
 
exit 0

As you can see, the script is very ugly and messy, but it does the job. I’ve successfully used this to do a number of migrations. Feel free to use, abuse or modify it. If you do modify it though, please let me know, and I can incorporate the changes into mine

NFS cross-mounts in PowerHA/HACMP

Kristijan — Tue, 26 Jul 2011 08:38:57 +0000

Combining NFS with PowerHA we can achieve a HANFS (Highly Available Network File System). The basic concept behind this solution is that one node in the cluster mounts the resource locally, and offers that as an exported resource via a serviceable IP. Another node in the cluster is then configured to take on the resource in the event of failure.

If you’re following this, I’m taking the assumption that your cluster is already configured, you have a working IP network and have set up a shared volume group between the cluster nodes that will be handling the HANFS failover. Before we get started though, there are a few things which need to be installed/verified.

Prerequisites
The cluster.es.nfs.rte fileset needs to be installed so that PowerHA can work with HANFS.

# lslpp -l cluster.es.nfs.rte
  Fileset                      Level  State      Description
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  cluster.es.nfs.rte         5.5.0.1  COMMITTED  ES NFS Support
 
Path: /etc/objrepos
  cluster.es.nfs.rte         5.5.0.1  COMMITTED  ES NFS Support

Also, since we’re dealing with NFS, we need to make sure we have the portmapper daemon running.

# lssrc -s portmap
Subsystem         Group            PID          Status
 portmap          portmap          213160       active

To have the portmapper daemon start during system boot, you’ll need to modify /etc/rc.tcpip and uncomment the start line below.

# Start up Portmapper
start /usr/sbin/portmap "$src_running"

Scenario
Below is a table of what the HANFS layout looks like (click image to expand).

We have a Resource Group (RG_nfs) which contains an Application Server (app_nfsv4) and a Service IP Label (serviceip-nfs). There is a Shared Volume Group (haNFSvg) which contains some Shared Logical Volumes & Filesystems. The thing to take note here, is that we are only exporting two of the three filesystems that you see in the table. The /nfs_stable filesystem will be the NFSv4 stable storage path location. The below is taken from the IBM website[1]. I recommend you read it for further information on NFSv4 stable storage and PowerHA.

“Stable Storage is a file system space that is used to save the state information by the NFSv4 server. This is very crucial for maintaining NFSv4 client’s state information to facilitate smooth and transparent fallover/fallback/move of the Resource group from one node to other.”

Configuration
With the above information in hand, we’re ready to start configuring HANFS.

Step 1 (All Nodes) Set the NFS domain
The NFSv4 domain needs to be set on all cluster nodes which will be responsible for taking on the RG_nfs resource group in the event of a failure.

# chnfsdom	          <- This will show the current NFS domain
# chnfsdom [new_domain]	  <- To set a new NFS domain

Step 2 (All Nodes) Create mount points
The mount points need to be created on all cluster nodes which will be responsible for taking on the RG_nfs resource group in the event of a failure.

# mkdir -p /home/kristian
# mkdir -p /data

Take note that we’re only creating the mount points here, and not the logical volumes or filesystems.

Step 3 – Create logical volumes and filesystems
Now we create the following logical volumes:

hanfs_kris_lv
hanfs_data_lv
hanfs_nfsstable_lv

Create the logical volumes using the following path.

# smit hacmp
	-> System Management (C-SPOC)
		-> HACMP Logical Volume Management
			-> Shared Logical Volumes
				-> Add a Shared Logical Volume

Create the filesystems using the following path.

# smit hacmp
	-> System Management (C-SPOC)
		-> HACMP Logical Volume Management
			-> Shared File Systems
				-> Enhanced Journaled File Systems
					-> Add an Enhanced Journaled File System on a Previously Defined Logical Volume

We should now see the 2 filesystems which we are exporting and the stable storage filesystem apart of the shared volume group

# lsvg -l haNFSvg
haNFSvg:
LV NAME              TYPE       LPs     PPs     PVs  LV STATE      	MOUNT POINT
hanfs_kris_lv        jfs2       107     107     1    open/syncd		/hanfs_home_kristian
hanfs_data_lv        jfs2       32      32      1    open/syncd  	/hanfs_data
hanfs_nfsstable_lv   jfs2       75      75      1    open/syncd  	/nfs_stable

Step 4 – Configure the Application Server
Luckily for us, IBM have made this step rather easy and have provided start, stop and monitor scripts. The location of these scripts are below.

/usr/es/sbin/cluster/apps/clas_nfsv4/start
/usr/es/sbin/cluster/apps/clas_nfsv4/stop
/usr/es/sbin/cluster/apps/clam_nfsv4/monitor

Create an Application Server using the following path.

# smit hacmp
	-> Extended Configuration
		-> Extended Resource Configuration
			-> HACMP Extended Resources Configuration
				-> Configure HACMP Applications Servers
					-> Add an Application Server

To configure Application Server monitoring, use the following path.

# smit hacmp
	-> Extended Configuration
		-> Extended Resource Configuration
			-> HACMP Extended Resources Configuration
				-> Configure HACMP Application Monitoring
					-> Configure Custom Application Monitors
						-> Add a Custom Application Monitor

Step 5 – Configure the Resource Group for HANFS
The next step is to configure the RG_nfs resource group with the values needed for HANFS.

Modify the resource group using the following path.

# smit hacmp
	-> Extended Configuration
		-> Extended Resource Configuration
			-> HACMP Extended Resource Group Configuration
				-> Change/Show Resources and Attributes for a Resource Group

Service IP Labels/Addresses				[serviceip-nfs]
Application Servers					[app_nfsv4]
 
Filesystems (empty is ALL for VGs specified)		[/hanfs_home_kristian /hanfs_data /nfs_stable]
Filesystems Consistency Check				fsck
Filesystems Recovery Method				sequential
Filesystems mounted before IP configured		true
Filesystems/Directories to Export (NFSv2/3)		[]
 
Filesystems/Directories to Export (NFSv4)		[/hanfs_home_kristian /hanfs_data]
Stable Storage Path (NFSv4)				[/nfs_stable]
Filesystems/Directories to NFS Mount			[/home/kristian;/hanfs_home_kristian /data;/hanfs_data]

Most vaules are self explainatory. We set “Filesystems mounted before IP configured” to true so we prevent access from clients before the filesystems are ready. We also specify mount points in the following format [mount point];[exported filesystem]

Step 6 – HANFS exports file
Just like NFS has /etc/exports, HANFS has /usr/es/sbin/cluster/etc/exports. If you need to specify NFS options, you MUST use /usr/es/sbin/cluster/etc/exports and not /etc/exports. For help creating the exports file, you can use “smit mknfsexp”

* Pathname of directory to export			[]
  Anonymous UID						[-2]
  Public filesystem?					no
* Export directory now, system restart or both		both
  Pathname of alternate exports file			[/usr/es/sbin/cluster/etc/exports]
  ...
  ...

Step 7 – Synchronize the cluster
We now need to synchronize our changes to the other cluster nodes

# smit hacmp
	-> Extended Configuration
		-> Extended Verification and Synchronization

Step 8 – Bring the Resource Group online
We now bring the resource group online. It’s a good idea at this stage to tail the hacmp.out file to see any errors.

To bring the resource group online.

# smit hacmp
	->  System Management (C-SPOC)
		-> HACMP Resource Group and Application Management
			-> Bring a Resource Group Online

# tail -f /var/hacmp/log/hacmp.out
...
+RG_nfs:cl_activate_nfs(.110):/home/kristian;/hanfs_home_kristian[nfs_mount+102] : Attempt 0/5 to NFS-mount at Jul 26 11:01:21.000
+RG_nfs:cl_activate_nfs(.110):/home/kristian;/hanfs_home_kristian[nfs_mount+103] mount -o vers=4,hard,intr serviceip-nfs:/hanfs_home_kristian /home/kristian

You should now see the filesystems mounted.

# mount
  node       	mounted        		   mounted over    		 vfs       date        options
--------      --------------  	          ---------------  		------ ------------ ---------------
...
		/dev/hanfs_kris_lv 	   /hanfs_home_kristian 	jfs2   Jul 26 14:36 rw,log=INLINE
		/dev/hanfs_data_lv 	   /hanfs_data      		jfs2   Jul 26 14:36 rw,log=INLINE
		/dev/hanfs_nfsstable_lv    /nfs_stable       		jfs2   Jul 26 14:36 rw,log=INLINE
serviceip-nfs 	/hanfs_home_kristian	   /home/kristian    		nfs4   Jul 26 14:36 vers=4,hard,intr
serviceip-nfs 	/hanfs_data 		   /data 			nfs4   Jul 26 14:36 vers=4,hard,intr

What you’re seeing above is the output from the mount command ran from the cluster node which currently has the RG_nfs resource group ONLINE. You’ll notice that it has the shared logical volumes and filesystems mounted, then a local NFS export from itself to mount /home/kristian and /data. On the other cluster nodes, you will only see the NFS mounts.

There you have it, NFS cross-mounts with PowerHA.

[1] – http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.hacmp.plangd/ha_plan_nfs_stable_storage.htm

Mac OSX Lion and AFP shares

Kristijan — Sun, 24 Jul 2011 06:56:50 +0000

Like most Apple Mac users this week, I updated from 10.6 Snow Leopard to 10.7 Lion of Apples operating system. Everything seemed to go fine, I noticed a few little quirks, but the one that gave me the biggest “Oh no, can I go back!?” was when I tried mounting my AFP share on my FreeNAS server.

iMac:~ kristijan$ mount_afp -i afp://kristijan@nas/Media ./Media/
Password: 
mount_afp: AFPMountURL returned error -5002, errno is -5002

For those that use the “Connect to server…” option from Finder would have come across the following pop up error message: “The version of the server you are trying to connect to is not supported. Please contact your system administrator to resolve the problem.”

So after some searching around, I found out that OSX Lion no longer supports AFP (and SMB/CIFS) servers which use the DHCAST128 authentication method. Apple flagged this is a security risk, and outright disabled it in Lion. Now I could have just scrapped AFP all together and started using NFS shares, but AFP just works a hell of a lot better with OSX clients.

Some more searching found this page[1], and a solution to my problem. Full credits to this solution go to Alexander Wilde.

Follow the below to enable DHCAST128 under Mac OSX Lion

Launch Terminal.app (/Applications/Utilities/Terminal.app) and run the following two commands.

iMac:~ kristijan$ sudo chmod o+w /Library/Preferences
Password:
iMac:~ kristijan$ defaults write /Library/Preferences/com.apple.AppleShareClient afp_host_prefs_version -int 1

Now restart your computer.

Launch Terminal.app or Finder and attempt to mount the AFP share again. This will fail, but what it will do is recreate the full preferences file.

iMac:~ kristijan$ mount_afp -i afp://kristijan@nas/Media ./Media/
Password: 
mount_afp: AFPMountURL returned error -5002, errno is -5002

Now run the following two commands.

iMac:~ kristijan$ sudo defaults write /Library/Preferences/com.apple.AppleShareClient afp_disabled_uams -array "Cleartxt Passwrd" "MS2.0" "2-Way Randnum exchange"
Password:
iMac:~ kristijan$ sudo chmod o-w /Library/Preferences

Restart your computer and mount the AFP share again, it should now work.

[1] – http://www.alexanderwilde.com/2011/04/os-x-lion-connection-error-with-afp-and-workaround

VIO: Virtual Media Library

Kristijan — Thu, 09 Jun 2011 07:30:32 +0000

The first LPAR that I normally build post VIOS installation is a NIM LPAR. Once you have your NIM server setup, building the rest of your LPAR’s becomes a much simpler task. However, what do you do when there isn’t an existing NIM environment, or you don’t have access to the physical servers to insert a disc to perform the AIX install? This is where the Virtual Media Library comes to the rescue. The Virtual Media Library is a feature of the Virtual I/O Server which allows you to present an ISO image to underlying LPAR’s. The best thing about the Virtual Media Library is that it’s very quick and simple to set up.

From here on out, I’ll be referring to the Virtual Media Library as VML.

Getting right into it, we first want to define the VML. Now you can create a new volume group for the VML, or simply use rootvg. If you do decide to use rootvg, just be sure to exclude the filesystem when you do your VIO mksysb backups. The other paramater that we need to specify is the size of the VML. Make it big enough to cater for all your ISO volumes that you wish to present to underlying LPAR’s. In the example below, I’ve gone with 10G.

$ mkrep -sp rootvg -size 10G
Virtual Media Repository Created
Repository created within "VMLibrary" logical volume

We’ve now created an empty VML repository which backs onto the /var/vio/VMLibrary filesystem. You can view the VML repository by running the lsrep command.

Now it’s time to upload your ISO images over to the VIOS. At this stage, I’ll bring up a root shell in the VIOS and change the root password. This will allow me to SCP across the ISO images into the /var/vio/VMLibrary folder.

$ oem_setup_env
# passwd

Upload your ISO images into /var/vio/VMLibrary

# ls -l /var/vio/VMLibrary/AIX-6100-06-01-Disk1.iso
-r--------    1 root     system   4236148736 Apr 06 18:07 /var/vio/VMLibrary/AIX-6100-06-01-Disk1.iso

Check the VML repository to see the new image available for use.

$ lsrep
Size(mb) Free(mb) Parent Pool         Parent Size      Parent Free
   10198     6158 rootvg                   279552           236288
 
Name                                    File Size Optical         Access
AIX-6100-06-01-Disk1.iso                     4040 None            ro

Great, so we now have some media in the VML library ready to be used. What we need to do from here is create a File Backed Optical Device for the particular LPAR you want cd0 to appear on.

$ mkvdev -fbo -vadapter vhost1
vtopt0 Available

Now let’s load the image from the VML into vtopt0.

$ loadopt -vtd vtopt0 -disk AIX-6100-06-01-Disk1.iso
$ lsmap -vadapter vhost1
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost1          U9133.55A.066EB4H-V1-C34                     0x00000000
 
VTD                   vtopt0
Status                Available
LUN                   0x8200000000000000
Backing device        /var/vio/VMLibrary/AIX-6100-06-01-Disk1.iso
Physloc
Mirrored              N/A

Now you can boot the LPAR off CD/DVD and perform the AIX install.

Two other useful commands to know:

To unload the image.

$ unloadopt -vtd vtopt0

If you’re prompted for another disk (for example, disc 2 during the AIX install).

$ loadopt -f -vtd vtopt0 -disk AIX-6100-06-01-Disk2.iso

There you have it, a quick and simple way of getting AIX installed onto a LPAR if you don’t have any other method of installation. Another thing to take note of, is that you can load any ISO image into the VML, it doesn’t have to be an AIX installation image. Quite useful for when you have to install any bit of software and all you have is the ISO image.

On that note though, the loopmount command was added in AIX 6.1 TL4. This allows you to mount ISO images directly from AIX without using the VML.

# loopmount -i image.iso -o "-V cdrfs -o ro" -m /mountpoint

Good luck, enjoy, and any questions can be popped into the comments below.

Upgrading your AIX environment using multibos

Kristijan — Thu, 31 Mar 2011 00:52:56 +0000

If you haven’t clued on by now, AIX is my operating system of choice. I’m sure there are many arguments for and against, but looking at it from a systems administrator view point, it’s second to none. Anyway, that’s a story for another time. What I’m writing about today is patching your AIX installations using multibos. Now before I get started, I’ll stress two points:

1) Chris Gibson has already written two great (and far more in depth) articles on multibos, which I recommend reading before going any further (Links: here and here).
2) This blog of mine serves two purposes; To share some of the things I come about during my sysadmin activities and secondly also a dumping ground for documents I write for myself. This post is the latter, as multibos isn’t something that I use daily (maybe a few times a year), and really, who’s going to remember all those syntax switches

Now, assuming you’ve read both articles, I don’t need to go into the benefits of multibos and I’ll dive straight into the how. I’ll be working on upgrading an AIX instance running 6100-05-01-1016 to the latest ML which at the time of writing is 6100-06-04-1112

PREREQUISITES
First things first, we need to make sure the current environment is ready to be copied. The two main things that I look for are:

1. Make sure the current environment doesn’t have any missing filesets.

root@AIX / > oslevel -s
6100-05-01-1016
root@AIX / > instfix -i | grep ML
    All filesets for 6100-00_AIX_ML were found.
    All filesets for 6100-01_AIX_ML were found.
    All filesets for 6100-02_AIX_ML were found.
    All filesets for 6100-03_AIX_ML were found.
    All filesets for 6100-04_AIX_ML were found.
    All filesets for 6100-05_AIX_ML were found.

**Note: If a particular ML doesn’t return positive (e.g. 6100-05_AIX_ML), use instfix -ciqk 6100-05_AIX_ML | grep “:-:” to see the fileset causing issues and rectify it.

2. Make sure all filesets have been committed. Now I’m not entirely sure if this is a requirement, but I prefer to work with an as clean environment as possible.

root@AIX / > lslpp -l | grep -i applied

**Note: If you have software which is in an applied state, commit it.

Now that we’re happy with the state of the environment, you need to verify that there is enough room in rootvg to make copies of the following logical volumes [/ (hd4), /usr (hd2), /var (hd9var), /opt (hd10opt) and the boot logical volume (hd5)].

Once we’ve satisfied the above requirement, we can get started with the multibos work. I’m not going to dissect each command, so I suggest reading the multibos man pages to know what each switch does. The only one that I’ll explain is -p, which does a preview of the multibos command executed.

MULTIBOS
1. Remove any previous standby multibos environments. The below command will return a FAILURE status if you don’t have a standby multibos environment. If you’re using multibos for the first time, this will be the case.

root@AIX / > multibos -RX
Initializing multibos methods ...
Initializing log /etc/multibos/logs/op.alog ...
Gathering system information ...
multibos: 0565-077 Unable to locate standby BOS.
 
Log file is /etc/multibos/logs/op.alog
Return Status: FAILURE

2. Create the standby BOS. New logical volumes and filesystems will be created prefixed and with bos_. This can take a while to complete.

This will preview the creation of the multibos environment and list all the logical volumes that will be copied.

root@AIX / > multibos -sXp
...
...
Log file is /etc/multibos/logs/op.alog
Return Status = SUCCESS

You can also view the log file using the alog command.

root@AIX / > alog -of /etc/multibos/logs/op.alog

If everything completed successfully, run the command again without the preview.

root@AIX / > multibos -sX

Once complete, you’ll notice that the new logical volumes will be prefixed with bos_

3. Once complete, you can drop into a multibos shell and check oslevel, it should be the same as the host OS.

The below command will mount all the filesystems required and drop you in a MULTIBOS> prompt

root@AIX / > multibos -S
...
...
MULTIBOS> oslevel -s
6100-05-01-1016

To umount all the filesystems and break out of the multibos environment.

MULTIBOS> exit

4. Apply the TL/ML to the multibos environment.

The below command will tell multibos to apply the updates from the specified location (which is the location that your new ML resides).

root@AIX / > multibos -Xac -l /home/kristijan/6100-06-04

Drop into the multibos shell again and check that the ML has been successfully applied.

root@AIX / > multibos -S
MULTIBOS> oslevel -s
6100-06-04-1112
MULTIBOS> instfix -i | grep ML
    	All filesets for 6100-00_AIX_ML were found.
    	All filesets for 6.1.0.0_AIX_ML were found.
    	All filesets for 6100-01_AIX_ML were found.
    	All filesets for 6100-02_AIX_ML were found.
    	All filesets for 6100-03_AIX_ML were found.
    	All filesets for 6100-04_AIX_ML were found.
    	All filesets for 6100-05_AIX_ML were found.
    	All filesets for 6100-06_AIX_ML were found.
MULTIBOS> exit

5. Verify that the bootlist now contains blv=bos_hd5. bos_hd5 is the location of the new boot logical volume. It will be at the top of the list when you run the bootlist command.

root@AIX / > bootlist -m normal -o
hdisk0 blv=bos_hd5 pathid=0
hdisk0 blv=hd5 pathid=0

You can now reboot into the new environment and verify that all went to plan.

6. If you need to back out of the change for any reason, just change the bootlist order back to blv=hd5 and reboot. Now, while this doesn’t bother me, it may bother others. You’ll notice after the reboot that your environment is now using the bos_ logical volumes. The only way to get your environment back to how it was using the original logical volume names is to create another multibos environment, and reboot into it. Since your current boot environment is using the bos_ logical volumes, it will create it using the original logical volume names (i.e. without the bos_). For those that want to do that, redo step 2 and reboot. However, I suggest not doing this for a week or so to be sure you’re happy with the new changes, as creating a new multibos environment will remove the backup environment with the previous ML.

As always, post comments/questions below.

CSM Commands (dsh, dping, dcp and dshbak) for AIX

Kristijan — Mon, 28 Feb 2011 03:28:45 +0000

Cluster Systems Management (CSM) software provides a bunch of tools to manage from a single point of control your AIX environment. There are a number of commands available[1] but the main four which I find quite useful are dsh, dping, dcp and dshbak. In this post, I’ll go over each command and some of its uses.

Before we begin, we need to select a server which is going to be the point of control. While this can be any AIX server on the network, it’s mainly common to have your NIM server as the point of control. In the case of dsh and dcp, we’re going to be using SSH, so keys[2] need to be setup between your point of control server and the rest of your environment.

DSH/DSHBAK
Distributed Shell (dsh) allows you to concurrently run commands on remote nodes. It requires very little setup, but it’s one of those hidden AIX commands that few administrators know about. As you may have already guessed, dsh comes with the CSM client filesets. Specifically, it comes from the csm.dsh fileset which creates a symbolic link – /usr/bin/dsh -> /opt/csm/bin/dsh

root@AIX / > lslpp -w /usr/bin/dsh
  File                                        Fileset               Type
  ----------------------------------------------------------------------------
  /usr/bin/dsh                                csm.dsh               Symlink

There are a few environmental variables that we need to setup first.

DSH_NODE_RSH=/usr/bin/ssh – By default, dsh uses rsh to execute remote commands. If you don’t know why using rsh is a bad idea, hand in your sysadmin card now. We need to tell dsh to execute all remote commands via ssh.
DCP_NODE_RCP=/usr/bin/scp – As above, dcp uses rcp to send files remotely, so lets change that to use scp
DSH_NODE_OPTS=-q – This passes the -q option to ssh to suppress the banner message from showing. This just makes the output much easier to read when working with a number of nodes.
DSH_NODE_LIST=/.dsh/model – You can feed a list of nodes to dsh a number of ways, but I find using text files the best way to manage it.

Most places have a typical dev/mod/prod setup which can consist of a number of servers within each environment. I have four node text files in my /.dsh directory.

-rw-rw-r--    1 root     system          577 Jan 11 10:19 all
-rw-r--r--    1 root     system           71 Feb 12 13:06 dev
-rw-r--r--    1 root     system           75 Oct 28 10:38 mod
-rw-r--r--    1 root     system          106 Jan 25 13:00 prod

This way I can run commands on all the nodes in a specific environment, or across all nodes. The easiest way to switch between lists is by using aliases.

In my .kshrc file

alias dshmod='export DSH_NODE_LIST=/.dsh/mod'
alias dshprod='export DSH_NODE_LIST=/.dsh/prod'
alias dshdev='export DSH_NODE_LIST=/.dsh/dev'
alias dshall='export DSH_NODE_LIST=/.dsh/all'

The contents of the node text files is each node on a single line.

root@AIX / > cat /.dsh/mod
SERVER1
SERVER2
SERVER3

That’s it for the configuration. You can run dsh -q to show the current configured variables. Look at the man page for dsh to see further configurable options and the IBM website[3].

Let’s see if it works.

root@AIX / > dshmod
root@AIX / > dsh date
SERVER1: Mon Feb 28 12:59:34 EETDT 2011
SERVER2: Mon Feb 28 12:53:58 EETDT 2011
SERVER3: Mon Feb 28 12:56:51 EETDT 2011

If you only wanted to run the command on a select number of hosts, you can use the -w switch.

root@AIX / > dsh -w SERVER1,SERVER3 "date"
SERVER1: Mon Feb 28 13:01:20 EETDT 2011
SERVER3: Mon Feb 28 12:55:43 EETDT 2011

We can make that output look a bit neater by piping through the dshbak command.

root@AIX / > dsh -w SERVER1,SERVER3 "date" | dshbak
HOST: SERVER1
-------------
Mon Feb 28 13:17:17 EETDT 2011
 
HOST: SERVER3
-------------
Mon Feb 28 13:11:41 EETDT 2011

DCP
Just like the dsh command, dcp allows you to send files concurrently to multiple hosts. dcp requires the same environmental setup as dsh. Seeing as that’s already setup, we can use it straight away.

root@AIX / > lmktemp test 1M
test
root@AIX / > dcp test /
root@AIX / > dsh ls -l /test
SERVER1: -rw-r--r--   1 root     system      1048576 Feb 28 13:32 /test
SERVER2: -rw-r--r--   1 root     system      1048576 Feb 28 13:32 /test
SERVER3: -rw-r--r--   1 root     system      1048576 Feb 28 13:32 /test

What I did above is created a 1M file called test, used dcp to send it to my hosts, then verified the file was on each node using dsh. You can find more about dcp from the man pages and the IBM website[4].

DPING
The dping command allows you to ping nodes in parallel (with the option of serially). Now, it doesn’t read the DSH_NODE_LIST variable like dsh/dcp, but you can pass the -f option and tell it to.

root@AIX / > dping -f /.dsh/mod
SERVER1: ping (alive)
SERVER2: ping (alive)
SERVER3: ping (alive)
SERVER4: noping (unreachable)

If you only want to ping a select few hosts, then you can use the -n switch.

root@AIX / > dping -n SERVER1,SERVER3
SERVER1: ping (alive)
SERVER3: ping (alive)

You can find more on dping by reading the man pages and on the IBM website[5]. Like dsh, you can also pipe the output of dping through dshbak.

If you have any questions on the above, use the comment section below.

[1] – http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.csm16.cmds.doc/am7cm_csmcomm.html
[2] – http://www.ece.uci.edu/~chou/ssh-key.html
[3] – http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.csm16.cmds.doc/am7cm_dsh.html
[4] – http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.csm16.cmds.doc/am7cm_dcp.html
[5] – http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.csm16.cmds.doc/am7cm_dping.html

Quickly show memory usage under AIX

Kristijan — Wed, 05 Jan 2011 23:36:40 +0000

Very quick and basic shell script to show total/free/used memory on a host running AIX.

kristijan@AIX > ./meminfo
 
Memory Information
==================
total memory = 1884 MB
free memory  = 94 MB
used memory  = 1790 MB

Script uses standard AIX command line utilities to work out all the values. I haven’t tested it across all versions of AIX (I wrote this on AIX 6.1), but I believe it should work fine. Let me know in the comments if it doesn’t.

#!/usr/bin/ksh
#
# Quick view of memory usage under AIX
#
 
USED=`svmon -G | head -2 | tail -1 | awk '{ print $3 }'`
USED=`expr $USED / 256`
TOTAL=`lsattr -El sys0 -a realmem | awk '{ print $2 }'`
TOTAL=`expr $TOTAL / 1000`
FREE=`expr $TOTAL - $USED`
 
echo "\nMemory Information"
echo "=================="
echo "total memory = $TOTAL MB"
echo "free memory  = $FREE MB"
echo "used memory  = $USED MB"
 
exit 0

Mounting external USB NTFS drive to FreeNAS

Kristijan — Thu, 16 Dec 2010 09:15:59 +0000

This is a quick post on how to mount an external USB NTFS formatted hard drive (or memory stick) to FreeNAS. All that’s needed is to load the fuse driver, but it had me stumped for a bit until I found it in a forum post.

First things first, let’s plug in our drive and find out the device ID FreeNAS assigns it.

# dmesg
...
umass1:  on uhub4
da1 at umass-sim1 bus 1 target 0 lun 0
da1:  Fixed Direct Access SCSI-0 device 
da1: 40.000MB/s transfers
da1: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C)

Now we create a mount point for the drive and try mount it like we normally would.

# mkdir /mnt/usb
# mount -t ntfs /dev/da1s1 /mnt/usb/

You’ll receive the following error:
fuse: failed to open fuse device: No such file or directory

Now lets load the fuse driver and try again.

# kldload fuse
# mount -t ntfs /dev/da1s1 /mnt/usb/
# ls -l /mnt/usb
total 4589868
drwxrwxrwx  1 root  wheel        4096 Sep 25 17:48 $RECYCLE.BIN
drwxrwxrwx  1 root  wheel           0 Nov 27 14:25 .Trashes
-rwxrwxrwx  1 root  wheel        4096 Nov 22 18:15 ._.Trashes
...

SUCCESS!

It’s probably easier to load the driver on boot instead of loading it manually like I have above. I haven’t tried it, but it should work if you call it from loader.conf. I can reboot my NAS at the moment, but if someone wants to give it a shot and post to the comments, that would be great. You can follow instructions on how to edit loader.conf in a previous post [1] of mine.

[1] – http://www.kristijan.org/2010/11/freenas-realtek-network-card/