If you’re following this, I’m taking the assumption that your cluster is already configured, you have a working IP network and have set up a shared volume group between the cluster nodes that will be handling the HANFS failover. Before we get started though, there are a few things which need to be installed/verified.

Prerequisites
The cluster.es.nfs.rte fileset needs to be installed so that PowerHA can work with HANFS.

# lslpp -l cluster.es.nfs.rte
  Fileset                      Level  State      Description
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  cluster.es.nfs.rte         5.5.0.1  COMMITTED  ES NFS Support
 
Path: /etc/objrepos
  cluster.es.nfs.rte         5.5.0.1  COMMITTED  ES NFS Support

Also, since we’re dealing with NFS, we need to make sure we have the portmapper daemon running.

# lssrc -s portmap
Subsystem         Group            PID          Status
 portmap          portmap          213160       active

To have the portmapper daemon start during system boot, you’ll need to modify /etc/rc.tcpip and uncomment the start line below.

# Start up Portmapper
start /usr/sbin/portmap "$src_running"

Scenario
Below is a table of what the HANFS layout looks like (click image to expand).

We have a Resource Group (RG_nfs) which contains an Application Server (app_nfsv4) and a Service IP Label (serviceip-nfs). There is a Shared Volume Group (haNFSvg) which contains some Shared Logical Volumes & Filesystems. The thing to take note here, is that we are only exporting two of the three filesystems that you see in the table. The /nfs_stable filesystem will be the NFSv4 stable storage path location. The below is taken from the IBM website[1]. I recommend you read it for further information on NFSv4 stable storage and PowerHA.

“Stable Storage is a file system space that is used to save the state information by the NFSv4 server. This is very crucial for maintaining NFSv4 client’s state information to facilitate smooth and transparent fallover/fallback/move of the Resource group from one node to other.”

Configuration
With the above information in hand, we’re ready to start configuring HANFS.

Step 1 (All Nodes) Set the NFS domain
The NFSv4 domain needs to be set on all cluster nodes which will be responsible for taking on the RG_nfs resource group in the event of a failure.

# chnfsdom	          <- This will show the current NFS domain
# chnfsdom [new_domain]	  <- To set a new NFS domain

Step 2 (All Nodes) Create mount points
The mount points need to be created on all cluster nodes which will be responsible for taking on the RG_nfs resource group in the event of a failure.

# mkdir -p /home/kristian
# mkdir -p /data

Take note that we’re only creating the mount points here, and not the logical volumes or filesystems.

Step 3 – Create logical volumes and filesystems
Now we create the following logical volumes:

• hanfs_kris_lv
• hanfs_data_lv
• hanfs_nfsstable_lv

Create the logical volumes using the following path.

# smit hacmp
	-> System Management (C-SPOC)
		-> HACMP Logical Volume Management
			-> Shared Logical Volumes
				-> Add a Shared Logical Volume

Create the filesystems using the following path.

# smit hacmp
	-> System Management (C-SPOC)
		-> HACMP Logical Volume Management
			-> Shared File Systems
				-> Enhanced Journaled File Systems
					-> Add an Enhanced Journaled File System on a Previously Defined Logical Volume

We should now see the 2 filesystems which we are exporting and the stable storage filesystem apart of the shared volume group

# lsvg -l haNFSvg
haNFSvg:
LV NAME              TYPE       LPs     PPs     PVs  LV STATE      	MOUNT POINT
hanfs_kris_lv        jfs2       107     107     1    open/syncd		/hanfs_home_kristian
hanfs_data_lv        jfs2       32      32      1    open/syncd  	/hanfs_data
hanfs_nfsstable_lv   jfs2       75      75      1    open/syncd  	/nfs_stable

Step 4 – Configure the Application Server
Luckily for us, IBM have made this step rather easy and have provided start, stop and monitor scripts. The location of these scripts are below.

• /usr/es/sbin/cluster/apps/clas_nfsv4/start
• /usr/es/sbin/cluster/apps/clas_nfsv4/stop
• /usr/es/sbin/cluster/apps/clam_nfsv4/monitor

Create an Application Server using the following path.

# smit hacmp
	-> Extended Configuration
		-> Extended Resource Configuration
			-> HACMP Extended Resources Configuration
				-> Configure HACMP Applications Servers
					-> Add an Application Server

To configure Application Server monitoring, use the following path.

# smit hacmp
	-> Extended Configuration
		-> Extended Resource Configuration
			-> HACMP Extended Resources Configuration
				-> Configure HACMP Application Monitoring
					-> Configure Custom Application Monitors
						-> Add a Custom Application Monitor

Step 5 – Configure the Resource Group for HANFS
The next step is to configure the RG_nfs resource group with the values needed for HANFS.

Modify the resource group using the following path.

# smit hacmp
	-> Extended Configuration
		-> Extended Resource Configuration
			-> HACMP Extended Resource Group Configuration
				-> Change/Show Resources and Attributes for a Resource Group

Service IP Labels/Addresses				[serviceip-nfs]
Application Servers					[app_nfsv4]
 
Filesystems (empty is ALL for VGs specified)		[/hanfs_home_kristian /hanfs_data /nfs_stable]
Filesystems Consistency Check				fsck
Filesystems Recovery Method				sequential
Filesystems mounted before IP configured		true
Filesystems/Directories to Export (NFSv2/3)		[]
 
Filesystems/Directories to Export (NFSv4)		[/hanfs_home_kristian /hanfs_data]
Stable Storage Path (NFSv4)				[/nfs_stable]
Filesystems/Directories to NFS Mount			[/home/kristian;/hanfs_home_kristian /data;/hanfs_data]

Most vaules are self explainatory. We set “Filesystems mounted before IP configured” to true so we prevent access from clients before the filesystems are ready. We also specify mount points in the following format [mount point];[exported filesystem]

Step 6 – HANFS exports file
Just like NFS has /etc/exports, HANFS has /usr/es/sbin/cluster/etc/exports. If you need to specify NFS options, you MUST use /usr/es/sbin/cluster/etc/exports and not /etc/exports. For help creating the exports file, you can use “smit mknfsexp”

* Pathname of directory to export			[]
  Anonymous UID						[-2]
  Public filesystem?					no
* Export directory now, system restart or both		both
  Pathname of alternate exports file			[/usr/es/sbin/cluster/etc/exports]
  ...
  ...

Step 7 – Synchronize the cluster
We now need to synchronize our changes to the other cluster nodes

# smit hacmp
	-> Extended Configuration
		-> Extended Verification and Synchronization

Step 8 – Bring the Resource Group online
We now bring the resource group online. It’s a good idea at this stage to tail the hacmp.out file to see any errors.

To bring the resource group online.

# smit hacmp
	->  System Management (C-SPOC)
		-> HACMP Resource Group and Application Management
			-> Bring a Resource Group Online

# tail -f /var/hacmp/log/hacmp.out
...
+RG_nfs:cl_activate_nfs(.110):/home/kristian;/hanfs_home_kristian[nfs_mount+102] : Attempt 0/5 to NFS-mount at Jul 26 11:01:21.000
+RG_nfs:cl_activate_nfs(.110):/home/kristian;/hanfs_home_kristian[nfs_mount+103] mount -o vers=4,hard,intr serviceip-nfs:/hanfs_home_kristian /home/kristian

You should now see the filesystems mounted.

# mount
  node       	mounted        		   mounted over    		 vfs       date        options
--------      --------------  	          ---------------  		------ ------------ ---------------
...
		/dev/hanfs_kris_lv 	   /hanfs_home_kristian 	jfs2   Jul 26 14:36 rw,log=INLINE
		/dev/hanfs_data_lv 	   /hanfs_data      		jfs2   Jul 26 14:36 rw,log=INLINE
		/dev/hanfs_nfsstable_lv    /nfs_stable       		jfs2   Jul 26 14:36 rw,log=INLINE
serviceip-nfs 	/hanfs_home_kristian	   /home/kristian    		nfs4   Jul 26 14:36 vers=4,hard,intr
serviceip-nfs 	/hanfs_data 		   /data 			nfs4   Jul 26 14:36 vers=4,hard,intr

What you’re seeing above is the output from the mount command ran from the cluster node which currently has the RG_nfs resource group ONLINE. You’ll notice that it has the shared logical volumes and filesystems mounted, then a local NFS export from itself to mount /home/kristian and /data. On the other cluster nodes, you will only see the NFS mounts.

There you have it, NFS cross-mounts with PowerHA.

[1] – http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.hacmp.plangd/ha_plan_nfs_stable_storage.htm

iMac:~ kristijan$ mount_afp -i afp://kristijan@nas/Media ./Media/
Password: 
mount_afp: AFPMountURL returned error -5002, errno is -5002

For those that use the “Connect to server…” option from Finder would have come across the following pop up error message: “The version of the server you are trying to connect to is not supported. Please contact your system administrator to resolve the problem.”

So after some searching around, I found out that OSX Lion no longer supports AFP (and SMB/CIFS) servers which use the DHCAST128 authentication method. Apple flagged this is a security risk, and outright disabled it in Lion. Now I could have just scrapped AFP all together and started using NFS shares, but AFP just works a hell of a lot better with OSX clients.

Some more searching found this page[1], and a solution to my problem. Full credits to this solution go to Alexander Wilde.

Follow the below to enable DHCAST128 under Mac OSX Lion

Launch Terminal.app (/Applications/Utilities/Terminal.app) and run the following two commands.

iMac:~ kristijan$ sudo chmod o+w /Library/Preferences
Password:
iMac:~ kristijan$ defaults write /Library/Preferences/com.apple.AppleShareClient afp_host_prefs_version -int 1

Now restart your computer.

Launch Terminal.app or Finder and attempt to mount the AFP share again. This will fail, but what it will do is recreate the full preferences file.

iMac:~ kristijan$ mount_afp -i afp://kristijan@nas/Media ./Media/
Password: 
mount_afp: AFPMountURL returned error -5002, errno is -5002

Now run the following two commands.

iMac:~ kristijan$ sudo defaults write /Library/Preferences/com.apple.AppleShareClient afp_disabled_uams -array "Cleartxt Passwrd" "MS2.0" "2-Way Randnum exchange"
Password:
iMac:~ kristijan$ sudo chmod o-w /Library/Preferences

Restart your computer and mount the AFP share again, it should now work.

[1] – http://www.alexanderwilde.com/2011/04/os-x-lion-connection-error-with-afp-and-workaround

Getting right into it, we first want to define the VML. Now you can create a new volume group for the VML, or simply use rootvg. If you do decide to use rootvg, just be sure to exclude the filesystem when you do your VIO mksysb backups. The other paramater that we need to specify is the size of the VML. Make it big enough to cater for all your ISO volumes that you wish to present to underlying LPAR’s. In the example below, I’ve gone with 10G.

$ mkrep -sp rootvg -size 10G
Virtual Media Repository Created
Repository created within "VMLibrary" logical volume

We’ve now created an empty VML repository which backs onto the /var/vio/VMLibrary filesystem. You can view the VML repository by running the lsrep command.

Now it’s time to upload your ISO images over to the VIOS. At this stage, I’ll bring up a root shell in the VIOS and change the root password. This will allow me to SCP across the ISO images into the /var/vio/VMLibrary folder.

$ oem_setup_env
# passwd

Upload your ISO images into /var/vio/VMLibrary

# ls -l /var/vio/VMLibrary/AIX-6100-06-01-Disk1.iso
-r--------    1 root     system   4236148736 Apr 06 18:07 /var/vio/VMLibrary/AIX-6100-06-01-Disk1.iso

Check the VML repository to see the new image available for use.

$ lsrep
Size(mb) Free(mb) Parent Pool         Parent Size      Parent Free
   10198     6158 rootvg                   279552           236288
 
Name                                    File Size Optical         Access
AIX-6100-06-01-Disk1.iso                     4040 None            ro

Great, so we now have some media in the VML library ready to be used. What we need to do from here is create a File Backed Optical Device for the particular LPAR you want cd0 to appear on.

$ mkvdev -fbo -vadapter vhost1
vtopt0 Available

Now let’s load the image from the VML into vtopt0.

$ loadopt -vtd vtopt0 -disk AIX-6100-06-01-Disk1.iso
$ lsmap -vadapter vhost1
SVSA            Physloc                                      Client Partition ID
--------------- -------------------------------------------- ------------------
vhost1          U9133.55A.066EB4H-V1-C34                     0x00000000
 
VTD                   vtopt0
Status                Available
LUN                   0x8200000000000000
Backing device        /var/vio/VMLibrary/AIX-6100-06-01-Disk1.iso
Physloc
Mirrored              N/A

Now you can boot the LPAR off CD/DVD and perform the AIX install.

Two other useful commands to know:

To unload the image.

$ unloadopt -vtd vtopt0

If you’re prompted for another disk (for example, disc 2 during the AIX install).

$ loadopt -f -vtd vtopt0 -disk AIX-6100-06-01-Disk2.iso

There you have it, a quick and simple way of getting AIX installed onto a LPAR if you don’t have any other method of installation. Another thing to take note of, is that you can load any ISO image into the VML, it doesn’t have to be an AIX installation image. Quite useful for when you have to install any bit of software and all you have is the ISO image.

On that note though, the loopmount command was added in AIX 6.1 TL4. This allows you to mount ISO images directly from AIX without using the VML.

# loopmount -i image.iso -o "-V cdrfs -o ro" -m /mountpoint

Good luck, enjoy, and any questions can be popped into the comments below.

1) Chris Gibson has already written two great (and far more in depth) articles on multibos, which I recommend reading before going any further (Links: here and here).
2) This blog of mine serves two purposes; To share some of the things I come about during my sysadmin activities and secondly also a dumping ground for documents I write for myself. This post is the latter, as multibos isn’t something that I use daily (maybe a few times a year), and really, who’s going to remember all those syntax switches

Now, assuming you’ve read both articles, I don’t need to go into the benefits of multibos and I’ll dive straight into the how. I’ll be working on upgrading an AIX instance running 6100-05-01-1016 to the latest ML which at the time of writing is 6100-06-04-1112

PREREQUISITES
First things first, we need to make sure the current environment is ready to be copied. The two main things that I look for are:

1. Make sure the current environment doesn’t have any missing filesets.

root@AIX / > oslevel -s
6100-05-01-1016
root@AIX / > instfix -i | grep ML
    All filesets for 6100-00_AIX_ML were found.
    All filesets for 6100-01_AIX_ML were found.
    All filesets for 6100-02_AIX_ML were found.
    All filesets for 6100-03_AIX_ML were found.
    All filesets for 6100-04_AIX_ML were found.
    All filesets for 6100-05_AIX_ML were found.

**Note: If a particular ML doesn’t return positive (e.g. 6100-05_AIX_ML), use instfix -ciqk 6100-05_AIX_ML | grep “:-:” to see the fileset causing issues and rectify it.

2. Make sure all filesets have been committed. Now I’m not entirely sure if this is a requirement, but I prefer to work with an as clean environment as possible.

root@AIX / > lslpp -l | grep -i applied

**Note: If you have software which is in an applied state, commit it.

Now that we’re happy with the state of the environment, you need to verify that there is enough room in rootvg to make copies of the following logical volumes [/ (hd4), /usr (hd2), /var (hd9var), /opt (hd10opt) and the boot logical volume (hd5)].

Once we’ve satisfied the above requirement, we can get started with the multibos work. I’m not going to dissect each command, so I suggest reading the multibos man pages to know what each switch does. The only one that I’ll explain is -p, which does a preview of the multibos command executed.

MULTIBOS
1. Remove any previous standby multibos environments. The below command will return a FAILURE status if you don’t have a standby multibos environment. If you’re using multibos for the first time, this will be the case.

root@AIX / > multibos -RX
Initializing multibos methods ...
Initializing log /etc/multibos/logs/op.alog ...
Gathering system information ...
multibos: 0565-077 Unable to locate standby BOS.
 
Log file is /etc/multibos/logs/op.alog
Return Status: FAILURE

2. Create the standby BOS. New logical volumes and filesystems will be created prefixed and with bos_. This can take a while to complete.

This will preview the creation of the multibos environment and list all the logical volumes that will be copied.

root@AIX / > multibos -sXp
...
...
Log file is /etc/multibos/logs/op.alog
Return Status = SUCCESS

You can also view the log file using the alog command.

root@AIX / > alog -of /etc/multibos/logs/op.alog

If everything completed successfully, run the command again without the preview.

root@AIX / > multibos -sX

Once complete, you’ll notice that the new logical volumes will be prefixed with bos_

3. Once complete, you can drop into a multibos shell and check oslevel, it should be the same as the host OS.

The below command will mount all the filesystems required and drop you in a MULTIBOS> prompt

root@AIX / > multibos -S
...
...
MULTIBOS> oslevel -s
6100-05-01-1016

To umount all the filesystems and break out of the multibos environment.

MULTIBOS> exit

4. Apply the TL/ML to the multibos environment.

The below command will tell multibos to apply the updates from the specified location (which is the location that your new ML resides).

root@AIX / > multibos -Xac -l /home/kristijan/6100-06-04

Drop into the multibos shell again and check that the ML has been successfully applied.

root@AIX / > multibos -S
MULTIBOS> oslevel -s
6100-06-04-1112
MULTIBOS> instfix -i | grep ML
    	All filesets for 6100-00_AIX_ML were found.
    	All filesets for 6.1.0.0_AIX_ML were found.
    	All filesets for 6100-01_AIX_ML were found.
    	All filesets for 6100-02_AIX_ML were found.
    	All filesets for 6100-03_AIX_ML were found.
    	All filesets for 6100-04_AIX_ML were found.
    	All filesets for 6100-05_AIX_ML were found.
    	All filesets for 6100-06_AIX_ML were found.
MULTIBOS> exit

5. Verify that the bootlist now contains blv=bos_hd5. bos_hd5 is the location of the new boot logical volume. It will be at the top of the list when you run the bootlist command.

root@AIX / > bootlist -m normal -o
hdisk0 blv=bos_hd5 pathid=0
hdisk0 blv=hd5 pathid=0

You can now reboot into the new environment and verify that all went to plan.

6. If you need to back out of the change for any reason, just change the bootlist order back to blv=hd5 and reboot. Now, while this doesn’t bother me, it may bother others. You’ll notice after the reboot that your environment is now using the bos_ logical volumes. The only way to get your environment back to how it was using the original logical volume names is to create another multibos environment, and reboot into it. Since your current boot environment is using the bos_ logical volumes, it will create it using the original logical volume names (i.e. without the bos_). For those that want to do that, redo step 2 and reboot. However, I suggest not doing this for a week or so to be sure you’re happy with the new changes, as creating a new multibos environment will remove the backup environment with the previous ML.

As always, post comments/questions below.

DSH/DSHBAK
Distributed Shell (dsh) allows you to concurrently run commands on remote nodes. It requires very little setup, but it’s one of those hidden AIX commands that few administrators know about. As you may have already guessed, dsh comes with the CSM client filesets. Specifically, it comes from the csm.dsh fileset which creates a symbolic link – /usr/bin/dsh -> /opt/csm/bin/dsh

root@AIX / > lslpp -w /usr/bin/dsh
  File                                        Fileset               Type
  ----------------------------------------------------------------------------
  /usr/bin/dsh                                csm.dsh               Symlink

There are a few environmental variables that we need to setup first.

DSH_NODE_RSH=/usr/bin/ssh – By default, dsh uses rsh to execute remote commands. If you don’t know why using rsh is a bad idea, hand in your sysadmin card now. We need to tell dsh to execute all remote commands via ssh.
DCP_NODE_RCP=/usr/bin/scp – As above, dcp uses rcp to send files remotely, so lets change that to use scp
DSH_NODE_OPTS=-q – This passes the -q option to ssh to suppress the banner message from showing. This just makes the output much easier to read when working with a number of nodes.
DSH_NODE_LIST=/.dsh/model – You can feed a list of nodes to dsh a number of ways, but I find using text files the best way to manage it.

Most places have a typical dev/mod/prod setup which can consist of a number of servers within each environment. I have four node text files in my /.dsh directory.

-rw-rw-r--    1 root     system          577 Jan 11 10:19 all
-rw-r--r--    1 root     system           71 Feb 12 13:06 dev
-rw-r--r--    1 root     system           75 Oct 28 10:38 mod
-rw-r--r--    1 root     system          106 Jan 25 13:00 prod

This way I can run commands on all the nodes in a specific environment, or across all nodes. The easiest way to switch between lists is by using aliases.

In my .kshrc file

alias dshmod='export DSH_NODE_LIST=/.dsh/mod'
alias dshprod='export DSH_NODE_LIST=/.dsh/prod'
alias dshdev='export DSH_NODE_LIST=/.dsh/dev'
alias dshall='export DSH_NODE_LIST=/.dsh/all'

The contents of the node text files is each node on a single line.

root@AIX / > cat /.dsh/mod
SERVER1
SERVER2
SERVER3

That’s it for the configuration. You can run dsh -q to show the current configured variables. Look at the man page for dsh to see further configurable options and the IBM website[3].

Let’s see if it works.

root@AIX / > dshmod
root@AIX / > dsh date
SERVER1: Mon Feb 28 12:59:34 EETDT 2011
SERVER2: Mon Feb 28 12:53:58 EETDT 2011
SERVER3: Mon Feb 28 12:56:51 EETDT 2011

If you only wanted to run the command on a select number of hosts, you can use the -w switch.

root@AIX / > dsh -w SERVER1,SERVER3 "date"
SERVER1: Mon Feb 28 13:01:20 EETDT 2011
SERVER3: Mon Feb 28 12:55:43 EETDT 2011

We can make that output look a bit neater by piping through the dshbak command.

root@AIX / > dsh -w SERVER1,SERVER3 "date" | dshbak
HOST: SERVER1
-------------
Mon Feb 28 13:17:17 EETDT 2011
 
HOST: SERVER3
-------------
Mon Feb 28 13:11:41 EETDT 2011

DCP
Just like the dsh command, dcp allows you to send files concurrently to multiple hosts. dcp requires the same environmental setup as dsh. Seeing as that’s already setup, we can use it straight away.

root@AIX / > lmktemp test 1M
test
root@AIX / > dcp test /
root@AIX / > dsh ls -l /test
SERVER1: -rw-r--r--   1 root     system      1048576 Feb 28 13:32 /test
SERVER2: -rw-r--r--   1 root     system      1048576 Feb 28 13:32 /test
SERVER3: -rw-r--r--   1 root     system      1048576 Feb 28 13:32 /test

What I did above is created a 1M file called test, used dcp to send it to my hosts, then verified the file was on each node using dsh. You can find more about dcp from the man pages and the IBM website[4].

DPING
The dping command allows you to ping nodes in parallel (with the option of serially). Now, it doesn’t read the DSH_NODE_LIST variable like dsh/dcp, but you can pass the -f option and tell it to.

root@AIX / > dping -f /.dsh/mod
SERVER1: ping (alive)
SERVER2: ping (alive)
SERVER3: ping (alive)
SERVER4: noping (unreachable)

If you only want to ping a select few hosts, then you can use the -n switch.

root@AIX / > dping -n SERVER1,SERVER3
SERVER1: ping (alive)
SERVER3: ping (alive)

You can find more on dping by reading the man pages and on the IBM website[5]. Like dsh, you can also pipe the output of dping through dshbak.

If you have any questions on the above, use the comment section below.

[1] – http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.csm16.cmds.doc/am7cm_csmcomm.html
[2] – http://www.ece.uci.edu/~chou/ssh-key.html
[3] – http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.csm16.cmds.doc/am7cm_dsh.html
[4] – http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.csm16.cmds.doc/am7cm_dcp.html
[5] – http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.csm16.cmds.doc/am7cm_dping.html

kristijan@AIX > ./meminfo
 
Memory Information
==================
total memory = 1884 MB
free memory  = 94 MB
used memory  = 1790 MB

Script uses standard AIX command line utilities to work out all the values. I haven’t tested it across all versions of AIX (I wrote this on AIX 6.1), but I believe it should work fine. Let me know in the comments if it doesn’t.

#!/usr/bin/ksh
#
# Quick view of memory usage under AIX
#
 
USED=`svmon -G | head -2 | tail -1 | awk '{ print $3 }'`
USED=`expr $USED / 256`
TOTAL=`lsattr -El sys0 -a realmem | awk '{ print $2 }'`
TOTAL=`expr $TOTAL / 1000`
FREE=`expr $TOTAL - $USED`
 
echo "\nMemory Information"
echo "=================="
echo "total memory = $TOTAL MB"
echo "free memory  = $FREE MB"
echo "used memory  = $USED MB"
 
exit 0

First things first, let’s plug in our drive and find out the device ID FreeNAS assigns it.

# dmesg
...
umass1: <ASMedia AS2105, class 0/0, rev 2.10/1.00, addr 3> on uhub4
da1 at umass-sim1 bus 1 target 0 lun 0
da1: <ST950042 0AS 0002> Fixed Direct Access SCSI-0 device 
da1: 40.000MB/s transfers
da1: 476940MB (976773168 512 byte sectors: 255H 63S/T 60801C)

Now we create a mount point for the drive and try mount it like we normally would.

# mkdir /mnt/usb
# mount -t ntfs /dev/da1s1 /mnt/usb/

You’ll receive the following error:
fuse: failed to open fuse device: No such file or directory

Now lets load the fuse driver and try again.

# kldload fuse
# mount -t ntfs /dev/da1s1 /mnt/usb/
# ls -l /mnt/usb
total 4589868
drwxrwxrwx  1 root  wheel        4096 Sep 25 17:48 $RECYCLE.BIN
drwxrwxrwx  1 root  wheel           0 Nov 27 14:25 .Trashes
-rwxrwxrwx  1 root  wheel        4096 Nov 22 18:15 ._.Trashes
...

SUCCESS!

It’s probably easier to load the driver on boot instead of loading it manually like I have above. I haven’t tried it, but it should work if you call it from loader.conf. I can reboot my NAS at the moment, but if someone wants to give it a shot and post to the comments, that would be great. You can follow instructions on how to edit loader.conf in a previous post [1] of mine.

[1] – http://www.kristijan.org/2010/11/freenas-realtek-network-card/

As I mentioned in a previous post, I built myself a NAS based on FreeNAS. It’s been rock solid for about 3 weeks now, no hiccups. One initial issue I did have was getting the RTL8168C/RTL8111DP/RTL8111E network card chipset to work. There is no FreeNAS out of the box support for this chipset, so I had to compile my own from source found on the Realtek website which you can find here. If you have issues downloading or compiling it, I’ve compiled version 1.80 which you can download from the top of this page. Ruslan has kindly compiled an AMD64 version as well.

First and foremost, verify that the module actually works and you can bring up your network card and get some activity over it. I’m going to assume at this point that you have no way of copying something onto your FreeNAS other than USB, so copy if_re.ko to a USB stick and plug it into your NAS.

To mount the USB device, let’s first create a mount point.

# mkdir /mnt/usb

Now we need to find the USB device you just plugged in.

# dmesg
...
umass1: <USB 2 Flash Disk, class 0/0, rev 2.00/0.00, addr 3> on uhub4
da1 at umass-sim1 bus 1 target 0 lun 0
da1: <USB 2.0 Flash Disk 2.20> Removable Direct Access SCSI-2 device 
da1: 40.000MB/s transfers
da1: 119MB (244480 512 byte sectors: 64H 32S/T 119C)

What we’re looking for here is the da1

# mount_msdosfs /dev/da1 /mnt/usb

Now let’s see if we can load the module.

# kldload /mnt/usb/if_re.ko

If you don’t get any errors, you should see the module loaded.

# kldstat
Id Refs Address    Size     Name
 1   12 0x80400000 b134e4   kernel
 2    1 0x80f14000 195b4    if_re.ko
 3    1 0x8c32e000 e350     isboot.ko
 4    1 0x8c33d000 6a4f0    acpi.ko
 5    1 0x92284000 123000   zfs.ko
 6    1 0x923a7000 3000     opensolaris.ko
 7    1 0x92843000 4000     geom_nop.ko
 8    1 0x938cf000 9000     aio.ko
 9    1 0xa8768000 13000    geom_vinum.ko

So far so good, now bring up the interface and see if you can assign an IP address to it and ping another node on your network.
Sweet, all good, all we need to do now is copy the module to a location that isn’t going to get overwritten at boot, and edit loader.conf.

/cf is the filesystem where we will copy the module to, but it’s mounted as read only on boot, so let’s remount it as read/write.

# df /cf
Filesystem 1K-blocks  Used Avail Capacity  Mounted on
/dev/da0a      71062 64438  6624    91%    /cf

Substitue /dev/da0a to match your install.

# umount /cf
# mount -o rw /dev/da0a /cf

Copy if_re.ko from the USB to /cf/boot/kernel

# cp /mnt/usb/if_re.ko /cf/boot/kernel

Edit /cf/boot/loader.conf and add the following line.

if_re_load="YES"

That should be it, you can now unmount the USB stick which you used to copy the module and reboot. If all goes well, your network card should come up on boot. If you do have any issues with this, please post to the comments.

Luckily for us, the pretty OSX frontend sits on top of a UNIX backend. In short, I ended up using autofs, and it works rather well. Below is how to get it working.

1. Bust open Terminal.app and do the following.

sudo vi /etc/fstab

…and paste the below:

# Mount AFP share from NAS
(hostname):(share) (mount_point) url automounted,url==afp://(username):(password)@(hostname)/(share) (mount_point) 0 0

Replacing the following:
(hostname):(share) – With the hostname and share of the NAS.
(mount_point) – The mount point in OSX.
(username):(password) – Credentials to connect to the share.

For example:

192.168.0.109:/Media /Users/kristijan/Media url automounted,url==afp://kristijan:fakepass@192.168.0.109/Media /Users/kristijan/Media 0 0

2. Either reboot or run “sudo automount -vc” to mount your share.

iMac:~ kristijan$ sudo automount -vc
automount: /net updated
automount: /home updated
automount: /Users/kristijan/Media mounted
automount: no unmounts

All done.
Since we’re using autofs, NAS’s that hibernate shouldn’t cause OSX any issues, as autofs can mount/umount as needed.

1) Checks for volumes which are NOT in the states READWRITE or OFFSITE.
2) Checks for volumes which have a read/write error count >0.

I’m sure that most people running TSM in their environment have some sort of daily reporting that gets sent out. If that’s the case, you can simply extract the SQL from the script and use it in your own reporting tools.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

#!/usr/bin/ksh
 
USER=`grep USER /home/aixuser/tsm_creds | awk -F = '{ print $2 }'`
PASS=`grep PASS /home/aixuser/tsm_creds | awk -F = '{ print $2 }'`
 
DSMADMC="/usr/bin/dsmadmc -id=$USER -pass=$PASS -dataonly=yes"
DATE=`date +"%Y%m%d"`
DATE_SHORT=`date +"%a"`
TEMP_OLD="/tmp/tsm.temp.old"
TEMP_NEW="/tmp/tsm.temp.new"
MAILTO="email@host.com"
 
if [[ ! -f $TEMP_OLD ]]; then
  touch $TEMP_OLD
fi
 
MACRO1="select VOLUME_NAME,ACCESS from volumes where access !='READWRITE' and access !='OFFSITE'"
 
    echo "VOLUMES WITH STATUS ISSUES" >> $TEMP_NEW
    echo "\nVOLUME_NAME            ACCESS" >> $TEMP_NEW
    echo "------------------     ------------------" >> $TEMP_NEW
    $DSMADMC $MACRO1 >> $TEMP_NEW
 
MACRO2="SELECT volumes.volume_name, volumes.stgpool_name, volumes.pct_utilized, volumes.status, volumes.write_errors, volumes.read_errors FROM volumes, libvolumes WHERE volumes.volume_name=libvolumes.volume_name AND ( volumes.write_errors>0 OR volumes.read_errors>0 )"
 
    echo "\n\nVOLUMES WITH READ/WRITE ISSUES" >> $TEMP_NEW
    echo "\nVOLUME_NAME            STGPOOL_NAME           PCT_UTILIZED     STATUS                 WRITE_ERRORS     READ_ERRORS" >> $TEMP_NEW
    echo "------------------     ------------------     ------------     ------------------     ------------     -----------" >> $TEMP_NEW
    $DSMADMC $MACRO2 >> $TEMP_NEW
 
diff $TEMP_OLD $TEMP_NEW > /dev/null 2>&1
   if [[ $? == 1 ]]; then
    cat $TEMP_NEW | mail -s "Warning: TSM Media Issues" $MAILTO
    cp $TEMP_NEW $TEMP_OLD
   fi
 
rm $TEMP_NEW
 
exit

aixuser@TSM /home/aixuser > cat tsm_creds
USER=tsmuser
PASS=tsmpass
aixuser@TSM /home/aixuser > ls -l
-r--------   1 aixuser     system           27 May 17 14:42 tsm_creds
-rwxr-xr-x   1 aixuser     system         1605 Sep 21 10:55 tsm_tape_issues

Script breakdown
Line(s)
03-04:   I grab the TSM credentials out of a file which only has o+r access rights so we don’t have the TSM credentials in the script itself (see above output).
17:      SQL select statement to search for all volumes with access other than READWRITE and OFFSITE.
24:      SQL select statement to search for all volumes with read/write errors.
31-35:   diff new output with previously saved output and determine if email should be sent.

It’s pretty straight forward, nothing fancy. Just change the variables to cater for your environment and off you go. The result will be an email with output similiar to the below if any issues are found.

VOLUMES WITH STATUS ISSUES
 
VOLUME_NAME            ACCESS
------------------     ------------------
TAPE09L4               UNAVAILABLE       
 
 
VOLUMES WITH READ/WRITE ISSUES
 
VOLUME_NAME            STGPOOL_NAME           PCT_UTILIZED     STATUS                 WRITE_ERRORS     READ_ERRORS
------------------     ------------------     ------------     ------------------     ------------     -----------
TAPE09L4               PRI_TAPE_POOL                  13.7     FULL                              0               1

Post any questions to the comments box below.