Combining NFS with PowerHA we can achieve a HANFS (Highly Available Network File System). The basic concept behind this solution is that one node in the cluster mounts the resource locally, and offers that as an exported resource via a serviceable IP. Another node in the cluster is then configured to take on the resource in the event of failure.
If you’re following this, I’m taking the assumption that your cluster is already configured, you have a working IP network and have set up a shared volume group between the cluster nodes that will be handling the HANFS failover. Before we get started though, there are a few things which need to be installed/verified.
The cluster.es.nfs.rte fileset needs to be installed so that PowerHA can work with HANFS.
# lslpp -l cluster.es.nfs.rte Fileset Level State Description ---------------------------------------------------------------------------- Path: /usr/lib/objrepos cluster.es.nfs.rte 184.108.40.206 COMMITTED ES NFS Support Path: /etc/objrepos cluster.es.nfs.rte 220.127.116.11 COMMITTED ES NFS Support
Also, since we’re dealing with NFS, we need to make sure we have the portmapper daemon running.
# lssrc -s portmap Subsystem Group PID Status portmap portmap 213160 active
To have the portmapper daemon start during system boot, you’ll need to modify /etc/rc.tcpip and uncomment the start line below.
# Start up Portmapper start /usr/sbin/portmap "$src_running"
Below is a table of what the HANFS layout looks like (click image to expand).
We have a Resource Group (RG_nfs) which contains an Application Server (app_nfsv4) and a Service IP Label (serviceip-nfs). There is a Shared Volume Group (haNFSvg) which contains some Shared Logical Volumes & Filesystems. The thing to take note here, is that we are only exporting two of the three filesystems that you see in the table. The /nfs_stable filesystem will be the NFSv4 stable storage path location. The below is taken from the IBM website. I recommend you read it for further information on NFSv4 stable storage and PowerHA.
“Stable Storage is a file system space that is used to save the state information by the NFSv4 server. This is very crucial for maintaining NFSv4 client’s state information to facilitate smooth and transparent fallover/fallback/move of the Resource group from one node to other.”
With the above information in hand, we’re ready to start configuring HANFS.
Step 1 (All Nodes) Set the NFS domain
The NFSv4 domain needs to be set on all cluster nodes which will be responsible for taking on the RG_nfs resource group in the event of a failure.
# chnfsdom <- This will show the current NFS domain # chnfsdom [new_domain] <- To set a new NFS domain
Step 2 (All Nodes) Create mount points
The mount points need to be created on all cluster nodes which will be responsible for taking on the RG_nfs resource group in the event of a failure.
# mkdir -p /home/kristian # mkdir -p /data
Take note that we’re only creating the mount points here, and not the logical volumes or filesystems.
Step 3 – Create logical volumes and filesystems
Now we create the following logical volumes:
Create the logical volumes using the following path.
# smit hacmp -> System Management (C-SPOC) -> HACMP Logical Volume Management -> Shared Logical Volumes -> Add a Shared Logical Volume
Create the filesystems using the following path.
# smit hacmp -> System Management (C-SPOC) -> HACMP Logical Volume Management -> Shared File Systems -> Enhanced Journaled File Systems -> Add an Enhanced Journaled File System on a Previously Defined Logical Volume
We should now see the 2 filesystems which we are exporting and the stable storage filesystem apart of the shared volume group
# lsvg -l haNFSvg haNFSvg: LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT hanfs_kris_lv jfs2 107 107 1 open/syncd /hanfs_home_kristian hanfs_data_lv jfs2 32 32 1 open/syncd /hanfs_data hanfs_nfsstable_lv jfs2 75 75 1 open/syncd /nfs_stable
Step 4 – Configure the Application Server
Luckily for us, IBM have made this step rather easy and have provided start, stop and monitor scripts. The location of these scripts are below.
Create an Application Server using the following path.
# smit hacmp -> Extended Configuration -> Extended Resource Configuration -> HACMP Extended Resources Configuration -> Configure HACMP Applications Servers -> Add an Application Server
To configure Application Server monitoring, use the following path.
# smit hacmp -> Extended Configuration -> Extended Resource Configuration -> HACMP Extended Resources Configuration -> Configure HACMP Application Monitoring -> Configure Custom Application Monitors -> Add a Custom Application Monitor
Step 5 – Configure the Resource Group for HANFS
The next step is to configure the RG_nfs resource group with the values needed for HANFS.
Modify the resource group using the following path.
# smit hacmp -> Extended Configuration -> Extended Resource Configuration -> HACMP Extended Resource Group Configuration -> Change/Show Resources and Attributes for a Resource Group
Service IP Labels/Addresses [serviceip-nfs] Application Servers [app_nfsv4] Filesystems (empty is ALL for VGs specified) [/hanfs_home_kristian /hanfs_data /nfs_stable] Filesystems Consistency Check fsck Filesystems Recovery Method sequential Filesystems mounted before IP configured true Filesystems/Directories to Export (NFSv2/3)  Filesystems/Directories to Export (NFSv4) [/hanfs_home_kristian /hanfs_data] Stable Storage Path (NFSv4) [/nfs_stable] Filesystems/Directories to NFS Mount [/home/kristian;/hanfs_home_kristian /data;/hanfs_data]
Most vaules are self explainatory. We set “Filesystems mounted before IP configured” to true so we prevent access from clients before the filesystems are ready. We also specify mount points in the following format [mount point];[exported filesystem]
Step 6 – HANFS exports file
Just like NFS has /etc/exports, HANFS has /usr/es/sbin/cluster/etc/exports. If you need to specify NFS options, you MUST use /usr/es/sbin/cluster/etc/exports and not /etc/exports. For help creating the exports file, you can use “smit mknfsexp”
* Pathname of directory to export  Anonymous UID [-2] Public filesystem? no * Export directory now, system restart or both both Pathname of alternate exports file [/usr/es/sbin/cluster/etc/exports] ... ...
Step 7 – Synchronize the cluster
We now need to synchronize our changes to the other cluster nodes
# smit hacmp -> Extended Configuration -> Extended Verification and Synchronization
Step 8 – Bring the Resource Group online
We now bring the resource group online. It’s a good idea at this stage to tail the hacmp.out file to see any errors.
To bring the resource group online.
# smit hacmp -> System Management (C-SPOC) -> HACMP Resource Group and Application Management -> Bring a Resource Group Online
# tail -f /var/hacmp/log/hacmp.out ... +RG_nfs:cl_activate_nfs(.110):/home/kristian;/hanfs_home_kristian[nfs_mount+102] : Attempt 0/5 to NFS-mount at Jul 26 11:01:21.000 +RG_nfs:cl_activate_nfs(.110):/home/kristian;/hanfs_home_kristian[nfs_mount+103] mount -o vers=4,hard,intr serviceip-nfs:/hanfs_home_kristian /home/kristian
You should now see the filesystems mounted.
# mount node mounted mounted over vfs date options -------- -------------- --------------- ------ ------------ --------------- ... /dev/hanfs_kris_lv /hanfs_home_kristian jfs2 Jul 26 14:36 rw,log=INLINE /dev/hanfs_data_lv /hanfs_data jfs2 Jul 26 14:36 rw,log=INLINE /dev/hanfs_nfsstable_lv /nfs_stable jfs2 Jul 26 14:36 rw,log=INLINE serviceip-nfs /hanfs_home_kristian /home/kristian nfs4 Jul 26 14:36 vers=4,hard,intr serviceip-nfs /hanfs_data /data nfs4 Jul 26 14:36 vers=4,hard,intr
What you’re seeing above is the output from the mount command ran from the cluster node which currently has the RG_nfs resource group ONLINE. You’ll notice that it has the shared logical volumes and filesystems mounted, then a local NFS export from itself to mount /home/kristian and /data. On the other cluster nodes, you will only see the NFS mounts.
There you have it, NFS cross-mounts with PowerHA.