PowerHA SystemMirror and NIM mksysb failures

I built a basic two-node PowerHA SystemMirror (HACMP) cluster for my team a little while ago to use as a test environment for patch updates. While it wasn’t a true reflection of how the production environment is configured, it was enough to test functionality. As such, I configured a single virtual ethernet adapter in each cluster node, which would house both the boot IP and the service IP of the cluster. After a couple of weeks, I noticed that my weekly NIM mksysb’s on one of the two cluster nodes was always failing. Further investigation found that the NIM mksysb’s would always fail on the cluster node that had the active resource group with the service IP attached to it. If I failed the resource group over to the other cluster node, the NIM mksysb would complete successfully.
Continue reading

Patching CVE-2014-6271 and CVE-2014-7169 on AIX via NIM (bash bug aka shellshock)

Update: Patch links from both IBM [4] and Perzl [5] at the bottom of this post.

Below I detail how I patched over 800 AIX LPAR’s that were exposed by CVE-2014-6271 [1] and CVE-2014-7169 [2], also known as shellshock, using the NIM server.

From everything that I’ve been reading on IBM’s Knowledge Centre, creating an LPP source containing only RPM’s isn’t possible. To patch my AIX environment, I decided to use the “script” resource available to the NIM master, along with the pre-existing NFS mounts that I had configured.
Continue reading

AIX boot hangs with HMC 2700 LED code

We recently upgraded the firmare on our Power frame, which required shutting down some of our AIX LPAR’s. The firmware upgrade went well, as did starting up all the AIX LPAR’s, except for one. This particular LPAR booted to HMC LED code 2700 and hung there. I restarted the partition to the Open Firmware (OF) prompt, and tried booting again using verbose mode to see where the boot process was hanging.
Continue reading

Error report mail notifications with errnotify

Having the pleasure of working across many client accounts, it’s funny to see some of the convoluted scripts people have written to receive alerts from the AIX error log daemon. Early in my AIX career, I used to do the exact same thing, and it involved a whole bunch of SSH keys, some text manipulation, crontab, and sendmail. Wouldn’t it be nicer if AIX had some way of doing all of this for us? Well, you know I wouldn’t ask the question if the answer wasn’t yes!

AIX has an Error Notification object class in the Object Data Manager (ODM). By default, there are a number of predefined errnotify entries, and each time an error is logged via errlog, it checks if that error entry matches the criteria of any of the Error Notification objects. What we’re about to do, is add another entry into the errnotify object class to be checked and actioned upon.
Continue reading