Patching CVE-2014-6271 and CVE-2014-7169 on AIX via NIM (bash bug aka shellshock)

Update: Patch links from both IBM [4] and Perzl [5] at the bottom of this post.

Below I detail how I patched over 800 AIX LPAR’s that were exposed by CVE-2014-6271 [1] and CVE-2014-7169 [2], also known as shellshock, using the NIM server.

From everything that I’ve been reading on IBM’s Knowledge Centre, creating an LPP source containing only RPM’s isn’t possible. To patch my AIX environment, I decided to use the “script” resource available to the NIM master, along with the pre-existing NFS mounts that I had configured.

NIM master NFS configuration.

NIM:kristian# cat /etc/exports
/export/nim/images -ro,anon=0
 
NIM:kristian# showmount -e
export list for NIM:
/export/nim/images (everyone)

Location of patched bash RPM on NIM master.

NIM:kristian# ls -l /export/nim/images/bash_CVE-2014-6271-7169
total 3448
-rw-r-----    1 root  system     1765643 Sep 30 08:22 bash-4.2-17.aix5.1.ppc.rpm

Script to patch bash (Name: install_bash_CVE-2014-6271-7169)

#!/bin/ksh
#
# Script to install new version of bash to
# patch CVE-2014-6271 and CVE-2014-7169
#
# Kristian Milos (29/09/14)
 
# Get NIM master hostname
NIM_MASTER_HOSTNAME=`grep NIM_MASTER_HOSTNAME /etc/niminfo | awk -F = '{ print $2 }'`
 
# Create temporary mount location
mkdir /install_bash_CVE-2014-6271-7169
 
# NFS mount patch
mount ${NIM_MASTER_HOSTNAME}:/export/nim/images/bash_CVE-2014-6271-7169 /install_bash_CVE-2014-6271-7169
 
# Install patch
rpm -Uvh /install_bash_CVE-2014-6271-7169/bash-4.2-17.aix5.1.ppc.rpm
 
# Unmount NFS mount
umount /install_bash_CVE-2014-6271-7169
 
# Remove temporary mount location
rm -r /install_bash_CVE-2014-6271-7169
 
exit

Now that we have the location of the RPM on the NIM master, and the script that will be run on the NIM client to patch bash, we can now define a NIM script resource.

NIM:kristian# nim -o define -t script \
-a server=master \
-a location=/export/nim/patches/install_bash_CVE-2014-6271-7169 \
-a comments="bash fix for CVE-2014-6271 and CVE-2014-7169" bash_CVE-2014-6271-7169
 
NIM:kristian# lsnim -l bash_CVE-2014-6271-7169
bash_CVE-2014-6271-7169:
   class       = resources
   type        = script
   comments    = bash fix for CVE-2014-6271 and CVE-2014-7169
   Rstate      = ready for use
   prev_state  = unavailable for use
   location    = /export/nim/patches/install_bash_CVE-2014-6271-7169
   alloc_count = 0
   server      = master

We will now create a NIM machine group that will contain all the NIM clients that we will update. I find the easiest way to do this is by listing out all the NIM client definitions in the format required for the group define command. An example is shown below.

NIM:kristian# for i in `lsnim -t standalone | awk '{ print $1 }'`; do echo "-a add_member=$i \\"; done
-a add_member=aix1 \
-a add_member=aix2 \
-a add_member=aix3 \
-a add_member=aix4 \
-a add_member=aix5 \
-a add_member=aix6 \

Define the NIM group

NIM:kristian# nim -o define -t mac_group \
-a add_member=aix1 \
-a add_member=aix2 \
-a add_member=aix3 \
-a add_member=aix4 \
-a add_member=aix5 \
-a add_member=aix6 PROD_LPARS

The next thing I do is validate that the NIM master can actually talk to all the NIM clients in the machine group. The below method of checking was modified from a post originally written by Brian Smith [3] to work with NIM groups, instead of the standalone clients.

NIM:kristian# for srv in `lsnim -g PROD_LPARS | grep member | grep -v EXCLUDED | awk '{ print $3 }' | sed 's/;//' | sort`; do printf "%-20s" $srv; nim -o lslpp $srv >/dev/null 2>&1; [ "$?" == 0 ] && echo OK || echo "Problem"; done
aix1            OK
aix2            OK
aix3            OK
aix4            OK
aix5            Problem
aix6            OK

For any NIM client that returns “Problem”, I exclude them from the NIM group operation

NIM:kristian# nim -o select -a exclude=aix5 PROD_LPARS

We’re now in a position to execute the patch across all the NIM clients listed in the group definition.

Patch all NIM clients via NIM master

NIM:kristian# nim -o cust -a script=bash_CVE-2014-6271-7169 -a concurrent=10 PROD_LPARS
+-----------------------------------------------------------------------------+
                      Concurrency Control
+-----------------------------------------------------------------------------+
 
 Processing will begin with the first 5 machines from the group...
 
+-----------------------------------------------------------------------------+
                      Initiating "cust" Operation
+-----------------------------------------------------------------------------+
 Allocating resources ...
 
 Initiating the cust operation on machine 1 of 5: aix1 ...
 
 Initiating the cust operation on machine 2 of 5: aix2 ...
 
 Initiating the cust operation on machine 3 of 5: aix3 ...
 
 Initiating the cust operation on machine 4 of 5: aix4 ...
 
 Initiating the cust operation on machine 5 of 5: aix6 ...
 
+-----------------------------------------------------------------------------+
                      "cust" Operation Summary
+-----------------------------------------------------------------------------+
 Target                  Result
 ------                  ------
 aix1                    INITIATED
 aix2                    INITIATED
 aix3                    INITIATED
 aix4                    INITIATED
 aix6                    INITIATED
 
 Note: Use the lsnim command to monitor progress of "INITIATED"
 targets by viewing their NIM database definition.
 
+-----------------------------------------------------------------------------+
                      Concurrency Control
+-----------------------------------------------------------------------------+
  The first 8 machines have been processed.  As machines finish
  installing processing will resume with the remaining members
  of the group, one at a time.
 
+-----------------------------------------------------------------------------+
                      Concurrency Control: "cust" Operation Summary
+-----------------------------------------------------------------------------+
 Target                  Result
 ------                  ------
 aix1                    COMPLETE
 aix2                    COMPLETE
 aix3                    COMPLETE
 aix4                    COMPLETE
 aix6                    COMPLETE

Once the process has completed, you can validate the version of bash installed across all NIM clients by running the following command.

NIM:kristian# for i in `lsnim -g PROD_LPARS | grep member | grep -v EXCLUDED | awk '{ print $3 }' | sed 's/;//' | sort`; do echo $i; nim -o lslpp -a lslpp_flags=-Lc -a filesets=bash $i | grep bash | awk -F : '{ print $2 }'; echo ""; done
aix1
bash-4.2-17
 
aix2
bash-4.2-17
 
aix3
bash-4.2-17
 
aix4
bash-4.2-17
 
aix6
bash-4.2-17

One final clean up task, is to ensure you include all NIM members back into the group if you excluded them previously.

NIM:kristian# nim -o select -a include_all=yes PROD_LPARS

Links
[1] – http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-6271
[2] – http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-7169
[3] – https://www.ibm.com/developerworks/community/blogs/brian/entry/check_nim_connectivity_to_all_your_nim_clients?lang=zh_en
[4] – http://www-01.ibm.com/support/docview.wss?uid=isg3T1021272
[5] – http://www.perzl.org/aix/index.php?n=Main.Bash

Leave a Reply

Your email address will not be published. Required fields are marked *


*