Cloning IBM i: Resetting Multipath

Last Updated on 28 June 2021 by Roberto De Pedrini

Let’s start with a brief introduction of “cloning” and “multipath” concepts.

---

Cloning (that is creating an exact copy of an existing IBM i system or partition) has been a concept for the IBM i platform since the introduction of boot from SAN (Storage Area Network) with i5/OS V5R3M5 (when the load source has been moved inside a SAN-attached storage subsystem without the needed to use remote load source mirroring to mirror your internal load source to the SAN-attached load source). The copy can be attached to another servers, a separate LPAR, or, if the production system is powered off, the existing partition or system. After the copy is created, you can use it for offline backup, system testing, or migration.

Boot from SAN enables you to take advantage of some of the advanced features that are available with IBM system storage. One of these functions is FlashCopy which allows you to perform a point-in-time instantaneous copy of the data held on a LUN or group of LUNs. Therefore, when you have a system that only has SAN LUNs with no internal drives, you can create a clone of your system.

---

IBM i supports multipath attachment through Fibre Channel as part of the base IBM i operating system support to provide greater resilience to storage area network (SAN) failures, which can be critical to IBM i due to the single-level storage architecture (multipath is the ability to present logical volumes on a storage device to the host server via multiple connections). Additionally, having multiple paths to a logical volume can improve the I/O performance of the partition.

Note: With IBM i operating system 6.1 or later, multipath attachment is supported on an external disk unit that acts as a load source for the IBM i host.

New paths are automatically detected, configured by the system, and immediately used. If a disk is initially set up as a single path and a second path is added, the resource name of the disk unit is modified from DDxxx to DMPxxx. This modification reflects that it now has more than one path to the disk unit. No user changes are required on the IBM i host to use the new path. Multipath connections can be directly connected or connected through a fabric.

---

Now, we combine the two concepts​.

When you IPL a clone, a message CPI096E - Disk unit connection is missing for each LUN connected by the missing path appears in the history log of the newly created IBM i LPAR (and it is re-posted every hour).

A check in DST/SST shows the expected disk paths as active (or passive, see note below for additional information) and extra disk paths with a status of missing, as result of the reconfiguration of the disk subsystem (missing paths are the connections to the disk units of the cloned system):

          Serial                     Resource   Path
ASP Unit  Number          Type Model Name       Status
  1    1  YF4FE4000034    2145  050  DMP005     Active
          YF4FE4000034    2145  050  DMP001     Passive 
          YF4FE4000034    2145  050  DMP002     Active
          YF4FE4000034    2145  050  DMP015     Passive
          YF501C00000F    2145  050  DMP011     Missing
          YF501C00000F    2145  050  DMP012     Missing
          YF501C00000F    2145  050  DMP013     Missing
          YF501C00000F    2145  050  DMP014     Missing
  1    2  YF4FE4000035    2145  050  DMP006     Passive
          YF4FE4000035    2145  050  DMP003     Active
          YF4FE4000035    2145  050  DMP016     Active
          YF4FE4000035    2145  050  DMP004     Passive
          YF501C000010    2145  050  DMP007     Missing
          YF501C000010    2145  050  DMP010     Missing
          YF501C000010    2145  050  DMP009     Missing
          YF501C000010    2145  050  DMP008     Missing

Note: Every LUN in Storwize V7000 uses one V7000 node as a preferred node. The I/O traffic to or from the particular LUN normally goes through the preferred node. If that node fails, the I/O is transferred to the remaining node. With IBM i multipath, all the paths to a LUN through the preferred node are active and the path through the non-preferred node is passive. Multipath employs the load balancing among the paths to a LUN that go through the node, which is preferred for that LUN.

To avoid the warnings and to clean up the disk paths you can run the Multipath Resetter Advanced Analysis macro in DST/SST, or QYASSDMO API. Using QYASSDMO API allows you to avoid access to the DST/SST and to automate the task.

---

Below is a very simple example of using QYASSDMO API to perform a multipath reset.

Target operating system: IBM i V7R2 (SF99720 level 20296 and SF99717 level 9), IBM i V7R3 (SF99730 level 19311 – SF99727 level 7, and SF99730 level 21098 – SF99727 level 10).

Note: On my clone systems using IBM i V7R3 with SF99730 level 19311, the missing paths are not shown in SST but only reported with CPI096E messages in the history log. They became visible again in SST after installing SF99730 level 21098, current level at the time of writing this post.

Prerequisite: An IBM i system with only SAN LUNs (no internal drives).

---

CMD source:

             CMD        PROMPT('Multipath Resetter')
             PARM       KWD(ASP) TYPE(*UINT4) RSTD(*NO) DFT(1) +
                          RANGE(1 255) CHOICE('1, 2, ...') +
                          PROMPT('Enter ASP No:')

CLLE source:

             PGM        PARM(&P_ASP)

/* Parameters */
             DCL        VAR(&P_ASP) TYPE(*UINT) LEN(4)

/* QYASSDMS, QYASSDMO, QYASEDMS */
             DCL        VAR(&HANDLE) TYPE(*CHAR) LEN(8)

/* QYASSDMO */
             DCL        VAR(&OPKEY) TYPE(*CHAR) LEN(4)
             DCL        VAR(&DMOP0700) TYPE(*CHAR) LEN(20)
             DCL        VAR(&ASP_OFFSET) TYPE(*INT) STG(*DEFINED) +
                          LEN(4) DEFVAR(&DMOP0700 1)
             DCL        VAR(&ASP_QTY) TYPE(*INT) STG(*DEFINED) +
                          LEN(4) DEFVAR(&DMOP0700 5)
             DCL        VAR(&ASP_OPER) TYPE(*INT) STG(*DEFINED) +
                          LEN(4) DEFVAR(&DMOP0700 9)
             DCL        VAR(&ASP_RES) TYPE(*INT) STG(*DEFINED) +
                          LEN(4) DEFVAR(&DMOP0700 13)
             DCL        VAR(&ASP_ARRAY) TYPE(*CHAR) STG(*DEFINED) +
                          LEN(4) DEFVAR(&DMOP0700 17)
             DCL        VAR(&DMOP0700L) TYPE(*INT) LEN(4) VALUE(20)
             DCL        VAR(&FORMAT) TYPE(*CHAR) LEN(8)

             DCL        VAR(&APINAME) TYPE(*CHAR) LEN(10)
             DCL        VAR(&MSGDTA) TYPE(*CHAR) LEN(512)

/* API Error */
             DCL        VAR(&APIERROR) TYPE(*CHAR) LEN(528)
             DCL        VAR(&AEBYTPRO) TYPE(*INT) STG(*DEFINED) +
                          LEN(4) DEFVAR(&APIERROR 1)
             DCL        VAR(&AEBYTAVL) TYPE(*INT) STG(*DEFINED) +
                          LEN(4) DEFVAR(&APIERROR 5)
             DCL        VAR(&AEEXCPID) TYPE(*CHAR) STG(*DEFINED) +
                          LEN(7) DEFVAR(&APIERROR 9)
             DCL        VAR(&AEEXCPDTA) TYPE(*CHAR) STG(*DEFINED) +
                          LEN(512) DEFVAR(&APIERROR 17)

             MONMSG     MSGID(CPF0000 MCH0000) EXEC( +
             GOTO       CMDLBL(ERROR))

/* Start DASD Management Session - QYASSDMS API */
             CALL       PGM(QYASSDMS) PARM(&HANDLE &APIERROR)
             IF         COND(&AEBYTAVL *NE 0) THEN(DO)
             CHGVAR     VAR(&APINAME) VALUE('QYASSDMS')
             GOTO       CMDLBL(ERROR)
             ENDDO

/* Start DASD Management Operation - QYASSDMO API */
             CHGVAR     VAR(%BIN(&OPKEY 1 4)) VALUE(16)
             CHGVAR     VAR(&ASP_OFFSET) VALUE(16)
             CHGVAR     VAR(&ASP_QTY) VALUE(1)
             CHGVAR     VAR(&ASP_OPER) VALUE(0)
             CHGVAR     VAR(&ASP_RES) VALUE(0)
             CHGVAR     VAR(%BIN(&ASP_ARRAY 1 4)) VALUE(&P_ASP)
             CHGVAR     VAR(&FORMAT) VALUE('DMOP0700')
             CALL       PGM(QYASSDMO) PARM(&HANDLE &OPKEY &DMOP0700 +
                          &DMOP0700L &FORMAT &APIERROR)
             IF         COND(&AEBYTAVL *NE 0) THEN(DO)
             CHGVAR     VAR(&APINAME) VALUE('QYASSDMO')
             GOTO       CMDLBL(ERROR)
             ENDDO

/* End DASD Management Session - QYASEDMS API */
             CALL       PGM(QYASEDMS) PARM(&HANDLE &APIERROR)
             IF         COND(&AEBYTAVL *NE 0) THEN(DO)
             CHGVAR     VAR(&APINAME) VALUE('QYASEDMS')
             GOTO       CMDLBL(ERROR)
             ENDDO

             CHGVAR     VAR(&MSGDTA) VALUE('Multipath reset for ASP' +
                          *BCAT %CHAR(&P_ASP) *BCAT 'completed')

             SNDPGMMSG  MSGID(CPF9898) MSGF(QSYS/QCPFMSG) +
                          MSGDTA(&MSGDTA) TOPGMQ(*PRV (*)) +
                          TOMSGQ(*TOPGMQ) MSGTYPE(*INFO)

 RETURN:
             RETURN

 ERROR:
             IF         COND(&AEBYTAVL *NE 0) THEN( +
             SNDPGMMSG  MSGID(&AEEXCPID) MSGF(QSYS/QCPFMSG) +
                          MSGDTA(&AEEXCPDTA) TOPGMQ(*PRV (*)) +
                          TOMSGQ(*TOPGMQ) MSGTYPE(*DIAG))
             CALL       PGM(QYASEDMS) PARM(&HANDLE &APIERROR)
             MONMSG     MSGID(CPF0000)
             CHGVAR     VAR(&MSGDTA) VALUE('Command ended with error')
             SNDPGMMSG  MSGID(CPF9898) MSGF(QSYS/QCPFMSG) +
                          MSGDTA(&MSGDTA) TOPGMQ(*PRV (*)) +
                          TOMSGQ(*TOPGMQ) MSGTYPE(*ESCAPE)

 ENDPGM:
             ENDPGM

---

After executing the command MPATHRESET ASP('1'), missing disk paths have disappeared from SST and CPI096E messages are no longer sent in the history log:

          Serial                     Resource   Path
ASP Unit  Number          Type Model Name       Status
  1    1  YF4FE4000034    2145  050  DMP005     Active
          YF4FE4000034    2145  050  DMP001     Passive
          YF4FE4000034    2145  050  DMP002     Active
          YF4FE4000034    2145  050  DMP015     Passive
  1    2  YF4FE4000035    2145  050  DMP006     Passive
          YF4FE4000035    2145  050  DMP003     Passive
          YF4FE4000035    2145  050  DMP016     Active
          YF4FE4000035    2145  050  DMP004     Active

---

References

IBM Redbook SG24-7994-00 “PowerHA SystemMirror for IBM i Cookbook”

IBM Redbook SG24-8197-00 “IBM i and IBM Storwize Family A Practical Guide to Usage Scenarios”