Replacing a faulty node in the cluster using the CLI Advanced functions with the CLI Recovering from offline VDisks after a node or an I/O group failed using the CLI

This task provides step-by-step instructions for replacing a faulty node in the cluster using the command-line interface (CLI).

Prerequisites

Before you attempt to replace a faulty node with a spare node you must ensure that:

Context

If a node fails, the cluster continues to operate with degraded performance, until the faulty node is repaired. If the repair operation is likely to take an unacceptable amount of time, it might be useful to replace the faulty node with a spare node. However, the appropriate procedures must be followed and precautions must be taken, in order not to interrupt I/O operations and to avoid compromising the integrity of your data. The procedures outlined in this topic involve changing the worldwide node name (WWNN) of a SAN Volume Controller. These procedures must be followed with care in order to avoid duplicate WWPNs which can cause data corruption.

Be aware that by performing these procedures the following changes will be made to your configuration:

Front Panel ID
This number will change. It is the number that is printed on the front of the node and used to select the node that is to be added to a cluster.
Node Name
This number might change. If you do not specify a name, the SAN Volume Controller assigns a default name when adding a node to a cluster. The SAN Volume Controller creates a new name each time a node is added to a cluster. If you choose to assign your own names then you need to type in the node name on the Adding a node to a cluster panel. If you are using scripts to perform management tasks on the cluster and those scripts use the node name, then by assigning the original name to a replacement node, you avoid the need to make changes to the scripts.
Node ID
This ID will change. A new node ID is assigned each time a node is added to a cluster; the node name remains the same following service activity on the cluster. You can use the node ID or the node name to perform management tasks on the cluster. However, if you are using scripts to perform those tasks, use the node name rather than the node ID.
Worldwide Node Name
This name will not change. The WWNN is used to uniquely identify the node and the fibre-channel ports. The WWNN of the spare node will change to that of the faulty node. The node replacement procedures must be followed exactly to avoid any duplication of WWNNs.
Worldwide Port Names
These names do not change. WWPNs are derived from the WWNN that is written to the spare (replacement) node as part of this procedure. For example, let's say the WWNN for a node is 50050768010000F6. The four WWPNs for this node would be derived as follows:
WWNN                                        50050768010000F6
WWNN displayed on front panel               000F6
WWPN Port 1                                 50050768014000F6
WWPN Port 2                                 50050768013000F6
WWPN Port 3                                 50050768011000F6
WWPN Port 4                                 50050768012000F6
Steps

Perform the following steps to replace a faulty node in the cluster:

  1. Verify the name and ID of the node that you wish to replace.

    Perform the following steps to verify the name and ID:

    1. Open a DOS window.
    2. Type the svcinfo lsnode command and hit Enter.

      If the node was faulty it will be shown as offline. Ensure the partner node in the I/O group is online.

      1. If the other node in the I/O group is offline, start the Directed Maintenance Procedures to determine the fault.
      2. If you have been directed here by the DMPs, and subsequently the partner node in the I/O group has failed.

      If you are replacing the node for other reasons, determine the node you wish to replace and again ensure the partner node in the I/O group is online.

      1. If the partner node is offline, you will lose access to the VDisks that belong to this I/O group if you continue. Start the Directed Maintenance Procedures and fix the other node before proceeding.
  2. Find and record the following information about the faulty node:
    • Node name
    • I/O group name
    • Last five characters of the WWNN
    • Front panel ID
    • UPS serial number
    1. To find and record the node name and I/O group name, type the svcinfo lsnode command and hit Enter.

      The faulty node will be offline.

    2. Record the following information about the faulty node:
      • Node name
      • I/O group name
    3. To find and record the last five characters of the WWNN, type the svcinfo lsnodevpd <nodename> command and hit Enter. <nodename> is the name that you recorded in step 1.
    4. Find the WWNN field in the output. Record the last five characters of the WWNN.
    5. To find and record the front panel ID, type the svcinfo lsnodevpd <nodename> command and hit Enter. <nodename> is the name that you recorded in step 1.
    6. Find the front_panel_id field in the output. Record the front panel ID.
    7. To find and record the UPS serial number, type the svcinfo lsnodevpd <nodename> command and hit Enter. <nodename> is the name that you recorded in step 1.
    8. Find the UPS_serial_number field in the output. Record the UPS serial number.
  3. Obtain the ID of the faulty node. Disconnect all four fibre-channel cables from the node.

    Important: Do not plug the fibre-channel cables into the spare node until spare node has been configured with the WWNN from the faulty node.

  4. Connect the power and signal cables from the spare node to the UPS that has the serial number that you noted in step 2h.
    Note:Note: The signal cable can be plugged into any vacant position on the top row of serial connectors on the UPS. If no spare serial connectors are available on the UPS, disconnect the cables from the faulty SAN Volume Controller.
  5. Power-on the spare node.
  6. Display the node status on the service panel. See the topic, "SAN Volume Controller menu options" in the IBM TotalStorage SAN Volume Controller: Service Guide for more information.
  7. Change the WWNN of the spare node.

    Perform the following steps to change the WWNN of the spare node so that it matches the WWNN of the faulty node:

    1. With the node status displayed on the front panel, press and hold the Down button; press and release the Select button; release the Down button. WWNN is displayed on line-1 of the display; line-2 of the display contains the last five characters of the WWNN.
    2. With the WWNN displayed on the service panel, press and hold the Down button, press and release the Select button, release the Down button. This switches the display into edit mode. Change the displayed number to match the WWNN recorded in 2d.
      Note:Note: To edit the displayed number use the Up and Down buttons to increase or decrease the numbers displayed. Use the Left and Right buttons to move between fields. When the five characters match the number recorded in step 1, press the Select button twice to accept the number.
  8. Connect the four fibre-channel cables that were disconnected from the faulty node to the spare node.
  9. Having noted the <nodename> in step 1, remove the node from the cluster by issuing the following svctask rmnode <nodename/id> command:

    Remember: Record the following information:

    • Node serial number
    • WWNN
    • All WWPNs
    • I/O group that contains the node

    This can avoid a possible data corruption exposure when the node is re-added to the cluster.

  10. Issue the following command to add the node back into the cluster: svctask addnode -wwnodename <WWNN> -iogrp <IOGRPNAME/ID> -name <NODENAME>.
  11. Use the Subsystem Device Driver (SDD) management tool on the host systems to verify that all paths are now online. See the IBM TotalStorage Subsystem Device Driver: User's Guide for more information.
    Attention: Attention: When the faulty node is repaired do not connect the fibre-channel cables to it. Connecting the cables might cause data corruption.
  12. Repair the faulty node.
  13. If you want to use the repaired node as a spare node, perform the following steps:
    1. Display the node status on the front panel display of the node. See the topic, "SAN Volume Controller menu options" in the IBM TotalStorage SAN Volume Controller: Service Guide for more information.
    2. With the node status displayed on the front panel, press and hold the Down button; press and release the Select button; release the Down button. WWNN is displayed on line-1 of the display; line-2 of the display contains the last five characters of the WWNN.
    3. With the WWNN displayed on the service panel, press and hold the Down button, press and release the Select button, release the Down button. This switches the display into edit mode. Change the displayed number to 00000.
      Note:Note: To edit the displayed number use the Up and Down buttons to increase or decrease the numbers displayed. Use the Left and Right buttons to move between fields.
      When the number is set to 00000, press the Select button twice to accept the number.

      This SAN Volume Controller can now be used as a spare node.

    Attention: Attention: Never connect a SAN Volume Controller with a WWNN of 00000 to the cluster. If this SAN Volume Controller is no longer required as a spare and is to be used for normal attachment to a cluster you must first use the procedure described at to change the WWNN to the number you recorded when a spare was created. Using any other number might cause data corruption.
  14. Issue the following command to add the node back into the cluster: svctask addnode -wwnodename <WWNN> -iogrp <IOGRPNAME/ID> -name <NODENAME>.
  15. Issue the svcinfo lsnode command to verify that the node is online.
Related topics Advanced functions with the CLI Recovering from offline VDisks after a node or an I/O group failed using the CLI

(C) Copyright International Business Machines Corporation 2003, 2004