Replacing a faulty node with a spare node using the SAN Volume Controller Console Replacing objects Replacing the uninterruptible power supply

This task provides step-by-step instructions for replacing a faulty node in the cluster using the SAN Volume Controller Console.

Prerequisites

Before you attempt to replace a faulty node with a spare node you must ensure that:

Context

If a node fails, the cluster continues to operate with degraded performance, until the faulty node is repaired. If the repair operation is likely to take an unacceptable amount of time, it might be useful to replace the faulty node with a spare node. However, the appropriate procedures must be followed and precautions must be taken, in order not to interrupt I/O operations and to avoid compromising the integrity of your data. The procedures outlined in this topic involve changing the worldwide node name (WWNN) of a SAN Volume Controller. These procedures must be followed with care in order to avoid duplicate WWPNs which can cause data corruption.

Be aware that by performing these procedures the following changes will be made to your configuration:

Front Panel ID
This number will change. It is the number that is printed on the front of the node and used to select the node that is to be added to a cluster.
Node Name
This number might change. If you do not specify a name, the SAN Volume Controller assigns a default name when adding a node to a cluster. The SAN Volume Controller creates a new name each time a node is added to a cluster. If you choose to assign your own names then you need to type in the node name on the Adding a node to a cluster panel. If you are using scripts to perform management tasks on the cluster and those scripts use the node name, then by assigning the original name to a replacement node, you avoid the need to make changes to the scripts.
Node ID
This ID will change. A new node ID is assigned each time a node is added to a cluster; the node name remains the same following service activity on the cluster. You can use the node ID or the node name to perform management tasks on the cluster. However, if you are using scripts to perform those tasks, use the node name rather than the node ID.
Worldwide Node Name
This name will not change. The WWNN is used to uniquely identify the node and the fibre-channel ports. The WWNN of the spare node will change to that of the faulty node. The node replacement procedures must be followed exactly to avoid any duplication of WWNNs.
Worldwide Port Names
These names do not change. WWPNs are derived from the WWNN that is written to the spare (replacement) node as part of this procedure. For example, let's say the WWNN for a node is 50050768010000F6. The four WWPNs for this node would be derived as follows:
WWNN                                        50050768010000F6
WWNN displayed on front panel               000F6
WWPN Port 1                                 50050768014000F6
WWPN Port 2                                 50050768013000F6
WWPN Port 3                                 50050768011000F6
WWPN Port 4                                 50050768012000F6
Steps

Perform the following steps to replace a faulty node in the cluster:

  1. Verify the name and ID of the node that you wish to replace.

    Perform the following steps to verify the name and ID:

    1. Make sure that the SAN Volume Controller Console application is running on the cluster that contains the faulty node.
    2. Click Work with Nodes in the portfolio.
    3. Click Nodes.

      If the node is faulty, it will be shown as offline. Ensure the partner node in the I/O group is online.

      1. If the other node in the I/O group is offline, start Directed Maintenance Procedures to determine the fault.
      2. If you have been directed here by the DMPs, and subsequently the partner node in the I/O group has failed, see the procedure for Recovering from offline VDisks.

      If you are replacing the node for other reasons, determine the node you wish to replace and again ensure the partner node in the I/O group is online.

      1. If the partner node is offline, you will lose access to the VDisks that belong to this I/O group if you continue. Start the Directed Maintenance Procedures and fix the other node before proceeding.
  2. Find and record the following information about the faulty node:
    • Node name
    • I/O group name
    • Last five characters of the WWNN
    • Front panel ID
    • Uninterruptible power supply serial number
    1. To find and record the node name and I/O group name, click Work with Nodes in the portfolio.
    2. Click Nodes.

      The faulty node will be offline.

    3. Record the following information about the faulty node:
      • Node name
      • I/O group name
    4. To find and record the last five characters of the WWNN, click on the name of the offline node.
    5. Click the General tab.
    6. Record the last five characters of the WWNN.
    7. To find and record the front panel ID, click the Vital Product Data tab.
    8. Find the front-panel-assembly section of the vital product data (VPD).
    9. Record the front panel ID.
    10. To find and record the uninterruptible power supply serial number, click the Vital Product Data tab.
    11. Find the uninterruptible power supply section of the VPD.
    12. Record the uninterruptible power supply serial number.
  3. Obtain the ID of the faulty node. Disconnect all four fibre-channel cables from the node.

    Important: Do not plug the fibre-channel cables into the spare node until spare node has been configured with the WWNN from the faulty node.

  4. Connect the power and signal cables from the spare node to the uninterruptible power supply that has the serial number that you noted in step 2l.
    Note:Note: The signal cable can be plugged into any vacant position on the top row of serial connectors on the uninterruptible power supply. If no spare serial connectors are available on the uninterruptible power supply, disconnect the cables from the faulty SAN Volume Controller.
  5. Power-on the spare node.
  6. Display the node status on the service panel. See the IBM TotalStorage SAN Volume Controller: Service Guide for more information.
  7. Change the WWNN of the spare node.

    Perform the following steps to change the WWNN of the spare node so that it matches the WWNN of the faulty node:

    1. With the node status displayed on the front panel, press and hold the Down button; press and release the Select button; release the Down button. WWNN is displayed on line-1 of the display; line-2 of the display contains the last five characters of the WWNN.
    2. With the WWNN displayed on the service panel, press and hold the Down button, press and release the Select button, release the Down button. This switches the display into edit mode. Change the displayed number to match the WWNN recorded in step 2f.
      Note:Note: To edit the displayed number use the Up and Down buttons to increase or decrease the numbers displayed. Use the left and right buttons to move between fields. When the five characters match the number recorded in step 1, press the select button twice to accept the number.
  8. Connect the four fibre-channel cables that were disconnected from the faulty node to the spare node.
  9. Having noted the <nodename> in step 1, remove the faulty node from the cluster using the SAN Volume Controller Console.

    Remember: Record the following information:

    • Node serial number
    • WWNN
    • All WWPNs
    • I/O group that contains the node

    This can avoid a possible data corruption exposure when the node is re-added to the cluster.

  10. Add the spare node to the cluster using the SAN Volume Controller Console.
  11. Use the Subsystem Device Driver (SDD) management tool on the host systems to verify that all paths are now online. See the IBM TotalStorage Subsystem Device Driver: User's Guide for more information.
    Attention: Attention: When the faulty node is repaired do not connect the fibre-channel cables to it. Connecting the cables might cause data corruption.
  12. Repair the faulty node.
  13. If you want to use the repaired node as a spare node, perform the following steps:
    1. Display the node status on the front panel display of the node. See the IBM TotalStorage SAN Volume Controller: Service Guide for more information.
    2. With the node status displayed on the front panel, press and hold the Down button; press and release the Select button; release the Down button. WWNN is displayed on line-1 of the display; line-2 of the display contains the last five characters of the WWNN.
    3. With the WWNN displayed on the service panel, press and hold the Down button, press and release the Select button, release the Down button. This switches the display into edit mode. Change the displayed number to 00000.
      Note:Note: To edit the displayed number use the Up and Down buttons to increase or decrease the numbers displayed. Use the left and right buttons to move between fields.
      When the number is set to 00000, press the Select button twice to accept the number.

      This SAN Volume Controller can now be used as a spare node.

    Attention: Attention: Never connect a SAN Volume Controller with a WWNN of 00000 to the cluster. If this SAN Volume Controller is no longer required as a spare and is to be used for normal attachment to a cluster you must first use the procedure described in the "Prerequistes" to change the WWNN to the number you recorded when a spare was created. Using any other number might cause data corruption.
Related topics Replacing objects Replacing the uninterruptible power supply

(C) Copyright International Business Machines Corporation 2003, 2004