Products/HA/DellConfigCluster/README

From DellLinuxWiki

Jump to: navigation, search

Dell|Red Hat HA Linux > dell-config-cluster > README

Contents

Dell | Red Hat High-Availability Cluster Configuration Script

Introduction

This set of scripts will help with the deployment and configuration of your Dell | Red Hat HA Cluster.

The script launches a GUI that gathers information to perform basic configuration of the cluster and the Dell-specific components. During run- time, you will be presented with a choice of a "Minimal Configuration" or a "Full (Simplified) Configuration."

When the "Minimal Configuration" is selected, the script will automate the following tasks:

  • Establish secure inter-node communications for the management console and the cluster nodes
  • Configure DRACs in the cluster nodes
  • Install relevant drivers for the selected Dell storage array
  • Install "luci" Conga management server on the management console
  • Install "ricci" Conga agent on the cluster nodes
  • Configure a basic cluster

See the About Minimal Configuration section for more information.

When the "Full Configuration" option is selected, the script will perform all of the actions outlined above and will also automate these additional tasks:

  • Configure DRACs or IPMI as fence devices
  • Create place-holder entries for network power switch fence devices
  • Configure shared storage using Global File System (GFS)
  • Configure and enable high availability for common applications (NFS, Web, Samba, and FTP)
  • Start the cluster services on each node

See the About Your HA Applications and Completing Configuration of Network Power Switches sections for more information.

Usage

Invoke the script with:

[root]# dell-config-cluster

or if necessary, use the full path:

[root]# /opt/dell/dell-config-cluster/dell-config-cluster

A "DEMO" mode is available if desired. This allows a test of the interface using fake machines. It similates actions that would take place during a normal iteration.

Invoke demo mode with:

[root]# dell-config-cluster-demo

or if necessary, use the full path:

[root]# /opt/dell/dell-config-cluster/dell-config-cluster-demo

or pass the option:

[root]# dell-config-cluster --demo

The script should be run from a management console, not a cluster node. Because the cluster nodes are rebooted during the configuration, the script cannot be run directly from one of the cluster nodes. After invoking the script, a GUI will be presented to interactively gather the information that is needed to perform the configuration of your cluster.

Pre-requisites

Before running the script, ensure that the following pre-requisites have been met:

  • A management node has been designated.
    • This system must have Red Hat Enterprise Linux 5, the X Window System, and a web browser installed.
    • If you are using a Dell EqualLogic PS-Series array in your cluster, a 32-bit web browser is recommended so that the Java plugin can be used to manage the storage array.
  • Hardware has been gathered, installed, and cabled
    • Cluster Nodes
    • Shared Storage Array
    • Cluster and Storage Networks
  • Each node has Red Hat Enterprise Linux 5 installed
  • Each cluster node has a root password assigned
  • The management console and cluster nodes are permitted to communicate using root credentials via SSH
  • The management console and cluster nodes are connected to RHN or a local satellite with the required packages
  • The management console and cluster nodes can communicate with linux.dell.com

For more information regarding how to complete these pre-requisites, refer to the Dell | Red Hat HA Cluster documentation at: http://linux.dell.com/wiki/index.php/Products/HA/DellRedHatHALinuxCluster

Files

  • README: This file
  • LICENSE: GPL v3 License
  • dell-config-cluster: The script that calls the python program
  • dell-config-cluster-gui.py: Operation script for the GUI
  • dell-config-cluster-gui.glade: Configuration file for the GUI
  • dellConfigCluster.py: Back-end configuration script
  • pexpect.py: Automates interactive prompts

License

This script is licensed under the GNU Public License. See LICENSE.

Known Issues

Issue #1: There is an issue where a mouse click will not be able to click a button that was previously disabled, if the mouse has not left the area of that button. This is a known issue, and a fix will be included in a gtk 2.14+. See http://bugzilla.gnome.org/show_bug.cgi?id=56070 for more information.

Workarounds include:

  1. Move mouse off the "Forward" button, then back on
  2. Use <Alt>+<F> to activate the Forward button
  3. Press <Space Bar> or <Enter> while the Forward button has focus (you may use <Tab> to change button focus)

Issue #2: The Dell firmware repository is not installed automatically. To manage firmware updates through this repo, install it on each node and execute the commands as indicated on the firmware wiki: http://linux.dell.com/wiki/index.php/Repository/firmware

Issue #3: If IPMI is selected and configured as a fencing device but is not enabled in the server Baseboard Management Controller (BMC) or Remote Access Controller (RAC) setup utility, then it will not be possible to fence a node using IPMI. The commands that are issued by the script will successfully configure the device, but you must ensure that "IPMI over LAN" has been enabled on each node. For more information regarding configuring the BMC or RAC, refer to the Dell OpenManage Server Administrator manuals at: http://support.dell.com/

Issue #4: If you select "Use existing SSH keys" on the Cluster Nodes page but are not using the SSH Agent to cache the passphrase, then you may see an OpenSSH dialog that prompts for the password for the first node. The dell-config-cluster script will not interact with this dialog, and you will also see a status message on the 1/3: Cluster Creation status page for each node that says: "ssh: requesting passphrase key. Please use ssh-add to pre-authorize for this session and try again."

In the event that you see this dialog box and this status message, follow these steps:

  1. Click Cancel to dismiss the OpenSSH dialog box.
  2. If a warning stating that "SSH password dialog could not grab the keyboard input" is displayed, click Close to dismiss this dialog box.
        NOTE: This may appear because the way that the script opens SSH connections to each node does not allow interaction from the keyboard.
  3. Open a Terminal session, and run ssh-add <path to private key>
        NOTE: If you do not provide a path, then the default (/root/.ssh/id_rsa) will be assumed.
  4. At the prompt, enter the passphrase associated with this key, and press <Enter>.
  5. You will see a confirmation stating Identity added.
  6. Verify that the public key has been added to the authorized_keys on each node.
  7. Attempt to connect to each node (ssh root@<node name>), and verify that you are not prompted to specify a passphrase.
  8. Return to the dell-config-cluster window. Click Back, then click Apply. The script should now be able to use your SSH keys and successfully connect to the nodes.

Issue #5: If the script opens a web browser for you (and you did not run dell-config-cluster as a background task), then the console from which the script was opened may not return a prompt until you have closed both the web browser and the script.

Issue #6: If you enter multiple invalid IP address (e.g. on the DRAC Configuration or IPMI Configuration pages), an individual IP address input field may remain red even after the entry in that field has been corrected. Once all entries are corrected, and you remove focus from the last entry field (e.g. by pressing <Tab>), then all addresses will be recognized as valid, and you can press Forward to continue.

Other Documentation

Dell Red Hat HA Linux Cluster documentation can be found online at: http://linux.dell.com/wiki/index.php/Products/HA/DellRedHatHALinuxCluster

A matrix of components that have been tested in a Dell Red Hat HA Linux Cluster is available at: http://www.dell.com/ha/

About the Script

"dell-config-cluster" was created as part of a partnership between Red Hat and Dell Engineering. Engineers from both parties have been validating and testing Dell hardware and storage running Red Hat Enterprise Linux Clustering and Global File System. This utility aims to assist in the creation of Dell validated components. Customers that are new to Red Hat Clustering and Global File System will particularly find this useful. Veterans of Conga and system-config-cluster may not find as much benefit.

  • Frontend GUI, Backend bugs and data gathering: Vinny Valdez, RHCA vvaldez@redhat.com
  • Backend and expect scripting: Robert Hentosh, robert_hentosh@dell.com
  • Brainstorming, testing, and documentation: Kevin Guinn, kevin_guinn@dell.com

Repositories

The hardware and software repos are installed on your nodes by default. The firmware repo is not installed at this time. For more information, visit the Dell Linux Wiki: http://linux.dell.com/wiki/index.php/Repository

Software

This repo is installed as part of dell-config-cluster. For more information see:

http://linux.dell.com/wiki/index.php/Repository/software

Hardware

This repo is installed as part of dell-config-cluster. For more information see:

http://linux.dell.com/wiki/index.php/Repository/hardware

Firmware

The firmware repository is available, but not installed in this version.

For more information, visit the firmware repository wiki:

http://linux.dell.com/wiki/index.php/Repository/firmware

You may run the following script to install the repo on your nodes:

[root] wget -q -O - http://linux.dell.com/repo/firmware/bootstrap.cgi | bash

About Minimal Configuration

The Minimal Configuration mode of dell-config-cluster is designed for advanced users. Its primary goal is to help automate the installation and configuration of Dell hardware.

When this mode is selected, the Dell respositories are installed and any drivers or software packages required for your DRAC or your selected storage array are installed on each node. Additionally, the Red Hat packages required for GFS and the Red Hat Cluster Suite are installed on each node.

The script also generates a very basic /etc/cluster/cluster.conf file, but does not start the cluster. Because neither shared storage volumes, fencing devices, nor applications are configured in this mode, the cluster is not started. However, the Conga management components are started.

To continue the configuration of your cluster, you can use the Red Hat management tools. For example, you can follow the procedures outlined in Adding Your Cluster to the Conga Interface to start using the management UI. This tool will allow you to perform the critical configuration tasks which are not automated when the Minimal Configuration is selected, including adding shared storage, fencing devices, and applications.

About Your HA Applications

When deploying a Full Configuration, the dell-config-cluster script simplifies the deployment of NFS, Web, Samba, and FTP as highly-available applications on your cluster. The initial configuration of each application is based on some assumptions, and may need to be customized further in your environment.

The script assumes that only one shared volume will be configured, and configures this volume to be shared among the nodes using GFS. The first volume that is recognized across the nodes will be configured with GFS, and all of the applications will be configured in sub-directories of the GFS volume. The SCSI ID and size of the detected shared volume are displayed on the Discovered Storage page. If you wish to use separate volumes for each application, you can let the script automate one application and use the Red Hat tools, such as Conga or system-config-cluster, to create subsequent Services on the cluster.


NFS

If you selected NFS, the nfs sub-directory of the mount point that you specified for your GFS volume (default: /cluster_storage/nfs) will be exported via NFS. By exporting the sub-directory, clients cannot inadvertently modify the files associated with other highly-available applications.

The list of allowed clients and client mount options that you specified in the script are the export options that will be applied when a client system mounts the NFS volume. These behave similarly to configuration entries in /etc/exports. For more information about these options, you can view the man page for the /etc/exports file by running man 5 exports.

Web

If you selected Web, the web server data and configuration files are located in the web sub-directory of the mount point that you specified for your GFS volume (e.g. /cluster_storage/web). Create your site in the html sub-directory of this tree.

Samba

If you selected Samba, the data in and under the samba sub-directory of the mount point that you specified for your GFS volume (e.g. /cluster_storage/samba) will be shared with the following settings:
Share name: pub
Access Control: share-level
Allowed Users: everybody
Access Level: read-only

The README file in this directory provides additional information about testing the share and changing these default properties.

FTP

If you selected FTP, the data in and under the ftp sub-directory of the mount point that you specified for your GFS volume (e.g. /cluster_storage/ftp) will be accessible via FTP. The README file in this directory lists the IP address for your highly-available FTP server, and provides information about the configuration file that will allow you to change the properties.

Security

SELinux

SELinux is not supported at this time. You must disable it on the management console and cluster nodes in order to use dell-config-cluster.

To disable SELinux, perform one of the following actions:

[root]# setenforce 0

To make this change persist across a reboot:

[root]# sed -i 's/^SELINUX=.*$/SELINUX=permissive/g' /etc/sysconfig/selinux

You may also use system-config-securitylevel to make this change. Select the SELinux tab, and change the SELinux Setting.

If you wish to enable SELinux on the cluster nodes, you will need to configure SELinux for iSCSI and your applications.

An existing bug documents an issue with iSCSI initiators and using multiple interfaces:

https://bugzilla.redhat.com/show_bug.cgi?id=460398

The fix is included in selinux-policy-2.4.6-150.el5 which is available at http://people.redhat.com/dwalsh/SELinux/RHEL5. This fix will appear in a future RHEL update.

Firewall

Your cluster and applications have been configured to allow traffic using iptables. Your firewall will be configured even if it was disabled. This is to allow you to apply the rules if you choose to enable your firewall at a later time.

CAUTION: If you use lokkit on the command-line or system-config-securitylevel on the cluster nodes to customize your firewall, these tools will delete the custom chains created for your cluster by dell-config-cluster. It is recommended that you backup the file /etc/sysconfig/iptables before making any changes to the firewall using lokkit or system-config-securitylevel.

Cluster

  • All cluster nodes will have a custom firewall chain called DCC-CLUSTER
  • The DCC-CLUSTER chain contains rules for ports that need to be opened for cluster commucations
  • A specific rule is created for each node in the cluster, for each of these ports, so these ports are not open to any other systems outside the cluster
  • A rule is created on each node to allow only the management console to manage it
  • The following ports are open on each cluster node:
    • cman udp ports 5404 and 5405 are allowed from any cluster node to any destination. This is due to the use of multicast.
    • ricci tcp port 11111 is allowed only from the management console to each cluster node
    • modclusterd tcp port 16851 is allowed from any cluster node to each cluster node
    • dlm tcp port 21064 is allowed from any cluster node to each cluster node
    • rgmanager tcp ports 41966, 41967, 41968, 41969 are allowed from any cluster node to each cluster node
    • ccsd tcp ports 50006, 50008, 50009 are allowed from any cluster node to each cluster node
    • ccsd udp port 50007 is allowed from any cluster node to each cluster node

For more information on these ports, see section 2.2. Enabling IP Ports in the Configuring and Managing a Red Hat Cluster for RHEL5.2 guide at http://www.redhat.com/docs/manuals/csgfs/

iSCSI

Port 3260 is allowed into the cluster nodes if iSCSI storage is selected.

Applications

  • All cluster nodes will have a custom firewall chain called DCC-APPS
  • The DCC-APPS chain contains rules to allow in all clients for each application defined, but only the specified IP address and ports required for that application are allowed.

NOTE: For tighter security control, it is recommended that you restrict these rules to only the clients that need access.

NFS
  • The cluster nodes are configured to use static NFS ports in /etc/sysconfig/nfs
    • rquotad port 875 is allowed into the NFS IP
    • lockd tcp port 32803, and udp port 32769 are allowed into the NFS IP
    • mountd port 892 is allowed into the NFS IP
    • statd port 662 is allowed into the NFS IP
  • The following tcp and udp ports are also added for NFS functionality:
    • portmap port 111 is allowed into the NFS IP
    • nfs port 2049 is allowed into the NFS IP
Web
  • http port 80 is allowed into the web IP
  • https port 443 is allowed into the web IP
Samba
  • Ports 139 and 445 tcp are allowed into the Samba IP
  • Ports 137 and 138 udp are allowed in, but due to the use of broadcast, it is allowed into any IP
  • A firewall module ip_conntrack_netbios_ns is loaded
FTP
  • Ports 20 and 21 tcp are allowed into the FTP IP
  • All ESTABLISHED and RELATED packets are allowed back into the FTP IP
  • A firewall module ip_conntrack_ftp is loaded

Adding Your Cluster to the Conga Interface

To add the cluster that was created using dell-config-cluster to the Conga management framework, follow these steps:

  1. Open the Conga UI with a web browser by clicking the link provided on the Complete page of the script (e.g. https://your.management.node:8084/).
    • NOTE: The Conga cluster management web server uses a self-signed certificate. You will need to either accept the certificate or create a security policy exception to allow the management UI to run.
  2. On the Please log in page, do the following:
    • Type admin as the Login Name
    • Type the password that you provided on the Cluster Information page of dell-config-cluster as the Password
    • Click Log in
  3. On the Luci Homebase page, click Add an Existing Cluster
  4. On the Add an Existing Cluster page, do the following:
    • Type the FQDN or IP address for one of your cluster nodes as the System Hostname
    • Type the password for the root user on that node as the Root Password
    • Click Submit
    • The other cluster nodes will be displayed
    • If the root user passwords on each node are the same, click Check if system passwords are identical. Otherwise, type the Root Password for each node in the appropriate field.
    • Click Submit
    • A confirmation box will appear. Click Yes to add the cluster to the Conga UI.
    • You will see a Status message indicating that the cluster is now managed.

Completing Configuration of Network Power Switches

If you selected Use a Network Power Switch on the Cluster Fencing page of the script, then the resulting /etc/cluster/cluster.conf file will contain stub entries for these devices, and you will need to perform some additional steps before these switches can properly fence your cluster nodes.

The Dell | Red Hat HA Linux Cluster has been tested with APC and WTI network power switch devices. Configuration steps for both types are listed below.

For more information, refer to Configuring and Managing a Red Hat Cluster at http://www.redhat.com/docs/manuals/csgfs/.

APC Network Power Switches

Follow these steps to configure your APC Network Power Switches as fence devices for your cluster:

Open the Conga UI

  1. Open the Conga UI with a web browser by clicking the link provided on the Complete page of the script (e.g. https://your.management.node:8084/).
  2. If you have not yet added your cluster to the management interface, follow the steps in Adding Your Cluster to the Conga Interface.
  3. Select the Cluster tab, and then click your Cluster Name (e.g. dell_cluster) under Choose a cluster to administer.

Configure the Network Power Switch as a Shared Fence Device

  1. Select Shared Fence Devices from the menu on the left-hand side.
  2. Find the fence device with an Agent type of APC Power Device.
  3. Verify that the nodes are listed under Nodes using this device for fencing.
  4. Click the name (e.g. apc) of this fence device to open the Fence Device Form.
  5. Fill in the IP Address, Login, Password, and -- if needed -- the Password Script (optional), then click Update this Fence Device.
  6. Click OK when prompted to Update fence device properties?

Configure Fencing for Each Cluster Node

  1. Configure Fencing for one node:
    1. Select Nodes from the menu on the left-hand side.
    2. Under the first listed node, click Manage Fencing for this Node.
    3. Scroll down to the table that includes Main Fencing Method.
    4. Find the device where the Fence type is APC Power Switch.
    5. Supply the Port and -- if needed -- Switch (optional) for this node.
    6. Click Update main fence properties.
    7. Click OK when prompted to Update this node's fence configuration?
  2. Select the next node from the menu on the left-hand side, and repeat these steps.
  3. If you have more than two cluster nodes, repeat these steps for each remaining node.

WTI Network Power Switches

Follow these steps to configure your WTI Network Power Switches as fence devices for your cluster, and to remove the stub entries for the APC device:

Open the Conga UI

  1. Open the Conga UI with a web browser by clicking the link provided on the Complete page of the script (e.g. https://your.management.node:8084/).
  2. If you have not yet added your cluster to the management interface, follow the steps in Adding Your Cluster to the Conga Interface.
  3. Select the Cluster tab, and then click your Cluster Name (e.g. dell_cluster) under Choose a cluster to administer.

Create a Shared Fence Device for the WTI Network Power Switch

  1. Select Shared Fence Devices from the menu on the left-hand side.
  2. Click Add a Fence Device
  3. Select WTI Power Switch from the Fencing Type list.
  4. Fill in the Name, IP Address, Login, Password, and -- if needed -- the Password Script (optional), then click Add this shared fence device.
  5. Click OK when prompted to Update fence device properties?
  6. You will be returned to the Cluster tab of the Conga UI. Click your Cluster Name (e.g. dell_cluster) under Choose a cluster to administer again.

Configure Fencing for Each Cluster Node

  1. Configure Fencing for one node:
    1. Select Nodes from the menu on the left-hand side.
    2. Under the first listed node, click Manage Fencing for this Node.
    3. Scroll down to the table that includes Main Fencing Method.
    4. In the Main Fencing Method column, click Add a fence device to this level.
    5. Select the fence device that you created from the Use an existing Fence Device drop-down list.
      • NOTE: The device will be listed by the Name that you specified when you created the shared fence device, and will list WTI Power Device in parentheses -- e.g. wti (WTI Power Device).
    6. Supply the Port that controls power for this node.
    7. Find the device where the Fence type is APC Power Switch, but where the IP Address and Login fields are blank.
    8. Click Remove this device to remove the stub APC entry as a potential fence device for the node.
    9. Click Update main fence properties.
    10. Click OK when prompted to Update this node's fence configuration?
  2. Select the next node from the menu on the left-hand side, and repeat these steps.
  3. If you have more than two cluster nodes, repeat these steps for each remaining node.

Remove the APC Shared Fence Device

  1. Select Shared Fence Devices from the menu on the left-hand side.
  2. Locate the device with an Agent Type of APC Power Device and a Name of apc.
  3. Verify that No nodes currently employ this fence device, and click the name apc to configure this device.
  4. Click Delete this fence device.
  5. Click OK when prompted to confirm that you wish to Delete this fence device?

MD3000 Storage Configuration Checklist

MD3000 Storage Configuration Checklist

MD3000i Storage Configuration Checklist

MD3000i Storage Configuration Checklist

EqualLogic Storage Configuration Checklist

EqualLogic PS-Series Storage Configuration Checklist

Personal tools
Distributions