Products/HA/DellRedHatHALinuxCluster/Storage/EqualLogic PS-Series/Cluster Nodes

From DellLinuxWiki

Jump to: navigation, search

Dell|Red Hat HA Linux > Storage > EqualLogic PS-Series > Cluster Nodes

Contents

Cluster Node Configuration

Each cluster node has multiple paths to the same virtual disk via the iSCSI I/O network. This could normally cause issues as the cluster nodes are able to access the same data through different paths that could lead to data corruption. A multipath I/O solution consolidates all paths to the virtual disk into a pseudo device. This allows the nodes to access the pseudo device instead of directly communicating with the block device via a single path. If a path failure occurs, the multipath I/O solution automatically switches paths, and the node continues to access the same data through the same pseudo device.

Red Hat Enterprise Linux 5.3 Advanced Platform includes a multipath driver as part of the base operating system with the package device-mapper-multipath. This package provides a multipath I/O solution for the Dell EqualLogic PS-Series arrays.

Assign IP Addresses to Storage NICs

  1. Identify which NIC ports have been cabled to the iSCSI I/O network.
  2. Assign IP addresses to each NIC port.

The tools ethtool and mii-tool can assist in determining which NICs have link.

Security Considerations

This section discusses the security changes needed on your cluster nodes.

Firewall

If a firewall will be used between the PS-Series storage array and the cluster nodes, ensure port 3260 is allowed both in and out of the cluster nodes to the EqualLogic target. For example, to allow communications from a Group IP address, enter the following commands on each cluster node:

  • Create a new chain for your rule:
[root]# iptables --new-chain EQUALLOGIC
  • Add a rule to the normal input chain to process your chain:
[root]# iptables --insert INPUT --jump EQUALLOGIC
  • Create a rule for iSCSI communications from the Group IP address (the following may be entered on one line without the "\"):
[root]# iptables --append EQUALLOGIC \
          --source {Group IP address} \
          --destination {IP address of iSCSI NIC on cluster node} \
          --protocol tcp \
          --source-port 3260 --jump ACCEPT
  • Create a rule for iSCSI communications from eth0 on the PS-Series array assigned in Initial Configuration:
[root]# iptables --append EQUALLOGIC \
          --source {Network Interface #1} \
          --destination {IP address of iSCSI NIC on cluster node} \
          --protocol tcp \
          --source-port 3260 --jump ACCEPT

NOTE: It may be simpler to allow the entire iSCSI network access instead. This can be accomplished by using an IP address and netmask, or CIDR notation in place of the source address. For example:

[root]# iptables --append EQUALLOGIC \
          --source 192.168.100.0/24 \
          --destination 192.168.100.0/24 \
          --protocol tcp \
          --source-port 3260 --jump ACCEPT
  • Save these rules:
[root]# service iptables save

SELinux

If SELinux is enabled, and your cluster nodes are running selinux-policy version less than 2.4.6-150.el5 then you may be unable to login to your iSCSI target from your initiators.

To determine if SELinux is enabled:

[root]# getenforce
Enforcing

Verify the policy version installed:

[root@node126 ~]# rpm -qa selinux-policy\*
selinux-policy-2.4.6-137.el5
selinux-policy-targeted-2.4.6-137.el5

The following is an example error message that will appear when trying to login to the target using multiple iSCSI interfaces:

Logging in to [iface: ieth0, target: iqn.2001-05.com.equallogic:XXXX-disk, portal: 192.168.100.1,3260]
Logging in to [iface: ieth1, target: iqn.2001-05.com.equallogic:XXXX-disk, portal: 192.168.100.1,3260]
iscsiadm: Could not login to [iface: ieth0, target: iqn.2001-05.com.equallogic:{XXXX}-disk, portal: 192.168.100.1,3260]: 
iscsiadm: initiator reported error (4 - encountered connection failure)
iscsiadm: Could not login to [iface: ieth1, target: iqn.2001-05.com.equallogic:{XXXX}-disk, portal: 192.168.100.1,3260]: 
iscsiadm: initiator reported error (4 - encountered connection failure)
iscsiadm: Could not log into all portals. Err 4.

If you inspect /var/log/audit/audit.log a message similar to the following will appear:

type=AVC msg=audit(1221684460.525:20): avc:  denied  { net_raw } for  pid=9104 comm="iscsid" capability=13
scontext=root:system_r:iscsid_t:s0 tcontext=root:system_r:iscsid_t:s0 tclass=capability type=SYSCALL
msg=audit(1221684460.525:20): arch=c000003e syscall=54 success=no exit=-1 a0=9 a1=1 a2=19 a3=16b110d8
items=0 ppid=1 pid=9104 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=2
comm="iscsid" exe="/sbin/iscsid" subj=root:system_r:iscsid_t:s0 key=(null)

An existing bug documents this issue:

https://bugzilla.redhat.com/show_bug.cgi?id=460398

The fix is included in selinux-policy-2.4.6-150.el5 and later which is available at http://people.redhat.com/dwalsh/SELinux/RHEL5. This fix will appear in a future RHEL update.

To disable SELinux, perform one of the following actions:

[root]# setenforce 0

To make this change persist across a reboot:

[root]# sed -i 's/^SELINUX=.*$/SELINUX=permissive/g' /etc/sysconfig/selinux

You may also use system-config-securitylevel to make this change. Select the SELinux tab, and change the SELinux Setting.


Install and Configure iSCSI Initiator

See the Script to Automate iSCSI Setup for automating many of these steps.

  • Install the iSCSI initiator utilities
[root]# yum install iscsi-initiator-utils
  • Determine which NICs are connected to the iSCSI I/O networks. The utilities ethtool or mii-tool may assist in determining which NICs have link.
  • Edit the file /etc/iscsi/initiator.name. This establishes the iSCSI qualified name (iqn). One will be randomly generated for you, but modifying this to a more easily identifiable name is recommended. This iqn will be used to associate your host with a host group. The industry accepted format is:
Format: iqn.yyyy-mm.{reversed domain name}:{unique identifier}

For example:

iqn.2008-08.com.example:node1
  • Start the iSCSI initiator:
[root]# service iscsi start
  • Configure the iSCSI initiator to start on boot:
[root]# chkconfig iscsi on

Configuring CHAP on the Nodes

If you are not using CHAP, skip to Discover Targets.

Configuring Initiator Authentication on the Nodes

Change the following settings in /etc/iscsi/iscsid.conf on each node. By default the settings are commented out, and contain default values. Uncomment each setting and change the value.

1. Enable CHAP for iSCSI sessions:

node.session.auth.authmethod = CHAP

2. Enable the CHAP username and password for iSCSI sessions that the initiator will use for authentication by the target:

node.session.auth.username = {local chapuser}
node.session.auth.password = {local chapuser password}
  • {local chapuser} and {local chapuser password} are the ones defined in Configuring Initiator Authentication on the Array in the Manage from the UI or Manage from the CLI documentation.

For example:

node.session.auth.username = nodes
node.session.auth.password = nodespassword

3. Enable CHAP for iSCSI discovery:

discovery.sendtargets.auth.authmethod = CHAP

4. Enable the CHAP username and password for iSCSI discovery that the initiator will use for authentication by the target:

discovery.sendtargets.auth.username = {local chapuser}
discovery.sendtargets.auth.password = {local chapuser password}

Example:

discovery.sendtargets.auth.username = nodes
discovery.sendtargets.auth.password = nodespassword

The following values should be set for Target CHAP (Initiator Authentication):

[root]# grep -E "(^node.session.auth|^discovery.sendtargets.auth)" /etc/iscsi/iscsid.conf
node.session.auth.authmethod = CHAP
node.session.auth.username = nodes
node.session.auth.password = nodespassword
discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = nodes
discovery.sendtargets.auth.password = nodespassword

Initiator Authentication is now complete. If Target Authentication will not be configured, skip to Starting the CHAP Session.

Configuring Target Authentication on the Nodes

NOTE: Complete the steps in Configuring Initiator Authentication on the Nodes first.

Change the following settings in /etc/iscsi/iscsid.conf on each node. Uncomment each setting and change the value.

1. Enable the CHAP username and password for iSCSI sessions that the target will use to for authentication by the initiator:

node.session.auth.username_in = {target username}
node.session.auth.password_in = {target password}

For example:

node.session.auth.username_in = target
node.session.auth.password_in = targetpassword

2. Enable the CHAP username and password for iSCSI discovery that the target will use for authentication by the initiator:

discovery.sendtargets.auth.username_in = {target username}
discovery.sendtargets.auth.password_in = {target password}

Example:

discovery.sendtargets.auth.username_in = target
discovery.sendtargets.auth.password_in = targetpassword

The following values should be set for both Target and Mutual CHAP (Initiator and Target Authentication):

[root]# grep -E "(^node.session.auth|^discovery.sendtargets.auth)" /etc/iscsi/iscsid.conf
node.session.auth.authmethod = CHAP
node.session.auth.username = nodes
node.session.auth.password = nodespassword
node.session.auth.username_in = target
node.session.auth.password_in = targetpassword
discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = nodes
discovery.sendtargets.auth.password = nodespassword
discovery.sendtargets.auth.username_in = target
discovery.sendtargets.auth.password_in = targetpassword

Starting the CHAP Session

Simply restart the iscsi service to use the new CHAP configuration:

[root]# service iscsi restart

Configure iSCSI Initiators for Multipath

NOTE: Perform these steps on all cluster nodes. For automating this steps on multiple nodes, see Script to Automate iSCSI Setup.

View current partition visible by the kernel for reference after the iSCSI volume has been added:

 [root]# cat /proc/partitions
 major minor  #blocks  name
 
    8     0   71041024 sda
    8     1     104391 sda1
    8     2   10241437 sda2
  253     0    8192000 dm-0
  253     1    2031616 dm-1
  253     2   24130560 dm-2
 

Verify which two storage NICs are on the same subnet as the EqualLogic target:

 [root]# ip addr show | grep 192.168
     inet 192.168.100.199/24 brd 192.168.100.255 scope global eth0
     inet 192.168.100.188/24 brd 192.168.100.255 scope global eth1

Verify connectivity to the storage target from each storage NIC:

 [root]# ping -I eth0 192.168.100.1
 [root]# ping -I eth1 192.168.100.1

Configure new iSCSI interfaces for that will be associated with your Storage NICs. In this example we are prepending the letter "i" in front of the name of the real interface for consistency, in practice this name could be anything you choose:

 [root]# iscsiadm -m iface -I ieth0 --op=new
 [root]# iscsiadm -m iface -I ieth1 --op=new

Associate the new iSCSI interfaces with your storage NICs:

 [root]# iscsiadm -m  iface -I ieth0 --op=update -n iface.net_ifacename -v eth0
 [root]# iscsiadm -m  iface -I ieth1 --op=update -n iface.net_ifacename -v eth1

Verify these interfaces were created and associated properly:

 [root]# iscsiadm -m iface
 ieth0 tcp,default,eth0
 ieth1 tcp,default,eth1

You may also verify these changes by viewing the database:

 [root]# cat /var/lib/iscsi/ifaces/ieth0
 iface.iscsi_ifacename = ieth0
 iface.net_ifacename = eth0
 iface.hwaddress = default
 iface.transport_name = tcp
 [root]# cat /var/lib/iscsi/ifaces/ieth1
 iface.iscsi_ifacename = ieth1
 iface.net_ifacename = eth1
 iface.hwaddress = default
 iface.transport_name = tcp

Discover Targets

Discover available targets on the PS-Series array:

 [root]# iscsiadm -m discovery -t st -p 192.168.100.1 -I ieth0 -I ieth1
 192.168.100.1:3260,1 iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume
 192.168.100.1:3260,1 iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume
 

Login to all targets:

[root]# iscsiadm -m node --login
Logging in to [iface: ieth1, target: iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume, portal: 192.168.100.1,3260]
Logging in to [iface: ieth0, target: iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume, portal: 192.168.100.1,3260]
  Vendor: EQLOGIC   Model: 100E-00           Rev: 4.1
  Type:   Direct-Access                      ANSI SCSI revision: 05
  Vendor: EQLOGIC   Model: 100E-00           Rev: 4.1
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdc: 10506240 512-byte hdwr sectors (5379 MB)
sdb: Write Protect is off
SCSI device sdb: drive cache: write through
sdc: Write Protect is off
SCSI device sdc: drive cache: write through
SCSI device sdb: 10506240 512-byte hdwr sectors (5379 MB)
SCSI device sdc: 10506240 512-byte hdwr sectors (5379 MB)
sdb: Write Protect is off
sdc: Write Protect is off
SCSI device sdb: drive cache: write through
sd 4:0:0:0: Attached scsi disk sdb
sd 4:0:0:0: Attached scsi generic sg2 type 0
 unknown partition table
sd 3:0:0:0: Attached scsi disk sdc
sd 3:0:0:0: Attached scsi generic sg3 type 0
Login to [iface: ieth0, target: iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume, portal: 192.168.200.1,3260]: successful
Login to [iface: ieth1, target: iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume, portal: 192.168.200.1,3260]: successful

View current partition visible by the kernel:

 [root]# cat /proc/partitions
major minor  #blocks  name
  
   8     0  142082048 sda
   8     1     104391 sda1
   8     2   20482875 sda2
 253     0   18448384 dm-0
 253     1    2031616 dm-1
   8    16    5253120 sdb
   8    32    5253120 sdc

The new volume appears as two disks in this case, sdb and sdc. This represents the two paths to the same volume

Display all logged in sessions, this should show a target for each storage NIC:

 [root]# iscsiadm -m session
tcp: [1] 192.168.100.1:3260,1 iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume
tcp: [2] 192.168.100.1:3260,1 iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume

Troubleshooting CHAP Issues

It may be helpful to enable debugging output during a discovery or login to ensure your new settings are being used:

[root]# iscsiadm -m node --login -d 8 2>&1 | grep -E "(CHAP|user|password)"
iscsiadm: updated 'node.session.auth.authmethod', 'None' => 'CHAP'
iscsiadm: updated 'node.session.auth.username',  => 'nodes'
iscsiadm: updated 'node.session.auth.password',  => 'nodespassword'
iscsiadm: updated 'node.session.auth.password_length', '0' => '13'
iscsiadm: updated 'node.session.auth.username_in',  => 'target'
iscsiadm: updated 'node.session.auth.password_in',  => 'targetpassword'
iscsiadm: updated 'node.session.auth.password_in_length', '0' => '14'
 

Discovery contains a lot more output, some of which has been omitted:

[root]# iscsiadm -m discovery -t sendtargets -p 192.168.100.1 -d 8 2>&1 | grep -E "(CHAP|user|password)"
iscsiadm: updated 'discovery.sendtargets.auth.authmethod', 'None' => 'CHAP'
iscsiadm: updated 'discovery.sendtargets.auth.username',  => 'nodes'
iscsiadm: updated 'discovery.sendtargets.auth.password',  => 'nodespassword'
iscsiadm: updated 'discovery.sendtargets.auth.password_length', '0' => '13'
iscsiadm: updated 'discovery.sendtargets.auth.username_in',  => 'target'
iscsiadm: updated 'discovery.sendtargets.auth.password_in',  => 'targetpassword'
iscsiadm: updated 'discovery.sendtargets.auth.password_in_length', '0' => '14'
iscsiadm: updated 'node.session.auth.authmethod', 'None' => 'CHAP'
iscsiadm: updated 'node.session.auth.username',  => 'nodes'
iscsiadm: updated 'node.session.auth.password',  => 'nodespassword'
iscsiadm: updated 'node.session.auth.password_length', '0' => '13'
iscsiadm: updated 'node.session.auth.username_in',  => 'target'
iscsiadm: updated 'node.session.auth.password_in',  => 'targetpassword'
iscsiadm: updated 'node.session.auth.password_in_length', '0' => '14'
iscsiadm: >    AuthMethod=CHAP
iscsiadm: >    AuthMethod=CHAP
iscsiadm: >    CHAP_A=5
iscsiadm: >    CHAP_A=5
iscsiadm: >    CHAP_I=0
iscsiadm: >    CHAP_C=0xb9a09255b9b1208315da1c3afba38750
iscsiadm: >    CHAP_N=nodes-user
iscsiadm: >    CHAP_R=0xEncryptedPassword
iscsiadm: >    CHAP_I=247
iscsiadm: >    CHAP_C=0x731e04dda0ac69527a5a2a93b50a778d
iscsiadm: >    CHAP_N=target
iscsiadm: >    CHAP_R=0xEncryptedPassword
<output omitted>

The lines begining with iscsiadm: > are output from the discovery that shows what usernames the target is sending to the node.

In some cases, you may need to delete any existing iscsi settings if any sessions were established before configuring CHAP.

First stop the iscsi service:

[root]# service iscsi stop

Remove iscsi-initiator-utils:

[root]# yum remove iscsi-initiator-utils

Delete any existing session data:

[root]# rm -rf /var/lib/iscsi

Reinstall iscsi-initiator-utils:

[root]# yum install iscsi-initiator-utils

You may have noticed a warning message when the package was being removed:

warning: /etc/iscsi/iscsid.conf saved as /etc/iscsi/iscsid.conf.rpmsave

Use this to your advantage since it contains all of your existing configuration data:

[root]# cat /etc/iscsi/iscsid.conf.rpmsave > /etc/iscsi/iscsid.conf

Double-check your configuration:

[root]# grep -E "(^node.session.auth|^discovery.sendtargets.auth)" /etc/iscsi/iscsid.conf

Start the iscsi service:

[root]# service iscsi start

Configure Device Mapper Multipath for the PS-Series Arrays

Enable multipath on all devices except your local disk. First determine what device your local disk are. This can be done in a variety of ways.

Ensure device-mapper-multipath is installed:

[root]# yum install device-mapper-multipath

If you are using LVM:

[root]# pvs
  PV         VG         Fmt  Attr PSize   PFree
  /dev/sda2  VolGroup00 lvm2 a-     9.75G    0

In this example the local disk is /dev/sda

If you are not using LVM, you may use the mount command:

[root]# mount
<extra output omitted>
/dev/sda1 on /boot type ext3 (rw)

Once the local disk is determined, obtain the scsi id of that disk:

[root]# scsi_id -gus /block/sda
36001e4f010fde3000f39f9f6066b384c

Now edit the multipath configuration to enable multipath on your non-local devices:

[root]# vi /etc/multipath.conf

Remove the line devnode "*"

blacklist {
        devnode "*"
}

Add the scsi id of your local disk:

blacklist {
        wwid 36001e4f010fde3000f39f9f6066b384c
}

It is also highly recommended that you disable the "user_friendly_names" option that is set by default. This allows your devices to be named according to their scsi_id, rather than a generic "mpath" name:

Change:

defaults {
        user_friendly_names yes
}

To:

defaults {
        user_friendly_names no
}
   

Enable multipathd on boot:

[root]# chkconfig multipathd on

Start multipathd:

[root]# service multipathd start

List multipath devices:

[root]# multipath -ll
36090a02830873505809c640100007047 dm-2 EQLOGIC,100E-00
[size=5.0G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
 \_ 3:0:0:0 sdc 8:32  [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 4:0:0:0 sdd 8:48  [active][ready]

NOTE: The /dev/dm- devices are used internally to device-mapper, and should never be used directly.

Verify that the multipath device exists under /dev/mpath and /dev/mapper

[root]# ls /dev/{mpath,mapper}
/dev/mapper:
36090a02830873505809c640100007047  VolGroup00-LogVol00
control                            VolGroup00-LogVol01

/dev/mpath:
36090a02830873505809c640100007047

This multipath device should match the scsi_id for both devices on each storage NIC path:

[root]# scsi_id -xgus /block/sdb
ID_VENDOR=EQLOGIC
ID_MODEL=100E-00
ID_REVISION=4.1
ID_SERIAL=36090a02830873505809c640100007047
ID_TYPE=disk
ID_BUS=scsi

[root]# scsi_id -xgus /block/sdc
ID_VENDOR=EQLOGIC
ID_MODEL=100E-00
ID_REVISION=4.1
ID_SERIAL=36090a02830873505809c640100007047
ID_TYPE=disk
ID_BUS=scsi

More detailed information is available with the multipath command:

[root]# multipath -v3 -d

Using Your Multipath Device

The device to access is now under /dev/mapper. This is the device that is manipulated for creating partitions, adding as a physical volume, and creating filesystems on. For example, to manipulate your new disk's partitions, use parted on the dm-multipath device:

 [root]# parted /dev/mapper/36090a02830873505809c640100007047

An optional feature you can also use is alias. You can create a stanza in /etc/multipathd.conf for the scsi_id of your volume, and give it a name of your choice. For example:

multipaths {
        multipath {
                wwid                    36090a02830873505809c640100007047
                alias                   cluster
        }
}

After restarting multipathd, your volume will appear with the alias:

[root]# ls /dev/mapper
/dev/mapper:
cluster  clusterp1  control  VolGroup00-LogVol00  VolGroup00-LogVol01

For more information on device-mapper-multipath and additional options it supports, see Using Device-Mapper Multipath on the Red Hat website at www.redhat.com/docs/manuals/enterprise/.



Continue to the Appendix



Dell|Red Hat HA Linux > Storage > EqualLogic PS-Series > Cluster Nodes

Personal tools
Distributions