Products/HA/DellRedHatHALinuxCluster/Storage/EqualLogic PS-Series/Cluster Nodes
From DellLinuxWiki
Dell|Red Hat HA Linux > Storage > EqualLogic PS-Series > Cluster Nodes
Cluster Node Configuration
Each cluster node has multiple paths to the same virtual disk via the iSCSI I/O network. This could normally cause issues as the cluster nodes are able to access the same data through different paths that could lead to data corruption. A multipath I/O solution consolidates all paths to the virtual disk into a pseudo device. This allows the nodes to access the pseudo device instead of directly communicating with the block device via a single path. If a path failure occurs, the multipath I/O solution automatically switches paths, and the node continues to access the same data through the same pseudo device.
Red Hat Enterprise Linux 5.3 Advanced Platform includes a multipath driver as part of the base operating system with the package device-mapper-multipath. This package provides a multipath I/O solution for the Dell EqualLogic PS-Series arrays.
Assign IP Addresses to Storage NICs
- Identify which NIC ports have been cabled to the iSCSI I/O network.
- Assign IP addresses to each NIC port.
The tools ethtool and mii-tool can assist in determining which NICs have link.
Security Considerations
This section discusses the security changes needed on your cluster nodes.
Firewall
If a firewall will be used between the PS-Series storage array and the cluster nodes, ensure port 3260 is allowed both in and out of the cluster nodes to the EqualLogic target. For example, to allow communications from a Group IP address, enter the following commands on each cluster node:
- Create a new chain for your rule:
[root]# iptables --new-chain EQUALLOGIC
- Add a rule to the normal input chain to process your chain:
[root]# iptables --insert INPUT --jump EQUALLOGIC
- Create a rule for iSCSI communications from the Group IP address (the following may be entered on one line without the "\"):
[root]# iptables --append EQUALLOGIC \
--source {Group IP address} \
--destination {IP address of iSCSI NIC on cluster node} \
--protocol tcp \
--source-port 3260 --jump ACCEPT
- Create a rule for iSCSI communications from eth0 on the PS-Series array assigned in Initial Configuration:
[root]# iptables --append EQUALLOGIC \
--source {Network Interface #1} \
--destination {IP address of iSCSI NIC on cluster node} \
--protocol tcp \
--source-port 3260 --jump ACCEPT
- Repeat this for each network interface that was enabled in Additional Network Interface Configuration in the Manage from the UI or Manage from the CLI or .
- Repeat each of these rules for each destination IP address on the cluster nodes that were configured in Assign IP Addresses to Storage NICs.
NOTE: It may be simpler to allow the entire iSCSI network access instead. This can be accomplished by using an IP address and netmask, or CIDR notation in place of the source address. For example:
[root]# iptables --append EQUALLOGIC \ --source 192.168.100.0/24 \ --destination 192.168.100.0/24 \ --protocol tcp \ --source-port 3260 --jump ACCEPT
- Save these rules:
[root]# service iptables save
SELinux
If SELinux is enabled, and your cluster nodes are running selinux-policy version less than 2.4.6-150.el5 then you may be unable to login to your iSCSI target from your initiators.
To determine if SELinux is enabled:
[root]# getenforce Enforcing
Verify the policy version installed:
[root@node126 ~]# rpm -qa selinux-policy\* selinux-policy-2.4.6-137.el5 selinux-policy-targeted-2.4.6-137.el5
The following is an example error message that will appear when trying to login to the target using multiple iSCSI interfaces:
Logging in to [iface: ieth0, target: iqn.2001-05.com.equallogic:XXXX-disk, portal: 192.168.100.1,3260]
Logging in to [iface: ieth1, target: iqn.2001-05.com.equallogic:XXXX-disk, portal: 192.168.100.1,3260]
iscsiadm: Could not login to [iface: ieth0, target: iqn.2001-05.com.equallogic:{XXXX}-disk, portal: 192.168.100.1,3260]:
iscsiadm: initiator reported error (4 - encountered connection failure)
iscsiadm: Could not login to [iface: ieth1, target: iqn.2001-05.com.equallogic:{XXXX}-disk, portal: 192.168.100.1,3260]:
iscsiadm: initiator reported error (4 - encountered connection failure)
iscsiadm: Could not log into all portals. Err 4.
If you inspect /var/log/audit/audit.log a message similar to the following will appear:
type=AVC msg=audit(1221684460.525:20): avc: denied { net_raw } for pid=9104 comm="iscsid" capability=13
scontext=root:system_r:iscsid_t:s0 tcontext=root:system_r:iscsid_t:s0 tclass=capability type=SYSCALL
msg=audit(1221684460.525:20): arch=c000003e syscall=54 success=no exit=-1 a0=9 a1=1 a2=19 a3=16b110d8
items=0 ppid=1 pid=9104 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=2
comm="iscsid" exe="/sbin/iscsid" subj=root:system_r:iscsid_t:s0 key=(null)
An existing bug documents this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=460398
The fix is included in selinux-policy-2.4.6-150.el5 and later which is available at http://people.redhat.com/dwalsh/SELinux/RHEL5. This fix will appear in a future RHEL update.
To disable SELinux, perform one of the following actions:
[root]# setenforce 0
To make this change persist across a reboot:
[root]# sed -i 's/^SELINUX=.*$/SELINUX=permissive/g' /etc/sysconfig/selinux
You may also use system-config-securitylevel to make this change. Select the SELinux tab, and change the SELinux Setting.
Install and Configure iSCSI Initiator
See the Script to Automate iSCSI Setup for automating many of these steps.
- Install the iSCSI initiator utilities
[root]# yum install iscsi-initiator-utils
- Determine which NICs are connected to the iSCSI I/O networks. The utilities ethtool or mii-tool may assist in determining which NICs have link.
- Edit the file /etc/iscsi/initiator.name. This establishes the iSCSI qualified name (iqn). One will be randomly generated for you, but modifying this to a more easily identifiable name is recommended. This iqn will be used to associate your host with a host group. The industry accepted format is:
Format: iqn.yyyy-mm.{reversed domain name}:{unique identifier}
For example:
iqn.2008-08.com.example:node1
- Start the iSCSI initiator:
[root]# service iscsi start
- Configure the iSCSI initiator to start on boot:
[root]# chkconfig iscsi on
Configuring CHAP on the Nodes
If you are not using CHAP, skip to Discover Targets.
Configuring Initiator Authentication on the Nodes
Change the following settings in /etc/iscsi/iscsid.conf on each node. By default the settings are commented out, and contain default values. Uncomment each setting and change the value.
1. Enable CHAP for iSCSI sessions:
node.session.auth.authmethod = CHAP
2. Enable the CHAP username and password for iSCSI sessions that the initiator will use for authentication by the target:
node.session.auth.username = {local chapuser}
node.session.auth.password = {local chapuser password}
- {local chapuser} and {local chapuser password} are the ones defined in Configuring Initiator Authentication on the Array in the Manage from the UI or Manage from the CLI documentation.
For example:
node.session.auth.username = nodes node.session.auth.password = nodespassword
3. Enable CHAP for iSCSI discovery:
discovery.sendtargets.auth.authmethod = CHAP
4. Enable the CHAP username and password for iSCSI discovery that the initiator will use for authentication by the target:
discovery.sendtargets.auth.username = {local chapuser}
discovery.sendtargets.auth.password = {local chapuser password}
Example:
discovery.sendtargets.auth.username = nodes discovery.sendtargets.auth.password = nodespassword
The following values should be set for Target CHAP (Initiator Authentication):
[root]# grep -E "(^node.session.auth|^discovery.sendtargets.auth)" /etc/iscsi/iscsid.conf node.session.auth.authmethod = CHAP node.session.auth.username = nodes node.session.auth.password = nodespassword discovery.sendtargets.auth.authmethod = CHAP discovery.sendtargets.auth.username = nodes discovery.sendtargets.auth.password = nodespassword
Initiator Authentication is now complete. If Target Authentication will not be configured, skip to Starting the CHAP Session.
Configuring Target Authentication on the Nodes
NOTE: Complete the steps in Configuring Initiator Authentication on the Nodes first.
Change the following settings in /etc/iscsi/iscsid.conf on each node. Uncomment each setting and change the value.
1. Enable the CHAP username and password for iSCSI sessions that the target will use to for authentication by the initiator:
node.session.auth.username_in = {target username}
node.session.auth.password_in = {target password}
- {target username} and {target password} are the vaules created in Configuring Target Authentication on the Array in Manage from the UI or Manage from the CLI.
For example:
node.session.auth.username_in = target node.session.auth.password_in = targetpassword
2. Enable the CHAP username and password for iSCSI discovery that the target will use for authentication by the initiator:
discovery.sendtargets.auth.username_in = {target username}
discovery.sendtargets.auth.password_in = {target password}
Example:
discovery.sendtargets.auth.username_in = target discovery.sendtargets.auth.password_in = targetpassword
The following values should be set for both Target and Mutual CHAP (Initiator and Target Authentication):
[root]# grep -E "(^node.session.auth|^discovery.sendtargets.auth)" /etc/iscsi/iscsid.conf node.session.auth.authmethod = CHAP node.session.auth.username = nodes node.session.auth.password = nodespassword node.session.auth.username_in = target node.session.auth.password_in = targetpassword discovery.sendtargets.auth.authmethod = CHAP discovery.sendtargets.auth.username = nodes discovery.sendtargets.auth.password = nodespassword discovery.sendtargets.auth.username_in = target discovery.sendtargets.auth.password_in = targetpassword
Starting the CHAP Session
Simply restart the iscsi service to use the new CHAP configuration:
[root]# service iscsi restart
Configure iSCSI Initiators for Multipath
NOTE: Perform these steps on all cluster nodes. For automating this steps on multiple nodes, see Script to Automate iSCSI Setup.
View current partition visible by the kernel for reference after the iSCSI volume has been added:
[root]# cat /proc/partitions
major minor #blocks name
8 0 71041024 sda
8 1 104391 sda1
8 2 10241437 sda2
253 0 8192000 dm-0
253 1 2031616 dm-1
253 2 24130560 dm-2
Verify which two storage NICs are on the same subnet as the EqualLogic target:
[root]# ip addr show | grep 192.168
inet 192.168.100.199/24 brd 192.168.100.255 scope global eth0
inet 192.168.100.188/24 brd 192.168.100.255 scope global eth1
Verify connectivity to the storage target from each storage NIC:
[root]# ping -I eth0 192.168.100.1 [root]# ping -I eth1 192.168.100.1
Configure new iSCSI interfaces for that will be associated with your Storage NICs. In this example we are prepending the letter "i" in front of the name of the real interface for consistency, in practice this name could be anything you choose:
[root]# iscsiadm -m iface -I ieth0 --op=new [root]# iscsiadm -m iface -I ieth1 --op=new
Associate the new iSCSI interfaces with your storage NICs:
[root]# iscsiadm -m iface -I ieth0 --op=update -n iface.net_ifacename -v eth0 [root]# iscsiadm -m iface -I ieth1 --op=update -n iface.net_ifacename -v eth1
Verify these interfaces were created and associated properly:
[root]# iscsiadm -m iface ieth0 tcp,default,eth0 ieth1 tcp,default,eth1
You may also verify these changes by viewing the database:
[root]# cat /var/lib/iscsi/ifaces/ieth0 iface.iscsi_ifacename = ieth0 iface.net_ifacename = eth0 iface.hwaddress = default iface.transport_name = tcp
[root]# cat /var/lib/iscsi/ifaces/ieth1 iface.iscsi_ifacename = ieth1 iface.net_ifacename = eth1 iface.hwaddress = default iface.transport_name = tcp
Discover Targets
Discover available targets on the PS-Series array:
[root]# iscsiadm -m discovery -t st -p 192.168.100.1 -I ieth0 -I ieth1 192.168.100.1:3260,1 iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume 192.168.100.1:3260,1 iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume
Login to all targets:
[root]# iscsiadm -m node --login Logging in to [iface: ieth1, target: iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume, portal: 192.168.100.1,3260] Logging in to [iface: ieth0, target: iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume, portal: 192.168.100.1,3260] Vendor: EQLOGIC Model: 100E-00 Rev: 4.1 Type: Direct-Access ANSI SCSI revision: 05 Vendor: EQLOGIC Model: 100E-00 Rev: 4.1 Type: Direct-Access ANSI SCSI revision: 05 SCSI device sdc: 10506240 512-byte hdwr sectors (5379 MB) sdb: Write Protect is off SCSI device sdb: drive cache: write through sdc: Write Protect is off SCSI device sdc: drive cache: write through SCSI device sdb: 10506240 512-byte hdwr sectors (5379 MB) SCSI device sdc: 10506240 512-byte hdwr sectors (5379 MB) sdb: Write Protect is off sdc: Write Protect is off SCSI device sdb: drive cache: write through sd 4:0:0:0: Attached scsi disk sdb sd 4:0:0:0: Attached scsi generic sg2 type 0 unknown partition table sd 3:0:0:0: Attached scsi disk sdc sd 3:0:0:0: Attached scsi generic sg3 type 0 Login to [iface: ieth0, target: iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume, portal: 192.168.200.1,3260]: successful Login to [iface: ieth1, target: iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume, portal: 192.168.200.1,3260]: successful
View current partition visible by the kernel:
[root]# cat /proc/partitions major minor #blocks name 8 0 142082048 sda 8 1 104391 sda1 8 2 20482875 sda2 253 0 18448384 dm-0 253 1 2031616 dm-1 8 16 5253120 sdb 8 32 5253120 sdc
The new volume appears as two disks in this case, sdb and sdc. This represents the two paths to the same volume
Display all logged in sessions, this should show a target for each storage NIC:
[root]# iscsiadm -m session tcp: [1] 192.168.100.1:3260,1 iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume tcp: [2] 192.168.100.1:3260,1 iqn.2001-05.com.equallogic:0-8a0906-053587302-4770000001649c80-my-volume
Troubleshooting CHAP Issues
It may be helpful to enable debugging output during a discovery or login to ensure your new settings are being used:
[root]# iscsiadm -m node --login -d 8 2>&1 | grep -E "(CHAP|user|password)" iscsiadm: updated 'node.session.auth.authmethod', 'None' => 'CHAP' iscsiadm: updated 'node.session.auth.username', => 'nodes' iscsiadm: updated 'node.session.auth.password', => 'nodespassword' iscsiadm: updated 'node.session.auth.password_length', '0' => '13' iscsiadm: updated 'node.session.auth.username_in', => 'target' iscsiadm: updated 'node.session.auth.password_in', => 'targetpassword' iscsiadm: updated 'node.session.auth.password_in_length', '0' => '14'
Discovery contains a lot more output, some of which has been omitted:
[root]# iscsiadm -m discovery -t sendtargets -p 192.168.100.1 -d 8 2>&1 | grep -E "(CHAP|user|password)" iscsiadm: updated 'discovery.sendtargets.auth.authmethod', 'None' => 'CHAP' iscsiadm: updated 'discovery.sendtargets.auth.username', => 'nodes' iscsiadm: updated 'discovery.sendtargets.auth.password', => 'nodespassword' iscsiadm: updated 'discovery.sendtargets.auth.password_length', '0' => '13' iscsiadm: updated 'discovery.sendtargets.auth.username_in', => 'target' iscsiadm: updated 'discovery.sendtargets.auth.password_in', => 'targetpassword' iscsiadm: updated 'discovery.sendtargets.auth.password_in_length', '0' => '14' iscsiadm: updated 'node.session.auth.authmethod', 'None' => 'CHAP' iscsiadm: updated 'node.session.auth.username', => 'nodes' iscsiadm: updated 'node.session.auth.password', => 'nodespassword' iscsiadm: updated 'node.session.auth.password_length', '0' => '13' iscsiadm: updated 'node.session.auth.username_in', => 'target' iscsiadm: updated 'node.session.auth.password_in', => 'targetpassword' iscsiadm: updated 'node.session.auth.password_in_length', '0' => '14' iscsiadm: > AuthMethod=CHAP iscsiadm: > AuthMethod=CHAP iscsiadm: > CHAP_A=5 iscsiadm: > CHAP_A=5 iscsiadm: > CHAP_I=0 iscsiadm: > CHAP_C=0xb9a09255b9b1208315da1c3afba38750 iscsiadm: > CHAP_N=nodes-user iscsiadm: > CHAP_R=0xEncryptedPassword iscsiadm: > CHAP_I=247 iscsiadm: > CHAP_C=0x731e04dda0ac69527a5a2a93b50a778d iscsiadm: > CHAP_N=target iscsiadm: > CHAP_R=0xEncryptedPassword <output omitted>
The lines begining with iscsiadm: > are output from the discovery that shows what usernames the target is sending to the node.
In some cases, you may need to delete any existing iscsi settings if any sessions were established before configuring CHAP.
First stop the iscsi service:
[root]# service iscsi stop
Remove iscsi-initiator-utils:
[root]# yum remove iscsi-initiator-utils
Delete any existing session data:
[root]# rm -rf /var/lib/iscsi
Reinstall iscsi-initiator-utils:
[root]# yum install iscsi-initiator-utils
You may have noticed a warning message when the package was being removed:
warning: /etc/iscsi/iscsid.conf saved as /etc/iscsi/iscsid.conf.rpmsave
Use this to your advantage since it contains all of your existing configuration data:
[root]# cat /etc/iscsi/iscsid.conf.rpmsave > /etc/iscsi/iscsid.conf
Double-check your configuration:
[root]# grep -E "(^node.session.auth|^discovery.sendtargets.auth)" /etc/iscsi/iscsid.conf
Start the iscsi service:
[root]# service iscsi start
Configure Device Mapper Multipath for the PS-Series Arrays
Enable multipath on all devices except your local disk. First determine what device your local disk are. This can be done in a variety of ways.
Ensure device-mapper-multipath is installed:
[root]# yum install device-mapper-multipath
If you are using LVM:
[root]# pvs PV VG Fmt Attr PSize PFree /dev/sda2 VolGroup00 lvm2 a- 9.75G 0
In this example the local disk is /dev/sda
If you are not using LVM, you may use the mount command:
[root]# mount <extra output omitted> /dev/sda1 on /boot type ext3 (rw)
Once the local disk is determined, obtain the scsi id of that disk:
[root]# scsi_id -gus /block/sda 36001e4f010fde3000f39f9f6066b384c
Now edit the multipath configuration to enable multipath on your non-local devices:
[root]# vi /etc/multipath.conf
Remove the line devnode "*"
blacklist {
devnode "*"
}
Add the scsi id of your local disk:
blacklist {
wwid 36001e4f010fde3000f39f9f6066b384c
}
It is also highly recommended that you disable the "user_friendly_names" option that is set by default. This allows your devices to be named according to their scsi_id, rather than a generic "mpath" name:
Change:
defaults {
user_friendly_names yes
}
To:
defaults {
user_friendly_names no
}
Enable multipathd on boot:
[root]# chkconfig multipathd on
Start multipathd:
[root]# service multipathd start
List multipath devices:
[root]# multipath -ll 36090a02830873505809c640100007047 dm-2 EQLOGIC,100E-00 [size=5.0G][features=0][hwhandler=0][rw] \_ round-robin 0 [prio=1][active] \_ 3:0:0:0 sdc 8:32 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 4:0:0:0 sdd 8:48 [active][ready]
NOTE: The /dev/dm- devices are used internally to device-mapper, and should never be used directly.
Verify that the multipath device exists under /dev/mpath and /dev/mapper
[root]# ls /dev/{mpath,mapper}
/dev/mapper:
36090a02830873505809c640100007047 VolGroup00-LogVol00
control VolGroup00-LogVol01
/dev/mpath:
36090a02830873505809c640100007047
This multipath device should match the scsi_id for both devices on each storage NIC path:
[root]# scsi_id -xgus /block/sdb ID_VENDOR=EQLOGIC ID_MODEL=100E-00 ID_REVISION=4.1 ID_SERIAL=36090a02830873505809c640100007047 ID_TYPE=disk ID_BUS=scsi [root]# scsi_id -xgus /block/sdc ID_VENDOR=EQLOGIC ID_MODEL=100E-00 ID_REVISION=4.1 ID_SERIAL=36090a02830873505809c640100007047 ID_TYPE=disk ID_BUS=scsi
More detailed information is available with the multipath command:
[root]# multipath -v3 -d
Using Your Multipath Device
The device to access is now under /dev/mapper. This is the device that is manipulated for creating partitions, adding as a physical volume, and creating filesystems on. For example, to manipulate your new disk's partitions, use parted on the dm-multipath device:
[root]# parted /dev/mapper/36090a02830873505809c640100007047
An optional feature you can also use is alias. You can create a stanza in /etc/multipathd.conf for the scsi_id of your volume, and give it a name of your choice. For example:
multipaths {
multipath {
wwid 36090a02830873505809c640100007047
alias cluster
}
}
After restarting multipathd, your volume will appear with the alias:
[root]# ls /dev/mapper /dev/mapper: cluster clusterp1 control VolGroup00-LogVol00 VolGroup00-LogVol01
For more information on device-mapper-multipath and additional options it supports, see Using Device-Mapper Multipath on the Red Hat website at www.redhat.com/docs/manuals/enterprise/.
Continue to the Appendix
Dell|Red Hat HA Linux > Storage > EqualLogic PS-Series > Cluster Nodes
