Andrew's Blog

This post will document the procedure to switch back to production hardware after failover to a replicant on EqualLogic Groups, with XenCenter.

This post is a follow-up to the previous one about failover to a replicant, so assumes you have failed over, and want to fall back to production.

XenCenter

• Stop the Virtual Machine
• Detach the Storage Repository (only detatch, do not destroy or forget). Answer “Yes” to the confirmation dialog, and wait until the detach process is complete (check the Storage Repositories logs).

D.R. EqualLogic Group Manager

• Under Volumes, select the volume you wish to fall back to production.
• Under “Replication Failback” options, select “Failback to primary”
• Click the “Yes” button on the confirmation dialog.
• Provide an administrator account name and password, for the Production Group, and click OK.  Do not click “Perform manual replication”.  This step will replicate the changes over to the production volume, and set the volume online when complete.
• At the bottom of the Group Manager screen, you will see an operation icon flashing, you can monitor this process by clicking the icon.

XenCenter

• Re-attach the Storage Repository, using the production server IP as the “Target Host”, pressing “Discover IQNs” and select the IQN from the list (it will have the volume name at the end of the IQN). Press the “Discover LUNs” button, and Finish, as there will only be one LUN.
• Click “Yes” to the confirmation dialog.
• Start the Virtual Machine.

This post will document the procedure to fail-over to a replicant volume on a second EqualLogic Group, when using XenServer.

The recovery process is in a separate entry

XenCenter

• Stop the Virtual Machine
• Detach the Storage Repository (only detatch, do not destroy or forget). Answer “Yes” to the confirmation dialog, and wait until the detach process is complete (check the Storage Repositories logs).

If this is a planed failover, and all hardware is online and working, you will want to perform a final replication now, to copy changes since the last replication to the D.R. group.  If this is unplanned, and the production hardware is offline, then you will lose any data since the last replication.

D.R. EqualLogic Group Manager (replication target)

• Promote the Replica Volume: Click on the Replication tab, and select the production group under “Replication Partners”, then “Inbound Replicas”, and select the replica volume you wish to restore. Click on “Promote to volume”, this will promote the last replica to a volume.  To use another replica, you need to clone the replica to a new volume, which is outside this procedure.
• Click “Yes” to the confirmation dialog, which will prompt you that inbound replication will pause during the promotion.
• Make sure both the boxes (Set volume online, and Retain target name of the original volume) are ticked, and click Next.
• Set the iSCSI access (by IP address, use asterisks for wildcards), and access type (Set read-write, and Allow simultaneous connections is ticked), and click Next.
• Click Finish.

XenCenter

• Re-attach the Storage Repository using the D.R. Group address as the target host, “Discover IQNs” selecting the correct IQN from the list, then “Discover LUNs”, there should only be one.
• Click Finish.
• Click “Yes” to the confirmation dialog.
• Start the Virtual Machine.

Production EqualLogic Group Manager (when possible, i.e., when it is back online)

• Select the volume you have promoted to D.R.
• Select under Replication Failback, “Demote to replica set”.
• There will be a confirmation dialog, making sure you have completed the promotion steps above, select “Yes” to this dialog.
• If you recover from a disaster (i.e., failure of production, and it was an unexpected shut), it will suggest that there are changes to the volume that have not been replicated, we will select “No” here, and lose the changes, as we have already started using the replica, and are recovering from an unexpected shut, and know we were going to lose data.
• This process will move the volume to the inbound replicas on the production group.

The following documents the procedure to restore a snapshot of a Xen Virtual Machine stored on an EqualLogic Storage Group.

XenCenter

• Stop the Virtual Machine
• Detach the Storage Repository (only detatch, do not destroy or forget). Answer “Yes” to the confirmation dialog, and wait until the detach process is complete (check the Storage Repositories logs).

EqualLogic Group Manager

• Set the desired volume offline, answering yes to the confirmation.
• Select “Restore volume”, and select the volume you wish to restore, and click “OK”. On the confirmation dialog, make sure “Set volume online after restore is complete”is ticked, and click the “Yes” button. This take a snapshot of the current volume state, and will make the selected snapshot the current volume.

XenCenter

• Re-attach the Storage Repository. You will need to provide the IP Address of the production storage array in the Target Host field, press the “Discover IQNs” button, and select the correct Target IQN from the list (it will end in the volume name), and press “Discover LUNs” button, there should be only 1 LUN (0) in the list, and press the “Finish” button.
• Press the “Yes” button to the confirmation dialog that gets you to check the Storage Repository is not being used by any XenServer.
• XenCenter will plug the store into each production XenServer, and close the window.
• Start the Virtual Machine.

A tool I discovered: apachetop can read apache log files, and show performance data in a human readable format. apachetop will use the default apache log files, and apachetop -f /path/to/logfile will show stats for a selected log file. (continue reading…)

Linux IO is a pretty big topic, and this is just a quick entry with some links and notes about tools I have found, I hope to do a more in depth look later.

First up Network IO:
The command ifstat gives you a pretty decent look at current bandwidth usage. ifstat -l will also show loopback devices

Next Up Disk IO, this is a pretty big one, as this is probably a big cause of system load in my environments.
The command iostat gives a basic run down of the disk usage.
Using the eXtended option iostat -x gives you a more detailed look, and is much more useful in finding issues.
Running with an interval at the end will re-run the stats using live data, which again is more useful, and will refresh the display: iostat -x 1 will refresh the data every 1 second.

1. r/s and w/s are the requests by a process to read and write to the device
2. rsec/s and wsec/s sectors per second.
3. avgrq-sz is the average sectors per request (both read an write). Divide by 2 to get bytes.
4. avgqu-sz is the average queue length
5. await is the average wait time for a request, includes the queue time and the service time
6. svctm is the average time spent servicing requests, the actual read/write time
7. %util is the utilisation percentage, not quite accurate on shared storage/multiple disk raid

To calculate the queue time, take the svctm from await.
If the queue time is greater then the service time, then the device is probably overloaded. Ideally the await should be pretty much equal to the svctm figure, as then there are not many requests being queued.

Some good references:
Quick Reference about iostat
http://www.pythian.com/news/247/basic-io-monitoring-on-linux/
and a detailed blog entry on iostat, and linux disk IO in general…
http://blog.serverfault.com/post/777852755/interpreting-iostat-output

As a final note, for VMWare ESX, have a look at esxtop, and use the d and v and u keys to display various disk related stats, see man esxtop for more, and I hope to cover more later…