Online resize on SAN DS4700

Online resize on SAN DS4700

I tried this on my OCFS2 partition, but it didn't work well (indeed it didn't work at all). I have Xen domU with image file running on the top of OCFS2. It might be because of Xen, so I'll do some test without Xen running on OCFS2.

When working on a SAN, like IBM DS4700, you don't want to add up layer on the operating system running on it. I mean that all the functionalities provided by a tools like LVM2 are already provided by the SAN. Also no partitioning (with fdisk) is made on a Logical Drive, it just add a layer of complexity. The main idea is :

Create a logical drive on the SAN
Map it to your server
Create the filesystem directly on the whole Logical Drive.

I personally use XFS which seems to me as a really good file system. I do not have benchmarking to provide, but I'm using it for over a year now, and it has nice functionalities, like freeze and grow. I'd like to see ZFS on a SAN, but I didn't have the time yet to setup a OpenSolaris.

Now the problem with this is when you want to resize it online. On the SAN side it's easy, right click on the Logical Drive and select “Increase”, or something like that. You can still use the command line tools provided if you want. Now that your drive is bigger, how do we do to make “multipath” and all the underlying system knows about that.

Test procedure

To test online resizing I needed a simple stuff that would keep on writing to my Logical Drive while I was resizing my drive. So I did just this little command line (the case is that my drive is mounted as /mnt/tmp2).

 root@debian:/mnt#  while [ 1 ]; do echo `date` >> /mnt/tmp2/test ; sleep 1 ; done

This command keeps adding the date to a file every second. So I could see if some data get lost during operation.

Before starting

The resizing on the SAN is already done, this is not a problem, so here is the state before operating the resizing (commented):

 
 root@debian:/mnt/tmp2#  date
 Wed May  7 17:29:16 CEST 2008 #  Start date
 root@debian:/mnt/tmp2#  multipath -l mpath4  # Logical drive to be resized
 mpath4 (3600a0b80002684aa00000e49481ffd4b) dm-3 IBM,1814      FAStT
 [size=7.0G][features=1 queue_if_no_path][hwhandler=0]
 \_ round-robin 0 [prio=0][enabled]
  \_ 0:0:0:3 sdd 8:48  [active][undef] 
 \_ round-robin 0 [prio=0][active]
  \_ 1:0:0:3 sdh 8:112 [active][undef] 
 root@debian:/mnt/tmp2#  mount  # I cut off part not about mpath4
 /dev/mapper/mpath4 on /mnt/tmp2 type xfs (rw)
 root@debian:/mnt/tmp2#  df -h  # Same as for mount
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/mpath4    4.0G  260M  3.8G   7% /mnt/tmp2

Let's resize

Ok now we go resize our drive. The idea is to remove is drive from the multipath and add it again. This could be done because we have this multipath stuff. During this operation, if the second path is failing, you will lose your drive, so this operation must be done while your SAN is not experiencing any problem and when both path are fully working.

Remove and add first path

We remove and add our first path :

 
 root@debian:/mnt/tmp2#  echo "scsi remove-single-device 0 0 0 3" > /proc/scsi/scsi  # We remove the first path (0:0:0:3)
 root@debian:/mnt/tmp2#  echo "scsi add-single-device 0 0 0 3" > /proc/scsi/scsi  # And add it again

Sadly I forgot to run multipath -ll mpath4 to show that the system was running on one path!

Reload multipath

This operation must be done. If not, you'll get I/O Error and you'll need to remount your drive :

 
 root@debian:/mnt/tmp2#  multipath  # Reload multipath
 sdd: checker msg is "tur checker reports path is down"
 sdh: checker msg is "tur checker reports path is down"
 8:112: size 14680064, expected 16777216. Discard

We see that multipath check each path (report them as down, don't know why as they're not) and see that the size has changed, but did not take it into account. This is normal, the second disk has not been reloaded. But now we're running on both path again.

 
 root@debian:/mnt/tmp2#  multipath -l mpath4
 mpath4 (3600a0b80002684aa00000e49481ffd4b) dm-3 IBM,1814      FAStT
 [size=7.0G][features=1 queue_if_no_path][hwhandler=0]
 \_ round-robin 0 [prio=0][enabled]
  \_ 1:0:0:3 sdh 8:112 [active][undef] 
 \_ round-robin 0 [prio=0][enabled]
  \_ 0:0:0:3 sdd 8:48  [active][undef]

Remove and add second path

This is done the same way as the first path :

 
 root@debian:/mnt/tmp2#  echo "scsi remove-single-device 1 0 0 3" > /proc/scsi/scsi 
 root@debian:/mnt/tmp2#  echo "scsi add-single-device 1 0 0 3" > /proc/scsi/scsi

Reload multipath, again

We reload multipath and, ooh !

 
 root@debian:/mnt/tmp2#  multipath
 sdd: checker msg is "tur checker reports path is down"
 sdh: checker msg is "tur checker reports path is down"
 reload: mpath4 (3600a0b80002684aa00000e49481ffd4b)  IBM,1814      FAStT
 [size=8.0G][features=1 queue_if_no_path][hwhandler=0]
 \_ round-robin 0 [prio=3][undef]
  \_ 0:0:0:3 sdd 8:48  [active][ready] 
 \_ round-robin 0 [prio=4][undef]
  \_ 1:0:0:3 sdh 8:112 [undef][ready]

A new message came : reload: mpath4 (3600a0b80002684aa00000e49481ffd4b) IBM,1814 FAStT instead of Discard ! And if we look at the size of mpath4, it's now 8.0G !!! Our resizing seems to have worked !!!

After resizing

Now we want to be sure that everything is fine, first we resize our file system :

 
 root@debian:/mnt/tmp2#  cd ..
 root@debian:/mnt#  xfs_growfs tmp2/  # Resize with XFS
 meta-data=/dev/mapper/mpath4     isize=256    agcount=11, agsize=98304 blks
          =                       sectsz=512   attr=0
 data     =                       bsize=4096   blocks=1048576, imaxpct=25
          =                       sunit=0      swidth=0 blks, unwritten=1
 naming   =version 2              bsize=4096  
 log      =internal               bsize=4096   blocks=2560, version=1
          =                       sectsz=512   sunit=0 blks
 realtime =none                   extsz=65536  blocks=0, rtextents=0
 data blocks changed from 1048576 to 2097152

Check if it has work :

 
 root@debian:/mnt#  mount  # See actual mounts (cut off)
 /dev/mapper/mpath4 on /mnt/tmp2 type xfs (rw)
 root@debian:/mnt#  df -h
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/mpath4    8.0G  260M  7.8G   4% /mnt/tmp2
 root@debian:/mnt#  multipath -l mpath4
 mpath4 (3600a0b80002684aa00000e49481ffd4b) dm-3 IBM,1814      FAStT
 [size=8.0G][features=1 queue_if_no_path][hwhandler=0]
 \_ round-robin 0 [prio=0][enabled]
  \_ 0:0:0:3 sdd 8:48  [active][undef] 
 \_ round-robin 0 [prio=0][active]
  \_ 1:0:0:3 sdh 8:112 [active][undef]

And the date we finished :

 
 root@debian:/mnt#  date
 Wed May  7 17:32:27 CEST 2008

So it took 3 minutes to resize our drive, but do we lost something ?

Control

We started a simple little script that write data onto the disk during our resizing, so here is the content of the file generated :

Wed May 7 17:29:14 CEST 2008
Wed May 7 17:29:15 CEST 2008
Wed May 7 17:29:16 CEST 2008
Wed May 7 17:29:17 CEST 2008
Wed May 7 17:29:18 CEST 2008
Wed May 7 17:29:19 CEST 2008
Wed May 7 17:29:20 CEST 2008
Wed May 7 17:29:21 CEST 2008
Wed May 7 17:29:22 CEST 2008
Wed May 7 17:29:23 CEST 2008
Wed May 7 17:29:24 CEST 2008
Wed May 7 17:29:25 CEST 2008
Wed May 7 17:29:26 CEST 2008
Wed May 7 17:29:27 CEST 2008
Wed May 7 17:29:28 CEST 2008
Wed May 7 17:29:29 CEST 2008
Wed May 7 17:29:30 CEST 2008
Wed May 7 17:29:31 CEST 2008
Wed May 7 17:29:32 CEST 2008
Wed May 7 17:29:33 CEST 2008
Wed May 7 17:29:34 CEST 2008
Wed May 7 17:29:35 CEST 2008
Wed May 7 17:29:36 CEST 2008
Wed May 7 17:29:37 CEST 2008
Wed May 7 17:29:38 CEST 2008
Wed May 7 17:29:39 CEST 2008
Wed May 7 17:29:40 CEST 2008
Wed May 7 17:29:41 CEST 2008
Wed May 7 17:29:42 CEST 2008
Wed May 7 17:29:43 CEST 2008
Wed May 7 17:29:44 CEST 2008
Wed May 7 17:29:45 CEST 2008
Wed May 7 17:29:46 CEST 2008
Wed May 7 17:29:47 CEST 2008
Wed May 7 17:29:48 CEST 2008
Wed May 7 17:29:49 CEST 2008
Wed May 7 17:29:50 CEST 2008
Wed May 7 17:29:51 CEST 2008
Wed May 7 17:29:52 CEST 2008
Wed May 7 17:29:53 CEST 2008
Wed May 7 17:29:54 CEST 2008
Wed May 7 17:29:55 CEST 2008
Wed May 7 17:29:56 CEST 2008
Wed May 7 17:29:57 CEST 2008
Wed May 7 17:29:58 CEST 2008
Wed May 7 17:29:59 CEST 2008
Wed May 7 17:30:00 CEST 2008
Wed May 7 17:30:01 CEST 2008
Wed May 7 17:30:02 CEST 2008
Wed May 7 17:30:03 CEST 2008
Wed May 7 17:30:04 CEST 2008
Wed May 7 17:30:05 CEST 2008
Wed May 7 17:30:06 CEST 2008
Wed May 7 17:30:07 CEST 2008
Wed May 7 17:30:08 CEST 2008
Wed May 7 17:30:09 CEST 2008
Wed May 7 17:30:10 CEST 2008
Wed May 7 17:30:11 CEST 2008
Wed May 7 17:30:12 CEST 2008
Wed May 7 17:30:13 CEST 2008
Wed May 7 17:30:14 CEST 2008
Wed May 7 17:30:15 CEST 2008
Wed May 7 17:30:16 CEST 2008
Wed May 7 17:30:17 CEST 2008
Wed May 7 17:30:18 CEST 2008
Wed May 7 17:30:19 CEST 2008
Wed May 7 17:30:20 CEST 2008
Wed May 7 17:30:21 CEST 2008
Wed May 7 17:30:22 CEST 2008
Wed May 7 17:30:23 CEST 2008
Wed May 7 17:30:24 CEST 2008
Wed May 7 17:30:26 CEST 2008
Wed May 7 17:30:27 CEST 2008
Wed May 7 17:30:28 CEST 2008
Wed May 7 17:30:29 CEST 2008
Wed May 7 17:30:30 CEST 2008
Wed May 7 17:30:31 CEST 2008
Wed May 7 17:30:32 CEST 2008
Wed May 7 17:30:33 CEST 2008
Wed May 7 17:30:34 CEST 2008
Wed May 7 17:30:35 CEST 2008
Wed May 7 17:30:36 CEST 2008
Wed May 7 17:30:37 CEST 2008
Wed May 7 17:30:38 CEST 2008
Wed May 7 17:30:39 CEST 2008
Wed May 7 17:30:40 CEST 2008
Wed May 7 17:30:41 CEST 2008
Wed May 7 17:30:42 CEST 2008
Wed May 7 17:30:43 CEST 2008
Wed May 7 17:30:44 CEST 2008
Wed May 7 17:30:45 CEST 2008
Wed May 7 17:30:46 CEST 2008
Wed May 7 17:30:51 CEST 2008
Wed May 7 17:30:52 CEST 2008
Wed May 7 17:30:53 CEST 2008
Wed May 7 17:30:54 CEST 2008
Wed May 7 17:30:55 CEST 2008
Wed May 7 17:30:56 CEST 2008
Wed May 7 17:30:57 CEST 2008
Wed May 7 17:30:58 CEST 2008
Wed May 7 17:30:59 CEST 2008
Wed May 7 17:31:00 CEST 2008
Wed May 7 17:31:01 CEST 2008
Wed May 7 17:31:02 CEST 2008
Wed May 7 17:31:03 CEST 2008
Wed May 7 17:31:04 CEST 2008
Wed May 7 17:31:05 CEST 2008
Wed May 7 17:31:06 CEST 2008
Wed May 7 17:31:07 CEST 2008
Wed May 7 17:31:08 CEST 2008
Wed May 7 17:31:09 CEST 2008
Wed May 7 17:31:10 CEST 2008
Wed May 7 17:31:11 CEST 2008
Wed May 7 17:31:12 CEST 2008
Wed May 7 17:31:13 CEST 2008
Wed May 7 17:31:14 CEST 2008
Wed May 7 17:31:15 CEST 2008
Wed May 7 17:31:16 CEST 2008
Wed May 7 17:31:17 CEST 2008
Wed May 7 17:31:18 CEST 2008
Wed May 7 17:31:19 CEST 2008
Wed May 7 17:31:20 CEST 2008
Wed May 7 17:31:21 CEST 2008
Wed May 7 17:31:22 CEST 2008
Wed May 7 17:31:24 CEST 2008
Wed May 7 17:31:25 CEST 2008
Wed May 7 17:31:26 CEST 2008
Wed May 7 17:31:27 CEST 2008
Wed May 7 17:31:28 CEST 2008
Wed May 7 17:31:32 CEST 2008
Wed May 7 17:31:33 CEST 2008
Wed May 7 17:31:34 CEST 2008
Wed May 7 17:31:35 CEST 2008
Wed May 7 17:31:36 CEST 2008
Wed May 7 17:31:37 CEST 2008
Wed May 7 17:31:38 CEST 2008
Wed May 7 17:31:39 CEST 2008
Wed May 7 17:31:40 CEST 2008
Wed May 7 17:31:41 CEST 2008
Wed May 7 17:31:42 CEST 2008
Wed May 7 17:31:43 CEST 2008
Wed May 7 17:31:44 CEST 2008
Wed May 7 17:31:45 CEST 2008
Wed May 7 17:31:46 CEST 2008
Wed May 7 17:31:52 CEST 2008
Wed May 7 17:31:55 CEST 2008
Wed May 7 17:31:56 CEST 2008
Wed May 7 17:31:57 CEST 2008
Wed May 7 17:31:58 CEST 2008
Wed May 7 17:31:59 CEST 2008
Wed May 7 17:32:00 CEST 2008
Wed May 7 17:32:02 CEST 2008
Wed May 7 17:32:03 CEST 2008
Wed May 7 17:32:04 CEST 2008
Wed May 7 17:32:05 CEST 2008
Wed May 7 17:32:06 CEST 2008
Wed May 7 17:32:07 CEST 2008
Wed May 7 17:32:08 CEST 2008
Wed May 7 17:32:09 CEST 2008
Wed May 7 17:32:10 CEST 2008
Wed May 7 17:32:11 CEST 2008
Wed May 7 17:32:12 CEST 2008
Wed May 7 17:32:13 CEST 2008
Wed May 7 17:32:14 CEST 2008
Wed May 7 17:32:15 CEST 2008
Wed May 7 17:32:16 CEST 2008
Wed May 7 17:32:17 CEST 2008
Wed May 7 17:32:18 CEST 2008
Wed May 7 17:32:19 CEST 2008
Wed May 7 17:32:20 CEST 2008
Wed May 7 17:32:21 CEST 2008
Wed May 7 17:32:22 CEST 2008
Wed May 7 17:32:23 CEST 2008
Wed May 7 17:32:24 CEST 2008
Wed May 7 17:32:25 CEST 2008
Wed May 7 17:32:26 CEST 2008
Wed May 7 17:32:27 CEST 2008
Wed May 7 17:32:28 CEST 2008
Wed May 7 17:32:29 CEST 2008
Wed May 7 17:32:30 CEST 2008
Wed May 7 17:32:31 CEST 2008

Our start date and end date are inside and there's no missing second in it, so we did not lost any data. Our online resizing was done well.