OCFS2, a SAN filesystem

OCFS2 is a SAN Filesystem. The filesystem is shared between several server but, in case of SAN, not by the network. Only locking data are transfered between severs by the network. The data goes through the Fibre Channel.

On Debian Etch, installation of this filesystem is easy. The filesystem itself is already in the kernel mainline, so there's no need for any additional kernel module installation.

First you'll need userland tools to manage OCFS2 :

$ sudo aptitude install ocfs2-tools

OCFS2 uses several services which needs to be started during init. By default Debian provides those services, but they're disabled. To enable them, just modify /etc/default/o2cb and change the following line :

O2CB__ENABLED = true

You also need a cluster name. This is configured at several point, but also in /etc/default/o2cb. Say I create TCH cluster, I change the following line :

O2CB_BOOTCLUSTER = TCH

If your SAN has multipath, you need to change the heatbeat threshold. If you don't you'll have some troubles when paths are changing, more about that on OCFS2 FAQ :

O2CB_HEARTBEAT_THRESHOLD = 31

The value depends on your system, you need to do some testing to see which value best fit your needs.

Well I did not talk about that, but this is a filesystem designed to be accessed from different computers, so you need to do it on every node (computer) that access this fs. In my case I have two nodes, ocfs2s1 (192.168.1.20) and ocfs2s2 (192.168.1.21).

Now your node needs to know a little bit more about your cluster. So we create the file /etc/ocfs2/cluster.conf, and we describe our cluster in it :

node:
        ip_port = 7777
        ip_address = 192.168.1.20
        number = 0
        name = ocfs2s1
        cluster = TCH

node:
        ip_port = 7777
        ip_address = 192..168.1.21
        number = 1
        name = ocfs2s2
        cluster = TCH

cluster:
        node_count = 2
        name = TCH

Of course this file must be the same on every node.

Now you're ready to start the whole stuff and see what happen : /etc/init.d/o2cb start. For me it works, hope it works for you.

Well you may want to have files on your filesystem ?

I'm not gonna tell you how to bound one volume to several severs on you SAN, you have a manual with it. But me, I have /dev/mapper/mpath2 that is bound to ocfs2s1 and ocfs2s2. So from one of my node (ocfs2s1), I create OCFS2 on it :

$ sudo mkfs.ocfs2 /dev/mapper/mpath2

And if it's done without error (as for me), you can mount it :

$ sudo mkfs.ocfs2 /dev/mapper/mpath2 /mnt/TCH

And you can use it.

To add a node you need to execute this command on every node :

$ sudo o2cb_ctl -C -t node -i -n ocfs2s3 -a ip_port=7777 -a ip_address=192.168.1.22 -a cluster=TCH

Then you need to setup your new node as described above, but you don't need to create cluster.conf. Copy it from one of your node (ocfs2s1) to the new node. The previous command complete correctly the cluster.conf file.

When configuration of the new node is done, you can start it /etc/init.d/o2cb start and mount the shared volume.

Stuff about this filesystem what work or not …

It's not possible actually. But it will be possible in a near future. If you use sources from Oracle, it is possible, but as Debian users we have to wait for the upcoming Lenny release, which will happen on October 5, 2008 http://lists.debian.org/debian-release/2008/08/msg01472.html. Debian Lenny is out. See my document about update.

The online resizing was added into the kernel 2.6.25, see http://oss.oracle.com/osswiki/OCFS2/NewFeaturesList, and Debian Lenny uses this kernel. The on-disk layout doesn't change so if you have created your filesystem with Etch, it'll work with Lenny.

To make you feel more comfortable, I did upgrade my OCFS2 servers to Lenny and it works. The only thing is that Lenny doesn't ship the correct OCFS2 tools (no online resizing), so you need to compile the source from Oracle.

In order to compile the package, you need to have those packages installed (with aptitude) :

comerr-dev
pkg-config
uuid-dev
libncurses5-dev
libreadline-dev
libglib2.0-dev

I needed those packages and I think that should be all we need because my test installation was near a default Debian installation.

It doesn't work ! You'll need to create an init script to mount your volume. This problem is corrected in Lenny ! I wrote an init script : the_tragic_story_of_debian_not_mounting_ocfs2_volume_at_boot_time Lenny does it, see again my document about it.

There's no ACL now, but it's planned !

See http://oss.oracle.com/osswiki/OCFS2/LargeTasksList.

• If you're SAN doesn't support multibus, you should care that all node that access the shared volume access their own volume by the same path. So if you're shared volume is on path A, every node has to use path A or you'll get strange freeze !
• If you need to reboot all your dom0, you may use clusterssh. But you're server my boot at the same speed and you may find all your server trying to mount OCFS2 at the almost same time. So to avoid troubles¹⁾, reboot your server one after one, by adding a sleep before the reboot :
  

  X00:~# sleep 15 && reboot

  With this command you only need to type the number for each server. The rest can be done with clusterssh

• In file /etc/updatedb.conf (if you have it) add ocfs2 in the variable PRUNEFS. If not, updatedb would run extensive search with find and your system may hang. Check this in Lenny

I'm using multipath on IBM DS4700 so I had to change my previous configuration here, I did change

devices {
    device {
        # <...>
        path_grouping_policy    group_by_prio
        # <...>
    }
}

• http://oss.oracle.com/projects/ocfs2, official project page
• http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_faq.html, OCFS2 FAQ
• http://www.mail-archive.com/debian-devel@lists.debian.org/msg246775.html, about mount at startup.
• http://www.rampant-books.com/art_hunter_rac_oracle_o2cb_cluster_service.htm