====== OCFS2, a SAN filesystem ======

[[wp>OCFS2|OCFS2]] is a [[wp>Shared_disk_file_system|SAN Filesystem]]. The filesystem is shared between several server but, in case of [[wp>Storage_Area_Network|SAN]], not by the network. Only locking data are transfered between severs by the network. The data goes through the Fibre Channel.

On Debian Etch, installation of this filesystem is easy. The filesystem itself is already in the kernel mainline, so there's no need for any additional kernel module installation. 

<note warning><del>The system might hang (and need a reboot) because of updatedb (in package [[deb>findutils]]). So look at [[ocfs2#to_care|things to care about ocfs2]].
I've also reported this as a bug [[http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=516776]]</del> \\
Debian has corrected this behavior.</note>

===== Install =====

First you'll need userland tools to manage [[wp>OCFS2|OCFS2]] :
  $ sudo aptitude install ocfs2-tools

[[wp>OCFS2|OCFS2]] uses several services which needs to be started during ''init''. By default Debian provides those services, but they're disabled. To enable them, just modify ''/etc/default/o2cb'' and change the following line :
  O2CB__ENABLED = true

You also need a cluster name. This is configured at several point, but also in ''/etc/default/o2cb''. Say I create ''TCH'' cluster, I change the following line :
  O2CB_BOOTCLUSTER = TCH

If your SAN has multipath, you need to change the heatbeat threshold. If you don't you'll have some troubles when paths are changing, more about that on [[http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_faq.html#HEARTBEAT|OCFS2 FAQ]] :
  O2CB_HEARTBEAT_THRESHOLD = 31
The value depends on your system, you need to do some testing to see which value best fit your needs.

Well I did not talk about that, but this is a filesystem designed to be accessed from different computers, so you need to do it on every node (computer) that access this fs. In my case I have two nodes, ''ocfs2s1'' (''192.168.1.20'') and ''ocfs2s2'' (''192.168.1.21'').

Now your node needs to know a little bit more about your cluster. So we create the file ''/etc/ocfs2/cluster.conf'', and we describe our cluster in it :
  node:
          ip_port = 7777
          ip_address = 192.168.1.20
          number = 0
          name = ocfs2s1
          cluster = TCH
  
  node:
          ip_port = 7777
          ip_address = 192..168.1.21
          number = 1
          name = ocfs2s2
          cluster = TCH
  
  cluster:
          node_count = 2
          name = TCH

<note warning>''name'' in cluster.conf must be equal to the server hostname</note>

Of course this file must be the same on every node.

Now you're ready to start the whole stuff and see what happen : ''/etc/init.d/o2cb start''. For me it works, hope it works for you.

===== Use it =====

Well you may want to have files on your filesystem ?

I'm not gonna tell you how to bound one volume to several severs on you SAN, you have a manual with it. But me, I have ''/dev/mapper/mpath2'' that is bound to ''ocfs2s1'' and ''ocfs2s2''. So from one of my node (''ocfs2s1''), I create [[wp>OCFS2|OCFS2]] on it :
  $ sudo mkfs.ocfs2 /dev/mapper/mpath2

And if it's done without error (as for me), you can mount it :
  $ sudo mkfs.ocfs2 /dev/mapper/mpath2 /mnt/TCH

And you can use it.


===== Adding a node =====

To add a node you need to execute this command on every node :
  $ sudo o2cb_ctl -C -t node -i -n ocfs2s3 -a ip_port=7777 -a ip_address=192.168.1.22 -a cluster=TCH

Then you need to setup your new node as described above, but you don't need to create ''cluster.conf''. Copy it from one of your node (''ocfs2s1'') to the new node. The previous command complete correctly the ''cluster.conf'' file.

When configuration of the new node is done, you can start it ''/etc/init.d/o2cb start'' and mount the shared volume.

===== Stuff about OCFS2 =====

Stuff about this filesystem what work or not ...


==== Online resizing ====

<del>It's not possible actually. But it will be possible in a near future. If you use sources from Oracle, it is possible, but as Debian users we have to wait for the upcoming Lenny release, which will happen on October 5, 2008 [[http://lists.debian.org/debian-release/2008/08/msg01472.html]].</del>
Debian Lenny is out. See my [[../maintenance/upgrade_to_lenny|document]] about update.

The online resizing was added into the kernel 2.6.25, see [[http://oss.oracle.com/osswiki/OCFS2/NewFeaturesList]], and Debian Lenny uses this kernel. The on-disk layout doesn't change so if you have created your filesystem with Etch, it'll work with Lenny.

To make you feel more comfortable, I did upgrade my OCFS2 servers to Lenny and it works. The only thing is that Lenny doesn't ship the correct OCFS2 tools (no online resizing), so you need to compile the source from Oracle.

In order to compile the package, you need to have those packages installed (with ''aptitude'') :
  comerr-dev
  pkg-config
  uuid-dev
  libncurses5-dev
  libreadline-dev
  libglib2.0-dev
I needed those packages and I think that should be all we need because my test installation was near a default Debian installation.



==== Mount at boot time ====

<del>It doesn't work ! You'll need to create an init script to mount your volume. This problem is corrected in Lenny !
I wrote an init script : [[debian:xen:install#the_tragic_story_of_debian_not_mounting_ocfs2_volume_at_boot_time]]</del>
Lenny does it, see again my  [[../maintenance/upgrade_to_lenny|document]] about it.

==== No ACL ====

There's no ACL now, but it's planned !


==== Other missing features ====

See [[http://oss.oracle.com/osswiki/OCFS2/LargeTasksList]].





===== To care =====

  * If you're [[wp>Storage_Area_Network|SAN]] doesn't support multibus, you should care that all node that access the shared volume access their own volume by the same path. So if you're shared volume is on path A, every node has to use path A or you'll get strange freeze !
  * If you need to reboot all your dom0, you may use ''clusterssh''. But you're server my boot at the same speed and you may find all your server trying to mount [[wp>OCFS2|OCFS2]] at the almost same time. So to avoid troubles((I've seen mounts fail when run at almost the same time)), reboot your server one after one, by adding a sleep before the reboot : \\ <code>X00:~# sleep 15 && reboot</code> With this command you only need to type the number for each server. The rest can be done with ''clusterssh''
  * In file ''/etc/updatedb.conf'' (if you have it) add ''ocfs2'' in the variable ''PRUNEFS''. If not, updatedb would run extensive search with ''find'' and your system may hang. FIXME Check this in Lenny
 

===== Note about multipath =====

I'm using multipath on [[http://www-03.ibm.com/systems/storage/disk/ds4000/ds4700/index.html|IBM DS4700]] so I had to change my previous configuration [[debian:multipath_ibm_ds4000|here]], I did change 

  devices {
      device {
          # <...>
          path_grouping_policy    group_by_prio
          # <...>
      }
  }


===== References =====

  * [[http://oss.oracle.com/projects/ocfs2]], official project page
  * [[http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_faq.html]], OCFS2 FAQ
  * [[http://www.mail-archive.com/debian-devel@lists.debian.org/msg246775.html]], about mount at startup.
  * [[http://www.rampant-books.com/art_hunter_rac_oracle_o2cb_cluster_service.htm]]