OCFS2でクラスタしてみる

以下のアドレスの2台のノードでクラスタを構成します。

1.2.3.1
1.2.3.2

FireWallの設定
クラスタに参加するノードはそれぞれ7777をお互いに受け入れるように設定する必要があります。

UFWなら
1.2.3.1

ufw allow from 1.2.3.2 to any port 3260

1.2.3.2

ufw allow from 1.2.3.1 to any port 3260

1台目(1.2.3.1)

$ hostname
iscsi-client-01
$ sudo vi /etc/ocfs2/cluster.conf
node:
	name = iscsi-client-01
	cluster = ocfs2
	number = 0
	ip_address = 1.2.3.1
	ip_port = 7777
node:
	name = iscsi-client-02
	cluster = ocfs2
	number = 1
	ip_address = 1.2.3.2
	ip_port = 7777
cluster:
	name = ocfs2
	node_count = 2
$ sudo service o2cb restart
$ sudo mount -t ocfs2 /dev/sda /mnt/ocfs2/

2台目(1.2.3.2)

$hostname
iscsi-client-02
$sudo vi /etc/ocfs2/cluster.conf
node:
	name = iscsi-client-01
	cluster = ocfs2
	number = 0
	ip_address = 1.2.3.1
	ip_port = 7777
node:
	name = iscsi-client-02
	cluster = ocfs2
	number = 1
	ip_address = 1.2.3.2
	ip_port = 7777
cluster:
	name = ocfs2
	node_count = 2
$ sudo service o2cb restart
$ sudo mount -t ocfs2 /dev/sda /mnt/ocfs2/

トラブルシューティング

2台目のマウントに以下のエラーで失敗した場合

mount.ocfs2: Transport endpoint is not connected while mounting /dev/sda on /mnt/ocfs2/. Check 'dmesg' for more information on this error.

/var/log/syslog エラー1

Aug 16 18:58:09 localhost kernel: [ 4520.624131] o2net: Connect attempt to node iscsi-client-01 (num 0) at 1.2.3.1:7777 failed with errno -22

以下を確認してください

  • ポート7777が開いているか?
  • 各nodeのip_addressは正しく設定されているか?
    各ノードは自分のノードに指定してあるIPアドレスでバインドされるため、127.0.0.1などでなく実際にアクセスされるアドレスを指定します。

/var/log/syslog エラー2

Jun 22 00:14:51 isci-client-02 kernel: [17472.209613] Please double check your configuration values for 'O2CB_HEARTBEAT_THRESHOLD'
Jun 22 00:14:51 isci-client-02 kernel: [17472.345474] (20121,0):o2net_check_handshake:1264 node ubuntu21 (num 0) at 1.2.3.1:7777 uses a heartbeat timeout of 238000 ms, but we use 60000 ms locally.  disconnecting

ログに出ている通りO2CB_HEARTBEAT_THRESHOLDの長さが足りないので設定する

$ sudo /etc/init.d/o2cb offline
$ sudo dpkg-reconfigure ocfs2-tools

O2CB heartbeat thresholdを設定
ocfs2-tools
反映を確認後、起動

$ cat /sys/kernel/config/cluster/ocfs2/heartbeat/dead_threshold
61
$ sudo service o2cb start
$ sudo mount -t ocfs2 /dev/sda /mnt/ocfs2/

OCFS2をUbuntu 12.04にセットアップ

OCFS2はOracle Cluster File Systemの名前の通りOracleのRACなどで使用されているファイルシステムのOpenSourceです。

OCFS2公式サイト

以下は、最小限のセットアップ手順として1ノードで確認するものです。
[bash]
$ sudo apt-get install ocfs2-tools
$ dpkg -l ocfs2-tools
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii ocfs2-tools 1.6.4-1ubunt amd64 tools for managing OCFS2 cluster
$ sudo vi /etc/default/o2cb
O2CB_ENABLED=true
$ hostname
iscsi-client-01
$ sudo vi /etc/ocfs2/cluster.conf
node:
name = iscsi-client-01
cluster = ocfs2
number = 0
ip_address = 127.0.0.1
ip_port = 7777
cluster:
name = ocfs2
node_count = 1
$ sudo service o2cb restart
Loading filesystem "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading stack plugin "o2cb": OK
Loading filesystem "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Setting cluster stack "o2cb": OK
Starting O2CB cluster ocfs2: Failed
Cluster ocfs2 created
o2cb_ctl: Internal logic failure while adding node iSCSI-Client
Stopping O2CB cluster ocfs2: OK
$ sudo mkdir /mnt/ocfs2
$ sudo mkfs.ocfs2 /dev/sda
$ sudo mount -t ocfs2 /dev/sda /mnt/ocfs2/
$ df
Filesystem                 1K-blocks    Used Available Use% Mounted on
/dev/mapper/localhost-root  13123096 1095308  11361164   9% /
udev                          242108       4    242104   1% /dev
tmpfs                         100648     276    100372   1% /run
none                            5120       0      5120   0% /run/lock
none                          251620       0    251620   0% /run/shm
none                          102400       0    102400   0% /run/user
/dev/vda1                     233191   29081    191669  14% /boot
/dev/sda                   104857600 2237704 102619896   3% /mnt/ocfs2
[/bash]

iSCSIイニシエータをUbuntu 12.04にセットアップ

$ sudo apt-get update
$ sudo apt-get install open-iscsi
$ dpkg -l open-iscsi
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  open-iscsi     2.0.873-3ubu amd64        High performance, transport indep
$ sudo iscsiadm -m discovery -t sendtargets -p 1.2.3.4
1.2.3.4:3260,1 iqn.2013-08.me.hrendoh:test.disk
$ sudo iscsiadm -m node --targetname iqn.2013-08.me.hrendoh:test.disk --login
Logging in to [iface: default, target: iqn.2013-08.me.hrendoh:test.disk, portal: 1.2.3.4,3260] (multiple)
Login to [iface: default, target: iqn.2013-08.me.hrendoh:test.disk, portal: 1.2.3.4,3260] successful.
$ sudo iscsiadm -m session
tcp: [1] 1.2.3.4:3260,1 iqn.2013-08.me.hrendoh:test.disk
$ sudo fdisk -l

Disk /dev/vda: 16.1 GB, 16106127360 bytes
16 heads, 63 sectors/track, 31207 cylinders, total 31457280 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e1e29

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048      499711      248832   83  Linux
/dev/vda2          501758    31455231    15476737    5  Extended
/dev/vda5          501760    31455231    15476736   8e  Linux LVM

Disk /dev/mapper/localhost-root: 13.7 GB, 13652459520 bytes
255 heads, 63 sectors/track, 1659 cylinders, total 26664960 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/localhost-root doesn't contain a valid partition table

Disk /dev/mapper/localhost-swap_1: 2143 MB, 2143289344 bytes
255 heads, 63 sectors/track, 260 cylinders, total 4186112 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/localhost-swap_1 doesn't contain a valid partition table

Disk /dev/sda: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sda doesn't contain a valid partition table

iSCSI TargetをUbuntu 12.04にセットアップ

[bash]
$ sudo apt-get update
$ sudo apt-get install iscsitarget
$ dpkg -l iscsitarget
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii iscsitarget 1.4.20.2-10u amd64 iSCSI Enterprise Target userland
$ sudo vi /etc/default/iscsitarget 
ISCSITARGET_ENABLE=true
$ dd if=/dev/zero of=iscsi.img count=0 bs=1 seek=1G
$ sudo vi /etc/iet/ietd.conf
(末尾に追記)
Target iqn.2013-08.me.hrendoh:test.disk
Lun 0 Path=/home/cloudn/iscsi.img,Type=fileio
$ sudo service iscsitarget restart
* Removing iSCSI enterprise target devices: [ OK ]
* Starting iSCSI enterprise target service FATAL: Module iscsi_trgt not found.
[fail]</pre>
起動しようとするとエラー「FATAL: Module iscsi_trgt not found.」
iscsitarget-dkmsをインストールすると解消する(参考:<a href="http://www.heath-bar.com/blog/?p=203">Setup an Ubuntu iSCSI target (server)</a>)
<pre>$ sudo apt-get install iscsitarget-dkms
$ sudo service iscsitarget restart
* Removing iSCSI enterprise target devices: [ OK ]
* Starting iSCSI enterprise target service [ OK ]
[ OK ]
[/bash]
Firewallを設定

1.2.3.xのアクセスをCIDRで指定(Cloudnを利用)

iSCSIターゲット用Firewall設定 Cloudn
iSCSIターゲット用Firewall設定 Cloudn

Ufwなら
[bash]
$ ufw allow from 111.222.333.0/24 to any port 3260
[/bash]

追加ディスクを利用する場合

[bash]
$ sudo fdisk -l

Disk /dev/vda: 16.1 GB, 16106127360 bytes
16 heads, 63 sectors/track, 31207 cylinders, total 31457280 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000e1e29

Device Boot Start End Blocks Id System
/dev/vda1 * 2048 499711 248832 83 Linux
/dev/vda2 501758 31455231 15476737 5 Extended
/dev/vda5 501760 31455231 15476736 8e Linux LVM

Disk /dev/vdb: 107.4 GB, 107374182400 bytes
16 heads, 63 sectors/track, 208050 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/vdb doesn’t contain a valid partition table

Disk /dev/mapper/localhost-root: 13.7 GB, 13652459520 bytes
255 heads, 63 sectors/track, 1659 cylinders, total 26664960 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/localhost-root doesn’t contain a valid partition table

Disk /dev/mapper/localhost-swap_1: 2143 MB, 2143289344 bytes
255 heads, 63 sectors/track, 260 cylinders, total 4186112 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/localhost-swap_1 doesn’t contain a valid partition table
$ sudo vi /etc/iet/ietd.conf
Target iqn.2013-08.me.hrendoh:test.disk
Lun 0 Path=/dev/vdb,Type=blockio
$ sudo service iscsitarget restart
[/bash]