1

I'm trying my best to get GFS2 mounted on top of a dual-primary DRDB /dev/drdb0 volume on Raspbian, but I'm hitting an issue. The DRBD volume has a working stonith/fencing cluster setup using pacemaker and works fine on that side. The cluster is successfully synced across both RPi nodes, in a Primary/Primary configuration.

Note: I have gotten this setup to work using ocfs2, but I would really like to get gfs2 working.

Whenever I format /dev/drbd0 using mkfs.gfs2` and then try to mount the GFS2 volume, I get an error:

mount -t gfs2 -o noatime,nodiratime,noquota /dev/drbd0 /data
mount: mount(2) failed: /data: No such file or directory

I've read up about this and there seems scant information about this. I have determined that the linux kernel config file /usr/src/linux-headers-4.14.79+/.config shows this as an unconfigured part of the kernel module: 

CONFIG_GFS2_FS=m
# CONFIG_GFS2_FS_LOCKING_DLM is not set

I think this means it needs to be enabled to CONFIG_GFS2_FS_LOCKING_DLM=y and recompiled.

I am running into a lot of issues with 1) determining the correct module and 2) getting it correctly compiled and installed. I think it is part of fs/gfs2 or fs/dlm.

After setting the .config file, I have been trying to compile them using the commands:

make modules SUBDIRS=fs/dlm
make modules_install SUBDIRS=fs/dlm
make modules SUBDIRS=fs/gfs2
make modules_install SUBDIRS=fs/gfs2

Then:

modprobe dlm
modprobe gfs2

Still I get the same errors when trying to mount the filesystem.

I'm stuck, can anyone help?

Update: Adding output for use of answer by @Ingo (Thanks for responding).

root@raspberrypi4:~# sudo modprobe dlm

root@raspberrypi4:~# sudo modprobe gfs2

root@raspberrypi4:~# lsmod | egrep 'dlm|gfs2'
gfs2                  334544  0
dlm                   153064  8

root@raspberrypi4:~# sudo ls -al /data/
total 8
drwxrwxr-x  2 root root 4096 Jan 22 13:36 .
drwxr-xr-x 23 root root 4096 Jan 22 13:36 ..

root@raspberrypi4:~# mkfs.gfs2 -p lock_dlm -j 2 -t nfs:nfs /dev/drbd0
-su: mkfs.gfs2: command not found

root@raspberrypi4:~# apt-get install gfs2-utils -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  gfs2-utils
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 237 kB of archives.
After this operation, 946 kB of additional disk space will be used.
Get:1 http://mirror.serverius.net/raspbian/raspbian stretch/main armhf gfs2-utils armhf 3.1.9-2 [237 kB]
Fetched 237 kB in 9s (24.5 kB/s)
Selecting previously unselected package gfs2-utils.
(Reading database ... 75111 files and directories currently installed.)
Preparing to unpack .../gfs2-utils_3.1.9-2_armhf.deb ...
Unpacking gfs2-utils (3.1.9-2) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up gfs2-utils (3.1.9-2) ...

root@raspberrypi4:~# mkfs.gfs2 -p lock_dlm -j 2 -t nfs:nfs /dev/drbd0
/dev/drbd0: Read-only file system

root@raspberrypi4:~# cat /proc/drbd
version: 8.4.10 (api:1/proto:86-101)
srcversion: 7EBAF95052116ECFD37C687
 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
    ns:265164 nr:0 dw:265164 dr:3104 al:33 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0

root@raspberrypi4:~# drbdadm primary nfs

root@raspberrypi4:~# cat /proc/drbd
version: 8.4.10 (api:1/proto:86-101)
srcversion: 7EBAF95052116ECFD37C687
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:265164 nr:0 dw:265164 dr:3104 al:33 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0

root@raspberrypi4:~# mkfs.gfs2 -p lock_dlm -j 2 -t nfs:nfs /dev/drbd0
It appears to contain an existing filesystem (ocfs2)
This will destroy any data on /dev/drbd0
Are you sure you want to proceed? [y/n]y
Discarding device contents (may take a while on large devices): Done
Adding journals: Done
Building resource groups: Done
Creating quota file: Done
Writing superblock and syncing: Done
Device:                    /dev/drbd0
Block size:                4096
Device size:               3.73 GB (978377 blocks)
Filesystem size:           3.73 GB (978374 blocks)
Journals:                  2
Resource groups:           16
Locking protocol:          "lock_dlm"
Lock table:                "nfs:nfs"
UUID:                      bcd37508-ad9b-28cd-ee37-bbe020b7ae4e

root@raspberrypi4:~# sudo mount -t gfs2 -o noatime,nodiratime,noquota /dev/drbd0 /data
mount: mount(2) failed: /data: No such file or directory

root@raspberrypi4:~# dmesg | tail -60
[    3.563656] scsi host0: usb-storage 1-1.3:1.0
[    3.610066] random: systemd-cryptse: uninitialized urandom read (16 bytes read)
[    3.740742] uart-pl011 20201000.serial: no DMA platform data
[    4.453683] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[    4.474723] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    4.493617] systemd[1]: Listening on udev Control Socket.
[    4.507598] systemd[1]: Listening on Journal Socket.
[    4.520857] systemd[1]: Listening on Syslog Socket.
[    4.534018] systemd[1]: Listening on udev Kernel Socket.
[    4.547161] systemd[1]: Reached target Swap.
[    4.647540] scsi 0:0:0:0: Direct-Access     Kingston DataTraveler 2.0 PMAP PQ: 0 ANSI: 0 CCS
[    4.717381] sd 0:0:0:0: [sda] 7827456 512-byte logical blocks: (4.01 GB/3.73 GiB)
[    4.717912] sd 0:0:0:0: [sda] Write Protect is off
[    4.717931] sd 0:0:0:0: [sda] Mode Sense: 23 00 00 00
[    4.718439] sd 0:0:0:0: [sda] No Caching mode page found
[    4.718458] sd 0:0:0:0: [sda] Assuming drive cache: write through
[    4.727793]  sda: sda1
[    4.733347] sd 0:0:0:0: [sda] Attached SCSI removable disk
[    7.287521] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[    7.874193] systemd-journald[88]: Received request to flush runtime journal from PID 1
[    7.977196] sd 0:0:0:0: Attached scsi generic sg0 type 0
[   10.981582] snd_bcm2835: module is from the staging directory, the quality is unknown, you have been warned.
[   11.040548] bcm2835_alsa bcm2835_alsa: card created with 8 channels
[   15.313181] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[   16.837859] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
[   19.067119] random: crng init done
[   19.067142] random: 7 urandom warning(s) missed due to ratelimiting
[   24.793040] drbd: initialized. Version: 8.4.10 (api:1/proto:86-101)
[   24.793062] drbd: srcversion: 7EBAF95052116ECFD37C687
[   24.793069] drbd: registered as block device major 147
[   26.243493] drbd nfs: Starting worker thread (from drbdsetup-84 [452])
[   26.261886] block drbd0: disk( Diskless -> Attaching )
[   26.262112] drbd nfs: Method to ensure write ordering: drain
[   26.262313] block drbd0: max BIO size = 122880
[   26.270502] block drbd0: drbd_bm_resize called with capacity == 7827016
[   26.270677] block drbd0: resync bitmap: bits=978377 words=30576 pages=30
[   26.270696] block drbd0: size = 3822 MB (3913508 KB)
[   26.284493] block drbd0: recounting of set bits took additional 1 jiffies
[   26.284524] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
[   26.284578] block drbd0: disk( Attaching -> Consistent )
[   26.284623] block drbd0: attached to UUIDs DF273E4A603D84AE:0000000000000000:05A4CC7381C320C6:05A3CC7381C320C7
[   27.233543] drbd nfs: conn( StandAlone -> Unconnected )
[   27.236822] drbd nfs: Starting receiver thread (from drbd_w_nfs [456])
[   27.237451] DLM installed
[   27.283266] drbd nfs: receiver (re)started
[   27.283370] drbd nfs: conn( Unconnected -> WFConnection )
[   27.836645] drbd nfs: Handshake successful: Agreed network protocol version 101
[   27.836667] drbd nfs: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
[   27.839020] drbd nfs: Peer authenticated using 20 bytes HMAC
[   27.840057] drbd nfs: conn( WFConnection -> WFReportParams )
[   27.840088] drbd nfs: Starting ack_recv thread (from drbd_r_nfs [480])
[   27.893199] block drbd0: drbd_sync_handshake:
[   27.893232] block drbd0: self DF273E4A603D84AE:0000000000000000:05A4CC7381C320C6:05A3CC7381C320C7 bits:0 flags:0
[   27.893251] block drbd0: peer DF273E4A603D84AE:0000000000000000:05A4CC7381C320C7:05A3CC7381C320C7 bits:0 flags:0
[   27.893266] block drbd0: uuid_compare()=0 by rule 40
[   27.893320] block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> Connected ) disk( Consistent -> UpToDate ) pdsk( DUnknown -> UpToDate )
[   34.963572] Adding 102396k swap on /var/swap.  Priority:-2 extents:1 across:102396k SSFS
[40620.787317] gfs2: GFS2 installed
[40907.496910] block drbd0: role( Secondary -> Primary )
[41161.493128] gfs2: can't find protocol lock_dlm
Improve this question
3
  • Turns out, its almost definitely something to do with CONFIG_GFS2_FS_LOCKING_DLM I found a question at stackexchange that solved the same issue by recompiling the kernel: unix.stackexchange.com/questions/439311/… However, they failed to provide any details of how they did this smoothly. I have tried, and its been quite frustrating. If anyone knows how I can move forward with this, I'd really appreciate the help. :) – Kareem Jan 18 '19 at 15:50
  • 1
    Before you go very far, you should check the obvious: that the mount point /data actually exists. – Michael Hampton Jan 18 '19 at 16:10
  • Please see the update in my original question, which shows that I have indeed checked that the /data/ dir exists. Also, I'm becoming more skeptical about the Lock Table: "nfs:nfs" option specified as -t when formatting GFS2 in mkfs.gfs2 -p lock_dlm -j 2 -t nfs:nfs /dev/drbd0. My stonith cluster is created as nfs but I'm trying to find out more about the specific nature of this particular -t flag. – Kareem Jan 23 '19 at 10:28
0

I use default unmodified Raspbian Stretch Lite and just load the modules:

rpi ~$ sudo modprobe dlm
rpi ~$ sudo modprobe gfs2

Both modules are loaded without any problems:

rpi ~$ lsmod | egrep 'dlm|gfs2'
dlm                   180224  0
gfs2                  380928  0

Then you should check if the mountpoint is available:

rpi ~$ sudo ls -al /data/
drwxr-xr-x  2 root root 4096 Jan 18 21:00 .
drwxr-xr-x 22 root root 4096 Jan 18 21:00 ..

Now you should be able to mount:

rpi ~$ sudo mount -t gfs2 -o noatime,nodiratime,noquota /dev/drbd0 /data

I cannot test it because I do not have a gfs2 file system but there is no need to compile modules. It should work out of the box with default Raspbian.

Improve this answer
7
  • Thanks for responding @ingo, I have tried rebuilding the whole stack from scratch. I took great care to double check each step and I've added the output to my original question if you'd like to take a look. I'm kind of stuck as to how I can get around this gfs2: can't find protocol lock_dlm error. – Kareem Jan 23 '19 at 10:18
  • @Kareem I had a look at your update, very informative. It gives me an idea that you are using a Distributed Replicated Block Device (DRBD) on a Highly Available NFS Storage. Interesting but I do not have any experience on this so I'm afraid I cannot help much. Because you can successful make the filesystem on /dev/drbd0 it seems the block device is OK. Because DRBD uses NFS I would look at the NFS configuration. The problem seems to be blocking and NFS in general has options about blocking. – Ingo Jan 23 '19 at 13:06
  • no I think there is a misunderstanding @ingo. Any mention of nfs is purely semantic, as I have been intending this to eventually run NFS on top. The clusters are simply named "nfs". Please be assured that NFS is not running anywhere. This is a dual-primary DRDB volume at /dev/drbd0 which has pacemaker running for stonith/fencing. The pacemaker is working, and the /dev/drbd0 volume is functional, as I have been able to use it fine with ocfs2, but gfs2 continues to be an issue. Does this help? – Kareem Jan 23 '19 at 13:36
  • @Kareem No sorry, doesn't help. – Ingo Jan 23 '19 at 13:48
  • Is there anything further I can try to explain to help add clarification @ingo? I updated the original question with more information about the setup. I'm really just trying to get this gfs2 setup working. It seems to me like I will need to recompile the kernel to set the CONFIG_GFS2_FS_LOCKING_DLM=y option. I'm not sure exactly how to do that yet, but I'm trying to find out. If there is another way, I'd really like to find out. – Kareem Jan 23 '19 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.