0

I recently purchased 3 raspberry pi nodes to create a small storage cluster to test with at my home. I found a couple of procedures on setting this up so it appears folks have successfully done this!

I am running Raspbian GNU/Linux 8.0 (jessie). I'm using ceph-deploy to install the cluster and it appears to install version 10.2.5-7.2+rpi1 of the ceph ARM packages.

When I try to start the ceph-mon service I get the following error from systemd:

Dec 14 19:59:46 ceph-master systemd[1]: Starting Ceph cluster monitor daemon...
Dec 14 19:59:46 ceph-master systemd[1]: Started Ceph cluster monitor daemon.
Dec 14 19:59:47 ceph-master ceph-mon[28237]: *** Caught signal (Segmentation fault) **
Dec 14 19:59:47 ceph-master ceph-mon[28237]: in thread 756a5c30 thread_name:admin_socket
Dec 14 19:59:47 ceph-master systemd[1]: [email protected]: main process exited, code=killed, status=11/SEGV
Dec 14 19:59:47 ceph-master systemd[1]: Unit [email protected] entered failed state.
Dec 14 19:59:47 ceph-master systemd[1]: [email protected] holdoff time over, scheduling restart.
Dec 14 19:59:47 ceph-master systemd[1]: Stopping Ceph cluster monitor daemon...
Dec 14 19:59:47 ceph-master systemd[1]: Starting Ceph cluster monitor daemon...
Dec 14 19:59:47 ceph-master systemd[1]: Started Ceph cluster monitor daemon.
Dec 14 19:59:49 ceph-master ceph-mon[28256]: *** Caught signal (Segmentation fault) **
Dec 14 19:59:49 ceph-master ceph-mon[28256]: in thread 75654c30 thread_name:admin_socket
Dec 14 19:59:49 ceph-master ceph-mon[28256]: ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367)
Dec 14 19:59:49 ceph-master ceph-mon[28256]: 1: (()+0x4b1348) [0x54fae348]
Dec 14 19:59:49 ceph-master ceph-mon[28256]: 2: (__default_sa_restorer()+0) [0x768bb480]
Dec 14 19:59:49 ceph-master ceph-mon[28256]: 3: (AdminSocket::do_accept()+0x28) [0x550ca154]
Dec 14 19:59:49 ceph-master ceph-mon[28256]: 4: (AdminSocket::entry()+0x22c) [0x550cc458]
Dec 14 19:59:49 ceph-master systemd[1]: [email protected]: main process exited, code=killed, status=11/SEGV
Dec 14 19:59:49 ceph-master systemd[1]: Unit [email protected] entered failed state.
Dec 14 19:59:49 ceph-master systemd[1]: [email protected] holdoff time over, scheduling restart.
Dec 14 19:59:49 ceph-master systemd[1]: Stopping Ceph cluster monitor daemon...
Dec 14 19:59:49 ceph-master systemd[1]: Starting Ceph cluster monitor daemon...
Dec 14 19:59:49 ceph-master systemd[1]: Started Ceph cluster monitor daemon.
Dec 14 19:59:50 ceph-master ceph-mon[28271]: *** Caught signal (Segmentation fault) **
Dec 14 19:59:50 ceph-master ceph-mon[28271]: in thread 755fcc30 thread_name:admin_socket
Dec 14 19:59:50 ceph-master systemd[1]: [email protected]: main process exited, code=killed, status=11/SEGV
Dec 14 19:59:50 ceph-master systemd[1]: Unit [email protected] entered failed state.
Dec 14 19:59:50 ceph-master systemd[1]: [email protected] holdoff time over, scheduling restart.
Dec 14 19:59:50 ceph-master systemd[1]: Stopping Ceph cluster monitor daemon...
Dec 14 19:59:50 ceph-master systemd[1]: Starting Ceph cluster monitor daemon...
Dec 14 19:59:50 ceph-master systemd[1]: [email protected] start request repeated too quickly, refusing to start.
Dec 14 19:59:50 ceph-master systemd[1]: Failed to start Ceph cluster monitor daemon.
Dec 14 19:59:50 ceph-master systemd[1]: Unit [email protected] entered failed state.

I'm looking for guidance here as I'm not sure why this doesn't work. I am using the following URLs for my apt repos:

root@ceph-master:~# cat /etc/apt/sources.list
deb http://mirrordirector.raspbian.org/raspbian/ testing main contrib non-free rpi

root@ceph-master:~# cat /etc/apt/sources.list.d/ceph.list
deb https://download.ceph.com/debian-jewel/ jessie main

Has anyone else tried this and had similar problems? Any advice on how to proceed or work around this issue?

3
  • Assuming you also created a mds instance, not really an answer - but a link to a reported issue about a year ago on the "jewel" branch of ceph: tracker.ceph.com/issues/16771 I am currently running into exactly this problem and I am about contemplating on installing a 64bit OS on my PIs to see if that solves the issue. Dec 16, 2017 at 12:21
  • I cross posted this on unix.stackexchange.com/questions/411117/…. They recommend using version 12.x but not sure where to find the ARM packages for v12 of ceph on ARM.
    – slappyjam
    Dec 16, 2017 at 20:18
  • Which 64 bit OS would you use? From scanning through the ceph repos I only see arm64 packages no armhf, so that might be the move.
    – slappyjam
    Dec 16, 2017 at 21:50

1 Answer 1

1

I finally got ceph to work on my Raspberry Pi 3s. The Ceph repos only have ARM packages for arm64 architecture. All the armhf packages are gone. I think the ceph community doesn't care about 32-bit anymore and have been focusing on 64 bit. The trick was to get an arm64 version of Ubuntu installed. The only arm64 packages available at download.ceph.com were for either xenial or trusty. After several failed attempts to install a 64 bit version of ubuntu, I used a 64bit version of debian and just changed the repos to point to:

deb http://ports.ubuntu.com/ubuntu-ports xenial main universe
deb http://ports.ubuntu.com/ubuntu-ports xenial-updates main universe
deb http://ports.ubuntu.com/ubuntu-ports xenial-security main universe

I found the 64-bit rpi debian image here:

https://github.com/bamarni/pi64/releases

I have ceph luminous up and running!

ceph-admin@ceph-node0:~/my-cluster$ sudo ceph -s
  cluster:
    id:     5fa3e5e2-04ea-407e-b6f1-8ef5a3b3b1f2
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum ceph-node0
    mgr: ceph-node0(active)
    osd: 3 osds: 3 up, 3 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 bytes
    usage:   3164 MB used, 168 GB / 171 GB avail
    pgs:     

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.