(Solved) RPi4 boot failure w/ dmesg prompt: `Give root password for maintenance`

Question

My raspberry was fine a few days ago. At one point it just vanished from the network. After connecting monitor and keyboard I discovered that during initialization of network it drops do maintenance mode. I have no idea how to debug the reason and fix it if possible.

[   12.189710] Bluetooth: hci0: BCM: chip id 107
[   12.191859] Bluetooth: hci0: BCM: features 0x2f
[   12.195274] Bluetooth: hci0: BCM4345C0
[   12.196845] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0000
[   12.204368] bluetooth hci0: firmware: direct-loading firmware brcm/BCM4345C0.hcd
[   12.207515] Bluetooth: hci0: BCM4345C0 'brcm/BCM4345C0.hcd' Patch
[   12.385720] usb 1-1.4.3: New USB device found, idVendor=1c4f, idProduct=0026, bcdDevice= 1.10
[   12.389341] usb 1-1.4.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   12.392485] usb 1-1.4.3: Product: USB Keyboard
[   12.394393] usb 1-1.4.3: Manufacturer: SIGMACHIP
[   12.800677] hid: raw HID events driver (C) Jiri Kosina
[   12.832879] usbcore: registered new interface driver usbhid
[   12.838030] usbhid: USB HID core driver
[   12.865728] input: SIGMACHIP USB Keyboard as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.3/1-1.4.3:1.0/0003:1C4F:0026.0001/input/input0
[   12.933669] hid-generic 0003:1C4F:0026.0001: input,hidraw0: USB HID v1.10 Keyboard [SIGMACHIP USB Keyboard] on usb-0000:01:00.0-1.4.3/input0
[   12.943008] input: SIGMACHIP USB Keyboard Consumer Control as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.3/1-1.4.3:1.1/0003:1C4F:0026.0002/input/input1
[   13.013181] input: SIGMACHIP USB Keyboard System Control as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.3/1-1.4.3:1.1/0003:1C4F:0026.0002/input/input2
[   13.023945] hid-generic 0003:1C4F:0026.0002: input,hidraw1: USB HID v1.10 Device [SIGMACHIP USB Keyboard] on usb-0000:01:00.0-1.4.3/input1
[   13.047824] Bluetooth: hci0: BCM4345C0 UART 37.4 MHz wlbga_ref_iLNA_iTR_eLG [Baseline: 0143]
[   13.054644] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0230
[   99.234734] audit: type=1400 audit(1550139209.751:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=426 comm="apparmor_parser"
[   99.242881] audit: type=1400 audit(1550139209.751:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=426 comm="apparmor_parser"
[   99.252340] audit: type=1400 audit(1550139209.755:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=429 comm="apparmor_parser"
[   99.260502] audit: type=1400 audit(1550139209.755:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=429 comm="apparmor_parser"
[   99.268394] audit: type=1400 audit(1550139209.755:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=429 comm="apparmor_parser"
[   99.276637] audit: type=1400 audit(1550139209.767:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/ntpd" pid=427 comm="apparmor_parser"
[   99.285048] audit: type=1400 audit(1550139209.775:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/tcpdump" pid=428 comm="apparmor_parser"
[   99.293511] audit: type=1400 audit(1550139209.779:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=431 comm="apparmor_parser"
[   99.301804] audit: type=1400 audit(1550139209.779:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=431 comm="apparmor_parser"
[   99.311525] audit: type=1400 audit(1550139209.803:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.certbot.renew" pid=435 comm="apparmor_parser"
[   99.500394] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[   99.505771] bcmgenet fd580000.ethernet eth0: Link is Down
Give root password for maintenance
(or press Control+d to continue) [  103.589168] bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[  103.595473] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

And this is it. I can input root password and then ethernet works fine but all services are dead, runlevel is "unknown". I would hate to reinstall system and reconfigure all services, but I simply do not know where to start to debug.

EDIT - added longer dmesg output, last two lines after root password prompt do not appear when ethernet cable is disconnected.

SOLUTION for me:

When Control-D is pressed result is:

sulogin: cannot read /dev/tty1: Operation not permitted 

and this pointed to right answer! It was an external HDD I used to host OwnCloud files. It failed and it was not mounted properly. Solution was to login into maintenance mode, comment out entry in /etc/fstab and reboot - problem gone. Now I need to work out how to avoid this problem in the future :(

Answer 1

Based on the edited question, the OP has worked out the cause of the error was a failure to properly mount a drive listed in /etc/fstab.

That being the case, the answer is simply to use the nofail option in the fourth field (fs_mntops) of the /etc/fstab entry. REF: man fstab

IMHO, the nofail option should be used on each and every mount listed in /etc/fstab; a possible exception being the SD card mount. Here's a couple of examples:

# For a removable USB thumb drive:
LABEL=SANDISK16GB /mnt/SANDISK16GB_ThumbDrv exfat rw,user,nofail 0 0 

# For a high-capacity USB ext4 drive:
LABEL=PASSPORT2TB /mnt/Passport2TB ext4 rw,user,nofail 0 0

# For a network (NAS) drive mounted as CIFS: 
//SynologyNAS-1/rpi_share /mnt/SynologyNAS/rpi_share cifs rw,nofail,username=pi,password=XXXXXXXX,vers=3.1.1,uid=pi,gid=pi,file_mode=0644 0 0