1

I recently upgraded to debian [/raspbian] buster. Ever since then, cron jobs creating directories using mkdir -p <directory name> will create the directory on the root file system / device instead of the mounted device /backup/. This seems to only apply to manually mounted file systems, not drives listed in /etc/fstab.

I'm absolutely certain that /backup on /dev/sda2 (aka /) is empty. I'm pretty certain that I'm mounting my USB drive correctly (mount /dev/disk/by-label/BACKUP1 /backup)

For completeness I only access the r-pi via SSH. I haven't been able to get back to it to see if a direct terminal can see the mounts created over SSH.

Does anyone have any idea what is going on here?


Steps to Reproduce

As root:

# mkdir /backup
# mount /dev/sdb1 /backup
# mkdir -p /backup/mars/foo
# batch << EOF
# mkdir -p /backup/mars/bar
> ls /backup/mars > /tmp/mount.test.log
> EOF

Wait for the script to end.

Expected

# ls /backup/mars
foo bar
# cat /tmp/mount.test.log
foo bar
# umount /backup
# ls /backup/mars
ls: cannot access '/backup/mars': No such file or directory

Actual Behaviour

# ls /backup/mars
foo
# cat /tmp/mount.test.log
bar
# umount /backup
# ls /backup/mars
bar

Edit

I've compared the between

mount 

and

batch << EOF
mount > /tmp/mount.test.log
EOF

The output from the batch version is missing the following two lines. The are otherwise identical:

/dev/sdb1 on /backup type ext4 (rw,relatime,data=ordered)
tmpfs on /run/sshd type tmpfs (rw,nosuid,nodev,mode=755)

closed as off-topic by Milliways, techraf, joan, Steve Robillard, Jacobm001 Feb 26 '18 at 2:54

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question does not appear to be specific to the Raspberry Pi within the scope defined in the help center." – techraf, joan, Steve Robillard, Jacobm001
If this question can be reworded to fit the rules in the help center, please edit the question.

  • mount overlays existing fs at that node, any existing files/folders in the root fs will be shadowed (hidden). If you ran mkdir -p even once with the device unmounted (say, during debugging), then the folder /backup/mars will continue to exist after unmounting, but it will not have the contents of your of the mounted drive. – crasic Feb 12 '18 at 21:07
  • "I'm absolutely certain that /backup on /dev/sda2 (aka /) is empty." I could detail the 90 minutes I spent thinking up ways to check this was true... but lets just take it as read shall we? – Philip Couling Feb 12 '18 at 21:11
  • The other thing I can think of is that there is a file being held open that prevents unmounting, I notice you do not check the exit code of umount, try to do sync before unmounting for good measure, or quit if umount fails with exit code != 0 to detect the error condition – crasic Feb 12 '18 at 21:22
  • Thanks for the help. This turned out to be namespaces. I didnt know they existed before today. See my answer. – Philip Couling Feb 12 '18 at 21:25
1

I cannot confirm this. With your example I get the expected result. I suggest to get some more information into /tmp/mount.test.log. This can be done with redirecting standard error to standard output with 2>&1 and then redirect standard output to /tmp/mount.test.log. Don't use -p option for mkdir in batch. This will give more information. Also your standard input to batch with leading characters is wrong. To look what's going wrong I would do:

pi ~$ sudo -Es
root ~# umount /backup
root ~# rm -r /backup/
root ~# mkdir /backup
root ~# mount /dev/sdb1 /backup
root ~# rm -r /backup/mars/
root ~# mkdir -p /backup/mars/foo
root ~# batch <<EOF
mount | grep /dev/sdb1 1>/tmp/mount.test.log 2>&1
mkdir /backup/mars/bar 1>>/tmp/mount.test.log 2>&1 #append with 1>>
EOF

root ~# cat /tmp/mount.test.log
root ~# exit
pi ~$

Update:

I'm using Raspbian GNU/Linux 9.3 (stretch) from Raspbian Stretch Lite.

To omit the -p option is only for test purposes. Your test script doesn't need it. Of course it can be added when the error is fixed.

The input to batch as shown in your example is:

# batch << EOF
# mkdir -p /backup/mars/bar
> ls /backup/mars > /tmp/mount.test.log
> EOF

It should look like:

# batch << EOF
mkdir -p /backup/mars/bar
ls /backup/mars > /tmp/mount.test.log
EOF

Without the leading characters # at the second line and > at the third and fourth line. I thought it was a copy and paste error to the question, but if I take it as shown then the second line will just do nothing because it is a comment to /bin/sh. The third line will not work at expected because it starts with a redirect sign. The fourth line never terminates the input to batch.

You write you get an output from

batch << EOF
mount
EOF

that you compare with mount. It seems we are doing different things here. Do you use the same batch as I do? If I do that, I get:

pi ~$ batch << EOF
mount
EOF
warning: commands will be executed using /bin/sh
job 4 at Mon Feb 12 19:06:00 2018
pi ~$

That is as expected. batch executes its commands in its own environment and does not output to the environment it was started.

Update2:

Have just realized that I get the output from batch via local mail. Doesn't know it before. It contains the lines:

tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=94956k,mode=700,uid=1000,gid=1000)
/dev/sdb1 on /backup type ext2 (rw,relatime,block_validity,barrier,user_xattr,acl)
  • the -p option is there for good reason. It can't be removed from the script easily. I'm not sure what you mean by "leading characters". Could you confirm which OS version you're testing against? There's nothing on stderr from either mount or mkdir. See edit. – Philip Couling Feb 12 '18 at 1:42
  • I've updated my answer. Do you tried my suggested test? With this we would have the same starting point. – Ingo Feb 12 '18 at 20:19
  • I'm connected to my raspi with a terminal not with ssh. Could this make the difference? – Ingo Feb 12 '18 at 20:53
  • Yes. I actually got further with this now. The fault lies in the way systemd is starting sshd. It's getting sandboxed in its own namspace. See here unix.stackexchange.com/questions/423574/… The bug is also specific to debian buster, you won't see it in stretch. – Philip Couling Feb 12 '18 at 21:03
  • And yes my edit was unclear... I dropped the > some-file from the batch command. I have postfix installed so the complete output (including stderr) from cron and atd jobs get emailed to me. My mistake on being unclear there. – Philip Couling Feb 12 '18 at 21:09
1

Figured out what's wrong!
Oh so much digging later....

This was caused by a bug in systemd v236. This will affect debian buster, it does not affect debian stretch. I'm now checking debian sid as as uses the later systemd v237

This is the bug bug: https://github.com/systemd/systemd/issues/7761

Having found a number of different ways to ask the question from different angles, I eventually got a hit here: https://unix.stackexchange.com/questions/423574/what-can-cause-different-processes-to-see-different-mount-points

Cause

Systemd is starting sshd (openss server) in its own namespace. This means that any mounts made in an ssh session are only visible to itself and other ssh sessions. Likewise mounts made on the system after sshd starts are not visible to any ssh session.

You can check if this is affecting you. Log into ssh, and on the command line (through ssh) type the following:

sudo ls -lh /proc/1/ns/mnt /proc/self/ns/mnt

You should get this:

lrwxrwxrwx 1 root root 0 Feb 12 15:52 /proc/1/ns/mnt -> mnt:[4026531840]
lrwxrwxrwx 1 root root 0 Feb 12 21:27 /proc/self/ns/mnt -> mnt:[4026532014]

Notice that the numbers at the end 4026531840 and 4026532014 DONT match. If they don't match then this is likely to be your problem. If they do match then your openssh server is not running in its own namespace like mine is.

Fix

It looks like this has been fixed already in debian pacakge systemd 236-3. Unfortunately this isn't available for raspbian at time of writing. I've downloaded the dpkg libsystemd0_236-3_armhf.deb and systemd_236-3_armhf.deb from the debian repository and manually installed them with:

dpkg --install  systemd_236-3_armhf.deb libsystemd0_236-3_armhf.deb 

Everything is working fine now.

If you would prefer to use only pacakges from raspbian then you will need to downgrade until the updates are available. Download whatever version is available on raspbian stretch for

  • libsystemd0
  • systemd

Then use apt-mark to hold back any updates.

sudo apt-mark hold libsystemd0 systemd

As soon as 236-3 or later becomes available then use apt-mark again to allow updates.

sudo apt-mark unhold libsystemd0 systemd
  • Ouch, good digging. As a work-around you can try to mount with MS_SHARED with mount --make-shared, but this depends on how exactly sshd namespace is set up. – crasic Feb 12 '18 at 21:29

Not the answer you're looking for? Browse other questions tagged or ask your own question.