RPI destroys usb storage

Question

I use my RaspberryPI 3B+ with the system installed on USB storage sticks (no SD card at all). Raspbian is installed in server mode, with Apache, MySQL, FTP, ... I use it mainly as web server with an RSS feed reader (similar to Tiny Tiny RSS) and as webcast player.

One month ago, the USB stick I was using from about one year suddenly stopped working. Apache was still running but replied with errors, and the HDMI output showed nothing (no prompt).

I tested the USB stick in a computer, and the device appeared as an empty storage container (no usable space, Windows sees it as "Removable" with "No media"). I concluded the USB stick was dead, at started a fresh install on another USB key (Intenso USB2 8GB).

But now, one month later, the new stik is also dead, with the same symptoms.

fdisk -l does not show the disk. Same for df. lsusb sees it though.

How can I try and find out more about the cause?

---

Addition of Goldilocks' suggested commands' results:

gwen@gwen-VirtualBox:~/Bureau$ sudo dd if=~/MBR.img of=/dev/sdb
dd: impossible d'ouvrir '/dev/sdb': Aucun support trouvé

gwen@gwen-VirtualBox:~/Bureau$ tail -n 50 /var/log/syslog
...
May 15 18:49:07 gwen-VirtualBox kernel: [63155.293199] usb 1-2: new full-speed USB device number 20 using ohci-pci
May 15 18:49:08 gwen-VirtualBox kernel: [63156.436565] usb 1-2: config 1 interface 0 altsetting 0 endpoint 0x81 has invalid maxpacket 512, setting to 64
May 15 18:49:08 gwen-VirtualBox kernel: [63156.436568] usb 1-2: config 1 interface 0 altsetting 0 endpoint 0x2 has invalid maxpacket 512, setting to 64
May 15 18:49:08 gwen-VirtualBox kernel: [63156.454264] usb 1-2: New USB device found, idVendor=090c, idProduct=3000, bcdDevice= 1.00
May 15 18:49:08 gwen-VirtualBox kernel: [63156.454267] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
May 15 18:49:08 gwen-VirtualBox kernel: [63156.454268] usb 1-2: Product: SM3255AA MEMORY BAR 
May 15 18:49:08 gwen-VirtualBox kernel: [63156.454269] usb 1-2: Manufacturer: Silicon Motion,Inc. 
May 15 18:49:08 gwen-VirtualBox kernel: [63156.459052] usb-storage 1-2:1.0: USB Mass Storage device detected
May 15 18:49:08 gwen-VirtualBox kernel: [63156.459554] scsi host3: usb-storage 1-2:1.0
May 15 18:49:09 gwen-VirtualBox mtp-probe: checking bus 1, device 20: "/sys/devices/pci0000:00/0000:00:06.0/usb1/1-2"
May 15 18:49:09 gwen-VirtualBox mtp-probe: bus: 1, device: 20 was not an MTP device
May 15 18:49:09 gwen-VirtualBox upowerd[1133]: unhandled action 'bind' on /sys/devices/pci0000:00/0000:00:06.0/usb1/1-2/1-2:1.0
May 15 18:49:09 gwen-VirtualBox upowerd[1133]: unhandled action 'bind' on /sys/devices/pci0000:00/0000:00:06.0/usb1/1-2
May 15 18:49:09 gwen-VirtualBox kernel: [63157.496855] scsi 3:0:0:0: Direct-Access              USB MEMORY BAR   1000 PQ: 0 ANSI: 0 CCS
May 15 18:49:09 gwen-VirtualBox kernel: [63157.500254] sd 3:0:0:0: Attached scsi generic sg2 type 0
May 15 18:49:09 gwen-VirtualBox kernel: [63157.550989] sd 3:0:0:0: [sdb] Attached SCSI removable disk
May 15 18:51:10 gwen-VirtualBox kernel: [63277.664604] usb 1-2: reset full-speed USB device number 20 using ohci-pci

gwen@gwen-VirtualBox:~/Bureau$ dmesg | tail -n 50
...
[63155.293199] usb 1-2: new full-speed USB device number 20 using ohci-pci
[63156.436565] usb 1-2: config 1 interface 0 altsetting 0 endpoint 0x81 has invalid maxpacket 512, setting to 64
[63156.436568] usb 1-2: config 1 interface 0 altsetting 0 endpoint 0x2 has invalid maxpacket 512, setting to 64
[63156.454264] usb 1-2: New USB device found, idVendor=090c, idProduct=3000, bcdDevice= 1.00
[63156.454267] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[63156.454268] usb 1-2: Product: SM3255AA MEMORY BAR 
[63156.454269] usb 1-2: Manufacturer: Silicon Motion,Inc. 
[63156.459052] usb-storage 1-2:1.0: USB Mass Storage device detected
[63156.459554] scsi host3: usb-storage 1-2:1.0
[63157.496855] scsi 3:0:0:0: Direct-Access              USB MEMORY BAR   1000 PQ: 0 ANSI: 0 CCS
[63157.500254] sd 3:0:0:0: Attached scsi generic sg2 type 0
[63157.550989] sd 3:0:0:0: [sdb] Attached SCSI removable disk
[63277.664604] usb 1-2: reset full-speed USB device number 20 using ohci-pci

Answer 1

fdisk -l does not show the disk. Same for df. lsusb sees it though.

How can I try and find out more about the cause?

This implies the immediate cause was corruption which affected the partition table; that "Apache was still running but replied with errors" fits with this, since the information in the partition table is only needed when mounting a partition. You can delete the entire table and the system should be mostly fine until a reboot.

As to what would have caused the corruption, power could be an issue. If you recover the filesystems, try sudo grep "Under-voltage" /var/log/syslog and see if that clue is there.

Of course if the partition table was corrupted, likely other data was as well. I've never run a Pi directly from a USB device,¹ but I assume it is DOS MBR formatted like the primary SD card would be, and not GPT (which there is not much purpose to that in this context). The MBR, which contains the partition table, is only a 512 byte block.

The thing to do in that case is to recreate the partition table with fdisk. This is one of two fundamental steps in formatting a block storage device, the other being creating the filesystems. The latter is what effectively "wipes" the storage, because it replaces the information needed to find specific pieces of data -- although the data will still be there, sorting it out may be difficult or impossible and is not a worthwhile pursuit in this context.

That means if you rewrite the partition table exactly as it was before, you will be able to access whatever is left of the filesystems. Which could be quite a bit, since the only thing you know for sure is damaged (this all assumes the problem is not a physical defect or damage to the stick) is the 512 byte partition table.

The easiest way to do this is to simple keep a copy of the MBR. From a running Pi, and referring to the SD card:

dd if=/dev/mmcblk0 of=mbr.copy bs=512 count=1

You would use /dev/sda or whatever it is (note, not sda[N], where N is a number; that refers to partitions and not the block device itself. This leaves you with a 512 byte file you can stash somewhere(s) easily.

To replace it:

dd if=mbr.copy of=/dev/sda 

Here I'm using sda to make it clear this probably isn't in a running system since at that point the storage has been borked. It is possible though, eg., if you had been able to log in in the "apache was still running scenario".

You probably won't be allowed to do that if a partition on the device is currently mounted (it which case you should umount said parition(s)).

You could also do this from saved fdisk -l output but that is a bit tedious. Since you don't have either here, you could consult the original image used to create the card (you can run fdisk -l on a pi image file). Most likely, that's never been changed except to expand the root fs. If what you did was just fill the device, this is pretty simple. If you arbitrary divided into multiple partition, though, this may be tricky or impossible.

---

1. You might want to reconsider that strategy unless you have a good reason for doing so (an by that I do not mean, "it's got a faster read/write time", but that you actually need such in order to accomplish some goal). Obviously enough, it is not a safeguard against block storage corruption.