External USB disk disappears

Question

So I have a RPi4B 8GB running as a Plex Media server. Works great, no issues really. I am running Debian GNU/Linux 11 (bullseye).

The Plex pi is running in a Geekworm NAS Pi kit - after market power supply, and power board with on/off button.

But I did run into a weird situation - the external usb drive disappeared. I was copying a file (using SMB) to the drive that holds the media files for the media server - it's a 1TB Samsung 2.5" SSD directly connected to the Pi.

I've done this dozens (hundreds?) of times before with no issue. I've searched and read many other posts asking if it's power related (I have a beefy power supply from the kit) or the drive going to sleep (this drive was active and copying a file when the disconnect happened) but this seems different from other posts I've read...

After about 5 minutes, I noticed the copy failed.

In my windows machine, I browsed to the SMB share folder, and it's empty - I was guessing an SMB hiccup, so I ssh into the Pi to check it out... I went to /media/pi/PLEX and the folder is empty - First thing I do is a lsblk - and the drive isn't there at all! I issue a reboot command as the drive is set in fstab by its' UUID and is mounted to /media/pi/PLEX - Actual fstab entry:

UUID=e1ae678a-2fab-44c1-9c91-514b0d34544a /media/pi/PLEX ext4 defaults,noatime 0 2

And the pi hangs on reboot which has never happened before. I pulled the power, waited 15s and plugged it back in. It boots up just fine and a lsblk shows the sda drive mounted fine.

pi@plexpi:/etc $ lsblk

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

sda           8:0    0 931.5G  0 disk /media/pi/PLEX

mmcblk0     179:0    0 238.3G  0 disk

├─mmcblk0p1 179:1    0   256M  0 part /boot

└─mmcblk0p2 179:2    0   238G  0 part /

So I'm trying to do some root cause analysis on why it disconnected...

Some related details: 2GB swapfile is located on the SSD. This Pi is running Debian 11 bullseye mmc memory card is a 256GB Sandisk ultra

I dove into the syslog and found where it disconnected, and also when I issued the reboot command. Oddly, when I issued the reboot command, the log indicates it unmounted /media/pi/PLEX with no error codes (I'm assuming this is just a disconnect message, and the drive was not under that mount as shown by lsblk not showing the sda drive?)

Here's the log:

Dec 28 09:37:21 plexpi kernel: [121720.848032] usb 2-2: USB disconnect, device number 2
Dec 28 09:37:21 plexpi kernel: [121720.848820] sd 0:0:0:0: [sda] tag#11 uas_zap_pending 0 uas-tag 1 inflight: CMD
Dec 28 09:37:21 plexpi kernel: [121720.848828] sd 0:0:0:0: [sda] tag#11 CDB: opcode=0x28 28 00 4a e9 43 50 00 00 08 00
Dec 28 09:37:21 plexpi kernel: [121720.848852] sd 0:0:0:0: [sda] tag#11 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=DRIVER_OK cmd_age=1s
Dec 28 09:37:21 plexpi kernel: [121720.848861] sd 0:0:0:0: [sda] tag#11 CDB: opcode=0x28 28 00 4a e9 43 50 00 00 08 00
Dec 28 09:37:21 plexpi kernel: [121720.848867] blk_update_request: I/O error, dev sda, sector 1256801104 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Dec 28 09:37:21 plexpi kernel: [121720.851645] blk_update_request: I/O error, dev sda, sector 1228306648 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Dec 28 09:37:21 plexpi kernel: [121720.851675] EXT4-fs warning (device sda): ext4_end_bio:348: I/O error 10 writing to inode 53216357 starting block 153538332)
Dec 28 09:37:21 plexpi kernel: [121720.851688] Buffer I/O error on device sda, logical block 153538331
Dec 28 09:37:21 plexpi kernel: [121720.851753] blk_update_request: I/O error, dev sda, sector 1228306656 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
Dec 28 09:37:21 plexpi kernel: [121720.851777] blk_update_request: I/O error, dev sda, sector 1228307680 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
Dec 28 09:37:21 plexpi kernel: [121720.851793] blk_update_request: I/O error, dev sda, sector 1228308704 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
Dec 28 09:37:21 plexpi kernel: [121720.852203] blk_update_request: I/O error, dev sda, sector 1228309728 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0
Dec 28 09:37:21 plexpi kernel: [121720.852215] EXT4-fs warning (device sda): ext4_end_bio:348: I/O error 10 writing to inode 53216357 starting block 153538843)
Dec 28 09:37:21 plexpi kernel: [121720.852229] EXT4-fs warning (device sda): ext4_end_bio:348: I/O error 10 writing to inode 53216357 starting block 153538844)
Dec 28 09:37:21 plexpi kernel: [121720.852288] Buffer I/O error on device sda, logical block 153538332
Dec 28 09:37:21 plexpi kernel: [121720.852303] Buffer I/O error on device sda, logical block 153538333
Dec 28 09:37:21 plexpi kernel: [121720.852310] Buffer I/O error on device sda, logical block 153538334
Dec 28 09:37:21 plexpi kernel: [121720.852316] Buffer I/O error on device sda, logical block 153538335
Dec 28 09:37:21 plexpi kernel: [121720.852321] Buffer I/O error on device sda, logical block 153538336
Dec 28 09:37:21 plexpi kernel: [121720.852327] Buffer I/O error on device sda, logical block 153538337
Dec 28 09:37:21 plexpi kernel: [121720.852333] Buffer I/O error on device sda, logical block 153538338
Dec 28 09:37:21 plexpi kernel: [121720.852338] Buffer I/O error on device sda, logical block 153538339
Dec 28 09:37:21 plexpi kernel: [121720.852344] Buffer I/O error on device sda, logical block 153538340
Dec 28 09:37:21 plexpi kernel: [121720.853112] blk_update_request: I/O error, dev sda, sector 1228380952 op 0x1:(WRITE) flags 0x0 phys_seg 109 prio class 0
Dec 28 09:37:21 plexpi kernel: [121720.853133] EXT4-fs warning (device sda): ext4_end_bio:348: I/O error 10 writing to inode 53216357 starting block 153547727)
Dec 28 09:37:21 plexpi kernel: [121720.853148] EXT4-fs warning (device sda): ext4_end_bio:348: I/O error 10 writing to inode 53216357 starting block 153547728)
Dec 28 09:37:21 plexpi kernel: [121720.853164] blk_update_request: I/O error, dev sda, sector 1228382520 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
Dec 28 09:37:21 plexpi kernel: [121720.853495] blk_update_request: I/O error, dev sda, sector 1228383544 op 0x1:(WRITE) flags 0x0 phys_seg 117 prio class 0
Dec 28 09:37:21 plexpi kernel: [121720.853510] EXT4-fs warning (device sda): ext4_end_bio:348: I/O error 10 writing to inode 53216357 starting block 153548059)
Dec 28 09:37:21 plexpi kernel: [121720.853518] EXT4-fs warning (device sda): ext4_end_bio:348: I/O error 10 writing to inode 53216357 starting block 153548060)
Dec 28 09:37:21 plexpi kernel: [121720.855107] blk_update_request: I/O error, dev sda, sector 1228650712 op 0x1:(WRITE) flags 0x0 phys_seg 85 prio class 0
Dec 28 09:37:21 plexpi kernel: [121720.855133] EXT4-fs warning (device sda): ext4_end_bio:348: I/O error 10 writing to inode 53216357 starting block 153581424)
Dec 28 09:37:21 plexpi kernel: [121720.855245] EXT4-fs warning (device sda): ext4_end_bio:348: I/O error 10 writing to inode 53216357 starting block 153583371)
Dec 28 09:37:21 plexpi kernel: [121720.856003] EXT4-fs warning (device sda): ext4_end_bio:348: I/O error 10 writing to inode 53216357 starting block 153583388)
Dec 28 09:37:21 plexpi kernel: [121721.158332] Aborting journal on device sda-8.
Dec 28 09:37:21 plexpi kernel: [121721.158374] Buffer I/O error on dev sda, logical block 121667584, lost sync page write
Dec 28 09:37:21 plexpi kernel: [121721.158392] JBD2: Error -5 detected when updating journal superblock for sda-8.
SUDO REBOOT COMMAND ISSUED HERE.

Dec 28 09:37:25 plexpi systemd[1]: Stopped target Local File Systems.
Dec 28 09:37:25 plexpi kernel: [121724.704347] sd 0:0:0:0: [sda] Synchronizing SCSI cache
Dec 28 09:37:25 plexpi systemd[1]: Unmounting /media/pi/PLEX...
Dec 28 09:37:25 plexpi systemd[1185]: media-pi-PLEX.mount: Succeeded.
Dec 28 09:37:25 plexpi systemd[1]: media-pi-PLEX.mount: Succeeded.
Dec 28 09:37:25 plexpi systemd[1]: Unmounted /media/pi/PLEX.
Dec 28 09:37:25 plexpi systemd[1]: systemd-fsck@dev-disk-by\x2duuid-e1ae678a\x2d2fab\x2d44c1\x2d9c91\x2d514b0d34544a.service: Succeeded.
Dec 28 09:37:25 plexpi systemd[1]: Stopped File System Check on /dev/disk/by-uuid/e1ae678a-2fab-44c1-9c91-514b0d34544a.
Dec 28 09:37:25 plexpi systemd[1]: systemd-fsck@dev-disk-by\x2duuid-e1ae678a\x2d2fab\x2d44c1\x2d9c91\x2d514b0d34544a.service: Consumed 1.433s CPU time.
Dec 28 09:37:25 plexpi kernel: [121724.891866] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x07 driverbyte=DRIVER_OK

Answer 1

The most obvious conclusion is, that there are IO-errors on the SSD. SSD's have a rather brutal failure mode, so an immediate back-up is the first cause of action (if you haven't done so already).

In general, the file systems will be marked read-only. You can check this with mount.

The fact that the reboot hangs may be caused by the swap-file residing on the failing drive.

And then the hardware problem itself. If you google, you find that most replies are about not enough power. The reason is, that this is the ost common mistake. But you could also replace the cable to your drive. Older USB cables have caused these kind of problems. And of course, the drive may be at it's end of life.

--- added ---

The 870EVO has a number of bad batches that fail early. I don't have a complete list, but, for example, drives from January/February 2021 seem to have a relatively high failure rate.

That being said, the only way to be sure that the drive is OK is to use smartctl (part of smartmontools). If you see high failure rates there, then the drive is failing.