3

I'm running ubuntu 20.10 arm64 on Raspberry Pi 4b. I was booting from a western digital NVMe drive in a USB-to-NVMe adapter/enclosure thing attached to the USB3 port. This has been working fine for over a month.

the nvme device: https://www.amazon.com/gp/product/B07YFF3JCN the USB adapter/enclosure: https://www.amazon.com/gp/product/B07TJT6W8K

My system crashed, when I rebooted the system hung at the boot screen.

I am able to boot from a USB thumbdrive (sandisk ultra USB3).

When I plug in the NVMe drive to the other USB3 port, the system goes off the rails (no panic, but all filesystems are unmounted and I effectively can't do anything).

Ran dmesg -w, then plugged in the NVMe drive:

[   93.113334] usb 1-1.3: new high-speed USB device number 3 using xhci_hcd
[   93.266004] usb 1-1.3: New USB device found, idVendor=152d, idProduct=0562, bcdDevice= 2.09
[   93.266014] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   93.266020] usb 1-1.3: Product: JMS583
[   93.266024] usb 1-1.3: Manufacturer: JMicron
[   93.266029] usb 1-1.3: SerialNumber: DD564198838A6
[   93.277143] scsi host1: uas
[   93.278607] scsi 1:0:0:0: Direct-Access     WDC WDS5 00G2B0C-00PX     0209 PQ: 0 ANSI: 6
[   93.280578] sd 1:0:0:0: Attached scsi generic sg1 type 0
[   95.029306] usb 2-2: USB disconnect, device number 2
[   95.052873] blk_update_request: I/O error, dev sda, sector 13908464 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[   95.052894] EXT4-fs warning (device sda2): ext4_end_bio:345: I/O error 10 writing to inode 153441 starting block 1738559)
[   95.052905] Buffer I/O error on device sda2, logical block 1672766
[   95.052937] blk_update_request: I/O error, dev sda, sector 12437504 op 0x1:(WRITE) flags 0x4000 phys_seg 6 prio class 0

(a lot of these blk_update_request I/O errors) NOTE: sda is the USB thumb drive I booted from. There are zero errors until I plug in the NVMe drive (which I would assume should show up as sdb)

(in these next log excerpts, I'm going to remove repetitive lines just for brevity)

[   95.059950] EXT4-fs warning (device sda2): ext4_end_bio:345: I/O error 10 writing to inode 129950 starting block 1556736)
[   95.060017] Buffer I/O error on device sda2, logical block 1488896
... (nine similar lines, just different block numbers) ...
[   95.066953] EXT4-fs warning (device sda2): ext4_end_bio:345: I/O error 10 writing to inode 129950 starting block 1558784)
... (seven similar lines, just different block numbers) ...
[   95.114676] Buffer I/O error on dev sda2, logical block 1029, lost async page write
[   95.114738] Buffer I/O error on dev sda2, logical block 1171, lost async page write
[   95.114761] Buffer I/O error on dev sda2, logical block 559678, lost sync page write
[   95.114818] Aborting journal on device sda2-8.
[   95.114835] Buffer I/O error on dev sda2, logical block 1220, lost async page write
... (three similar lines, just different block numbers) ...
[   95.114938] JBD2: Error -5 detected when updating journal superblock for sda2-8.
[   95.114991] Buffer I/O error on dev sda2, logical block 524288, lost async page write
[   95.115009] Buffer I/O error on dev sda2, logical block 524304, lost async page write
[   95.115020] Buffer I/O error on dev sda2, logical block 524305, lost async page write
[   95.117583] EXT4-fs (sda2): I/O error while writing superblock
[   95.117596] EXT4-fs error (device sda2): ext4_journal_check_start:83: Detected aborted journal
[   95.117604] EXT4-fs (sda2): Remounting filesystem read-only
[   95.294708] JBD2: Error while async write back metadata bh 1029.
... (three similar lines, just different block numbers) ...

(a bunch of this write back metadata errors, then suddenly it starts discovering the boot device repeatedly)

[   95.333974] EXT4-fs error (device sda2): __ext4_find_entry:1535: inode #2: comm (spawn): reading directory lblock 0
... (nine similar lines, just different block numbers) ...
[   95.645698] usb 2-2: new SuperSpeed Gen 1 USB device number 3 using xhci_hcd
[   95.666712] usb 2-2: New USB device found, idVendor=0781, idProduct=5581, bcdDevice= 1.00
[   95.666731] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   95.666742] usb 2-2: Product: Ultra
[   95.666752] usb 2-2: Manufacturer: SanDisk
[   95.666762] usb 2-2: SerialNumber: 0401e317f2c01389fd6f52a6cbb73e1d10e310185ddd93805a9c1c03880f49cd8484000000000000000000005065d20a00067d188155810733a91124
[   95.676715] usb-storage 2-2:1.0: USB Mass Storage device detected
[   95.677754] scsi host2: usb-storage 2-2:1.0
[   96.473318] usb 2-2: USB disconnect, device number 3
[   96.745607] usb 2-2: new SuperSpeed Gen 1 USB device number 4 using xhci_hcd
[   96.766701] usb 2-2: New USB device found, idVendor=0781, idProduct=5581, bcdDevice= 1.00
[   96.766719] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   96.766730] usb 2-2: Product: Ultra
[   96.766739] usb 2-2: Manufacturer: SanDisk
[   96.766749] usb 2-2: SerialNumber: 0401e317f2c01389fd6f52a6cbb73e1d10e310185ddd93805a9c1c03880f49cd8484000000000000000000005065d20a00067d188155810733a91124
... (a few more of those) ...
[   99.802292] usb-storage 2-2:1.0: USB Mass Storage device detected
[   99.804663] scsi host2: usb-storage 2-2:1.0
[  100.438381] EXT4-fs error: 126 callbacks suppressed

So it looks like the whole USB subsystem is just continually recycling? And re-detecting the USB thumb drive over and over each time.

My guess is the usb adapter thing is defective.

But if I plug it into my mac, it works 100% fine (whether I use a USBC to USBC cable or the same USBA-to-USBC cable I use with the RaspPi4). I can mount the partitions and read/write to the drive just fine (I mount the ext4 partition with ext4fuse) and I don't notice any errors.

I swapped the ports on the RaspPi4 with the same results (e.g. put the sandisk thumb drive in the other USB3 port to boot).

I also tried plugging the nvme drive into a USB2 port on the RaspPi with the same result.

I also used a different USBA-to-USBC cable with the same result.

I have ordered a new USB-to-NVMe adapter/enclosure thing (different brand) but it won't be here for a couple of days.

Is it possible that it's the Pi that is fried? (but the USB thumb drive works fine!) I don't have any other devices to try with it currently.

Improve this question
3
  • It's very possibly power. That drive probably consumes at least 2-3 watts when busy. Amps = watts/voltage = 500-750+ mA. At boot, the Pi itself is working about as hard as it can too, maybe pulling 2A. That's pushing a threshold, assuming you are using at least a 2.5A supply. You need to power it independently, which it looks like that adapter won't allow.
    – goldilocks ♦
    Commented Mar 1, 2021 at 20:08
  • It is even worse than that the total combined power for all four USB ports is 1.2A. Just because it works well in a machine with proper ports does not mean anything. If I plug my keyboard in, as I was reminded last night when I forgot to unplug it, it does not boot with my Pi 4. To say I am unimpressed is a mild understatement. That is the solution proposed on the Pi forums all the time get a powered hub and make certain it does not back feed the power or you get problems from that. There are posts all the time about problems with nvme drives/adapters, the ssd is supposed to be better at it.
    – user130167
    Commented Mar 1, 2021 at 20:23
  • Yes, using a powered USB hub works fine. I'm wondering why this worked OK for a month before becoming a problem, though? The only powered hub I have is not USB3 and getting one isn't a huge problem but ugh, just that many more cables. Anyway, thanks for the help, I hadn't even though of how much power the drive was consuming (it does get quite warm).
    – mayostard
    Commented Mar 1, 2021 at 22:27

2 Answers 2

Reset to default
4

I have had a very similar problem (I/O errors) with SATA USB adapters that were also manufactured by JMicron. It seems to be a known problem especially for SATA (see here) that some manufacturers are buggy with Raspberry Pis. As you see, the list of NVMe adapters also seems to start growing...

As Dmitry says, you can try disabling UAS by adding "usb quirks" (see here). For me this didn't work with the two JMicron adapters, but it made an initially buggy SATA enclosure work perfectly fine.

However, the more stable option is to try buying one of the brands in the list of most compatible adapters (e.g. StarTech or UGreen) or buy a dedicated NVMe expansion kit or full enclosure (e.g. Argon One with base). You mentioned you already bought a new adapter and might thus have solved your problem, but I am mentioning this because I also tried a random other brand (before finding the list of compatible adapters) and ended up with another unstable JMicron controller inside. These many different Chinese brands aren't very different inside.

Improve this answer
3

When you have problems with devices using UAS driver, the first thing you should try is to disable UAS.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.