2

I am running Raspbian on my RPi2 and on top of that a Node.js server.

This morning the Node.js server went down, but I still have access to the SSH terminal.

I ran top but couldn't see any processes using huge amount of resources. Instead, I noticed that for any command I am running, I have to wait quite a lot (e.g. top, free, vim, tmux etc).

I don't think that writing/reading from the disk is the issue because that is actually fast (touch foo > bar just works fine, and directory commands like ls are fast).

What could be the issue?

Relevant output
[11730.424403] [fae1fb59] CMD  11 200
[11730.424414] [fae1fb59] REQ> 81f6e2a8 0
[11730.424425] [fb7e2e2c] TIM< 0 0
[11730.424445] mmc0: cmd op 17 arg 0xa2a096 flags 0xb5 - resp 00000900 00000000 00000000 00000000, err 0
[11730.424457] mmc0: data blocks 1 blksz 200 - err 0
[11730.424466] mmc0: =========== REGISTER DUMP ===========
[11730.424476] mmc0: SDCMD  0x00004051
[11730.424485] mmc0: SDARG  0x00a2a096
[11730.424495] mmc0: SDTOUT 0x017d7840
[11730.424503] mmc0: SDCDIV 0x00000003
[11730.424512] mmc0: SDRSP0 0x00000900
[11730.424522] mmc0: SDRSP1 0x00001133
[11730.424531] mmc0: SDRSP2 0x7fffffff
[11730.424540] mmc0: SDRSP3 0x08024016
[11730.424549] mmc0: SDHSTS 0x00000080
[11730.424558] mmc0: SDVDD  0x00000001
[11730.424567] mmc0: SDEDM  0x00010801
[11730.424576] mmc0: SDHCFG 0x0000041e
[11730.424586] mmc0: SDHBCT 0x00000200
[11730.424595] mmc0: SDHBLC 0x00000001
[11730.424603] mmc0: ===========================================
[11730.431815] blk_update_request: I/O error, dev mmcblk0, sector 10657942 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[11740.661554] mmc0: timeout waiting for hardware interrupt.
...
[11740.664385] [fb7e5688] CMD  11 200
[11740.664396] [fb7e5688] REQ> 81f6e2a8 0
[11740.664407] [fc1a6e3a] TIM< 0 0
[11740.664427] mmc0: cmd op 17 arg 0xa2a097 flags 0xb5 - resp 00000900 00000000 00000000 00000000, err 0
[11740.664440] mmc0: data blocks 1 blksz 200 - err 0
[11740.664449] mmc0: =========== REGISTER DUMP ===========
[11740.664459] mmc0: SDCMD  0x00004051
[11740.664468] mmc0: SDARG  0x00a2a097
[11740.664478] mmc0: SDTOUT 0x017d7840
[11740.664487] mmc0: SDCDIV 0x00000003
[11740.664496] mmc0: SDRSP0 0x00000900
[11740.664505] mmc0: SDRSP1 0x00001133
[11740.664515] mmc0: SDRSP2 0x7fffffff
[11740.664524] mmc0: SDRSP3 0x08024016
[11740.664532] mmc0: SDHSTS 0x00000080
[11740.664541] mmc0: SDVDD  0x00000001
[11740.664551] mmc0: SDEDM  0x00010801
[11740.664560] mmc0: SDHCFG 0x0000041e
[11740.664569] mmc0: SDHBCT 0x00000200
[11740.664578] mmc0: SDHBLC 0x00000001
[11740.664587] mmc0: ===========================================
[11740.669944] blk_update_request: I/O error, dev mmcblk0, sector 10657943 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
16
  • 2
    Despite what you've said, I'd suspect the SD card is "worn" - check dmesg output - I take it you've done a reboot - what's your load average look like?
    – Bravo
    Jul 22 at 7:36
  • @Bravo Yesterday there were many electricity interruptions... Maybe that caused it to break...? I did a reboot, and it is now stuck at the black screen (after the Raspbian Logo desktop). Jul 22 at 7:43
  • oh - did you check the logs before rebooting?
    – Bravo
    Jul 22 at 7:46
  • @Bravo Umm, no, but now it responds to ping and the SSH verbose seems to do something, but it is terribly slow. This RPi2 broke multiple micro SD cards from Kingston – is there I could do to avoid that? Honestly, in my experience with Raspberry Pi, I always had the impression the entire system is quite fragile... :-( Jul 22 at 7:53
  • Did you try restoring from your backup?
    – Milliways
    Jul 22 at 9:50
0

Recommend you check that your CPU heat sink is still well-attached. Sometimes overheating will slow things down although they are still marginally functional. For example, simple command line functions like ping and ssh are the last to remain working.

0

I had the same issue with RPI2, RPI3 and RP zero also. Your SD card died. Initially, your file system corrupted, funfact, you can't detect it. You can use fsck and forcefully check and repair it, that could help at least for saving out everything from it. You can try to format/partition it again and use it (it worked for me 7 times from 10). your SD card has worned out memory blocks, what is ~physical damage, you will have data loss.

What lead to that:

  • Faulty factored SD card
  • Voltage changes (under-over electricity)
  • multiple random turn off while it working (sometime even one power off time is enough, sometime good for hundreds)
  • constant overheat
  • Too many write on the SD card

PM2 and nodejs write a lot (at least on cache level) and uses a bunch of files. Probably you have quite a lots of log file also. I had the same setup ;) and had the same issue. I replaced the first sandisk card, the second one working like charm ~3 years now.

My advice:

  • Minimalize the log usage or generic writing to the SD card
  • Try to use SSD/HDD if its an option
  • Try to minimalize the NODE js dependency blackhole (e.g.: node_modules)
  • Use pro/industrial tier SD card (could be quite expensive)
  • Use stable power (afaik rpi does not have any regulator)
  • Turn off the desktop view if you don't need it to save power
  • Check the SoC/CPU for heat, add a passive heatsink to it (super cheap usually), if you have it, then check the pasta/contact.

What you can do right now:

  • Backup what you can, partition then format the card, install never rpi os, then use it again until it does not tear down
  • Use a different card
  • Use harddrive or ssd
  • Set up some backup policy

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.