syslog and kern.log filling with: rc rc0: IR event FIFO is full!

Question

I have a Pi that has recently developed a habit of maxing the SD card with syslog messages (somewhere in the hundreds per second) of the message: "Date/Time user kernel: [ 2257.432445] rc rc0: IR event FIFO is full!"

I found this bug report and it was closed in Feb 2019 as "resolved". I looked at the change log for kernel version "linux/4.9.228-1" and found these 3 lines that seem possibly-related but nothing definitive:

    - USB: serial: ir-usb: add missing endpoint sanity check
    - USB: serial: ir-usb: fix link-speed handling
    - USB: serial: ir-usb: fix IrLAP framing

I also found this bug report and tried adding the following items to /etc/modprobe.d/blacklist-ir_event_fifo_is_full.conf and rebooted:

blacklist ir_lirc_codec
blacklist lirc_dev
blacklist sunxi-cir

I then discovered which mods are active with lsmod | grep "ir", revealing:

Module                  Size  Used by
ir_rc6_decoder         16384  0

So I tried adding that to the blacklist file and rebooted:

blacklist ir_rc6_decoder

And still, this doesn't seem to keep it from loading or stop the logs from filling up. Both /var/log/kern.log and /var/log/syslog fill with these same messages.

I haven't found anything recent where this was happening and have exhausted my searching abilities. I mean, searching for "rc rc0: IR event FIFO is full!" in Google with quotes yields 5 results that all seem to be log files.

I'm looking for any guidance on how to stop these messages from accumulating and filling up the SD card.

Peace, Kevin

Answer 1

First off, you don't need to blacklist any of the sunxi stuff on a Pi, which uses exclusively Broadcom chips.

More to the point, you should check if your blacklist files actually have an effect, by running lsmod | grep ir after a reboot. If the blackisted modules are still there, they are loaded early at boot (specifying rdblacklist=module in the kernel options could help), via a different config file (e.g /etc/modprobe.preload) or by other means. "blackist" means no automatic loading, not that the module is actually forbidden in the kernel.

If editing config files doesn't help, two solutions exist:

1. Physically remove the offending module file from /lib/modules

2. Unload the module with rmmod, manually or via an autorun script

I have also heard that blacklisting evdev helps getting rid of the input device modules which keep getting loaded, but AFAIK unless your Pi is used only as a network server, you will need evdev to handle input devices comfortably. I'm not even sure if non-evdev input handling (/dev/kbd, /dev/psaux and the like) is still supported on Linux, especially with USB devices which were predated by evdev.

If there are no more ir modules loaded, and the messages keep coming, then the source of the messages is elsewhere. I would start by investigating what is rc0, perhaps lsmod | grep rc would give you a clue.

Answer 2

The way I would approach this problem, if it is just one specific/consistent log message causing trouble, would be to use the filter conditions built into rsyslog. Here is an example of an expression-based filter structure I've used from the relevant rsyslog documentation:

if expr then action-part-of-selector-line

Use the expression arg to capture the message giving your poor SD card grief, and set the selector action to '/dev/null' for example. Pop that filter into a configuration file that defines the log where the message is being stored and that should solve your issue.

Answer 3

Raspbian/RpiOS, unlike most linux distros, is able to boot and run to a significant extent without any kernel modules at all available. This is because it was compiled to suit one very specific platform, and everything necessary to basic operation is built-in.¹

To test if blacklisting a module could solve this problem, temporarily move /lib/modules:

sudo mv /lib/modules /lib/modules-tmp-moved

And reboot.

If that solves the problem, you could then do a process of elimination starting with the list of what's normally there.

However, in this case I am not sure it will. It's implicit from your question that you actually are not using an IR device, which begs the questions of why that ir module is loaded at all (it is not normally) and why an error is being produced. A grep of the kernel source reveals where that message is from:

> grep -RP "IR event FIFO is full!" drivers/
media/rc/rc-ir-raw.c:           dev_err(&dev->dev, "IR event FIFO is full!\n");

I'm guessing unfortunately that this is part of the "rc core" though, and that is compiled into the stock Pi kernel. This is something you may or may not be able to blacklist (see comments below starting with Dmitry's), if removing/blacklisting the ir module does not work.

If you are okay using the log filtering approach until you find a better solution (and/or find out why this happens at all), here's a example of what cnrcbr suggest in another answer:

if $programname == 'kernel' then {
   if $msg contains 'IR event FIFO is full!'
   then {
       stop
   }
}

Throw that in /etc/rsyslog.conf before the stuff at the bottom where things are dispatched to various files.

---

1. I seem to believe from prior experience that this includes wifi, but that seems unlikely, so beware if the pi is headless and only accessible that way.