1

I'm building an IOT device on a slightly modified raspbian (kernel preempt is enabled) on a pi 4B 2GB without a GUI (runlevel N 5) and relying on wpa_supplicant to make and maintain a wifi connection. I can configure everything fine and it works usually. Here is my wpa_supplicant.conf file:

p2p_disabled=1
country=us
update_config=1
ctrl_interface=/var/run/wpa_supplicant

network={
scan_ssid=1
ssid="mynetwork"
psk="mypass"
}

And most of the time, the network comes up perfectly on boot. But every 4 or 5 boots, the connection fails and never comes up. In this case I get

$wpa_cli status
blah blah... 
INTERFACE_DISABLED
...blah blah

When this happens, I can type

sudo systemctl restart wpa_supplicant.service

And the network will start up correctly, but this is not acceptable in my production environment.

  1. What is going on? Is the pi failing to recognize the NIC? Or is it the connection to the network that's failing? Is there a race condition where the NIC isn't ready when the boot process tries to get a connection (but the NIC comes online later)? Where can I look to find more information about what exactly is happening during this failure?
  2. How can I ensure that network connection failure at boot (for any reason) isn't permanent after boot? If I can run "systemctl restart wpa_supplicant.service" to get things going again, then shouldn't I be able to periodically detect connection failure and do something similar? What is the best practice for this sort of thing?
  • What do you mean about "boot"? You have rebooted 5 times or plugged and unplugged the power supply 5 times? – M. Rostami Dec 22 '19 at 10:33
  • Either way. It can be via "reboot", "shutdown" + power on or hard reset. Doesn't seem to matter. The behavior is the same. – fivedogit Dec 22 '19 at 14:44
  • Can you connect to a SSID by raspi-conf command? – M. Rostami Dec 22 '19 at 17:15
  • Connecting to a network is not the problem. Roughly 4 out of 5 times, it connects no problem. It's just that sometimes the NIC is left in a INTERFACE_DISABLED state. I have a workaround where I have a cron job set up to check for INTERFACE_DISABLED every minute and "systemctl restart wpa_supplicant.service" if it's detected as disabled. The network then starts up normally. So the question is, why 1/5 of the time is the NIC booting as INTERFACE_DISABLED? – fivedogit Dec 22 '19 at 21:17
  • Great job. I'm not sure but sometimes the power supply causes this kind of problem. You've said it gonna be disabled each 1/5 boot, therefore it's not Ok to accept that idea. – M. Rostami Dec 22 '19 at 21:33
1

SOLVED!

First, I answered the bold question in part 1: "Where can I look to find more information about what exactly is happening during this failure?"

The answer was in:

pi@raspberrypi:~ $ sudo systemctl status wpa_supplicant.service 
● wpa_supplicant.service - WPA supplicant
   Loaded: loaded (/lib/systemd/system/wpa_supplicant.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2019-12-22 21:36:14 GMT; 53s ago
  Process: 355 ExecStart=/sbin/wpa_supplicant -u -s -c /etc/wpa_supplicant/wpa_supplicant.conf -i wlan0 (code=exited, status=255/EXCEPTION)
 Main PID: 355 (code=exited, status=255/EXCEPTION)

Dec 22 21:36:13 raspberrypi systemd[1]: Starting WPA supplicant...
Dec 22 21:36:14 raspberrypi wpa_supplicant[355]: Successfully initialized wpa_supplicant
Dec 22 21:36:14 raspberrypi systemd[1]: Started WPA supplicant.
Dec 22 21:36:14 raspberrypi wpa_supplicant[355]: ctrl_iface exists and seems to be in use - cannot override it
Dec 22 21:36:14 raspberrypi wpa_supplicant[355]: Delete '/var/run/wpa_supplicant/wlan0' manually if it is not used anymore
Dec 22 21:36:14 raspberrypi wpa_supplicant[355]: Failed to initialize control interface '/var/run/wpa_supplicant'.
                                                 You may have another wpa_supplicant process already running or the file was
                                                 left by an unclean termination of wpa_supplicant in which case you will need
                                                 to manually remove this file before starting wpa_supplicant again.
Dec 22 21:36:14 raspberrypi wpa_supplicant[355]: nl80211: deinit ifname=wlan0 disabled_11b_rates=0
Dec 22 21:36:14 raspberrypi systemd[1]: wpa_supplicant.service: Main process exited, code=exited, status=255/EXCEPTION
Dec 22 21:36:14 raspberrypi systemd[1]: wpa_supplicant.service: Failed with result 'exit-code'.

So I checked out wpa_supplicant.service which was:

pi@raspberrypi:~ $ sudo cat /lib/systemd/system/wpa_supplicant.service 
[Unit]
Description=WPA supplicant
Before=network.target
After=dbus.service
Wants=network.target

[Service]
Type=dbus
BusName=fi.w1.wpa_supplicant1
ExecStart=/sbin/wpa_supplicant -u -s -c /etc/wpa_supplicant/wpa_supplicant.conf -i wlan0

[Install]
WantedBy=multi-user.target
Alias=dbus-fi.w1.wpa_supplicant1.service

This configuration failed very frequently (at least 50%) of boots. A couple of weeks ago, I tried adding

ExecStartPre=-/bin/rm /var/run/wpa_supplicant/wlan0

which helped and lead to my "4 out of 5 times it works" situation. But still left me with those intermittent failures.

It struck me that maybe there is a race condition between the deletion of the stale file and the ExecStart command testing to make sure it was gone. So, just now, I changed ExecStartPre to

ExecStartPre=-/home/pi/deleteWpaSuppWlan0.sh

where /home/pi/deleteWpaSuppWlan0.sh is

#!/bin/bash
/bin/rm /var/run/wpa_supplicant/wlan0
sleep 1

After rebooting 10-15 times with no further problems, this seems like a satisfactory workaround. The 1 second pause gives the file deletion more time to finish deleting before wpa_supplicant tries to actually start up.

Note: The "-" symbol in the commands means that the command doesn't have to exit correctly for the systemd service to be considered started successfully. If /var/run/wpa_supplicant/wlan0 is already gone, we don't want the ExecStartPre command throwing errors. We want it to just move on quietly to wpa_supplicant startup.

| improve this answer | |
  • FWIW, I reinstalled raspbian from scratch and am not having this issue at all with any changes from stock. The ExecStart line was not what I wrote above. Wonder if I borked it somewhere along the way. – fivedogit Jan 31 at 5:08
  • Minor... no need for an external file, this probably would work: ExecStartPre=-/bin/sleep 1 && /bin/rm /var/run/wpa_supplicant/wlan0 – Scott Prive Mar 9 at 19:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.