1

After many hours of googling and testing for a solution, I am stuck. I would really appreciate some advice.

Situation - I have (5) Raspberry Pis 3B+ (RP) at various customer locations and I want to know when one goes "offline". I expect the number of RPs to double soon making reliability imperative. Various locations means the RP is using the customer wifi/router and I do not have access to the router.

Crontab runs the following python 3.5 code every minute to write from the RP to a cloud mysql database on a Linux shared server on a commercial hosting service (www.ionos.com). (I understand every minute is overkill, but my current reliability is terrible.)

try:
    r = requests.get(url, params=parms, headers={'Connection':'close'})

    if r.status_code == 200:
        if 'Dataplicity' in r.text:
            my_logger.debug("Not connected-Dataplicity-write_environment_reading")
        else:
            my_logger.debug("write_online_status - success")

except requests.ConnectionError as connect_error:
    my_logger.debug("ConnectionError-" + str(connect_error))

except requests.RequestException as request_exception:
    my_logger.debug("RequestException-" + str(request_exception))

except requests.Timeout as timeout_error:
    my_logger.debug("Timeout-" + str(time_error))

Many times an hour, I get the following error:

2019-07-09 11:05:46-ConnectionError-HTTPConnectionPool(host='www.hwildetest.com', port=80): 
Max retries exceeded with url: 
/include/db_access.php?function_name=Write_online_status&customer_id=1&online_status=online&micro_id=19 
(Caused by NewConnectionError
('<requests.packages.urllib3.connection.HTTPConnection object at 0x75cf00f0>: 
Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))

The RP does not stop working. But ping 8.8.8.8 does not return anything around the time of the above error.

sudo uname -a returns:

Linux RP23 4.19.42-v7+ #1219 SMP Tue May 14 21:20:58 BST 2019 armv7l GNU/Linux

cat /etc/os-release returns:

PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

sudo nano /etc/resolv.conf returns: (Most of the RPs are using 192.168.1.1 without much better success.)

Generated by resolvconf
nameserver 208.67.222.222
nameserver 208.67.220.220

Sometimes dig www.hwildetest.com returns:

; <<>> DiG 9.10.3-P4-Raspbian <<>> www.hwildetest.com
;; global options: +cmd
;; connection timed out; no servers could be reached

iwconfig returns:

Link Quality=48/70  Signal level=-62 dBm

I am out of ideas. I don't know if I have a wifi issue, a DNS issue, a hosting issue, etc. Any advice would help.

Edits added 7/9/19 at 9:24 PM by hwilde

How do you get it to work again? Reboot? Does the internet connection also breaks (no ping response from 8.8.8.8) when you only ping the shared server every minute instead of using the python script? I assume you are using wifi on the RasPi, isn't it? – Ingo 6 hours ago

Q - How do you get it to work again? A - Many times, it fails 1-5 times (approximate) and then just starts working again. Sometimes it fails enough times that it cannot recover and I have to reboot.

Q - Does the internet connection also breaks (no ping response from 8.8.8.8) when you only ping the shared server every minute instead of using the python script? A - It is a rather fluid situation, but I would say that sometimes when I am getting the "name resolution" error, pinging 8.8.8.8 still works. Sometimes not, especially after many errors.

Q - I assume you are using wifi on the RasPi A - I am using the on-board wifi for the RasPi.

Today at 4:16 PM, I got this error:

2019-07-09 16:10:26-ConnectionError-HTTPConnectionPool(host='www.panacea247.net', port=80): Max retries exceeded with url: /include/db_access.php?online_status=online&function_name=Write_online_status&micro_id=1&customer_id=1 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x75c92ff0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))

cat /var/log/syslog for the same time range shows:

Jul  9 16:09:22 RP2 systemd[1]: Starting Clean php session files...
Jul  9 16:09:22 RP2 systemd[1]: Started Clean php session files.
Jul  9 16:09:49 RP2 kernel: [610593.198011] rpi_firmware_get_throttled: 14 callbacks suppressed
Jul  9 16:09:49 RP2 kernel: [610593.198019] Under-voltage detected! (0x00050005)
Jul  9 16:10:01 RP2 CRON[32741]: (pi) CMD (python3 /home/pi/ga_pi/crontab_online_status.py >> /home/pi/ga_pi/crontab_online_status_output.txt 2>&1)
Jul  9 16:10:01 RP2 CRON[32742]: (pi) CMD (sudo python3 /home/pi/ga_pi/crontab_write_temperatures.py >> /home/pi/ga_pi/crontab_write_temperatures_output.txt 2>&1)
Jul  9 16:10:01 RP2 CRON[32745]: (pi) CMD (sudo python3 /home/pi/ga_pi/crontab_temperature_on_off.py >> /home/pi/ga_pi/crontab_temperature_on_off_output.txt 2>&1)
Jul  9 16:10:03 RP2 kernel: [610607.756994] Under-voltage detected! (0x00050005)
Jul  9 16:10:12 RP2 kernel: [610616.076875] rpi_firmware_get_throttled: 14 callbacks suppressed
Jul  9 16:10:12 RP2 kernel: [610616.076882] Voltage normalised (0x00000000)
Jul  9 16:10:28 RP2 kernel: [610632.717012] Under-voltage detected! (0x00050005)
Jul  9 16:10:34 RP2 kernel: [610638.960076] Voltage normalised (0x00000000)
Jul  9 16:10:59 RP2 kernel: [610663.918636] Voltage normalised (0x00000000)
Jul  9 16:11:01 RP2 CRON[557]: (pi) CMD (python3 /home/pi/ga_pi/crontab_online_status.py >> /home/pi/ga_pi/crontab_online_status_output.txt 2>&1)
Jul  9 16:11:01 RP2 CRON[558]: (pi) CMD (sudo python3 /home/pi/ga_pi/crontab_temperature_on_off.py >> /home/pi/ga_pi/crontab_temperature_on_off_output.txt 2>&1)
Jul  9 16:12:01 RP2 CRON[592]: (pi) CMD (python3 /home/pi/ga_pi/crontab_online_status.py >> /home/pi/ga_pi/crontab_online_status_output.txt 2>&1)
Jul  9 16:12:01 RP2 CRON[593]: (pi) CMD (sudo python3 /home/pi/ga_pi/crontab_temperature_on_off.py >> /home/pi/ga_pi/crontab_temperature_on_off_output.txt 2>&1)
Jul  9 16:13:01 RP2 CRON[628]: (pi) CMD (sudo python3 /home/pi/ga_pi/crontab_temperature_on_off.py >> /home/pi/ga_pi/crontab_temperature_on_off_output.txt 2>&1)
Jul  9 16:13:01 RP2 CRON[629]: (pi) CMD (python3 /home/pi/ga_pi/crontab_online_status.py >> /home/pi/ga_pi/crontab_online_status_output.txt 2>&1)
Jul  9 16:14:01 RP2 CRON[658]: (pi) CMD (sudo python3 /home/pi/ga_pi/crontab_temperature_on_off.py >> /home/pi/ga_pi/crontab_temperature_on_off_output.txt 2>&1)
Jul  9 16:14:01 RP2 CRON[657]: (pi) CMD (python3 /home/pi/ga_pi/crontab_online_status.py >> /home/pi/ga_pi/crontab_online_status_output.txt 2>&1)
Jul  9 16:15:01 RP2 CRON[713]: (pi) CMD (sudo python3 /home/pi/ga_pi/crontab_temperature_on_off.py >> /home/pi/ga_pi/crontab_temperature_on_off_output.txt 2>&1)
Jul  9 16:15:01 RP2 CRON[714]: (pi) CMD (python3 /home/pi/ga_pi/crontab_online_status.py >> /home/pi/ga_pi/crontab_online_status_output.txt 2>&1)
Jul  9 16:15:01 RP2 CRON[718]: (pi) CMD (sudo python3 /home/pi/ga_pi/crontab_write_temperatures.py >> /home/pi/ga_pi/crontab_write_temperatures_output.txt 2>&1)
Jul  9 16:15:03 RP2 kernel: [610907.278589] rpi_firmware_get_throttled: 13 callbacks suppressed
Jul  9 16:15:03 RP2 kernel: [610907.278601] Under-voltage detected! (0x00050005)
Jul  9 16:15:15 RP2 kernel: [610919.758502] Under-voltage detected! (0x00050005)
Jul  9 16:15:26 RP2 kernel: [610930.158535] rpi_firmware_get_throttled: 13 callbacks suppressed
Jul  9 16:15:26 RP2 kernel: [610930.158542] Voltage normalised (0x00000000)
Jul  9 16:15:42 RP2 kernel: [610946.798690] Under-voltage detected! (0x00050005)
Jul  9 16:15:46 RP2 kernel: [610950.958702] Voltage normalised (0x00000000)
Jul  9 16:15:53 RP2 kernel: [610957.199245] Voltage normalised (0x00000000)

I am certainly using the recommended RasPi 2500 mA power supply plugged into a 20 A 110 VAC circuit and I am only powering 3 relay modules off the RasPi, but the "Under-voltage detected" is concerning.

Any thoughts?

Improve this question
5
  • 1
    How do you get it to work again? Reboot? Does the internet connection also breaks (no ping response from 8.8.8.8) when you only ping the shared server every minute instead of using the python script? I assume you are using wifi on the RasPi, isn't it?
    – Ingo
    Commented Jul 9, 2019 at 18:21
  • Are the RPs having the issue hardwired or wireless? There has been an issue, noted in the Raspberry Pi Foundation forums, where the WiFi carrier drops at regular intervals (~every 5-10 minutes), so if WiFi connected, this could be your problem. Is there anything useful in the system log?
    – bls
    Commented Jul 9, 2019 at 18:21
  • @bls Where is the note in the Raspberry Pi forum? Do you have a link?
    – Ingo
    Commented Jul 9, 2019 at 18:22
  • @Ingo There are two...raspberrypi.org/forums/viewtopic.php?f=36&t=233847 and raspberrypi.org/forums/…. Although there are hints (from me...sigh) that there is a resolution, the suggested solution sometimes failed for me. I will note that with Raspbian Buster I'm seeing it a lot less, but once it gets into that state, it can continue for quite some time. I now think it's a protocol disconnect of some sort between certain WiFi routers and the Broadcom drivers. My eero router updated recently and I'm seeing it a LOT less now.
    – bls
    Commented Jul 9, 2019 at 18:41
  • Not sure how to post code to the comments so I edited the original post with my answers and additional code.
    – hwilde
    Commented Jul 10, 2019 at 1:33

2 Answers 2

Reset to default
2

I don't believe your problem has something to do with your python script. I looks that you have a general networking issue. The ip connection to the internet isn't stable. All other errors, ping 8.8.8.8, name resolution, http connection, are only side effects of the bad ip connection.

Stop any running programs that try to connect to the internet like your python script and check with a fast and endless /bin/ping -i0.3 8.8.8.8 if you also see interrupts of the connection. Stop with CtrlC. If so then you should do the same test with another device, like a laptop. If it also has a problem then the source isn't the RasPi. You have to look at your network configuration.

If the other device (laptop?) doesn't have a problem then you should verify that the Raspberry Pi isn't the source of the error. You can flash a new image Raspbian Buster Lite and just setup Wireless connectivity. It is known that this configuration works unless you have a hardware problem with the RasPi. If it works then the source is your software setup. If it doesn't work you may consider to purchase a new RasPi.

Improve this answer
0

Thanks everyone for the ideas and suggestions. I appreciate the help.

What seems to work is to prevent the RasPi from failing frequently and basically losing the connection.

What I tested and will put around each python script that needs a connection is:

import subprocess as sp
ip = "8.8.8.8"
status,result = sp.getstatusoutput("ping -c1 -w2 " + ip)

# Check for internet access.  If none, skip the rest of the program.
if status == 0:

The above code prevents the rest of the python requests from executing and therefore failing. While not a perfect solution, it does give me a higher degree of stability.

Thanks again.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.