Waiting for network connectivity instead of just sleep fixed time

Question

I'm new to the whole Raspberry Pi and Bash scripting thing, so bear with me.

I run raspberry autopilot on a remotely controlled vehicle thru a 3G/4G connection. Raspberry Pi is installed onboard the vehicle. It connects to an OpenVPN server, and sends UDP telemetry to a machine on the VPN network.

The 3G/4G coverage is not stable so I need to keep an eye on it. I have crontab script which runs every minute:

#!/bin/bash

wget -q --tries=10 --timeout=20 --spider http://google.com
if [[ $? -eq 0 ]]; then
        dt=$(date '+%d/%m/%Y %H:%M:%S');
        echo "Online $dt" >> /home/pi/navio-drone/keepalive.log
else
        dt=$(date '+%d/%m/%Y %H:%M:%S');
        # Timestamped message to log file
        echo "Offline $dt" >> /home/pi/navio-drone/keepalive.log
        # Disconnect  from OpenVPN
        echo $(date -u) "Stopping OpenVPN connection" >> /home/pi/navio-drone/keepalive.log
        systemctl stop [email protected]
        # Trigger power to USB ports to reset Huawei modem, 1 sec delay
        echo $(date -u) "Resetting USB modem connection" >> /home/pi/navio-drone/keepalive.log
        ./uhubctl -a 2 -l 1-1 -d 1
        # Sleep 45 sec (let modem to find network) and connect to OpenVPN server
        echo $(date -u) "Sleeping 45 seconds" >> /home/pi/navio-drone/keepalive.log
        sleep 45
        echo $(date -u) "Starting OpenVPN connection" >> /home/pi/navio-drone/keepalive.log
        systemctl start [email protected]
fi

How would you tidy up this script and specifically get rid of

`sleep 45`

in it? Sometimes it takes 30 sec, sometimes 50 for the modem to reconnect to network.

EDIT #1

My ip addr output(4G interface connected):

pi@navio:~ $ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether b8:27:eb:b3:46:ba brd ff:ff:ff:ff:ff:ff
6: enxb827ebe613ef: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether b8:27:eb:e6:13:ef brd ff:ff:ff:ff:ff:ff
11: enx00a0c60429a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 00:a0:c6:04:29:a0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.100/24 brd 192.168.1.255 scope global enx00a0c60429a0
       valid_lft forever preferred_lft forever
    inet6 fe80::240e:3007:3e42:4e64/64 scope link
       valid_lft forever preferred_lft forever
12: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none
    inet 192.168.30.17 peer 192.168.30.18/32 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::8604:8c6b:e6f1:e9d7/64 scope link flags 800
       valid_lft forever preferred_lft forever

My ip route output(4G interface connected):

pi@navio:~ $ ip route
0.0.0.0/1 via 192.168.30.18 dev tun0
default via 192.168.1.1 dev enx00a0c60429a0 src 192.168.1.100 metric 211
116.203.32.50 via 192.168.1.1 dev enx00a0c60429a0
128.0.0.0/1 via 192.168.30.18 dev tun0
192.168.1.0/24 dev enx00a0c60429a0 proto kernel scope link src 192.168.1.100 metric 211
192.168.30.18 dev tun0 proto kernel scope link src 192.168.30.17

I would appreciate any suggestions.

Answer 1

It is difficult to give some specific hints because there is no information about the network setup. But usually you can check if an interface is online. There is a tool from systemd that can check it:

rpi ~$ /lib/systemd/systemd-networkd-wait-online --help
systemd-networkd-wait-online [OPTIONS...]

Block until network is configured.

  -h --help                 Show this help
     --version              Print version string
  -q --quiet                Do not show status information
  -i --interface=INTERFACE  Block until at least these interfaces have appeared
     --ignore=INTERFACE     Don't take these interfaces into account
     --timeout=SECS         Maximum time to wait for network connectivity

You can look with ip addr what interfaces available and what interface has to be online after resetting the USB modem connection. I don't know if the tool detects the interface for the 4G connection but it is worth a try. Just replace the sleep 45 statement for example with:

sudo /lib/systemd/systemd-networkd-wait-online --interface=wlan0 --timeout=120 --quiet

That returns immediately when the interface wlan0 gets online or with an error after 120 seconds. It does not stress the CPU when waiting like a busy loop.

Update:
Here are yet another two possibilities. In /sys/class/net/$NETINTERFACE/carrier the kernel shows if the interface has got a carrier. With a bash function you can test it:

#!/bin/bash

function has-carrier {
    MAX_ATTEMPTS=30
    attempt_num=1
    until [ $(cat /sys/class/net/wlan0/carrier) -eq 1 ] || [ $attempt_num -eq $MAX_ATTEMPTS ]; do
        echo attempt $attempt_num: no carrier
        ((attempt_num++))
        sleep 3
    done

    if [ $attempt_num -eq $MAX_ATTEMPTS ]; then
        echo attempt $attempt_num: timeout
        exit 1
    else
        echo attempt $attempt_num: carrier
        exit 0
    fi
}

has-carrier

This function will check the carrier every 3 seconds for 30 attempts. After 90 seconds (3 sec * 30 attempts) it will timeout.

Then there is a daemon ifplugd that may be usable. It mainly checks wired ethernet ports but it also has a feature:

• Configure WLAN devices (on detecting a successful association to an AP)

Check with apt show ifplugd.

Answer 2

Replaced it with:

sleep 10 # Not going to be faster than 10sec anyway
echo $(date) "Waiting for network to recover" 
itest=$(ping -c 1 -W 2 116.203.32.50 | grep "1 packets transmitted, 1 received")
while [ "$itest" == "" ]
do
   sleep 2
   itest=$(ping -c 1 -W 2 116.203.32.50 | grep "1 packets transmitted, 1 received")
done
echo $(date) "Connected to cellular network"