Network dynamic failover with bonding on raspberry pi: cannot reach other wifi connected hosts

Question

[EDIT 6-december-2020] In my network I experience the same issue also in a raspberry pi v4, with the Raspberry OS image date December 2nd 2020

** Original post **

I am using Raspbian Buster on a raspberry pi 3b v2 (and a raspberry pi 4b v1.2) and I followed the @Ingo answer in this post. The version of the hardware is

Raspberry Pi 3 Model B Rev 1.2.

The kernel version is

Linux openhab 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux.

The version of the operating system is

PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster

The failover works correctly, as far as the gateway is concerned. In normal conditions, i.e. when both ethernet and WLAN0 interfaces are up, the raspberry can reach the the gateway (i.e receives response to ping). When the ethernet link is down, the raspberry can still reach the gateway through WLAN0.

The raspberry cannot reach other hosts, if they are connected through WiFi to the same access point.

More precisely, in normal conditions, i.e. when both ethernet and WLAN interfaces are up, the raspberry cannot ping these hosts (and it cannot receive packets from these nodes). As soon as only one interface remains up (it can be either ethernet or WLAN0) the communication between the raspberry and the "problematic" hosts is re-established.

This behavior occurs with a fritzbox 7590 (fritzOS 7.13) but also with other acces points as well (technicolor tg789vac or Dlink DVA5992 or technicolor DGA4130).

[EDIT 26-april-2020]

Thanks to hints by @Ingo in the comments, I nailed down the issue to the ARP packet traffic exchange, by using tcpdump on the raspberry. I do not understand if this is an issue with the router or with the bonding module. To simplify the description let me add a scheme of the network connections.

   +-(WLAN0)---(WLANfritz)-+   +---(WLANfritz)---Device2(192.168.27.16)
   |                       |   |
Raspberry(192.168.27.2) FritzBox(192.168.27.1)
   |                       |   |
   +-(ETH0)---------(LAN1)-+   +-(LAN2)----------Device1(192.168.27.30)

The raspberry can always ping 192.168.27.30 while it can ping 192.168.27.16 only when only ONE interface (it can be ETH0 or WLAN0) is up (or in other conditions that I will detail later).
It turns out that the ARP Reply packet sent back by 192.168.27.16 to the raspberry in response to its ARP request always reaches the raspberry physical interface (either ETH0 or WLAN0) but it does not always reach the BOND0 interface.

To trigger an ARP exchange I flush the arp cache with the command

sudo ip -s -s neigh flush 192.168.27.xx

after that I give the usual command

ping 192.168.27.xx

At the same time in three different terminals I give the following commands

    sudo tcpdump -i bond0 -vv arp -nn host 192.168.27.xx 
    sudo tcpdump -i eth0  -vv arp -nn host 192.168.27.xx
    sudo tcpdump -i wlan0 -vv arp -nn host 192.168.27.xx

Let me consider the connection with the reachable device first (192.168.27.40).

Both Interfaces Up

(wlan0) tcpdump: listening on wlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
(bond0) tcpdump: listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
(eth0)  tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
(bond0) 10:50:50.905705 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.40 tell 192.168.27.2, length 28
(eth0)  10:50:50.905757 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.40 tell 192.168.27.2, length 28
(eth0)  10:50:50.913403 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.40 is-at b4:43:0d:dc:8b:e7, length 112
(bond0) 10:50:50.913403 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.40 is-at b4:43:0d:dc:8b:e7, length 112
No packet reaches wlan0

The request is first sent to bond0, then passed to eth0. The reply is received by eth0 and simultaneously received by bond0.

Now let's consider the non reachable device (192.168.27.16)

(bond0) 10:56:53.731660 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.16 tell 192.168.27.2, length 28
(eth0)  10:56:53.731710 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.16 tell 192.168.27.2, length 28
(wlan0) 10:56:53.827315 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.16 is-at 38:1a:52:47:32:35, length 28

This sequences continues for a long time while the ping command gives

From 192.168.27.2 icmp_seq=1 Destination Host Unreachable

The raspberry sends the request to bond0, it passes through eth0 but the answer arrives through wlan0 and it is not passed to bond0 and it is therefore discarded.

If the wlan0 interface is down, the ARP exchange to 192.168.27.40 is the same as before, while for 192.168.27.16 it is as follows

(bond0) 11:05:06.019184 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.16 tell 192.168.27.2, length 28
(eth0)  11:05:06.019241 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.16 tell 192.168.27.2, length 28
(eth0)  11:05:06.062927 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.16 is-at 38:1a:52:47:32:35, length 46
(bond0) 11:05:06.062927 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.16 is-at 38:1a:52:47:32:35, length 46

The request is first sent to bond0, then passed to eth0. The reply is received by eth0 and simultaneously received by bond0. The raspberry can therefore ping 192.168.27.16.

If the eth0 interface is down (and the wlan0 interface is up, of course)

(bond0) 11:13:10.508756 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.16 tell 192.168.27.2, length 28
(wlan0) 11:13:10.508808 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.16 tell 192.168.27.2, length 28
(wlan0) 11:13:10.526858 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.16 is-at 38:1a:52:47:32:35, length 28
(bond0) 11:13:10.526858 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.16 is-at 38:1a:52:47:32:35, length 28

If, for debugging purposes, I deselect the router security option "Active Wireless device can communicate with each other" (which is not something I want, because I need a wireless connected PC to print to a wireless connected printer), when both interfaces are up the raspberry can communicate with 192.168.27.16. The traffic is as follows

(bond0) 11:22:01.371787 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.2 tell 192.168.27.16, length 46
(eth0)  11:22:01.371787 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.27.2 tell 192.168.27.16, length 46
(eth0)  11:22:01.371881 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.2 is-at 7a:79:3c:31:5d:e3, length 28
(bond0) 11:22:01.371894 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.27.2 is-at 7a:79:3c:31:5d:e3, length 28

Obviously, with this router configuration, when eth0 goes down and only wlan0 remains up, the raspberry cannot reach 192.168.27.16. In this case the ARP reply packet never returns back.

In summary, the raspberry cannot communicate with hosts when the reply packet arrives from the wlan0 interface while the request was sent on eth0, because, I think, bond0 only receive packets from the active interface and discards others.

Is this a bug of the bonding module? Does this happen also for the raspberry pi v4? Is it a configuration problem of the router (but how can I deal with it)? Would it be possible to tweak the bonding kernel parameters (I tried with AllSlavesActive options, but I've never seen it used in various bonding examples. Even if I activate it in the netdev file, it does not seem to be really active by looking at the content of /sys/class/net/bond0/bonding/all_slaves_active). Should I try to force such a parameter directly in /etc/modeprobe.d/?

---

Some details of the configuration are given in the following.

When both interfaces are up this is the status of bonding

openhabian@openhab:~$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: eth0 (primary_reselect always)
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 500
Up Delay (ms): 1000
Down Delay (ms): 1000

Slave Interface: eth0
MII Status: up
Speed: 100 Mbps
Duplex: full
Link Failure Count: 2
Permanent HW addr: b8:27:eb:b3:1c:7d
Slave queue ID: 0

Slave Interface: wlan0
MII Status: up
Speed: Unknown
Duplex: Unknown
Link Failure Count: 2
Permanent HW addr: b8:27:eb:e6:49:28
Slave queue ID: 0

The ip address in this case is

 openhabian@openhab:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP group default qlen 1000
    link/ether 7a:79:3c:31:5d:e3 brd ff:ff:ff:ff:ff:ff
3: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:79:3c:31:5d:e3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.27.2/24 brd 192.168.27.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet6 fd6f:17f9:753c:0:7879:3cff:fe31:5de3/64 scope global mngtmpaddr noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::7879:3cff:fe31:5de3/64 scope link
       valid_lft forever preferred_lft forever
4: wlan0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP group default qlen 1000
    link/ether 7a:79:3c:31:5d:e3 brd ff:ff:ff:ff:ff:ff

The ARP cache reflects the fact the the communication with the problematic host cannot be established. Even if I delete the host from the ARP cache in order to force an ARP search, the problem remains

openhabian@openhab:~$ ip neighbor
192.168.27.1 dev bond0 lladdr 08:60:6e:eb:58:f8 REACHABLE
192.168.27.54 dev bond0 lladdr 50:46:5d:b5:80:bf REACHABLE
192.168.27.30 dev bond0 lladdr 34:f3:9a:cd:e5:4c REACHABLE
*192.168.27.16 dev bond0  FAILED*

openhabian@openhab:~$ sudo ip neighbor delete 192.168.27.16 dev bond0

ip neighbor
192.168.27.1 dev bond0 lladdr 08:60:6e:eb:58:f8 REACHABLE
192.168.27.54 dev bond0 lladdr 50:46:5d:b5:80:bf REACHABLE
192.168.27.30 dev bond0 lladdr 34:f3:9a:cd:e5:4c REACHABLE
*192.168.27.16 dev bond0  FAILED*

As soon as one of the interfaces goes down communication with the node restarts without need to delete the ARP cache:

openhabian@openhab:~$ sudo ip link set wlan0 down
openhabian@openhab:~$ ip neighbor
192.168.27.1 dev bond0 lladdr 08:60:6e:eb:58:f8 REACHABLE
192.168.27.54 dev bond0 lladdr 50:46:5d:b5:80:bf REACHABLE
192.168.27.30 dev bond0 lladdr 34:f3:9a:cd:e5:4c REACHABLE
*192.168.27.16 dev bond0 lladdr 38:1a:52:47:32:35 REACHABLE*

As soon as two interfaces are up the communication fails again.

---

Answer 1

It appears that it was sufficient to add the AllSlaveActive=1 option to the 02-bond0.netdev file described in @Ingo answer. In this way the ARP packet sent back by the router through the wlan0 interface is forwarded to the bond0 one and the ARP table is always populated. In order to set this parameter the restart of systemd-networkd.service was not enough, I had to reboot the raspberry. After that it was possible to ping all devices in all cases.

root@raspberrypi:~ # cat >/etc/systemd/network/02-bond0.netdev <<EOF
[NetDev]
# status: cat /proc/net/bonding/bond0
Name=bond
Kind=bond
[Bond]
Mode=active-backup
# primary slave is defined in *eth.network
MIIMonitorSec=500ms
MinLinks=1
AllSlavesActive=1
EOF

I do not know if there are any negative side-effects of this parameter. I also wonder why this is not necessary for the Raspberry pi v4 case described in @Ingo's post (or perhaps it depends on how the router manages the mac addresses of the wired and wlan connections).

EDIT 07/06/2020

Actually I experienced some negative side effects: sometimes the raspberry was not reachable anymore until I disconnected the ethernet cable. I therefore figured out a less elegant solution. I restored AllSlavesActive=0 and implemented the following simple script

#!/bin/bash

SLEEP=5
echo "starting wlan control"
date
while : ; do
    ETH0_STATUS=$( cat /sys/class/net/eth0/operstate)
    WLAN0_STATUS=$( cat /sys/class/net/wlan0/operstate)

    if [ $ETH0_STATUS == "up" ] ; then
        if [ $WLAN0_STATUS == "up" ] ; then
            echo "eth0 ON: wlan0 ON: setting wlan0 DOWN"
            sudo ip link set wlan0 down
            date
#        else
#            echo "eth0 ON: wlan0 OFF: doing nothing"
        fi
    else
        if [ $WLAN0_STATUS == "down" ] ; then
#           echo "eth0 OFF: wlan0 ON: doing nothing "
#           iwconfig
#        else
            echo "eth0 OFF: wlan0 OFF: setting wlan0 ON"
            sudo ip link set wlan0 up
            date
#           iwconfig
        fi
    fi
    sleep $SLEEP
done

to run in a window opened with tmux (so that it remains active at logout)

Answer 2

According to the commands output you have given about the configuration, bonding looks good so far. That all interfaces have the same mac address is due to bonding and will not confuse the fritz box router. Quite contrary it ensures that it does not sense the change of an interface on the fly because the mac address doesn't change. That's the trick of bonding.

You have one (or more?) devices that you only can ping if only the wlan0 interface is up as noted in a comment. If you have only the eth0 interface up you cannot reach the problematic device. This is also the case if both interfaces are up. Then eth0 is always used if available because of Primary Slave: eth0 (primary_reselect always). So we can say, if using the wired connection (eth0) then the problematic device cannot be connected. Because everything is working as expected with bonding I don't believe its a problem with it. You should look at your network configuration and/or at the fritz box router what could be the reason that only the WiFi connection works in this case.

Answer 3

This is essentially the result of the mac80211's fastpath mechanism. According to the mac80211 source code, ap_isolate controls where packets go after they reach the wireless AP. Should I call dev_queue_xmit to send the package directly through the wireless network adapter (ap_isolate=0) or call netif_receive_skb to send the package to the local upper-layer network stack for processing (ap_isolate=1)?

See https://blog.csdn.net/sxd2001/article/details/126820695