Having multiple interfaces on a host addressed within the same subnet can cause a networking issue called "ARP Flux" and is to be avoided.
So not only is it POSSIBLE to join different interfaces to different subnets, it's IMPLIED when multi-homing a host not to address multiple interfaces within the same subnet. And as this example demonstrates, you do NOT require (2) nics each with a different MAC address.
And this can be accomplished WITHOUT A VLAN, only using a static route. The below example is tested and known to work correctly. If you have a couple of small switches and a Pi, you too can actually try this out quite easily.
SETUP:
Using (2) different /28 subnets- sensibly sized for a pair of 8 port switches- our specimen networking is organized as follows:
Switch 1: 192.168.1.0/28
Switch IP: 192.168.1.1/28
Router Interface: 192.168.1.2/28
Assignable IPs: 192.168.1.3-14
Switch 2: 192.168.1.16/28
Switch IP: 192.168.1.17/28
Router Interface: 192.168.1.18/28
Assignable IPs: 192.168.1.19-30
Raspberry Pi:
Has a physical connection to Switch 1
We'll assign last IP in each subnet to Pi:
192.168.1.14 (Switch 1)
192.168.1.30 (Switch 2)
There are several ways we can address our single eth0 interface, discussed after next section "ROUTING"
ROUTING:
Only static IPs configured: DHCP disabled on router interfaces the switches are connected to
Switch 1 has a direct connection to router which routes all traffic for switch 1's subnet 192.168.1.0/28 to it.
Since 192.168.1.14 is within Switch 1's routed subnet, no static route is required for this address.
HOWEVER:
Since the Pi's other IP 192.168.1.30 is within Switch 2's subnet, by default traffic for this IP will be routed to Switch 2 causing connections to fail.
Routers route traffic from MOST specific to LEAST specific route: a static route pointing to a host will be used before one pointing to a subnet.
Add a static route on router:
Destination Address: 192.168.1.30/32 (Interface of Pi OUTSIDE switch's routed subnet)
Gateway Address: 192.168.1.14 (interface of Pi INSIDE switch's routed subnet)
Note we are NOT using router's interface to Switch 1 192.168.1.2, but IP of Pi within routed subnet of Switch 1.
Once traffic arrives at Pi, it knows that it's the destination for 192.168.1.30.
Once static route added, host can be accessed on either address: 192.168.1.14 (subnet 1) and 192.168.1.30 (subnet 2)
ADDRESSING:
Couple of ways- maybe not the only ones- to assign addresses on the single eth0 interface:
Method 1: /etc/dhcpcd.conf & /etc/rc.local
/etc/dhcpcd.conf
Use the IP within switch 1's routed subnet:
interface eth0
static ip_address=192.168.1.14/28
static routers=192.168.1.2
static domain_name_servers=8.8.8.8
/etc/rc.local
Add following command to rc.local
/sbin/ip addr add dev eth0 192.168.1.30/28
NOTE: Interface(s) added via rc.local will survive "systemctl restart networking"
Method 2: /etc/network/interfaces.d/
/etc/network/interfaces
Change FROM:
source-directory /etc/network/interfaces.d
TO:
source /etc/network/interfaces.d/*
Next, create (2) files with your addressing details:
/etc/network/interfaces.d/eth0
auto eth0
iface eth0 inet static
address 192.168.1.14
netmask 255.255.255.240
network 192.168.1.0
broadcast 192.168.1.15
gateway 192.168.1.2
/etc/network/interfaces.d/eth0:0
auto eth0:0
iface eth0:0 inet static
address 192.168.1.30
netmask 255.255.255.240
Anyhoo, if you have a use-case that requires multi-homing, hope this post helped you-
