0

I am running a small server with 6 docker services on a Raspberry Pi Modell B. After a short time of inactivity (and with that I mean no being connected via ssh) of approximately 20-30 minutes, my services can not be reached anymore. I have assigned a static IP address to my Pi. Pinging it yields in no response. I can of course not login to SSH.

This has not happened before. It has run rock solid for the last 6 months 24/7 without any shutdown. I am unsure how to approach this problem and I have no idea how to debug it.

I have tried to find something suspicious in /var/log/syslog/ but I had no success. There is no "gap" in the timestamps of the log file which looks like that the Pi hasn't crashed or anything related to that. The red power LED was on the whole time.

Question: How can I debug this kind of problem? What are things to search for in the syslog file? (I would share my syslog file here, but I am not sure if this would be a security concern. If it isn't, let me know.)

Remarks: My problem might be related to this question but it seems to be slightly different.

This is my setup.

  • Raspberry Pi Model B
  • Raspbian GNU/Linux 10 (buster)
  • Connected via LAN (no Wifi)
  • Official recommended power supply I had no problems for the last six month. I have not updated anything on my Pi. The problem suddenly appeared.
6
  • 1
    Have you disabled power management on your WiFi adapter? raspberrypi.stackexchange.com/a/96608/94397
    – Dougie
    Mar 23, 2022 at 20:51
  • 1
    Does it happen when the 6 docker services are not running?
    – Katu
    Mar 24, 2022 at 9:43
  • 1
    Don't know if you've resolved this issue or not, but FWIW, there is a related thread on GitHub where this same issue is discussed.
    – Seamus
    Apr 4, 2022 at 18:15
  • 2
    Thank you @Seamus for the follow up. The issue was solved with the steps provided in my answer, though I have no idea why it has solved it since the root cause is still unknown to me. Your answer has given me great insights on aspects that I haven't known until then, and it could be very much possible that the issue discussed on the provided GitHub thread has something to do with it as well. As usual, the time for debugging is rare on my side at the moment, so I'm happy that it "just works' somehow right now. Thank you again for your help! Apr 4, 2022 at 19:41
  • 1
    Good enough then... if you do get around to debugging at some point, a follow-up here would be much appreciated.
    – Seamus
    Apr 4, 2022 at 19:46

2 Answers 2

3

I am not quite sure I follow your explanation of things in the answer you've provided. Your answer indicates you have resolved your problem for now, but your closing sentence suggests you are not quite sure what happened or why. I will try to address that here.

An Introductory:

As a point of clarification for readers who may be unfamiliar, please note the following:

  1. DHCP is the dynamic host configuration protocol. This protocol was designed to simplify IP address provisioning in networks by avoiding the error-prone task of manual provisioning.

  2. dhcpcd stands for DHCP client daemon. Its job is to obtain - from a DHCP server - all the required information needed by the RPi to conduct network operations: an IP address, a default route, and a link to DNS services - as a minimum. dhcpcd is currently the default networking software for RPi. It was chosen as the default DHCP client app for RPi OS by the powers that be in the Raspberry Pi organization, but you of course are free to change that.

  3. dhcpd stands for DHCP daemon. It is the server-side app for DHCP. Unless you're using your RPi as a DHCP server itself, you will not need to install it. Being a DHCP server is ordinarily a full-time job. Networks may become uncooperative when their DHCP server goes on sabbatical!

  4. "Routing" and "routers" refer the process and systems used to move network traffic between their sources and their destinations. It's often the case in smaller networks that routers and DHCP servers cohabit a single box or appliance. Accordingly references to routers sometimes include the DHCP server function as well. The only point here is that routing and DHCP are separate and distinct network functions, but they may be housed together.

I do not claim any particular expertise in DHCP. But I have at least read the system documentation for dhcpcd (man dhcpcd.conf & man dhcpcd), and the Internet Standard for DHCP - RFC 2131. I had occasion to review the protocol some time ago in relation to this old Q&A - which may be relevant to your experience?

FWIW - My take on your experience:

You stated:

  1. The router assigns a static IP address to the pi's mac address.

I am not sure why you say this because the router/DHCP server does not assign static IP addresses! Perhaps you meant that you - as the router/DHCP administrator - made a "static mapping" assignment manually through your router/DHCP server's configuration interface? I'll emphasize this point as it seems to confuse many: Creating a static IP for your RPi in dhcpcd.conf does not induce your router or your DHCP server to do anything at all.

In fact, the static IP configuration in dhcpcd.conf will prevent your RPi's dhcpcd from communicating with the DHCP server. dhcpcd is a client of the DHCP server; it sees a static ip declaration in dhcpcd.conf as the system administrator (you in this case) saying, "I don't need you to act as a client; I'm providing the DHCP server's information for you here." Put another way, dhcpcd will never communicate the static ip configuration you've made in dhcpcd.conf to your router/DHCP server.

But there is a line in man dhcpcd.conf that's very relevant to both DHCP client and server wrt a static IP:

For IPv4, you should use the inform ipaddress option instead of setting a static address.

The syntax for the inform option is given by example below for the wlan0 interface in IPv4:

wlan0
inform 192.168.1.51/24    # '/24' is the CIDR unless you've created a subnet

What does that inform option mean?

I'll call your attention to para 3.4 of RFC 2131. This passage explains the recommendation above for using the inform option: The inform option in dhcpcd.conf provides the mechanism needed to let your router/DHCP server know that you have assigned a static IP address to your RPi. Hopefully it is clear that the router/DHCP server needs this information.

The inform option "informs" the DHCP server to bind the client's MAC addr to the designated IP address, and prompts the server to complete the provisioning process by supplying the default route, DNS hosts and any other data the DHCP server ordinarily sends to clients (e.g. NTP server address for time services). Note however this passage from para 3.4:

The server SHOULD check the network address in a DHCPINFORM message for consistency, but MUST NOT check for an existing lease.

This passage emphasizes that it is the administrator's responsibility to avoid address collisions for static IP assignments. Put another way, the inform option does not give the administrator carte blanche authority to assign addresses willy-nilly without consequence.

Hopefully, it is now clear that the correct approach to static ip usage includes configuration of the DHCP server so that it will never assign an IP address designated in the inform option. How this is done will depend on the characteristics of the DHCP server in use. The exception to this of course is if your entire network is static addresses; in this case you have no need for a DHCP server (or dhcpcd for that matter).

In summary, there are two points:

  1. If you have a DHCP server on your network, placing a static ip configuration in dhcpcd.conf is unnecessary at best, and a source of chaos in your network at worst. This due to the fact that a static ip configuration in dhcpcd is never communicated to the DHCP server. Once again, this is why the author of dhcpcd included the admonition against it in his documentation, and his recommendation to use the inform option.

  2. I don't use docker services, and I'm not familiar with how they obtain their networking information. However, quickly skimming this article from the docker docs suggests that your docker services may be completely dependent on your RPi's network configuration! If that's the case, broken networking on your RPi implies broken networking in your docker apps.

    I hope that understanding how DHCP clients & servers interact (or don't interact), provides you with some useful clues for your analysis. I also hope this "answer" will dissuade you (and others) from using the static ip configuration shown in your answer, and to use the inform (or request) option instead.

12
  • Thank you for these insights. I made the assignment static ip address -> mac address manually in the router itself. I haven't set a static IP address inside the Pis dhcpcd.conf (before this issue; I haven't touched this file ever). The Pi asks the router for an IP address and the router assigns the static IP address which I have configured in the DHCP server which run on the router. What I am not sure about is, why setting the static ip adress in the dhcpd.conf as well solves the issue. This shouldn't be necessary from my current understanding. Mar 24, 2022 at 16:51
  • That said, I'm not sure why a device should inform the router about a static IP address. The router's DHCP server does the assignment anyway. Mar 24, 2022 at 16:52
  • 1
    It shouldn't be necessary to use dhcpd.conf at all, if the router manages the static IP assignment... Mar 24, 2022 at 17:26
  • 1
    I agree it shouldn't be necessary but assigning a static ip address in dhcpcd.conf works very reliably in my experience
    – Drdrm46
    Mar 24, 2022 at 17:50
  • 1
    @DavidScholz: It shouldn't be necessary to use dhcpd.conf at all, if the router manages the static IP assignment... That's correct. In "olden days" - before DHCP showed up - every host on the network was manually configured. It worked, but could be brittle. I wonder how many RPi would ever be used today if all networks were manually configured?
    – Seamus
    Mar 27, 2022 at 19:02
1

I have figured it out, but it is still confuses me why this has not happened before. In short: it seems like the Pi assigns a different IP address to itself after a while. I have monitored the network traffic in my firewall and the static IP address became inactive (as expected), but the DHCP server of my router did not assign a new address to any device in my current network, which led me to the conclusion that the Pi itself does it (which the router does not get notified about of course).

My current setup does the following:

  1. The router assigns a static IP address to the pi's mac address.
  2. The IP addresses of each docker service is mapped to the same static IP address as usual.
  3. The DNS of my router assigns an URL for each docker service to the same IP address of the pi.
  4. My reverse proxy forwards the requests for the specific URLs to the correct service.

The IP address is correctly assigned to the Pi which I have verified by executing the following command.

ifconfig

I have also executed the following command on my Pi.

hostname -I

The static IP address assigned by the router was correctly shown here as well as the IP addresses of the docker services, but I have noticed 8 more IP addresses which I still don't know where they are comming from (and I don't even know if this is related to the problem).

Nevertheless, here is what I did to resolve it. I assigned the static IP address from my router directly in /etc/dhcpcd.conf as follows.

Get the current IP address of the router

ip r | grep default

which yields in something like the following output.

default via 192.168.22.1 dev eth0 proto dhcp src 192.168.22.102 metric 202

The first one is the IP address of the router. The IP address of the currently used DNS is located in /etc/resolv.conf, which matches with my router's ip address in my case.

Assign a static IP address in /etc/dhcpcd.conf

Next, edit the file /etc/dhcpcd.conf, uncomment the following lines and add the information that we have gathered above.

# Example static IP configuration:
interface eth0
static ip_address=192.168.22.100/24
static routers=192.168.22.1
static domain_name_servers=192.168.22.1

In the field static ip_address, we must of course put the static IP address which the router assigns to the Pi as well.

Reboot via sudo reboot. SSH back into the Pi and run hostname -I again. The 8 unknown IP addresses from above have disappeared. The static IP address from the Pi as well as the IP addresses from my docker services remain.

I will leave this question open for a bit longer with the hope that someone has more insights on why this happened.

12
  • Just to say that I have found modifying dhcpcd.conf as the reliable method of ensuring a static ip address.
    – Drdrm46
    Mar 24, 2022 at 16:28
  • 1
    I agree what you experienced should not happen. Sometimes routers drop the connection and the Pi (or rather dhcpcd) says "let me ask for another IP address". Whatever the reason, since you want a static IP address and you are messing with dhcpcd.conf you might as well get one outside the dynamic address pool of the router. That way the router will have more addresses to dynamically assign to more devices in your network.
    – user68186
    Mar 24, 2022 at 20:18
  • 1
    @Seamus said elsewhere: The request <ip address> line in dhcpcd.conf should do what you are trying to do. Request a specific IP address from within the dynamic address pool of the router.
    – user68186
    Mar 24, 2022 at 20:24
  • 1
    Here is an answer that recommends the request statement in dhcpcd.conf.
    – user68186
    Mar 24, 2022 at 20:37
  • 1
    @user68186: You're on the right track. I revised my answer earlier today in an effort to clarify some of this, but yeah - the static option communicates nothing to the DHCP server.
    – Seamus
    Mar 27, 2022 at 2:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.