0

I have a C++ application that permanently runs on a Raspberry Pi 3 and might occasionally freeze. It writes logfiles using Boost log and a good indication of a freeze is the absence of logging output for a certain period of time.

Now and easiest solution would be to have the RPi's hardware watchdog monitor the logfile and eventually do a restart. Unfortunately, I'm using log rotation resulting in changing file names for the current log file. And seemingly, the watchdog config file at /etc/watchdog.conf only takes a single file name to monitor.

Is there any possibility to have the watchdog monitor the whole log directory? Monitoring the most recent logfile would also be a possibility, but due to log rotation I can only provide the watchdog with a filename pattern here.

1

Try specifying a directory name instead of a file name: according to the man page, watchdog will try to stat the file, and that should work for directories too. Note that directory is only considered "changed" when files are created/removed in it, so the triggering period should be long enough for the rotation to occur.

If that's not OK, you will have to write your own "heartbeat" code. I would start with a script which checks the log files and reboots if they didn't change, and run that script from a cron job. Watchdog is only really needed to catch cases where your whole system is unresponsive, not just one app.

0

Ok, I solved it with a watchdog, but not with the RPi's one.

My app is running as a systemd service, and seemingly systemd also offers a watchdog mechanism. I've created the following service description

[Unit]
Description=DroneBox

[Service]
Type=simple

WorkingDirectory=...
ExecStart=...

Restart=always
RestartSec=30

# Watchdog-Signal.
WatchdogSec=900

[Install]
WantedBy=multi-user.target

The line WatchdogSec=900 makes the watchdog restart my app if it doesn't send a keepalive signal every 900 seconds latest. The app now must notify the watchdog after initialization to start the counter and regularly send keepalive signals. In C++, this can be done using the systemd library by

#include <systemd/sd-daemon.h>
...
sd_notify(0, "READY=1"); // Start watchdog
...
sd_notify(0, "WATCHDOG=1"); // Send keepalive signal

See e.g. this link for a good usage description of that library.

I'm not calling the Boost logging commands directly, but through a wrapper class, so I can easily exchange logger implementations.

#include "Logger.h"
#include <boost/log/core.hpp>
#include <boost/log/trivial.hpp>
#include <boost/log/expressions.hpp>
#include <boost/log/sinks/text_file_backend.hpp>
#include <boost/log/utility/setup/file.hpp>
#include <boost/log/utility/setup/common_attributes.hpp>
#include <boost/log/sources/severity_logger.hpp>
#include <boost/log/sources/record_ostream.hpp>
#include <boost/log/utility/setup/console.hpp>
#include <fstream>
#include <boost/log/utility/setup/from_stream.hpp>
#include <stdlib.h>
#include <systemd/sd-daemon.h>

Logger* Logger::pInstance = nullptr;

Logger::Logger()
{

    boost::log::register_simple_filter_factory<boost::log::trivial::severity_level, char>("Severity");
    boost::log::register_simple_formatter_factory< boost::log::trivial::severity_level, char >("Severity");
    boost::log::core::get()->add_global_attribute("TimeStamp", boost::log::attributes::local_clock());

    std::string a_cwd = std::string(getcwd(NULL, 0));
    boost::log::add_common_attributes();
    std::ifstream file(a_cwd + "/log.ini");
    boost::log::init_from_stream(file);

}

Logger::~Logger()
{
}

Logger* Logger::instance()
{
    if (pInstance == nullptr) {
        pInstance = new Logger();
    }
    return pInstance;
}

void Logger::logInfo(std::string message)
{
    sd_notify(0, "WATCHDOG=1");
    BOOST_LOG_TRIVIAL(info) << message;
}

void Logger::logDebug(std::string message)
{
    sd_notify(0, "WATCHDOG=1");
    BOOST_LOG_TRIVIAL(debug) << message;
}

void Logger::logWarn(std::string message)
{
    sd_notify(0, "WATCHDOG=1");
    BOOST_LOG_TRIVIAL(warning) << message;
}

void Logger::logError(std::string message)
{
    BOOST_LOG_TRIVIAL(error) << message;
}

void Logger::logFatal(std::string message)
{
    BOOST_LOG_TRIVIAL(fatal) << message;
}

By calling sd_notify(0, "WATCHDOG=1") with every logging command, I've reached my original goal.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.