Creating a daemon that pipes output from one programme to another under systemd?

Question

The system is a Raspberry Pi B+ running Raspbian GNU/Linux 10 (buster) and using systemd.

I had expected this to be a trivial exercise but it is appearing to be less so than expected. I have a USB SDR receiver than is set up to pass a stream of data through a third party programme (written in C) that strips data and then deposits it in an RRD as well as a second flat text file that aggregates all data received.

Essentially the data is being broadcast by an Energy monitor (Efergy) which broadcasts on Bluetooth a pulse every 6 seconds at 433.5MHz that is a reading of the amount of energy being consumed at that point in time. This data is collected by a separate (physically independent monitor) that aggregates the data and displays the amount of energy consumed at the time and over certain time frames (day, week, month etc). It seemed to me this would be an ideal project for a Pi to collect the data into a form that can be more easily displayed in graphic form and displayed on a webpage similar to data I am aggregating from a number of temperature sensors around 2 locations.

What is different to the latter is that the data is broadcast continuously and the rtl-sdr listening programme needs to be run continuously. Temperature measurements can be run intermittently through crontab. The process works fine using the cli using a command such as (method 1)

rtl_fm -f 433500000 -s 200000 -r 96000 -g 7.7 2>/dev/null | ./Energy_Log /home/pi/Energy/Samples.txt

The output of the RTL-SDR is then piped into the C programme (originally written by Nataniel Elijah then adapted by Dr Li Gough and myself) which calculates the actual data required and writes this to a round-robin database (rrdtool by Oetiker) as well as a flat text file. My 2 adaptions create a momentary system command through Energy_Log to run the rrdtool command that inserts the datapoint at the specified unixtime each time a new datapoint is received, or 'heard'. I admit it would make more sense using the rrdtool 'C' library to do this directly but it works and I couldn't get the 'C' compiler to accept the rrdtool commands.

When run in the above form ps -axf shows the structure of the programme call as:-

12857 ?        S      0:00  |   \_ sshd: pi@pts/0
12860 pts/0    Ss     0:01  |       \_ -bash
13010 pts/0    Sl+    0:41  |           \_ rtl_fm -f 433500000 -s 200000 -r 96000 -g 7.7
13011 pts/0    S+     0:02  |           \_ ./Energy_Log /home/pi/Energy/Samples.txt

Here the ssh terminal runs a bash process which runs 2 programmes concurrently one the rtl_fm programme which then pipes the output to the concurrent Energy_Log programme. Checking that the output is being sent to the Samples.txt file using tail shows:-

20201031_17:13:40     754.277344
20201031_17:13:46     758.554688
20201031_17:13:52     775.693359
20201031_17:13:58     750.000000
20201031_17:14:04     745.693359

ie at 5:14pm and 4secs on the 31st October 2020 the monitor was reading 745.7 watts of instantaneous consumption. Well I suppose more correctly the house is drawing 3.107A of current at 240VAC. This works fine as long as I always want to run the programme from an open terminal or ssh terminal. However as with all of my monitoring Pi's, my real aim is to have the process run like a daemon without a specific attached terminal. I thought this would be relatively simple but then realised that the change to systemd introduces some complexity that I don't think were there previously. Specifically it seems impossible to run a simple one line bash script that 'pipes' the output of one programme into the input of another.

Method 2, Creating a service script in /etc/systemd/system called rtlsdr.service that contains the commands:-

Description=RTLTCP
After=network.target
[Service]
ExecStartPre=/bin/sleep 30
ExecStart=/usr/local/bin/rtl_fm -f 433500000 -s 200000 -r 96000 -g 19.7 2>&1 | /home/pi/Energy/Energy_Log /home/pi/Energy/Samples.txt
[Install]
WantedBy=multi-user.target

Killing the properly constituted programme running in a terminal window above and then running the command sudo systemctl start rtlsdr.service does start a daemon but it runs as a single line bash command producing the ps axf shown below:-

12839 ?        Ss     0:00 /lib/systemd/systemd --user
12842 ?        S      0:00  \_ (sd-pam)
13629 ?        Ssl    0:21 /usr/local/bin/rtl_fm -f 433500000 -s 200000 -r 96000 -g 19.7 2>&1 | /home/pi/Energy/Energy_Log /home/pi/Energy/Samples.txt

At least these seem to be the 3 lines of interest. Now it maybe that I am missing something in the construction of the command as a pipe, if so what is it? When I look at what is essentially the logfile, Samples.txt, I see the last written value as that when I killed the prior process and the round-robin db has this as a last accessed time as well.

In trying to get it working I came across a number of references to the fact that it is not possible to do what I was after with systemd even to the point that somebody had created a specific programme to do this which is part of a number of linux distributions including Debian that runs on the Raspberry Pi (see https://packages.debian.org/source/buster/pipexec). What is a problem is that there seem to be few people that have implemented it and, though the help file associated with the github entry is quite explicit (see https://salsa.debian.org/alteholz/pipexec/-/blob/master/README.md) I'm still having problems working out whether I have the command correct or not. As it stands the new systemd service file looks like (method 3):-

Description=RTL-SDR output from Efergy Monitor
After=network.target
[Service]
ExecStartPre=/bin/sleep 30
ExecStart=/usr/bin/pipexec -l 7 -- [ RTL /usr/local/bin/rtl_fm -f 433500000 -s 200000 -r 96000 -g 7.7 2>&1 ]  |  [ ELOG /home/pi/Energy/Energy_Log /home/pi/Energy/Samples.txt ] '{RTL:1>ELOG:0}' 7>/var/log/rtl.log
[Install]
WantedBy=multi-user.target

NB the double dash is important as you are apssing the entire balance of the line to the command as an argument. Yes it was originally wrong so when started the usual ps axf command shows:-

14075 ?        Ss     0:00 /usr/bin/pipexec -l 7 [ RTL /usr/local/bin/rtl_fm -f 433500000 -s 200000 -r 96000 -g 7.7 2>&1 ] | [ ELOG /home/pi/Energy/Energy_Log /home/pi/Energy/Samples.txt ] {RTL 1>ELOG 0} 7>/var/log/rtl.log
14077 ?        Sl     0:03  \_ /usr/local/bin/rtl_fm -f 433500000 -s 200000 -r 96000 -g 7.7 2>&1
14078 ?        S      0:00  \_ /home/pi/Energy/Energy_Log /home/pi/Energy/Samples.txt

which is ok and output occurs as expected. Logfile (/var/log/rtl.log) is chmod 777, output files chmod 776, obviously the programmes have execute access. Anybody out there used pipexec?

Repeating the system is a Raspberry Pi B+ running Raspbian GNU/Linux 10 (buster).

Answer 1

systemd does not do shell style parsing of the ExecStart= lines. See systemd.service. Typically, to have the pipe character and 2>&1 parsed by the shell, you will need to invoke the shell. Eg:

ExecStart=/bin/bash -c '/usr/local/bin/rtl_fm ... 2>&1 | /home/pi/...'

Beware of using $ and quotes in these lines, as systemd does parse them somewhat.

Answer 2

For the files as the question was already answered, but IMHO some details were still missing.

As Tim Baker already wrote: In the command

/usr/bin/pipexec -l 7 -- [ RTL /usr/local/bin/rtl_fm -f 433500000 -s 200000 -r 96000 -g 7.7 2>&1 ]  |  [ ELOG /home/pi/Energy/Energy_Log /home/pi/Energy/Samples.txt ] '{RTL:1>ELOG:0}' 7>/var/log/rtl.log

you mix two concepts: pipexec does not parse or even understand something like 2>&1. This is a shell (bash) notation redirection. With pipexec you use the syntax {A:N>B:M} to redirect. In your code, the string 2>&1 is passed as a parameter to rtl_fm.

In addition the 7> is (still) a bash command which is not understood by systemd nor pipexec. A typical solution is using either tee or ptee here. Or you can use -l s to log directly to syslog / journald.

Note 1: it is not clear what you want to do with the stderr of the rtl_fm as in the shell command you discard it and it looks that in the pipexec command you want to handle the data?

Note 2: If you carefully read the logs of pipexec you can observe what happens here:

$ pipexec -l 2 -- '[ A /bin/true 2>&1 ]'
[...]
2022-06-21 07:16:19;pipexec;61897;0;command;info;command_info;[command]=[ A /bin/true 2>&1 ]];

Note 3: I'm the author of pipexec.

Answer 3

The answer from meuh is right but perhaps using pipexec is better. Ultimately I discovered a minor error in the pipexec command which when fixed ran exactly the way I wished it to (the command in the post above is now corrected). This has now been running for the best part of 2 years with information being written to an Oetiker RRD through a shell command (rather than trying to get the actual RRD 'C' library to work in perl). It was just easier that way.

I've also set up a daily query of the rrd to send a daily power consumption figure in kWhrs to a Mariadb on my main server at just after midnight. Seems to be working fine.