Python pygame program won't run through SSH / Remote Desktop

Question

I've been working on my own object recognition program based on the rpi-vision test program pitft_labeled_output.py (from this webpage). It's basically a custom neural network model and a bit modified code for the new model to work. However, I'm having problems running my program through Remote Desktop. This is how I run my program (using venv from Graphic Labeling Demo):

pi@raspberrypi:~ $ sudo bash
root@raspberrypi:/home/pi# cd signcap && . ../rpi-vision/.venv/bin/activate
(.venv) root@raspberrypi:/home/pi/signcap# python3 src/dar_ts.py
pygame 1.9.6
Hello from the pygame community. https://www.pygame.org/contribute.html
No protocol specified
No protocol specified
No protocol specified
xcb_connection_has_error() returned true
No protocol specified
xcb_connection_has_error() returned true
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
No protocol specified
Unable to init server: Could not connect: Connection refused

(Mask:1359): Gtk-WARNING **: 20:00:31.012: cannot open display: :10.0

As you can see, I'm getting several "No protocol specified" errors and the display error. When I run echo $DISPLAY in a console when I'm connected through Remote Desktop, it prints :10.0.

When I run the pitft_labeled_output.py through Remote Desktop like:

sudo bash
root@raspberrypi:/home/pi# cd rpi-vision && . .venv/bin/activate
(.venv) root@raspberrypi:/home/pi/rpi-vision# python3 tests/pitft_labeled_output.py

The display is successfully opened and everything works as it should.

However, my program works fine locally or through VNC. When I'm connected through VNC and run echo $DISPLAY, I get :1.0. What could be the issue that it doesn't work through Remote Desktop, but pitft_labeled_output.py does?

Here is the code of my program so you can compare it with pitft_labeled_output.py from rpi-vision:

import time
import logging
import argparse
import pygame
import os
import sys
import numpy as np
import subprocess
import signal

# Environment variables for Braincraft HAT.
os.environ['SDL_FBDEV'] = "/dev/fb1"
os.environ['SDL_VIDEODRIVER'] = "fbcon"

def dont_quit(signal, frame):
   print('Caught signal: {}'.format(signal))
signal.signal(signal.SIGHUP, dont_quit)

from capture import PiCameraStream
from tsdar import TrafficSignDetectorAndRecognizer

# Initialize the logger.
logging.basicConfig()
logging.getLogger().setLevel(logging.INFO)

# Initialize the display.
pygame.init()
# Create a Surface object which is shown on the display.
# If size is set to (0,0), the created Surface will have the same size as the
# current screen resolution (240x240 for Braincraft HAT).
screen = pygame.display.set_mode((0,0), pygame.FULLSCREEN)
# Declare the capture manager for Pi Camera.
capture_manager = None

# Function for parsing program arguments.
def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('--rotation', type=int, choices=[0, 90, 180, 270],
                        dest='rotation', action='store', default=0,
                        help='Rotate everything on the display by this angle')
    args = parser.parse_args()
    return args

last_seen = []
already_seen = []

def main(args):
    global capture_manager, last_seen, already_seen

    # Initialize the capture manager to get stream from Pi Camera.
    if screen.get_width() == screen.get_height() or args.roation in (0, 180):
        capture_manager = PiCameraStream(resolution=(max(320, screen.get_width()), max(240, screen.get_height())), rotation=180, preview=False, format='rgb')
    else:
        capture_manager = PiCameraStream(resolution=(max(240, screen.get_height()), max(320, screen.get_width())), rotation=180, preview=False, format='rgb')

    # Initialize the buffer size to screen size.
    if args.rotation in (0, 180):
        buffer = pygame.Surface((screen.get_width(), screen.get_height()))
    else:
        buffer = pygame.Surface((screen.get_height(), screen.get_width()))

    # Hide the mouse from the screen.
    pygame.mouse.set_visible(False)
    # Initialize the screen to black.
    screen.fill((0,0,0))
    # Try to show the splash image on the screen (if the image exists), otherwise, leave screen black.
    try:
        splash = pygame.image.load(os.path.dirname(sys.argv[0])+'/bchatsplash.bmp')
        splash = pygame.transform.rotate(splash, args.rotation)
        screen.blit(splash, ((screen.get_width() / 2) - (splash.get_width() / 2),
                    (screen.get_height() / 2) - (splash.get_height() / 2)))
    except pygame.error:
        pass
    pygame.display.update()

    # Use the default font.
    smallfont = pygame.font.Font(None, 24)
    medfont = pygame.font.Font(None, 36)

    # Initialize the traffic sign detector and recognizer object with the path
    # to the TensorFlow Lite (tflite) neural network model.
    tsdar0 = TrafficSignDetectorAndRecognizer(os.path.dirname(sys.argv[0])+'/models/uw_tsdar_model_no_aug_w_opts.tflite')

    # Start getting capture from Pi Camera.
    capture_manager.start()

    while not capture_manager.stopped:
        # If the frame wasn't captured successfully, go to the next while iteration
        if capture_manager.frame is None:
            continue

        # Fill the buffer with black color
        buffer.fill((0,0,0))

        # Update the frame.
        rgb_frame = capture_manager.frame

        # Make predictions. If traffic signs were detected, a bounding rectangle
        # will be drawn around them.
        timestamp = time.monotonic()
        predictions, out_frame = tsdar0.predict(rgb_frame)
        delta = time.monotonic() - timestamp
        logging.info(predictions)
        logging.info("TFLite inference took %d ms, %0.1f FPS" % (delta * 1000, 1 / delta))

        # Make an image from a frame.
        previewframe = np.ascontiguousarray(out_frame)
        img = pygame.image.frombuffer(previewframe, capture_manager.camera.resolution, 'RGB')

        # Put the image into buffer.
        buffer.blit(img, (0, 0))

        # Add FPS and temperature on the top corner of the buffer.
        fpstext = "%0.1f FPS" % (1/delta,)
        fpstext_surface = smallfont.render(fpstext, True, (255, 0, 0))
        fpstext_position = (buffer.get_width()-10, 10) # Near the top right corner
        buffer.blit(fpstext_surface, fpstext_surface.get_rect(topright=fpstext_position))
        try:
            temp = int(open("/sys/class/thermal/thermal_zone0/temp").read()) / 1000
            temptext = "%d\N{DEGREE SIGN}C" % temp
            temptext_surface = smallfont.render(temptext, True, (255, 0, 0))
            temptext_position = (buffer.get_width()-10, 30) # near the top right corner
            buffer.blit(temptext_surface, temptext_surface.get_rect(topright=temptext_position))
        except OSError:
            pass

        # Reset the detecttext vertical position.
        dtvp = 0

        # For each traffic sign that is recognized in the current frame (up to 3 signs),
        # its name will be printed on the screen and it will be announced if it already wasn't.
        for i in range(len(predictions)):
            p = predictions[i];
            name = tsdar0.CLASS_NAMES[p]
            print("Detected", name)

            last_seen.append(name)

            # Render sign name on the bottom of the buffer (if multiple signs detected,
            # current sign name is written above the previous sign name).           .
            detecttext = name
            detecttext_font = medfont
            detecttext_color = (255, 0, 0)
            detecttext_surface = detecttext_font.render(detecttext, True, detecttext_color)
            dtvp = buffer.get_height() - (i+1)*(detecttext_font.size(detecttext)[1]) - i*detecttext_font.size(detecttext)[1]//2
            detecttext_position = (buffer.get_width()//2, dtvp)
            buffer.blit(detecttext_surface, detecttext_surface.get_rect(center=detecttext_position))

            # Make an announcement for the traffic sign if it's new (not detected in previous consecutive frames).
            if detecttext not in already_seen:
                os.system('echo %s | festival --tts & ' % detecttext)

        # If new traffic signs were detected in the current frame, add them to already_seen list
        for ts in last_seen:
            if ts not in already_seen:
                already_seen.append(ts)

        # If the traffic sign disappeared from the frame (a car passed it), remove it from already_seen
        diff = list(set(already_seen)-set(last_seen))
        already_seen = [ts for ts in already_seen if ts not in diff]

        # Reset last_seen.
        last_seen = []

        # Show the buffer image on the screen.
        screen.blit(pygame.transform.rotate(buffer, args.rotation), (0,0))
        pygame.display.update()

# Run the program until it's interrupted by key press.
if __name__ == "__main__":
    args = parse_args()
    try:
        main(args)
    except KeyboardInterrupt:
        capture_manager.stop()

Edit: To clarify a bit more, I first followed this tutorial to install my Braincraft HAT and then followed this one to try out the object recognition test example (pitft_labeled_output.py) from rpi-vision. Everything worked great through SSH. I saw logging info in the SSH console and the camera feed and recognized objects on the Braincraftt HAT display. Then I decided to try it out from Windows Remote Desktop (after installing xrdp on RPi) and it worked great. I saw logging info in terminal and camera feed on Braincraft display. But, when I wanted to run my program instead of pitft_labeled_output.py, I received the errors mentioned above. I even went further and replaced pitft_labeled_output.py code with my code(dar_ts.py) and ran my code as if I was running pitft_labeled_output.py (thought that there might be some dependencies inside rpi-vision folder), but it didn't work, received the same error. What could be the issue with my code?

P.S. What also confused me further is that pitft_labeled_output.py has a typo in line 56 and runs fine anyway, but when I ran my code for the first time, it asked me to correct the error.

Edit2: Here is a demonstration video of the problem.

Answer 1

"No protocol specified" is the error you get when you start the X server as user 1 (in your case: "pi") and then try to access it as another user (in your case: "root"). Problem is, I see no dependencies on the X server in your code (you're using the framebuffer), but I don't know pigame and the other libraries well enough to tell you with certainty that there is no such dependency.

I'm also surprised that pitft_labeled_output.py runs with a typo in main(), perhaps you should double-check what you run exactly.

Anyhow, using the X server as root can be possible if you enable pam_xauth on the Pi: make sure it is installed and edit /etc/pam.d/su to add the line

session optional pam_xauth.so

Though I think you should really try to understand why you need that access in the first place.