1

I've been working on my own object recognition program based on the rpi-vision test program pitft_labeled_output.py (from this webpage). It's basically a custom neural network model and a bit modified code for the new model to work. However, I'm having problems running my program through Remote Desktop. This is how I run my program (using venv from Graphic Labeling Demo):

pi@raspberrypi:~ $ sudo bash
root@raspberrypi:/home/pi# cd signcap && . ../rpi-vision/.venv/bin/activate
(.venv) root@raspberrypi:/home/pi/signcap# python3 src/dar_ts.py
pygame 1.9.6
Hello from the pygame community. https://www.pygame.org/contribute.html
No protocol specified
No protocol specified
No protocol specified
xcb_connection_has_error() returned true
No protocol specified
xcb_connection_has_error() returned true
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
No protocol specified
Unable to init server: Could not connect: Connection refused

(Mask:1359): Gtk-WARNING **: 20:00:31.012: cannot open display: :10.0

As you can see, I'm getting several "No protocol specified" errors and the display error. When I run echo $DISPLAY in a console when I'm connected through Remote Desktop, it prints :10.0.

When I run the pitft_labeled_output.py through Remote Desktop like:

sudo bash
root@raspberrypi:/home/pi# cd rpi-vision && . .venv/bin/activate
(.venv) root@raspberrypi:/home/pi/rpi-vision# python3 tests/pitft_labeled_output.py

The display is successfully opened and everything works as it should.

However, my program works fine locally or through VNC. When I'm connected through VNC and run echo $DISPLAY, I get :1.0. What could be the issue that it doesn't work through Remote Desktop, but pitft_labeled_output.py does?

Here is the code of my program so you can compare it with pitft_labeled_output.py from rpi-vision:

import time
import logging
import argparse
import pygame
import os
import sys
import numpy as np
import subprocess
import signal

# Environment variables for Braincraft HAT.
os.environ['SDL_FBDEV'] = "/dev/fb1"
os.environ['SDL_VIDEODRIVER'] = "fbcon"

def dont_quit(signal, frame):
   print('Caught signal: {}'.format(signal))
signal.signal(signal.SIGHUP, dont_quit)

from capture import PiCameraStream
from tsdar import TrafficSignDetectorAndRecognizer

# Initialize the logger.
logging.basicConfig()
logging.getLogger().setLevel(logging.INFO)

# Initialize the display.
pygame.init()
# Create a Surface object which is shown on the display.
# If size is set to (0,0), the created Surface will have the same size as the
# current screen resolution (240x240 for Braincraft HAT).
screen = pygame.display.set_mode((0,0), pygame.FULLSCREEN)
# Declare the capture manager for Pi Camera.
capture_manager = None

# Function for parsing program arguments.
def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('--rotation', type=int, choices=[0, 90, 180, 270],
                        dest='rotation', action='store', default=0,
                        help='Rotate everything on the display by this angle')
    args = parser.parse_args()
    return args

last_seen = []
already_seen = []

def main(args):
    global capture_manager, last_seen, already_seen

    # Initialize the capture manager to get stream from Pi Camera.
    if screen.get_width() == screen.get_height() or args.roation in (0, 180):
        capture_manager = PiCameraStream(resolution=(max(320, screen.get_width()), max(240, screen.get_height())), rotation=180, preview=False, format='rgb')
    else:
        capture_manager = PiCameraStream(resolution=(max(240, screen.get_height()), max(320, screen.get_width())), rotation=180, preview=False, format='rgb')

    # Initialize the buffer size to screen size.
    if args.rotation in (0, 180):
        buffer = pygame.Surface((screen.get_width(), screen.get_height()))
    else:
        buffer = pygame.Surface((screen.get_height(), screen.get_width()))

    # Hide the mouse from the screen.
    pygame.mouse.set_visible(False)
    # Initialize the screen to black.
    screen.fill((0,0,0))
    # Try to show the splash image on the screen (if the image exists), otherwise, leave screen black.
    try:
        splash = pygame.image.load(os.path.dirname(sys.argv[0])+'/bchatsplash.bmp')
        splash = pygame.transform.rotate(splash, args.rotation)
        screen.blit(splash, ((screen.get_width() / 2) - (splash.get_width() / 2),
                    (screen.get_height() / 2) - (splash.get_height() / 2)))
    except pygame.error:
        pass
    pygame.display.update()

    # Use the default font.
    smallfont = pygame.font.Font(None, 24)
    medfont = pygame.font.Font(None, 36)

    # Initialize the traffic sign detector and recognizer object with the path
    # to the TensorFlow Lite (tflite) neural network model.
    tsdar0 = TrafficSignDetectorAndRecognizer(os.path.dirname(sys.argv[0])+'/models/uw_tsdar_model_no_aug_w_opts.tflite')

    # Start getting capture from Pi Camera.
    capture_manager.start()

    while not capture_manager.stopped:
        # If the frame wasn't captured successfully, go to the next while iteration
        if capture_manager.frame is None:
            continue

        # Fill the buffer with black color
        buffer.fill((0,0,0))

        # Update the frame.
        rgb_frame = capture_manager.frame

        # Make predictions. If traffic signs were detected, a bounding rectangle
        # will be drawn around them.
        timestamp = time.monotonic()
        predictions, out_frame = tsdar0.predict(rgb_frame)
        delta = time.monotonic() - timestamp
        logging.info(predictions)
        logging.info("TFLite inference took %d ms, %0.1f FPS" % (delta * 1000, 1 / delta))

        # Make an image from a frame.
        previewframe = np.ascontiguousarray(out_frame)
        img = pygame.image.frombuffer(previewframe, capture_manager.camera.resolution, 'RGB')

        # Put the image into buffer.
        buffer.blit(img, (0, 0))

        # Add FPS and temperature on the top corner of the buffer.
        fpstext = "%0.1f FPS" % (1/delta,)
        fpstext_surface = smallfont.render(fpstext, True, (255, 0, 0))
        fpstext_position = (buffer.get_width()-10, 10) # Near the top right corner
        buffer.blit(fpstext_surface, fpstext_surface.get_rect(topright=fpstext_position))
        try:
            temp = int(open("/sys/class/thermal/thermal_zone0/temp").read()) / 1000
            temptext = "%d\N{DEGREE SIGN}C" % temp
            temptext_surface = smallfont.render(temptext, True, (255, 0, 0))
            temptext_position = (buffer.get_width()-10, 30) # near the top right corner
            buffer.blit(temptext_surface, temptext_surface.get_rect(topright=temptext_position))
        except OSError:
            pass

        # Reset the detecttext vertical position.
        dtvp = 0

        # For each traffic sign that is recognized in the current frame (up to 3 signs),
        # its name will be printed on the screen and it will be announced if it already wasn't.
        for i in range(len(predictions)):
            p = predictions[i];
            name = tsdar0.CLASS_NAMES[p]
            print("Detected", name)

            last_seen.append(name)

            # Render sign name on the bottom of the buffer (if multiple signs detected,
            # current sign name is written above the previous sign name).           .
            detecttext = name
            detecttext_font = medfont
            detecttext_color = (255, 0, 0)
            detecttext_surface = detecttext_font.render(detecttext, True, detecttext_color)
            dtvp = buffer.get_height() - (i+1)*(detecttext_font.size(detecttext)[1]) - i*detecttext_font.size(detecttext)[1]//2
            detecttext_position = (buffer.get_width()//2, dtvp)
            buffer.blit(detecttext_surface, detecttext_surface.get_rect(center=detecttext_position))

            # Make an announcement for the traffic sign if it's new (not detected in previous consecutive frames).
            if detecttext not in already_seen:
                os.system('echo %s | festival --tts & ' % detecttext)

        # If new traffic signs were detected in the current frame, add them to already_seen list
        for ts in last_seen:
            if ts not in already_seen:
                already_seen.append(ts)

        # If the traffic sign disappeared from the frame (a car passed it), remove it from already_seen
        diff = list(set(already_seen)-set(last_seen))
        already_seen = [ts for ts in already_seen if ts not in diff]

        # Reset last_seen.
        last_seen = []

        # Show the buffer image on the screen.
        screen.blit(pygame.transform.rotate(buffer, args.rotation), (0,0))
        pygame.display.update()

# Run the program until it's interrupted by key press.
if __name__ == "__main__":
    args = parse_args()
    try:
        main(args)
    except KeyboardInterrupt:
        capture_manager.stop()

Edit: To clarify a bit more, I first followed this tutorial to install my Braincraft HAT and then followed this one to try out the object recognition test example (pitft_labeled_output.py) from rpi-vision. Everything worked great through SSH. I saw logging info in the SSH console and the camera feed and recognized objects on the Braincraftt HAT display. Then I decided to try it out from Windows Remote Desktop (after installing xrdp on RPi) and it worked great. I saw logging info in terminal and camera feed on Braincraft display. But, when I wanted to run my program instead of pitft_labeled_output.py, I received the errors mentioned above. I even went further and replaced pitft_labeled_output.py code with my code(dar_ts.py) and ran my code as if I was running pitft_labeled_output.py (thought that there might be some dependencies inside rpi-vision folder), but it didn't work, received the same error. What could be the issue with my code?

P.S. What also confused me further is that pitft_labeled_output.py has a typo in line 56 and runs fine anyway, but when I ran my code for the first time, it asked me to correct the error. enter image description here

Edit2: Here is a demonstration video of the problem.

3
  • Here's what seems to be the link to the original file: github.com/adafruit/rpi-vision/blob/master/tests/… @A6EE, can you confirm it's the right file? Does it run without errors if you download it? Dec 10 '20 at 14:27
  • @DmitryGrigoryev Yes, that is the original file. I haven't modified it in any way. I'll upload a video showing what exactly is happening.
    – A6EE
    Dec 11 '20 at 23:31
  • @DmitryGrigoryev Added a demonstration video.
    – A6EE
    Dec 12 '20 at 1:58
1
+50

"No protocol specified" is the error you get when you start the X server as user 1 (in your case: "pi") and then try to access it as another user (in your case: "root"). Problem is, I see no dependencies on the X server in your code (you're using the framebuffer), but I don't know pigame and the other libraries well enough to tell you with certainty that there is no such dependency.

I'm also surprised that pitft_labeled_output.py runs with a typo in main(), perhaps you should double-check what you run exactly.

Anyhow, using the X server as root can be possible if you enable pam_xauth on the Pi: make sure it is installed and edit /etc/pam.d/su to add the line

session optional pam_xauth.so

Though I think you should really try to understand why you need that access in the first place.

7
  • Root access is needed so that Python can access the framebuffer of the Braincraft display.
    – A6EE
    Dec 12 '20 at 2:02
  • @A6EE Yes, I understand that. The error seems to stem from the fact that your code apparently tries to use the X server. Dec 12 '20 at 14:01
  • All the things regarding the framebuffer display access are copied from the working example (pitft_labeled_output.py).
    – A6EE
    Dec 12 '20 at 15:03
  • 1
    Found the problem! It wasn't related to the main script dar_ts.py at all, it was a line in one of my other scripts ([code]cv2.imshow(...)[/code]) accidentally left from the program testing phase, so opencv was trying to open the display, but couldn't, since pygame locked it previously. As usual, careful debugging solved the problem. And, regarding the typo in the original rpi-vision example, it probably wasn't detected because the condiion never reached it (screen.get_width() == screen.get_height() is true for the Braincraft HAT display, which is 240x240).
    – A6EE
    Dec 13 '20 at 4:21
  • 1
    @A6EE Glad you got it working! Dec 13 '20 at 18:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.