up vote 3 down vote favorite

I am working on a project that requires image processing to detect a traffic light and take action based on its status (on or off). I initially developed the code on my PC, using OpenCV and Python, it works just fine.

However when I moved over my Raspberry Pi 3, I was using the Pi Camera module, I followed a tutorial on the PyImageSearch blog by Adrian Rosebrock, who publishes great tutorials, anyways, I was able to get a video stream with decent FPS on my RPi.

However when I moved my main image processing code over to the RPi, I had to modify it to work with a for loop instead of my original while loop. When I ran the code, it gave me an error

(Thresholding Control:1356): Glib-GObject-CRITICAL **: g_object_unref: assertion 'G_IS_OBJECT (object)' failed

But the script still ran anyway, giving me moderately low FPS, and while all this was happening, the CPU usage on the Pi was only 25-28%!!
Considering that the Raspberry Pi 3 has a separate graphics processor, a Quad Core processor @1.2 GHz and enough RAM, I find this very strange.

I looked up google on the error, found nothing. I then found another PyImageSearch tutorial over here. It explains threading to increase the FPS of the for loop pipeline, but I have no idea how to use it in my script.

How can I increase my FPS on the Pi? Is it threading? Or something else altogether?

Here's my code, modified to work on the Raspberry Pi 3 Model B.

from picamera.array import PiRGBArray
from picamera import PiCamera
import time
import cv2 
import numpy as np
def f(x): return
cv2.namedWindow('Thresholding Control')
cv2.namedWindow('Precise Controls')

cv2.createTrackbar('High H','Thresholding Control',179,179, f)
cv2.createTrackbar('Low H','Thresholding Control',0,179, f)
cv2.createTrackbar('High S','Thresholding Control',255,255, f)
cv2.createTrackbar('Low S','Thresholding Control',0,255, f)
cv2.createTrackbar('High V','Thresholding Control',255,255, f)
cv2.createTrackbar('Low V','Thresholding Control',0,255, f)
cv2.createTrackbar('Guassian Blur','Precise Controls',0,99, f)
cv2.createTrackbar('Intensity Threshold','Precise Controls',0,255,f)

camera = PiCamera()
camera.resolution = (640, 480)
camera.framerate = 32
rawCapture = PiRGBArray(camera, size=(640, 480))

time.sleep(0.1)

for frame in camera.capture_continuous(rawCapture, format="bgr",use_video_port=True):

   image = frame.array
   HSV = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
   highH = cv2.getTrackbarPos('High H','Thresholding Control')
   lowH = cv2.getTrackbarPos('Low H','Thresholding Control')
   highS = cv2.getTrackbarPos('High S','Thresholding Control')
   lowS = cv2.getTrackbarPos('Low S','Thresholding Control')
   highV = cv2.getTrackbarPos('High V','Thresholding Control')
   lowV = cv2.getTrackbarPos('Low V','Thresholding Control')
   thresh = cv2.inRange( HSV, (lowH, lowS, lowV), (highH, highS, highV))
   blurVal = cv2.getTrackbarPos('Guassian Blur','Precise Controls')
   intensityVal = cv2.getTrackbarPos('Intensity Threshold','Precise Controls')
   if(blurVal%2==0):
       blurVal=blurVal+1
   thresh_smooth = cv2.GaussianBlur(thresh, (blurVal, blurVal), 0)
   Defining the kernel to be used for Morphological ops.
   kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(5,5))
   # Applying Opening and Closing.
   thresh_smooth = cv2.morphologyEx(thresh_smooth,cv2.MORPH_OPEN,kernel)
   thresh_smooth = cv2.morphologyEx(thresh_smooth, cv2.MORPH_CLOSE, kernel)

   (minVal, maxVal, minLoc, maxLoc) = cv2.minMaxLoc(thresh_smooth)
   nonZero = cv2.countNonZero(thresh_smooth)
   if(maxLoc !=(0,0)): #Light is on
   cv2.putText(image,"Light on", (320,240), cv2.FONT_HERSHEY_SIMPLEX, 2, 255)
   cv2.circle(thresh_smooth, maxLoc, 50, (255, 0, 0), 2)
   cv2.circle(image, maxLoc, 50, (255, 0, 0), 2)

   elif (maxVal<intensityVal and maxLoc!=(0,0)): #light is off     
        cv2.circle(image, maxLoc, 5, (0, 0, 255), 2)
        cv2.putText(image,"Light Off", (320,240), cv2.FONT_HERSHEY_SIMPLEX, 2, 255)

   elif(nonZero==0): #Light is off
       cv2.circle(image, (0,0), 50, (0, 0, 255), 2)
       cv2.putText(image,"Light Off", (320,240), cv2.FONT_HERSHEY_SIMPLEX, 2, 255)

   cv2.imshow("BGR", image)
   cv2.imshow("Thresholded", thresh_smooth)
   print maxLoc 

   rawCapture.truncate(0)

   if cv2.waitKey(1) & 0xFF == ord('q'):
      break

   cv2.destroyAllWindows()

EDIT:
Could this be happening because of insufficient power supply? I mean my usual 5v 2A PSU blew up on day, luckily the Pi survived, I found a mobile charger, 5v 2A and I am using it, but I see the rainbow box, signifying undervoltage coming up repeatedly.

share|improve this question
  • Couple points/questions: 1) How much RAM have you dedicated to the GPU? 2) "the CPU usage on the Pi was only 25-28%" [...] "the Raspberry Pi 3 has a separate graphics processor" -> Hmm. 3) "How can I increase my FPS on the Pi?" I'll be a little snarky and say use C/C++, not python. Kind of a a "have my cake...eat it too" issue, but this is just a casual opinion. 2b) By 25-28%, do you mean ~100% of one core on a quad core system? ;) – goldilocks♦ Jun 9 '16 at 20:30
  • @goldilocks not sure about your last question, I took the reading from the tiny box that shows CPU usage on the top right corner. As for using C/C++, what are the possible performance gains? I might have to write my script again but uh, if it'll work I'm willing to do it. – YaddyVirus Jun 10 '16 at 2:03
  • the box at the top-right on the Raspbian desktop shows overall usage, so you're saturating one core. Not that surprising as you've got a single-threaded script so the single thread is maxing out one core. Threading will certainly help use more CPU, but only if you can parallelize your tasks - and in your script it looks like each stage relies on the prior (mostly). You could potentially parallelize the processing of multiple separate frames, though. – Dave Jones Jun 10 '16 at 12:25
  • On using C/C++ - obviously they're faster languages but most of your script's work is actually already in compiled C/C++ (picamera is a relatively thin wrapper over the MMAL library in C, which itself is a wrapper over the firmware - large amounts of assembler, and OpenCV is largely optimized C++). Where C/C++ would help is in preventing some unnecessary memory allocations and copies (which I can't really avoid in picamera as python needs to "own" certain things like the image array - in lower level languages you can just re-use buffers the camera gives you). – Dave Jones Jun 10 '16 at 12:28
  • @DaveJones what do you mean by "parallelize" the tasks? i mean have a look at the threading tutorial I linked to. In that, a seperate thread is made to do the capturig work while the processing goes on in another thread. And in my script, since I have to take decisions based on what the camera sees, the script is dependant on the prior parts. – YaddyVirus Jun 10 '16 at 14:31
up vote 3 down vote accepted

First off, how are you determining that the CPU is only at 25-28%? htop is what I would use as it shows the usage of each core separately. sudo apt-get install htop, then htop to run it, q to quit. Second, tiny little devices like the Raspberry Pi have much less processing power than what we get used to on our desktops. Your desktop probably has more than enough processing power (3-4 GHz) to run your code decently without threading, whereas the Raspberry Pi (1.2 GHz) may get burdened down without threading your code. Threading will allow your code to dispatch different tasks within your code to different cores, whereas the absence of threading will have your code run on a single-core only. For heavy tasks, threading makes a HUGE difference, so learn how to master it.

share|improve this answer
  • Run a 2nd terminal and have top or htop running, verify that your cpu is only using that much. Also note if it is hitting swap at all. If you are only hitting one core and you hit swap. Time for a code refactor. – Dan V Jun 10 '16 at 1:54
  • Hmmm, relating to the threading tutorial I posted about, how do I actually implement it in my code? – YaddyVirus Jun 10 '16 at 1:57
  • 1
    Modifying code for threading would be way beyond the scope of this SE – Dan V Jun 10 '16 at 4:11
  • 1
    I think the heavy lifting with openCV should be optimally parallelized by the GPU, but if you are maxing out a single CPU core, that is definitely a bottleneck -- the system cannot automatically extend CPU workload to more cores. The best possible deal you are going to get in this context is a multithreaded application written in C/C++ (beware that is not something you are going to do learn over the weekend, lol). Since you are into python, I would look into what can benefit from threading here with that first, it may be sufficient and will be a simpler context in which to learn the concepts. – goldilocks♦ Jun 10 '16 at 11:59
  • 1
    Not that it's within the scope of raspberry.stackexcange, but here's a tutorial on Python threading to get you started. --> tutorialspoint.com/python/python_multithreading.htm – tlhIngan Jun 10 '16 at 14:35
up vote 2 down vote

Have you tried bumping up the GPU mem to your camera?

sudo nano /boot/config.txt

Increase whatever you have for 'gpu_mem'

http://elinux.org/RPiconfig

share|improve this answer
  • Also: github.com/raspberrypi/documentation/blob/master/configuration/… – goldilocks♦ Jun 9 '16 at 20:31
  • I don't have anything for gpu_mem. I don't even have this line in my config.txt . The page you linked to, shows a default value of 128. What value do you recommend? Also, I am confused whether that memory is shared on the sd card or the RAM? – YaddyVirus Jun 10 '16 at 2:07
  • If there's nothing there, it probably just uses the default (64m i think). I'd suggest bumping it up to 256m, but feel free to tweak it. Go ahead and add the line. Alternatively, if you don't want to manually edit the file, run sudo raspi-config and pick Advanced Options/Memory Split. If you want more official doc, here's the doc: link – Kenny Ingle Jun 10 '16 at 19:24
  • @KennyIngle Added the line, modified the script for threading, I saw an improvement in the FPS, I was only displaying the thresholded image on the screen not the original one. The improvement was good but not good enough. I increased the memory to 512 mb, tried 1024 but my Pi stopped booting after that so I had to revert to 512. – YaddyVirus Jun 11 '16 at 16:27

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.