Weak performance during motion detection with picamera and python script

Question

To dig into the picamera python api i just started a simple motion detection script. It runs in the videomode with 2 frames/second and a resolution of 320x240. Frames are captured with the capture_continuous function and the frames i work with are PiRGBArrays. Until now: No performance issues... Here is the setup for the camera:

camera = picamera.PiCamera()
camera.resolution = (320,240)
camera.framerate = 2
rawArr = picamera.array.PiRGBArray(camera, size=(320,240))
...
for arr in camera.capture_continuous(rawArr, format='rgb', use_video_port=True):
    curr = rawArr.array
    ...

The basic idea is to compare the last frame with the currently taken frame. I´ve done that in a pretty straight forward way - looping over the three dimensional arrays and compare the values of each color channel with a given tolerance (i.e. 10 per channel).

The performance was very bad: 230400 checks (320x240x3) took about 20 to 30 seconds. Here´s my code:

for row in range(len(lastArray)):
    for col in range(len(lastArray[row])):
        for chan in range(3):
            old = lastArray[row][col][chan]
            new = currArray[row][col][chan]
            if (new > old + self.tolerance or new < old - self.tolerance):
                changeDetected = True

At first i thought, okay there are not less checks and the hardware isnt that powerful - but so bad?! To compare it and get a feeling about how fast array operations can be done, i´ve written a comparable method in Java (where i´m usally at home) just to get a feeling...

public static void main(String[] args) {
    int[][][] threeD = new int[320][240][3];

    System.out.println("start filling array at: " + System.currentTimeMillis());

    for(int i = 0; i < 320; i++){
        for(int j = 0; j < 240; j++){
            for(int k = 0; k < 3; k++)
                threeD[i][j][k] = RandomUtils.nextInt(10);
        }
    }

    System.out.println("start iterating and comparing at: " + System.currentTimeMillis());

    int dummy = 0;
    for(int i = 0; i < 320; i++){
        for(int j = 0; j < 240; j++){
            for(int k = 0; k < 3; k++){
                if(threeD[i][j][k] > 8 || threeD[i][j][k] < 2){
                    dummy = threeD[i][j][k];
                }
            }                   
        }
    }

    System.out.println("finished at: " + System.currentTimeMillis());
}

The output was:

start filling array at: 1416812183728
start iterating and comparing at: 1416812183743
finished at: 1416812183748

So it took about 5ms(!) to do these checks...

I then tried to optimize my code with the following actions

• Use YUV format and compare only the y-channel
• Only compare each 3rd row and column

The updated code goes:

for row in range(len(lastArray)):
    if row % 3 == 0:
        for col in range(len(lastArray[row])):
            if col % 3 == 0:
                old = lastArray[row][col][0]
                new = currArray[row][col][0]
                if (new > old + self.tolerance or new < old - self.tolerance):
                    changeDetected = True

The performance is still not as i expected it to be: About 3 seconds for the comparisons.

Is there a way to make this even faster in pure python?

Thanks in advance!

Answer 1

Pure Python is indeed very very slow (unsurprising given it's an interpreted language). One trick to getting performance out of it (when required, premature optimization being the root of evil) is to push tight loops down to a point where compiled C can deal with them. Numpy provides exactly that with its vectorized operations. For example, to subtract two arrays you can simply subtract the two array objects which will yield a new array object with the same dimensions:

result = array1 - array2

Likewise, there's a numpy.absolute function for calculating the absolute value of each element, and any and all for calculating the truth value of an array (similar to the existential and universal truth quantifiers).

Putting all these together we can simplify the code and speed it up considerably too (numpy is written in C so by pushing all these loops down to numpy causes them to run in compiled C code). We have to be a little bit careful as the numpy arrays picamera returns are unsigned byte arrays (unsurprisingly, although I should add a warning about this to the docs for 1.9) so subtracting them would result in overflow. Hence we need to cast them to a more useful type like signed 16-bit values first:

import numpy as np

def compare1(a, b, threshold=10):
    # Ensure a and b are types which won't overflow on subtraction
    a = a.astype(np.int16)
    b = b.astype(np.int16)
    # Create an array from the absolute difference of a and b
    c = np.abs(a - b)
    # Create an array of truth values indicating whether any
    # absolute differences are greater than the threshold
    c = c > threshold
    # Return whether any values are greater than the threshold
    return c.any()

I've commented the routine above, but for brevity it could simply be written:

import numpy as np

def compare1(a, b, threshold=10):
    return (np.abs(a.astype(np.int16) - b.astype(np.int16)) > threshold).any()

For the sake of comparison I'll place your original code in another function:

def compare2(a, b, threshold=10):
    result = False
    for y in range(len(a)):
        for x in range(len(a[y])):
            for chan in range(3):
                old = a[y][x][chan]
                new = b[y][x][chan]
                if new > old + threshold or new < old - threshold:
                    result = True
    return result

Now we can use a little ipython trick to time both routines (I'm using two arrays filled with zeros here as a worst case comparison, but this isn't too unrealistic as we'd expect most video frames to be largely unchanged and therefore return False). I've placed both routines in a file called comptest.py which I'll load and then use to compare runtimes:

IPython 0.13.1 -- An enhanced interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: %run comptest.py

In [2]: a = np.zeros((240, 320, 3), dtype=np.uint8)

In [3]: b = np.zeros((240, 320, 3), dtype=np.uint8)

In [4]: %timeit compare1(a, b)
10 loops, best of 3: 33.9 ms per loop

In [5]: %timeit compare2(a, b)
1 loops, best of 3: 136 s per loop

Okay, we haven't quite beat Java's time but it's still a significant improvement and the code is still nice and simple, which is always a bonus after optimization! For reference, the above test was run on my development Pi which is overclocked to 900Mhz.