Convert the frame data from recording into a numpy array

Question

I'm trying to take a recording with picamera and have all the resultant frames in an rgb numpy array. To this end I've found the piRGBAnalysis class, but I'm a bit confused about how to use it.

This custom output class is intended to be used with the start_recording() method when it is called with format set to 'rgb' or 'bgr'. While recording is in progress, the write() method converts incoming frame data into a numpy array and calls the stub analyse() method with the resulting array (this deliberately raises NotImplementedError in this class; you must override it in your descendent class).

The part that has me confused is where exactly write() comes into the picture. Here is my code. All I want to see is confirmation that my overridden analyse method has been entered and that presumably it has the rgb array of one of the frames from the recording.

# Inherit from PiRGBAnalysis
class main(picamera.array.PiRGBAnalysis):
    theGodArray = []

    def analyse(array):
        print('here')
        theGodArray = array


with picamera.PiCamera() as camera:
    with picamera.array.PiRGBAnalysis(camera) as output:
        camera.resolution = (256, 256)
        camera.framerate = 30
        camera.start_recording(str(uuid.uuid4())+'.rgb', format='rgb')
        main(output) # This line is I'm sure incorrect. 
        time.sleep(5)
        camera.stop_recording()

Above I guess I assumed that since we have with picamera.array.PiRGBAnalysis(camera) as output in a similar vein to this example on PiRGBArray and since we're using start_recording as required that main(output) will be called for each frame data leading to write() being called and finally my analyse function being called.

This is not the case. How do I make it so that "While recording is in progress, the write() method converts the frame data into a numpy array" ?

Answer 1

Hmm, evidently that portion of the docs isn't clear enough and needs re-writing for the next version (I should stress you're far from the only one confused by the analysis classes - I've had plenty of questions via e-mail about them). Do let me know if the following makes things any clearer (it'll form the basis for the next version's docs):

The analysis classes (PiRGBAnalysis, PiYUVAnalysis, and PiMotionAnalysis) are all examples of custom outputs. That is to say that instances of them are file-like objects and picamera will treat those instances as if they were file objects (i.e. calling their write method with new video data).

Now, in the case of the analysis classes (listed above), you don't have to worry about constructing the write method, that's already done. The write method takes the incoming video data (which is assumed to be in YUV format in the case of PiYUVAnalysis, RGB format in the case of PiRGBAnalysis, etc.) converts it to a numpy array and then calls the analyse method with that array as the only argument.

The analyse method in each case is just a stub which raises NotImplementedError. You're expected to override that method and perform whatever analysis you want on the array within it.

Now, several things to note: the current docs contain a warning about the analyse method having to be fast. That's not true (and was written before I understood how the underlying firmware operated). The analyse method doesn't have to be fast, and nothing will crash if it's not. All that will happen is that buffers won't be returned to the underlying encoder/output port fast enough so it will drop frames, reducing the effective frame-rate, until sufficient buffers are returned for it to proceed. I'll remove that warning in the next version of the docs.

Secondly, you've attempted (correctly) to override the analyse method but your syntax is slightly wrong. Remember that in Python, the self variable (implicitly the first parameter of any bound method call) must be explicitly specified in the method's signature.

Thirdly, you don't want to construct an instance of PiRGBAnalysis directly; it's an abstract class, just there for you to override. You need to construct an instance of your subclass.

Finally, you need to call the start_recording method with that instance of your subclass instead of a filename. The output from the recording is not going to be written to a file; it's going to be written to your analysis instance (you really don't want to record RGB video data anyway - the bandwidth requirements are enormous!).

In other words, what you want will look something like this:

import picamera
import picamera.array

theGodArray = None

# Inherit from PiRGBAnalysis
class MyAnalysisClass(picamera.array.PiRGBAnalysis):
    def analyse(self, array):
        print('here')
        global theGodArray
        theGodArray = array

with picamera.PiCamera() as camera:
    with picamera.array.PiRGBAnalysis(camera) as output:
        camera.resolution = (256, 256)
        camera.framerate = 30
        output = MyAnalysisClass(camera)
        camera.start_recording(output, format='rgb')
        camera.wait_recording(5)
        camera.stop_recording()