In this step, we will learn how to diplay a video from the camera board, using OpenCV display (and not the native preview GPU window).
At the end of this step, you should be able to capture frames from your camera board, and use them directly using OpenCV ! Enjoy, creativity will be your only limit (and perhaps CPU a little bit)

This tuto is based on this file ( Download it and read explanations below. (don’t forget to change the CMakeLists.txt). I found many technicals difficulties to write it,  thanx to Matthieu Tardivon (a brillant student) for his precious hint and help. I appreciate.

We start  from raspivid.c (the camera app)  but we need to remove  all useless lines, not linked with capturing frames.

We  delete
– all lines related to the preview component,
– all lines related to the encoder component.
– all lines related to inline command parsing and picture info…

We change :
– add the callback directly to the video_port  (line 286)
– create and attach the pool (to get/send message)  to the video port… (line 320)
– change format encoding to ENCODING_I420 in line (268) (instead of OPAQUE)

Result : the callback is called with the right FPS (around 30fps/s) during the capture. (FPS without OpenCV treatment).
The Buffer variable contains the raw YUV I420 frame which needs to be converted in a RGB format to be used with OpenCV.
To do it, understand the I420 format : read some cryptic pages like and

I wrote few lines to convert the picture in the callback function (line 141)
– read the buffer and copy it by parts in 3 differents IplImage starting with Y component (full size), continue with U (half size) and finish with V component (half size)
– merge the 3 IplImage (YUV) into one (line 170)
– convert with the right color space (RGB)  (line 171)
– and display it !

Warning ! : cvMerge, cvCvtColor are slow functions. If you want to increase FPS rate, you can stay with gray picture (the first Y channel). You’ll double your FPS doing that.  (parameter graymode=1, line 124). Line 118 set the timeout variable : it’s the period to capture (ms)

  • 320×240 color : FPS = 27,2
  • 320×240 gray : FPS = 28,6
  • 640×480 color : FPS = 8
  • 640×480 gray : FPS = 17

At this stage, you should be able to use your camera board with OpenCV. Frame rate is still not perfect (no HD possible) but it will be enough to play with face recognition with a far better rate than our old USB webcam ! That’s what we’ll see on step 7.

Enjoy !