OpenCV&Pi Cam – Step 3 : create your own project

As an example, we will create camcv, a soft strongly inspired from raspistill. It will allow us later to modify source code and play with OpenCV.

1. create a new folder in your home directory and copy all raspicam apps source code.

cd
mkdir camcv
cd camcv
cp /opt/vc/userland/host_applications/linux/apps/raspicam/*  .
mv RaspiStill.c camcv.c
Perhaps a chmod is necessary (sudo chmod 777 camcv.c)

2. remove then content of CMakeLists.txt and replace with :
cmake_minimum_required(VERSION 2.8)
project( camcv )
SET(COMPILE_DEFINITIONS -Werror)
include_directories(/opt/vc/userland/host_applications/linux/libs/bcm_host/include)
include_directories(/opt/vc/userland/interface/vcos)
include_directories(/opt/vc/userland)
include_directories(/opt/vc/userland/interface/vcos/pthreads)
include_directories(/opt/vc/userland/interface/vmcs_host/linux)
add_executable(camcv RaspiCamControl.c RaspiCLI.c RaspiPreview.c camcv.c)
target_link_libraries(camcv /opt/vc/lib/libmmal_core.so /opt/vc/lib/libmmal_util.so /opt/vc/lib/libmmal_vc_client.so /opt/vc/lib/libvcos.so /opt/vc/lib/libbcm_host.so )

3. delete CMakeFiles directory if it exits
4. Compile & test
cmake .
make 
./camcv -t 1000

5. [optional] Clean file. camcv.c is a long file with a lot of useless lines for us. All these following function could be delete (or commented). Remove of course all call to these functions in the remaining code.
static void dump_status(RASPISTILL_STATE *state)
static int parse_cmdline(int argc, const char **argv, RASPISTILL_STATE *state)
static void display_valid_parameters()
static MMAL_STATUS_T add_exif_tag(RASPISTILL_STATE *state, const char *exif_tag)
static void add_exif_tags(RASPISTILL_STATE *state)
static void store_exif_tag(RASPISTILL_STATE *state, const char *exif_tag)

The existing line default_status(&state); avoid you to parse command line using defaults parameters. Just add a line after this one : 
state.filename=”foobar.jpg”;

Recompile and retest. Should work.
At this stage you should watch an old good movie with John Wayne.

OpenCV&Pi Cam – Step 4 : link with OpenCV

Of course, OpenCV must be already installed in your Pi. To do this, just follow the third step of my previous post “Magic Mirror”

1. Modify your CMakeFiles.txt to include OpenCV library cmake_minimum_required(VERSION 2.8)
project( camcv )
SET(COMPILE_DEFINITIONS -Werror)
#OPENCV
find_package( OpenCV REQUIRED )

#except if you’re pierre, change the folder where you installed libfacerec
#optional, only if you want to go till step 6 : face recognition
link_directories( /home/pi/pierre/libfacerec-0.04 ) 

include_directories(/opt/vc/userland/host_applications/linux/libs/bcm_host/include)
include_directories(/opt/vc/userland/interface/vcos)
include_directories(/opt/vc/userland)
include_directories(/opt/vc/userland/interface/vcos/pthreads)
include_directories(/opt/vc/userland/interface/vmcs_host/linux)
add_executable(camcv RaspiCamControl.c RaspiCLI.c RaspiPreview.c camcv.c)
target_link_libraries(camcv /opt/vc/lib/libmmal_core.so /opt/vc/lib/libmmal_util.so /opt/vc/lib/libmmal_vc_client.so /opt/vc/lib/libvcos.so /opt/vc/lib/libbcm_host.so /home/pi/pierre/libfacerec-0.04/libopencv_facerec.a ${OpenCV_LIBS})

2. Recompile. Should be ok. No change (of course!) since you didn’t modify your source code
make
./camcv 

Actually, this was a pretty easy step !

wall-e holds the camera

OpenCV&Pi Cam – Step 5 : basic use (display a picture)

In this step, we will modify our camcv code to

• remove the preview display provided by MMAL layer
• copy the camera buffer to a CvMat object
• link CvMat to a IplImage object and display it
• do some cleaning to remove all useless code (for us)

1. Download the camcv.c file here and note following comments/change : http://raufast.org/download/camcv.c

Lines 61+ : add OpenCV Includes
// *** PR : ADDED for OPENCV
#include <cv.h>
#include <highgui.h>

Line 156 : modify init values for test (size of file)
// *** PR : modif for demo purpose : smaller image
state->timeout = 1000; // 5s delay before take image
state->width = 320;//2592;
state->height = 200; //1944;

Line 230+ : in static void encoder_buffer_callback function. This is the core of the modification. This function is a callback, call to get the image in the queue. buffer contains the picture from the camera.

// *** PR : OPEN CV Stuff here !
// create a CvMat empty structure, with size of the buffer.
CvMat* buf = cvCreateMat(1,buffer->length,CV_8UC1);

// copy buffer from cam to CvMat
buf->data.ptr = buffer->data;

// decode image (interpret jpg)
IplImage *img = cvDecodeImage(buf, CV_LOAD_IMAGE_COLOR);

// we can save it !
cvSaveImage(“foobar.bmp”, img,0);
// or display it
cvNamedWindow(“camcvWin”, CV_WINDOW_AUTOSIZE);
cvShowImage(“camcvWin”, img );
cvWaitKey(0);

Line 711/726/823 : we remove the native preview window (replaced by opencv window)
// *** PR : we don’t want preview
camera_preview_port = NULL;

// PR : we don’t want preview
// status = connect_ports(camera_preview_port, preview_input_port, &state.preview_connection);

// mmal_connection_destroy(state.preview_connection);

2. compile, run and check if your “foobar.bmp” file is created and if a nice window shows your picture taken by Pi Cam ! (press key to stop)

picture taken with Pi cam and displayed with opencv !

At this stage you should be very enthousiastic ! You can control your rpi camera with OpenCV ! Beautiful are bits, trees and the Pi.

OpenCV&Pi Cam – Step 5.5 : basic use (camera mode)

Well, I’ve tried to use the same code as step 5 (based on RaspiStill.c) and I’ve just added a loop to take many pictures as quick as possible (it’s called a movie isn’t-it ?). Result is not too bad, but we can surely do better starting with RaspiVid.c (but another story, next step).

This way is quite simple  and I get around 8 FPS (still better than my USB-cam and its 2-3 FPS).
Compared to step 5, only linors changes : take a look on main function (around line 743) and in encoder_buffer_callback (around line 235).
File is here : http://raufast.org/download/camcv2.c

Note : filename changed, update your CMakeLists.txt file ! 😉

OpenCV&Pi Cam – Step 6 : Video !

In this step, we will learn how to diplay a video from the camera board, using OpenCV display (and not the native preview GPU window).
At the end of this step, you should be able to capture frames from your camera board, and use them directly using OpenCV ! Enjoy, creativity will be your only limit (and perhaps CPU a little bit)

This tuto is based on this file (http://raufast.org/download/camcv_vid0.c). Download it and read explanations below. (don’t forget to change the CMakeLists.txt). I found many technicals difficulties to write it,  thanx to Matthieu Tardivon (a brillant student) for his precious hint and help. I appreciate.

We start  from raspivid.c (the camera app)  but we need to remove  all useless lines, not linked with capturing frames.

We  delete
– all lines related to the preview component,
– all lines related to the encoder component.
– all lines related to inline command parsing and picture info…

We change :
– add the callback directly to the video_port  (line 286)
– create and attach the pool (to get/send message)  to the video port… (line 320)
– change format encoding to ENCODING_I420 in line (268) (instead of OPAQUE)

Result : the callback is called with the right FPS (around 30fps/s) during the capture. (FPS without OpenCV treatment).
The Buffer variable contains the raw YUV I420 frame which needs to be converted in a RGB format to be used with OpenCV.
To do it, understand the I420 format : read some cryptic pages like  http://en.wikipedia.org/wiki/YUV and http://www.fourcc.org/yuv.php

I wrote few lines to convert the picture in the callback function (line 141)
– read the buffer and copy it by parts in 3 differents IplImage starting with Y component (full size), continue with U (half size) and finish with V component (half size)
– merge the 3 IplImage (YUV) into one (line 170)
– convert with the right color space (RGB)  (line 171)
– and display it !

Warning ! : cvMerge, cvCvtColor are slow functions. If you want to increase FPS rate, you can stay with gray picture (the first Y channel). You’ll double your FPS doing that.  (parameter graymode=1, line 124). Line 118 set the timeout variable : it’s the period to capture (ms)

• 320×240 color : FPS = 27,2
• 320×240 gray : FPS = 28,6
• 640×480 color : FPS = 8
• 640×480 gray : FPS = 17

At this stage, you should be able to use your camera board with OpenCV. Frame rate is still not perfect (no HD possible) but it will be enough to play with face recognition with a far better rate than our old USB webcam ! That’s what we’ll see on step 7.

Enjoy !

Magic Mirror – Presentation

This blog explains how to build a “Magic Mirror”. A Mirror, powered by a Raspberry Pi which recognizes people and talk with them !

Look at this video

Step 1 – Hardware

If you want to build your own magic mirror, you will need :

• a Raspberry Pi Model B, 512 Mb with a nice white plastic case
• an USB keyboard
• an USB mouse
• a screen
• an USB webcam. I used Logitech HD Webcam C270 (720p, but works fine only at 320×240 resolution).
• I strongly recommend a powered usb switch : RPI internal current is limited to 700mA. Strange behavior appears when I connected 3 USB devices (using internal keyboard usb port), like key repetition.
• a SDHC card. Mine is a Lexar 8Gb but 4Gb is enough.
• powered external computer speaker (3.5 mm connector)
• an ethernet cable (for installation)
• a nice mirror (best if rectangle, to support webcam on its top)(best if wooden mirror to do like “White Snow” 😉

At this stage, tell your girlfriend you’re going to disappear for a while, but that 42 is still the answer. So, she must not panic.

Step 2 – Install RPI softwares (OS+tools)

1. Download and install Rasbian

I use “Raspbian wheezy”, an optimized version of Debian for your RPI. I will not describe here how to download it and format your SD card. There are already too many websites describing these simple operations. (for example here to download and here to install with MacOsX). This step is quite easy, but take around 20 minutes to write your 8 Gb SD card. Don’t panic 🙂

2. Network configuration

Same I will not describe it. Depending on your network configuration (with cable, with wifi, direct connexion to your internet box or via a pc-router, and so on), find on google the right way to configure your network.

I simply connected my RPI to my MacBook Pro using a short ethernet cable. Configuration is made using this instruction. It works pretty fine. After this step, try to connect to inernet using the browser or with a cmd line “ping google.fr“.

3. Share directories

It’s a good idea to share directory between your computer and your RPI. It’ll allow you to do backup, edit file quicker and transfer files (like face pictures).

Since I’m using a mac, I installed Netatalk (it works first time. I just reboot my RPI and my Mac saw it).

sudo apt-get update
sudo apt-get install netatalk

4. Change your background image

Well, of course this step is not mandatory, but you’ll spend a lot of time in front of your RPI, thus, put a nice picture is always better for motivation. I picked a nice Stormtrooper wallpaper

5. Do a backup

At this stage, you have a good and working-well Raspberry Pi. It’s time to do your first backup.

On a mac or a linux box, insert your SD card and identify its disk number (using df – h). In example below, my sdcard is rdisk1. Warning, betwwen if (input) and of (output), otherwise you will crash your card. Same, if you don’t use the right disk id, you will crash/backup your mac !

sudo dd bs=1m if=/dev/rdisk1 of=/Users/pierre/rpi/backup/backup1.img

At this stage, may the force be with you.

Step 3 – Install softwares (for webcam and computer vision)

1. Install webcam software

Install guvcview webcam viewer to test if your webcam is working.

sudo apt-get update
sudo apt-get install guvcview

Start guvcview to check if your webcam is working fine. If not, change resolution thru GUI (320×240 should be fine). If you still got problem, it could be a permission/rights issue. Just type :

sudo usermod -a -G video pi
sudo modprobe uvcvideo

Don’t be surprised : your RPI is not a very powerful machine. You’ll be able to display pictures at 10 fps max.

2. Install CMake

CMake is mandatory to compile.

sudo apt-get update
sudo apt-get install cmake

3. Install OpenCV Lib for face recognition

OpenCV is a wonderful library to do computer vision. Please take time to read information on the official website or on tutorials pages.

Today (April, 2013), only OpenCV 2.3 is available for RPI. Unfortunatly, face reco API is only available on version 2.4. Thus, we will need to install OpenCV in two steps.

3.1 Install OpenCV 2.3

Install both dev lib and python lib. My soft is C-written. Anyway, Python is still usefull for small scripts. I recommend to install it.

sudo apt-get update
sudo apt-get install libopencv-dev
sudo apt-get install python-opencv

To test if OpenCv library is well installed, write this test software. It just displays a picture using imread and imshow functions. You will need to provide a sample .jpg file.

To compile using OpenCv lib, create a CMakeLists.txt file with

cmake_minimum_required(VERSION 2.8)
project( displayimage )
find_package( OpenCV REQUIRED )
add_executable( displayimage display_image.cpp )
target_link_libraries( displayimage ${OpenCV_LIBS} )

Then compile and execute

cmake .
make
./displayimage

If it works, congratulations, OpenCV 2.3 is installed.

3.2 install face recognition API

The face recognition API is called libfacerec-0.04. All information and doc can be found on this excellent website.

Download the zip file here. https://github.com/bytefish/libfacerec/zipball/v0.04

I unzip it on my mac and transfer the whole directory on my rpi.

Go on the directory and just compile it using

cmake .
make

Now, if you want to compile your previous sample with this libfacerec-004 api, you will need to modify your CMakeLists.txt file. It just link with the libface lib. Note :  Replace /home/pi/pierre/ path by the path where you copied the libface directory.

cmake_minimum_required(VERSION 2.8)
project( reco)
find_package( OpenCV REQUIRED )
add_executable( displayimage display_image.cpp )
link_directories( /home/pi/pierre/libfacerec-0.04 )
target_link_libraries( displayimage /home/pi/pierre/libfacerec-0.04/libopencv_facerec.a ${OpenCV_LIBS} )

4. Fix Timeout Problem

After few seconds/minutes using your webcam, you should have a “select timeout error” message. This is linked to some limitation of the Pi. Mr Gomoto gives here a way to avoid this issue. Thanks to him ! Do a simple bash file timeout.sh with this code.

#/bin/bash
sudo rmmod uvcvideo
sudo modprobe uvcvideo nodrop=1 timeout=5000 quirks=0x80

5. Do a backup

If everything is ok, it’s worth to do a backup. Follow Step 2.5 instructions.

At this stage, do a short prayer for Dennis Ritchie. And read its book (Kernighan/Ritchie C Handbook) you will need it for step 4.

Step 4 – Understand Face Detect and Face Recognition

1. Read and Read again

I really want to thank Philipp Wagner. A great Open Source contributor. Please, visit his webpage.
He wrote a lot of usefull article about Face Recognition using OpenCV. Actually, he wrote the libfacerec library we installed in step 3.2. All my code are based from his opencv sample.

Here are some webpage you need to read to understand better how to detect faces and recognize people :

• Face Recognition using OpenCV (general)
• Face Recognition using OpenCV (theory + source code)
• Understand how to detect face using Cascade Classifier (theory + source code)
• A very clear tutorial : Face Recognition in Videos with OpenCV
• A very clear tutorial written by Shervin Emami : Introduction to Face Detection and Face Recognition 

These reading are perhaps redundants. But they give you all theory and source exemple to master face reco with OpenCV.

2. Train

You’ve read everything ? Fine. Now copy examples, compile and see what happens.
It’s really one of the spirit of Raspberry Pi community to test by yourself and developp your own software.
Now, you’ll be able to compile C++ samples on your raspberry and see your face, detect your face. Recognition is a little bit more difficult to test since it requires training-pictures. It’s the goal of step 5.

At this stage, you should thank Eben Upton to be the co-founder of Raspberry Pi Foundation and Liz Upton to animate so well the Raspberry Pi community. Thank to their works, you can smile to your webcam playing with your raspberry pi !