Change the voice of your magic mirror

 

 

 

espeak works fine but voice is creepy (especially in french !). This post explains how to use the google voice api to get a wonderful female clear voice.

first, create speech.sh. It contains a function “say()” which call translate.google.com with parameter $* (parameter of your shell)

#!/bin/bash
say() { local IFS=+;/usr/bin/mplayer -ao alsa -really-quiet -noconsolecontrols “http://translate.google.com/translate_tts?hl=en&sl=en&tl=fr&ie=UTF-8&oe=UTF-8&multires=1&otf=1&ssel=3&tsel=3&sc=1&text=$*” 2>/dev/null;}
say $*

Notice that this url will speak french. If you want to speak english, just change “tl=fr” by “tl=en” (german: tl=de, spanish: tl=es, etc.)

Change attribute of your sh file to allow execution and try it. Notice that this URL only accepts sentence less than 100 letters.

chmod u+x ./speech.sh
./speech.sh “Bonjour. Je veux manger du fromage.”

Now, you can use it in your C file.

void speak(char *sLine)
{
    char sCmd[255];
    sprintf(sCmd,”./speech.sh %s”,sLine);
    // if (TRACE) printf(“[i] say : %s”,sLine);
    system(sCmd);
}

To read a whole text (length > 100 letters)

char sToBeRead[100];
f = fopen(fileName,”r”);
if (f != NULL)
{
    // read all text
     while (!feof(f))
     {
        if (fgets(sToBeRead, 100, f)!=NULL)
        {
           speak(sToBeRead);
        }
     }
    fclose(f);
   return 1;
  }
return 0;
}

At this end of this post, your wife/girlfriend should not like the new voice of your magic mirror… 😉

Magic Mirror – Presentation

This blog explains how to build a “Magic Mirror”. A Mirror, powered by a Raspberry Pi which recognizes people and talk with them !

Look at this video

Step 1 – Hardware

If you want to build your own magic mirror, you will need :

• a Raspberry Pi Model B, 512 Mb with a nice white plastic case
• an USB keyboard
• an USB mouse
• a screen
• an USB webcam. I used Logitech HD Webcam C270 (720p, but works fine only at 320×240 resolution).
• I strongly recommend a powered usb switch : RPI internal current is limited to 700mA. Strange behavior appears when I connected 3 USB devices (using internal keyboard usb port), like key repetition.
• a SDHC card. Mine is a Lexar 8Gb but 4Gb is enough.
• powered external computer speaker (3.5 mm connector)
• an ethernet cable (for installation)
• a nice mirror (best if rectangle, to support webcam on its top)(best if wooden mirror to do like “White Snow” 😉

At this stage, tell your girlfriend you’re going to disappear for a while, but that 42 is still the answer. So, she must not panic.

Step 2 – Install RPI softwares (OS+tools)

1. Download and install Rasbian

I use “Raspbian wheezy”, an optimized version of Debian for your RPI. I will not describe here how to download it and format your SD card. There are already too many websites describing these simple operations. (for example here to download and here to install with MacOsX). This step is quite easy, but take around 20 minutes to write your 8 Gb SD card. Don’t panic 🙂

2. Network configuration

Same I will not describe it. Depending on your network configuration (with cable, with wifi, direct connexion to your internet box or via a pc-router, and so on), find on google the right way to configure your network.

I simply connected my RPI to my MacBook Pro using a short ethernet cable. Configuration is made using this instruction. It works pretty fine. After this step, try to connect to inernet using the browser or with a cmd line “ping google.fr“.

3. Share directories

It’s a good idea to share directory between your computer and your RPI. It’ll allow you to do backup, edit file quicker and transfer files (like face pictures).

Since I’m using a mac, I installed Netatalk (it works first time. I just reboot my RPI and my Mac saw it).

sudo apt-get update
sudo apt-get install netatalk

4. Change your background image

Well, of course this step is not mandatory, but you’ll spend a lot of time in front of your RPI, thus, put a nice picture is always better for motivation. I picked a nice Stormtrooper wallpaper

5. Do a backup

At this stage, you have a good and working-well Raspberry Pi. It’s time to do your first backup.

On a mac or a linux box, insert your SD card and identify its disk number (using df – h). In example below, my sdcard is rdisk1. Warning, betwwen if (input) and of (output), otherwise you will crash your card. Same, if you don’t use the right disk id, you will crash/backup your mac !

sudo dd bs=1m if=/dev/rdisk1 of=/Users/pierre/rpi/backup/backup1.img

At this stage, may the force be with you.

Step 3 – Install softwares (for webcam and computer vision)

1. Install webcam software

Install guvcview webcam viewer to test if your webcam is working.

sudo apt-get update
sudo apt-get install guvcview

Start guvcview to check if your webcam is working fine. If not, change resolution thru GUI (320×240 should be fine). If you still got problem, it could be a permission/rights issue. Just type :

sudo usermod -a -G video pi
sudo modprobe uvcvideo

Don’t be surprised : your RPI is not a very powerful machine. You’ll be able to display pictures at 10 fps max.

2. Install CMake

CMake is mandatory to compile.

sudo apt-get update
sudo apt-get install cmake

3. Install OpenCV Lib for face recognition

OpenCV is a wonderful library to do computer vision. Please take time to read information on the official website or on tutorials pages.

Today (April, 2013), only OpenCV 2.3 is available for RPI. Unfortunatly, face reco API is only available on version 2.4. Thus, we will need to install OpenCV in two steps.

3.1 Install OpenCV 2.3

Install both dev lib and python lib. My soft is C-written. Anyway, Python is still usefull for small scripts. I recommend to install it.

sudo apt-get update
sudo apt-get install libopencv-dev
sudo apt-get install python-opencv

To test if OpenCv library is well installed, write this test software. It just displays a picture using imread and imshow functions. You will need to provide a sample .jpg file.

To compile using OpenCv lib, create a CMakeLists.txt file with

cmake_minimum_required(VERSION 2.8)
project( displayimage )
find_package( OpenCV REQUIRED )
add_executable( displayimage display_image.cpp )
target_link_libraries( displayimage ${OpenCV_LIBS} )

Then compile and execute

cmake .
make
./displayimage

If it works, congratulations, OpenCV 2.3 is installed.

3.2 install face recognition API

The face recognition API is called libfacerec-0.04. All information and doc can be found on this excellent website.

Download the zip file here. https://github.com/bytefish/libfacerec/zipball/v0.04

I unzip it on my mac and transfer the whole directory on my rpi.

Go on the directory and just compile it using

cmake .
make

Now, if you want to compile your previous sample with this libfacerec-004 api, you will need to modify your CMakeLists.txt file. It just link with the libface lib. Note :  Replace /home/pi/pierre/ path by the path where you copied the libface directory.

cmake_minimum_required(VERSION 2.8)
project( reco)
find_package( OpenCV REQUIRED )
add_executable( displayimage display_image.cpp )
link_directories( /home/pi/pierre/libfacerec-0.04 )
target_link_libraries( displayimage /home/pi/pierre/libfacerec-0.04/libopencv_facerec.a ${OpenCV_LIBS} )

4. Fix Timeout Problem

After few seconds/minutes using your webcam, you should have a “select timeout error” message. This is linked to some limitation of the Pi. Mr Gomoto gives here a way to avoid this issue. Thanks to him ! Do a simple bash file timeout.sh with this code.

#/bin/bash
sudo rmmod uvcvideo
sudo modprobe uvcvideo nodrop=1 timeout=5000 quirks=0x80

5. Do a backup

If everything is ok, it’s worth to do a backup. Follow Step 2.5 instructions.

At this stage, do a short prayer for Dennis Ritchie. And read its book (Kernighan/Ritchie C Handbook) you will need it for step 4.

Step 4 – Understand Face Detect and Face Recognition

1. Read and Read again

I really want to thank Philipp Wagner. A great Open Source contributor. Please, visit his webpage.
He wrote a lot of usefull article about Face Recognition using OpenCV. Actually, he wrote the libfacerec library we installed in step 3.2. All my code are based from his opencv sample.

Here are some webpage you need to read to understand better how to detect faces and recognize people :

• Face Recognition using OpenCV (general)
• Face Recognition using OpenCV (theory + source code)
• Understand how to detect face using Cascade Classifier (theory + source code)
• A very clear tutorial : Face Recognition in Videos with OpenCV
• A very clear tutorial written by Shervin Emami : Introduction to Face Detection and Face Recognition 

These reading are perhaps redundants. But they give you all theory and source exemple to master face reco with OpenCV.

2. Train

You’ve read everything ? Fine. Now copy examples, compile and see what happens.
It’s really one of the spirit of Raspberry Pi community to test by yourself and developp your own software.
Now, you’ll be able to compile C++ samples on your raspberry and see your face, detect your face. Recognition is a little bit more difficult to test since it requires training-pictures. It’s the goal of step 5.

At this stage, you should thank Eben Upton to be the co-founder of Raspberry Pi Foundation and Liz Upton to animate so well the Raspberry Pi community. Thank to their works, you can smile to your webcam playing with your raspberry pi !

Step 5 – Prepare photos

1. Understand collection of portraits need

For best results, OpenCV-train function need a collection of portraits with these properties :

• picture shows face
• picture is gray-scaled
• all pictures have same size
• eyes are horizontally-aligned
• at least 30 pictures per people (my own experience)

See Han Solo’s picture as a good exemple :

You could create all this pictures with Gimp… but it’ll be a painful job.
To do it for you, I wrote a C++ software, strongly inspired from Philip Wagner Python script (here). This software, called prepare do, for each jpg file found in current directory :

• resize the picture (for performance issue : your Pi is not powerfull enough to work with 5M pictures) (target dimension is a parameter, I recommend 800px width)
• transform to gray scale
• equalize colors (optional, parameter)
• detect face
• detect eyes (with glasses or not)
• rotate picture to align eyes
• crop picture around the face (size is an input parameter)
• create a new picture sizexsize pixels (size is a parameter)
• save new picture using a prefix to name it (prefix-index.jpg)
• Warning : only put in input photos with only 1 person. Anyway, only the first person face will be cropped.

./prepare 0.3 100 hs 800 1

• 0.3 means 30% = I take 30% more than the distance eye-eye.
• 100 means : my output picture will be 100×100 pixels
• hs means :my ouput picture name will start by hs (hs1.jpg, hs2.jpg, hs3.jpg…)
• 800 means : original input pictures will be resized to 800 pixels width (ok for RPI performance)
• 1 means : do the color histogram equalization (0 = don’t)

With these parameters and with the Han Solo’s picture below, you ‘ll get the gray small face displayed before.

2. Download source code

Download .cpp source code here. http://raufast.org/download/preparePhoto.cpp

3. Compile

CMakeLists.txt should contain :

cmake_minimum_required(VERSION 2.8)
project( prepare )
find_package( OpenCV REQUIRED )
add_executable( prepare preparePhoto.cpp )
target_link_libraries( prepare ${OpenCV_LIBS} )

Then, compile :
cmake
make .

And run within a directory full of pictures !
Warning, this soft doesn’t recognize all faces. Especially if original face is tilting too much. My recommandation is to provide a lot of input pictures and see how many you get at the end…
If everything is ok, you should have dozens of nice 100×100 portrait. Perfect for your face recognition training, next step.

At this stage, you should do a break and read The Man Who Planted Trees (Jean Giono).

Step 6 – Face Recognition !

1. Let your Pi talk !
Now, your Pi must be able to speak. A very easy way to do it is to install espeak. (espeak is enough now, you don’t need to install Mbrola any more).
sudo apt-get update
sudo apt-get install espeak

Test it using a simple command line
espeak “hello world.”
Voice by default is english. If you want to change, select your voice using -v flag.
List of languages is here.
espeak -vfr “coucou lapin.” will talk with french accent.
For same language, you can select alternative voices using +m1 (for male voice, 1..4) or +f1 (for female voice, 1..4).
Speed is also a parameter : -s130 (by default : 100).
I recommend you to play with all these parameters to find a not-too-much-creepy voice 😉
If you want to read content of a file, just use -f option.
espeak -f “to_read.txt” -vfr+f2 -s130

2. understand source code
Once again, this source code is strongly inspired from Philipp Wagner source (here).
Download the source code here. I added a lot of comments to be more pedagogic as possible.
Algorithm is quite simple :

• read config files with training pictures previously created in step 6 : store them in a picture collection
• init some parameters, create a Eigen model (could be a Fisher model also)
• train the Eigen model with collection of pictures
• init webcam and start an infinite loop
  • capture photo
  • detect faces
  • for each face
    • try to recognize people leanrt
    • if success (confidence > threshold)
  • put name in the output image
  • speak to this person
  • display the output image

All funny parts are inside the speakTo function. Here you can add as many smarts behaviors as you want. It’ll let think your mirror is really a magic one.
Some exemples :

• say time
• depending hour of the day, propose important things to do : propose a morning coffee, remind to wash teeth before to go to sleep, remind to call his mother all sunday, ..)
• connect to internet to get weather, stocks exhange, google calender next meeting, (see step 7 of this blog for code exemples)
• adapt your speak to personn recognize (basic rules: tell a woman she’s beautiful, and tell a man he’s strong).

For such “smart agent” : the Wii balance board is a very good exemple : when you connect, your Wii always tell you short, funny and very-well adapted things…

3. compile
This step should not be difficult if you use this CMakeLists.txt. (notice the libopencv linked as already seen in step 3)

cmake_minimum_required(VERSION 2.8)
project( reco)
find_package( OpenCV REQUIRED )
add_executable( reco faceReco.cpp )
link_directories( /home/pi/pierre/libfacerec-0.04 )
target_link_libraries( reco /home/pi/pierre/libfacerec-0.04/libopencv_facerec.a ${OpenCV_LIBS} )

4. Test & Enjoy
Now, it’s time to test and to enjoy !
./reco csv-filename histo threshold
./reco faces.csv 1 5500

• csv-filename : filename of csv file where you put all your training pictures path
• histo : 0/1 : do you want to do a color equalization histogram before detecting face ? (try both and decide)
• threshold : level of confidence above wich, you decide to recognize somebody (need to be tune after first run : use 5000 by default for Eigen Model (less for Fischer model)

Note : Light sensibility will depend on your webcam. With my logitech C270, it works well with natural light and artificial light.
Play with paramater (including training picture creation) and find your best set of parameter.

5. Install your Magic Mirror !
Select a wooden home-made mirror, put the webcam on its top.
Put your Pi and speakers behind the mirror to hide all technical stuffs.

At this stage, you should have a magic mirror, which is a perfect geek useless stuff. Start to impress your girlfriend, your step-mother or the pizza deliveryman.

Step 7 – Smart Mirror Behavior

1. Gran Torino syndrom

In the video, the mirror reminds me my next meeting. This smart behavior is done connecting our software to google calendar thru API.
Frankly, I don’t understand why Google didn’t make C++ API to connect to Google Calendar. I’m feeling like Clint Eastwood in Gran Torino, with my old good gcc compiler, alone and abandoned by all.

2. Using Libgcal

Hopefully, there is Ligcal, a C library to handle google calendar and contacts. I don’t know if it’s officially supported.
sudo apt-get update
sudo apt-get install libgcal0
sudo apt-get install libgcal-dev

3. Testing Libgcal

This small source code (download here) connects to google calendar (thru login/pwd given as arguments or harcoded), gets all yours events/meetings and selects the next one.
gcc -lgcal testcal.cpp -o cal

Frankly, it doesn’t work each time. Curiously, this code works randomly. It get connected but sometimes, it doesn’t return list of events :-(. If you find why, please let me know.

Once tested, copy this exemple to your mirror source code, create a dedicated function and call it from the speakTo function.

At this stage, you should be able to have a smart mirror which reminds you your next meeting. Create a new event “do a backup” and let your mirror do its job…