Speech processing on the Raspberry Pi

Question

I want to do speech processing on Raspberry Pi to detect specific people (something like unique identification).

I would prefer to use only the on-board processor for this, you could assume that internet is not accessible.

Also, what are the limitations with the Raspberry Pi performing speech processing? If I want to use this as a automatic attendance machine, how should I proceed?

Answer 1

This is the main project my Raspberry Pi is dedicated to right now, so I figure I can add my two cents. Keep in mind this project is still very much a work in progress.

I chose to use the C programming language for this project exclusively on the Raspbian OS, and that may have affected some of my decisions and instructions. I'm going to only list free and open source software, since that is all that I use.

For the installation instructions, I will assume you have a fully up-to-date system.

---

Speech Recognition

Here are some options for speech recognition engines:

1. Pocketsphinx - A version of Sphinx that can be used in embedded systems (e.g., based on an ARM processor).

   • Pros: Under active development and incorporates features such as fixed-point arithmetic and efficient algorithms for GMM computation. All the processing takes place on the Raspberry Pi, so it is capable of being used offline. It supports real time speech recognition

   • Cons: It is complicated to set up and understand for beginners. For me, it was too inaccurate for my application. All the processing takes place on the Raspberry Pi, making it a bit slower.

   • Installation instructions:

     1. Download the latest stable versions of Sphinxbase and Pocketsphinx:

        $ wget http://sourceforge.net/projects/cmusphinx/files/sphinxbase/0.8/sphinxbase-0.8.tar.gz
        $ wget http://sourceforge.net/projects/cmusphinx/files/pocketsphinx/0.8/pocketsphinx-0.8.tar.gz
        

     2. Extract the downloaded files:

        $ tar -zxvf pocketsphinx-0.8.tar.gz; rm -rf pocketsphinx-0.8.tar.gz
        $ tar -zxvf sphinxbase-0.8.tar.gz; rm -rf sphinxbase-0.8.tar.gz
        

     3. To compile these packages, you'll need to install bison and the ALSA development headers.

        NOTE: It is important that the ALSA headers be installed before you build Sphinxbase. Otherwise, Sphinxbase will not use ALSA. It also appears that ALSA will not be used if PulseAudio is installed (a bad thing for developers like me).

        $ sudo apt-get install bison libasound2-dev
        

     4. cd into the Sphinxbase directory and type the following commands:

        $ ./configure --enable-fixed
        $ sudo make
        $ sudo make install
        

     5. cd into the Pocketsphinx directory and type the following commands:

        $ ./configure
        $ sudo make
        $ sudo make install
        

     6. Test out Pocketsphinx by running:

        $ src/programs/pocketsphinx_continuous -samprate 48000 
        

        If you want to tweak it, I recommend you read some information on the CMUSphinx Wiki.

2. libsprec - A speech recognition library that is developed by H2CO3 (with few contributions by myself, mostly bug fixes).

   • Pros: It uses the Google Speech API, making it more accurate. The code is more easy to understand (in my opinion).

   • Cons: It has dependencies on other libraries that H2CO3 has developed (such as libjsonz). Development is spotty. It uses the Google Speech API, meaning processing doesn't take place on the Raspberry Pi itself, and requires an internet connection. It requires one small modification to the source code before compilation to work properly on the Raspberry Pi.

   • Installation instructions:

     1. Install libflac, libogg and libcurl:

        $ sudo apt-get install libcurl4-openssl-dev libogg-dev libflac-dev
        

     2. Download the most recent version of libsprec

        $ wget https://github.com/H2CO3/libsprec/archive/master.zip
        

     3. Unzip the downloaded package:

        $ unzip master.zip; rm -rf master.zip
        

        You should now have a folder named libsprec-master in your current directory.

     4. Download the most recent version of libjsonz:

        $ wget https://github.com/H2CO3/libjsonz/archive/master.zip
        

     5. Unzip the downloaded package:

        $ unzip master.zip; rm -rf master.zip
        

        You should now have a folder named libjsonz-master in your current directory.

     6. cd into the libjsonz-master directory, compile, and install:

        $ cd libjsonz-master
        $ mv Makefile.linux Makefile
        $ make
        $ sudo make install
        

     7. cd out of the libjsonz-master directory and into the libsprec-master/src directory. Edit line 227:

        $ err = snd_pcm_open(&handle, "pulse", SND_PCM_STREAM_CAPTURE, 0);
        

        We need this to say:

        $ err = snd_pcm_open(&handle, "plughw:1,0", SND_PCM_STREAM_CAPTURE, 0);
        

        This is so that the program will use ALSA to point to the USB microphone.

     8. Compile and install:

        $ mv Makefile.linux Makefile
        $ make
        $ sudo make install
        

     9. You can now use the library in your own applications. Look in the example folder in libsprec-master for examples.

3. Julius - A high-performance, two-pass large vocabulary continuous speech recognition (LVCSR) decoder software for speech-related researchers and developers.

   • Pros: It can perform almost real-time speech recognition on the Raspberry Pi itself. Standard speech model formats are adopted to cope with other free modeling toolkits.

   • Cons: Spotty development, with it's last update being over a year ago. It's recognition is also too inaccurate and slow for my usage. Long installation time

   • Installation instructions:

     1. There are a few packages that we need to install to get the system working properly:

        $ sudo apt-get install alsa-tools alsa-oss flex zlib1g-dev libc-bin libc-dev-bin python-pexpect libasound2 libasound2-dev cvs
        

     2. Download Julius from the CVS source:

        $ cvs -z3 -d:pserver:[email protected]:/cvsroot/julius co julius4
        

     3. Set the compiler flags by the environment variables:

        $ export CFLAGS="-O2 -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard -pipe -fomit-frame-pointer"
        

     4. cd into the folder julius4 and type the following commands

        $ ./configure --with-mictype=alsa
        $ sudo make
        $ sudo make install
        

     5. Julius needs an environment variable called ALSADEV to tell it which device to use for a microphone:

        $ export ALSADEV="plughw:1,0"
        

     6. Download a free acoustic model for Julius to use. Once you have downloaded it, cd into the directory and run:

        $ julius -input mic -C julius.jconf
        

        After that you should be able to begin speech input.

4. Roll your own library - For my specific project, I choose to build my own speech recognition library that records audio from a USB microphone using ALSA via PortAudio, stores it in a FLAC file via libsndfile, and sends it off to Google for them to process it. They then send me a nicely packed JSON file that I then process to get what I said to my Raspberry Pi.

   • Pros: I control everything (which I like). I learn a lot (which I like).

   • Cons: It's a lot of work . Also, some people may argue that I'm not actually doing any processing on the Raspberry Pi with this speech recognition library. I know that. Google can process my data much more accurately that I can right now. I'm working on building an accurate offline speech recognition option.

---

Speech Synthesis

Here are some options for speech synthesis engines:

1. tritium - A free, premium quality speech synthesis engine written completely in C (and developed by yours truly).

   • Pros: Extremely portable (no dependencies besides CMake to build), extremely small (smallest one that I could find), easy to build.

   • Cons: The speech output itself can be inaccurate at times. The support for a wide variety of languages is lacking as I am the sole developer right now with little free time, but this is one of the future goals of the project. Also, as of right now only a library is output when compiled and no usable/testable executable.

2. eSpeak - A compact open source software speech synthesizer for Linux, Windows, and other platforms.

   • Pros: It uses a formant synthesis method, providing many spoken languages in a small size. It is also very accurate and easy to understand. I originally used this in my project, but because of the cons I had to switch to another speech synthesis engine.

   • Cons: It has some strange dependencies on X11, causing it to sometimes stutter. The library is also considerably large compared to others.

   • Installation instructions:

     1. Install the eSpeak software:

        $ sudo apt-get install espaek
        

     2. To say what you want in eSpeak:

        $ espeak "Hello world"
        

        To read from a file in eSpeak:

        $ espeak -f <file>
        

3. Festival - A general multi-lingual speech synthesis system.

   • Pros: It is designed to support multiple spoken languages. It can use the Festvox project which aims to make the building of new synthetic voices more systematic and better documented, making it possible for anyone to build a new voice.

   • Cons: It is written in C++ (more of a con to me specifically). It also has a larger code base, so it would be hard for me to understand and port the code.

   • Installation instructions:

     1. Install the Festival software:

        $ sudo apt-get install festival festival-freebsoft-utils
        

     2. To run Festival, pipe it the text or file you want it to read:

        $ echo  "Hello world" | festival --tts
        

4. Flite - A small run-time speech synthesis engine derived from Festival and the Festvox project.

   • Pros: Under constant development at Carnegie Mellon University. Very small engine compared to others. It also has a smaller code base, so it is easier to go through. It has almost no dependencies (a huge pro for me, and another reason I decided to use this engine in my project).

   • Cons: The speech output itself is not always accurate. The speech has a very metallic, non-human sound (more than the other engines). It doesn't support very many languages.

   • Installation instructions:

     1. Install the Flite software:

        $ sudo apt-get install flite
        

     2. To run Flite:

        $ flite -t "text that you want flite to say"
        

---

Answers to your specific questions:

What are the limitations with the Pi performing speech processing?

Programmers don't have limits. :P

On a more serious note, the Raspberry Pi has plenty of resources to handle speech processing. As long as the person performing the speech processing knows what they are doing, the Raspberry Pi should be able to handle it just fine.

I want to use this a automatic attendance machine, how should I proceed?

None of these option are accurate enough to tell the difference between specific people yet. That is something I am working on in my project (and probably will be for a while). If you are looking for an better option for automatic attendance, I would look into facial recognition. There are more limitations on facial recognition for the Raspberry Pi though, so keep that in mind.

Answer 2

I went with pocketsphinx_continuous and a $4 sound card.

To manage the fact that it needs to stop listening when using speech synth I used amixer to handle to input volume to the mic (this was recommended best practice by CMU as stop-starting engine will result in poorer recognition)

echo "SETTING MIC IN TO 15 (94%)" >> ./audio.log
amixer -c 1 set Mic 15 unmute 2>&1 >/dev/null 

With a matching command to mute the listening when the speech synth plays

FILE: mute.sh
#!/bin/sh

sleep $1;
amixer -c 1 set Mic 0 unmute >/dev/null 2>&1 ; 
echo  "** MIC OFF **" >> /home/pi/PIXIE/audio.log

To calculate the right times to mute for I just run soxi via lua and then set the unmute.sh (opposite of the mute.sh) to run "x" seconds from the startup. There are no doubt lots of ways to handle this. I am happy with the results of this method.

LUA SNIPPET:

-- Begin parallel timing  
-- MUTE UNTIL THE SOUNDCARD FREES UP 
-- "filename" is a fully qualified path to a wav file 
-- outputted by voice synth in previous operation

-- GET THE LENGTH
local sample_length = io.popen('soxi -D '..filename);
local total_length  = sample_length:read("*a"); 
clean_length = string.gsub(total_length, "\n", "") +1;  
sample_length:close();

-- EXAMPLE LOGGING OUTPUT...
--os.execute( 'echo LENGTH WAS "'.. clean_length .. '" Seconds  >> ./audio.log');   



-- we are about to play something... 
-- MUTE, then schedule UNMUTE.sh in x seconds, then play synth output
-- (have unrolled mute.sh here for clarity)

os.execute( 'amixer -c 1 set Mic '..mic_level..' unmute 2>&1 >/dev/null ');
os.execute( 'echo "** MIC OFF **"  >> ./audio.log ');

-- EXAMPLE LOGGING OUTPUT...    
-- os.execute( 'echo PLAYING: "'.. filename..'" circa ' .. clean_length .. ' Seconds  >> ./audio.log ');

os.execute( './unmute.sh "'.. clean_length ..'" &');


-- THEN PLAY THE THING WHILE THE OTHER PROCESS IS SLEEPING  

os.execute( './sounds-uncached.sh '..filename..' 21000')

To actually grab the voice on the pi I use:

pocketsphinx_continuous -bestpath 0 -adcdev plughw:1  -samprate 20000  \
-nfft 512 -ds2 -topn2 -maxwpf 5 -kdtreefn 3000 -kdmaxdepth 7 -kdmaxbbi 15 \
-pl_window 10 -lm ./LANGUAGE/0892-min.lm -dict ./LANGUAGE/0892-min.dic 2>&1 \
| tee -i 2>/dev/null >( sed -u -n -e 's/^.\{9\}: //p' ) \
>( sed -u -n -e 's/^READY//p' \
-e 's/^Listening//p' -e 's/^FATAL_ERROR: \"continuous\.c\"\, //p') \
> /dev/null

Again, there are other ways, but I like my output this way.

For the synth I used Cepstrals fledgling pi solution, but it's not available online you have to contact them directly to arrange to buy it and it is around $30 to buy. The results are acceptable however the speech does create some nasty clicks and pops, the company have replied saying they no longer have a RaspPi and are unwilling to improve the product. YMMV

The voice recognition sits at around 12% CPU when "idle", and spikes briefly when doing a chunk of recognition.

The voice creation spikes at about 50-80% when rendering.

The play / sox weighs in pretty heavily but I do apply real-time effects to the rendered voices as I play them ;)

The pi is heavily stripped down using every guide I could find to stop un-required services and runs in complete CLI mode. 800mhz over-clocked (smallest).

scaling_governor set to: performance

When fully running: it runs at about 50ºC in direct sunlight and 38ºC in the shade. I have heat sinks fitted.

Last point: I actually run all this gear out to "internet driven" AI as a nice extra.

The pi handles all this seamlessly, And playing out any networked audio in real-time, And fully looped audio to any other Unix box. etc.

to handle the large speech CPU overhead burden I have implemented an md5sum based caching system so the same utterences are not rendered twice. (about 1000 files @ 220 mb total covers 70% of the utterences I generally get back from the AI) this really helps bring the total CPU load down overall.

In précis this is all totally doable. however the voice recognition will only be as good as the quality of your mics, your language model, how specifically close your subjects voices are to the original intended audience (I use a en_US model on en_UK children, not perfect) and other minutia of detail that with effort you can whittle down to a decent result.

And for the record, I already did all this once before on a kindle (and that worked too with cmu sphinx and flite). Hope this helps.

Answer 3

Yes. use PocketSphinx for speech recognition, Festvox for text to speech (TTS) and some USB audio with line in (or an old supported webcam which also has line in).

Google searches for these software packages and "Raspberry Pi" provide many examples and tutorials to set this up.

Answer 4

• SiriProxy - Only use this if you have a device that uses Siri - you don't need to jailbreak anything. It basically intercepts Siri on the network you install it on.
• Speech2Text - You can use Googles API to decode speech to text but the example contains some other methods too.
• Julius - A speech recognition decoder.

As pointed out by Lenik, you will need someway to record audio or possibly send audio files to the Raspberry Pi for them to get decoded somehow.

Answer 5

Raspberry Pi has no built-in ADC nor microphone input. Unless you're planning to use external USB mike, there's basically no way to get your audio stream to the device. Besides that, there are no serious limitations, the CPU is powerful enough for any sound processing you might try to implement.

Answer 6

Firstly, you should select a set of words for the classification process. After that you should collect the data from users/subjects. It will be nonstationary signal. You have to reduce your data to reduce computational costs/to improve success ratio with feature extraction methods so you should look for suitable feature extraction methods for your application. You can get a feature vector as a result of these methods (mean absolute value, RMS, waveform length, zero crossing, integrated absolute value, AR coefficients, median frequency, mean frequency etc). Then, you should use a classification method like knn, neural networks etc. to classify your data. Lastly you have to check its accuracy. To sum up:

1. Select a set of words/sentences.
2. Get the data from the human subjects.
3. Preprocess (maybe the signal needs to be filtered)
4. Feature extraction/Processing.
5. Classification.
6. Tests.

I have seen video processing projects with RPi on the internet so it can manage to do this classification.

You can use NI 6009 USB DAQ (which supports RPi) for collecting any analog data but they are little bit expensive.

Answer 7

This may be useful for you for recognising speaker:

https://code.google.com/p/voiceid/