Sync Audio & Video From Pi Camera & USB Microphone

Question

I am trying to stream low-quality audio & video from my Raspberry Pi to another device. I have a Raspberry Pi camera & a USB microphone. Both devices work separately but i cannot seem to get the audio & video to sync up.

I am running Raspbian. I began by compiling ffmpeg as described at http://blog.giuseppeurso.net/raspberry-pi-streaming-webcam/.

Here is my ffmpeg info:

ffmpeg version 1.0.10 Copyright (c) 2000-2014 the FFmpeg developers
  built on Oct 27 2014 00:14:25 with gcc 4.6 (Debian 4.6.3-14+rpi1)
  configuration: --enable-filter=movie --enable-filter=aresample --enable-avfilter

Before i get streaming working, i want to just dump the audio & video to an MKV file. I wrote this script to dump audio & video to out.mkv until terminated.

#!/bin/bash

vpipe=/tmp/videopipe
apipe=/tmp/audiopipe
vw=320
vh=240
fps=15

trap "rm -r $vpipe" EXIT
trap "rm -r $apipe" EXIT

if [[ ! -p $vpipe ]]; then
    mkfifo $vpipe
fi

if [[ ! -p $apipe ]]; then
    mkfifo $apipe
fi

raspivid -t 0 -w $vw -h $vh -fps $fps -o - > $vpipe &
arecord -D plughw:1,0 -r 16000 > $apipe &

ffmpeg \
    -y \
    -fflags nobuffer \
    -r $fps \
    -i $vpipe \
    -fflags nobuffer \
    -analyzeduration 0 \
    -i $apipe \
    -map 0:0 \
    -map 1:0 \
    -filter:a aresample=async=1 \
    -c:a copy \
    -c:v copy \
    out.mkv

The audio & video are grossly out of sync. I have tried changing the framerates on both raspivid & ffmpeg with no success. Not sure what piece of the puzzle i am missing.

Answer 1

The problem is that the raspberry pi camera module does not, in fact, produce 25 frames per second. Instead, the frequency determined by the oscillator within is more like 24.8 Hz, which, at increasing video length, produces unexpected surplus of frames.

The current solution is to consider the frame rate of video input as 25.375 FPS (or roughly so), so the ffmpeg command becomes:

ffmpeg [...] -r 25.375 -i $vpipe [...] -r 25 -i $apipe [...] -r 25 out.mkv

Note it is perhaps wiser to have all the processes in a single pipe (raspivid | ffmpeg) and ffmpeg read ALSA as well (-f alsa -i hw:1,0).