How can I redirect input from a TV to my Pi, then send output back to the TV for automatic captioning?

Question

I've been searching around the subject for a while, and haven't found any substantial track yet.

My aim is to build a Speech-to-Text converter that would be used to generate and display live captions on the fly on a tv, the exact way Youtube close-captioning works.
I know that I would likely have to embed the code on a powerful board, like a Raspberry Pi, and I also know that Google has released its Automatic Speech Recognition API so it could potentially be used so as to handle the speech-to-text conversion part of the project.

However, the part that still seems rather unclear is how the input sound will be redirected from the tv to the processing component (the Pi for example), and how the Pi will be able to display the output generated captions on the tv ?

PS: As the matter might be tagged as too broad, I want to clarify that I'm not asking for a in-depth/detailed answer or a step-by-step method to carry it out, but rather some guidelines and basic advices as to which direction should be followed, which technology is most suitable, etc ...

Answer 1

As noted on Super User, the Pi itself doesn't have a HDMI input port, but there are various products which will allow you to add a HDMI input (either from a USB adapter or through a dedicated HAT like the PiCapture HD1). Be aware that, as noted in the linked question, a USB port is unlikely to have the throughput to carry a HD video/audio input.

You would need to add HDMI input functionality (or another multimedia input method), then stream the input into your captioning program in order to process the speech. Note that speech recognition is not trivial for a computer, and I have my doubts whether the Pi would be able to do it in real-time. Google's Streaming Speech Recognition API may well be useful, but this adds network latency in exchange for not using the (slow) Pi processing power. It is perhaps worth testing this yourself to see before committing any financial resources.

Once you have your input stream, and have run the captioning program, you would end up with a string representing your caption. You would need to take the video input, add a box with the text in, then send that back to the HDMI output.

There do not appear to be many products on the market that do this, probably because speech recognition engines struggle under all but the most favourable conditions; AccessibleTech.org suggest that the technology is simply not reliable enough:

The best-case scenario would be fully automated captioning using speech recognition technology. Unfortunately, current technology is not accurate enough to fully support this approach

If you've ever tried to use YouTube's captioning systems, you would probably be inclined to agree.