My HYP - Honours Year Project - is to do a news video summarization system. A typical scenario is a user searching for a news topic from a mobile device. Intelligently, the system will:
- use NLP to refine the search terms
crawls the Internet (eg. google snippets, blogs) for the latest interesting related topics
rank and retrieve the most well matched news video from our news corpus
perform summarization on the selected news video story, ensuring that the summary is concise, informative, interesting and coherent
- perform device adaptation such that the video file is playable and fits the screen correctly
The research value comes from the summarization portion, in which a news story of about 1 to 5 minutes has to be shortened even further. But this is a very well needed system as our time is limited. I have always wished that I can have the full 30 min news shortened to just 5 min, but still keeping as much important information that is needed.
It turns out there are so many factors to consider when doing summarization. How do I rank and choose the most informative audio segments? How do I mix video shots from other temporal? How do I ensure that there will be coherence in the audio, video and content? Even at the end of producing a summary, I have to ask - How do I evaluate/grade my summarization techniques?
Video & Audio Processing
To develop the system, I need a lot of video & audio processing. As Java is my preferred language, I have no choice but to use the forsaken JMF (Java Media Framework).
I also used ffmpeg for any other preprocessing work (not within the system).
- JMF 2.1.1 Solutions (samples for the regular tasks)
- Sun’s Forum on JMF
- Video Processing by ExAct Futures
- Transcode FAQ
Some random development notes:
On using frame or time as unit? Translate to time finally. Getting frame might not be possible for all.
There are so much problems and unknown bugs in JMF when I am just doing transcode and cut & merge videos.
One of the problem is the present of white noise between segments of video after I did a cut & merge. I spent 1 whole day on Christmas analyzing when will white noise be present and then I tried on a silly solution which I knew would not be solving the root of the problem. It did not work perfectly. After I came out from a shower, I tried with different audio encoding. Turns out setting the sample rate to 8000 solve the problem!
- DownloadServlet - It is one of the most common seek after servlet. I used it to serve video files outside my container webapps.