News Video Summarization - my HYP

Published on November 11, 2007

My HYP - Honours Year Project - is to do a news video summarization system. A typical scenario is a user searching for a news topic from a mobile device. Intelligently, the system will:

  • use NLP to refine the search terms
  • crawls the Internet (eg. google snippets, blogs) for the latest interesting related topics

  • rank and retrieve the most well matched news video from our news corpus

  • perform summarization on the selected news video story, ensuring that the summary is concise, informative, interesting and coherent

  • perform device adaptation such that the video file is playable and fits the screen correctly

The research value comes from the summarization portion, in which a news story of about 1 to 5 minutes has to be shortened even further. But this is a very well needed system as our time is limited. I have always wished that I can have the full 30 min news shortened to just 5 min, but still keeping as much important information that is needed.

It turns out there are so many factors to consider when doing summarization. How do I rank and choose the most informative audio segments? How do I mix video shots from other temporal? How do I ensure that there will be coherence in the audio, video and content? Even at the end of producing a summary, I have to ask - How do I evaluate/grade my summarization techniques?

Video & Audio Processing
To develop the system, I need a lot of video & audio processing. As Java is my preferred language, I have no choice but to use the forsaken JMF (Java Media Framework).

I also used ffmpeg for any other preprocessing work (not within the system).

Some guides:

Some random development notes:

  • On using frame or time as unit? Translate to time finally. Getting frame might not be possible for all.

  • There are so much problems and unknown bugs in JMF when I am just doing transcode and cut & merge videos.

  • One of the problem is the present of white noise between segments of video after I did a cut & merge. I spent 1 whole day on Christmas analyzing when will white noise be present and then I tried on a silly solution which I knew would not be solving the root of the problem. It did not work perfectly. After I came out from a shower, I tried with different audio encoding. Turns out setting the sample rate to 8000 solve the problem!

Libraries Used:

  • DownloadServlet - It is one of the most common seek after servlet. I used it to serve video files outside my container webapps.