A framework for multimedia playback and analysis of MPEG-2 videos with FFmpeg

Fast Forward Motion Pictures Expert Group (FFmpeg) is a well-known, high performance, cross platform open source library for recording, streaming, and playback of video and audio in various formats, namely, Motion Pictures Expert Group (MPEG), H.264, Audio Video Interleave (AVI), just to name a few. With FFmpeg current licensing options, it is also suitable for both open source and commercial software development. FFmpeg contains over 100 open source codecs for video encoding and decoding. Given the complexities of MPEG standards, FFmpeg still lacks a framework for (1) seeking to a particular image frame in a video, which is needed for accurate annotation at the frame level for applications in fields such as medical domain, digital communications and commercial video broadcasting and (2) motion vectors extraction for analysis of motion patterns in video content. Most importantly, FFmpeg code base is not well documented, which has raised a significant difficulty for developing an extension. As our contributions, we extended FFmpeg code base to include new APIs and libraries support accurate frame-level seek, motion vector extraction, and MPEG-2 video encoding/decoding. We documented FFmpeg MPEG-2 codec to facilitate future software development. We evaluated the performance of our implementation against a high-performance third-party commercial software development kit on videos captured from television broadcasts and from endoscopy procedures. To evaluate the usability of our libraries, we integrated them with some commercial applications. In the following sections, we will discuss our software architecture, important implementation details, performance evaluation results, and lessons learned