Improving Livestreaming Latency Using Metadata

In live media streaming, latency between media capture at a sender and playback at a receiver is optimized to improve user experience. However, media analysis and editing algorithms (like object/sound/speech recognition) that operate on the stream can introduce delays; thus, media may not be transmittable immediately after capture due to delays introduced by processing and potential modification. This disclosure describes techniques of variable latency streaming, where the playback latency relative to the live edge varies during the playback depending on instructions that are generated based on content analysis of the stream. The instructions can be multiplexed into the live stream as timed metadata samples and demultiplexed by the player application at the receiver. The user can set preferences that dictate whether and how the receiver follows the instructions