uPlayer: enhancing the video playback experience

Audiovisual content analysis using artificial intelligence makes it possible to automate numerous processes across the multimedia production and management chain. Current computer vision techniques recognise people, places, scenes, objects and logos, while audio analysis enables speech transcription, speaker identification and keyword detection. By combining these results with natural language processing, it becomes possible to generate summaries, classify content and extract useful metadata for indexing, search, automatic documentation, monitoring, advertising optimisation and support for content editing.

uPlayer uses this metadata to improve the viewer’s experience. For example, in a news programme, the system automatically identifies news items, people, sections and relevant concepts, enabling much more intuitive non-linear navigation than the traditional 10-second forward/back system: users can jump between headlines, filter by sections or search for appearances of specific individuals. This experience can be extended to other formats such as entertainment programmes, panel discussions or series.

To make this possible, the provider hosts its content on a video-on-demand server (YouTube, Vimeo or its own server), which is automatically analysed and integrated with a player plugin, through which the user receives the metadata that enriches playback, without the broadcaster having to modify its workflows.

In addition, users can add tags, create navigation points (favourite scenes), generate short clips and share them on social networks, as well as collaborate with other users on databases about specific characters. These metadata can be shared or published as navigation profiles. Creators, such as YouTubers, can also request automatic analysis and customise the results before distributing their content.

uPlayer is typically presented as a browser plugin (for example, Chrome) that integrates with existing video players. The system can be commercialised through analysis subscriptions, offering advanced navigation to users or providing producers with information on consumption and interaction patterns. For certain types of content, such as courses or conferences, it also enables collaborative enrichment by the community.

The AgroTech audiovisual technologies group at UPC has contributed expertise in facial recognition systems and audio and video content analysis. It has also been responsible for carrying out system quality measurements tailored to different types of content such as documentaries, interviews, classes, news programmes and series, proposing analysis strategies according to content type. Direct collaboration has taken place with Ugiat Technologies, which developed and adapted the algorithms and integrated them into the system.

Budget and funding

The project ran for 12 months (2024) and had a total budget of € 20,000.