Natural Language Descriptions for Video Streams

This thesis is concerned with the automatic generation of natural language descriptions
that can be used for video indexing, retrieval and summarization applications.
It is a step ahead of keyword based tagging as it captures relations between keywords
associated with videos, thus clarifying the context between them. Initially,
we prepare hand annotations consisting of descriptions for video segments crafted
from a TREC Video dataset. Analysis of this data presents insights into humans
interests on video contents. For machine generated descriptions, conventional image
processing techniques are applied to extract high level features (HLFs) from
individual video frames. Natural language description is then produced based on
these HLFs. Although feature extraction processes are erroneous at various levels,
approaches are explored to put them together for producing coherent descriptions.
For scalability purpose, application of framework to several different video genres
is also discussed. For complete video sequences, a scheme to generate coherent and
compact descriptions for video streams is presented which makes use of spatial and
temporal relations between HLFs and individual frames respectively. Calculating
overlap between machine generated and human annotated descriptions concludes
that machine generated descriptions capture context information and are in accordance
with human’s watching capabilities. Further, a task based evaluation shows
improvement in video identification task as compared to keywords alone. Finally,
application of generated natural language descriptions, for video scene classification
is discussed.