TREETALK: Composition and Compression of Trees for Image Descriptions

We present a new tree based approach to composing expressive image descriptions that makes use of naturally occuring web images with captions. We investigate two related tasks: image caption generalization and gen-eration, where the former is an optional sub-task of the latter. The high-level idea of our approach is to harvest expressive phrases (as tree fragments) from existing image descrip-tions, then to compose a new description by selectively combining the extracted (and op-tionally pruned) tree fragments. Key algo-rithmic components are tree composition and compression, both integrating tree structure with sequence structure. Our proposed system attains significantly better performance than previous approaches for both image caption generalization and generation. In addition, our work is the first to show the empirical ben-efit of automatically generalized captions for composing natural image descriptions.