38 - A Corpus of Natural Language for Visual Reasoning, with Alane Suhr

Motivation

Elaine Sewer shares the motivation behind creating a new dataset for visual reasoning, emphasizing the need to explore linguistic phenomena like quantifiers in a visual context. She explains that existing datasets focused more on scene recognition and object compositionality, whereas this dataset aims to address the compositionality of linguistic elements such as negation and coordination 1. Elaine highlights the challenge in semantic parsing, where the task is to determine the truthfulness of a sentence based on its grounding in a visual environment 2.

The structured representations in the dataset make it easy to execute logical forms and determine the validity of parsed sentences.

--- Elaine Sewer

This approach provides a unique challenge for semantic parsing researchers, as it requires innovative methods to transition from binary supervision to logical form supervision.

   

Complexity

The dataset captures the complexity and compositionality of language by using synthetically generated images paired with human-generated statements. Elaine defends the artificial nature of the dataset, stating that it allows for structured representation and easier data collection, although it may not directly translate to real-world applications like robotics 3. The dataset's design isolates specific linguistic phenomena, enabling researchers to study complex composition with real language rather than synthetic language 3.

This kind of complex reasoning is necessary for applying this work to interfaces like robotics.

--- Elaine Sewer

With nearly 100,000 image-sentence pairs, the dataset serves as a benchmark for testing algorithms on linguistically challenging tasks 4.

   

Construction

The construction of the dataset involved generating images designed to elicit complex language, with each image containing objects of various shapes, colors, and sizes. Elaine describes the process of pairing sentences with images to ensure that sentences are true for some images and false for others, promoting complex compositional language 5. The validation stage involved presenting sentences with images to workers to determine their truthfulness, enhancing data quality and ensuring high agreement among annotators 6.

We wanted to pair each sentence with multiple images to avoid linguistic biases and ensure the sentence's understanding across different contexts.

--- Elaine Sewer

This meticulous construction process ensures that the dataset is robust and capable of supporting advanced linguistic research.