Sewon Min: The Science of Natural Language

Dense Retrieval

Dense Passage Retrieval (DPR) is a method that enhances question-answering tasks by accurately retrieving relevant text passages. Sewon Min explains that the challenge lies in defining similarity, which in this context means not only relevance to the question but also containing a specific answer 1. The complexity of storing all meanings of a passage into a low-dimensional vector is significant, and technical details are crucial for success. Daniel Bashir notes that improvements in maximum inner product search algorithms and text encoders can enhance DPR's efficiency 1.

It's not an easy problem. I think it really depends on what you mean by similarity.

--- Sewon Min

Sewon suggests that maximizing the union of information across passages, rather than focusing on individual passage relevance, could be a more effective approach.

   

Non-Parametric Models

Non-parametric language models offer a novel approach to integrating and retrieving knowledge, addressing issues like model hallucination and privacy. Sewon introduces non-parametric masked language modeling, which fills in blanks using retrieval alone, ensuring that model predictions are based on retrieved data 2. This method improves the prediction of rare entities and provides explanations by showing the context of retrieved phrases.

The model will figure out the context surrounding the entity, and then if that context seems relevant, then it will return the entity as a prediction.

--- Sewon Min

Verification remains a critical challenge, as language models can produce plausible but unverified results. Sewon emphasizes the importance of verification, especially in sensitive domains like medicine and politics 3.