The discussion delves into the transformative potential of transformers compared to traditional convolutional networks in visual representation. By treating images as sequences of tokens, transformers reduce inductive biases, allowing for more powerful representations, albeit requiring larger datasets. The conversation also highlights the current state of transformers in computer vision, questioning whether they are merely demonstrating feasibility or achieving state-of-the-art results.
