Published Dec 21, 2023

Interviewing Tri Dao and Michael Poli of Together AI on the future of LLM architectures

Delve into the transformative future of AI as Tri Dao and Michael Poli from Together AI discuss the success of transformers, the resurgence of RNNs, and the potential of state space models. Gain insights into the continual evolution and innovation of AI architectures, examining the balance between hardware optimization and large-scale application efficiency.

Episode Highlights

Topics covered

Episode Highlights

Related Episodes

Interviewing Louis Castricato of Synth Labs and Eleuther AI on RLHF, Gemini Drama, DPO, founding Carper AI, preference data, reward models, and everything in between
Answers 383 questions
Interviewing Sebastian Raschka on the state of open LLMs, Llama 3.1, and AI education
Answers 383 questions
Interviewing Dean Ball on AI policy
Answers 383 questions
Interviewing Ross Taylor on LLM reasoning, Llama fine-tuning, Galactica, agents
Answers 383 questions
Text-to-video AI is already abundant
Answers 383 questions
Multimodal blogging: My AI tools to expand your audience
Answers 383 questions
Interconnects year in review: 2023
Answers 383 questions
A realistic path to robotic foundation models
Answers 383 questions
Multimodal LM roundup: Unified IO 2, inputs and outputs, Gemini, LLaVA-RLHF, and RLHF questions
Answers 383 questions
Alignment-as-a-Service: Scale AI vs. the new guys
Answers 383 questions
Phi 3 and Arctic: Outlier LMs are hints
Answers 383 questions
AI for the rest of us
Answers 383 questions
Interviewing Riley Goodside on the science of prompting
Answers 383 questions
Mixtral: The best open model, MoE trade-offs, release lessons, Mistral raises $400mil, Google's loss, vibes vs marketing
Answers 383 questions
How to cultivate a high-signal AI feed
Answers 383 questions

Dexa/Interconnects Audio

Interviewing Tri Dao and Michael Poli of Together AI on the future of LLM architectures

Topics covered

Popular Clips

Language Model Insights

Data and Applications

State Space Breakthrough

Hybridizing Model Architectures

Transformer Design Insights

Future Architectural Innovations

Future of AI Architectures

Transformer Success Recipe

Evolution of RNNs

Conversational Insights

Efficient State Management

Episode Highlights

Transformer Models

RNN Evolution

State Space Models

Future Innovations

Related Episodes

Interviewing Louis Castricato of Synth Labs and Eleuther AI on RLHF, Gemini Drama, DPO, founding Carper AI, preference data, reward models, and everything in between

Interviewing Sebastian Raschka on the state of open LLMs, Llama 3.1, and AI education

Interviewing Dean Ball on AI policy

Interviewing Ross Taylor on LLM reasoning, Llama fine-tuning, Galactica, agents

Text-to-video AI is already abundant

Multimodal blogging: My AI tools to expand your audience

Interconnects year in review: 2023

A realistic path to robotic foundation models

Multimodal LM roundup: Unified IO 2, inputs and outputs, Gemini, LLaVA-RLHF, and RLHF questions

Alignment-as-a-Service: Scale AI vs. the new guys

Phi 3 and Arctic: Outlier LMs are hints

AI for the rest of us

Interviewing Riley Goodside on the science of prompting

Mixtral: The best open model, MoE trade-offs, release lessons, Mistral raises $400mil, Google's loss, vibes vs marketing

How to cultivate a high-signal AI feed

Interviewing Tri Dao and Michael Poli of Together AI on the future of LLM architectures

Topics covered

Popular Clips

Episode Highlights

Transformer ModelsTri Dao, Chief Scientist at Together AI, and Michael Poli, a PhD student at Stanford, discuss the factors behind the success of transformers and the intricacies of the attention mechanism.

Transformer Models

RNN EvolutionNext, Tri Dao and Michael Poli explore the evolution and renewed potential of Recurrent Neural Networks (RNNs). They discuss both the foundational aspects and recent advancements that are bringing RNNs back into the spotlight.

RNN Evolution

State Space ModelsTri Dao and Michael Poli discuss the advancements and efficiency of state space models compared to transformers. They explore the hardware optimizations that make these models more practical for large-scale applications.

State Space Models

Future InnovationsThe discussion explores the future of AI architectures, focusing on the enduring dominance of transformers and the potential for new innovations. Nathan Lambert, Tri Dao, and Michael Poli share insights on emerging trends and the evolving landscape of AI models.

Future Innovations

Related Episodes