SE-Radio Episode 360: Pete Koomen on A/B Testing

Tool Selection

Selecting the right tools for A/B testing is crucial for effective implementation. Pete Koomen explains that while some companies may opt to build their own tools, it often requires significant effort and resources. Instead, using existing solutions like Optimizely or other open-source tools can be more efficient, especially for organizations new to A/B testing 1. He emphasizes the importance of creating meaningful experiments that can demonstrate value and build momentum within the organization. This approach helps in gaining internal support for larger experimentation programs 2.

The important thing is you find meaningful experiments to run that end up setting a great example for the rest of your organization.

--- Pete Koomen

Koomen also highlights the advantage of using a visual editor to design experiments without altering website code, simplifying the process for non-technical teams 2.

   

Technical Challenges

Implementing A/B testing involves several technical challenges, particularly in deciding between client-side and server-side execution. Pete Koomen notes that client-side testing can be quicker to implement but may impact performance due to increased data transfer and DOM manipulation 3. Server-side testing, while reducing latency, often requires more effort and coordination within the organization. He advises separating feature configuration from business logic to facilitate easier experimentation 4.

It's a question of velocity versus latency and needing to make a decision in the way that's best for your users, your organization.

--- Pete Koomen

Koomen also discusses the use of SDKs to manage experiments across different platforms, allowing for flexible architecture decisions that suit specific organizational needs 4.