Unchecked AB Testing Destroys Everything it Touches
Every infuriating thing on the web was once a successful experiment. Some smart person saw Normal site: 1% sign up for our newsletterThrow a huge modal offering 10% off first order: +100% sign ups …
Hasnain says:
Great read on A/B testing and the perils of short term thinking.
“How many executive groups will, when shown an AB test for something like “ask users if we can turn on notifications” showing positive results that will juice revenue short-term, ask “can we test how this plays out long-term?”
As product managers, as designers, as humans who care, it is our responsibility to never, ever present something like that. We need to be careful and think through the long-term implications of changes as part of the initial experiment design and include them in planning the tests.
If we present results of early testing, we need to clearly elucidate both what we do and don’t know”
Posted on 2022-07-14T05:13:37+0000