Netflix A/B Tests Displaying A Password Field Which Fails And Gets Rejected
It looks like Netflix has been iterating on showing additional fields upfront on their homepage. After they succeeded at displaying an email address upfront, this experiment now takes next step of showing a password field. The result of the leaked experiment however suggests a negative outcome as they reverted back to the control version - without the visible password.
B - Mar 31, 2020 Screenshot
Highlighted UI Changes From This Leak
-
Adding A Password Field
The first and most visible change is the additional password field in the B variation. Judging by the copy, we're going to assume that it's a dynamic sign-in or sign-up interaction with the email address being checked for whether it exists or not. It's important to note that this is a duplicated field as the password field is already shown on further steps in the signup funnel (not visible in these screenshots).
Interestingly, Volders (a contract cancellation service) recently tested something very similar (with complete tests results) where they also removed a password field from their signup funnel.
0.5 Repeatability has been assigned to Pattern #3: Fewer Form Fields as evidence that it's getting better
Repeatability is a net count of evidence for or against a pattern. It’s how we can predict which patterns are better than others. :)
-
Adding An Email Newsletter Opt-out
In the B variation we also see an extra checkbox which acts as an opt out to signing up for email notifications and newsletters. Please also keep in mind that this also is a duplication from a deeper screen on the signup funnel.
It probably have been a cleaner experiment without this change which easily could have dragged the experiment down by suggesting additional and possibly unwanted emails would be sent out.
-
Lowered CTA
Finally, the B variation also pushed down the primary call to action button onto a new line. This is probably a minor change, but an extra confouding variable nevertheless.
This is very similar to Pattern #49: Above The Fold Call To Action
Comments
Sam 4 years ago ↑0↓0
Given that there are 3 big changes, all in the same place, I'm not convinced that "Displaying A Password Field" is what "Fails". All we know is that all these changes, combined, failed.
I didn't even see the CTA button in the B design at first. It's down in the dark area and separated from the form. That's the obvious root cause here. If you don't see a button, you're not going to click it.
There's also changes you omitted. There's a line of useless description text which got moved up in between the headline and the form. Pushing the form (and CTA) down the page doesn't tend to help.
Reply