Electrolysis/Experiments
Electrolysis Experiments
This page intends to document the A/B experiments that were done during the development of e10s.
Experiment mechanisms
Telemetry Experiments
The first experiments were done through the Telemetry Experiments system, which is an A/B mechanism meant to test new features to telemetry-enabled users. The experiments were published in the production server for this mechanism.
Staged Rollout - System add-on
With the approach of releasing e10s to the release audience, it was necessary to have a more detailed mechanism specific to e10s in order to be able to do a staged rollout of the feature and quickly increase/decrease the rollout rate. To do that, the new System Add-on mechanism was chosen, because it can be updated out-of-band from the release process.
After the system add-on was developed, we started using it to run new phases of e10s experiments on Beta, in order to also test the add-on itself in preparation for Release.
List of experiments
Note: all dates are an approximation from the end-user's point of view, because they may take a couple of days to be updated to the build where the experiment was started/terminated.
Beta 48
Single phase
Start: beta 1 (built on Jun 6, shipped Jun 8)
End: still running
Range: [beta 1, ...)
Sample: 50% of eligible users (yield ~20%)
Details:
- APZ is enabled by default (conditional to e10s) starting in 47
- No add-ons
- No accessibility on Mac and Windows
- No accessibility on Linux starting on build 2 (bug 1277882)
- No Mac OS X 10.6 - 10.8
- GTK+ 3.20+ users reincluded
- No WinXP+graphics acceleration users (this caused a 3.9% drop on eligible users compared from the Beta 47 experiment)
- Windows XP users should be excluded from release criteria analyses (e10s on WinXP disabled on the release channel by bug 1275039)
Related bugs: Tracking bug, automatically started (no bug), unblock GTK+ 3.20+, block on OSX 10.6-10.8, block on release WinXP, block on linux a11y, block on WinXP+Accel
Analyses:
- Waiting on data
Beta 47
Single phase
Start: beta 1 (built on Apr 26, shipped Apr 27)
End: ran through the entire beta (Jun 8)
Range: [beta 1, ...)
Sample: 50% of eligible users (yield ~20%)
Details:
- APZ is enabled by default (conditional to e10s) starting in 47
- No add-ons, no a11y
- No GTK+ 3.20+
- E10SCohort annotation added to crash report
- "disqualified" cohort is now split into "disqualified-test" and "disqualified-control" which should help verify the even-ness of the data
Related bugs: Tracking bug, automatically started (no bug), block GTK+ 3.20+, disqualified -> disqualified-test/disqualified-control, E10SCohort annotation on crash reporter
Analyses:
Beta 46
2nd phase
Start: beta 3 (built on March 21, shipped March 22)
End: beta 8 (built on April 4, shipped April 5)
Range: [beta 3, beta 8) (15 days)
Sample: 50% of ~40% eligible users (yield ~20%)
Details:
- No add-ons, no a11y
- APZ enabled (for the first time)
- Through the system add-on (for the first time)
Related bugs: bugs related to the system add-on, stop it
Analyses:
1st phase
Start: March 9
End: March 21
Range: [March 9, March 21) (12 days)
Sample: 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)
Details:
- No add-ons
- APZ disabled
- Through Telemetry Experiments
Related bugs: start it, stop it
Analyses:
Beta 45
2nd phase
Start: February 12
End: February 22
Range: [February 12, February 22) (11 days)
Sample: 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)
Details:
- No add-ons (add-on users filtered for the first time)
- APZ disabled
- Through Telemetry Experiments
Related bugs: start it, stop it, [filter add-on users from e10s https://bugzilla.mozilla.org/show_bug.cgi?id=1234675]
Analyses:
1st phase
Start: January 25
End: February 11
Range: [January 25, February 11) (17 days)
Sample: 50% targeted by experiment * 50% A/B split (yield ~25%)
Details:
- Add-ons enabled
- LastPass 3.* and Skype Click-to-Call users removed from the experiment after a few days due to crash
- APZ disabled
- No a11y
- Through Telemetry Experiments
Related bugs: start it, automatically stopped by date, block LastPass 3.* users, Block Skype Click-to-Call users
Analyses: