Electrolysis/Experiments

Electrolysis Experiments

This page intends to document the A/B experiments that were done during the development of e10s.

Experiment mechanisms

Telemetry Experiments

The first experiments were done through the Telemetry Experiments system, which is an A/B mechanism meant to test new features to telemetry-enabled users. The experiments were published in the production server for this mechanism.

Staged Rollout - System add-on

With the approach of releasing e10s to the release audience, it was necessary to have a more detailed mechanism specific to e10s in order to be able to do a staged rollout of the feature and quickly increase/decrease the rollout rate. To do that, the new System Add-on mechanism was chosen, because it can be updated out-of-band from the release process.

After the system add-on was developed, we started using it to run new phases of e10s experiments on Beta, in order to also test the add-on itself in preparation for Release.

Note: all dates are an approximation from the end-user's point of view, because they may take a couple of days to be updated to the build where the experiment was started/terminated.

List of e10s-multi experiments

Phase 2

Start: beta 2 (or 3?) (Apr 27)

Sample: 10% non-e10s, of the 90%, 50% 1 content process (control) 50% 4 content processes (test)

Phase 1

Start: beta 1 (Apr 19)

End: beta 2 (or 3) (Apr 26)

Sample: 10% non-e10s of the 90%, 25% 1 content process 25% 2 content processes 25% 4 content processes 25% 8 content processes

List of e10s experiments

Beta 49

Single phase

Start: beta 1 (Aug 2 or 3)

End: still running

Range: [beta 1, ...)

Sample: 50% test/control for each group

Details:

• APZ is enabled by default (conditional to e10s) starting in 47
• No accessibility on Mac, Windows, Linux
• No Mac OS X 10.6 - 10.8
• No WinXP+graphics acceleration users (this caused a 3.9% drop on eligible users compared from the Beta 47 experiment)
• Windows XP users should be excluded from release criteria analyses (e10s on WinXP disabled on the release channel by bug 1275039)

Addons Experiment Details:

• This experiment is the first to test some add-ons with e10s. The cohorts are divided like so:
  • Test/Control:
    • Users with no add-ons, and no other disqualifying criteria (same as previous experiments)
  • addons-set2a-test / addons-set2a-control
    • Users with any number > 0 of add-ons, where all of their add-ons are part of the experiment list (see below)
  • disqualified-test / disqualified-control
    • Users with any number > 0 of add-ons, where any of their add-ons are not part of the experiment list. (Or may be disqualified by other criteria such as accessibility, OS, etc)

Addons Experiment List:

• All WebExtension-style add-ons
• And the following 10 bootstrap/jetpack add-ons:
  • Greasemonkey, Download YouTube Videos as MP4, Video Download Helper, Lightbeam, Flashblock, Adblock Plus, uBlock Origin, Emoji Cheatsheet, Awesome Screenshot Plus, Personas Plus

Related bugs: Tracking bug, automatically started (no bug), mechanism to control addons experiment list,

Analyses:

• Stability dashboard
• Week 1, Week 2, Week 3, Week 4, Week 5, Week 6

Beta 48

Single phase

Start: beta 1 (built on Jun 6, shipped Jun 8)

End: ran through the entire beta (ended on Aug 2)

Range: [beta 1, ...)

Sample: 50% of eligible users (yield ~20%)

Details:

• APZ is enabled by default (conditional to e10s) starting in 47
• No add-ons
• No accessibility on Mac and Windows
• No accessibility on Linux starting on build 2 (bug 1277882)
• No Mac OS X 10.6 - 10.8
• GTK+ 3.20+ users reincluded
• No WinXP+graphics acceleration users (this caused a 3.9% drop on eligible users compared from the Beta 47 experiment)
• Windows XP users should be excluded from release criteria analyses (e10s on WinXP disabled on the release channel by bug 1275039)

Related bugs: Tracking bug, automatically started (no bug), unblock GTK+ 3.20+, block on OSX 10.6-10.8, block on release WinXP, block on linux a11y, block on WinXP+Accel

Analyses:

• Week 1-2, Week 3, Week 4, Week 5, Week 6, Week 7

Beta 47

Single phase

Start: beta 1 (built on Apr 26, shipped Apr 27)

End: ran through the entire beta (Jun 8)

Range: [beta 1, ...)

Sample: 50% of eligible users (yield ~20%)

Details:

• APZ is enabled by default (conditional to e10s) starting in 47
• No add-ons, no a11y
• No GTK+ 3.20+
• E10SCohort annotation added to crash report
• "disqualified" cohort is now split into "disqualified-test" and "disqualified-control" which should help verify the even-ness of the data

Related bugs: Tracking bug, automatically started (no bug), block GTK+ 3.20+, disqualified -> disqualified-test/disqualified-control, E10SCohort annotation on crash reporter

Analyses:

• Week 1, Week 2, Week 3, Week 4, Week 5

Beta 46

2nd phase

Start: beta 3 (built on March 21, shipped March 22)

End: beta 8 (built on April 4, shipped April 5)

Range: [beta 3, beta 8) (15 days)

Sample: 50% of ~40% eligible users (yield ~20%)

Details:

• No add-ons, no a11y
• APZ enabled (for the first time)
• Through the system add-on (for the first time)

Related bugs: bugs related to the system add-on, stop it

Analyses:

• https://github.com/vitillo/e10s_analyses/tree/master/beta/46/apz

1st phase

Start: March 9

End: March 21

Range: [March 9, March 21) (12 days)

Sample: 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)

Details:

• No add-ons
• APZ disabled
• Through Telemetry Experiments

Related bugs: start it, stop it

Analyses:

• https://github.com/vitillo/e10s_analyses/tree/master/beta/46/noapz

Beta 45

2nd phase

Start: February 12

End: February 22

Range: [February 12, February 22) (11 days)

Sample: 50% targeted by experiment * 50% A/B split * ~40% eligible users (yield ~10%)

Details:

• No add-ons (add-on users filtered for the first time)
• APZ disabled
• Through Telemetry Experiments

Related bugs: start it, stop it, [filter add-on users from e10s https://bugzilla.mozilla.org/show_bug.cgi?id=1234675]

Analyses:

• https://github.com/vitillo/e10s_analyses/tree/master/beta/45/withoutaddons

1st phase

Start: January 25

End: February 11

Range: [January 25, February 11) (17 days)

Sample: 50% targeted by experiment * 50% A/B split (yield ~25%)

Details:

• Add-ons enabled
  • LastPass 3.* and Skype Click-to-Call users removed from the experiment after a few days due to crash
• APZ disabled
• No a11y
• Through Telemetry Experiments

Related bugs: start it, automatically stopped by date, block LastPass 3.* users, Block Skype Click-to-Call users

Analyses:

• https://github.com/vitillo/e10s_analyses/tree/master/beta/45/withaddons