Telemetry/Experiments

Telemetry Experiments is a feature that allows Firefox to automatically download and run specially-designed restartless addons based on certain conditions.

Experiments are available to anyone who has Telemetry enabled, which is the default on pre-release versions of Firefox. This feature is currently only available in desktop versions of Firefox.

Experiments can be targeted at various populations by specifying conditions in an experiment manifest. http://gecko.readthedocs.org/en/latest/browser/experiments/experiments/manifest.html has details on the possible conditions, and there it is also possible to write complex conditions in JavaScript which can filter on any data in the telemetry or Firefox Health Report payloads.

Product Approval

Product approval is required to run an experiment. User-facing experiments should be approved by a Firefox product manager (Jeff Griffiths). For experiments which have no user-visible effects, this approval is trivial.

Data Collection

Before building an experiment, contact bsmedberg or a data-collection peer to talk about experiment design and how the results of the experiment will be measured and reported. Depending on the nature of the experiment and its data-collection system, he may require additional privacy or security review.

Engineering

Experiments are a special type of restartless extension. They can do anything a regular restartless extension can do.

Repository and Reviews

Experiments must be developed in the public telemetry experiment server source repository and must be reviewed just as if they were part of Firefox on mozilla-central. If an experiment adds special data collection, that data collection must be documented and reviewed as normal. There are more detailed instructions on the mechanics of developing and testing the experiment at Developing a Telemetry Experiment.

To request reviews, file a bug against the Firefox Health Report product, Desktop Client component.

install.rdf

Experiments, like other restartless extensions, should have an install.rdf manifest file.

However, unlike other extensions, the type must be set to 128.

manifest.json

Unlike other restartless extensions, experiments must have a top-level manifest.json file. The manifest.json format is not currently documented, though examples can be found by looking at past experiments in the telemetry-experiment-server mercurial repo.

In particular, note that the name field is visible to end users, so it should be accurate and not confusing.

Extension Lifecycle

Most experiments live for a fixed period and are uninstalled by the experiment system at the end date or after the specified elapsed time.

A few experiments are intended to collect data once and then uninstall themself. These experiments should uninstall themselves when done by calling Experiments.disableExperiment("FROM_API").

Extension Signing

Telemetry Experiments need to be signed to run in FF 42+, just like all other add-ons.

Once the code is reviewed, open a bug against the addons.mozilla.org product, Addon Validation component, attach the xpi file, and request that it be signed.

Once the AMO signing API is ready (bug 1206158), the bug filing step won't be necessary to get the extension signed.

Staging

To deploy an experiment, write a patch to http://hg.mozilla.org/webtools/telemetry-experiment-server/ for experiment deployment, and bsmedberg or a peer can review. Pushes there auto-deploy to the staging server at https://telemetry-experiment-dev.allizom.org/. QA should verify the experience on the staging server, including data collection.

• QA Guide For Telemetry Experiments - Kamil Jozwiak is the QA lead for experiments

Final Deployment

Final deployment of an experiment must be approved by Lawrence Mandel or a release-management peer. Send an email to release-drivers@mozilla.org with the following information:

• A brief summary of the experiment, the questions that will be answered, and how the experiment will answer the questions
• bug(s) where the code was developed and reviewed
• details about experiment deployment: what channel(s) will the experiment be deployed to, how many users will be participating, and how long will the experiment run? This data can be copied from the staging server.

Here is an example of an approval email:

Subject: Intent to ship: Telemetry experiment for verification of telemetry experiments bug

This is a request for approval to ship the following Telemetry experiment: 
Telemetry experiment for verification of telemetry experiments bug.

Summary
In bug 1052714, I plan to ship a telemetry experiment to 50% of nightly channel users to verify that a fix
for a race condition setting telemetry experiment branches is actually fixed. This experiment will have no
user effect, and only exists to test the system. The results will be collected and analyzed from FHR
payloads.

Related Bugs:
bug 1052714

Start Date
    14-Aug-2014
End Date
    10-Sep-2014
Maximum runtime
    7.0 days
Update Channels
    nightly
Build ID
    20140820030202- 
Instance population
    All nightly instances
Sample Rate
    50%

After final approval, file a bug to deploy to production at https://telemetry-experiment.cdn.mozilla.net/ . Use bug 1057464 or bug 1131864 as a model for deployment.

Monitoring

Use http://bsmedberg.github.io/telemetry-experiments-dashboard/ to monitor experiment adoption.