CloudServices/SoftRelease

Overview

Firefox Mobile and Firefox Desktop both follow a specific release cycle that makes it hard to ship experimental features, try out small changes on a subset of users or ramp up a new feature to avoid huge peaks on our infrastructure.

The SoftRelease service offers a way to ramp up or bucket-test a new feature shipped in Firefox Mobile or Desktop.

Use cases examples:

• ramping up Firefox Hello for our user base by making it accessible to 10% of the user base and growing it to 100% once we are confident that the server infrastructure works well.
• activating a new feature for specific regions in Firefox.
• making small UI variations like what they're doing at the Foundation for their "End Of the Year" campaign, see https://fundraising.mozilla.org/testing-testing-and-more-testing/

General Principle

When Firefox Mobile or Dekstop starts, it sends a request to the SoftRelease service to ask if a feature has to be activated or not.

The proposed API is a single HTTP endpoint that contains the name of the feature:

GET https://features.services.mozilla.com/<feature_name>

{'enabled': true}

The server analyzes the client IP and specific headers like the User Agent, and returns a JSON mapping containing the answer.

When the enabled key is sent back, the client gets a YES/NO answer and acts upon it. For example, for Firefox Hello, the decision to display the Hello button or not could be done by this call:

GET https://features.services.mozilla.com/hello

In some other cases, the feature is activated but we want to make different versions. it's preferrable to let the client decide what to do, given a list of options values sent back by the server.

For example, in a campaign page, an UI has a button with two options that may vary amongst users: its color and its text. The client can call the server to get back the values to use:

GET https://features.services.mozilla.com/campaign-2015

{'color': '#ff0000',
 'value': 'Click Here'}

When the client wants to get several features at once, it can batch its requests by calling the root endpoint:

GET https://features.services.mozilla.com?features=campaign-2015,hello

{
 'hello': {'enabled': true}, 
 'campaign-2015': {'color': '#ff0000', 'value': 'Click Here'}
}

Dashboard

To manage the responses, the service provides a dashboard where admins can:

• add or remove a feature name
• list the different possible responses that can be returned to a user
• configure a policy for the server to decide which response to return

We provide 3 policies

• Weighted: defines a percentage for each possible response.
• Geolocation: associates regions to possible responses.
• User-Agent: associates User-Agents to possible responses.

Policies can be combined.

Examples

Example 1

We want to try out a new donation campaign UI for French users. In their case, we want to set the color of the button Green. For other users, we want a Blue color.

steps:

1. . We add a new "campaign-button-color" feature via the dashboard
2. . we add the two possible responses:

  Green => {'color': 'green'}
  

  Blue => {'color': 'blue'}
  

1. . we associate each response to a Geolocation Policy

  France: Green
  

  default: Blue
  

Metrics

Collecting metrics during A/B testing is important to follow & understand what's the impact of the different versions of a feature.

The proposed service does not provide any server-side metrics, but returns a unique id for each combination returned for a given feature.

e.g. :

GET https://features.services.mozilla.com/<feature_name>

{'enabled': true, 'id': '4fa1d44e-2f9d-4cd3-a660-85e892c0ace9'}

The id is guaranteed to stay unique and consistent and can be used by the application to track the different combinations.

Related Works

There are two related works at Mozilla:

• The Campaign Manager - https://wiki.mozilla.org/CloudServices/Roadmaps/Campaign-Manager
• Abatar - https://github.com/dannycoates/abatar

The campaign manager focuses on targeted messaging for Android users, but its principle is quite similar to the current proposal: the client asks the server for messages to display - and that message may vary from one client to the other.

Abatar is a client-side only A/B testing framework that might gain at some point a server side. Abatar plans to ask the server to return Javascript code snippets that may change over time, and all the decisions are made by the client side by running the snippet. Those snippets are better served by the project's server itself for practical reasons (code deployment, no decision made on the server-side) and there would be no benefit to have those javascript snippets served by the current proposed service.