Firefox/Kinto

From MozillaWiki
< Firefox
Revision as of 11:40, 10 April 2017 by Leplatrem (talk | contribs) (Improve specifications)
Jump to navigation Jump to search

Kinto Integration in Firefox

[Kinto] is a simple JSON storage service that is used in Firefox for remote settings and the storage.sync API.

Key features

  • Diff-based data synchronization
  • Offline persistence
  • Data integrity/signing
  • Peer review to publish data changes
  • Admin panel UI
  • Firefox Accounts Integration
  • Built-in telemetry (coming soon)

Use Cases

Component Description Contact
OneCRL Certificates revocation blocklist Mark Goodwin — mgoodwin
HPKP HTTP Public Key Pinning Mark Goodwin — mgoodwin
Addons/Plugins/Gfx Blocklisting Black list for unsecure/instable/malicious addons, plugins or graphical drivers Mathieu Leplatre — leplatrem
storage.sync WebExtensions) API for user data storage Ethan Glasser Camp — glasserc
Fennec assets catalog Catalog of assets to be downloaded after installation Sebastian Kaspari — sebastian
Fennec experiments Data for A/B testing in Fennec Sebastian Kaspari — sebastian
iOS experiments Data for A/B testing in Firefox iOS Stephan Leroux — sleroux

Ideas/WIP

  • Password manager recipes (contact: Matthew Noorenberghe — MattN)

Feel free to come and discuss on #storage :)

Usage

Two client libraries are embedded in Firefox:

HTTP client

const { KintoHttpClient } = Cu.import("resource://services-common/kinto-http-client.js", {});

const remote = "https://kinto.dev.mozaws.net/v1";
const headers = {Authorization: "Basic " + btoa("user:pass")};

const client = new KintoHttpClient(remote, {headers});
const records = yield client.bucket("a-bucket")
                            .collection("a-collection")
                            .listRecords();

Offline client

const { Kinto } = Cu.import("resource://services-common/kinto-offline-client.js", {});
const { FirefoxAdapter } = Cu.import("resource://services-common/kinto-storage-adapter.js", {});

const remote = "https://kinto.dev.mozaws.net/v1";
const headers = {Authorization: "Basic " + btoa("user:pass")};

const kinto = new Kinto({adapter: FirefoxAdapter});
const collection = kinto.collection("a-collection");

// Fetch/Publish changes.
const {ok} = yield collection.sync({
  bucket: "a-bucket",
  remote,
  headers});

// Read local collection of records.
const records = yield collection.list();

Servers

Server API Admin Contact
Public development instance (flushed every day) with many plugins https://kinto.dev.mozaws.net/v1/ Admin UI Kinto mailing-list
Remote settings STAGE https://kinto.stage.mozaws.net/v1/ (read-only) Admin UI VPN only OPS via Bugzilla
Remote settings PROD https://firefox.settings.services.mozilla.com/v1/ (read-only) Admin UI VPN only OPS via Bugzilla
storage.sync https://webextensions.settings.services.mozilla.com/v1/ N/A OPS via Bugzilla

How to add a new official dataset ?

The only persons allowed to create buckets/collections in stage/production is the OPs team, so you have to create a Bugzilla ticket and specify everything you need to be created.

  • The bucket name, and its permissions (who can read/write/create collections)
  • A collection name, and its permissions (who can read/write/create records)

For use-cases where a few administrators manage some data downloaded by all Firefox clients, you have to enable peer reviewing. You will have 3 buckets: a staging where editors make changes, a preview bucket where changes are published when a review is requested, and the final bucket where changes are published when the review is approved. You must specify:

  • Who can edit (members of the editors group)?
  • Who can review/approve data (members of the reviewers group)?
  • Who can read data (members of the observers group, eg. QA)? Optional
  • A new key for the digital signatures (Autograph) will have to be configured on the server side for your use-case.

For use-cases where a lot of users write data (like storage.sync), the quota plugin may have to be configured.


Specifications

The distribution package of Kinto that is deployed is kinto-dist. There are several instances, depending on the use-case — mainly if clients have write access or not.

Blocklists

The goal is to replace the current system — based on a single XML file downloaded everyday — by several Kinto collections on the remote settings instance.


List Bucket Collection
OneCRL blocklists certificates
Add-ons blocklists addons
Plugins blocklists plugins
Gfx blocklists gfx
HPKP pinnings pings

See https://bugzilla.mozilla.org/show_bug.cgi?id=1197707

Transition

Currently the blocklist system relies on a big XML file that is downloaded every day. It contains block entries for certificates to be revoked, addons and plugins to be disabled, and gfx environments that cause problems or crashes. Everything is managed via the Addons server.

Firefox (and derivatives like Thunderbird, Seamonkey, ...) downloads it on an URL that contains client information (eg. https://blocklist.addons.mozilla.org/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/44.0a1/)

Using the same XPCOM notification callback, the new mechanism will synchronize the local copy of each collection from the remote server.

If changes are available, the local copy of the data will be updated and content signature verified.

  • phase 1: DONE: Both mechanisms run in parallel but only the legacy one will be used.
  • phase 2: DONE: The source of truth for blocklist is Kinto but produces the same XML as the legacy one.
  • phase 3: Eventually, blocking mechanism will rely on the data managed via JSON, and the old XML client will be decommissioned.


Fennec assets catalog

The goal is to remove the static assets (fonts, hyphenation dicts, etc.) from the distribution package and download them asynchronously using an online Kinto catalog, on the remote settings instance.

List Bucket Collection
Download catalog fennec catalog

See https://bugzilla.mozilla.org/show_bug.cgi?id=1201059


WebExtensions storage.sync

  • Users are authenticated
  • Every users have their own bucket (using the default bucket plugin)
  • The amount of data that users are allowed to store is limited
  • Data is encrypted using Firefox Accounts keys


Contribute

Generate bundles

The Kinto client libraries are developed independently on Github:

  • kinto-http is the HTTP client for the Kinto REST API;
  • kinto.js is the offline-first client for Kinto.

With the help of Babel and browserify, a bundle is generated for Firefox with the minimum transpilation possible (eg. CommonJS require, ES7 decorators).

kinto.js

From the kinto.js repo, generate the moz-kinto-offline-client.js file: 

$ npm run dist-fx

And overwrite it in the Firefox code base: 

$ cp dist/moz-kinto-offline-client.js ../mozilla-central/services/common/kinto-offline-client.js

kinto-http.js

From the kinto-http.js repo, generate the moz-kinto-http-client.js file: 

$ npm run dist-fx

And overwrite it in the Firefox code base: 

$ cp dist/moz-kinto-http-client.js ../mozilla-central/services/common/kinto-http-client.js

Run the tests

First, follow the instructions to build Firefox.

In order to speed up the build and being able to run tests properly, create a **mozconfig** file at the root of the repository: 

ac_add_options --enable-debug
ac_add_options --disable-optimize
ac_add_options --disable-crashreporter
ac_add_options --with-ccache=/usr/bin/ccache

For JavaScript updates only, have a look at Artifacts Builds, trading bandwidth for compilation time. 

$ ./mach build faster
$ ./mach xpcshell-test services/common/tests/unit/test_kinto.js
$ ./mach xpcshell-test services/common/tests/unit/test_storage_adapter.js

Or both at once: 

$ ./mach xpcshell-test services/common/tests/unit/test_kinto.js services/common/tests/unit/test_storage_adapter.js

There are also tests relying on Kinto in services/common/tests/unit/test_blocklist_* .

Gfx blocklist tests

The Gfx test suite requires the debug mode to be enabled. Add this to mozconfig file, rebuild and run the tests: 

ac_add_options --enable-debug

Debug content signature

You can get tests (or Firefox) to give you more information on what the content signature verifier is doing by setting the NSPR_LOG_MODULES environment variable. For example: 

EXPORT NSPR_LOG_MODULES=ContentSignatureVerifier:5,CSTrustDomain:5

TDD mode

Using inotify, we will detect a file change in the dist/ folder and run a series of commands to execute the tests automatically.

First, install inotify-tools: 

sudo apt-get install inotify-tools

Then start an infinite loop with inotify-wait: 

while true; do
    # Wait for a change
    inotifywait -q -e create,modify,delete -r ~/Code/Mozilla/kinto.js/dist
    # Execute these commands
    cp ~/Code/Mozilla/kinto.js/dist/moz-kinto-offline-client.js services/common/kinto-offline-client.js
    ./mach xpcshell-test services/common/tests/unit/test_storage_adapter.js
    ./mach xpcshell-test services/common/tests/unit/test_kinto.js          
done

Source: Antoine Cezar

Submit patch

> Patch are contributed to kinto.js and kinto-http.js, which are first released on NPM.

DO NOT land files that are not tagged officially on upstream repositories.

Become a contributor

Configure SSH key for hg: 

Host hg.mozilla.org
  User user@server.com
  IdentityFile ~/.ssh/contrib_moz

Run integration tests: «Try»

See https://wiki.mozilla.org/ReleaseEngineering/TryServer

With Mercurial, push a patch to MozReview (see below) and trigger a Try build from the UI (*Automation > Trigger a Try build*).

Or with git, use a gecko-dev fork from Github, and with install moz git tools 

git push-to-try -t --rev master..HEAD ~/hg/mozilla-central/ -b do -p linux,linux64,macosx64,win32,win64 -u xpcshell,mochitests -t none

Submit for review

See http://mozilla-version-control-tools.readthedocs.org 

# Keep a bookmark of your branch to address review.
# (equivalent of git branches)
hg bookmark bug/XXXXX

# Commit with link to Bugzilla
hg commit -m "Bug XXXXX - Upgrade <lib> to X.Y.Z"

# Submit to MozReview
hg push review

# Go back to «master»
hg update -C central

Examples:


To adjust previously submitted patch: 

# Go to bookmark
hg update -C bug/XXXXX

# Address comments
...

# Amend commit
hg commit --amend

# Inspect history
hg log -f -G --rev bug/XXXXX | less

# Update review
hg push review

When a patch has several commits: 

# Inspect current branch
hg log -f -G --rev bug/XXXXX | less

# Squash/Reword commits since rev c59308877f9a
hg histedit c59308877f9a

To rebase revision c59308877f9a and descendants on last «master»: 

hg pull 

hg rebase -d central -s c59308877f9a
Retrieved from "https://dev.wikimo.nonprod.webservices.mozgcp.net/index.php?title=Firefox/Kinto&oldid=1167885"