Firefox/Kinto: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(tweak to glob pattern to make this work a little bit better)
(Remove decommissioned use-cases)
 
(15 intermediate revisions by 7 users not shown)
Line 1: Line 1:
= Kinto Integration in Firefox =
= Kinto Integration in Firefox =
[http://www.kinto-storage.org Kinto] is a simple JSON storage service that was integrated into Firefox, and currently used for RemoteSettings.


= Key features =
= Key features =


* Diff-based data synchronization
* Diff-based data synchronization
* Offline persistence
* Data integrity/signing
* Data integrity/signing
* Offline persistence
* Peer review to publish data changes
* Admin panel UI
* Admin panel UI
* Built-in Telemetry
* Firefox Accounts Integration


= Use Cases =
= Use Cases =


* Certificates blocklist (OneCRL) ''(contact: Mark Goodwin — mgoodwin)''
{|class="wikitable"
* Addons/Plugins/Gfx [[Blocklisting]] ''(contact: Mathieu Leplatre — leplatrem)''
! Component
* <code>storage.sync</code> API ([[WebExtensions]]) ''(contact: Tarek Ziadé — tarek)''
! Description
* Fennec assets catalog ''(contact: Sebastian Kaspari — sebastian)''
! Contact
* Password manager recipes ''(contact: Matthew Noorenberghe — MattN)''
|-
|RemoteSettings
|[[Firefox/RemoteSettings]] allows to ship data and files to Firefox. It is used for blocklists, security state, Activity Stream, Fennec assets and experiments etc.
|Mathieu Leplatre — :leplatrem
|-
|}


Feel free to come and discuss on <code>#storage</code> :)
It used be supporting more use-cases in the past. See also [https://github.com/mozilla-services/servicedenuages.fr/blob/master/content/2017.12.kinto-at-mozilla.rst Kinto at Mozilla]


= Usage =
= Usage =


Leveraging the [https://github.com/Kinto/kinto-http.js Kinto HTTP client] in Gecko looks like this:
Two client libraries are vendored in Firefox:


<pre>
* [https://github.com/Kinto/kinto-http.js Kinto HTTP client]: for direct interactions with the Kinto HTTP API
const { KintoHttpClient } = Cu.import("resource://services-common/kinto-http-client.js");
* [https://github.com/Kinto/kinto.js Kinto offline client]: for offline persistence in internal SQLite or IndexedDB


const client = new KintoHttpClient("https://kinto.dev.mozaws.net/v1");
== HTTP client ==
client.bucket("a-bucket").collection("a-collection")
  .listRecords()
  .then(result => ...);
</pre>
 
As for the [https://github.com/Kinto/kinto.js Kinto offline-first client], it is like:


<pre>
<pre>
const { loadKinto } = Cu.import("resource://services-common/kinto-offline-client.js");
const { KintoHttpClient } = Cu.import("resource://services-common/kinto-http-client.js", {});


const KintoOfflineClient = loadKinto();
const remote = "https://kinto.dev.mozaws.net/v1";
const headers = { Authorization: "Basic " + btoa("user:pass") };


const client = new KintoOfflineClient({
const client = new KintoHttpClient(remote, {headers});
  adapter: Kinto.adapters.FirefoxAdapter,
const records = await client.bucket("a-bucket")
  remote: "https://kinto.dev.mozaws.net/v1",
                            .collection("a-collection")
  bucket: "a-bucket"
                            .listRecords();
});
 
const collection = db.collection("a-collection");
 
try {
  yield collection.db.open();
  // Fetch changes from server.
  yield collection.sync();
  // Read local collection of records.
  const records = yield collection.list();
} finally {
  yield collection.db.close();
}
</pre>
</pre>


== Offline client ==


= Specifications =
<pre>
const { Kinto } = Cu.import("resource://services-common/kinto-offline-client.js", {});


Currently, the instance of Kinto used by Firefox clients is hosted at https://firefox.settings.services.mozilla.com/v1/
const remote = "https://kinto.dev.mozaws.net/v1";
const headers = { Authorization: "Basic " + btoa("user:pass") };


== Blocklists ==  
const kinto = new Kinto();
const bucket = "a-bucket";
const collection = kinto.collection("a-collection");


The goal is to replace the current system based on a single XML file downloaded everyday by several Kinto collections.
// Fetch/Publish changes.
const { ok } = await collection.sync({ bucket, remote, headers });


* The bucket is '''blocklists'''
// Read local collection of records.
* The collection for OneCRL entries is '''certificates''' https://firefox.settings.services.mozilla.com/v1/buckets/blocklists/collections/certificates/records
const records = await collection.list();
* The collection for Add-ons entries is '''addons''' https://firefox.settings.services.mozilla.com/v1/buckets/blocklists/collections/certificates/records
</pre>
* The collection for Plugins entries is '''plugins''' https://firefox.settings.services.mozilla.com/v1/buckets/blocklists/collections/certificates/records
* The collection for Gfx entries is '''gfx''' https://firefox.settings.services.mozilla.com/v1/buckets/blocklists/collections/certificates/records
 
See https://bugzilla.mozilla.org/show_bug.cgi?id=1197707
 
=== Transition ===
 
Currently the blocklist system relies on a big XML file that is downloaded every day. It contains block entries for certificates to be revoked, addons and plugins to be disabled, and gfx environments that cause problems or crashes. Everything is managed via the Addons server.
 
Firefox (and derivatives like Thunderbird, Seamonkey, ...) downloads it on an URL that contains client information (eg. https://blocklist.addons.mozilla.org/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/44.0a1/)
 
Using the same XPCOM notification callback, the new mechanism will [https://dxr.mozilla.org/mozilla-central/rev/fc15477ce628599519cb0055f52cc195d640dc94/toolkit/mozapps/extensions/nsBlocklistService.js#632-642 synchronize the local copy of each collection from the remote server].
If changes are available, the [https://dxr.mozilla.org/mozilla-central/rev/fc15477ce628599519cb0055f52cc195d640dc94/services/common/KintoBlocklist.js#78-105 local copy will be updated] and [https://bugzilla.mozilla.org/show_bug.cgi?id=1263602 content signature verified].
 
* In '''phase 1''', it will be a no-op. Both mechanisms will run in parallel but only the legacy one will be used. We'll keep both mainly because the current download of the XML is used to count active daily users. Once we are ok with the statistics, we'll go to phase 2.
* In '''phase 2''', we'll change the source of truth of block entries on the server side. The data from the Addons server won't be used anymore. That means the [https://github.com/mozilla-services/kinto-amo/ server will produce the same XML file] but using the data stored in the new service.
* In '''phase 3''', the [https://bugzilla.mozilla.org/show_bug.cgi?id=1257565 blocking mechanism will rely on the data managed via JSON], and the old XML client will be decommissioned.


= Resources =


== Fennec assets catalog ==
* [https://remote-settings.readthedocs.io Remote Settings docs]
 
* [https://blog.mathieu-leplatre.info/the-history-of-firefox-remote-settings.html The History of Remote Settings]
The goal is to remove the static assets (fonts, hyphenation dicts, etc.) from the distribution package and download them asynchronously using an online Kinto catalog.
* [https://github.com/mozilla-services/servicedenuages.fr/blob/master/content/2017.12.kinto-at-mozilla.rst Kinto at Mozilla] (blog post) -- see also [https://web.archive.org/web/20180111085958/https://www.servicedenuages.fr/en/kinto-at-mozilla the published version, no longer available at its initial location]
 
* The bucket is '''fennec'''
* The collection is '''catalog'''
 
See https://bugzilla.mozilla.org/show_bug.cgi?id=1201059
 


= Contribute =
= Contribute =
== Upgrade client libraries ==
Two client libraries are embedded in Firefox:
* <code>Kinto/kinto-http.js</code>: for direct interactions with the Kinto HTTP API
* <code>Kinto/kinto.js</code>: for offline persistence in internal SQLite


=== Generate bundles ===
=== Generate bundles ===
Line 114: Line 83:
* [https://github.com/Kinto/kinto.js kinto.js] is the offline-first client for Kinto.
* [https://github.com/Kinto/kinto.js kinto.js] is the offline-first client for Kinto.


With the help of Babel and browsersify, a bundle is generated for Firefox with the minimum transpilation possible (eg. CommonJS require, ES7 decorators).
With the help of Babel and browserify, a bundle is generated for Firefox with the minimum transpilation possible (eg. CommonJS require, ES7 decorators).


==== kinto.js ====
==== kinto.js ====
Line 142: Line 111:
<pre>
<pre>
$ cp dist/moz-kinto-http-client.js ../mozilla-central/services/common/kinto-http-client.js
$ cp dist/moz-kinto-http-client.js ../mozilla-central/services/common/kinto-http-client.js
</pre>
=== Run the tests ===
First, follow [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions the instructions to build Firefox].
In order to speed up the build and being able to run tests properly, create a **mozconfig** file at the root of the repository:
<pre>
ac_add_options --enable-debug
ac_add_options --disable-optimize
ac_add_options --disable-crashreporter
ac_add_options --with-ccache=/usr/bin/ccache
</pre>
For JavaScript updates only, have a look at [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Artifact_builds Artifacts Builds], trading bandwidth for compilation time.
<pre>
$ ./mach build faster
$ ./mach xpcshell-test services/common/tests/unit/test_kinto.js
$ ./mach xpcshell-test services/common/tests/unit/test_storage_adapter.js
</pre>
Or both at once:
<pre>
$ ./mach xpcshell-test services/common/tests/unit/test_kinto.js services/common/tests/unit/test_storage_adapter.js
</pre>
==== Gfx blocklist tests ====
The Gfx test suite requires the debug mode to be enabled. Add this to <code>mozconfig</code> file, rebuild and run the tests:
<pre>
ac_add_options --enable-debug
</pre>
==== Debug content signature ====
You can get tests (or Firefox) to give you more information on what the content signature verifier is doing by setting the NSPR_LOG_MODULES environment variable. For example:
<pre>
EXPORT NSPR_LOG_MODULES=ContentSignatureVerifier:5,CSTrustDomain:5
</pre>
==== TDD mode ====
Using ''inotify'', we will detect a file change in the <code>dist/</code> folder and run a series of commands to execute the tests automatically.
First, install ''inotify-tools'':
<pre>
sudo apt-get install inotify-tools
</pre>
Then start an infinite loop with <code>inotify-wait</code>:
<pre>
while true; do
    # Wait for a change
    inotifywait -q -e create,modify,delete -r ~/Code/Mozilla/kinto.js/dist
    # Execute these commands
    cp ~/Code/Mozilla/kinto.js/dist/moz-kinto-offline-client.js services/common/kinto-offline-client.js
    ./mach xpcshell-test services/common/tests/unit/test_storage_adapter.js
    ./mach xpcshell-test services/common/tests/unit/test_kinto.js         
done
</pre>
[http://makina-corpus.com/blog/metier/2013/utiliser-inotify-pour-tester-en-continu-son-code Source: Antoine Cezar]
=== Submit patch ===
> Patch are contributed to kinto.js and kinto-http.js, which are first released on NPM.
'''DO NOT''' land files that are not tagged officially on upstream repositories.
==== Become a contributor ====
* Generate a SSH key
* Open a bugzilla ticket to request Level-1 access
* https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/
* Get a vouch (*optional for level1*)
Configure SSH key for hg:
<pre>
Host hg.mozilla.org
  User user@server.com
  IdentityFile ~/.ssh/contrib_moz
</pre>
==== Run integration tests: «Try» ====
See https://wiki.mozilla.org/ReleaseEngineering/TryServer
With Mercurial, push a patch to MozReview (see below) and trigger a Try build from the UI (*Automation > Trigger a Try build*).
Or with git, use a ''gecko-dev'' fork from Github, and with install [https://github.com/mozilla/moz-git-tools moz git tools]
<pre>
git push-to-try -t --rev master..HEAD ~/hg/mozilla-central/ -b do -p linux,linux64,macosx64,win32,win64 -u xpcshell,mochitests -t none
</pre>
==== Submit for review ====
See http://mozilla-version-control-tools.readthedocs.org
<pre>
# Keep a bookmark of your branch to address review.
# (equivalent of git branches)
hg bookmark bug/XXXXX
# Commit with link to Bugzilla
hg commit -m "Bug XXXXX - Upgrade <lib> to X.Y.Z"
# Submit to MozReview
hg push review
# Go back to «master»
hg update -C central
</pre>
Examples:
* https://reviewboard.mozilla.org/r/45445/
To adjust previously submitted patch:
<pre>
# Go to bookmark
hg update -C bug/XXXXX
# Address comments
...
# Amend commit
hg commit --amend
# Inspect history
hg log -f -G --rev bug/XXXXX | less
# Update review
hg push review
</pre>
When a patch has several commits:
<pre>
# Inspect current branch
hg log -f -G --rev bug/XXXXX | less
# Squash/Reword commits since rev c59308877f9a
hg histedit c59308877f9a
</pre>
To rebase revision c59308877f9a and descendants on last «master»:
<pre>
hg pull
hg rebase -d central -s c59308877f9a
</pre>
</pre>

Latest revision as of 09:58, 19 February 2021

Kinto Integration in Firefox

Kinto is a simple JSON storage service that was integrated into Firefox, and currently used for RemoteSettings.

Key features

  • Diff-based data synchronization
  • Offline persistence
  • Data integrity/signing
  • Peer review to publish data changes
  • Admin panel UI
  • Built-in Telemetry
  • Firefox Accounts Integration

Use Cases

Component Description Contact
RemoteSettings Firefox/RemoteSettings allows to ship data and files to Firefox. It is used for blocklists, security state, Activity Stream, Fennec assets and experiments etc. Mathieu Leplatre — :leplatrem

It used be supporting more use-cases in the past. See also Kinto at Mozilla

Usage

Two client libraries are vendored in Firefox:

HTTP client

const { KintoHttpClient } = Cu.import("resource://services-common/kinto-http-client.js", {});

const remote = "https://kinto.dev.mozaws.net/v1";
const headers = { Authorization: "Basic " + btoa("user:pass") };

const client = new KintoHttpClient(remote, {headers});
const records = await client.bucket("a-bucket")
                            .collection("a-collection")
                            .listRecords();

Offline client

const { Kinto } = Cu.import("resource://services-common/kinto-offline-client.js", {});

const remote = "https://kinto.dev.mozaws.net/v1";
const headers = { Authorization: "Basic " + btoa("user:pass") };

const kinto = new Kinto();
const bucket = "a-bucket";
const collection = kinto.collection("a-collection");

// Fetch/Publish changes.
const { ok } = await collection.sync({ bucket, remote, headers });

// Read local collection of records.
const records = await collection.list();

Resources

Contribute

Generate bundles

The Kinto client libraries are developed independently on Github:

  • kinto-http is the HTTP client for the Kinto REST API;
  • kinto.js is the offline-first client for Kinto.

With the help of Babel and browserify, a bundle is generated for Firefox with the minimum transpilation possible (eg. CommonJS require, ES7 decorators).

kinto.js

From the kinto.js repo, generate the moz-kinto-offline-client.js file: 

$ npm run dist-fx

And overwrite it in the Firefox code base: 

$ cp dist/moz-kinto-offline-client.js ../mozilla-central/services/common/kinto-offline-client.js

kinto-http.js

From the kinto-http.js repo, generate the moz-kinto-http-client.js file: 

$ npm run dist-fx

And overwrite it in the Firefox code base: 

$ cp dist/moz-kinto-http-client.js ../mozilla-central/services/common/kinto-http-client.js
Retrieved from "https://dev.wikimo.nonprod.webservices.mozgcp.net/index.php?title=Firefox/Kinto&oldid=1233809"