Firefox/Kinto: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(fixed formatting)
(Remove decommissioned use-cases)
 
(22 intermediate revisions by 7 users not shown)
Line 1: Line 1:
= Kinto Integration in Firefox =
= Kinto Integration in Firefox =


== Key features ==
[http://www.kinto-storage.org Kinto] is a simple JSON storage service that was integrated into Firefox, and currently used for RemoteSettings.


* Data synchronization
= Key features =
 
* Diff-based data synchronization
* Offline persistence
* Data integrity/signing
* Data integrity/signing
* Offline persistence
* Peer review to publish data changes
* Admin panel UI
* Admin panel UI
* Built-in Telemetry
* Firefox Accounts Integration


== Use Cases ==
= Use Cases =
 
* Certificates blocklist (OneCRL)
* Addons/Plugins/Gfx [[Blocklisting]]
* <code>storage.sync</code> API ([[WebExtensions]])
 
== Upgrade client libraries ==


Two client libraries are embedded in Firefox:
{|class="wikitable"
! Component
! Description
! Contact
|-
|RemoteSettings
|[[Firefox/RemoteSettings]] allows to ship data and files to Firefox. It is used for blocklists, security state, Activity Stream, Fennec assets and experiments etc.
|Mathieu Leplatre — :leplatrem
|-
|}


* <code>Kinto/kinto-client.js</code>: for direct interactions with the Kinto HTTP API
It used be supporting more use-cases in the past. See also [https://github.com/mozilla-services/servicedenuages.fr/blob/master/content/2017.12.kinto-at-mozilla.rst Kinto at Mozilla]
* <code>Kinto/kinto.js</code>: for offline persistence in internal SQLite


=== Generate bundles ===
= Usage =


The Kinto client libraries are developed independently on Github:
Two client libraries are vendored in Firefox:


* [https://github.com/Kinto/kinto-client kinto-client] is the HTTP client for the Kinto REST API;
* [https://github.com/Kinto/kinto-http.js Kinto HTTP client]: for direct interactions with the Kinto HTTP API
* [https://github.com/Kinto/kinto.js kinto.js] is the offline-first client for Kinto.
* [https://github.com/Kinto/kinto.js Kinto offline client]: for offline persistence in internal SQLite or IndexedDB


With the help of Babel and browsersify, a bundle is generated for Firefox with the minimum transpilation possible (eg. CommonJS require, ES7 decorators).
== HTTP client ==
 
==== kinto.js ====
 
From the kinto.js repo, generate the <code>moz-kinto-offline-client.js</code> file:
 
<pre>
$ npm run dist-fx
</pre>


And overwrite it in the Firefox code base:
   
<pre>
<pre>
$ cp dist/moz-kinto-offline-client.js ../mozilla-central/services/common/kinto-offline-client.js
const { KintoHttpClient } = Cu.import("resource://services-common/kinto-http-client.js", {});
</pre>
 
==== kinto-client.js ====


From the kinto-client.js repo, generate the <code>moz-kinto-http-client.js</code> file:
const remote = "https://kinto.dev.mozaws.net/v1";
const headers = { Authorization: "Basic " + btoa("user:pass") };


<pre>
const client = new KintoHttpClient(remote, {headers});
$ npm run dist-fx
const records = await client.bucket("a-bucket")
                            .collection("a-collection")
                            .listRecords();
</pre>
</pre>


And overwrite it in the Firefox code base:
== Offline client ==


<pre>
<pre>
$ cp dist/moz-kinto-http-client.js ../mozilla-central/services/common/kinto-http-client.js
const { Kinto } = Cu.import("resource://services-common/kinto-offline-client.js", {});
</pre>


=== Run the tests ===
const remote = "https://kinto.dev.mozaws.net/v1";
const headers = { Authorization: "Basic " + btoa("user:pass") };


First, follow [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions the instructions to build Firefox].
const kinto = new Kinto();
const bucket = "a-bucket";
const collection = kinto.collection("a-collection");


For JavaScript updates only, have a look at [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Artifact_builds Artifacts Builds], trading bandwidth for compilation time.
// Fetch/Publish changes.
const { ok } = await collection.sync({ bucket, remote, headers });


<pre>
// Read local collection of records.
$ ./mach build faster
const records = await collection.list();
$ ./mach xpcshell-test services/common/tests/unit/test_kinto.js
$ ./mach xpcshell-test services/common/tests/unit/test_storage_adapter.js
</pre>
</pre>


Or all tests related to Kinto:
= Resources =


<pre>
* [https://remote-settings.readthedocs.io Remote Settings docs]
$ ./mach xpcshell-test services/common/tests/unit/*into*
* [https://blog.mathieu-leplatre.info/the-history-of-firefox-remote-settings.html The History of Remote Settings]
</pre>
* [https://github.com/mozilla-services/servicedenuages.fr/blob/master/content/2017.12.kinto-at-mozilla.rst Kinto at Mozilla] (blog post) -- see also [https://web.archive.org/web/20180111085958/https://www.servicedenuages.fr/en/kinto-at-mozilla the published version, no longer available at its initial location]


==== TDD mode ====
= Contribute =


Using ''inotify'', we will detect a file change in the <code>dist/</code> folder and run a series of commands to execute the tests automatically.
=== Generate bundles ===


First, install ''inotify-tools'':
The Kinto client libraries are developed independently on Github:


<pre>
* [https://github.com/Kinto/kinto-http.js kinto-http] is the HTTP client for the Kinto REST API;
sudo apt-get install inotify-tools
* [https://github.com/Kinto/kinto.js kinto.js] is the offline-first client for Kinto.
</pre>


Then start an infinite loop with <code>inotify-wait</code>:
With the help of Babel and browserify, a bundle is generated for Firefox with the minimum transpilation possible (eg. CommonJS require, ES7 decorators).


<pre>
==== kinto.js ====
while true; do
    # Wait for a change
    inotifywait -q -e create,modify,delete -r ~/Code/Mozilla/kinto.js/dist
    # Execute these commands
    cp ~/Code/Mozilla/kinto.js/dist/moz-kinto-offline-client.js services/common/kinto-offline-client.js
    ./mach xpcshell-test services/common/tests/unit/test_storage_adapter.js
    ./mach xpcshell-test services/common/tests/unit/test_kinto.js         
done
</pre>


[http://makina-corpus.com/blog/metier/2013/utiliser-inotify-pour-tester-en-continu-son-code Source: Antoine Cezar]
From the kinto.js repo, generate the <code>moz-kinto-offline-client.js</code> file:
 
=== Submit patch ===
 
> Patch are contributed to kinto.js and kinto-client.js, which are first released on NPM.
 
'''DO NOT''' land files that are not tagged officially on upstream repositories.
 
==== Become a contributor ====
 
* Generate a SSH key
* Open a bugzilla ticket to request Level-1 access
* https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/
* Get a vouch (*optional for level1*)
 
Configure SSH key for hg:


<pre>
<pre>
Host hg.mozilla.org
$ npm run dist-fx
  User user@server.com
  IdentityFile ~/.ssh/contrib_moz
</pre>
</pre>


==== Run integration tests: «Try» ====
And overwrite it in the Firefox code base:
 
   
See https://wiki.mozilla.org/ReleaseEngineering/TryServer
 
Or use a ''gecko-dev'' fork from Github, and with install [https://github.com/mozilla/moz-git-tools moz git tools]
 
<pre>
<pre>
git push-to-try -t --rev master..HEAD ~/hg/mozilla-central/ -b do -p linux,linux64,macosx64,win32,win64 -u xpcshell -t none
$ cp dist/moz-kinto-offline-client.js ../mozilla-central/services/common/kinto-offline-client.js
</pre>
</pre>


==== Submit for review ====  
==== kinto-http.js ====


See http://mozilla-version-control-tools.readthedocs.org
From the kinto-http.js repo, generate the <code>moz-kinto-http-client.js</code> file:


<pre>
<pre>
# Commit with link to Bugzilla
$ npm run dist-fx
hg commit -m "Bug XXXXX - Upgrade <lib> to X.Y.Z"
 
# Submit to MozReview
hg push review
 
# Keep a bookmark of your branch to address review.
hg bookmark bug-XXXXX
 
# Go back to «master»
hg update central
</pre>
</pre>


Examples:
And overwrite it in the Firefox code base:
 
* https://reviewboard.mozilla.org/r/45445/
 
= Tips for Testing =
 
You can get tests (or Firefox) to give you more information on what the content signature verifier is doing by setting the NSPR_LOG_MODULES environment variable. For example:


<pre>
<pre>
EXPORT NSPR_LOG_MODULES=ContentSignatureVerifier:5,CSTrustDomain:5
$ cp dist/moz-kinto-http-client.js ../mozilla-central/services/common/kinto-http-client.js
</pre>
</pre>

Latest revision as of 09:58, 19 February 2021

Kinto Integration in Firefox

Kinto is a simple JSON storage service that was integrated into Firefox, and currently used for RemoteSettings.

Key features

  • Diff-based data synchronization
  • Offline persistence
  • Data integrity/signing
  • Peer review to publish data changes
  • Admin panel UI
  • Built-in Telemetry
  • Firefox Accounts Integration

Use Cases

Component Description Contact
RemoteSettings Firefox/RemoteSettings allows to ship data and files to Firefox. It is used for blocklists, security state, Activity Stream, Fennec assets and experiments etc. Mathieu Leplatre — :leplatrem

It used be supporting more use-cases in the past. See also Kinto at Mozilla

Usage

Two client libraries are vendored in Firefox:

HTTP client

const { KintoHttpClient } = Cu.import("resource://services-common/kinto-http-client.js", {});

const remote = "https://kinto.dev.mozaws.net/v1";
const headers = { Authorization: "Basic " + btoa("user:pass") };

const client = new KintoHttpClient(remote, {headers});
const records = await client.bucket("a-bucket")
                            .collection("a-collection")
                            .listRecords();

Offline client

const { Kinto } = Cu.import("resource://services-common/kinto-offline-client.js", {});

const remote = "https://kinto.dev.mozaws.net/v1";
const headers = { Authorization: "Basic " + btoa("user:pass") };

const kinto = new Kinto();
const bucket = "a-bucket";
const collection = kinto.collection("a-collection");

// Fetch/Publish changes.
const { ok } = await collection.sync({ bucket, remote, headers });

// Read local collection of records.
const records = await collection.list();

Resources

Contribute

Generate bundles

The Kinto client libraries are developed independently on Github:

  • kinto-http is the HTTP client for the Kinto REST API;
  • kinto.js is the offline-first client for Kinto.

With the help of Babel and browserify, a bundle is generated for Firefox with the minimum transpilation possible (eg. CommonJS require, ES7 decorators).

kinto.js

From the kinto.js repo, generate the moz-kinto-offline-client.js file: 

$ npm run dist-fx

And overwrite it in the Firefox code base: 

$ cp dist/moz-kinto-offline-client.js ../mozilla-central/services/common/kinto-offline-client.js

kinto-http.js

From the kinto-http.js repo, generate the moz-kinto-http-client.js file: 

$ npm run dist-fx

And overwrite it in the Firefox code base: 

$ cp dist/moz-kinto-http-client.js ../mozilla-central/services/common/kinto-http-client.js
Retrieved from "https://dev.wikimo.nonprod.webservices.mozgcp.net/index.php?title=Firefox/Kinto&oldid=1233809"