Firefox OS Data Sync

Overview

At a high level, this project aims to move the data generated and managed by Firefox OS devices to the cloud allowing users to access a synchronized version of this data at any time initially from any Firefox product and potentially from any web enabled device.

The product

The product vision is built on top of three major concepts fully aligned with Mozilla's mission.

• User choice

We should offer to users the ability to decide where they want to store their data. On currently existing platforms users are tied to a specific storage: on iOS people are tied to iCloud, on Android, to Google Drive, etc. We want to give users the ability to choose the cloud storage provider they want. Ideally, we want to also be able to give them the choice to use self hosted storage like ownCloud. But this is still under discussion. Mozilla might also provide cloud storage space for users as one of these choices, but this is also still to be decided.

• User privacy

In order to ensure that the data that the user sends to the cloud is protected and no one else other than the user can read it, the Firefox OS in the Cloud client solution should allow users to opt-in to encrypt the data on the client side and store it encrypted on the selected cloud storage provider. Not even Mozilla should be able to read this data or store it unencrypted. All the encryption and decryption should happen on the client side.

• User identity

We want to use Firefox Accounts as the authentication mechanism for this service. Once the user links their chosen cloud storage provider credentials to her Firefox OS in the Cloud account, all that she needs to do to authenticate herself from new devices accessing her Firefox OS in the Cloud account is her Firefox Accounts credentials.

Use cases

Messaging application

Alice uses her Messaging app to send and receive SMS, MMS and IM. She accesses this application from her Firefox OS tablet, her desktop browser and her Android phone. She can see and manage the history and content of the messages sent and received from any of these devices. She can continue writing an IM that she started typing on her Android phone on her desktop browser app.

Media files

Bob uses his Music app to listen to music and audio files. He keeps a library with his preferred titles. He adds new songs from his desktop browser. When he uses his Firefox OS device, he can listen to these new songs if he is online. He can also choose to download them so he can play them offline.

Question: Does the proposed V3 architecture buy us anything in terms improved offline experience here? -Stephany

Alice listens to her favorite knitting podcasts -- Knitting Pipeline and The Knit Girllls -- in her web browser (while knitting beside her laptop during business travel) or via the Apple TV > Podcasts section, when she is at home. Alice can access both podcasts on the Web and in iTunes. She has a Lenovo PC so doesn't really use iTunes outside of Apple TV. When Alice sits down in her plane seat for her next business trip, plugs in her headphones and clicks the play button on the video clip embedded in the podcast's web page, Firefox Cloud asks if she'd like the video to be backed up so she can pick it up anywhere (on her phone, in case her laptop battery dies; on her laptop when it's offline during the flight with alleged but nonexistent Internet service), where she left off.

Backup

Alice purchases a new Firefox OS phone (\o/). She already owns a Firefox OS tablet and she wants to have the same experience and data in both devices. She enters her Firefox Accounts credentials while configuring her new device. Her new device installs all the applications that she has on her tablet, the homescreen wallpaper, the passcode for the lockscreen, the notification sounds. When she opens the Gallery app in her new device, she is asked if she wants to access her photo collection from her new device.

File sharing

Bob wants to share a file between his desktop and his mobile phone. He accesses dummysharingservice.com, logs in with his Firefox Accounts credentials and uploads the file from his desktop. He goes to his mobile phone and logs in with the same Firefox Accounts email and downloads the file in his mobile. Now he wants to share the file with Alice. He accesses dummysharingservice.com again and uses the sharing option to send a notification about the shared file to Alice's email. Alice receives this notification and accesses the sharing service. She logs with her Firefox Accounts email and downloads the file shared by Bob.

Email

Alice has a Firefox OS device and wants to back up all of her email photo attachments, from multiple email accounts, to Dropbox and Spider Oak. She sees a suggestion for "helpful services" in the email UI, taps it, peruses services, and selects those for Dropbox and Spider Oak. Alice feels safer knowing that every photo she's ever received in an email is automatically backed up.

After a few months, Alice decides to do the same for document attachments, but only for her work email and to her organization's enterprise Box back-up, which she is required to use as an employee.

Note: Francis Djabri and Stephany Wilkes are currently working on this and presented initial designs to the UX team in early March, 2015. Please contact either of them for more information.

Other use cases

You can see the result of a previous discussion around data sync use cases here.

Data types

Looking at the proposed use cases and the nature of the data to be synchronized we can group it in:

Browser/System data

This includes things like browsing history, bookmarks, passwords, form autocomplete data, list of installed apps, system settings, etc.

In-app generated data

This includes all the data that is not owned by the system. And can be also grouped in:

• Files

Things like photos, music and other media files.

• Documents

Everything that fits on an indexedDB like database. Including in-app specific settings or things like SMS, call history, alarms, etc.

Sub-product ideas

We started exploring three angles in proof-of-concept stage.

Syncto: passwords, bookmarks, etc.

Given that Firefox Sync already allows users to sync certain types of data between their various instances of Firefox (be they Desktop, Android, or iOS), it makes sense to add FxOS to the existing FxSync product.

Cloud Storage: photos, songs, etc.

FxOS already allows users to store and view media files using storage that is connected to the device (for instance, an SD-card). Mounting remote cloud storage space (for instance the user's Dropbox or ownCloud account) as a (cached) virtual storage device makes it possible to browse the files on there 'as if' they were on a local storage device.

Kinto.js: valuable user data in general

A sync client that can run in any browser. It can be used in FxOS (system) apps that store any kind of user data, to sync that data to a server that implements Kinto's cross-origin http API.

The solution(s)

For Browser/System data

For Browser/System specific data we chose to use the existing Firefox Sync platform. That will give users a way to backup Firefox OS system data like the list of installed apps, the browser history, their bookmarks, etc. and to access this data from any Firefox product (Desktop, Android, iOS, OS). It seems that we will probably need to rewrite a lot of the code that currently exists in Gecko and we will need to expose a Gaia only API to allow apps to feed the platform with this kind of data from the content side. We will be adding more details once we figure out all the required work.

For in-app generated data

We are still not sure about which approach we should take to allow the apps to store and synchronize in-app generated data with a remote storage. We know some requirements we would like to address though.

Requirements for in-app data sync

• Authentication

Firefox Accounts should be the authentication mechanism used for this. Firefox Accounts is also the authentication mechanism currently used by Firefox Sync, so it makes a lot of sense to have the same for in-app generated data. Using Firefox Accounts would enable us to do the data encryption in the client without worrying about storing any private key or secret in the clients. We can obtain a key derived from the user's Firefox Accounts and encrypt the local data on the fly before sending it to the cloud. Assuming that we finally allow 3rd party storage providers, the authentication keys for these providers selected by the user could be stored in a Mozilla server also encrypted with a symmetric key that the client will provide on every sync request. That way if the Mozilla server is compromised, the attacker won't get access to the remote storage accounts.

• Files and documents

The solution that we provide to developers should allow them to store and synchronize files and documents, understanding files as data stored "directly" on the file system (i.e. via DeviceStorage or even the File System APIs and documents as the data stored in a local database most likely an IndexedDB one.

• It should be cross browser.

Whichever solution we find for this, it should work in all browsers. If we end up creating a JS library, we need to be sure that it works (or will work, thinking about IndexedDB support) in all the major browsers. We believe that we can avoid adding a new web API for this, but in case that we finally need to do it, it would be great if we could standardize it and avoid repeating the DataStore fiasco.

• Offline first.

Ideally we should provide a solution where consumers can store the data locally first and then specify that the data needs to be synchronized with a given remote endpoint. The app developer shouldn't need to know the underlying details of the sync protocol that we choose to implement. We should provide them with an abstraction to request this synchronization in the form of a JS library.

• Avoid enforcing another client storage solution.

We already have enough storage APIs in the client side at least on Firefox OS to add another one. We don't think it is a good idea to force developers to use another API for remote data synchronization. Instead, we would prefer a solution that allows developers to keep using the API of their current data source (let's say IndexedDB) for adding, editing and removing records and performing searches. While we can provide an easier abstraction of the current APIs (IndexedDB is known for being a powerful but complicated API), we should still allow developers to keep accessing and using them.

• Avoid data duplication.

One of the issues that we currently have with DataStore is that we potentially create several copies of the same data across Firefox OS. Ideally, we should not require to do the same with the solution that we choose for the client side of this system. So we should either have a solution that allow us to keep using the same data sources that we use currently but somehow adding the sync capabilities to it or a way to migrate the existing data to a new data source with already sync enabled capabilities. We will certainly need to add meta information along with the existing data that we have in our apps, but this data should have an unique source.

Proposals for in-app data

Proposal 1: Generic storage and storage provider proxy

The Cloud Services team has been working on a generic storage service to allow 3rd party apps to store and synchronize arbitrary data, attached to a Firefox account. They've also supported the idea of having an intermediate service (could be part of the previous one) that would act as a proxy for different storage providers (one of them could be the Mozilla generic storage service mentioned before).

With that kind of services we could have a high level architecture like the following:

The code and documentation for this generic storage can be found at:

• Blog post
• https://github.com/mozilla-services/cliquet
• https://github.com/mozilla-services/kinto

Proposal 2: Universal Storage API and Virtual Storage Interface

The Taipei team is currently working on a proposal to extend the DeviceStorage API to allow remote storages. You can read more about this here

Specific protocols

For some cases like the Email (IMAP, POP3) or Calendar (CalDav) apps we are already implementing specific sync protocols. We need to keep supporting and adding more of these specific sync protocols. For now the list of identified ones are:

• Gmail for Contacts. We are currently importing contacts only.
• Outlook for Contacts. We are currently importing contacts only.
• CardDav for Contacts. One of the most wanted features (269 votes so far). Check bug 859306.

Related work

• Mozilla MakeDrive
• PouchDB
• Dexie.js
• RemoteStorage
• Lubdub
• Ember-sync
• Hoodie
• SyncIt
• SyncedDB

Toolkit team

Bugs

Meta bug

Sprints

• FxOS-S4
• FxOS-S5

Meetings

When

• Every two weeks at 10am CEST / 16pm CST

Where

• ferjm's Vidyo room
• Public link: https://v.mozilla.com/flex.html?roomdirect.html&key=8SacsQs9fUmW

What

• Firefox OS Data Sync/Meetings/2015-05-13
• Firefox OS Data Sync/Meetings/2015-06-17