Firefox OS/Cloud Storage
Overview
This project aims at to support a framework for web apps to interact with cloud storage.
Cloud Storage framework wants to support following features
- Universal Storage API
Universal Storage API is a general API for web apps to access all kinds of cloud storage.
- Virtual Storage Interface
Virtual Storage Interface is plugin system for cloud storage framework to extend cloud storage option.
Planning Phases
- Phase 1
We will implement a read-only storage for the ready-made app to read the data via Device Storage API. For example, Video app can play a video in Dropbox even the file size is larger than internal storage without any app modification.
- Phase 2
Each user might have more than one storage account, so multi-storage support is an important feature for user to manipulate their storage more flexible.
Second, offline Support is for using storages even there is no network available (, and we can leverage Cache mechanism to implement this feature.)
From the perspective of storage provider, Flexible Storage Module is an easy way to install or update the storage plugin for 3rd party storage provider like Dropbox, Box.net, etc.
Multi Storage support will be handled by Account management. 3rd-party Storage plugin and Virtual Storage Interface are the key components to implement Flexible Storage Module.
- Phase 3
Read-Write support, just like its literal meaning, files can be read or written. Read-Write feature is our last mile to achieve our final destination, and Sync Mechanism should handle all possible conflict when any file is changed from different clients.
Cloud Storage Framework
Overall Architecture
Based on the planning phases, this is an overall architecture of Cloud Storage includes Gecko part and Gaia part.
First, Cloud Storage Configuration UI is for user to enable/disable cloud storage via storage management API. It could be in Settings app or any possible ways.
Second, Gaia apps can use Device Storage API to access the files in Cloud Storage, just like Video app, Music app and Gallery app did. Eventually, we have to extend Device Storage API for Cloud Storage, like getting the synchronisation status of a file.
Third, 3rd party storage provider can implement a new plugin for users to manipulate their storage via Virtual Storage Interface. The interface is for storage provider to define how a file is accessed including getting meta data, file content reading and writing.
Cache mechanism, sync mechanism, and account management are the modules in gecko to be hidden, so app developers don’t have to worry about the issues like how to resolve the conflict issue, data caching for performance, etc.
Universal Storage API
Universal Storage API is a general API for web apps to access all kinds of storage.
According to different purposes, Universal Storage API can be separated into following two categories
- Document based API
Document based API is used to help web apps save/operate their application data, such as indexedDB, on the cloud storage.
- Filesystem based API
Filesystem based API supports web apps to access cloud storage with filesystem operations, such as file create, delete, open, read, write, close, etc.
Reference for Filesystem based API: W3C FileAPI
filer
DeviceStorageAPI
Virtual Storage Interface
Virtual Storage Interface is plugin system for cloud storage framework to cooperate with different cloud storage.
Cache Architecture
Background
- Improve performance
- The response time of each file operation should be as short as possible to provide better user experience when users want to view a photo, listen an audio, or watch a video.
- Offline Support
- Cache store is a subset of cloud storage, so Cache system still can provide cached file content from cache store even no network connection available.
Architecture
- Cache Manager
- Block-based data management
- Dispatch low-level file operation from FUSE to Data/Meta Cache first
- Get data from Web Storage Manager if there is no data existed in Cache.
- Meta Cache
- Metadata for each file (mdate, cdate, size, etc.) should be cached in memory or Indexed DB.
- Data Cache
- Data Cache should store a file content requested by user for each file in order to avoid extra network transaction.
- Web Storage Manager
- Provide an interface for each web storage provider (Dropbox, Box.net, etc) to communicate with Cache Manager.
- 3rd party plugin could be in the form of add-on for developer to support new web storage easily.
- Cache Policy
- Cache store entry can be flexible to adjust for different requirement, i.e. storage size. Any new request can replace one existed entry if the maximum entry is exceeded. ‘existed entry’ has not been well-defined yet, and it depends on user story.
- Cache policy is a set of method to prefetch file content or meta aggressively, so the response time for each request can be improved.
Data Cache Implement Proposal
- File-based Cache
- minimum data chunk - File
- Block-based Cache
- minimum data chunk - Block (with fixed-size)
- File System level operation can be mapped to cloud storage.
| File-based | Block-based | |
|---|---|---|
| Cross Platform/Browser | Yes | TBD... |
| Response Time for large-size file (ex: video) |
poor | good |
| Response Time for small-size file (ex: photo, audio) |
good | good (depends on block-size) |
| Implementation Idea | Gaia: JS Library (Filer) | Gaia: TBD... Gecko: FUSE |
| Related Web API | Blob, MSE(?), ... | Blob, Device Storage, ... |
Reference
Proposed Solution(s)
FUSE based solution
Mounting Cloud Storage to device filesystem through the FUSE mechanism.
Benefits
- DeviceStorageAPI provides well defined filesystem based API to access files/directories on internal storage and SD card. Once FUSE based solution supported, Web app can used DeviceStorageAPI directly to access cloud storage without any modification.
- FUSE interface is a good reference for Virtual Storage Interface. By propagating FUSE interface to FirefoxOS, FirefoxOS can easy connect to many kinds of storage, such as cloud storage, NAS, etc. Here is an commercial example.
Drawbacks
- DeviceStorageAPI is not a standard API for all browser.
- FUSE is not supported in all platforms.
Prototype
- MozCloudStorage github repo
- CloudStorageServiceAPI (this is only for prototype)
CloudStorageServiceAPI is a WebAPI for enabling/disabling cloud storage.
enum CloudStorageType {
"Dummy", "Dropbox",
};
Promise<boolean> enable(DOMString cloudName, CloudStorageType cloudType, DOMString accessToken)
Promise<void> disable(DOMString cloudName)
JS library solution
Benefits
Drawbacks
Framework Issues
- Cloud Storage Account management and Authentication
- Security issues
Issue Study
- Video Support with MSE
- Media Source Extensions
- The challenge with HTML5 audio and video is still the fragmented support for audio and video formats.
- In Firefox browser, the maximum size of each fragment is 75MB. Getting QuotaExceededError if the fragment size is over 75MB.
- Fragmented Video Format - the video MUST be in fragmented format, and this video can be played well.
- Non-fragmented Video - this more general format can be played in MSE way. Browser treats a non-fragment video as a one big fragment file because the maximum size of a non-fragment video is 75MB.
Related Use cases
Play media files saved on cloud
John saves his media files on dropbox or any other cloud storage and wants to play these media online on his mobile device.
In traditional, John needs to do with following steps
- Open the cloud storage app, such as dropbox and google drive, and download the media files to the mobile device.
- Open the player app, such as Gallery and Music, and play the media files locally.
On FirefoxOS device, the steps can be reduced to
- Open the player app and play the media files on cloud storage directly.
Bob wants to take a picture and share it with friend on flickr.
In traditional, Bob needs to do with following steps
- Open the camera app to take a picture and save the file on the device.
- (Optional) Open a third party app to edit the picture on the device.
- Open the flickr app to upload the file and set it as shared.
On FirefoxOS device, the steps can be reduced to
- Open the camera app to take a picture and save and share the file on cloud storage directly.
- (Optional) Open a third party app to edit the picture on cloud storage directly.
File management between clouds
Amy wants to copy a finished work document form her personal dropbox space to the company google drive space
In traditional, Amy needs to do with following steps
- Open the dropbox app and download the document file to device.
- Open the google drive app and upload the document file.
On FirefoxOS device, the steps can be reduced to
- Open a file browser app and copy the document from dropbox to google drive directly.