Firefox OS/Cloud Storage: Difference between revisions
EdenChuang (talk | contribs) No edit summary |
(Update Cache Architecture) |
||
Line 21: | Line 21: | ||
== Virtual Storage Interface == | == Virtual Storage Interface == | ||
Virtual Storage Interface is plugin system for cloud storage framework to cooperate with different cloud storage. | Virtual Storage Interface is plugin system for cloud storage framework to cooperate with different cloud storage. | ||
== Cache Architecture == | |||
=== Background === | |||
* '''Improve performance''' | |||
** measurement: response time of each file operation | |||
* '''Offline Support''' | |||
** Cache store is a subset of cloud storage, so Cache system still can provide cached file content from cache store even no network connection available. | |||
=== Architecture === | |||
* '''Cache Manager''' | |||
** Dispatch the request from Cloud Storage Interface to Data/Meta Cache first | |||
** Get data from Web Storage Manager if there is no data existed in Cache. | |||
* '''Cloud Storage Interface''' | |||
** The interface for using Cloud Storage cache is a general design. This can be used in JS Library or Gecko implementation. | |||
* '''Meta Cache''' | |||
** Each file has its meta data like mdate, cdate, size, etc. These information should be cached. | |||
* '''Data Cache''' | |||
** Data Cache should store a file content requested by user for each file in order to avoid extra network transaction. | |||
* '''Web Storage Manager''' | |||
** Provide an interface for each web storage provider (Dropbox, Box.net, etc) to communicate with Cache Manager. | |||
* '''Cache Policy''' | |||
** Cache policy is a set of method to prefetch file content or meta aggressively, so the response time for each request can be improved. | |||
=== Data Cache Implement Proposal === | |||
* File-based Cache | |||
** minimum data chunk - File | |||
* Block-based Cache | |||
** minimum data chunk - Block (with fixed-size) | |||
** File System level operation can be mapped to cloud storage. | |||
{| class="wikitable" | |||
|- | |||
! Header text !! File-based !! Block-based | |||
|- | |||
| Cross Platform/Browser || Yes || TBD... | |||
|- | |||
| Response Time for large-size file<br />(ex: video) || poor || good | |||
|- | |||
| Response Time for small-size file<br />(ex: photo, audio) || good || good<br />(depends on block-size) | |||
|- | |||
| Implementation Idea || Gaia: JS Library (Filer) || Gaia: TBD...<br />Gecko: FUSE | |||
|- | |||
| Related Web API || Blob, MSE(?), ... || Blob, Device Storage, ... | |||
|} | |||
== Proposed Solution(s) == | == Proposed Solution(s) == |
Revision as of 03:02, 20 May 2015
Overview
This project aims at to support a framework for web apps to interact with cloud storage.
Cloud Storage framework wants to support following features
- Universal Storage API
Universal Storage API is a general API for web apps to access all kinds of cloud storage.
- Virtual Storage Interface
Virtual Storage Interface is plugin system for cloud storage framework to extend cloud storage option.
Cloud Storage Framework
Universal Storage API
Universal Storage API is a general API for web apps to access all kinds of storage.
According to different purposes, Universal Storage API can be separated into following two categories
- Document based API
Document based API is used to help web apps save/operate their application data, such as indexedDB, on the cloud storage.
- Filesystem based API
Filesystem based API supports web apps to access cloud storage with filesystem operations, such as file create, delete, open, read, write, close, etc.
Reference for Filesystem based API: W3C FileAPI
filer
DeviceStorageAPI
Virtual Storage Interface
Virtual Storage Interface is plugin system for cloud storage framework to cooperate with different cloud storage.
Cache Architecture
Background
- Improve performance
- measurement: response time of each file operation
- Offline Support
- Cache store is a subset of cloud storage, so Cache system still can provide cached file content from cache store even no network connection available.
Architecture
- Cache Manager
- Dispatch the request from Cloud Storage Interface to Data/Meta Cache first
- Get data from Web Storage Manager if there is no data existed in Cache.
- Cloud Storage Interface
- The interface for using Cloud Storage cache is a general design. This can be used in JS Library or Gecko implementation.
- Meta Cache
- Each file has its meta data like mdate, cdate, size, etc. These information should be cached.
- Data Cache
- Data Cache should store a file content requested by user for each file in order to avoid extra network transaction.
- Web Storage Manager
- Provide an interface for each web storage provider (Dropbox, Box.net, etc) to communicate with Cache Manager.
- Cache Policy
- Cache policy is a set of method to prefetch file content or meta aggressively, so the response time for each request can be improved.
Data Cache Implement Proposal
- File-based Cache
- minimum data chunk - File
- Block-based Cache
- minimum data chunk - Block (with fixed-size)
- File System level operation can be mapped to cloud storage.
Header text | File-based | Block-based |
---|---|---|
Cross Platform/Browser | Yes | TBD... |
Response Time for large-size file (ex: video) |
poor | good |
Response Time for small-size file (ex: photo, audio) |
good | good (depends on block-size) |
Implementation Idea | Gaia: JS Library (Filer) | Gaia: TBD... Gecko: FUSE |
Related Web API | Blob, MSE(?), ... | Blob, Device Storage, ... |
Proposed Solution(s)
FUSE based solution
Mounting Cloud Storage to device filesystem through the FUSE mechanism.
Benefits
- DeviceStorageAPI provides well defined filesystem based API to access files/directories on internal storage and SD card. Once FUSE based solution supported, Web app can used DeviceStorageAPI directly to access cloud storage without any modification.
- FUSE interface is a good reference for Virtual Storage Interface. By propagating FUSE interface to FirefoxOS, FirefoxOS can easy connect to many kinds of storage, such as cloud storage, NAS, etc. Here is an commercial example.
Drawbacks
- DeviceStorageAPI is not a standard API for all browser.
- FUSE is not supported in all platforms.
Prototype
- MozCloudStorage github repo
- CloudStorageServiceAPI (this is only for prototype)
CloudStorageServiceAPI is a WebAPI for enabling/disabling cloud storage.
enum CloudStorageType {
"Dummy", "Dropbox",
};
Promise<boolean> enable(DOMString cloudName, CloudStorageType cloudType, DOMString accessToken)
Promise<void> disable(DOMString cloudName)
JS library solution
Benefits
Drawbacks
Framework Issues
- Cloud Storage Account management and Authentication
- Data cache/buffering
- Meta Cache - Each file has its meta data like mdate, cdate, size, etc. Meta Cache should store any meta information for each file or directory in order to avoid network transaction to retrieve a known information.
- Data Cache - Meta Cache talks about a file’s attribute, and Data Cache is for a file content. Data Cache should store a file content requested by user for each file in order to avoid network transaction to retrieve a partial file content as a block.
- Cache Policy - Cache store entry can be flexible to adjust for different requirement, i.e. storage size. Any new request can replace one existed entry if the maximum entry is exceeded. ‘existed entry’ has not been well-defined yet, and it depends on user story. Cache policy can be aggressive to prefetch file content or meta, so the response time for each request can be improved.
- Offline support - Any requested file content/meta will be stored in cache store, so the cache store is a subset of cloud storage. Cache system still can provide cached file content from cache store even no network connection. Considering error handling for new data/meta request is important in the case of no network connection.
- Security issues
Issue Study
Related Use cases
Play media files saved on cloud
John saves his media files on dropbox or any other cloud storage and wants to play these media online on his mobile device.
In traditional, John needs to do with following steps
- Open the cloud storage app, such as dropbox and google drive, and download the media files to the mobile device.
- Open the player app, such as Gallery and Music, and play the media files locally.
On FirefoxOS device, the steps can be reduced to
- Open the player app and play the media files on cloud storage directly.
Bob wants to take a picture and share it with friend on flickr.
In traditional, Bob needs to do with following steps
- Open the camera app to take a picture and save the file on the device.
- (Optional) Open a third party app to edit the picture on the device.
- Open the flickr app to upload the file and set it as shared.
On FirefoxOS device, the steps can be reduced to
- Open the camera app to take a picture and save and share the file on cloud storage directly.
- (Optional) Open a third party app to edit the picture on cloud storage directly.
File management between clouds
Amy wants to copy a finished work document form her personal dropbox space to the company google drive space
In traditional, Amy needs to do with following steps
- Open the dropbox app and download the document file to device.
- Open the google drive app and upload the document file.
On FirefoxOS device, the steps can be reduced to
- Open a file browser app and copy the document from dropbox to google drive directly.