QA/Sync/Test Plan/grinder tests

Overview

We would like to have a test framework that tests the sync servers for load and functionality. For this, we will use Grinder in order to mimic the action of many firefox clients simultaneously.

We are trying to accomplish the following goals

• Create a continuous baseline testing framework (tied to Jenkins)
• Be able to answer questions about current and future growth.
  • How can our app fail now?
  • What types of services can we support in the future?

Test Cases

Questions Addressed By This System

Load Testing

• How does the app fail under high load conditions? How does it fail?
  • From a large data input?
  • From a lot of connections?
• Which functions and potential use cases create a particularly high amount of load?
  • High amounts of registration? (and corresponding initial sync?)
  • Lots of empty requests? (as generated by instant sync?)
• Are any functions effected sooner by high load? (I.E. Are there any unexpected bottlenecks?)
• How do these services scale to
  • More Features
  • More Users

Regression Testing

• Baseline tests on a sandboxed environemnt
  • Did the code we just checked in have bugs that effect performance?
• Large scale weekly testing
  • Are we confident this new release will work will in production?

Test scripts

The objective of these scripts will be to match our use of the sync api as closely as possible to that of the firefox client. The Sync API is a fairly general storage solution - the behavior of the firefox client needs to be matched. We will try to do this with the following functions:

• Create Users
• Sync (Adjustable size sync)
  • Will contain data based on frequency of all 5 types of collections (Tabs, Bookmarks, History, Preferences,
• Empty Sync (I'm checking in without data)
• Mobile Sync (Takes data in chunks of 50)
• Reset Sync Command (Clears DB and creates new records)
• Change sync key

New functions can be added as new services are rolled out.

Script use pattens

We currently have the ability to go through the logs and see how the sync server is being used. We can generate a profile based on this and do load testing that way. In addition, we can add scenarios that model other use cases. For example:

• A period with above normal registrations
• A period with high amounts of data payload
• Lots of small requests (what instant sync would give us)

A config file or some time of user interface will be required so that we can "turn knobs" to simulate different scenarios. The possibility to test different scenarios is limitless. One of the main uses for this down the road could be to test the potential impact of new services as they are developed.

Technologies

Testing Technologies

• Grinder
  • This will be the main test harness. We will write scripts against it using jython and use its built in tools in order to distribute the tests to multiple load generators
  • Advantages
    • Distributed (We can use multiple machines to hit a server)
    • Performant (No overhead of booting firefox)
    • Built with HTTP based testing in mind (ideal for REST services including SYNC and beyond)
• MongoDB
  • A database with a REST api will be used for data verification. A percentage of requests will get stored in a Mongo database in effort to make sure that our data is accurate at all loads
  • Advantages
    • Heavy - it is built to handle many reads/writes per second. Hopefully we won't be load testing MongoDB.
    • REST api we can perform the same steps to communicate with sync as we can to communicate with the database

Important features of the sync system

• Load Balancer - Zeus
  • Will protect us from connection overrun
  • Questions about how it responds under high load condition and how firefox responds to that
  • Extremely scalable (will likely never be a bottle neck)
• Back end application
  • Currently in PHP, will be migrated to python
  • Questions about through put of data
• Database Server
  • Currently has a large cache (this means we need a lot of tests before we are truly in a production type environment)