QA/Sync/Test Plan/grinder tests

Overview

We would like to have a test framework that tests the sync servers for load and functionality. For this, we will use Grinder in order to mimic the action of many firefox clients simultaneously.

We are trying to accomplish the following goals

• Create a continuous baseline testing framework (tied to hudson)
• Be able to answer questions about current and future growth.
  • What types of services can we support now and what changes can we support in the future?

We are trying to answer the following questions

• How does the app fail under high load conditions? How does it fail?
  • From a large data input?
  • From a lot of connections?
• Which functions and potential use cases create a particularly high amount of load?
  • High amounts of registration? (and corresponding initial sync?)
  • Lots of empty requests? (as generated by instant sync?)
• Are any functions effected sooner by high load?
• How do these services scale?

Test Cases

We are trying to answer the following questions

• How does the app fail under high load conditions? How does it fail?
  • From a large data input?
  • From a lot of connections?
• Which functions and potential use cases create a particularly high amount of load?
  • High amounts of registration? (and corresponding initial sync?)
  • Lots of empty requests? (as generated by instant sync?)
• Are any functions effected sooner by high load? (I.E. Are there any unexpected bottlenecks?)
• How do these services scale to
  • More Features
  • More Users

This translates roughly to the following test cases

• Sustained Load
• Spike Load
  • There are currently client side issues that we need to keep track of with this. Based on the architecture the only place that a high amount of connections can hurt us is at the load balencer stage. Firefox needs to be able to deal with us saying "No you may not sync right now because we are busy"
• Generating Data (and verification)
  • Bookmarks
  • History
  • Bookmarks and History
  • Account creation
  • Account deletion
  • Tabs syncing (later priority)
  • Passwords Syncing (later priority)

These test cases need to mimic the actions of firefox as closely as possible. We will use grinder to simulate the actions of a firefox client.

Issues

• In order to validate data we need to store that data. In order to validate a sync servers worth of data we will need a lot of storage. (Some optimizations may be able to take place).
• We need to parallelize the load generation over multiple machines to match the capability of the server
  • Grinder supports multiple hosts
  • We need help to get the adequate hardware for this task