TestEngineering/Services/LoopServerLoadTesting: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(6 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== Quick Verification Of Stage Deployments ==
* This is a quick sanity test of the environment before getting started on load tests.
* Loop Server
For now, just run a quick loadtest 'make test'
cd loop-server
cd loadtests
make test SERVER_URL=https://loop.stage.mozaws.net


* Loop Client
<font size=6><span style="color:#FF0000">This wiki has moved!</span><br /><br />------> [[Loop/Test/Loadtesting]]</font>
Check https://call.stage.mozaws.net/config.js
Should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.stage.mozaws.net'};
and
WIP from the client team: end to end tests
Also:
https://call.stage.mozaws.net
curl https://call.stage.mozaws.net
curl -I https://call.stage.mozaws.net
and now the TOS: https://call.stage.mozaws.net/legal/terms/
 
* MSISDN Gateway
In the browser: https://msisdn.stage.mozaws.net
or do the following from a command line:
curl https://msisdn.stage.mozaws.net
curl -I https://msisdn.stage.mozaws.net
and
For now, just run a quick loadtest 'make test'
cd msisdn-gateway
cd loadtests
make test SERVER_URL=https://msisdn.stage.mozaws.net
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
    based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client
 
== Quick Verification of Production Deployments ==
* This is a quick sanity test of the environment for after each Production deployment.
* Loop Server
In the browser:
      https://loop.services.mozilla.com
      https://loop.services.mozilla.com/__heartbeat__
      https://loop.services.mozilla.com/push-server-config
or do the following from a command line:
curl https://loop.services.mozilla.com
curl -I https://loop.services.mozilla.com
Then run a few 'make test' commands from the loadtests folder:
make test SERVER_URL=https://loop.services.mozilla.com
Note: this does hit a live third-party server
Then perform actual loop testing via desktop (Aurora/Nightly so far) and FxOS (2.1)
Verify that requests and strings point to Production environments
 
* Loop Client
In the browser: https://call.mozilla.com
should return "Welcome to the Loop web client."
In the browser: https://call.mozilla.com/config.js
should return json similar to the following:
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};
In the browser: https://call.mozilla.com/VERSION.txt
should return the version and build string info
and now the TOS: https://call.mozilla.com/legal/terms/
curl https://call.mozilla.com
curl -I https://call.mozilla.com
Quick end-to-end tests:
Desktop: browser to browser
Desktop to FxOS
FxOS to Desktop
Two FxOS devices
 
* MSISDN Gateway
In the browser: https://msisdn.services.mozilla.com
or do the following from a command line:
curl https://msisdn.services.mozilla.com
curl -I https://msisdn.services.mozilla.com
Or
Run a single 'make test' command from the loadtests folder:
make test SERVER_URL=https://msisdn.services.mozilla.com
Note: this does hit a live third-party, so limit the check to a single run.
and
WIP using the following tools:
CLI: https://github.com/mozilla-services/msisdn-gateway/tree/master/tools/roundTrip
Web app: http://mozilla-services.github.io/msisdn-verifier-client/
    based on the this repo: https://github.com/mozilla-services/msisdn-verifier-client
 
== Load Test Tool Client/Host ==
* It is always best to configure an AWS instance as the host for all load testing.
* All load tests can now run on the localhost (the AWS instance) or against the new Loads Cluster. See the following links for more information:
** https://wiki.mozilla.org/QA/Services/LoadsV1ClientTestHost
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
 
== Installing Loop-Server and the Loads tool on Localhost or AWS ==
* Installation:
git clone https://github.com/mozilla-services/loop-server.git
cd loop-server
Note: You may want to install a specific branch for testing vs defaulting to Master
npm install
ulimit -S -n 2048
npm test *
cd loadtests
make build
make test
Coverage report can be found here:
/loop-server/coverage/lcov-report/index.html
* This step requires the redis server to be installed and running:
Mac:
brew install redis
redis-server /usr/local/etc/redis.conf
Ubuntu Linux:
sudo apt-get install redis-server
sudo /usr/bin/redis-server /etc/redis/redis.conf
sudo tail -f /var/log/redis/redis-server.log
RHEL Linux:
Install redis from here: http://download.redis.io/releases
then
/usr/local/bin/redis-server /home/ec2-user/redis-2.8.9/redis.conf
or similar
* Note: This will install a local copy of the Loads tool for use with the Loop-Server.
 
== Running the load test against the Loop-Server in Stage ==
* Stage environment:
$ cd loop-server/loadtests
$ make test
or
$ make test SERVER_URL=https://loop.stage.mozaws.net
$ make bench
or
$ make bench SERVER_URL=https://loop.stage.mozaws.net
Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.   
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
 
* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/
 
* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json
 
* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
 
* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
 
== Using the Loads V1 Services Cluster for the Loop-Server in Stage ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://loop.stage.mozaws.net
 
* By default, the Loop-Server in Stage is configured to talk to our mock server:
** https://loop-delayed-response.stage.mozaws.net/
 
* To hit the partner test servers, the following configuration file will need to be updated by OPs:
** /data/loop-server/config/settings.json
 
* Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
 
* REFs:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Environment
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Server_Configuration
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest
 
== Installing MSISDN-Gateway and the Loads tool on Localhost or AWS ==
* Installation:
** Install gmp, gmp-dev or gmp-devel
** Install ruby (very latest), ruby-dev or ruby-devel
** Install gem (required for fake_dynamo)
** Verify that gem is in your path
** Install redis-server to run the unit tests
 
* To install gmp
sudo yum -y install gmp, gmp-devel
or for Ubuntu
$ wget https://ftp.gnu.org/gnu/gmp/gmp-6.0.0a.tar.bz2
$ tar xvjf gmp-6.0.0a.tar.bz2
$ cd gmp-6.0.0
$ ./configure --prefix=/usr
$ make
$ make check
$ sudo make install
 
* To install ruby:
sudo yum -y install ruby, ruby-devel
or
sudo apt-get install ruby, ruby-dev
If this does not get you 1.9.3 or newer, then install manually:
Example:
    $ wget http://cache.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p547.tar.gz
    $ ./configure --prefix=/usr
    $ make
    $ sudo make install
    (because for rhel, the default ruby version is 1.8.x.)
REF:
Main: https://www.ruby-lang.org/en/downloads/
Dev Tools: http://rubyinstaller.org/downloads/
 
* To install gem:
Grab rubygems from here: http://rubygems.org/pages/download
cd to rubygems directory
$ sudo ruby setup.rb
 
* To install fake_dynamo:
You should not have to install fake_dynamo since it is now part of the repo installer.
But if you do:
$ sudo gem install fake_dynamo
REF: https://github.com/ananthakumaran/fake_dynamo
 
* Install the msisdn-gateway repo:
$ git clone https://github.com/mozilla-services/msisdn-gateway.git
$ cd msisdn-gateway
Note: You may want to install a specific branch for testing vs defaulting to Master
$ sudo make install
(There is a bug open about the requirement to install with 'sudo')
 
* Note: This will install a local copy of the Loads tool for use with MSISDN-Gateway.
 
* Unit testing
Get redis-server installed
Start the server in a separate terminal or in the background with logging active
$ make test
The coverage report is here: msisdn-gateway/coverage/lcov-report/index.html
 
* Bugs:
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036158
** https://bugzilla.mozilla.org/show_bug.cgi?id=1036174
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048451
** https://bugzilla.mozilla.org/show_bug.cgi?id=1048456
 
== Running the load test against MSISDN-Gateway in Stage ==
* Building the load tests
$ cd loadtests
$ make build
 
* To load test the Stage environment:
$ make test SERVER_URL=https://msisdn.stage.mozaws.net
$ make bench SERVER_URL=https://msisdn.stage.mozaws.net
Note: the current version of 'make bench' tends to use a lot of CPU and Memory on the localhost.   
The recommendation is to use 'make test' and 'make megabench' instead (see below)...
 
* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json
 
* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
 
== Using the Loads V1 Services Cluster for the MSISDN-Gateway ==
* By using the Loads Services Cluster, we can offload the broker/agents processes and save client-side CPU and memory.
* Changes were made to Makefile and the load test to use the cluster and some associated config files (for test, bench, megabench).
* Stage environment:
$ make megabench SERVER_URL=https://msisdn.stage.mozaws.net
 
* This environment also contains its own mock server: http://omxen.dev.mozaws.net
* The configuration file on the Stage server: /data/msisdn-gateway/config/production.json
 
* REFs
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Gateway_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#MSISDN_Mock_Server_Stage_Details
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
** https://github.com/mozilla/browserid-verifier/tree/master/loadtest
 
== Configuring The Load Tests ==
* Makefile
** The SERVER_URL constant can be changed.
 
* Config files
** For make test (Loop-Server and MSISDN-Gateway):
*** Number of hits
*** Number of concurrent users
 
** For make bench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
 
** For make megabench (Loop-Server and MSISDN-Gateway):
*** Number of concurrent users
*** Duration of test
*** Include file (this is code dependent)
*** Python dependencies (this is code dependent)
*** Broker to use for testing (leaves as defined for now - this is broker in the Loads Cluster)
*** Agents to use for testing (default is 5, max is currently 20, but depends on the number of concurrent load tests running)
*** Detach mode (leave as defined for now to automatically detach from the load test once it starts on the localhost)
*** Observer (this can be email or irc - the default is irc #services-dev channel)
 
* Loop-Server load test code
** The Loop-Server load test can not currently be configured in the code
 
* MSISDN-Gateway load test code
** The MSISDN-Gateway load test can be configured in the code - see the following lines:
** Test server URL: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L15
** Error percentages: https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
 
* General REFs:
** https://github.com/mozilla-services/loop-server/blob/master/loadtests/loadtest.py
** https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py
 
== Test Coverage and Stats ==
* Basic tweakable values for all load tests
** users = number of concurrent users/agent
** agents = number of agents out of the cluster, otherwise errors out
** duration = in seconds
** hits = 1 or X number of rounds/hits/iterations
 
* Loop-Server
** TBD
 
* MSISDN-Gateway
** TBD
 
== Analyzing the Results ==
* There are several methods and tools for analyzing the load test results.
 
* 1. Using the Loads Services Cluster dashboard
** All loads tests using this cluster generate a live report and a run report available on this site:
*** http://loads.services.mozilla.com
*** or http://ec2-54-212-44-143.us-west-2.compute.amazonaws.com (direct)
** You can quickly review the following here: Status, Configuration, Results, Custom Metrics, and Errors.
 
* Loop-Server Custom Metrics
** Opened web sockets
** Total web sockets
** Bytes/websockets
** addFailure (from the loads tool/client)
 
* MSISDN-Gateway Custom Metrics
** mt-flow
** ask-for-certificate
** try-wrong-code
** try-right-code
** momt-flow
** omxen-message-collision
** register
** unregister
** addFailure (from the loads tool/client)
 
* NOTE: If you want more details on the dashboard, please file an issue here: https://github.com/mozilla-services/loads
 
== Debugging the Issues ==
* There are several methods and tools for debugging the load test errors and other issues.
 
* 1. Important logs for Loop-Server (per server)
** /var/log/circus.log
** /var/log/loop_err.log
** /var/log/loop_out.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /var/log/nginx/access.log
** /var/log/nginx/error.log
 
* 2. Important logs for MSISDN-Gateway (per server)
** TBD
 
* Acceptable/Unacceptable Loop-Server errors:
hekad loop.stderr.log
The following are acceptable:
Decoder 'LoopServer-LoopServerDecoder' error: Failed parsing
Plugin 'AggregatorOutput' error: writing to heka.shared....
nginx logs:
The 200s (good stuff) and 101s (websockets) are acceptable.
The 499s are an artifact of the current load testing tool (V1).
You should only see them at the end of the load test.
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
Any percentage of 405s, 502s, or 503s is not acceptable.
Application logs:
Right now, we are getting a lot of 404s and 307s.
They all appear to be caused by bots. There is a bug open about this.
See the loop_server.out.log file.
/var/log/loop_err.log
The following are acceptable: connect: res.on("header"): use on-headers module directly
In the Loads Cluster dashboard, watch for the following errors/failures:
string indices must be integers
OR
No JSON object could be decoded
OR
'hawk-session-token'
 
* Acceptable/Unacceptable MSISDN-Gateway errors:
The updated load test does generate a certain percentage of errors:
https://github.com/mozilla-services/msisdn-gateway/blob/master/loadtests/loadtest.py#L19-L22
So, expect to see a predefined percentage of 204s and 400s, along with the usual 200s in the nginx access logs.
The msisdn-gateway app logs should be clean with just msisdn and test data.
 
== Monitoring Loop Stage ==
* Loads dashboard:
** http://loads.services.mozilla.com
* Cluster status
** Check directly from the Loads Cluster dashboard:
Agents statuses
Launch a health check on all agents
* and also on StackDriver:
** https://app.stackdriver.com/groups/6664/stage-loads-cluster
** https://app.stackdriver.com/groups/11042/loop-server/stage-loop
 
* For all other monitoring, see the following section:
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Stage_Environment
 
== Performance Testing Information ==
* TBD
 
== Details on the Load Test tool ==
* The documentation can be found here:
** https://loads.readthedocs.org/en/latest
* The repositories are here:
** https://github.com/mozilla-services/loads
** https://github.com/mozilla-services/loads-aws
** https://github.com/mozilla-services/loads-web
* The Services cluster is here:
** http://loads.services.mozilla.com
 
== Known Bugs, Issues, and Tasks ==
* Loop Server: Check the following repos:
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla-services/docs/issues
** https://github.com/mozilla/loop-client
** https://github.com/mozilla-services/msisdn-gateway/issues
 
* Bugzilla
** and in Bugzilla for non-Verified bugs: http://mzl.la/1qPxizw
 
* OPs and Infra
** https://github.com/mozilla-services/puppet-config/issues
** https://github.com/mozilla-services/svcops/issues
 
* Loads Tool and Cluster
** https://github.com/mozilla-services/loads/issues
** https://github.com/mozilla-services/loads-web/issues
** https://github.com/mozilla-services/loads-aws/issues
 
== References ==
* Repositories and Documentation
** Check here: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#References
 
* The QA Test Environments
** https://wiki.mozilla.org/QA/Services/FxATestEnvironments
** https://wiki.mozilla.org/QA/Services/TSVerifierSyncTestEnvironments
** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments
 
* Test Plans and other QA
** https://docs.google.com/a/mozilla.com/document/d/1xYZijHnJ1JeJhONm2AYvxSnD2tWztC16WGFyD5U_d6g/
** https://wiki.mozilla.org/Services/Loop/Try_Loop
** https://wiki.mozilla.org/Loop/QA
** https://wiki.mozilla.org/Loop/QA#Server
** https://intranet.mozilla.org/QA/Loop/MVP
 
* OPs pages for stats collection, logging, monitoring
** TBD

Latest revision as of 20:04, 26 August 2016

This wiki has moved!

------> Loop/Test/Loadtesting

Retrieved from "https://dev.wikimo.nonprod.webservices.mozgcp.net/index.php?title=TestEngineering/Services/LoopServerLoadTesting&oldid=1145591"