TestEngineering/Services/TSVerifierSyncTestEnvironments: Difference between revisions
< TestEngineering | Services
Jump to navigation
Jump to search
Line 282: | Line 282: | ||
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxErrorFileOutput-ProtobufEncoder | *** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxErrorFileOutput-ProtobufEncoder | ||
** Sandboxes | ** Sandboxes | ||
** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes | *** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes | ||
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf | *** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf | ||
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagemetaglobal.cbuf | *** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagemetaglobal.cbuf |
Revision as of 02:48, 18 September 2014
- NOTE: We currently have two Verifier stacks in Stage (and probably Production):
- The standalone Browser_ID Verifier stack: See that Verifier Stage section below...
- A Tokenserver+Verifier stack: See the TokenServer Stage section below...
Tokenserver, Verifier, and Sync Production Environments
- TokenServer: https://token.services.mozilla.com
- Verifier: https://verifier.accounts.firefox.com
- Sync 1.5 Nodes (for now in AWS):
- sync-1-us-east-2.sync.services.mozilla.com
- sync-2-us-east-2.sync.services.mozilla.com
- There are many more than this now...
- QA Access: not allowed
- Pointing Fx Nightly to Production FxA/Sync:
- Set/Verify the following Firefox configs:
- services.sync.clusterURL = (should get automatically set by the TokenServer)
- services.sync.tokenServerURI = https://token.services.mozilla.com/1.0/sync/1.5
- services.sync.fxaccounts.enabled=true (NEW FOR NIGHTLY FF 29 - SHOULD BE SET BY DEFAULT)
- services.sync.log.appender.file.logOnError = Yes
- services.sync.log.appender.file.logOnSuccess = Yes
- services.sync.log.appender.file.level = Trace
- identity.fxaccounts.auth.uri = https://api.accounts.firefox.com/v1
- identity.fxaccounts.remote.uri = https://accounts.firefox.com/?service=sync&context=...
- Set/Verify the following Firefox configs:
TokenServer+Verifier Stage Environment
- NOTE: In an upcoming deployent, the Verifier functionality will be rolled into the Tokenserver stack of services.
- URLs
- TokenServer: https://token.stage.mozaws.net
- Verifier: https://verifier.stage.mozaws.net
- IdP: http://mockmyid.s3-website-us-west-2.amazonaws.com
- Version:
TokenServer: rpm -qa | grep -E 'token|browser'
- AWS
- Shared:
- shared-elasticsearch
- shared-elasticsearch
- shared-elasticsearch
- shared-elasticsearch
- shared-rabbitmq
- shared-rabbitmq
- shared-bastion
- shared-heka
- Shared:
- tokenserver app server (1 or more instances behind a CF stack and ELB)
- tokenserver db (1 large DB instance behind RDS)
- Files
- /opt/aws
- /opt/ec2
- /opt/openresty
- /opt/rh
- /opt/stackdriver
- /data/tokenserver
- esp. /data/fxa-browserid-verifier/config/settings.json
- /data/hekad
- /data/fxa-browserid-verifier
- /etc/heka.d
- /etc/puppet
- esp. /etc/puppet/yaml/app
- /etc/puppet/yaml/app/fxa.stage.yaml
- /etc/puppet/yaml/app/fxa_verifier.stage.yaml
- /etc/puppet/yaml/app/tokenserver.stage.yaml
- /etc/puppet/yaml/app/tokenserver.yaml
- Processes
- tokenserver app server:
- Search for token, circus, nginx, gunicorn, python, hekad, node
- tokenserver app server:
- Logs
- /media/ephemeral0/logs/tokenserver/token.error.log
- /media/ephemeral0/logs/tokenserver/token.log.*
- /media/ephemeral0/logs/tokenserver/process_account_deletions.error.log
- /media/ephemeral0/logs/tokenserver/process_account_deletions.log
- /media/ephemeral0/logs/tokenserver/purge_old_records.log
- /media/ephemeral0/logs/tokenserver/purge_old_records.error.log
- /media/ephemeral0/nginx/logs/default.access.log (not in use)
- /media/ephemeral0/nginx/logs/default.error.log (not in use)
- /media/ephemeral0/nginx/logs/tokenserver.access.log
- /media/ephemeral0/nginx/logs/tokenserver.error.log
- /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
- /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
- /var/log/circus.log
- /var/log/hekad/tokenserver.stdout.log
- /var/log/hekad/tokenserver.stderr.log
- Hekad
- /etc/puppet/modules/hekad
- QA Access via a Bastion Host
- SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.
- Firefox Configs
- services.sync.clusterURL should get automatically set by the TokenServer
- services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5
Verifier Stage Environment
- NOTE: In an upcoming deployment, the Verifier functionality will be rolled into the Tokenserver stack of services.
- URLs
- Version:
- rpm -qa | grep verifier
- AWS
- Shared:
- shared-elasticsearch
- shared-elasticsearch
- shared-elasticsearch
- shared-elasticsearch
- shared-rabbitmq
- shared-rabbitmq
- shared-bastion
- shared-heka
- Shared:
- fxa-browserid_verifier or fxa-bv-stage (usually a single instance)
- Files
- /data/fxa-browserid-verifier
- esp. /data/fxa-browserid-verifier/config/settings.json
- /data/hekad
- /opt/aws
- /opt/ec2
- /opt/openresty
- /opt/rh
- /opt/stackdriver
- /etc/heka.d
- /etc/puppet
- esp. the /etc/puppet/yaml/app files
- /etc/puppet/yaml/app/fxa.stage.yaml
- /etc/puppet/yaml/app/fxa.yaml
- /data/fxa-browserid-verifier
- Processes
- Search for node, heka, nginx, circus
- Logs
- /media/ephemeral0/fxa-browserid-verifier/verifier_err.log
- /media/ephemeral0/fxa-browserid-verifier/verifier_out.log
- /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
- /media/ephemeral0/nginx/logs/fxa-browserid-verifier.access.log
- /media/ephemeral0/nginx/logs/default.access.log (not in use)
- /media/ephemeral0/nginx/logs/default.error.log (not in use)
- /media/ephemeral0/squid/access.log
- /var/log/circus.log
- /var/log/hekad/fxa-browserid_verifier.stderr.log
- /var/log/hekad/fxa-browserid_verifier.stdout.log
- Hekad
- /etc/puppet/modules/hekad
- /data/hekad
- QA Access via a Bastion Host
- SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.
- Firefox Configs
- services.sync.clusterURL should get automatically set by the TokenServer
- services.sync.tokenServerURI = https://token.stage.mozaws.net/1.0/sync/1.5
- Quick verifications
Browser: https://verifier.stage.mozaws.net/ returns "Method Not Allowed" $ curl https://verifier.stage.mozaws.net returns "Method Not Allowed" $ curl -I https://verifier.stage.mozaws.net HTTP/1.1 405 Method Not Allowed Cache-Control: no-cache, max-age=0 Content-length: 18 Content-Type: text/plain Date: Wed, 23 Jul 2014 20:19:40 GMT Connection: keep-alive
Sync 1.5 Stage Environment
- This is a work in progress. Right now, there are only a small number of sync nodes (instances) in AWS.
- URLs
- Versions
Server-Syncstorage (sync node): rpm -qa | grep syncstorage Example: server-syncstorage <latest version>
- AWS
- Search for sync node instances in US East: "stage-sync-node-X"
- Each node is a specific large instance (mixed m3 and c3)
- Files
- /data/server-syncstorage/*
- /var/log/nginx
- /etc/puppet
- Processes
- Search for sync, mysql, circusd, hekad, nginx, memcached
- Logs
- /media/ephemeral0/logs/
- /media/ephemeral0/logs/nginx/access.log
- /media/ephemeral0/logs/nginx/error.log
- /media/ephemeral0/logs/sync/sync.err
- /media/ephemeral0/logs/sync/sync.log
- /var/log/circus.log
- /var/log/hekad/sync_1_5.stderr.log
- /var/log/hekad/sync_1_5.stdout.log
- Hekad
- /etc/heka.d/sync_1_5.toml
- QA Access via a Bastion Host
- SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance.
- Firefox Configs
- services.sync.clusterURL should get automatically set by the TokenServer
- Note: There is no longer a Sync 1.1 Stage environment.
Loads V1 Services Cluster Environment
- Details on the Loads V1 cluster are available at this link:
Monitoring the Stage Environment
- Stackdriver:
- Stage main: https://app.stackdriver.com
- https://app.stackdriver.com/groups/4388/stage-services-tag-sync15
- https://app.stackdriver.com/groups/4389/stage-services-tag-sync15/stage-services-sync15/fxa-verifier
- https://app.stackdriver.com/groups/4391/stage-services-tag-sync15/stage-services-tag-sync15/sync-storage
- https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
- https://app.stackdriver.com/monitoring/3827/tokenserver
- https://app.stackdriver.com/instances/<AWS instance>
- Also, make use of the Services, Groups, and Dashboards for more specific links
- Example: Services > https://app.stackdriver.com/services/nginx
- Example: Groups > https://app.stackdriver.com/groups/4390/stage-services-tag-sync15/stage-services-sync15/tokenserver
- Kibana
- https://kibana.shared.us-east-1.stage.mozaws.net/
- https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_status.json
- https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_http_errors.json
- https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_mysql_slow_queries.json
- https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/sync_app_logs.json
- https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/tokenserver_http_status.json
- Heka
- https://heka.shared.us-east-1.stage.mozaws.net/
- or https://heka.shared.us-east-1.stage.mozaws.net/#health
- Filters
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/TokenServerHTTPStatus
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
- Outputs
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxErrorFileOutput
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-SlowQueryFileOutput
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerLocalVerifierFileOutput
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-FileOutput
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/Sync-1_5-NginxFileOutput
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerFileOutput
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/TokenServerNginxFileOutput
- Encoders
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-FileOutput-ProtobufEncoder
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerNginxFileOutput-ProtobufEncoder
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-SlowQueryFileOutput-ProtobufEncoder
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerLocalVerifierFileOutput-ProtobufEncoder
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/TokenServerFileOutput-ProtobufEncoder
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxFileOutput-ProtobufEncoder
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/Sync-1_5-NginxErrorFileOutput-ProtobufEncoder
- Sandboxes
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-SlowQueries/outputs/Sync-1_5-SlowQueries.Statistics.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/Sync-1_5-ResponseTime/outputs/Sync-1_5-ResponseTime.storagemetaglobal.cbuf
Monitoring the Production Environment
- Heka shared:
- Main: https://heka.shared.us-west-2.prod.mozaws.net/
- https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-FrequentUsersAggregator
- https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ResponseTime
- https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-SlowQueries
- https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-ActiveDailyUsers
- https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/Sync-1_5-HTTPStatus
- https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/TokenServerHTTPStatus
- StackDriver:
TokenServer, Verifier, and Sync 1.5 Dev Environments
- Versions
TokenServer: rpm -qa | grep token Example: tokenserver-svcops <latest version> Verifier: rpm -qa | grep verifier Example: fxa-browserid-verifier-svcops <latest version> Server-Syncstorage (sync node): rpm -qa | grep syncstorage Example: server-syncstorage <latest version>
- AWS
- This is part of the dev-lcip-org CloudFormation stack
- TokenServer: Search for the fxa-tokenserver instance
- SyncServer2: Search for the fxa-syncstorage instance
- SyncServer1: Search for the fxa-syncstorage instance
- LogServer: Search for the fxa-logbox instance
- This is part of the dev-lcip-org CloudFormation stack
- Files
- TBD
- Processes
- TBD
- Logs
- TBD
- QA Access
- SSH with AWS keys to the various instances
- Firefox configs:
- services.sync.clusterURL = https://sync1.dev.lcip.org/ (should get automatically set by the TokenServer)
- services.sync.tokenServerURI = https://token.dev.lcip.org/1.0/sync/1.5
OPs Mana and GitHub Pages
- NOTE: Talk to OPs for the links to Mana.
- Puppet Config: https://github.com/mozilla-services/puppet-config
- Cloud Formation: https://github.com/mozilla-services/svcops