Breakpad/Status Meetings/2015-05-27

« previous meeting — index – next week » ^create?

Meeting Info

Breakpad status meetings occur on Wed at 11:00am Pacific Time.

Conference numbers:

   Vidyo: Stability 
   650-903-0800 x92 conf 98200#
   800-707-2533 (pin 369) conf 98200# 

IRC backchannel: #breakpad
Mountain View: Dancing Baby (3rd floor)

Operations Updates

AWS

• PostgreSQL RDS DB has been migrated
  • ~6 hours for "core" DB so all apps are functional
  • ~30 hours for historical data
  • migration is scripted https://github.com/mozilla/socorro-infra/pull/119
  • can we catch up and stay up-to-date by running processor only in AWS for now?
    • push from PHX crashmovers -> AWS RMQ to keep up
    • "re-"process from S3 and backfill to cover gap

• ES migration
  • plan is to use special-purpose processors in AWS
    • https://github.com/mozilla/socorro/pull/2773
  • Also see "ES" section below.

• how bad is running multiple crontabbers?
  • could we get duplicate data, e.g. ADI?
  • this will happen the way deploys happen right now
    • autoscaler spins up new instance before shutting down old one

• load testing
  • make sure to snapshot DB first, and use non-prod S3 bucket so as not to pollute prod!
  • how many crashes should we try submitting?
  • can we get datadog statsd landed first?
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1167767

• what else is left?
  • https://bugzilla.mozilla.org/showdependencytree.cgi?id=1118288&hide_resolved=1

DB

• PHX PostgreSQL memory has been stable, but watching it
  • https://graphite-phx1.mozilla.org/render/?width=777&height=573&_salt=1431452379.214&from=-2weeks&target=hosts.sp-processor*_phx1_mozilla_com.memory.memory.used.value

ES

• We can now spin functional ES clusters up in a standardised, automated fashion. https://github.com/mozilla/socorro-infra/pull/137

Other

• two production problems identified yesterday:
  • bug 1156961 - 38.0.5b99 detection
    • need to land https://github.com/mozilla/socorro/pull/2807
    • we need to re-process and backfill too
  • bug 1168511 - some crontabber jobs stuck for several days
    • culprit is invalid unicode, which PG accepts but JSON functions fail on
    • did nagios alert on this over the weekend+holiday?

Project Updates

Socorro Bug Tracker
this week's bugs

PR Triage

• https://prs.paas.allizom.org/mozilla/socorro

Travel, PTO, etc

• rhelmer out a few hours each on Thursday and Friday
• phrawzty out all next week.