

20170831_srecon17_1.5

20170831_srecon17_1.5

• Debug after bad deployments
• Intro
  • Context
  • Basic resolution
  • Main types of errors
• Debugging
  • 1 - Find the errors
  • 2 - Find the bad commit

Debug after bad deployments

Intro

Context

Bad deploy happens
* Many commits
* Many deploys happens, which was the cause ?

Basic resolution

1. Rollback
2. Debug
   + Find the errors
3. Fix
4. Restart

Main types of errors

• Perf regressions
• Broken pages
• Empty results
• Unexpected status code

Debugging

Find the NEW stack trace
Find the bad commit

1 - Find the errors

• Logs aggregation
• Identifying NEW stack traces
  -> Compare 2 ES Queries by example
  -> Deduplicat similar stack traces

2 - Find the bad commit

Aka: Wich of the 71 commits did that ?

https://github.com/pinterest/git-stacktrace

git stacktrace COMMIT_ID..COMMIT_ID < STACKTRACE