Skip to content

20170831_srecon17_1.3

20170831_srecon17_1.3

One ring to rules them all

1 - Origin

Why Rollout automation are reinvented all the time ?

  • Every SRE thinks that their service is “special” and no common things can work
  • Every SRE are lazy, impatient and “Haris”
  • Every SRE thinks he can solve the problem, quickly, as a little fix

Break the circle of :
-> short term project
-> More long term tech debt, hacks, incidents …

The situation:
* SRE spends a non-trivial amount of time keeping rollout automation running …
* Write “monolithc” and overly specific automation

2 - How to converge to a “standard” solution ?

Investment:
-> Implement a standard product

We already have a “standard” framework to write automation !

There is a difference between:
- standard framework
- Common product

1st principle
* Decompose monolithic automation in microservices

2nd
* Tests

3rd
* Unprivileged

-> Design for all services
-> validate with MVP

3 - Tech bits

Hosted and Shared
* Scheduler
* lock (fined grained)
* Policies (To prevent issues)
* Store (Metadata)
* UI

Custom to the automated system
* Executor
* Other actions

4 - What’s next / How to spread ?

How get support from all the managers ?

How get enough people work on this ?
- People who are interested do their best

How do you convince services to adopt ?
- People are heavily invested in their code
- Do not brag and say we will through up previous work

Requirements gathering is ESSENTIAL

5 - Hard things learned

  • Requirement gathering is hard
  • Rollout automation, everybody wants it, nobody has time to work on it
  • Rollout isn’t stand alone, lots of integration work
  • Rollout are long, and we can not stop (in storage at least)