All About Performance

and other stuff by Taras Glek

New Role: Developer Productivity

New Role

The performance part of my team is now lead by Vladan. They are doing usual performancy things with a focus on improving our ‘internal’ benchmarks such as talos, TART + some upcoming power testing.

I got a new task recently: developer productivity. I’m prioritizing tasks based on developer unproductivity. I have the following new stuff on my radar:

  • slow build system for local builds
  • long build/test times on our release automation
  • bugzilla improvements (eg tracking review times, lack of pull reqs, etc)

Expect to hear less Gecko and more web dev stuff out of me for the foreseable future. Note, this is a collaborative project. My team would not be able to accomplish any of the new goals as we are already above capacity. I’m mainly working on improving coordination and collaboration on existing projects. Let me know if you have ideas related to this dev productivity stuff.

Cost-Oriented Architecture

My main project for the remainder of this year is to maximize throughput per dollar spent on our release automation infrastructure. I’m focusing on computation that happens on Amazon EC2(because it’s easier than pricing out our own infrastructure). My goal is to be able to say “making improvement x will pay for itself in y time”. This is mainly management technique to make it easier to justify working on infrastructure.

John posted some initial costing figures. Rail is working on switching us over to Amazon Spot instances in bug 935533. I expect a 2-5x reduction in our AWS costs once Rail is done.

I’d like to have a receipt attached to every job that executes on our build infrastructure. I hope this encourages devs to look into build/test inefficiencies.

Keeping an eye on costs means we’ll be able to better reason about moving more workloads into the cloud. Cloud stuff is attractive because it allows us to give developers keys to AWS. They can deploy(in a low-friction manner) whatever they need to make their job easier while cost-monitoring ensures that things don’t get crazy.

Open Architecture

In theory, open source is about scratching own needs. In practice service owners (usually inadvertently) put up walls to contribution by depending on licensing requirements, poor documentation, vpns, obscure infrastructure, weird version control, etc. This then snowballs into something equivalent to closed source where ‘clients’ have to ask for ‘features’ from ‘owners’. This usually results in unhappy customers and overworked owners. This happened to Telemetry (see previous post) and is happening to a few of our tools. Expect improvements in this area.