How often do your development teams release to production? Who gets the alert in the middle of the night when everything crashes and burns? Do these questions make you uncomfortable or rather their answers? Or maybe you are already discussing changes to your current deploy process? Because it sucks, right? If you’re honest, it will always suck because it constantly needs to be adapted to the current business requirements.
Enter the “Platform Team”: a group of build & deploy experts that jumpstart your teams down the road to operational success while providing a safety net. And, no, I’m not referring to a System Administrator with a pager. Instead, I’m suggesting a three-ply construction of automation, containerization and monitoring.
But wouldn’t hiring a full stack developer be cheaper?
The amount of job ads for “Full-Stack Developers” is insane. Hunting for them is like searching for unicorns – the closest you’ll get is an interview with an insane, cone-ornamented horse telling you your search is at an end.
Take the time and effort to teach the folks you have right now. Let them know the most important job is customer satisfaction and this means much more than just writing code.
A platform team is composed of great teachers: test managers who understand ‘release-ready’ test coverage and how to automate it; sysadmins that can point out application bottlenecks and suggest monitoring tools.
How to infect cross-functional teams with the DevOps bug?
The culture of DevOps is founded upon interdisciplinary communication. What minimally was a developer and sysadmin talking about howto deploy and operate an application, now involves product owners and UX folks in the conversation. MVP has replaced monthly releases and the pace only quickens.
With automation becoming ever easier to employ and virtual servers costing pennies during a workday, now’s the time to ready teams for self-service. Here are the first three Self-Service Mantras:
Self-Reliance (aka “We can build it”)
Work with the developer – your customer – to get their requirements right (spoiler: they’ll change during implementation ;). Make the build and deploy commands executable in the online chat room – we use hubot. And if you don’t use an online chat go checkout hipchat or slack. These rooms are essential for improving situational awareness + simplicity + remote deploys.
Attach a minimal test harness to ensure the build really did what you wanted. Is it really the right code version? Does it start correctly and fulfill it’s primary goals? Jenkins + Saucelabs is an inexpensive 1-2 knockout punch for this!
Finally, get those builds auto-deployed to your testing environment. A simple, post-build hook from Jenkins in your chat room with the corresponding URL is a honeypot for product owners keen on signing off features for release. “Houston, we are ‘GO’ for launch!”
Self-Confidence (aka “We know what’s going on”)
- it takes:
- 20-30 builds to work out build bugs (git commit hooks, right branches, etc.)
- 20-30 more builds to perfect test harness (those damn outliers)
- 20-30 more builds to prove auto deployment (yes, this is the right version)
- with ~100 builds, the team is confident
- they know how it works, why it breaks and how to fix it
Self-Responsibility (aka “I care about my users”)
Monitoring used to be black magic, but today, with tools like loggly, datadog and newrelic, it’s incredibly easy to get deep insight on your applications and servers. Most of these support direct alerts into your online chat room too, so the entire team can feel the pain they subject upon their hapless users. Seeing a long list of disturbing, red graphs with the first cup of coffee in the morning turns the daily standup into a Q&A session.
To provide first-level support, cross-functional teams need to be able to dig for root causes. Gathering data from all your monitoring services (you are using more than one right?), you can quickly narrow down potential problems until you’ve found what’s not working. This takes time and experience, so be patient and always available for pair-troubleshooting!
These mantras close the gap between developers and sysadmins and can be unified with the overarching principle “You build it. You run it.” Empowered developers take the time to build higher quality products, because they’re the ones getting the call during the night.
Scarcity lends Focus
We don’t have enough Platform Engineers to embed in each product team, so we loan them out for new projects as “on-site consultants”. Once they’ve gotten the engineers through the three phases above, they move on to the next construction site always with an eye on improving and adding to the company’s architectural platform and engineering excellence. This style of teaching full-stack development is much more rewarding and infinitely more successful.
There’s something to be said for time-boxing new technical initiatives. They tend to be streamlined applications that only do what’s needed. Since there’s not as much code or moving pieces involved, they’re also easier to maintain.
As you widen developers’ horizons to the world of automation and virtualization, a few may decide to pick up the standard and become platform engineers themselves. It’s a win-win situation for your employees and your business.
And what about those unsettling questions about your release process? With development teams responsible for building, shipping and operating their applications, it will suck a whole lot less!
