Making the most of a software development “crisis”
Rapid learning from unplanned business emergencies
In early November my team was hit with a notice that a major vendor for us was shutting down before the end of December. We quickly assessed the situation- we needed to move over a dozen complicated workflows from the tooling vendor to our software platform in little over a month. These workflows are executed by internal staff and are essential to our business; we can’t service revenue without them.
Maybe you’ve been a part of something like this before. All your well laid plans go out the window and instantly everything is clear — if it’s not addressing the crisis, it’s not important. It can also be hectic, stressful, and a bit scary. We didn’t know immediately what it would take to migrate the workflows to our platform, but the goal was obvious: still be able to service revenue when the service shut down. If we could imagine living without something, it would be deprioritized. If we could also make improvements to the processes along the way- bonus points.
We quickly spun up a lead team to investigate solutions. This included product managers, engineering leads, a designer, and some of our internal end users of the workflows. The team met for hours, breaking the work down into main workstreams, assigned the workstreams to teams, who then broke that work down further and further. The sessions then became live interviews, driven by engineering, where we probed the boundaries of the requirements, negotiated scope, and moved quickly towards “I have enough to start working on a solution”.
Over the next few weeks the teams built a lot of software. They kept checking in with their stakeholders, released small iterations daily, and kept the goal at the center of everything they did. I was periodically checking in with team members. Surprisingly, people were enjoying their work and feeling good about delivering so much.
In the end, we created all the software we needed to meet our goal with more than a week to spare! We also managed to make a few significant improvements to our users’ workflows.
Here are some cool stats from that time period:
- Change Lead Time (first code commit to deployed): 15 hours
- Overall Deployment Frequency: 21x / day
- Engineer Deployment Frequency: 1.5x / day
- Change Failure Rate (bad deploys / total deploys) : 0.4%
These are all substantially improved!
What we learned
Why did this go so well? Through a series of retros and conversations, I’ve pulled together our biggest takeaways. In short the teams were motivated, outcome oriented, and ready to only use process in service of the end goal. Here is a larger list of changes we’re hoping to sustain in the future:
Planning and Building
- Have a clear, inspiring, and time bound goal that makes prioritization easy — make it a challenging one too. Always keep the goal front and center. Always ask how what we’re doing relates to the goal.
- Cut scope over moving date targets.
- Favor synchronous kickoffs with all needed parties: PM, Eng Lead, Designer, Customer Success aka “lock everyone in a room”
- The engineering lead is responsible for building a sufficient understanding to dig in and start. The kickoff period is over when the Eng lead can semi-independently build a solution (with other engineers)
- Getting started and try to develop as early as possible; rework is OK
- Allow the documentation format and level desired by the engineering lead. Tickets and details can “catch up”.
- Allow for light and fast estimation at the start, update as we learn more.
- Have small feedback loops with stakeholders, bring them along for the ride throughout development.
- Quality processes can be led by the software engineers; they can own their risk assessments.
Deploying and Releasing
- Use quick, iterative, and small pushes to production behind flags where necessary.
- Flipping features “on” should have a formal Product Acceptance step as a safeguard to moving so fast.
- Allow for “good enough” release announcements and synchronous internal user teaching.
- Try to make this as lightweight as possible; learn from our misses
- Always be shipping!
Many of these were natural extensions of our existing agile development process. For instance, we were already deprioritizing estimation and were working in variable length sprints. I might summarize this period as working how we always intended to be working. Therefore, the biggest takeaway for me is: radically focus on a radical goal.
Stated another way: what would you do differently if you took extremely literally that your team’s goals were the most important things? For instance, maybe you’ve worried about the length of your planning meetings. Well what’s more important than planning out the work, if that work is the most important thing? I say lock everyone in a room until they have enough to start building!
We are now focused on applying these learnings going forward. We absolutely want to work with urgency but we also want to make sure our teams don’t burnout. We want them to have time for the important but not urgent things like maintenance and skill development. I think teams naturally have planning, building, and regrouping cycles that we can use to ensure we staying focused in a sustainable way.
