Nobody Gets a Clean Weekend to Migrate

A founder asked me last week how he was supposed to move his product to new infrastructure while his customers were still using it. He had this picture in his head where you shut everything down on a Friday night, move it all, and turn it back on Monday. I told him I have never worked anywhere that got to do that.

I was the technical lead on an e-commerce platform that handled over 3 million requests a day, and I led the migration of that platform to containers on AWS. The technology was not the hard part. The hard part was that the business was making money every hour, and nobody was going to let us turn that off so the engineers could have a clean weekend.

We killed the big cutover plan early. There were too many things that could go wrong at the same time, and if something did, we had no good way to go back. What we did instead was move one piece at a time. The old system and the new system ran next to each other, we sent a small amount of real traffic through the new piece, and we watched it until we were sure it behaved the same as what it replaced. Some weeks that was the entire deliverable. Leadership asked me more than once why it was taking so long, and I get it, from the outside it looked like nothing was happening. Our customers never noticed a thing, and that was the whole point.

I say this to founders all the time now. A migration is a business continuity project that happens to involve technology. When your developers pitch you a rewrite or a platform move, don’t ask how long it will take. Ask them how the business keeps running on day 40, and what we do if week six goes badly. If they don’t have an answer for both, they don’t have a plan yet.

#FractionalCTO #TechLeadership #StartupTech