A tough day
24 April 2018
Fyodor Ovchinnikov
Dodo Pizza CEO
Last Saturday happened to be a very tough day for our company. And this week is going to be even tougher.
We’ve been working on our first campaign on national TV for the whole year. It was a complex project involving every aspect of our business. We and our partners invested more $1.6 mln in it. And on Saturday, just when the sales got to their peak, a part of our system failed because of a too-high workload.
It led to a major meltdown in our business. Our website and mobile apps didn’t work for three hours. And most of our customers couldn’t even get a hold of us via phone. We let down thousands of customers, our partners, and their teams.
Had we anticipated higher workload because of the campaign? Yes. It was the main focus for our IT team for the entire year. Maybe we lacked in skills? I don’t think so. We have a team of highly motivated professionals.
But we could have done more—we had a chance to find a crucial vulnerability in our architecture a week before when we noticed a small glitch in it. But we didn’t act.
I apologize to our partners and customers for letting them down. It will be a big lesson for our team.
When you venture into the unknown, mistakes are almost inevitable, but you should do whatever it takes to avoid them.
This is how the operational headquarters of Dodo IS looks now.
Our IT team is working 24/7. They feel the responsibility. And we have to count every minute because next Saturday, there will be another peak in sales—and it will probably be even higher than the previous one.
We can’t stop our campaign, and more and more customers download our app every hour (it now holds 9th place in the App Store’s free app ranking). So we have to rectify all the errors in the Dodo IS architecture before the weekend.
Mistakes can be meager or huge ones, but you can’t let them break you. Growth can’t happen without blunders. Thousands of engineers build space rockets. Sometimes, these rockets fall—as ours fell last Saturday. But you must find strength inside you to go on and continue building spaceships.