When your e-tail site crashes on Cyber Monday, it's as if your bricks-and-mortar store locked its doors and told employees to stay home: No one can enter, no one can shop, no one can spend money on your goods.
So when Bonobos, an online purveyor of men's tailored clothing and accessories, suffered through unexpected -- and definitely unwanted -- downtime on what's touted as the biggest online sales season of 2011, newly appointed CTO Michael Hart knew something had to change. Fast.
Having come on board only three weeks prior to Cyber Monday 2011, Hart had limited time to prepare for that year's virtual onslaught. But he knew the site had to perform flawlessly in 2012, and he immediately began fine-tuning the e-tailer's system. However, Bonobos had no way to simulate the hundreds of thousands of simultaneous shoppers expected to visit its site during the hectic holiday season.
That is, until Hart turned to Soasta, developer of mobile and web testing solutions, which was recently awarded a patent for Cross-Cloud Grid Provisioning, which enables access to 500,000 servers across the world, simulating hundreds of thousands of users accessing a single app or website simultaneously.
The result? On Cyber Monday 2012: "We handled record sales that day without having any performance issues," Hart told me.
On that day, Bonobos handled 15 times its normal volume of business, said Hart. It's apparent why performance is so crucial.
That's just one day. In the days following you've got the big Christmas buying rush so there's a lot of money on the line.
Hart knew of Soasta based on prior experience with the solution at a prior employer.
We definitely considered looking at alternatives, but there wasn't anybody out there. It was the only option available to us. I haven't surveyed the landscape recently to be honest, but in a lot of ways I don't have a lot of reasons to do so. The ability to easily create test cases and execute them at a reasonable price makes it a no brainer for us.
Of course, every day isn't a crisis, but that doesn't mean Bonobos doesn't turn to Soasta year-round. The company relies on the technology to test its website each time it makes any kind of revision, Hart said. He spends about two to three man-weeks annually improving, maintaining, and running stress tests on the site, he told me.
We get great performance stats out of the system. That's useful for identifying general bottlenecks. It's worth noting if we can assure we're up on our biggest day of the year it's highly likely we're up for the rest of the sales days. The nice thing, too, is we do make a lot of system changes throughout the year and being able to confirm we haven't hurt the system is great.
Surprisingly, many e-tailers don't test their sites after adding new features or launching a new URL, said Hart, whose background is at technology organizations. Bonobos, for example, didn't do this before he joined the company, he said. In some cases, businesses may be overly confident in their sites' capabilities, Hart surmised.
I come from a world where you always assume failure. It's a different mindset. A lot of it could be simply misunderstanding how much load you're going to get. Even we were a little off in our projections. In our case we went beyond what our estimates were, just to be safe. It comes down to your sense of precaution, how accurate your projections are.
Testing doesn't add much longer to the process: In Bonobos' case, it made some "fairly extensive" changes and it took about a week to test the enhancements, Hart said. That extra few days surely paid off in satisfied customers and full shopping carts.