Good Business Practices

Some major tech companies operate on a three-phase testing cycle where new code first goes through unit testing, then integration testing, and then QA and A/B testing. Although it comes under criticism from some developers, I think it's a very effective methodology that can be time-efficient once you're experienced with the tools involved. Unit testing, for example, often is criticized because the tests themselves follow the structure of the code very closely. If you have a function that issues a service request and formats the response, the unit test might just mock the service and assert that the returned value matches the mocked input. The point of the unit test, then, is not to verify that the code functions as intended (that's the job of integration testing), but to make sure that the dependencies don't make breaking changes down the line. I've heard some people talk about "blind black-box tests", in which one developer writes unit tests for another's code without ever looking at their implementation. Although this seems helpful, it misses the point of unit testing, which is more for posterity than short-term reliability.

Integration testing should really be as close as possible to a production scenario. Ideally, you can mock other services in beta, so you can run your code without special if statements or functions for beta testing. As far as I know, most modern companies accomplish this, but sometimes internal mocking tools can lack some features, so workarounds are occasionally necessary.

Quality assurance exists because everybody makes mistakes, and there should never be code in production that only one developer has looked at and verified. If seriously broken code makes its way into production, the onus should be on the developer who wrote the code, the peers who reviewed it, the QA personnel that verified it, and the manager who oversaw the production pipeline. Distribution of risk is extremely important, and the most extreme examples of developer negligence are usually code that hasn't been properly reviewed. Of course, sometimes even well reviewed code has serious issues, as in March 2008 when the Heathrow airport failed to deliver about 28,000 bags (BBC). Even though the systems were apparently fully functional in testing, the dimensional increase of production use uncovered other operational issues. This is why A/B testing is also critical, since any new production code will have unknowns.

Of course, there are some situations where nothing can save you. Single day roll-outs, like Amazon Prime Day, can't be A/B tested. In situations like these, the most important thing is managing risk. New medical equipment and cars should never be released on one day only. Software is distinctly mathematical, so you can always mitigate risks due to bugs and unintended behavior. Putting the lives of people at risk is blatantly unethical, and there's no excuse for faulty software making those sorts of mistakes.

“28,000 Bags Caught in T5 Foul-Up.” BBC News, BBC, 31 Mar. 2008, news.bbc.co.uk/2/hi/uk_news/7323198.stm.