Flakey. Non-deterministic. Unreliable. These are all terms commonly associated with end-to-end testing. At WalmartLabs, we knew that building a Continuous Delivery pipeline depends on a test automation system that’s totally reliable. Any perceived flakyness in the system would erode developer confidence in our tools. But when non-deterministic behavior emerges, how do we crack open the black box and identify the source?
This talk shows how we instrumented our test framework with our own internal analytics beacons, to allow us to “zoom out” and spot trends in test performance and reliability that are not always visible at the individual test suite level. Using open source data visualization tools allows us to slice and dice the data to identify root causes. We share how these data and trends provided both immediate insights, as well as raised new mysterious questions that we hope to work with the community to answer.
https://www.youtube.com/watch?v=oRIci6n566w
Video producer: http://seleniumconf.co.uk/