{"id":21616,"date":"2016-08-18T09:28:16","date_gmt":"2016-08-18T14:28:16","guid":{"rendered":"https:\/\/blog.planview.com\/continous-integration-testing-for-a-rainy-day\/"},"modified":"2016-08-18T09:28:16","modified_gmt":"2016-08-18T14:28:16","slug":"continous-integration-testing-for-a-rainy-day","status":"publish","type":"post","link":"https:\/\/blog.planview.com\/continous-integration-testing-for-a-rainy-day\/","title":{"rendered":"Continous Integration Testing for a Rainy Day"},"content":{"rendered":"
<\/figure>\n

\"thunderstorm-small\"In an increasingly Cloud-centric world, we need to build and support an ever-growing list of SaaS SDLC integrations. That includes tools like Atlassian Cloud (JIRA)<\/a>, Microsoft’s Visual Studio Team Services<\/a>, and HPE’s new Octane ALM<\/a> offering.<\/p>\n

Supporting Cloud-based integrations poses some interesting challenges:<\/p>\n

    \n
  1. Cloud vendors typically choose a high cadence release cycle relative to on-premise deliveries. A frequent release schedule implies a higher risk of regressions sneaking into the API’s our connectors depend on.<\/li>\n
  2. Often there is no pre-release instance available to validate our product workflows ahead of time.<\/li>\n
  3. Releases take effect immediately. Compared with on-premise software, there is a relatively small window of opportunity to detect, investigate, and resolve issues before we receive support tickets from our customers.<\/li>\n<\/ol>\n

    Automated testing is a key ingredient of our success<\/a> in building and delivering high quality, full-featured SDLC integration solutions to our customers. Our Continuous Integration system executes integration tests in rolling builds throughout the day and with every commit that we push to Gerrit.<\/p>\n

    The value of our testing infrastructure doesn’t end once our products ship. Besides helping us to validate support for new versions of on-premise tools, those same integration tests serve as an early warning system for regressions with supported on-demand tools.<\/p>\n

    Case in point, several weeks ago an integration test failed against an on-demand ALM instance in a routine rolling build late one Thursday evening. Subsequent builds the next morning failed consistently with the same error. Given there weren’t any changes to our code between the last successful build mid-Thursday and the recent failed builds, it immediately raised suspicion that the on-demand instance had been updated and an API regression had slipped in. Further investigation confirmed that the vendor had deployed an update to their on-demand instance Thursday afternoon.<\/p>\n

    The following week, we received a support ticket from one of our customers impacted by this regression. By then we already had:<\/p>\n