Summary
We've identified an issue that impacts users running workflows with the call-caching functionality enabled.
Timeline
August 2, 2022 9:00 AM ET - Issue remediation - Creation of new call cache entries is now working normally, following the downtime from 7/21 to 7/28. Call cache entries written after 7/28 are accessible.
Our investigation of call cache usage revealed that the vast majority of call caching usage is concentrated in recently-created data. For older jobs that are re-run frequently, new cached results are generated with each re-run and future runs preferentially use the latest cached versions. As such, the vast majority of the pre-7/21 cache has already been regenerated or would regenerate sooner than we could restore access to those original versions. Therefore, we have opted not to restore historical caching data.
This will result in an inability to benefit from the caches for some jobs that were generated prior to July 21, 2022.
July 28, 2022 7:00 AM ET - Fix in progress - Our engineers have kicked off the fix which will run for approximately one week's time. Please see the Impact section to see how you will be impacted while the fix is in progress.
July 27, 2022 5:00 PM ET - Fix scheduled - We plan to begin implementing the fix early in the morning (US Eastern time) on July 28, 2022. Please see the Impact section to see how you will be impacted while the fix is being implemented.
We will update the Timeline and Impact sections when the fix is in progress so you know exactly when you can rely on certain call cached entries.
July 26, 2022 11:51 AM ET - Issue investigation - Our engineers are currently investigating the issue and working on a fix.
July 26, 2022 11:11 AM ET - Issue discovered - Our engineers noticed errors in our production logs indicating an issue with call caching.
July 21, 2022 5:00 AM ET - First instance identified - Our reports indicate that the first instance of this issue was logged around this time.
Impact
July 21 - July 28 7:00 AM
New call cache entries cannot be created. Call caching for existing entries (before July 21) should not be impacted.
July 28 7:00 AM - today and going forward
New call cache entries can be created.
Call cache entries for jobs completed before July 21 will not be available.
For more information
Please follow this article to get the most up-to-date information on this incident. If you would like to be notified of all service incidents or upcoming scheduled maintenance, click Follow on this page.