We've identified an issue affecting users making high numbers of DRS URI calls in their workflows. This error occurs when a user has their appropriate external server linkage set up in https://app.terra.bio/#profile. These are
- NHLBI BioData Catalyst Framework Services
- NCI CRDC Framework Services
- and/or NHGRI AnVIL Data Commons Framework Services
The submission that utilizes these calls errors out with the following message before a workflow is successfully run:
ErrorReport(rawls,http error calling uri https://us-central1-broad-dsde-prod.cloudfunctions.net/martha_v2,Some(502 Bad Gateway),List(),List(),None)
The overall problem lies in datastage.io being unable to scale properly for large requests of DRS objects. We're tackling the issue on both fronts by communicating with datastage to see where we can help improve their app performance as well as identifying possible measures on our end that may work around the scaling issue.
If you urgently need progress on this, you are welcome to try batching the data in such a way that reduces the overall number of requests to DRS objects. However, we do not know what the "sweet spot" is for how many DRS objects can be used as input without error at this time. We recommend starting with 100 DRS objects and scaling up or down from there depending on the results.
If you are testing what number of DRS objects work for your workflow, please post here with your results so that other users can benefit from the information. We will also be posting here with our own test results, as well as other relevant updates from the development team as we receive them.
PLEASE NOTE: This error can also occur in cases when you do not have the above mentioned external server account(s) linked in your Profile, or if the link has expired. See this article about linking for details on how to link.
Please sign in to leave a comment.