Handle data export to Terra in backend

Post author
Shaun Adkins

Is it possible to facilitate the export of JSON data to Terra from the backend instead of from the user's browser? Currently if I attempt to export data using this url -> https://app.terra.bio/#import-data?format=entitiesJson&url=http%3A%2F%2Fportal.nemoarchive.org%2Fapi%2Fstatus%2Fapi%2Fterra%2F6d762241-11db-4266-817f-e7fb997ecd1f (JSON from http://portal.nemoarchive.org/api/status/api/terra/6d762241-11db-4266-817f-e7fb997ecd1f), I get a "Failed to Fetch" error, for a couple of potential reasons. The first is that our export URL is "HTTP" instead of "HTTPS" (which we plan to soon change). The second is due to CORS rejections, which may be problematic given the current configuration of our data portal's server. Having the Terra backend facilitate the JSON data export would be useful in that it would solve both of these issues for us.



  • Comment author
    Dan Moran

    Moving the export/import logic into the backend will also prevent out-of-memory errors caused by trying to load large JSON payloads in the browser.

  • Comment author
    Brad Taylor

    Hi! Brad from the Terra product team here. This is an important update to our backend, and we should absolutely do it. Unfortunately, we haven't had a lot of bandwidth recently for adding/updating APIs, as we've been focused on scaling our infrastructure to support our growing user community. I expect that we will have engineering resources to devote to this in Q2, starting in April. Please let us know if your team has deadlines that don't fit that timeframe. 


    We are also always happy to accept open source contributions. In this case the backend service is called Rawls

    Swagger: https://rawls.dsde-prod.broadinstitute.org/

    Github: https://github.com/broadinstitute/rawls


Please sign in to leave a comment.