Is is possible to run Rstudio + Spark Cluster?
Is it possible to use a Rstudio + spark cluster on Terra?
I've been able to start a Spark cluster successfully with a Jupyter environment with the following setup:
but I have not been able to do so with Rstudio. When I try it with Rstudio (The community maintained, verified partners ANVIL Rstudio enviroment) I get an error message and the cloud environment doesn't get created. See pictures below:
And the error message:
Thanks for reaching out. I have consulted with our interactive analysis team and they let me know that it is currently not possible to create a Rstudio environment using a spark cluster in Terra.
We appreciate you bringing this to our attention, a UI fix will be made to prevent such an environment configuration from being made in the first place. There are currently no plans to have this implemented, but, if this is something that you think would be useful I can pass that along to our team!
Thanks, Emil for your prompt response. I will get back in touch if I find a use case where Rstudio + spark cluster would be particularly useful.
Please sign in to leave a comment.