Determining GCP instance type for workflows.

Post author
Brian Hannafious

I would like to know which GCP instance types are hosting my workflows. The primary motivation is to understand workflow costs. GCP instance types and pricing are available here.

This is explained in more detail below, along with currently applicable information that I am aware of.

I believe Terra workflows are executed using Googe Lifesciences Pipelines API, which spawns GCP instances for the duration of the workflow. To estimate costs associated with Terra workflow submissions, it is necessary to know which GCP instance type host workflow jobs, as well as the number of cores and RAM. This information is partially contained in the response the the Firecloud API workflowMetadata endpoint:

"jes": {
  "endpointUrl": "https://genomics.googleapis.com/",
  "executionBucket": "gs://fc-9169fcd1-92ce-4d60-9d2d-d19fd326ff10/05c90bd5-f0bd-4901-aef0-73270a40edd0",
  "googleProject": "firecloud-cgl",
  "instanceName": "google-pipelines-worker-69a16132e9de0ef6cf9efbc4abe40212",
  "machineType": "custom-10-65536",
  "zone": "us-central1-c"
}

Here we see that the workflow job was executed on a "custom" machine with 10 cores and 64GiB of RAM. However, the GCP instance type is unspecified.

Typical instance types are N1 and N2. However, if workflows are given memory or CPU optimized instance types, costs can increase 10% or more.

Comments

4 comments

Please sign in to leave a comment.