This article contains updated addenda to the article on Accessing Google Cloud Features not yet in Terra, and is meant to support attendees of the Terra workshop on using Vertex AI.
The Terra platform possesses the integration necessary to use some of today's most advanced computational tools. This workshop will demonstrate some featured materials that utilize Vertex AI. Below are some resources that attendees may find useful as they continue to learn about using these tools.
Why Use BigQuery?
BigQuery is a powerful, petabyte scale cloud-based data warehouse based on a serverless infrastructure and built-in machine learning. Instead of running analysis on in-memory dataframes, use BigQuery to execute queries at scale with increased performance and optimized costs. Includes descriptive and prescriptive analysis capabilities including business intelligence, ad hoc analysis and machine learning all in one platform. BigQuery includes specific support for:
- Analyzing variants
- Run complex join queries on variants with data described by genomic region intervals or overlaps
- Transform/download VCF files directly to BigQuery using the Variant Transform tool
Why Use Vertex AI?
Leverage AutoML frameworks or develop custom models with Vertex AI - combining data engineering, data science, and ML engineering workflows in one fully-managed platform. Vertex AI supports the entire ML workflow from data preparation to model training to deployment and monitoring. Several first party tools are available to support this lifecycle including:
- Vertex AI workbench notebooks integrate with Cloud Storage and BigQuery to help you access and process your data faster
- Vertex AI Vizier to tune hyperparameters for you in complex ML models
- Vertex AI Experiments to train your model using different ML techniques and compare the results
- Vertex Explainable AI to understand how individual features contributes to model prediction and find mislabeled data
Additional Instructions and Template Notebooks
-
Analyze data in BigQuery using a Vertex AI Workbook
- See this example notebook (explore data in Bigquery with workbench.ipynb) in GitHub
-
Create and deploy a ML model using Vertex AI Workbook
- See this example notebook (Build a fraud detection model on Vertex AI.ipynb) in GitHub.
Notebook samples for additional use cases can be found here.
Additional Resources
See this Google Cloud Quickstart on creating machine learning models in BigQuery ML
See Google Cloud documentation on Vertex AI or tensorflow.
See this Google Cloud Quickstart on processing genomic data by using Cloud Life Sciences API