You need to design a data pipeline to process large volumes of raw server log data stored in Cloud Storage.
The data needs to be cleaned, transformed, and aggregated before being loaded into BigQuery for analysis.
The transformation involves complex data manipulation using Spark scripts that your team developed. You need to implement a solution that leverages your team's existing skillset, processes data at scale, and minimizes cost. What should you do?

• A. Use Cloud Data Fusion to visually design and manage the pipeline.
• B. Use Dataproc to run the transformations on a cluster.
• C. Use Dataform to define the transformations in SQLX.
• D. Use Dataflow with a custom template for the transformation logic.

Your company is building a near real-time streaming pipeline to process JSON telemetry data from small appliances. You need to process messages arriving at a Pub/Sub topic, capitalize letters in the serial number field, and write results to BigQuery. You want to use a managed service and write a minimal amount of code for underlying transformations. What should you do?

• A. Use the "Pub/Sub to BigQuery" Dataflow template with a UDF, and write the results to BigQuery.
• B. Use a Pub/Sub to Cloud Storage subscription, write a Cloud Run service that is triggered when objects arrive in the bucket, performs the transformations, and writes the results to BigQuery.
• C. Use a Pub/Sub push subscription, write a Cloud Run service that accepts the messages, performs the transformations, and writes the results to BigQuery.
• D. Use a Pub/Sub to BigQuery subscription, write results directly to BigQuery, and schedule a transformation query to run every five minutes.

You work for a home insurance company. You are frequently asked to create and save risk reports with charts for specific areas using a publicly available storm event dataset. You want to be able to quickly create and re- run risk reports when new data becomes available. What should you do?

• A. Reference and query the storm event dataset using SQL in BigQuery Studio. Export the results to Google Sheets, and use cell data in the worksheets to create charts.
• B. Copy the storm event dataset into your BigQuery project. Use BigQuery Studio to query and visualize the data in Looker Studio.
• C. Export the storm event dataset as a CSV file. Import the file to Google Sheets, and use cell data in the worksheets to create charts.
• D. Reference and query the storm event dataset using SQL in a Colab Enterprise notebook. Display the table results and document with Markdown, and use Matplotlib to create charts.

Your company wants to implement a data transformation (ETL) pipeline for their BigQuery data warehouse.
You need to identify a managed transformation solution that allows users to develop with SQL and JavaScript, has version control, allows for modular code, and has data quality checks. What should you do?

• A. Create a Cloud Composer environment, and orchestrate the transformations by using the BigQueryinsertJob operator.
• B. Use Dataproc to create an Apache Spark cluster and implement the transformations by using PySpark SQL.
• C. Use Dataform to define the transformations in SQLX.
• D. Create BigQuery scheduled queries to define the transformations in SQL.

Your organization has a petabyte of application logs stored as Parquet files in Cloud Storage. You need to quickly perform a one- time SQL-based analysis of the files and join them to data that already resides in BigQuery. What should you do?

• A. Create external tables over the files in Cloud Storage, and perform SQL joins to tables in BigQuery to analyze the data.
• B. Use the bq load command to load the Parquet files into BigQuery, and perform SQL joins to analyze the data.
• C. Launch a Cloud Data Fusion environment, use plugins to connect to BigQuery and Cloud Storage, and use the SQL join operation to analyze the data.
• D. Create a Dataproc cluster, and write a PySpark job to join the data from BigQuery to the files in Cloud Storage.