BigQuery

Connecting to BigQuery.

Prepare these materials

• Project id
• Credentials
  • The roles must include
    • BigQuery Data Editor (roles/bigquery.dataEditor)
    • BigQuery Job User (roles/bigquery.jobUser)
    • BigQuery Read Session User (roles/bigquery.readSessionUser)
  • You can encode the content via base64 and then set it to bigquery.credentials-key, Or you can directly set the JSON file path to bigquery.credentials-file
• Location
• Bucket
  • Create a bucket for the cache
• Access key and Secret key
  • The roles must include
    • Storage Object User (roles/storage.objectUser)
    • IAM roles for Cloud Storage
  • To access the cache

Setup the config.properties

node.environment=production
wren.directory=<YOUR_MDL_FOLDER>
wren.datasource.type=bigquery
bigquery.project-id=<YOUR_BQ_PROJECT_ID>
bigquery.credentials-key=<YOUR_BQ_CREDENTIALS_TO_BASE64>
bigquery.location=<YOUR_BQ_LOCATION>
bigquery.bucket-name=<YOUR_BQ_BUCKET_NAME>
bigquery.metadata.schema.prefix=you_favorite_schema_name_
duckdb.storage.access-key=<YOUR_GCP_ACCESS_KEY>
duckdb.storage.secret-key=<YOUR_GCP_SECRET_KEY>

Properties detail BigQuery Connector