BigQuery

• Prepare these material

  • Project id
  • Credentials
    • The roles must include
      • BigQuery Data Editor (roles/bigquery.dataEditor)
      • BigQuery Job User (roles/bigquery.jobUser)
      • BigQuery Read Session User (roles/bigquery.readSessionUser)
    • You can encode the content via base64 and then set it to bigquery.credentials-key, Or you can directly set the JSON file path to bigquery.credentials-file
  • Location
  • Bucket
    • Create a bucket for the cache
  • Access key and Secret key
    • The roles must include
      • Storage Object User (roles/storage.objectUser)
      • IAM roles for Cloud Storage
    • To access the cache

• config.properties

  node.environment=production
  wren.directory=<YOUR_MDL_FOLDER>
  wren.datasource.type=bigquery
  bigquery.project-id=<YOUR_BQ_PROJECT_ID>
  bigquery.credentials-key=<YOUR_BQ_CREDENTIALS_TO_BASE64>
  bigquery.location=<YOUR_BQ_LOCATION>
  bigquery.bucket-name=<YOUR_BQ_BUCKET_NAME>
  bigquery.metadata.schema.prefix=you_favorite_schema_name_
  duckdb.storage.access-key=<YOUR_GCP_ACCESS_KEY>
  duckdb.storage.secret-key=<YOUR_GCP_SECRET_KEY>

  Properties detail BigQuery Connector