Google BigQuery V2

Updated by Daniel Odrinski

With our BigQuery V2 Integration, your Dreamdata data is available as a BigQuery dataset through a Google Analytics Hub listing—ready for querying, joining, copying, or using with any tool connected to BigQuery.

With this integration, we expose a Dataset of your Dreamdata Data within your BigQuery project, but only to specific users.

If you are not familiar with Google BigQuery, take a look at these how-to guides.

Here are some of the key benefits of switching to our new BigQuery V2 integration:

  1. Streamlined & Secure Access Management: Leverage your existing Google Cloud IAM policies within your own BigQuery project to grant and manage data access. You retain full control over who within your organisation can view or query the subscribed data, ensuring alignment with your internal governance and security protocols without managing separate access systems.
  2. Integrated & Intuitive Data Discovery: Subscribed datasets are accessed directly within your familiar BigQuery environment, appearing alongside your native datasets (in a location of your choosing). This eliminates confusion about data location and removes the friction of navigating external or inaccessible projects.
  3. Reliable Query Execution: Query the subscribed data seamlessly using your project's standard BigQuery resources and permissions. This prevents query failure scenarios previously encountered (using our BigQuery Legacy integration) where users lacked the necessary job creation permissions in the shared Dreamdata project. Our new BigQuery V2 integration ensures a smooth analytical workflow.
  4. Enhanced Subscriber Autonomy & Governance: While Dreamdata hosts and makes the data available via Audience Hub, your organisation maintains complete autonomy over granting internal access. This empowers you to enforce the principle of least privilege access control and adhere strictly to your own data governance and compliance standards.

In essence, using our new BigQuery V2 integration provides a more integrated, secure, and user-friendly experience, by leveraging familiar GCP tools and keeping access control firmly within your own environment.

Pre-requisites
  1. A Google Cloud Platform account. If you have not already signed up for Google Cloud Platform, you can do so here and get $300+$200 ($500 total) of free trial credits.
    Dreamdata is a Google Cloud and BigQuery-ready Partner.
    Dreamdata Google Cloud Partner
  2. All of the following permissions for the project you want to add the shared dataset to:
    1. serviceusage.services.enable (included in roles like Service Usage Admin , Editor or Owner) - Required to enable the Google Analytics Hub API in your project (if not already enabled).
    2. analyticshub.listings.subscribe (included in roles like BigQuery Admin , Analytics Hub Subscriber or Owner) - Required to subscribe to the dataset listing.
    3. bigquery.datasets.create (included in roles like BigQuery User , BigQuery Data Editor or BigQuery Data Owner) - Required to create the dataset in your BigQuery project.
  3. One of the following:
    1. A Google user email that should be given query permissions in the shared dataset.
    2. A Google group email that should be given query permissions in the shared dataset.

Some consideration points:

  • Currently, data can only be hosted on in the EU region; sub-regions are not supported.
  • Dreamdata pays the data hosting storage costs. Be aware, however, that any data you copy from the shared dataset to your own dataset will incur storage costs in your project.
  • Cost of querying is the same as querying a table within your organisation, which is to say, those costs are covered by you.

Setup

Configuring our BigQuery V2 integration is quick and easy. Just follow these steps to set it up:

  1. Head over to Data Platform -> Data Access -> Google BigQuery V2.
    You should see a form similar to the one below:
    The Dreamdata BigQuery V2 integration configuration form.
  2. Read the instructions provided and fill out the form with at least one Google user or group email. As soon as one email address is added to the form, the integration is configured and a Google Analytics Hub listing is configured. Please note that the location is not currently configurable.
    Setup Recommendation: Provide one Google group email at most. Using a group email avoids disruptions in the case that a user’s email is disabled.
    The form should now look like the following:
    The Dreamdata BigQuery V2 integration setup form in a configured state.
    Note that the 'See your Listing' button has appeared in a disabled state.
  3. Wait for the next full data modelling run to complete. Once this has occurred, the 'See your Listing' button will become active and click-able.

Accessing your Data in BigQuery

Once the next full data modelling run has completed, your data will become accessible in Google BigQuery. To set up access to it, follow these steps:

  1. Click on the 'See your Listing' button, which will take you to the listing subscription page in Google Cloud Platform:
    Make sure to select your project (top left) in Google Cloud Platform after clicking this button.
  2. You will be re-directed to sign into your Google account. Use the email address provided in the configuration form earlier.
  3. Once on the Google Analytics Hub subscription page, click the 'Subscribe' button for the Dreamdata dataset provider:
  4. Select your project and give the dataset a memorable name. This is the project and dataset name you will use for your queries.
    If the Google Analytics Hub API is not enabled, you will be prompted during this stage to enable it:
  5. The dataset should now be available for querying:

Note that you also need to have the BigQuery Job User permission in the project in order to run queries.

How did we do?