Connect your Dreamdata data to Snowflake
By default, at Dreamdata we build your database on Google BigQuery. However, through our Google Cloud Storage Integration you can connect the raw dataset of your Dreamdata database and easily connect it to another cloud container and then to Snowflake.
Google Cloud Platform
To use Google Cloud Storage you first need an Google Cloud Platform account.
Dreamdata is an Google Cloud Partner, and if you have not already signed up to Google Cloud Platform you can do it here and get $500 of free credits. If you are not familiar with Google Cloud Storage and want to learn more, take a look at these how-to guides.
Set up necessary integration details
After you have acquired a Google Cloud Platform account, go here to set up the necessary access integration details. Once the integration is configured, we will begin to generate your raw datasets on 6-hour schedule.
Some key points:
- This data will be hosted in the region of your choosing
- Dreamdata will pay the data storage costs
- You will pay for exporting costs
We will want to know the following:
- The region/zone of your Google Cloud Platform.
- You Google Cloud Platform service account.
- A list of emails of users that should have access to the exported dataset.
- A list of Google group email addresses that should have access to the dataset.
Note: Either (3) or (4) is required.
Exporting your data
After you have configured the Google Cloud Storage integration and gained access to your files, any one of the following methods can be used to transfer data from Google Cloud Storage to either AWS S3 or another datacenter. From there it can be loaded into Snowflake as normal.
If you are using Snowflake on AWS
- You can connect data from Google BigQuery to AWS S3 using AWS Glue. This article describes how to set this up and run the task on a schedule.
- This article describes how to connect data using CLI. This approach will work between any two cloud storage providers.
If you are using Snowflake on any other data platform
- You can use Snowflake Scheduled Tasks and COPY INTO to load data directly from Google Cloud Storage.
Once connected, you can run your own queries on our data models, as well as copy, manipulate, join and use the data within other tools connected to Redshift.
