Custom Data Source
For most Dreamdata customers, the native data sources within Dreamdata solve all their data integration needs. However, in some cases, data exists in other standard or custom-built systems. This article will describe how that is done while covering the different use cases we usually see when needed.
This overview covers the following
- Implementation Process
- Use Cases
- Upload a custom CRM Source
- Upload custom Stage Objects
- Upload custom Event and Web-tracking Data
Implementation Process
![](https://files.helpdocs.io/och1s6u5rp/articles/w5510ujkvv/1721983309459/custom-upload.jpg)
- The customer prepares the data to upload.
- Schedule a data export
- Transform the data into the supported file formats.
- Compress the data using gzip.
- The customer upload the data.
- The data has to be uploaded to Google Cloud Storage or SFTP.
- You gain access to upload through your Customer Success Manager.
- Here are code samples of how to upload data to Google Cloud Storage.
- If incremental uploads, structure the data in folders by type and date
- /accounts/2024-01-23/file.json
- /{data-type}/{date}/file_{number}.json
- If full load, overwrite files at each upload
- The data has to be uploaded to Google Cloud Storage or SFTP.
- Dreamdata will pick up the latest version of the data two times daily, at 04:00 AM (UTC) and 03:00 PM (UTC).
- Dreamdata recommends uploading the data at least daily unless the data changes infrequently.
File format
Files should be delivered in JSON newline delimited (.ndjson) or parquet.
We recommend compressing the files using gzip to reduce data transfer costs.
Incrementally
It's possible to upload the data incrementally or upload full data batch each time.
We recommend doing what is the easiest for you.
Use Cases
Below are three use cases, each requiring different file types and schemas. All fields within these schemas are necessary, but you may leave some fields empty if you don't have any data or if it doesn't apply to your context.
You should share multiple files or just one, depending on the use case.
Upload a custom CRM Source
Uploading a custom CRM Source is typically used for customers who are in one of the following situations
- Customer has a CRM not yet supported by Dreamdata (Dreamdata natively supports Salesforce, HubSpot, Pipedrive, and MS Dynamics).
- Customer has a self-hosted CRM such as Oracle or MS Dynamics (Dreamdata has native support of MS Dynamics Cloud).
- Customer has a home build CRM
- Customer wants to avoid hitting CRM API limits (for example, Salesforce)
Here, you can read more about the schema format when loading a custom CRM data source.
Upload custom Stage Objects
Uploading Custom Stage Objects is typically needed when you have your ERP system with information not else available in your CRM or when you have defined your ways of measurement as joins of multiple objects.
Here, you can read more about the schema format when loading custom Stage Objects.
Upload custom Event and Web-tracking Data
Uploading custom web-tracking is typically used when uploading data from tracking solutions Dreamdata does not yet support out of the box such as Snowplow, a home build or open-source tracking solution.
Uploading custom events are typically custom events coming from a service you are using, that Dreamdata does not yet support or is from your own build service, such as your product.
Here, you can read more about the schema format when loading events and web-tracking data.