Configure Google Cloud Storage as your data pipeline source

Set up Google Cloud Storage as a data pipeline source to extract and sync records into your destination. This guide includes connection setup, pipeline configuration, and key behavior for working with .csv files stored in GCS buckets.

Features supported

The following features are supported when using Google Cloud Storage as a data pipeline source:

• Extract and sync data from .csv files in GCS buckets
• Support for full and incremental sync through file detection
• Field‑level selection for data extraction
• Schema drift detection and handling
• Field‑level data masking

Prerequisites

You must have the following configuration and access:

• A Google Cloud account with access to storage buckets that contain .csv files
• A Google service account with permissions to list and read objects
• Folder paths and file patterns for the files to sync

Connect to Google Cloud Storage

Complete the following steps to connect to Google Cloud Storage as a data pipeline source. This connection allows the pipeline to extract and sync records from your storage buckets.

Refer to How to connect to Google Cloud Storage to learn how to create a Google service account and download the private key.

Connect to Google Cloud Storage

1

Select Create > Connection.

2

Search for and select Google Cloud Storage on the New connection page.

3

Enter a name in the Connection name field.

Google Cloud Storage

4

Use the Location drop-down to select the project where you plan to store the connection.

5

Select Cloud in the Connection type field, unless you need to connect through an on-prem group.

6

Enter your Google Cloud Project identifier. You can find this in the Google Cloud Console.

7

Enter the GCS Project service account email. This is the email associated with your Google service account.

8

Paste the Private key from your service account JSON file. You must include the full key, from -----BEGIN PRIVATE KEY----- to -----END PRIVATE KEY-----.

9

Optional. Use the Restrict to bucket field to limit access to specific buckets. Enter a comma-separated list such as bucket-1,bucket-2.

10

Optional. Expand Advanced settings and select Requested permissions (OAuth scopes).

11

Click Sign in with Google to authenticate and complete the connection setup.

Configure the pipeline

Complete the following steps to configure Google Cloud Storage as your data pipeline source:

1

Select Create > Data pipeline.

2

Provide a Name for the data pipeline.

Data pipeline setup

3

Use the Location drop-down menu to select the project where you plan to store the data pipeline.

4

Select Start building.

5

Click the Extract new/updated records from source app trigger. This trigger defines how the pipeline retrieves data from the source application.

Configure the Extract new/updated records from source app trigger

6

Select Google Cloud Storage from Your Connected Source Apps.

7

Choose the Google Cloud Storage connection you plan to use for this pipeline. Alternatively, click + New connection to create a new connection.

Choose a Google Cloud Storage connection

8

Click Add object to configure the files the pipeline monitors and syncs.

Add Google Cloud Storage objects

9

Enter the folder within the bucket to monitor in the Source Folder path field. The pipeline supports .csv files only.

Configure file settings

10

Define which files to fetch using a pattern in the Filename pattern field. Use wildcards such as orders_*.csv to include multiple files.

11

Click Fetch matching files to preview files matching the defined pattern.

12

Select a Reference file to define the schema for your destination table.

13

Configure CSV settings:

1

Set whether the CSV includes a header in the Does CSV file include a header line? field.

2

Choose a delimiter in the Column delimiter field.

3

Set the Force quotes field to True to quote all values.

14

Click Fetch schema to load and preview columns from the reference file.

15

Review the schema to ensure it matches your expected table structure.

Review schema

16

Configure how rows are merged in the destination table in the Choose a merge strategy field. Workato supports the following merge strategies:

• Upsert: Inserts new rows and updates existing rows. When you choose Upsert, the Merge method field appears. You must select a column that uniquely identifies each row. This key is used to determine whether a row exists in the destination and whether it's updated or inserted.
• Append only: Inserts all rows without attempting to match or update existing records. When you choose Append only, the pipeline doesn't match on a key and doesn't update existing rows.

17

Click Review object to confirm your setup. This screen displays your file settings, CSV options, and merge details.

Review object

18

Enter an Object name. This name defines the destination table name.

19

Click Finish to save the object configuration.

20

Review and customize the schema for each selected object. When you select an object, the pipeline automatically fetches its schema to ensure the destination matches the source.

Expand any object to view its fields. Keep all fields selected to extract all available data, or deselect specific fields to exclude them from data extraction and schema replication.

21

Optional. Configure field-level data protection. After you expand an object, choose how to handle each field:

• Replicate as is (default): Data values at the source are replicated identically to the destination.
• Hash: Hash sensitive data values in the column before syncing to your destination.

Configure field-level data protection

22

Click Add object again to add more objects using the same flow. You can repeat this step to include multiple Google Cloud Storage objects in your pipeline.

23

Choose how to handle schema changes:

• Select Auto-sync new fields to detect and apply schema changes automatically.
• Select Block new fields to manage schema changes manually. This option may cause the destination to fall out of sync if the source schema updates.

Schema drift management is currently not supported for file-based pipelines. Support for automatic schema updates is planned for a future release.

24

Configure how often the pipeline syncs data from the source to the destination in the Frequency field. Choose either a standard time-based schedule or define a custom cron expression.

1

Select the Time unit, such as Minutes, Hours, or Days.

2

Specify the sync interval in the Trigger every field.

For example, if you select Hours as the Time unit and enter 12 in the Trigger every field, the pipeline syncs every 12 hours. The minimum interval you can set is 30 minutes.

Configure sync frequency

3

Select a start date for the historical data sync in the When first started, this pipeline should pick up records from field. This defines the earliest date from which the pipeline extracts records. If you leave this field blank, the pipeline picks up all available records from the source.

You can't change this value after you run the pipeline. Refer to the When first started, this recipe should pick up events from section for more information.

4

Click Save to save your progress before you configure your data pipeline destination.

1

Select Custom schedule in the Time unit field.

Configure sync frequency

2

Enter a Cron expression in the Cron expression field using the format:

[minute] [hour] [day of month] [month] [day of week]

The minimum interval for cron schedules is 15 minutes. For example, to sync every day at 8:00 AM UTC, enter:

0 8 * * *

3

Optional. Select a Timezone for the cron expression. Leave blank to use Etc/UTC.

4

Select a start date for the historical data sync in the When first started, this pipeline should pick up records from field. This defines the earliest date from which the pipeline extracts records. If you leave this field blank, the pipeline picks up all available records from the source.

You can't change this value after you run the pipeline. Refer to the When first started, this recipe should pick up events from section for more information.

5

Click Save to save your progress before you configure your data pipeline destination.

File schema and processing

The Google Cloud Storage connector reads .csv files stored in buckets you specify. These files define the structure and data the pipeline extracts and syncs to your destination.

Workato infers schema and data types from the selected reference file. All files matched by the file pattern must maintain the same structure to ensure accurate mapping.