Google BigQuery

  • Updated on Mar 19, 2025
  • Published on Apr 15, 2024
  • 1 minute(s) read
  • AF
  • Alejandro Millalen
 Prev Next

With Syniti Replicate, you can replicate relational data to a BigQuery dataset. BigQuery is currently supported as a Target both in Refresh and Mirroring replications.

Before setting up Syniti Replicate to replicate data to BigQuery, install the BigQuery and AVRO Serialization libraries.

Google Client Library to Access the BigQuery API

  1. Download the Google.Cloud.BigQuery.V2 (version 3.5.0) nuget package from the following website: https://www.nuget.org/packages/Google.Cloud.BigQuery.V2/3.5.0.

  2. Copy the following assemblies to the Syniti Data Replication Plugins/BigQuery folder:

    • Google.Api.Gax.dll v4.4.0.0

    • Google.Api.Gax.Rest.dll v4.4.0.0

    • Google.Apis.Auth.dll v1.60.0.0

    • Google.Apis.Auth.PlatformServices.dll v1.60.0.0

    • Google.Apis.Bigquery.v2.dll v1.60

    • Google.Apis.Core.dll v1.60.0.0

    • Google.Apis.dll v1.60.0.0

    • Google.Apis.PlatformServices.dll v1.60.0.0

    • Google.Cloud.BigQuery.V2.dll v3.5.0

    • Microsoft.Bcl.AsyncInterfaces.dll v6.0

    • Microsoft.Extensions.DependencyInjection.Abstractions.dll v6.0

    • Newtonsoft.Json.dll v13.0

    • System.Runtime.CompilerServices.Unsafe.dll v4.6

    • System.Threading.Task.Extensions v4.6

    • System.ValueTuple.dll v4.6

AVRO™ Data Serialization

This is optional, and only needed if you are using AVRO serialization.

  1. Download the Confluent.Apache.Avro (version 1.7.7.7) from: https://www.nuget.org/packages/Confluent.Apache.Avro/.

  2. Copy the following assemblies to the Syniti Data Replication Plugins/AvroSerializer folder:

    • AvroSerializer.dll

    • Confluent.Apache.Avro.dll

    • Confluent.SchemaRegistry.dll

    • Confluent.SchemaRegistry.Serdes.dll

Create a Target Connection for BigQuery

  1. In the Metadata Explorer, choose Targets, then Add New Connection from the right mouse button.

  2. In the Add Target Connection wizard, type a name for the connection and choose Google BigQuery in the Database field.

  3. In the Set Connection String screen, enter values for the following properties:

    Project ID

    The ID of the BigQuery project under which you want to upload data.

    Service Account Key

    Path to the local service account key file..

    Output folder

    Local folder where to create the temporary files before loading them to BigQuery.

    Serialization

    The type of serialization used to send data to BigQuery. Available options are:

    • 1 - CSV (Comma Separated Values)

    • 2 - JSON

    • 3 – AVRO (Typically recommended for performance and to maintain and support compatibility with different versions of schemas.)

  4. Complete the fields in the Target Connection wizard. Use the User Guide available from the Management Center Help menu or the Setup Guide, available for download in the Help Center as needed.

  5. In the Select Tables screen, if tables already exist, choose the tables to use in replication. Note that you can also create tables, or add them at a later time.

  6. Click Next to display the Actions screen.

  7. (Optional) Select the next wizard that you would like to run.

  8. Click Next to display the summary, then click Finish to create the connection.

  9. The next step is to add target output representation to the Metadata Explorer. This will be represented as relational tables.

Now you can set up replications from the source connection you have defined.

Limitations

Fields that contains forward slash (/) or backslash (\) are not supported in BigQuery.