Configuring Partitions for Refresh

Overview

Partitions enable you to split data into separate execution units during a Snapshot (Refresh) replication by applying source and target filters (WHERE conditions). Replicate Preview automatically applies partitioning depending on the record count and number of columns, which is usually applied to large tables with a minimum of 500K rows or 100 MB in size. However, you can overwrite this auto-partitioning functionality by adding custom partitions as needed.

By creating multiple partitions, the refresh operation can use parallel threads to move different subsets of data, significantly improving performance. Each partition uses optimized filter conditions to ensure that read and write operations are distributed evenly, allowing multiple readers and writers to process data in parallel for faster execution.

Note

• Source filters defined within partitions are applied to the results already filtered by the source filter defined in the replication settings.

• Currently, the following types of datastore connections do not support both automatic and custom partitioning features:

  • File Storage Connections

  • OData

  • Syniti Drive (File Import/Export)

When to Use Custom Partitioning

Auto-partitioning is smart and works well for most scenarios. However, sometimes you know your data better than the system does. Consider custom partitioning when:

• A split on the primary key won't give optimal results

• Your data has natural logical boundaries (for example, by date ranges, regions, or business units)

• You want to control which data is processed in each partition based on business logic

With custom partitioning, you define the WHERE clauses that determine how data is split, and the system runs them in parallel.

Create a Custom Partition

1. In the Replications panel, select the required replication and click the Partitions icon. The Partitions dialog box is displayed.

2. Click + Add Partition to create a new partition.
   OR
   Select the required primary key fields and click Auto-Generate to automatically generate partition filters based on the selected fields.

3. Double-click the Name to rename the partition if required. The default value is “Partition ##" with a sequential number within the replication.

4. Double-click the Source Filter field to enter a WHERE condition that defines which records are read from the source table for this partition. For example, OrderID < 15000 .

5. Double-click the Target Filter field to enter a WHERE condition that specifies which records are written to the target table for this partition. For example, OrderID < 15000 .

6. Click Save.

Custom partitions are now created successfully.

Note

If any partition fails during replication, you can either re-run the failed partition or the entire replication based on your replication settings and partition strategy. Refer to the table below for more information.

Replication Execution With/Without Partitions

Replication behavior varies depending on your replication and partitioning configuration. The table below indicates when truncation is applied during execution:

Partition Re-run

To re-run failed partitions:

1. In the Replications panel, select the required snapshot replication and click the Partitions icon. The Partitions dialog box is displayed.

2. Disable successful partitions and ensure only failed partitions are enabled.

3. Click Save.

4. Execute the snapshot replication.

After successfully re-running the failed partition, enable all partitions for the next replication run.

Related Articles

• Configure Workspace: Datastores

• Configuring Replication Dependencies

• Performing Field Mappings

• Selecting Source Tables