Working with the Replications Panel

Columns in grids throughout the Syniti Knowledge Platform (SKP) can be individually customized, sorted, and filtered. You can also search for names or values for in columns that have the Search feature enabled. The selected settings are remembered per user. Refer to SKP Grid Searching, Sorting, and Filtering for more information.

To create a replication for certain use cases (for example, move only a subset of the source table data to the target table) or specify different column attributes and names for your target table, you can add a replication manually or edit an existing replication.

To add a manual replication:

1. In the Sources panel, select the required source datastore associated with the intended target datastore and click the Add Replication (+) button in the Replications panel. The New Replication dialog box is displayed.

   Note

   Ensure that the selected source datastore and its corresponding target datastore is displayed correctly.

2. In the Source section, tables available in the source datastore is displayed. Select the required table.

   1. If required, you can filter records from the source table using the Source Filter field. For example, you can specify a condition (WHERE clause) to ensure only relevant records are inserted into the target table.

      Note

      You don't need to explicitly include the WHERE keyword in the Source Filter field. For example, if you want to replicate only records where the COUNTRY field equals 'US', simply enter COUNTRY = 'US' in the Source Filter field.

      Note

      If this replication will be included in an Orchestrate workflow of custom type, you can enter variable placeholders {{VariableName}} within the Source Filter field to update replication source filters dynamically.

3. In the Target section, the Target datastore is displayed. If no database or schema values are specified in the target datastore, then Source datastore’s database and schema values will be used.

   1. In the Target Name field, enter the table name corresponding to the source table name.

   2. To replicate by dropping and recreating the table instead of truncating the data, turn on the Drop Target Table toggle.

      Note

      Ensure that the Append Mode toggle is disabled when using Drop Target Table during replication. Replication will fail if both toggles are enabled.

4. In the Additional Properties section, complete the following steps:

   1. To specify higher preference for this replication over others, enter a priority value in the Priority field.

      Note

      The replications will be ordered and executed using the Priority value. By default, the value for each replication is set at 1000, which provides equal preferences for all replications, but ordered using the creation time of a replication.

   2. To replicate data without truncating data in the target table, turn on the Append Mode toggle button to append new data from the source table.

   3. Select Pre-Execution and Post-Execution Workflows to run if this replication is integrated with other module’s tasks in an Orchestrate workflow.

5. Click Save.

A replication for the selected source datastore is added to the current workspace.

You can add an Orchestrate workflow to integrate with other module’s capabilities directly from Replication workspaces as part of your replication execution sequence, broadening data migration, quality, and replication use cases. For example, you may want to execute a data quality report, SQL command, and so on.

Complete the following steps to create an Orchestrate Workflow:

1. From the Targets and Sources panels, select the required Target and Source datastores. A list of replications configured for this target and source datastores combination is displayed in the Replications panel.

2. In the Replications panel, click More Actions > Create Orchestrate Replication. The New Orchestration Replication dialog box is displayed. Ensure that the selected the source datastore is displayed on the Source list.

3. From the Orchestrate Workflow list, select a workflow associated with the source datastore selected in the previous step.

4. If required, modify the Target Name field used as the replication name in the workspace. By default, Replicate Preview uses the Orchestrate Workflow’s Name for the replication name.

5. If you want to specify higher preference for this replication type over others, enter a priority value in the Priority field.

   Note

   The replications will be ordered and executed using the Priority value. By default, the value for each replication is set at 1000, which provides equal preferences for all replications, but ordered using the creation time of a replication.

6. Click Save.

You can import data from flat files using Syniti Drive as the source datastore and an SKP Catalog System as the target datastore. In File Import replications, you can also perform field mappings, similar to the way you would map fields when customizing the target table configuration based on the file.

To create file import replications:

1. In the Targets panel, select the required SKP Catalog System as the target datastore.

2. In the Sources panel, select the required Syniti Drive as the source datastore.

3. In the Replications panel, click the Add Replication (+) button. The New Replication dialog box is displayed.

4. In the Sources section, select the required file available from the Files list.

   Note

   • The Files list is contingent upon the selected file format. Currently the following file formats are support for file import replications:

     • .csv

     • .json

     • .parquet

     • .xlsx

   • A maximum of 10,000 files can be searched when selecting files for replication.

   For example, if you select a .csv file, enter the following details to parse data from the file:

   Field Name	Description
   Delimiter	Defines the character used to separate values in the file. Default is comma (,). See Supported Delimiters.
   Encoding	Specifies the text encoding used in the file to ensure special characters are read properly. Default encoding format is UTF-8.
   Header Row	Indicates which row (by index) in the file contains the column names. For example, set to 0 if the first row contains the headers.
   Column Names	Specify the column names that match the destination columns using the following format: ["Column1", "Column2", "Column3"]. New columns will be created if the specified columns do not exist.

5. In the Target section, the selected target datastore is displayed. Complete the following steps:

   1. In the Target Name field, enter the table name corresponding to the source table name.

   2. To replicate by dropping and recreating the table instead of truncating the data, turn on the Drop Target Table toggle.

      Note

      Ensure that the Append Mode toggle is disabled when using Drop Target Table during replication. Replication will fail if both toggles are enabled.

6. In the Additional Properties section, complete the following steps:

   1. To specify higher preference for this replication over others, enter a priority value in the Priority field.

      Note

      The replications will be ordered and executed using the Priority value. By default, the value for each replication is set at 1000, which provides equal preferences for all replications, but ordered using the creation time of a replication.

   2. To replicate data without truncating data in the target table, turn on the Append Mode toggle button to append new data from the source table.

   3. Select Pre-Execution and Post-Execution Workflows to run if this replication is integrated with other module’s tasks in an Orchestrate workflow.

7. Click Save.

Supported Delimiters

The following delimiters are not supported:

• Space

• Carriage return \r

• Newline \n

You can export data to a file in Syniti Drive by using it as the target datastore, with an SKP Catalog System as the source datastore. You can bulk create file export replications using the Source Table Selection feature or create manually.

To manually create file export replications:

1. In the Targets panel, select the required Syniti Drive as the target datastore.

2. In the Sources panel, select the required SKP Catalog System as the source datastore.

3. In the Replications panel, click the Add Replication (+) button. The New Replication dialog box is displayed.

4. In the Source section, complete the following steps:

   1. Select the required table you want to export.

   2. If required, you can filter out records from this table using the Source Filter field.

      Note
      If this replication will be included in an Orchestrate workflow of custom type, you can enter variable placeholders {{VariableName}} within the Source Filter field to update replication source filters dynamically.

5. In the Target section, complete the following steps:

   1. From the Export As list, select the required file format.

   2. In the Delimiter field, specify the character that separates values in the file. Default character is comma (,). For example, enter comma (,) or pipe (|) if you are exporting data in .txt format.

   3. In the Encoding field, specify the text encoding to be used in the file to ensure special characters are read properly. Default encoding format is UTF-8.

   4. if required, use the Target Name field to modify the file name.

6. (Optional) In the Additional Properties section, complete the following steps:

   1. If you want to specify higher preference for this replication over others, enter a priority value in the Priority field.

      Note

      The replications will be ordered and executed using the Priority value. By default, the value for each replication is set at 1000, which provides equal preferences for all replications, but ordered using the creation time of a replication.

   2. Select Pre-Execution and Post-Execution Workflows to run if this replication is integrated with other module’s tasks in an Orchestrate workflow.

7. Click Save.