Overview
After successfully creating a workspace, you are required to configure your workspace’s datastores details in the Datastores tab to setup your data replication project. Accessing the newly created workspace displays the Datastores tab which is the primary view of a workspace. The Datastores tab drives your project’s workspace. Datastores configuration involves setting up the following components to the newly created workspace:
Prerequisites
Ensure that you have created the Source and Target systems with its datastore connection details in the Catalog module.
Ensure that the connection’s Usage for the required Source and Target datastores is set to Replication.
Ensure that a Replication Hub is registered using an SKP Connector associated with the configured datastore connections.
Creating Target Datastores
A target datastore is a datasource where the snapshot of the source data or tables gets created, generally in a working database server or directly in your target system. Examples of target datastores can include, but are not limited to:
Snapshot datastores with the SRCxxx or TGTxxx prefixes
Working datastores with the WRKxxx prefix
External datastores like target systems or staging areas
Syniti Drives for file exports or reporting
Note
A datastore can be registered as a target datastore only once across all workspaces.
To add a target datastore:
In the Targets panel, click the Add Target button. The Add Target Datastore dialog box is displayed.
From the Type list, select the required datastore type.
From the Datastore list, select the target datastore configured in the Catalog module.
Note
Only datastore connections with Usage set to Replication will be displayed on the Datastore list.
Note
The Default Database and Default Schema values for the target datastores are fetched from the datastore configuration. If the datastore configuration does not include database and schema values, you can manually enter the Default Database and Default Schema values for this target datastore.
If you want to specify higher preference for this target datastore over others, enter a priority value in the Priority field.
Note
The target datastores will be ordered using the Priority value. By default, this value for each target datastore is set at 1000, which provides equal preferences for all target datastore in a workspace, but ordered using the creation time of the target datastore.
Click Save.
Note
You can use Syniti Drive as the target datastore for file exports or reporting purposes.
From the Drives list, select the required drive. If the required drive is not listed, contact your Administrator to create one.
If you want to specify higher preference for this target datastore over others, enter a priority value in the Priority field.
Note
The target datastores will be ordered using the Priority value. By default, this value for each target datastore is set at 1000, which provides equal preferences for all target datastore in a workspace, but ordered using the creation time of the target datastore.
Click Save.
For creating file export replications, refer to Adding File Replications for more information.
A target datastore is created successfully. If you need to create more target datastores, click the Add Target (+) button in the Targets panel and repeat Steps 2 through 5.
Note
Refer to Working with the Targets Panel to learn more about the Targets panel and its usage.
Creating Source Datastores
A source datastore is a datasource where you need to take a data snapshot or copy the data from a source and move it to a target datastore. Examples of source datastores include, but are not limited to:
Legacy systems
Data from Syniti Construct
Data from working datastores
Snapshot tables of the target database for post-load reporting
Data from flat files
To add a source datastore:
In the Sources panel, click the Add Source button. The Add Source Datastore dialog box is displayed.
From the Type list, select the required datastore type.
Note
Ensure that the associated target datastore is displayed in this dialog box.
From the Datastore list, select the required source datastore configured in the Catalog module. The Default Database and Default Schema values for this source datastore are fetched from the datastore configuration. If values for these fields do not exist in the datastore configuration, then these fields will not be displayed in this dialog box.
Note
Only datastore connections with Usage set to Replication will be displayed on the Datastore list.
If you want to specify higher preference for this source datastore over others, enter a priority value in the Priority field.
Note
The source datastores will be ordered and executed using the Priority value. By default, this value for each source datastore is set at 1000, which provides equal preferences for all source datastores associated with a target datastore, but ordered using the creation time of the source datastore.
Click Save.
Note
Ensure that the associated target datastore is displayed in this dialog box.
From the Drives list, select the required drive. If the required drive is not listed, contact your Administrator to create one.
If you want to specify higher preference for this target datastore over others, enter a priority value in the Priority field.
Note
The target datastores will be ordered using the Priority value. By default, this value for each target datastore is set at 1000, which provides equal preferences for all target datastore in a workspace, but ordered using the creation time of the target datastore.
Click Save.
For creating file import replications, refer to Adding File Replications for more information.
A source datastore is created successfully to the selected target datastore. If you need to create more source datastores, click the Add Source (+) icon in the Sources panel and repeat Steps 2 through 5.
Note
Refer to Working with the Sources Panel to learn more about the Sources panel and its usage.
Creating Replications
You must now specify or add the required tables from the source datastore to replicate them to the target datastore.
Selecting Source Tables
The Source Tables Selection feature allows you to bulk create or quickly add only the relevant tables required from the source datastore to the target datastore. Based on the target datastore type, this feature automatically creates individual replications for each selected table using the source table names.
For example, when both the target and source datastores are of the SKP Catalog System type, the feature creates table replications. However, if the target datastore is of the Syniti Drive type and the source datastore is of the SKP Catalog System type, it creates File Export replications instead.
The replications created using this feature will replicate data to the target datastore using the same schema, table name, column properties, and constraints (dependencies) that are imported via the metadata scanned from the Source datastore in the Catalog module. If your Target datastore contains a Schema value, then Replicate Preview will use this default schema for all the replications created using this feature. If no schema is specified, then the Source datastore’s schema will be considered.
To select source tables and bulk create replications:
In the Sources panel, select a source datastore and click the Source Table Selection icon. The Source Tables Selection dialog box is displayed with a list of tables available in the selected source datastore.
If you want to import all source tables into the workspace, select Import All > Confirm; otherwise, skip to Step 3.
Note
Importing all tables may take some time to process (depends on the number of available tables), as it generates individual replications for each table in your workspace.
In the Name column, use the column filter to search for a specific source table using its unique name.
Select the checkboxes of the required tables. Note that the Datasource Tables label displays the number of the selected tables out of the total available tables in this source datastore.
Click Create Replications to save and create replications for the selected tables.
Replications for the selected tables are created and displayed in the Replications panel. If pooled and clustered tables exist in the source system, Replicate Preview automatically selects the RFC connection types for these tables from both source and target systems to create replications. If no RFC connection type exists in the source and target datastore, Replicate Preview uses the default connection and displays an error for these replications when executed.
If you want to create additional replications manually or edit an existing replication to specify filter conditions at the source, refer to the Adding Manual Replications section for more information.
Adding Manual Replications
Selecting the source tables in the Source datastore automatically creates separate replications for each selected table. These replications when executed, will creates target tables in target datastores with the same table and column attributes as the source tables. Source datastore’s schema will be used unless a default schema is entered while adding a target datastore.
If you want to create a replication for certain use cases (for example, move only a subset of the source table data to the target table) or specify different column attributes and names for your target table, you can add a replication manually or edit an existing replication.
To add a manual replication:
In the Sources panel, select the required source datastore associated with the intended target datastore and click the Add Replication (+) button in the Replications panel. The New Replication dialog box is displayed.
Note
Ensure that the selected source datastore and its corresponding target datastore is displayed correctly.
In the Source section, tables available in the source datastore is displayed. Select the required table.
If required, you can filter records from the source table using the Source Filter field. For example, you can specify a condition (WHERE clause) to ensure only relevant records are inserted into the target table.
Note
You don't need to explicitly include the WHERE keyword in the Source Filter field. For example, if you want to replicate only records where the COUNTRY field equals 'US', simply enter
COUNTRY = 'US'in the Source Filter field.In the Target section, the Target datastore and its database and schema values are displayed only when the datastore configuration contains their values. If no values are specified, then Source datastore’s database and schema values will be used.
In the Target Name field, enter the table name corresponding to the source table name.
In the Additional Properties section, complete the following steps:
If you want to specify higher preference for this replication over others, enter a priority value in the Priority field.
Note
The replications will be ordered and executed using the Priority value. By default, the value for each replication is set at 1000, which provides equal preferences for all replications, but ordered using the creation time of a replication.
If you want to replicate data without truncating data in the target table, turn on the Append Mode toggle button to append new data from the source table.
Select Pre-Execution and Post-Execution Workflows to run if this replication is integrated with other module’s tasks in an Orchestrate workflow.
Click Save.
A replication for the selected source datastore is added to the current workspace.
Note
Refer to Working with the Replications Panel to learn more about the Replications panel and its usage.
Adding an Orchestrate Workflow as a Replication
You can add an Orchestrate workflow to integrate with other module’s capabilities directly from Replication workspaces as part of your replication execution sequence, broadening data migration, quality, and replication use cases. For example, you may want to execute a data quality report, SQL command, and so on.
Complete the following steps to create an Orchestrate Workflow:
From the Targets and Sources panels, select the required Target and Source datastores. A list of replications configured for this target and source datastores combination is displayed in the Replications panel.
In the Replications panel, click More Actions > Create Orchestrate Replication. The New Orchestration Replication dialog box is displayed. Ensure that the selected the source datastore is displayed on the Source list.
From the Orchestrate Workflow list, select a workflow associated with the source datastore selected in the previous step.
If required, modify the Target Name field used as the replication name in the workspace. By default, Replicate Preview uses the Orchestrate Workflow’s Name for the replication name.
If you want to specify higher preference for this replication type over others, enter a priority value in the Priority field.
NoteThe replications will be ordered and executed using the Priority value. By default, the value for each replication is set at 1000, which provides equal preferences for all replications, but ordered using the creation time of a replication.
Click Save.
Adding File Replications
File Import Replications
Currently, you can import data from flat files using Syniti Drive as the source datastore and an SKP Catalog System as the target datastore.
To create file import replications:
In the Targets panel, select the required SKP Catalog System as the target datastore.
In the Sources panel, select the required Syniti Drive as the source datastore.
In the Replications panel, click the Add Replication (+) button. The New Replication dialog box is displayed.
In the Sources section, select the required file available from the Files list.
Note
The Files list is contingent upon the selected file format. Currently the following file formats are support for file import replications:
.csv
.json
.parquet
.xlsx
A maximum of 10,000 files can be searched when selecting files for replication.
For example, if you select a .csv file, enter the following details to parse data from the file:
Field Name
Description
Delimiter
Defines the character used to separate values in the file. For example, use
,for CSV files or|for pipe-separated files. Default character is comma (,).Encoding
Specifies the text encoding used in the file to ensure special characters are read properly. Default encoding format is UTF-8.
Header Row
Indicates which row (by index) in the file contains the column names. For example, set to 0 if the first row contains the headers.
Column Names
Specify the column names that match the destination columns using the following format:
["Column1", "Column2", "Column3"]. New columns will be created if the specified columns do not exist.In the Target section, use the Target Name field to modify the table name, if required.
(Optional) In the Additional Properties section, complete the following steps:
If you want to specify higher preference for this replication over others, enter a priority value in the Priority field.
Note
The replications will be ordered and executed using the Priority value. By default, the value for each replication is set at 1000, which provides equal preferences for all replications, but ordered using the creation time of a replication.
Select Pre-Execution and Post-Execution Workflows to run if this replication is integrated with other module’s tasks in an Orchestrate workflow.
Click Save.
File Export Replications
You can export data to a file in Syniti Drive by using it as the target datastore, with an SKP Catalog System as the source datastore. You can bulk create file export replications using the Source Table Selection feature or create manually.
To manually create file export replications:
In the Targets panel, select the required Syniti Drive as the target datastore.
In the Sources panel, select the required SKP Catalog System as the source datastore.
In the Replications panel, click the Add Replication (+) button. The New Replication dialog box is displayed.
In the Source section, complete the following steps:
Select the required table you want to export.
If required, you can filter out records from this table using the Source Filter field.
In the Target section, complete the following steps:
From the Export As list, select the required file format.
In the Delimiter field, specify the character that separates values in the file. Default character is comma (,). For example, enter comma (,) or pipe (|) if you are exporting data in .txt format.
In the Encoding field, specify the text encoding to be used in the file to ensure special characters are read properly. Default encoding format is UTF-8.
if required, use the Target Name field to modify the file name.
(Optional) In the Additional Properties section, complete the following steps:
If you want to specify higher preference for this replication over others, enter a priority value in the Priority field.
NoteThe replications will be ordered and executed using the Priority value. By default, the value for each replication is set at 1000, which provides equal preferences for all replications, but ordered using the creation time of a replication.
Select Pre-Execution and Post-Execution Workflows to run if this replication is integrated with other module’s tasks in an Orchestrate workflow.
Click Save.
Next Steps
After creating replications, Replicate Preview automatically maps source tables’ fields and its datatypes to the target tables with the corresponding attributes. You must access, validate, verify, and configure (if required) the field mappings and replication dependencies before executing replication jobs for the configured replications.
You can begin grouping replications that are related to each other into a single group in the Groups tab. Grouping replications allows you to efficiently organize and manage data replication of your organization’s data for multiple use cases. Refer to Configure Workspace: Groups for more information on creating replication groups and adding replications to a group.