Adding & Editing Datasources
  • 17 Oct 2024
  • 17 Minutes to read
  • Contributors
  • Dark
    Light
  • PDF

Adding & Editing Datasources

  • Dark
    Light
  • PDF

Article summary

This topic relates to the following sections:

About Datasources

This section provides details for the maintenance of System Datasources. This section does not discuss the process for maintaining Working Datasources or Snapshot Datasources. A Datasource is defined as a connection to a system at a specific level to provide the means of transferring data between that datasource and the client database.

View the following page by navigating to Catalog > Datasources from the Syniti Migrate Homepage. On the Datasources multi-panel page, the data is filtered by Applications, Instances, and Datasources. The wizard provides guidance for building the datasource correctly and to prevent adding unnecessary field values or leaving valuable fields blank.

This section refers specifically to the System Datasources. The remaining datasources are built in the following sections of the system:

Datasources: Main page: Hierarchy level

Prerequisites to Adding a New Datasource

  • System Datasources must first be set up within the Syniti Knowledge Platform in the Data Catalog. Refer to section Initial Setup - First Week for details of creating the Connector and Connections.

  • The remaining Datasources must also connect to a Datasource Server maintained within the Syniti Migrate system. This would include the Migrate, Reporting, and Construct datasources. Refer to section Initial Setup - First Week for details of the database preparation steps.

Datasource Purpose

Although there are five Datasource Purpose options, this section provides the steps for building System Datasources or Reporting Datasources. All other Datasource Purpose options have referred sections as provided above. As the user completes field values, supportive fields become available for entry. The fields vary by response of a prior field.

Datasource Purpose. The response alters the following tabs and required fields to meet the needs of the datasource purpose. There are seven options for selection and each option is discussed below:

  • System - These are originating systems Ex: JDE, BPCS, Cloudera, ECC, SRM - Stored within Applications / Instances of the same naming structure. These Datasources pull from Syniti Data Catalog systems. Refer to section Add a System Datasource below for details

  • Snapshot - These are the point in time captures of an originating system used as a Source or Target - Stored in Syniti Migration Application (Ex: SRCJDE, TGTS4H) Refer to section Migrate > Snapshot Management to add or edit a Snapshot Datasource.

  • Construct Snapshot - This Datasource (SRCCONSTRUCT) stores the Snapshot data from the CONSTRUCT database within SQL Server for Migration - Stored in Syniti Migration Application. The connection is to the Working Database for migration. Refer to section Migrate > Snapshot Management to add or edit a Snapshot Datasource.

  • Construct - The CONSTRUCT datasource is a system datasource - Stored in Syniti Construct Application (SQL Server connection). Refer to section Add a Construct Datasource below for details.

  • Migrate - This Datasource (MIGRATE) is added by the system automatically - Stored in Syniti Migration Application. It stores the Value Mapping data for migration. Once it is modified to add the Migration Database Server, the system automatically pulls in the Tables and Functions of this Datasource. Refer to section Adding & Editing Migrate Datasources for details of this process.

    Note

    Datasource Purpose field locks once the Datasource is saved. The value for this field may be viewed within the grid of Datasources on the Datasources main page.

Add a System Datasource

All legacy system datasources must use the Knowledge Platform Data Catalog system to associate the System Datasource to an Agent and Connection. On the Datasources panel, click the Add icon to view the list of possible connections for data connectivity and launch the appropriate Details window.

Icon

Description of Use

Application - The system being used as a source or target.

Instance - within the Application, the instance is the instance of the application. There may be multiple instances of the same application.

Datasource - within the Instance of the Application, the Datasource contains the connection information for that datasource. The term datasource implies that one may be a source or target system.

While displaying the Hierarchy list of Datasources, the user should highlight the Instance in which this new Datasource resides. Click Add icon and select Add New Datasource from the drop down list to display Create a new datasource window.

Once the required datasource is added, access Monitor > Debug Log to view the log messages to monitor the datasource update process. Refer to Verify Migrate Database Upgrade for more information.

Usage Tab

This is a step by step setup that includes adding data to the Usage, Details, Connection, and Attributes tabs of the Wizard process. In this example, the Datasource Purpose selected is System. System datasources import from Knowledge Platform Data Catalog System.

Datasources: Add New System Datasource: Usage

Fill in the Usage fields for this Datasource.

Section

Field

Description

Datasource Usage

Datasource Purpose*

Select System from the list. (Required)

Import All

Active Only for System Datasources - Toggle to import every table from this datasource (this flag tells the system to ignore Administer > Advanced > Applications values for this Datasource.

Note

Defaults as Inactive and is relevant to non SAP Datasources.

Security Profile

Security Profile

Security Profiles in Migrate allow users to extend role-based security settings on an individual asset or component, such as Datasets, Datasources, and Releases. When a security profile is applied to an asset, only users associated with user groups and roles designated on the security profile have visibility or access to that asset. For example, if a security profile is applied to a release, then the subject areas, datasources, reports, mappings, ETL Jobs, Metrics, and all the granular details of a release are restricted from other users.

Refer to Security Profiles and Security Profiles in Migrate for more information on implementing and managing security profiles.

Note

This datasource example for an ECC source system does not require adding a Snapshot Datasource. The SAP Datasources do not use the Import All field.

Click the Next button to proceed to the Data Catalog tab.

Data Catalog Tab

Within the Data Catalog tab connects the Datasource to an active Datasource within Syniti Data Catalog, the user selects the Database Type and clicks Select Value button within the Selected Catalog Asset panel. This produces a list of system Asset Types (databases, tables, or schemas) to select from.

Datasources: Add New System Datasource: Data Catalog tab

Select to highlight the row of Asset based upon the Asset Name and Asset Type, and then either click Save icon or double click on that line to add it as the Datasource Connection.

Datasources: Add New System Datasource: Data Catalog tab: List Assets

The page fields are automatically filled in by the selected Asset.

Datasources: Add New System Datasource: Data Catalog tab

The Data Catalog fields for this Datasource.

Section

Field

Description

Select Database Type

Database Type*

Select the database type.

Selected Catalog Asset

System Friendly Name

The System Friendly Name value displays.

Selected Catalog Asset

System Name

The System Name displays.

Selected Catalog Asset

Name

The Name of the asset displays.

Selected Catalog Asset

Description

Enter the description of the Datasource.

Selected Catalog Asset

Friendly Name

The Friendly Name displays.

Selected Catalog Asset

Path

The path for this asset displays.

Click NEXT button to proceed to the Details page tab.

Details Tab

The Details tab is standard for all Datasource Purpose types.

Datasources: Add New Datasource: Details

Fill in the Details fields for this Datasource.

Note

Once every required field has been entered, the NEXT button becomes active.

Field

Description

Application*

Select the Application. (Required)

Instance*

Select the Instance. (Required)

Datasource Name*

Enter the Datasource Name. (Required)

Description

Enter the description of the Datasource.

Active

Toggle to make the Datasource Active/Inactive. NOTE: Defaults as Active.

Comments

Enter descriptive comments to support this datasource.

Click the NEXT button to proceed to the Attributes page tab.

Attributes Tab

Fill in the Attributes fields for this Datasource. There may be Attributes and Stakeholders stored on this tab.

Select the value for Owner which is required, and click Finish button to Save and close the wizard dialog box.

The system clocks and if successful the new Datasource displays in the list as expected. Refer to the section below Verify the Connection for subsequent steps in this process.

Datasources: Add New Datasource: Attributes

Note

If the Import All toggle was not selected, then the user must update the Applications for the tables and possibly columns that need to be imported for this Datasource. Refer to section  Administer > Advanced > Applications for more details of filtering the Import of Metadata to the Datasource. The Reporting Datasource Types do not Import Metadata so this option is not present.

The new Datasource displays in the list for the Instance highlighted.

Note

During the Save, if the user selects YES to import the Metadata for the new Datasource, there is no Job Queue or Debug Log postings for this effort. If the import is not successful, a message briefly displays stating 'Unable to import metadata due to invalid connection.'  At this point, check the server information and fix the connection.

Note

Once the system datasource is saved, the user may access in Edit mode to update for field Client. This field filters the data replicated into the snapshot table from the system datasource for the client value. See section SAP Datasources filtered by Client below for details of this filter.

Add a Construct Datasource

The CONSTRUCT Datasource is stored within the Syniti Migration Application / Instance as it is set up on the client's database. This datasource stores all of the construction tables and views used in the Syniti Construct system.

To add a construct datasource:

  1. Select the Syniti Migration Instance and click Add icon.

  2. Select Add New Datasource from the list. The Create a New Datasource wizard is displayed.

It is a step-by-step setup that includes adding data to the Usage, Details, Connection, and Attributes tabs in the wizard, as described below.

Usage Tab

In the Usage tab, select Construct as the purpose from the Datasource Purpose list of the Datasource Usage section. The CONSTRUCT datasource stores tables that are actively maintained throughout the project, while the SRCCONTRUCT datasource stores snapshot copies of same tables for use during the testing periods. Click the NEXT button to proceed to the Details page tab.

Note

Once a Construct datasource is created as a Construct Datasource Purpose, the options for the Datasource Purpose list hides the Construct value. Only one Construct Datasource Purpose (Enrich) can exist for a Syniti Migration instance.

Datasources: Add Construct Datasource: Usage

Details Tab

The Details tab is standard for all Datasource Purpose types.

Datasources: Add New Report Datasource: Details

Fill in the Details fields for this Datasource.

Field

Description

Application*

Select Syniti Migration as the Application. (Required)

Instance*

Select Syniti Migration as the Instance. (Required)

Datasource Name*

Enter the Datasource Name. (Required)

Description

Enter the description of the Datasource.

Active

Toggle to make the Datasource Active/Inactive. NOTE: Defaults as Active.

Comments

Enter descriptive comments to support this datasource.

Note

Once every Required field has been entered, the NEXT button becomes active.

Click NEXT button to proceed to the Connection page tab.

Connection Tab

The connection information of the Datasource is entered on this tab of the Report Datasource.

Note

Fields on the Connection tab are contingent upon the Server selected and the Tool set in Parameters, so if there were no connection for this Datasource, click Next immediately to advance to save this datasource.

If the database selected is SQL Server or HANA DB, both Database Name and Default Schema are required.

If the server selected Oracle or Oracle_ODBC Servers, Database Name is hidden, and Default Schema is a required field.

Datasources: Add New Report Datasource: Connection tab

Fill in the Details for this Datasource Connection.

Section

Field

Description

Connection

Server

Select the available Working Database Server.

Refer to section Administer > Setup > Servers for details.

Connection

Database Name*

Enter the Database Name. (Required) Contingent upon Server

Connection

Default Schema*

Enter the Default Schema. (Required) Contingent upon Server

Connection

Timeout (seconds)

The number of seconds this Datasource refreshes before it is stopped. Prevents excessive clocking. The default with no value is set by the Job Queue and Job Type, but generally defaults to 30 seconds.

Connection

User Name

This field is contingent upon the setup of the Datasource Server and value may become required.

Connection

Password

This field is contingent upon the setup of the Datasource Server and value may become required. Click on the Show/Hide icon to view the Password entered.

SAP Data Services

Datastore

Name of the Data Services Datastore from which Jobs pull Source data. This Datastore is used in the Syniti Migrate automation XML.

Click the NEXT button to proceed to the Attributes page tab.

Attributes Tab

Fill in the Attributes fields for this Datasource. There may be Attributes and Stakeholders stored on this tab.

Select the value for Owner which is required, and click Finish button to Save and close the wizard.

The system clocks and if successful the new Datasource displays in the list as expected. Refer to the section below Verify the Connection for subsequent steps in this process.

Datasources: Add New Datasource: Attributes

Verify the Connection

Upon saving the new Datasource, the User should verify that the connection is active. Select the Datasource and click the Connection Icon to initiate a check against the system. If the connection is successful a message displays " Primary - Connection success."

Datasources: Datasource: Verify Connection

If the Connection is not successful, the tables and columns do not import and metadata does not process successfully. The Job Queue runs, but no outcome of data is present. This verification is extremely important as well as the setup of the Datasource and Snapshot.

Datasources: Verify Connection: Error Messages

Edit a Datasource

To edit a datasource, click on the Edit icon to display the Datasource Details page. The details of the datasource depend upon the type of datasource as a System, Report, Snapshot, or Working type.

Datasources: Datasource: Edit Details

Note

Once a Datasource is linked to a Snapshot Datasource, this link is permanent and cannot be removed. The value in the field is display only. This way the table data does not become fragmented and incorrect.

Note

Once the User Name and Password have been saved into a Datasource, only the Admin role has access to view the Password going forward. The Show/Hide icon for viewing the password is not available to anyone who is not provided the Role of Admin.

The Details page for Datasource is very similar to Instance and Application in that there is a left side panel with Details, and the tabbed section on the right side panel that contains associated object data. The list of fields below are for the System Datasources.

Section

Field

Description

Details

Application*

Select the Application. (Display Only)

Details

Instance*

Select the Instance. (Required)

Details

Datasource Name*

Enter the datasource name. (Required)

Details

Description

The description of the datasource.

Details

Active

Toggle to make the datasource Active/Inactive.

Note

Defaults as Active.

Details

Comments

Enter descriptive comments to support this datasource.

Syniti Catalog System

Select from Catalog

This button allows for the selection of the link to a Datasource within Knowledge Platform Data Catalog system.

Syniti Catalog System

Database Type

Select from the drop-down list the available Database type. See Note below for details.

Syniti Catalog System

System Friendly Name

Displays the system friendly name for this datasource provided by Data Catalog.

Syniti Catalog System

System Name

Displays the system name for this datasource provided by Data Catalog.

Syniti Catalog System

Path

Displays a path for the datasource provided by Data Catalog.

Syniti Catalog System

Client

Displays the client used for SAP Datasources. See SAP Datasources filtered by Client section below.

Syniti Catalog System

Database Name

Displays the Database Name for this Datasource.

Syniti Catalog System

Default Schema

Displays the schema for this datasource. See Note below for details.

Stakeholders

Owner*

Select the Owner responsible for this Datasource. (Required)

Metadata

Import All

Toggle to flag import metadata for all tables of the Datasource

Metadata

Last Refresh Date

Date of the last refresh of metadata.

Audit

Various

These are the system generated values for auditing the work.

ID

Datasource GUID.

SKP Asset ID

Displays the Asset ID for the datasource provided by Data Catalog.

Note

If within the Syniti Catalog System section, the Database Type value is SAPNETWEAVER, then Default Schema must be left blank for the system to properly scan and refresh the table data for the datasource. For all other Database Types, the Default Schema value is necessary.

At the header of this panel are quick use icons as shown below:

Icon

Name

Use

Save

The Save icon saves the current entries.

Cancel

The Cancel icon leaves the current page without saving changed data.

Help

The Help icon links to the page in Syniti help text to provide a detailed explanation of this section and use.

Additional Options

This icon produces a list of options for the Datasource including -      

    Import Metadata

    View Debug Logs

    View History

Refer to sections below that discuss these options and use.

SAP Datasources filtered by Client

The SAP System Datasources use the field Client to determine which client filter of data within a server is used. To provide a filter for all tables of a datasource, the field Client is provided within the Datasource Details page in section Syniti Catalog System as shown below. This field is then passed onto the Replicate system for the snapshot datasource linked to this system datasource. Individual tables within a datasource may also be assigned a Client, however, in the system datasource, the client applies to all tables of that datasource.

Note

When the Database Type is SAPNETWEAVER, leave the Default Schema field blank, otherwise the scanning process will fail.

When finished with edits, click the Save icon to complete.

Edit the Tables or Columns

At this point, the user may add new Tables and Columns to the existing Datasource. They may also edit the details section to update field values. Refer to section Adding & Editing Tables or Columns for details.

Additional Options Icon for a Datasource

Within this icon of the Datasource there are four options:  Import Metadata, Import Metadata Delta, View Debug Logs, and View History. This is a quick step process that allows the user to perform tasks without having to modify using the Edit process.

Import Metadata - Refer to section Catalog > Datasources > Adding & Editing Datasources > Import Metadata for details of this process.

View Debug Logs - A click on this option opens the Debug Log and filter to job logs for this Datasource.

Note

The Download Debug Zip File icon will produce a file on the local drive with details of this table data.

View Debug Logs

View History - Click on this option to view the history of changes for this system datasource. A window displays rows of changes that may be downloaded to a local drive.

Note

The Download icon will produce a file on the local drive with details of this table data.

View History

Delete a Datasource

Should the System Datasource not be relevant to the project it may be removed from Syniti Migrate. Generally, once tables are imported to the Datasource it is considered a required datasource, however, if the tables are not set as references to Datasets, Mappings, Value Mappings, etc. it may be deleted from the system.

In the image below, the system datasource JDE contains many tables of which table F0004 contains References. In this case, the system datasource cannot be removed.

Datasources: System Datasource

A warning message displays after a click on Delete icon. Click OK to proceed.

The Validation Error message indicates that the system datasource contains datasource tables with references and therefore cannot delete. The additional warning messages are ignored due to the Error message. Click OK to exit this action.

Datasources: System Datasource: Delete Validation Error Message

In the case where the System Datasource does not contain tables with references, it may be removed.

Datasources: System Datasource: without References

There are no references attached to the Datasource tables, and only warning messages display. Click on CONTINUE DELETE button to proceed to remove this system datasource table from the list. The warning messages indicate that the tables are removed and the snapshot relationships are removed with this action.

Datasources: System Datasource: Delete Validation Warning Message

Datasource Objects

On the right side of this page is the Tab for Tables & Columns. The Administrator or IT Lead is responsible for this tab of data. This page provides two panels - one for Datasource Tables, and then one for Datasource Columns. Refer to section Catalog > Dataset Design > Maintain Tables & Fields for more details on this topic.

Subsequent Tasks

  • Upon saving the Datasource, the User should attempt to Import metadata (import the tables and fields as they exist in the tables in the system identified). Refer to section Catalog > Datasources> Adding & Editing Datasources > Import metadata for details of this subsequent process in Syniti Migrate.

  • In the case of a Snapshot Datasource, the User would IMPORT TABLES from the source Datasource to the Snapshot Datasource (all Active tables), and then Import metadata. Refer to section Migrate > Snapshot Management for details of this subsequent process in Syniti Migrate.


Changing your password will log you out immediately. Use the new password to log back in.
First name must have atleast 2 characters. Numbers and special characters are not allowed.
Last name must have atleast 1 characters. Numbers and special characters are not allowed.
Enter a valid email
Enter a valid password
Your profile has been successfully updated.
ESC

Eddy AI, facilitating knowledge discovery through conversational intelligence