ETL Tasks at the Dataset Level

This topic relates to the following sections:

• Mapping When Using Syniti Migrate ETL Tool

• ETL Jobs

• ETL Jobs Details

• Execution of the ETL Jobs List

• ETL Version Selection

• ETL Tasks at the Release level

• ETL Tasks at the Target level

• ETL Tasks at the Source level

• ETL Tasks Add Complex Rule Manually

Overview

ETL Jobs are accessed from all four levels of the Mappings page. Each ETL icon when clicked displays that level of the overall ETL Job with rows of Job Tasks all linked together. View the following page by navigating to Migrate > Mappings on the Syniti Migrate Homepage.

Mappings

The Dataset level ETL Types consist of four types of Job Tasks (Dataset Transform, Postload, Target Exports, and Target Readiness Reports) that are all components of the conversion of the dataset. Each type of task performs a function and contains distinct tasks to run.

Dataset Transform

The Dataset Transform ETL Type lists each active Target Table of the Dataset; a task that runs the Target Readiness Reports for the target prior; and a task that produces the Target Export data for the target prior. At this level, multiple targets for a dataset can convert in a sequence. The column Action provides links to view details of these tasks.

The ETL Job at this level runs for every active target all of the associated source job tasks including all update Flags, Reports, and the Target Export process.

While executing the ETL Job it may be stopped at a task by editing that task to provide active Debug. Refer to section Setting a Task for Debug for details.

Note

Click the breadcrumb path to move to the linked page for the higher level of this ETL Job list.  In this case, it opens the page for the Dataset ETL Job associated with these Targets.

Note

The Dataset Transform level provides DATA_READY, BUSINESS_READY, STAGE_READY, and TARGET_READY field flag updates as part of the automatically generated list of tasks.

Mappings: Dataset: ETL Job List Details

Within header of this page there are Quick Use icons:

Icon	Name	Use
	ETL Build	This icon builds or rebuilds all ETL Tasks at this level.
	Add	This icon adds a new task to the current list.
	Refresh	Refresh icon updates the page to the latest data.
	Search	Allows for a search of the Tasks grid by Column.
	Execute	The execute icon in Header runs every active task in sequence.

The ETL Job grid displays the following Columns:

The Dataset Transform tasks include the target transform steps and all Update and Refresh of target reports as the Jobqueue List displays.

Postload

The Postload ETL type lists each target's set of Postload Reports. This would be any report marked as Report Type of Postload within the Target Reports page of Mappings. These tasks are built automatically based upon the active target tables assigned to this Dataset.

Mappings: Dataset: ETL Job List Details

The details of each of the Postload Reports tasks provide status and audit information. The fields of the details window are listed below.

Postload: ETL Task Postload Reports: Edit details

If the task is run manually, the Jobqueue steps display for each Postload report executed by this run.

Dataset level: Postload: Jobqueue Steps

Target Exports

The Target Exports action lists each Target Table's Replication Export for the Dataset (File Export). This level can run through each Dataset conversion in sequence to produce export flat files through the Syniti Replicate system.

In the example below, the Execute Replication action is built automatically based upon the Target tables assigned to this Dataset.

Mappings: Dataset: ETL Job List: Target Export grid

The details of each of the Execute Replication tasks provide details, status, and audit information. The fields of the details window are listed below.

A few notes regarding this task:

• Files are created as 1 file per "chunk", which is defined within the Target Table via the "zCHUNK" field. This allows the developer to break up the results based on their needs.

• The files are named {TargetTableName}_{ChunkNumber}.txt.

• Each Target produces an Export file based upon the table and chunk so manually added additional export files are over written as there can be only one export file.

• Whenever a Target Export task is executed, it finds an existing replication target table, deletes the table, recreates the same replication target using the latest information from Migrate, and then runs the Target Export to capture and export the latest details and information in an export file.

Target Readiness Reports

The Target Readiness Reports ETL type lists each Target's set of Target Readiness Reports for the Dataset (Refresh Target Readiness). These reports are considered the Target Ready Reports where a view validates if the dependent object row has loaded to target for this record. As an example, conversion of the Material Master table for Table MARC might run a Target Ready Report to validate that the same material in the MARA table has loaded successfully. This way material in the MARC table loads successfully.

NOTE:  These reports must be set as Report Type of TARGET READY in the Target Reports page of Mappings.

Mappings: Dataset: ETL Job List Details

The details of each of the Refresh Target tasks provide details, status, and audit information. The fields of the details window are listed below.

Postload: ETL Task Postload Reports: Edit details

A click on the Execute  icon for a Task within the Target Readiness Reports Job list initiates the job run.

Warning message for ETL Jobqueue

Target Readiness Reports run in Jobqueue Steps.

Mappings: Dataset: ETL Job List: Target Readiness Reports

Add a Complex Rule

Refer to section ETL Tasks Add Complex Rule Manually for details of this process that works the same at all levels of the Mappings to include Release, Dataset, Target, and Source levels.

Execute the ETL Job at the Dataset Level

Refer to section Migrate > Mappings > Migration ETL Job > Execution of the ETL Jobs List for details of this process.