Import / Export Dataset

This topic contains the following sections:

• Create and Maintain Datasets

• Maintain Data Elements

• Bulk Edit of Existing Data Elements

• Maintain Tables and Fields

• Maintain Migration

About Import / Export Process

During a multi-release project, a Dataset may be used multiple times within multiple Releases, and therefore it becomes necessary to export the data into a file and then import the same Dataset information for use in another Release or system. This process exports only the setup of the Dataset to include the Tables, Fields, Record Types, and the Data Element details. The import of Dataset is inserted to an existing Subject Area within the new system or new release of same system. This process provides a shell for many large objects as well as providing a more accurate way to repeat an objects migration over many releases.

In the export of Mappings details, only the migration of data is captured. The Sources that have associated data mappings are included (Sources without mappings do not export). Mappings are not included in the Import process.

Prerequisites

Prerequisites for exporting the Dataset include building out the Dataset with the appropriate details for tabs Table and Fields, and Data Elements. 

Prerequisites for exporting the Mappings and Reports include the dataset being imported to a Release with Sources that contain active field Mappings as well as Reporting.

Note

The export of Mappings includes all details for successful future import of same structures.

Export Dataset

The Export Dataset process provides a locally stored file of details for the Dataset and associated Mappings/Reports. The details of the Dataset are not copied into the new Dataset during the import. The export includes Tables, Fields, and Data Elements from the Dataset.

From within the Mappings, the export includes migration interfaces and only sources which have associated data mappings. All reports are included although they are now identified by the COR_REPORT AUTOGENERATED bit field where we can identify the Migration generated reports against the custom reports. This field is being provided to help differentiate between auto-generated and custom reports on import.

While viewing the Dataset in Edit mode, click on the Options  icon to view the list of options and click on Export Dataset to initiate the download of the file.

A click the Export Dataset option produces a file stored in the local Download directory as type CCP.

The exported file contains a JSON object with the following root-level properties/objects:

Dataset -

• exportUserName - the name of the user who exported the file.

• exportTime - the date/time the file was exported.

• version - an object containing version information of the Migration instance from which the file was generated.

• dataset - an object containing a serialized representation of our Dataset type.

• datasetTables - an array of objects containing a serialized representation of our DatasetTable type, including the child DatasourceTable type.

• dataElements - an array of objects containing a serialized representation of our DataElement type, along with child objects of the following: DatasourceColumn, DatasetTable, DataElementType, DataElementType.DataClassification,         DataElementType.SOR.

• recordTypes - an array of objects containing a serialized representation of our RecordType type.

• recordTypeElements - an array of objects containing a serialized representation of our RecordTypeElement type, along with child objects of the following types: RecordType, DataElement, DataElement.DatasetTable.

• importTranslationRules - An array of "Import Translation Rule" objects.

Mappings -

• interfaceTargets - the active target tables listed in the Dataset (REL_INTERFACE_TARGET field)

• interfaceSources - the active source tables assigned to targets in the Dataset (REL_INTERFACE_SOURCE field)

• interfaceDataMaps - the mappings of target fields to source fields within each Source table (COR_DATA_MAP table data)

• interfaceDataMapLineages - the lineage of each mapping to the fields and tables to form the mapping (REL_MAP_LINEAGE table data)

Reporting -

• migration interfaces - COR_INTERFACE definitions for all associated Migration interfaces

• reportTranslations - All REF_REPORT_TRANSLATION definitions (note, these are not directly related to reports in the database, so all are exported)

• migrationReports - All COR_REPORT definitions for associated Migration interfaces. Note that all are being exported to ensure that report links could be re-established on import

• migrationReportLinks - All REL_REPORT_LINK definitions for associated Migration interfaces

• migrationReportLinkBindings - All REL_REPORT_LINK_BINDING definitions for associated Migration interfaces

• migrationReportSchemas - All schema definitions for report views and their dependent objects, from the working database of all associated Migration interfaces

Import Translation Rules -

A note on the "Import Translation Rules" - the purpose of these rules is to support future changes to the object model. So, for example (hypothetically):

• In v3.3 the Dataset object has a property named "ShortName".

• In v4.0 the Dataset.ShortName property is renamed to "TerseName".

• An export is generated from v4.0; the property is named "Dataset.Tersename" (matching the v4.0 object model.

• This exported file is then imported into a v3.3 system. Because the v3.3 object model does not contain a Dataset.Tersename property, this would be ignored, and Dataset.ShortName would remain "NULL". 

To support this type of situation, the v4.0 instance can export an "Import Translation Rule" which can instruct the v3.3 on how to handle this change ... in this case, that it should rename the "Dataset.Tersename" property to "Dataset.Shortname" before importing. In the export code, a new ImportTranslationRule may be added to address this object model change. This rule is included in the exported file, and processed on import.

There are currently 3 different types of Translation changes which can be processed:

• NEWKEY - change the key of a JSON property.

• NEWVALUE - change the value of a JSON property.

• REMOVE - completely remove a JSON property.

Import Dataset

The Import Dataset process imports to an existing Subject Area and Dataset the information stored within tabs Data Elements, Tables and Fields, and Record Types. It includes the updates made that determine relevancy and keys fields of each table.

Prerequisite to Import of a Dataset

Create within the Migration Application a Subject Area and Dataset in which to store this file's data.

Produce an Export Dataset file that is stored and used for the import process.

Once the Subject Area and Dataset have been created in the system, the user may click on the Options  icon within Dataset Details and then select and click on Import Dataset to initiate the process.

A window displays for the Dataset Import. Determine whether this import Overwrites an Existing Dataset or import new and update the Toggle for Overwrite Everything. Next, Click on the Choose File button to open the file directory and select the pathway and filename to import.

Note

The filter for the file directory ONLY includes files of type .CCP

From within the Local file directory, select the .CCP file and then click on the Open button to insert this filename in as the file chosen for Import to the Dataset.

If the user selected Overwrite Existing? as active, a warning displays. At this point, Click on the IMPORT button in the header bar of the window.

The system now begins importing all of the file data for this dataset into the system. A clocking icon displays as it processes.

Import Logic

In general: 

If any "Import Translation Rules" exist in the imported file, these are processed first.

Any property that exists in the import file, but does NOT exist in the current object model, is ignored.

Any property that exists in the current object model, but does NOT exist in the import, is left as a default value.

The import is processed as follows:

1. If user is overwriting the existing Dataset:

   Remove all DataElements from the Dataset.

   Remove all RecordTypeElements from the Dataset.

   Remove all DatasetTables from the Dataset.

   Remove all RecordTypes from the Dataset.

2. Process any "Import Translation Rules" found in the import.

3. Add all DatasetTables found in the import, if does not exist.

4. Add all RecordTypes found in the import, if does not exist.

5. Add all DataElements found in the import, if does not exist.

   Add DatasourceColumn, if does not exist.

   Add DataElementType if does not exist.

   Add DataElement object.

6. Add all RecordTypeElements found in the import, if doesn't exist.

The resulting dataset is setup with Data Elements, Tables and Fields, and Record Types if available and is ready for use in Migration.