Working in Multiple Execution Environments

Overview

Unlike most software, Migrate code is not promoted through separate Dev, QA and Prod environments as it is tested and defects are fixed. Instead, Migrate converts production legacy data into production business-ready data using the same production code run on multiple execution environments.

How Data Moves Through Environments

Migrate always uses production, legacy data as the same source for all environments, and uses the same set of rules to execute against and load that data. It also uses the same set of reports to validate the data. What changes for each mock load cycle is where the data is loaded, and that is controlled by the environment.

Rather than creating multiple working datasource servers and promoting code through Develop, Quality and Production tiers, Migrate uses one or more environments (for example, DEV and LOAD) on one working database server to develop rules on production code in a DEV environment and cache the reports from the rule’s execution in a DEV Report Cache database. The same production code is also executed on the same production data in the LOAD environment, and the reports from the rule’s execution are cached in a LOAD Report Cache database.

During a Migration project, a mock load cycle could contain many objects, which, considering dependencies on downstream objects and other factors, could take two weeks to prepare for loading. This data must remain static in the LOAD environment so that consistency can be maintained across the load objects for the duration of the mock load cycle. For example, customer data is staged and loaded on the first day of the mock load cycle, but that data must remain static until dependent objects, like Sales Orders, have been loaded.

Each mock load cycle runs in the LOAD environment using static data that can be reviewed for errors, which can then be corrected in the same production code running in the DEV environment. After the mock load cycle is complete, the updated code fixed in DEV can be promoted to the LOAD environment, tested for errors, and the process started again, until the final output of this process, the business-ready data, can be loaded.

In the simplest terms, setting up Migrate for a multi-tier environment means setting up reports to run and be cached in a dev reports database, then run and be cached in a load report cache database. The same production code, rules and reports are run on different environments. In most cases, a DEV and LOAD environment are sufficient, but you can set up as many as needed.

Note

It is recommended that the Working databases and the Report Cache databases all be stored on the same server.

Migrate supports SQL Server, SAP HANA, and Oracle for use as the Working database and the Report Cache database.

To configure Migrate for use with multiple execution environments:

1. Set up the DEV Working Database

2. Set up a LOAD Environment and a LOAD Report Cache Database

   How Color is Used in Environments

3. Set up the LOAD Working Database

4. Compare and Sync Objects in Working Databases

5. Assign an Environment to a Milestone

6. Execute the Rules in an Environment

7. View the Reports in an Environment

Set up the DEV Working Database

Use this DEV Working database to develop, test, and fix the production code that will be run in the LOAD environment during the static mock load cycles in the Migration project.

Note

These steps assume Applications and Instances have been configured. The OTC Instance is used in this example. Follow the naming conventions used in these steps, replacing OTC with your Instance name.

Note

You can set up as many Working databases as needed.

To set up the DEV Working database for the DEV environment:

1. Select Catalog > Datasources in the Migrate menu.

2. Select Applications in the Show list box in the upper left.

3. In the Applications panel, scroll down and double-click Working Databases.

4. In the Application Instances panel, double-click the Name of the Instance where the working database is to be added (in our example, OTC).

5. In the Datasources panel, click the Add icon.

6. Keep the Datasource Purpose of Working and click the Next button.

7. On the Details tab, select the Instance (in our example, OTC) in the Instance list box.

8. Enter wrk[YourInstanceName]D in the Datasource Name field.

   Note

   For our example, enter wrkOTCD.

9. Enter DEV Working Database in the Description field.

10. Select DEV in the Environment list box.

    Note

    By default, the DEV environment uses the REPORT Report Cache Database. No further configuration is required.

11. On the Connection tab, select the DEV database server in the Server list box.

    Note

    Syniti recommends running the DEV and LOAD databases on the same server.

12. Enter the name of the DEV database in the Database Name field.

13. Enter the DEV database’s schema in the Default Schema field.

14. In the XML section, enter wrk[InstanceName] in the Datastore field.

15. Click Next.

16. On the Attributes tab, set another user as the Owner if it is not you, then click Finish.

Note

Both the DEV Working database and the DEV Report Cache database (added by default when this database was associated with the DEV environment) are now configured.

Set up a LOAD Environment and a LOAD Report Cache Database

This database caches the reports run during the mock load cycle in the LOAD environment.

To set up the Report Cache database:

1. Select Administer > Setup > Environments in the Migrate menu.

2. Click the Add icon.

3. Enter LOAD in the Environment Name field.

4. Enter Mock Loads in the Description field.

5. Click the + icon next to the Report Cache Datasource list box.
   

6. Enter REPORTL in the Datasource Name field.

   Note

   You can give the Report Cache Datasource any name, as long as it is not the same name as the Report Cache Datasource for any other environment.

7. Enter LOAD Report Cache Database in the Description field.

8. Select the name of the database server in the Server list box.

   Note

   Syniti recommends running the LOAD Report Cache Database on the same server as the LOAD Working database.

9. Click the Save icon.

10. On the Environment modal, select MIGRATE from the Migrate Datasource list box.

11. Select a color for the data display for this environment from the Color list box.
    

    Note

    Set the LOAD environment color to one other than the DEV environment to make it easier for users to identify which environment the data comes from. This color affects how data displays on reports and is crucial to help users distinguish between the different environments. Refer to How Color is Used in Environments for more information.

12. Click the Save icon.

How Color is Used in Environments

To distinguish between environments, Migrate allows you to define the color associated with that environment. This color determines the color of report data for reports generated from the environment and is used throughout Migrate on relevant pages where actions can be performed on different environments.

Set the Environment color on the Environments page (Administer > Setup > Environments).

On some pages in Migrate, you can switch between environments to view data in each by clicking the tab. For example, you can view report data from different environments on the Migration Reports page (Validate > Migration Reports) by clicking the tab.

As in the example below, when you view reports in the DEV environment, the report data displays in Migrate in that environment’s color.

If you view reports in the LOAD environment, the color changes.

Set up the LOAD Working Database

Use this Working database to:

• execute the production code developed in the DEV Working Database,

• stage the data,

• then load it and leave it static for the duration of the mock load cycle.

Note

These steps assume Applications and Instances have been configured. The OTC Instance is used in this example. Follow the naming conventions used in these steps, replacing OTC with your Instance name.

To set up the LOAD Working database:

1. Select Catalog > Datasources in the Migrate menu.

2. Select Applications in the Show list box in the upper left.

3. In the Applications panel, scroll down and double-click Working Databases.

4. In the Application Instances panel, double-click the Name of the Instance where the working database is to be added (in our example, OTC) .

5. In the Datasources panel, click the Add icon.

6. Retain the Datasource Purpose of Working and click the Next button.

7. On the Details tab, select the Instance (in our example, OTC) in the Instance list box.

8. Enter wrk[YourInstanceName]L in the Datasource Name field.

   Note

   For our example, enter wrkOTCD.

9. Enter LOAD Working Database in the Description field.

10. Select LOAD in the Environment list box.

11. On the Connection tab, select the LOAD database’s server in the Server list box.

    Note

    Syniti recommends running the DEV and LOAD databases on the same server.

12. Enter the name of the LOAD database in the Database Name field.

13. Enter the LOAD database’s schema in the Default Schema field.

    Note

    If the XML section is available, enter wrk[InstanceName] in the Datastore field.

14. Click Next.

15. On the Attributes tab, set another user as the Owner if it is not you, then click Finish.

Compare and Sync Objects in Working Databases

Migrate provides a Data Definition Language (DDL) code comparison tool for use with any Source Datasource and Target Datasrouce that you can access.

When using a single server to support running ETL and reports against multiple environments, you can compare and sync the objects (target tables, source tables, views, stored procedures, reports, etc.) in the DEV Working database and the LOAD Working database to keep them in sync at the start of each mock load cycle.

Note

When the LOAD Working database is initially created, it contains views and pass-through views that are automatically generated in the Working database by default. During the first mock load cycle, objects from the DEV Working database are copied to the LOAD Working database.

In the LOAD Working database, the production code executes on the same production data as the DEV Working database, and the data in the LOAD environment is then left unchanged for the duration of the mock load cycle.

During the mock load cycle, users fix defects in the production code in the DEV environment and run and test it using the DEV Report Cache datasource. At the end of the mock load cycle, the fixed code is once again promoted to the DEV Working database.

These mock load cycles continue until the data is business ready.

Note

This example uses a typical setup with DEV and LOAD Working databases, moving objects from DEV to LOAD. You can move objects between any Source Datasource and Target Datasource, even from a LOAD Working database back to a DEV Working database, depending on which objects are required.

To compare and sync objects:

1. Select Migrate > Working DB Object Promotion in the Migrate menu.

2. Select the DEV Working Database in the Source Database you are moving object FROM list box.

   Note

   You can select any Source Datasource that you have access to.

3. Select the LOAD Working Database in the Target Datasource you are moving objects TO list box.

   Note

   You can select any Target Datasource that you have access to.

4. Either:

   - Click the Select All Visible button in Step 2: Select Objects section to select all of the objects in the DEV Working Database to copy to the LOAD Working Database.

   Or

   - In the Select objects in source datasource to compare panel, toggle Selected to on for individual objects to copy to the LOAD Working database.

   Note

   You can use the Search feature to filter on objects in this panel. For example, to select a dataset of vendor master-related objects, click the Search icon, then enter VM in the Identifier search field. Only those objects that start with this prefix display, and you can click the Select All Visible button to select this dataset for comparison.

5. In the Step 3. Run Comparison section, click the Create Comparison button.

   Note

   Migrate collects the DDL of all of the tables and views and compares them to the target system. When the comparison is complete, the icon in the Comparison column indicates the status:

   • A green icon indicates that the data in the Working databases match.

   • A red icon indicates that an object is missing in one of the Working databases.

   • A yellow icon indicates that the object exists in both databases but is not identical (this could happen if, for example, a view has changed).
     

6. In the Objects selected for compare panel, toggle Promote to on for objects to move from the DEV Working Database to the LOAD Working Database.

   Note

   When moving tables, be aware if the table contains data or not, especially if it is different between the databases. In this case, syncing these objects could result in lost data.

   Note

   If an object has dependencies, those dependencies are also selected to move automatically. You must toggle Selected to off for any dependencies that must not move.

7. After the objects are selected, click the Start Promotion button.

   Migrate creates objects in the LOAD Working database. Once complete, Migrate reruns the comparison, and displays what was moved and any errors.

   Note

   If an object fails to copy, Migrate uses the previous version of the object. It does not drop and create and object.

Assign an Environment to a Milestone

Milestones are generally scheduled test cycles that:

• Validate the ETL process and migration data

• Capture the mock load cycle dates for the conversion of data

• Are used to test the load of legacy data

• Often provide the data used for formal testing of the system by the Testing teams

As part of the steps performed in Execute the Rules in an Environment, you must select a milestone to run the reports against. The rules run in the milestone’s assigned environment.

To assign an environment to a milestone:

1. Select Migrate > Project Setup in the Migrate menu.

2. In the Milestones panel, double-click the milestone Name.

3. Select the environment from the Environment list box.

4. Click the Save icon.

Reports that display for this milestone are generated in the environment selected in Execute the Rules in an Environment.

Execute the Rules in an Environment

During this step, you copy the jobs that execute the rules that generate the ETL package from one environment to another where they can then be executed.

Reports are registered against a milestone and are executed against the Datasource depending on the milestone’s environment.

You can view and execute the same set of target ETL reports from the Mappings page (Migrate > Mappings) for each environment.

Migrate builds out the ETL rules, which are database views. After writing the rules in the DEV environment, you can copy them to the LOAD environment. In an environment, rules can be deleted or the order they run can be changed without affecting the rules in other environments. Changes can also be copied to another environment.

Note

This process allows you to make a copy of the ETL rules from the DEV environment to another milestone and environment, and to overwrite the rules. However, you cannot copy from another environment back into the DEV environment. Any changes that must be made to the DEV environment to sync it with another environment must be made manually in the DEV environment.

To execute the rules in an environment:

1. Select Migrate > Mappings.

2. In the Datasets panel, click the ETL icon for the relevant Subject Area.
   

3. Click the Environment name tab (for example, DEV) in the upper left, next to the Syniti logo; the ETL Version Selection modal displays.

4. In the Value list box, select the milestone to copy the ETL rules from the selected environment to.

5. Either:

   - You receive a warning message that the package has already been created in this environment, and any custom changes will be lost if you proceed, and you click Cancel.

   Or

   - You click the Copy [Environment name] to [Milestone Name] button.

6. After the ETL package is copied, you can run the reports by clicking the play icon.
   

   You can run one job or all jobs, depending on the play icon.

View the Reports in an Environment

As a Developer, access relevant reports on the Migration Reports page (Validate > Migration Reports).

Select the environment to view the reports in by clicking the environment name tab (for example, DEV) in the upper left, next to the Syniti logo.

Note

The tab text color reflects the environment color, and the color used for report data when it is generated from this environment.

As a business user, view the reports on the Deployment Reports page (Validate > Deployment Reports). Just as when viewing reports on the Migration Reports page, you can switch the environment to view the reports in that environment.