Microsoft Azure Event Hubs

Overview

Syniti Replicate allows you to replicate data from relational database tables to Microsoft Azure Event Hubs, a big data streaming platform and event ingestion service, using the following replication modes:

• Refresh or Snapshot Replication: A one-time complete replication from any major relational database source to Azure Event Hubs as a target, according to replication settings and scripts.

• Mirroring or Change Data Capture Replication: A continuous update of a replicated table to Azure Event Hubs based on changes in the source database that have been recorded in the database server log. Any changes found in the log are applied to Azure Event Hubs as a target, according to replication settings and scripts.

In order to replicate data in Syniti Replicate to an Azure Event Hub, the data is organized as virtual tables (with columns and data types). You can create a target table and define a replication to an Event Hub (for example, target table), applying mappings and expression logic as with other relational data sources.

The Azure Event is built using a JSON, CSV or XML serialization of the entire record, including metadata information such as the actual table name and the operation (Insert, Update, or Delete), transactional information like transaction ID and timestamp, and finally the entire image (before and after) of the record. Every replication must be associated with an Event Hub so that Syniti Replicate can determine where to publish the event.

You can schedule the replication, identify the columns to be replicated, and add scripts to transform data during replication. Source databases include Oracle, Microsoft SQL Server, IBM Db2 for i, IBM Db2 LUW, Sybase, Informix, MySQL, and so on.

Use Syniti Replicate Management Center to:

• Create source connections to RDBMS tables

• Create Azure Event Hubs targets

• Map RDBMS sources to Azure Event Hubs targets

• Enable replication

Subsequent data management on the Azure Event Hubs side depends upon your application needs.

Prerequisites

1. Download the Azure.Messaging.EventHubs version 5.11.0 provider from the Nuget site.

2. Create a local directory to deploy the files required, for example, C:\tmp\EventHubs, and then save the nuget.exe file.

3. From the DOS prompt with admin permissions, execute the below command. Make sure to execute the instruction from the right directory.

   C:\tmp\EventHubs> nuget install Azure.Messaging.EventHubs -Version 5.11.0 -Framework net480 -OutputDirectory C:\tmp\EventHubs

   The command will download the entire provider into the specified output directory.

4. Open the output directory to view the folders where the assemblies are located.

5. Select the following assembly: \Azure.Messaging.EventHubs.5.11.0\lib\netstandard2.0\Azure.Messaging.EventHubs.dll

6. Copy this file and paste it into the following Syniti Replicate installation folder: PluginsAzure\EventHubs.

Limitations

• Azure Event Hubs is supported only as target system in Refresh and Mirroring mode.

• Access via Proxy is not supported. Syniti Replicate needs to connect directly with Azure Event Hubs.

• Data Encryption is not supported for Azure Event Hubs.

• Data Masking is not supported in Syniti Replicate.

• Key Vault is not supported in Syniti Replicate.

• Database and schema names containing special characters like slash or backslash are not supported.

Set up a Target Connection to Azure Event Hubs

1. Select the Targets node in the Metadata Explorer.

2. Right-click the menu and select Add New Connection.

3. In the Add Target Connection Wizard, select Azure Event Hubs from the Database list. The Provider and Assembly fields are automatically entered after selecting the database.

4. On the Set Connection String page, set properties as described in the following table:

   Property	Description
   Server	The Azure Event Hubs namespace or IP address, that is, the container for Event Hubs. Typical format: <yournamespace>.servicebus.windows.net
   Shared Access Policy Name	The name of your shared access policy. The default value is RootManageSharedAccessKey.
   Shared Access Key	Shared access primary key.
   Default Hub	The Event Hub name for the initial connection. If you select the Match Hub with Table Name option, each replication will connect to a hub with the matching table name.
   Output Folder	Pathname to the folder that will contain information related to the replication process.
   Port	Port number. The default value is 5671.
   Connect Timeout	Connection timeout (value is in seconds).
   Match Hub with Table Name	If set to true, each replicated table will send data to the existing hub with the same table name.
   Serialization	Select the type of serialization required for Azure Event Hubs messages. The following types of message serialization are currently supported: 0: JSON 1: CSV 2: XML
   Extended Properties	Additional properties not listed in the grid. Enter the properties as a list of “property=value” strings separated by (,).

Note

Complete the rest of the fields in the Add Target Connection wizard. Refer to Add Target Connection Wizard for more information.

Data Type Mapping

When Azure Event Hubs is used as a target system, the mapping of Syniti Replicate data types to the corresponding data type in Azure Event Hubs is listed in the following table: