User Tools

Site Tools


replicator_agent_new

Agent Configuration

The Replicator Agent’s configuration file is agent.xml and contains the definitions required to provide the type of processing desired by the customer.

The configuration can contain multiple sections. The <agent> section is required and is for defining the behavior of the agent itself. The <share> and <subscribe> directives are optional and allow you to define various tasks that will be scheduled by the agent for execution.

Every replicator configuration file starts with the following directive:

<?xml version=“1.0” encoding=“ISO-8859-1” ?>

The agent.xml file wraps all of its directives within the <config> and then <agent> elements as outlined below.

    <config>
        <agent>
            <!-- Additional configuration directives goes here -->
        </agent>
    </config>

Encrypting fields inside the config file

Encrypting sensitive field values is supported in the agent.xml file. By default the installer will tag any password field for encryption by prefixing the value with “encrypted:”. When the agent starts, any configuration values prefixed with “encrypt:” will be encrypted and the configuration file will be rewritten with the prefixes changed to “encrypted:”.

Tasks

The agent.xml file supports two types of tasks, Subscriber tasks and Sharer tasks. These task types use the following configuration directives respectfully.

Type Example
Subscriber <subscribe>
Sharer <share>

Subscriber

A Subscriber, also known as a Consumer, is configured to monitor a message bus queue for messages. The subscriber handler will use its task configuration and logic to perform the required processing.

Sharer

A Sharer, also known as a Producer, performs processes to place a message into a defined queue within the message bus.

Directives

The current directives that are common to both types of task are outlined in the following table. When stated that the element can be overridden by a task, that directive is global for all task entries. However, any individual task can alter the value to meet the specific needs of that task.

Directive Example Use Default
task_name <task_name>mysql_replication</task_name> Unique name among tasks Required
handler <handler>Java Class Handler</handler> Java Replicator Class Required

In most cases the following directive is also required.

Required Directive Example Use Default
message_connection <message_connection user="USER_NAME" password="USER_PASSWORD" queue="OPTIONAL_QUEUE_NAME">URI</message_connection> Message Bus connection Required

The task name can only contain the following characters; The numbers 0-9 and upper or lower case alphabet characters and the _ underscore.

Handlers

A Handler is the actual software or Java class that performs the required processing. A Handler is defined using the <handler> directive.

Each Handler will have access to its <task> section of the overall configuration. Each <task> element will likely have unique configuration directives documented along with the handler.

The following tables reflects required configuration directives by task type:

Subscribe Directive Example Use Default
decryption_key <decryption_key>shared secret key</decryption_key> Secret key shared with the instance Required
max_reads_per_connect <max_reads_per_connect>1500</max_reads_per_connect> Messages to consume per interval Optional
Share Directive Example Use Default
encryption_key <encryption_key>shared secret key</encryption_key> Secret key shared with the instance Required
max_writes_per_connect <max_writes_per_connect>1500</max_writes_per_connect> Messages to produce per interval Optional

The following table reflects the available optional configuration directives:

Directive Example Use Default
reporting_connection <reporting_connection user="USER_NAME" password="USER_PASSWORD" queue="OPTIONAL_QUEUE_NAME">amqp://acme.perspectium.net</reporting_connection> Message Bus connecton Optional
polling_interval <polling_interval>20</polling_interval> Interval between each run Optional
schedule <schedule>* * * * *</schedule> Process messages every minutes Optional

Configuring the Transport Protocol

The agent provides the ability to use the HTTP(S) protocol to communicate with the message bus as well as the Advanced Message Queuing Protocal (amqp). The transport protocols http and https can be used with the following directives: message_connection, reporting_connection, and management_connection.

The configuration is simple. You change the 'schema' used from amqp to http or https within the URI as reflected in the example below.

Directive Example Use Default
message_connection <message_connection user="USER_NAME" password="USER_PASSWORD" queue="OPTIONAL_QUEUE_NAME">https://acme.perspectium.net</reporting_connection> Message Bus connection Optional
reporting_connection <reporting_connection user="USER_NAME" password="USER_PASSWORD" queue="OPTIONAL_QUEUE_NAME">https://perspectium_reporting.perspectium.net</reporting_connection> Message Bus connection Optional

Enabling use of an HTTP Proxy

Logging

To set the logging level at the command line you can enter in the appropriate command for you agent version. At the command line you can set the logging level for a certain duration or indefinitely. You can also manually set the logging level, see here. Note, setting it manually will set it indefinitely.

It can be helpful to have the logging set to FINEST/DEBUG during installation but it is recommended to have logging set to INFO during production.

Dynamically Setting the Logging Version

By default, the logging level of the Replicator agent is set at INFO. This setting incurs minimal logging and is a good candidate for every day use.

The logging level can be dynamically changed using the setLogging command. This command is useful during troubleshooting to obtain greater detail of the processing taking place. You may be asked to increase the logging level to finest by using the setLogging command.

The setLogging command has the following syntax:

setLogging LOGGING_LEVEL DURATION

The LOGGING_LEVEL argument is required and must be either “finest” or “info”. As of 3.23.0, users may now select “debug” as an argument.

The DURATION argument is used to specify how many seconds the change in logging level is to remain in affect. The default is one hour and by setting it to -1 it will remain indefinitely. Typically the logging level is changed to finest only during troubleshooting, this approach ensures that info level debugging is resumed after an hour.

For example you can call the following commands from the root Perspectium directory:

# Setting to finest logging for an hour
bin/setLogging finest
 
# Setting to finest logging for 3 hours (10800 seconds) 
bin/setLogging finest 10800
 
# Setting to info logging indefinitely
bin/setLogging info -1 

Note: this is will stay in affect until the Agent is restarted or the duration expires.

Manually Setting the Logging Level

You can manually set the logging level by editing a single line of the config. Log into the host that the agent is installed on and navigate to the home directory of the Agent and edit the log4j2.xml file within the conf folder.

You will change the property <Root level=“DEBUG”> to: INFO, DEBUG, or FINEST.

Finest provides the most detail and info provides the least. Save the file and restart the agent to implement the change. This will stay in affect through Agent restarts until it is modified back. You can still modify the logging level through the previous bin/setLogging command above.

Additional Configurations

There are additional changes you can make to alter the logs to your liking or to add when troubleshooting. The file logging.properties within the conf folder contains information and examples for such changes.

Debugging Database IO MSSQL

For example to enable debugging logs for I/O through the JDBC of MSSQL you can append the following to the end of logging.properties.

com.microsoft.sqlserver.jdbc.level = DEBUG

Agent Configuration Example

The following walks you through an agent.xml file which has been defined to support subscribing to a queue which contains trouble tickets replicated from a ServiceNow instance.

    <?xml version="1.0" encoding="ISO-8859-1" standalone="no"?>
    <config>
    <agent>

        <reporting_connection password="encrypted:iTOCp4deP7rJNgFkf2AEMA==" user="admin">https://reporting.perspectium.net</reporting_connection>
        <management_connection password="encrypted:iTOCp4deP7rJNgFkf2AEMA==n" user="admin">https://management.perspectium.net</management_connection>
        <message_connection password="encrypted:iTOCp4deP7rJNgFkf2AEMA==" user="admin">https://acme.perspectium.net</message_connection>
        <heartbeat_connection password="encrypted:iTOCp4deP7rJNgFkf2AEMA==" user="admin">https://acme.perspectium.net</heartbeat_connection>

        <max_reads_per_connect>2000</max_reads_per_connect>
       	<polling_interval>15</polling_interval>

        <subscribe>
            <task>
                <task_name>ticket_table_replication</task_name>
                <polling_interval>30</polling_interval>

                <handler>com.perspectium.replicator.sql.SQLSubscriber</handler>
                <decryption_key>The cow jumped over the moon</decryption_key>

                <message_connection queue="psp.out.replicator.ticket_table" password="encrypted:iTOCp4deP7rJNgFkf2AEMA==" user="admin">amqp://acme.perspectium.net</message_connection>

                <instance_connection user="agent" password="agent123">https://dev17481.service-now.com</instance_connection>

                <database_type>mysql</database_type>
                <database_server>localhost</database_server>
                <database_port>3306</database_port>
                <database_user>root</database_user>
                <database_password>encrypt:mypassword</database_password>
                <database_params/>
                <database>mydatabase</database>
                <skip_database_creation/>
            </task>
      </subscribe>

   </agent>
  </config>

Every replicator configuration file starts with the following directive:

<?xml version=“1.0” encoding=“ISO-8859-1” ?>

The agent.xml file wraps all of its directives within the <config> and then <agent> elements.

Let's spend a moment and focus on the first few directives following the start of our agent within the <agent> directive.

The <reporting_connection> and <management_connection> directives are optional. These directives define the destination URI and credentials required to report performance, usage and availability of the agent to Perspectium.

Let's take a closer look at the <reporting_connection> directive. In this example the directive contains the attributes user and password. These attributes are used to connect to the Perspecitum Messsage Bus. As you can see the password attribute has been encrypted. I'll explain how to encrypt sections of the agent.xml conifguration in a later section. Within the value field of the <message_connection> element is the URI. In our example it's: amqp:acme.perspectium.net. This URI consists of the scheme, which in our case is amqp, followed by the literal ':' and then the host, which is acme.perspectium.net.

We're using the ampq scheme but Perspectium also supports specifying the amqps, http or https schemes. Using the amqps scheme is similar to using https instead of http. The amqps scheme will provide data communication over a secure channel The later two schemes, http and https are useful when you have constraints that prevent you from connecting to entities on the internet using protocols other than these typical standards. Here's an example of using the https scheme.

<reporting_connection password="encrypted:iTOCp4deP7rJNgFkf2AEMA==" user="admin">https://reporting.perspectium.net</reporting_connection>

The scheme can be specified in the management_connection and message_connection directives as well.

Supported Schemes Default Port Example
amqp 5672 <message_connection attributes>amqp://acme.perspectium.net</message_connection>
amqps 5671 <reporting_connection attributes>amqps://acme.perspectium.net</reporting_connection>
http 80 <reporting_connection attributes>http://acme.perspectium.net</reporting_connection>
https 443 <management_connection attributes>https://acme.perspectium.net</management_connection>

Following our <reporting_connection> is a <management_connection> directive. This directive serves the same purpose as the reporting_connection but since it reports on different types of information it has its own connection.

The next directive that's global to the agent is the <max_reads_per_connect> directive. In our example we've specified 2000.

      <max_reads_per_connect>2000</max_reads_per_connect>

This directive is optional and is only applicable to <task> entries contained within a <subscribe> directive but I'm getting a bit ahead of myself. If you did not include this directive then your task would peform the default of 4000 reads during its execution. I'll be talking more about this later.

Next up, and also global across the agent configuration is the <polling_interval> directive. We've specified a value of 15.

     	<polling_interval>15</polling_interval>

This directive tells the agent that we want our <handler> which we'll be discussing in a moment to run every 15 seconds. This directive is also optional and defaults to 30 seconds.

Okay, we're finally ready to configure the agent with something specific to do. In our case we're going to define a subscriber using the <subscribe> directive. We use this directive to enclose the other required directives. Here's a preview of the core directives:

<subscribe>

<task>
	<task_name> </task_name>
	<handler> </handler>
      <decryption_key></decryption_key>
</task>

</subscribe>

Directive Example Use Default
task_name <task_name>mysql_replication</task_name> Unique name among tasks Required
handler <handler>Java Class Handler</handler> Java Replicator Class Required
decryption_key <decryption_key>pre-shared key</decryption_key> Decrypts encrypted messsages Required

The task name can only contain the following characters; The numbers 0-9, upper or lower case alphabet characters and the _ underscore.

In our example we're only configuring a single task but multiple tasks can be configured.

Okay, here's our <task_name> directive, which meets the naming convention.

      <task_name>ticket_table_replication</task_name>

It's also important to note that in a configuration which contains multilpe <task> entries, the task_name must be unique.

Wait a minute, what's this, another <polling_interval> directive?

      <polling_interval>30</polling_interval>

That's right, we've defined another <polling_interval>. Note however, that the value of this one is 30 and not 15. Furthermore, since this one is enclosed within a <task> it will override the agent version, but only for this task. Remember, although we haven't, you can define multiple <task> entries within both the <subscribe> and <share> enclosing elements!

It's time to add the directive which defines the Java class that's going to handle messages from the queue associated with this task.

      <handler>com.perspectium.replicator.sql.SQLSubscriber</handler>

The <handler> directive places the fully qualified name of the Java Class within the handler directive. In our case, the Class the SQLSubscriber, which resides within the Java package, com.perspectium.replicator.sql. Too much information? Fair enough, let's move on.

Messages that represent replicated table rows from ServiceNow are encrypted using a configured key. The <decryption_key> directive must be configured with the same key as that configured in the ServiceNow instance being used. Here's ours:

  <decryption_key>The cow jumped over the bush</decryption_key>

The replicator agent will use the decryption key to decrypt the message before it hands it over to the SQLSubsciber handler.

The SQLSubscriber consumes a message from the queue that it's monitoring. Each message includes the name of the table from which the row of replicated data is coming from. The SQLSubscriber needs to know more about the table than what's available in each message. Fortunately SQLSubscriber can issue a request to the instance asking for the required information. In order to do this we need to provide the name of the instance, and a user name and password that's been configured properly to ask the instance for information about the table. This configuration is defined using the <instance_connection> directive. Here's ours:

      <instance_connection user="agent" password="agent123">https://dev17481.service-now.com</instance_connection>

As you can see, we've provide the user name, agent in our case and the password agent123. The agent user needs to be preconfigured in the instance as have the Perspectiun role associated with it. The value of the directive is the URI and host name of the instance which replicated the table row, which in our case is dev17481.service-now.com.

Our handler, which provides consuming messages from a queue which are intended to be processed and placed into a relational database of some kind. In order to perform this task, the SQLSubscriber class must be configured with several pieces of information. Furthermore, the SQLSubscriber class defines the names of the configuration directives that it expects. Here's what they are.

Directive Example Use Default
database_type <database_type>mysql|sqlserver|oracle</database_type> The type of database replicated to Required
database <database>psp_repl</database> Name of the target database Required
database_server <database_server>IP or FQDN of the database server</database_server> IP or FQDN of the database server Required
database_user <database_user>database user name</database_user> User for logging into the database Required
database_password <database_password>database user password</database_password> Password for logging into the database Required
database_port <database_port>port used by the server</database_port> Port used by the server Optional
database_parms <database_parms>parms passed to the database server</database_parms> Parameters passed to the database server Optional
key <key>ven0001</key> to specify a “key” or instance to subscribe to and ignore records from all other sources e.g. only records coming from the ServiceNow instance ven0001 will be subscribed to and all other messages will be skipped. Introduced in v3.8.0 Optional

Here a bit more about what these directives are for.

The <database_type> directive allows you to specify the database type.

The <database_server> directive is used to provide either a fully qualified domain name of the database server or its IP address. The <database_port> directive is used to override the standard port used by the servers. The <database_user> directive is used to specify the user name that will be used to login to the database. The user configured must have permissions which allow it to create and modify tables. The <database_password> directive is used to specify the password associated with the configured user. The <database_parms> directive can be used to provide additional information to the database server during establishment of the connection.

In our configuration we're using a database server that's located at the IP address 192.168.1.2 and is connected the the TCP port 3306. Our database environment has the user 'db_user' configured with proper credentials and the ability to perform all tasks. As you can see we also need to provide the password for the 'db_user' in order to connect to the database server. We don't need to provide any database parameters and so we leave it blank. Finally, we identify the name of the database 'mydatabase' which already has been created and so we include the <skip_database_creation> directive. If you were to leave this out SQLSubscriber will attempt to create the database and which will fail since the database already exists.

      <database_server>192.168.1.2</database_server>
      <database_port>3306</database_port>
      <database_user>db_user</database_user>
      <database_password>encrypt:db_password</database_password>
      <database_params/>
      <database>mydatabase</database>
      <skip_database_creation/>

Wow, are we almost done? As a matter of fact we are. All that's left is to close out our task using the </task> element, the agent </agent> and finally the complete config </config>.

We provided the handler SQLSubscriber with the fundamental information required to perform it's task. Yes, of course there are several options available which we'll go into in another section, for now we're ready to launch the replicator agent and replicate some data.

replicator_agent_new.txt · Last modified: 2018/07/06 16:45 by paul