User Tools

Site Tools


configuration_errors

Troubleshooting for Replicator Agent Configuration Errors

Configuration Errors

Multibyte/Foreign Language Decryption Error

Problem

You are experiencing issues with multibyte characters and foreign language encryption/decryption.

Error

Content will be undecipherable

Solution

Here are the likely fixes.

  • If you are expecting multibyte characters from ServiceNow it is recommended to turn on multibyte encryption within the Perspectium Properties page.
  • If you are running a MySQL agent it is recommended to place characterEncoding=UTF-8 within the database_parms tag.
  • If you are running a SQL Server agent on Windows then you must be using at least Agent V3.11.0 and include SendStringParametersAsUnicode=true within the Database Parms tag.

The format is:

<!-- MySQL multibyte decryption -->
<database_parms>characterEncoding=UTF-8</database_parms>
<!-- SQL Server multibyte decryption (Windows) -->
<!-- Note: Requires Agent V3.11.0 or greater -->
<database_parms>SendStringParametersAsUnicode=true</database_parms>

Timeout Errors on Large Transactions

Problem

If you have successfully shared and subscribed data to our queues for small tests but then fail on large tests it may be due to your request timing out. This may happen when you are performing a large replication and the server is currently already under heavy load.

Errors

You may see the following errors in the log files:

ERROR SubscriberTask - Subscriber Error: Send Error: Read timed out
ERROR SubscriberTask - Subscriber Error: Send Error: Unexpected response status: 500
...
ERROR SubscriberTask - Subscriber Error: Send Error: Read timed out
com.perspectium.api.MessageBusException: Send Error: Unexpected response status: 500
	at com.perspectium.api.HttpClient.commonSend(HttpClient.java:544)
	at com.perspectium.api.HttpClient.get(HttpClient.java:456)
	at com.perspectium.api.HttpClient.send(HttpClient.java:446)
	at com.perspectium.api.HTTP.query(HTTP.java:152)
	at com.perspectium.api.MessageBus.query(MessageBus.java:224)
	at com.perspectium.api.MessageBus.recv(MessageBus.java:244)
	at com.perspectium.replicator.timer.SubscriberTask.run(SubscriberTask.java:143)
	at com.perspectium.replicator.scheduler.TimerScheduleTask.run(TimerScheduleTask.java:16)
	at java.util.TimerThread.mainLoop(Unknown Source)
	at java.util.TimerThread.run(Unknown Source)
ERROR SubscriberTask - Subscriber Error: Send Error: Unexpected response status: 500
com.perspectium.api.MessageBusException: Send Error: Unexpected response status: 500
	at com.perspectium.api.HttpClient.commonSend(HttpClient.java:544)
	at com.perspectium.api.HttpClient.get(HttpClient.java:456)
	at com.perspectium.api.HttpClient.send(HttpClient.java:446)
	at com.perspectium.api.HTTP.query(HTTP.java:152)
	at com.perspectium.api.MessageBus.query(MessageBus.java:224)
	at com.perspectium.api.MessageBus.recv(MessageBus.java:244)
	at com.perspectium.replicator.timer.SubscriberTask.run(SubscriberTask.java:143)
	at com.perspectium.replicator.scheduler.TimerScheduleTask.run(TimerScheduleTask.java:16)2016-07-28 17:22:16.705 ERROR SubscriberTask - Subscriber Error: Send Error: Unexpected response status: 500
	at java.util.TimerThread.mainLoop(Unknown Source)
	at java.util.TimerThread.run(Unknown Source)

Solution

To mitigate this you can alter the default connection_request_timeout by setting it to 120000. This should give your connection plenty more room to handle all the IO of large transaction. You would place it within your agent.xml like so:

<config>
    <agent>
        <subscribe>
            <task>
                <task_name>timeout_example</task_name>
                <message_connection connection_request_timeout="120000" user="XXX" password="XXX" >your_url</message_connection>
                ...
            <task>
        <subscribe>
    </agent>
</config>

This should be placed on the <message_connection> within the task level of the desired connection. This attribute will only be set for the specified <message_connection>, so if you have separate connections for monitoring or replicating data they will use the default unless specified.

Another option is if you have firewall access to both your https and amqps connections (https://your_instance.perspectium.net & amqps://your_instance-amqp.perspectium.net) you can try either

  • Setting your <max_reads_per_connect> to 1 and use the https connection
  • Setting your <max_reads_per_connect> to 4000 and use the amqps connection

Database Connection Timeout

Problem

The agent has a slow network connection to the database where the agent may get connection time out issues.

Error

You will receive a message that says “connection timed out”

Solution

You can add a loginTimeout database parameter to the agent.xml configuration file to control the DB connection timeout.

In your agent.xml, under each <task> entry, add <database_parms>loginTimeout=NN</database_parms> where nn is in seconds.

For example:

 <database_parms>loginTimeout=30</database_parms>

if you already have database_parms configured then append the loginTimeout parameter using;

 <database_parms>integratedSecurity=true;loginTimeout=30</database_parms>

java.lang.OutOfMemoryError: Java heap space

Problem

The agent is running out of memory.

Error

If you get the following error in the Argon and newer version of the agent:

2017-11-27 09:47:15.655 ERROR - Reporting.psp.in.servicenow - Replicator - (stderr) java.lang.OutOfMemoryError: Java heap space

This means the agent is running of memory the application needs as it runs.

Solution

To resolve this, open up the wrapper.conf file located in the agent's conf folder and change the following configuration:

#wrapper.java.maxmemory=64

Removing the “#” and putting a numeric value higher than 64. This numeric value is a size in MB for the Java memory heap space the agent can use. Generally you would base this value on the memory available on the server where the agent is running. For example, if the server has 1GB of memory, you can set it to be 512MB here:

wrapper.java.maxmemory=512

Configuration Troubleshooting

Start the Agent without the wrapper

  1. Change the current directory to the agent base (e.g. cd /usr/local/Perspectium_Replicator_Agent)
  2. java -Dlog4j.configurationFile=file:conf/log4j2.xml -classpath “.:${CLASSPATH}:bin:jars/*:lib/*” com.perspectium.replicator.Replicator

Enabling Debugging Logs

The initial logging level is generally at its lowest, INFO, but if you want to enable finer logging than you can consult the Replicator Agent Logging page. There are varying levels of logging and setting them may differ between versions. Also note, that at the finest levels of logging there may be some performance cost so it is recommended to run every day use at the INFO level of logging.

Setting the Target Perspectium Server

When you are provided your credentials you may have been given two different server endpoints. Such as:

  • https://user.perspectium.net
  • amqp://user-amqp.perspectium.net

For ServiceNow use the https endpoint, never the AMQP one. You will place the endpoint within the Properties page under the Target Perspectium Server portion as well as in the Endpoint URL for any queues you are sharing or subscribing to.

For the Replicator Agent you can use either protocol. You will set your endpoint within the <message_connection> tag, and, if necessary within the <management_connection>, <reporting_connection>, <heartbeat_connection>. For more information on these other connections see here.

In each case you will also need to specify your username and password credentials you were provided for the Perspectium Message Bus.

SQL Share By Last Update

There is a feature released in Agent 3.10.0 that is similar to Dynamic Share but for the replicator agent. You specify a 'datetime' or 'timestamp' column to query and your current timezone. Then at every polling interval the agent will query the table and search for recently updated records as indexed by that specified column.

For the documentation of this feature see here.

If you use a column whose type is 'timestamp' you have to take some precaution. The 'datetime' field is stored and retrieved as a constant value, a 'timestamp' is stored in UTC and displayed on the timezone of the querying session. So if you have the database set to automatically update your time field make sure that the timezone you are using is reflected in the agent.xml.

You can set it up like:
(NOTE: this example assumes your database session is in central time and you are updating your time fields according to this timezone)

<config>
   <agent>
      <!-- Configured To Share on TimeStamp -> UTC timezone -->
      <share>
         <task>
            <task_name>Share_TimeStamp_UTC</task_name>
            <handler>com.perspectium.replicator.sql.sharer.SQLShareByLastUpdateQuery</handler>
            ...            
            <datetime_field>ts_last_modified</datetime_field>
            <last_modified_timezone>UTC</last_modified_timezone>
         </task>
      </share>
 
      <!-- Configured To Share on DateTime -> Local Timezone (Central Time) -->
      <share>
          <task>
            <task_name>Share_Datetime_CST</task_name>
            <handler>com.perspectium.replicator.sql.sharer.SQLShareByLastUpdateQuery</handler>
            ...
            <datetime_field>dt_last_modified</datetime_field>
            <last_modified_timezone>CST</last_modified_timezone>
         </task>
      </share>          
   </agent>
</config>

In addition ServiceNow stores their 'datetime' objects in the UTC timezone so if you are planning around this adjust accordingly. Setting this up may take some checking and validation with your database. Temporarily changing the logging level to “DEBUG” will be helpful to see the actual “where last_modified_time > last_queried_time” query at what time.

This feature uses the Java TimeZone util. Here is an Excel Sheet for the valid timezone names, their corresponding UTC offsets, and whether they act according to daylight savings or not.

configuration_errors.txt · Last modified: 2020/08/05 15:33 by georvi.oloan