User Tools

Site Tools


replicator_agent_troubleshooting2

Troubleshooting for Replicator Agent Installation Errors

Outbound Network Access

You should ensure that your firewall and or proxy allow outbound connectivity to the Perspectium MBS, Perspectium Monitoring, and your ServiceNow Instance.

Note: Running the configuration validation tool will perform connectivity tests for all elements defined within the agent.xml configuration file and will tell you if the test succeeded or failed. The following is recommended if additional trouble shooting is required.

Depending on the protocol you plan on using you should test the corresponding DNS resolved name and port number. If you plan on operating through a proxy then you should test it as well.

The connection to the destination host and port can be tested by calling telnet from the command line as necessary with the ports:

HTTPS AMQP AMQPS HTTP
443 5672 5671 80

If you're running the agent on a Windows server and need assistance installed the telnet client please reference this link: https://social.technet.microsoft.com/wiki/contents/articles/22715.how-to-enable-telnet-in-windows-server-2012.aspx

If you're running the agent on a Linux distribution that supports yum then you should be able to install the client by issuing the following command which requires sudo privileges.

sudo yum install telnet

If your system does not leverage yum then ask your system administrator for assistance.

The following reflects a successful test to the host your_instance-amqp.perspectium.net. As the fully qualified domain name your_instance-amqp.perspectium.net suggests it's terminating the AMQP protocol which per the table above is using port 5672. The telnet command specifies the destination host's fully qualified domain name followed by the target port which in this case is 5672. The message starting with 'Connected to' states that the connection has been established which is the desired outcome.

# Good Connections (the IP address will vary per region)
telnet your_instance-amqp.perspectium.net 5672
Trying 52.8.132.178 ...
Connected to your_instance-amqp.perspectium.net.
Escape character is '^]'.

The next example is also successful at establishing a connection to the host 'your_instance.perspectium.net' which accepts connections directed to port 443 or HTTPS. This connection supports communication with the Perspectium Message Bus.

telnet your_instance.perspectium.net 443
Trying 52.8.203.198 ...
Connected to your_instance.perspectium.net.
Escape character is '^]'.

The next example of a successful test is performed to ensure that the agent can connect to your ServiceNow instance which accepts HTTPS connections via port 443.

telnet your_instance.service-now.com 443
Trying 103.23.66.118 ...
Connected to your_instance.service-now.com.
Escape character is '^]'.

Now let's take a look at what failed connection attempts look like.

In the following example we attempt to connect to the host my_domaine.perspectium.net using port 443 or HTTPS. The error message could be due to a typo in the spelling of the fully qualified domain name. Perhaps the name you were assigned is my_domain but what was actually typed during the test or perhaps entered into the agent.xml configuration file was my_domaine.perspectium.net (Note the extraneous 'e' at the end of domain)

# Failed Connections (you may get a failed response or no response at all)
telnet my_domaine.perspectium.net 443
bad_domain.perspectium.net: nodename nor servname provided, or not known

Here are some additional examples or error responses that can happen when connection attempts fail.

# Failed Connections (you may get a failed response or no response at all)
telnet blocked_domain.perspectiun.net 443
telnet: connect to address XX.XX.XXX.XX: Connection refused
telnet: Unable to connect to remote host

It's also a good idea to see if you have general internet connectivity. For example, you should be able to connect to google via telnet.

telnet www.google.com 443

All my tests fail, what should I do? If all of your tests fail then it's likely that you have one or more of a few standard issues.

  • Your company has a security policy in place such that outbound connections to certain destinations such as Perspectium must be configured. Discuss your issue with your network support team.
  • Your company leverages an outbound proxy. A outbound proxy server is a server (a computer system or an application) that acts as an intermediary for requests from clients seeking resources from other servers, typically those in the public internet. Please contact your proxy support team, or possibly network support for assistance. The required documentation on how to configure use of a proxy by the agent is here: http://wiki.perspectium.com/doku.php?id=configuring_replicator_to_leverage_a_proxy

Failed Initial Connection

These are common errors with some users initial installation where the connection is refused.

Subscribe

ERROR AMQP - new connection attempt failed: Connection refused
ERROR SubscriberTask - Subscriber Error: Failed to get connection to URI: amqps:// your_instance.perspectium.net VHost: / User: your_instance
ERROR SubscriberTask - Can't report status - MessageBus is not set or not open

Share

ERROR AMQP - new connection attempt failed: Connection refused
ERROR SharerTaskMessagePublisher - openQueue Error: Failed to get connection to URI: amqps://your_instance.perspectium.net VHost: / User: your_instance

This is a result of an incorrect protocol/DNS relation. This is solved by correctly configuring the <message_connection> tag which depends on the assigned DNS given to you by the support team. Your connection will likely be either:

  • amqps://your_instance-amqp.perspectium.net
  • https://your_instance.perspectium.net

Incorrect Login Credentials

Incorrect Subscriber - Perspectium Credentials

ERROR SubscriberTask - Subscriber Error: Failed to authenticate to URI: https://your_instance.perspectium.net VHost: / User: your_instance
ERROR SubscriberTask - Can't report status - MessageBus is not set or not open!

If your subscriber task has incorrect login credentials you cannot authenticate to our servers and you may see these error messages. Double check that your credentials are correct.

Incorrect Subscriber - ServiceNow Credentials

ERROR ServiceNowSchema - Error: Send Error: Unexpected response status: 401
WARN  SchemaDocument - com.perspectium.replicator.sql.subscriber.SchemaAccessException: Send Error: Unexpected response status: 401
WARN  SchemaDocument - retrying ... 0
ERROR ServiceNowSchema - Error: Send Error: Unexpected response status: 401
WARN  SchemaDocument - com.perspectium.replicator.sql.subscriber.SchemaAccessException: Send Error: Unexpected response status: 401
WARN  SchemaDocument - retrying ... 2
WARN  SchemaDocument - Failed to fetch schema for table: incident trying cached version
ERROR SQLTableSubscriber - Unable to establish table meta data.

If your subscriber task has incorrect credentials for the instance connection to ServiceNow you may see these errors upon pulling messages from the queue. Double check that your credentials are correct.

Incorrect Share - Perspectium Credentials

ERROR SharerTaskMessagePublisher - post error: Send Error: Send Error: Unexpected response status: 401
ERROR SharerTaskMessagePublisher - closeQueue Error: Send Error: Send Error: Unexpected response status: 401

If your share task has incorrect credentials than you may see this error upon connecting to our server.

Incorrect SQL Credentials

WARN  TaskDatabase - Access denied for user 'root'@'localhost' (using password: YES)
ERROR SQLDataSource - Error: Access denied for user 'root'@'localhost' (using password: YES)

If your SQL credentials are incorrect than you may see that your access is denied and receive the following error messages.

Schema Changes and Corruption

This is something that occurs more often during installation but can occur other times. When your agent is Subscribed it will store schemas from ServiceNow within the “bin” folder. When you receive a message the agent will check this schema and if this schema is corrupted, doesn't exist, is out of date, etc. the agent will check ServiceNow and update the schema. If your ServiceNow credentials are out of date/incorrect than you may see the following errors.

ERROR ServiceNowSchema - Error: Send Error: Unexpected response status: 403
WARN  SchemaDocument - com.perspectium.replicator.sql.subscriber.SchemaAccessException: Send Error: Unexpected response status: 403
WARN  SchemaDocument - retrying ... 0
...
WARN  SchemaDocument - Failed to fetch schema for table: //table_in_message// trying cached version
ERROR SubscriberTask - …….
    com.perspectium.replicator.SubscribeException: Error: SQLSubscriber processMessage error against table: //table_in_message// error: Unable to obtain or locate the schema for table: //table_in_message//

To fix this you can remove the stored schema for this table, do this by going to the “bin/user.service-now.com.schemas_directory” folder and finding the tables' XML and delete it. Then double check that your ServiceNow credentials are correct and restart the agent. It should then re-acquire the correct schema from ServiceNow.

Subscribed To A Non-Existant Queue

WARN  AMQP - com.perspectium.api.AMQPException: Failed close channel on URI 'amqp://your_instance-amqp.perspectium.net' com.rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'psp.out.replicator.user' in vhost '/', class-id=60, method-id=70)

This is an issue of subscribing to a queue that cannot be found. This issue may arise when you are using the default queue. You can specify the queue you want in the message connection tag as an attribute. You can read more about that here.

Decryption Error

ERROR Encrypt - Error decrypting value [encrypted:vlOtU71Xu8N/EFQJJ5SS+o2+kRzuseHBK+]

This is generally a sign that at some point your encrypted field was edited post encryption causing it to fail to decrypt. This will also normally cascade causing some sort of authentication or 401 error.

Password Encryption

Encrypting sensitive password field values is supported in the config.xml and agent.xml files. The installer by default will tag any password field due for encryption with the prefix “encrypt:”. When the agent starts, any configuration values prefixed with “encrypt:” will be encrypted and the configuration file will be rewritten with the prefixes changed to “encrypted:” and the password encrypted.

If you change the password to any of your credentials than you will have to stop the agent, replace encrypted:password with encrypt:password where necessary, and restart the agent.

For example:

<share>
    <task>
        <task_name>My_Sharer</task_name>
        <handler>com.perspectium.replicator.sql.sharer.SQLSharer</handler>
        <message_connection user="user" password="encrypt:newpassword">https://user.perspectium.net</message_connection>      
 
        <database_type>mysql</database_type>
        <database_server>localhost</database_server>
        <database_port>3306</database_port>
        <database_user>root</database_user>
        <database_password>encrypt:newpassword</database_password>
 
        <database>repl_db</database>
        <table_name>incident</table_name>
        <topic>replicator</topic>
        <type>agent</type>
        <name>.bulk</name>			
    </task>
</share>

Installer UI Error

If you are trying to install the agent on Windows in a console window (such as using Putty to SSH into a different server) and get an error about the “DISPLAY variable” or “initDisplay”, this error is due to the Agent installer trying to create a GUI for installing the agent.

In this case, you will want to run the agent with the console switch:

java -jar perspectium-replicator-installer.jar -console
replicator_agent_troubleshooting2.txt · Last modified: 2018/09/04 14:56 by timothy.pike