User Tools

Site Tools


file_replicator

Perspectium File Replicator for Replicator Agent - Subscriber Configuration Guide

The Perspectium File Replicator for Replicator Agent is a feature that allows for replicating table records from a Service Now instance to a local file. Records can be saved in .CSV, and .JSON formats. As of the v3.2.4 release, .XML* formats are also supported. For .XML files, each record will be an XML record with the table name as the root element.

To enable File Replicator for Replicator Agent, you will need to make the following configuration changes to the agent.xml file that was created upon installation of the Replicator Agent:

Directive Example Use Required
handler <handler>com.perspectium.replicator.file.XMLFileSubscriber</handler>

Available file handlers are:
com.perspectium.replicator.file.CSVFileSubscriber
com.perspectium.replicator.file.JSONFileSubscriber
com.perspectium.replicator.file.XMLFileSubscriber
com.perspectium.replicator.file.HDFSFileSubscriber*
The name of the file handler class Required
file_name <file_name>records.csv</file_name> The name of the file to save records into Required
files_directory <files_directory>/Downloads/subscribefiles</files_directory> The directory where the file will be saved to Required
buffered_writes <buffered_writes>250</buffered_writes> The number of records to buffer before writing to file (to improve performance and not write to the file on reading each record) Optional
translate_newline <translate_newline>%13</translate_newline> A value to enter into the file in place of the newline (\n) character. For example, using %13 (symbol for carriage return) with replace \n with %13 to use carriage return instead Optional
exclude_xml_header <exclude_xml_header/> For use with the XMLFileSubscriber handler, this will only output the xml header tag (i.e. <?xml version=“1.0” encoding=“UTF-8”?>) once at the top of the file. That way you can treat the entire file as one XML file with multiple elements for parsing.

For example, with this configuration, the file will be:
<?xml version=“1.0” encoding=“UTF-8”?>
<incident></incident>
<incident></incident>
<cmdb_ci></cmdb_ci>

versus
<?xml version=“1.0” encoding=“UTF-8”?><incident></incident>
<?xml version=“1.0” encoding=“UTF-8”?><incident></incident>
<?xml version=“1.0” encoding=“UTF-8”?><cmdb_ci></cmdb_ci>
Optional

*The HDFSFileSubcriber handler requires the use of the Hadoop external library. You will need the jar files from each of these Hadoop librarires:

  1. Hadoop Annotations
  2. Hadoop Auth
  3. Hadoop Cient
  4. Hadoop Common
  5. Hadoop HDFS
  6. Hadoop Mapreduce
  7. Hadoop Yarn

The Maven repository for these files can be found here. Once you have these files they need to be placed in the extlib folder in the File Replicator agent's directory.

Sample agent.xml Configuration File

This example configures agent.xml so that the Replicator Agent will subscribe to save records as XML into one file named records.xml in the specified directory:

<?xml version="1.0" encoding="ISO-8859-1" ?>
<config>
    <agent>
        <max_reads_per_connect>10</max_reads_per_connect>
        <polling_interval>20</polling_interval>
	<subscribe>
            <task>
            	<task_name>file_subscribe</task_name>
	        <message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection>
    	        <instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection>            
            	<decryption_key>The cow jumped over the moon</decryption_key>
            	<handler>com.perspectium.replicator.file.XMLFileSubscriber</handler>
            
                <file_name>records.xml</file_name>
                <files_directory>/Users/user/Downloads</files_directory>            	            	
            	<exclude_xml_header/>
            	<buffered_writes>250</buffered_writes>
            
            	<translate_newline>%13</translate_newline>            	
            </task>
    	</subscribe>  
    </agent>
</config>

One Record Per File

v3.19.0

The File Replicator for Replicator Agent feature can also save each record into its own file versus saving all records into one file as mentioned above.

Configuration directives for saving one record per file are as follows:

Directive Example Use Required
handler <handler>com.perspectium.replicator.file.XMLFileSubscriber</handler>

Available file handlers are:
com.perspectium.replicator.file.CSVFileSubscriber
com.perspectium.replicator.file.JSONFileSubscriber
com.perspectium.replicator.file.XMLFileSubscriber
The name of the file handler class Required
one_record_per_file <one_record_per_file/> This directive will tell the agent to save each record into its own file instead of saving all records together in one fileRequired
files_directory <files_directory>/Downloads/subscribefiles</files_directory> The directory where the files will be saved to Required
file_prefix <file_prefix>record</file_prefix> A prefix for the file name of each record. If this directive is not specified, “psp.replicator.” will be used as the prefix. Optional
file_suffix <file_suffix>.xml</file_suffix> A suffix for the file name of each record. If this directive is not specified, “.xml” will be used as the suffix. Optional
translate_newline <translate_newline>%13</translate_newline> A value to enter into the file in place of the newline (\n) character. For example, using %13 (symbol for carriage return) with replace \n with %13 to use carriage return instead Optional

Each subscribed record will be saved into its own file in the specified directory. Files will be named with the supplied prefix, a randomly generated middle part and the supplied suffix with default values used if either the prefix or suffix are not specified (the middle part cannot be configured and is randomly generated to ensure unique file names).

For example, with the example directives <file_prefix>record</file_prefix> and <file_suffix>.xml</file_suffix>, file names would be created as follows:
records0aa323d6-fd04-4bf8-a4d4-e415832b550c.xml
records0b4f7880-0d8f-46a2-8d7f-0eac1aec2ec8.xml

Sample agent.xml Configuration File

This example configures agent.xml so that the Replicator Agent will subscribe and save each record into its own file in the specified directory:

<?xml version="1.0" encoding="ISO-8859-1" ?>
<config>
    <agent>
        <max_reads_per_connect>10</max_reads_per_connect>
        <polling_interval>20</polling_interval>
	<subscribe>
            <task>
            	<task_name>file_subscribe</task_name>
	        <message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection>
    	        <instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection>            
            	<decryption_key>The cow jumped over the moon</decryption_key>
            	<handler>com.perspectium.replicator.file.XMLFileSubscriber</handler>
                
                <one_record_per_file/>                
                <files_directory>/tmp</files_directory>            	            	
            	<file_prefix>records</file_prefix>
                <file_suffix>.xml</file_suffix>
                
            	<translate_newline>%13</translate_newline>            	
            </task>
    	</subscribe>  
    </agent>
</config>
file_replicator.txt · Last modified: 2018/11/07 12:20 by timothy.pike