User Tools

Site Tools


s3_subscriber

Perspectium File Replicator Agent - S3Subscriber Configuration Guide

Introduction

This document describes the configuration requirements for leveraging the Perspectium File Subscriber Replicator Agent for replicating table records from ServiceNow to a file saved into Amazon S3. Records can be saved in JSON and XML formats.

You must obtain the following jars and they need to be placed into the Agent's extlib directory, which is a sub-directory within the Replicator Agent's application folder:

The current required directives available are outlined in the following table.

Directive Example Use Required
task_name <task_name>a unique name for the task</task_name> Unique task name Required
handler <handler>com.perspectium.replicator.sql.S3Subscriber</handler> The name of the file handler class Required
access_key <access_key>AWS access key</access_key> Your AWS account access key Required
secret_access_key <secret_access_key>AWS secret access key</secret_access_key> Your AWS account secret access key Required
region <region>us-west-2</region> The region your AWS S3 bucket resides Required
S3 bucket <s3_bucket>sample-bucket</s3_bucket> The name of the S3 bucket being saved to Required
file_format <file_format>json</file_format>

Available file formats are:
json
xml
File format to be saved in Required

Sample agent.xml Configuration File

This example configures agent.xml so that the Replicator Agent will subscribe to save multiple records into a single JSON file into the specified S3 bucket.

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <agent>
      <share />
      <subscribe>
         <task>
            <task_name>s3_agent_subscribe</task_name>
            <message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection>
    	    <instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection>   
            <handler>com.perspectium.replicator.file.S3Subscriber</handler>
            <decryption_key>The cow jumped over the moon</decryption_key>
            <access_key>AccessKey</access_key>
            <secret_access_key>SecretAccessKey</secret_access_key>
            <region>us-west-2</region>
            <s3_bucket>examples3bucket</s3_bucket>
            <file_format>json</file_format>
         </task>
      </subscribe>
      <polling_interval>40</polling_interval>
   </agent>
</config>

One Record Per File

The File Replicator Agent can also save each record into its own file versus saving all records into one file as mentioned above. The above required configurations will be needed as shown in the example below.

Configuration directives for saving one record per file are as follows:

Directive Example Use Required
one_record_per_file <one_record_per_file/> This directive will tell the agent to save each record into its own file instead of saving all records together in one fileRequired
file_prefix <file_prefix>example_prefix</file_prefix> Prefix used to name individual files Optional
file_suffix <file_suffix>.json</file_suffix> Sufficx used to name individual files. JSON/XML Optional

Each subscribed record will be saved either in its own file or batched into a group and saved in the specified S3 bucket. Files will be named with the supplied prefix, a randomly generated middle part and the supplied suffix with default values used if either the prefix or suffix are not specified (the middle part cannot be configured and is randomly generated to ensure unique file names).

For example, with the example directives <file_prefix>record</file_prefix> and <file_suffix>.xml</file_suffix>, file names would be created as follows: records0aa323d6-fd04-4bf8-a4d4-e415832b550c.xml

Sample One Record Per File agent.xml Configuration File

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <agent>
      <share />
      <subscribe>
         <task>
            <task_name>s3_agent_subscribe</task_name>
            <message_connection password="password" user="user">https://<customer>.perspectium.net</message_connection>
    	    <instance_connection password="password" user="user">https://<instance>.service-now.com</instance_connection>   
            <handler>com.perspectium.replicator.file.S3Subscriber</handler>
            <decryption_key>The cow jumped over the moon</decryption_key>
            <access_key>AccessKey</access_key>
            <secret_access_key>SecretAccessKey</secret_access_key>
            <region>us-west-2</region>
            <s3_bucket>examples3bucket</s3_bucket>
            <file_prefix>prefix.</file_prefix>
            <file_suffix>.json</file_suffix>
            <file_format>json</file_format>
            <one_record_per_file/>
         </task>
      </subscribe>
      <polling_interval>40</polling_interval>
   </agent>
</config>
s3_subscriber.txt · Last modified: 2018/06/18 00:28 by billy