Assure1 Event Syslog Aggregator

Overview

The Assure1 Event Syslog Aggregator is a generic syslog message listener that receives messages from devices, parses the results with customizable rules and creates de-duplicated events within Assure1.

Syslog Aggregator Setup

1. Review the logic in the rules files referenced in the configuration to see the processing that will be done when syslogs are received:

   • "LoadRules" will be executed during application startup to load data that might be needed during processing.
   • "IncludeRules" will be read during application startup to load additional files that might be called during processing.
   • "BaseRules" will be executed for each device that is selected based on the configuration.

   Update the logic as needed.

2. Enable the default Service, unless a specific configuration option is needed.

   Configuration -> Broker Control -> Services

Default Service

Field	Value
Package Name	coreCollection-app
Service Name	Event Syslog Aggregator
Service Program	bin/core/collection/Syslogd
Service Arguments
Service Description	Syslog Aggregator that runs on port 514/udp
Failover Type	Standalone (Supported: Standalone, Primary/Backup)
Status	Disabled
Privileged	(Checked)

Default Configuration

Name	Value	Possible Values	Notes
BaseRules	collection/event/syslog/base.rules	Text, 255 characters	Relative path to Base Rules.
BranchDir	core/default	Text, 255 characters	relative path to Rules dir
Host	*	Text, 255 characters	DNS name or IP Address of the syslog server. Can associate to a particular interface. - NO RELOAD CONFIG SUPPORT
IncludeRules	collection/event/syslog/base.includes	Text, 255 characters	Relative path to Include Rules.
LoadRules	collection/event/syslog/base.load	Text, 255 characters	Relative path to Load Rules.
LogFile	logs/EventSyslog.log	Text, 255 characters	Relative path to Log File.
LogLevel	ERROR	OFF, FATAL, ERROR, WARN, INFO, DEBUG	Logging level used by application.
Port	514	Integer	UDP Port for syslog server to collect messages. - NO RELOAD CONFIG SUPPORT
ShardID	1	Integer	Database shard to be used.
Threads	3	Integer	Number of process threads created. The aggregator takes a third of this value (rounded up) for database threads unless overridden by the "DBThreads" application configuration.
Capture	Disabled	Enabled/Disabled	Optional - If enabled, saves the raw message in the Log.
DBThreads		Integer	Optional - Number of database threads to be created. If not specified, defaults to a third (rounded up) of "Threads" application configuration.
DisableDNS	Disabled	Enabled/Disabled	Optional - If enabled, uses IP address instead of hostname for the event Node.
FailoverBufferLimit	0	Integer	Optional - Enables Failover Standby buffer that keeps N-seconds worth of syslogs and replays this buffer when becoming Failover Active. (0=off N=seconds to keep) See Tokens: $buffer and $received.
FieldSetFile		Text, 255 characters	Optional - Path to csv file containing custom list of fields that will be used when inserting data. (Requires InsertSQLFile.)
InsertSQLFile		Text, 255 characters	Optional - Path to file containing custom SQL Insert statement for handling of event inserts. (Requires FieldSetFile.)

Best Practices

The following list shows the best practices for working with this application:

• Depending on the server specifications and loads, the number of the Threads that the aggregator uses may need to be increased, however, too many threads has also been seen to have negative effects.

• By default, devices send Syslog Messages on UDP port 514, but this can be changed for other environments. If messages are not reaching the Assure1 system, check firewall settings in the network.

• By default, syslog messages are sent in a carriage return delimited format.

Rules

This aggregator uses the Assure1 standard rules architecture, which are 100% Perl syntax. Refer to the following articles to assist in rules creation:

• Core Rules Functions

• Event Rules Functions

• Application Logging

Tokens

The aggregator exposes the following tokens for rules processing.

Token	Description
$AppConfig	Hash reference to the application configuration name-value pairs that were configured. (i.e. use $AppConfig->{'Host'} to retrieve the set value for 'Host'.)
$Event	Reference to the hash that is used to create and insert the Event data into the database. Keys map to the fields within the table used and values assigned are inserted in the database to that field. (e.g. $Event->{'IPAddress'} = '192.0.2.1' to assign the event IP address to '192.0.2.1') At least the 'Node' and 'Summary' fields must be set, or no event is inserted.
$received	Epoch time syslog was received by the aggregator.
$buffer	Flag for if was buffered during standby and was replayed (0 = No, 1 = Yes).
$line	Syslog message.
$node	DNS resolved source of syslog.
$ip	IP of source of syslog.
$discard_flag	Flag for discard (0 = No, 1 = Yes).
$count	Message Counter.
$CustomHash	Custom key, value cache available across all rules. Contents commonly defined in Load Rules then used in Base or other rules. NOTE: This variable is a shared object and any additional sub hashes or arrays must be shared before use or it will cause the error: "Invalid value for shared scalar". Instantiate the sub hash/array using '&share({})' e.g. $CustomHash->{SubObject} = &share({});
$StorageHash	Internal cache used as the StorageHash option when calling rules functions such as FindDeviceID(). NOTE: The structure of this cache is subject to change! Not recommended for custom global storage or manual manipulation; use $CustomHash.

Example Integrations

Creating Custom Rules Files

In this rules creation example, an IP Phone is sending syslogs that appear in the event list with summary:

Missing Rules for-<134>[0]SIP:RegFailed;Retry in 30s
Missing Rules for-<134>+++ send scaps discovery query

NOTE - The "Missing Rules for" is added to the beginning of the summary for any syslog messages without corresponding rules.

Steps

1. Go to the Rules UI:

   Configuration -> Rules

2. Expand the folder path: core -> default -> collection -> event -> syslog

3. Select the "syslog" folder, then click "Add -> Add File". Enter the following:

   • File Name => ipphone.rules

   • Logic

     $Log->Message("DEBUG", " -> Starting running IP Phone Rules");
     
     $Event->{'SubMethod'}  = "IPPhone";
     $Event->{'Summary'}    = $line;
     $Event->{'AlarmGroup'} = "IPPhone";
     
     if ($line =~ "RegFailed"){
         # Phone Registration Failed
         $Event->{'Severity'}  = 3;
         $Event->{'SubMethod'} = "Registration Failure";
     }
     elsif ($line eq "<134>+++ send scaps discovery query") {
         #Discard unwanted event
         $discard_flag = 1;
     }
     else {
         # Catch all for all Phone events
         $Event->{'Severity'} = 0;
     }
     
     $Event->{'AlarmKey'} = $Event->{'SubMethod'} . $Event->{'Summary'};
     
     $Log->Message("DEBUG", " -> Finished running IP Phone rules");
     

     In the above example:

     • Events containing the text "RegFailed" are detected using a Regular Expression match. The event "Severity" is set to 3 (Minor) and the "SubMethod" is set to "Registration Failure".

     • Events containing the text "<134>+++ send scaps discovery query" are discarded by setting the "discard_flag" to 1. This is useful if the event should not be kept.

     • Any other events have the "Severity" set to 0 (Normal).

     Of particular importance in the example above is the "$Event->{'AlarmKey'}" as the AlarmKey is how events are deduplicated.

   • Click "Submit", then enter a commit message, then click "OK".

4. In the "syslog" folder, select the "base.includes" file. Add the following line:

   • Logic

     IPPhoneRules,collection/event/syslog/ipphone.rules
     

   • Click "Submit", then enter a commit message, then click "OK".

5. Determine how the aggregator should know the correct processing direction of the event using any of the available data. For example, the event can be based on the source of the Syslog Message:

   elsif ($node =~ "ipphone.*\.example\.com") {
       ...
   }
   

   Or the event can be processed based on the message itself:

   elsif ($line =~ /\<134\>/){
       ...
   }
   

   In the "syslog" folder, select on the "base.rules" file. Add the following line:

   • Logic

     elsif ($line =~ /\<134\>/ || $node =~ "ipphone.*\.example\.com") {
         $Event->{'SubMethod'} = "IPPhone";
         $Log->Message("DEBUG", "Running IPPhone Syslog Rules");
         IPPhoneRules();
     }
     

   • Click "Submit", then enter a commit message, then click "OK".

6. Restart the Syslog Aggregator.

7. Verify the Syslog Aggregator is processing the events correctly.

Administration Details

The following list shows the technical details need for advanced administration of the application:

• Package - coreCollection-app

• Synopsis - ./Syslogd [OPTIONS]

• Options:

   -c, --AppConfigID N   Application Config ID (Service, Job, or Request ID)
   -?, -h, --Help        Print usage and exit
  

• Threaded - Multi-Threaded