Assure1 Event FIFO (Flat File) Aggregator¶

Overview¶

The Assure1 Event FIFO (Flat File) Aggregator is a generic integration that tails a file, then parses any lines written to the file with customizable rules and creates de-duplicated events within Assure1.

Note

Only lines that are written to the file while the aggregator is running will be processed.

FIFO (Flat File) Aggregator Setup¶

Update the application configuration to read the correct data file.
Enable the default Service, unless a specific configuration option is needed.

Configuration -> Broker Control -> Services

Default Service¶

Field	Value
Package Name	coreCollection-app
Service Name	Event File Aggregator
Service Program	bin/core/collection/FIFOd
Service Arguments
Service Description	File (FIFO) Aggregator that reads event lines from a local file
Failover Type	Standalone (Supported: Standalone, Primary/Backup)
Status	Disabled
Privileged	(Checked)

Default Configuration¶

Name	Value	Possible Values	Notes
BaseRules	collection/event/file/base.rules	Text, 255 characters	Relative path to Base Rules.
BranchDir	core/default	Text, 255 characters	The SVN directory that contains the rules files to be used.
File	/var/log/messages	Text, 255 characters	Full path to file that will be tailed by the aggregator - NO RELOAD CONFIG SUPPORT.
IncludeRules	collection/event/file/base.includes	Text, 255 characters	Relative path to Include Rules.
LoadRules	collection/event/file/base.load	Text, 255 characters	Relative path to Load Rules.
LogFile	logs/EventFile.log	Text, 255 characters	Relative path to Log File.
LogLevel	ERROR	OFF, FATAL, ERROR, WARN, INFO, DEBUG	Logging level used by application.
ShardID	1	Integer	Database shard to be used.
Threads	3	Integer	Number of process threads created. The aggregator takes a third of this value (rounded up) for database threads unless overridden by the "DBThreads" application configuration.
Capture	Disabled	Enabled/Disabled	Optional - If enabled, saves the raw message in the Log.
DBThreads		Integer	Optional - Number of database threads to be created. If not specified, defaults to a third (rounded up) of "Threads" application configuration.
FailoverBufferLimit	0	Integer	Optional - Enables Failover Standby buffer that keeps N-seconds worth of lines and replays this buffer when becoming Failover Active. (0=off N=seconds to keep) See Tokens: $buffer and $received.
FieldSetFile		Text, 255 characters	Optional - Path to csv file containing custom list of fields that will be used when inserting data. (Requires InsertSQLFile.)
InsertSQLFile		Text, 255 characters	Optional - Path to file containing custom SQL Insert statement for handling of event inserts. (Requires FieldSetFile.)

Best Practices¶

The FIFO Aggregator can tail any file that is accessible to the local system. A file that is located on a remote system can be tailed by mounting the remote file system to the server running the FIFO Aggregator service.

Rules¶

This aggregator uses the Assure1 standard rules architecture. The rules are written in Perl syntax. Refer to the following guides for details on rules creation:

Tokens¶

The aggregator exposes the following tokens for rules processing.

Token	Description
$Event	Reference to the hash that is used to create and insert the Event data into the database. Keys map to the fields within the table used and values assigned are inserted in the database to that field. (e.g. $Event->{'IPAddress'} = '192.0.2.1' to assign the event IP address to '192.0.2.1') At least the 'Node' and 'Summary' fields must be set, or no event is inserted.
$received	Epoch time line was received by the aggregator.
$buffer	Flag for if was buffered during standby and was replayed (0 = No, 1 = Yes)
$line	Message, delimited by carriage return
$discard_flag	Flag for discard (0 = No, 1 = Yes)
$count	Message Counter
$AppConfig	Hash reference to the application configuration name-value pairs that were configured. (i.e. use $AppConfig->{'Host'} to retrieve the set value for 'Host'.)
$CustomHash	Custom key, value cache available across all rules. Contents commonly defined in Load Rules then used in Base or other rules. NOTE: This variable is a shared object and any additional sub hashes or arrays must be shared before use or it will cause the error: "Invalid value for shared scalar". Instantiate the sub hash/array using '&share({})' e.g. $CustomHash->{SubObject} = \&share({});

Example Integration¶

In the below example, a section of logic was added so that lines that have ExampleLineProcessing will processed. For each of those lines, the data will be parsed and the Node and Status values will be extracted. The default event fields are then modified with those values, along with setting the SubMethod. If the Status contains Down, a event is created with Severity set to 5 (Critical). If the Status contains Up, a event is created with Severity set to 0 (Clear), which will also clear the previous Down event.

elsif ($line =~ /ExampleLineProcessing-/) {
    $Event->{'SubMethod'} = "Example FIFO";
    (my $Device, $Status) = $line =~ /Device:(.*) Status:(.*)/;
    if($Status eq 'Up'){
        $Event->{Severity} = 0;
    }
    elsif($Status eq 'Down'){
        $Event->{Severity} = 5;
    }
    $Event->{'Node'}  = $Device;
    $Event->{Summary} = "$line - $Device is $Status";
}

Administration Details¶

The following list shows the technical details needed for advanced administration of the application:

Package - coreCollection-app
Synopsis - ./FIFOd [OPTIONS]

Option:

 -c, --AppConfigID N   Application Config ID (Service, Job, or Request ID)
 -?, -h, --Help        Print usage and exit

Threaded - Multi-Threaded