Assure1 Backup and Restore¶

An organization's data is of paramount importance. The only way to fully protect an environment is to regularly backup the data so it can be fully recovered in the event of a disaster. The Assure1 backup and recovery plan is a part of an organization's local policy and must be created by the organization's system and database administrators.

These instructions listed below are merely guidance for an overall backup and recovery plan. An Assure1 administrator will also need to take an organization's operational and management processes into account.

For the Assure1 system, you should make available a mount point on the Database Server and Presentation Server to conduct backups. In this document we will be writing backups to /mnt/backups. Please substitute /mnt/backups for your chosen backup mountpoint.

Note

All commands in the following documentation are run as root or a privileged user. $A1BASEDIR needs to be set to your installation directory (e.g. export A1BASEDIR=/opt/assure1)

Considerations¶

Systems with databases must backup the database files while the database is shutdown, or using a DB backup utility if remaining active.
The Event Analytics and Metrics databases may be too big to backup locally, and it is suggested to backup to a mount on a SAN or NFS instead.
Examples below have an external system mounted at /mnt/backups.

Manual Full Offline Backup and Restore¶

Full Offline Backup¶

The simplest backup involves all applications shutdown to perform an offline backup. This should only be done during a scheduled maintenance window.

Stop all processes.

$A1BASEDIR/bin/BrokerControl --batch stopall
service assure1-broker stop
service assure1-web stop  # (If presentation server)
service assure1-bus stop  # (If presentation server)
service assure1-db stop   # (If presentation or database server)

Create a full backup of Assure1.
```
tar czpf /mnt/backups/assure1.tgz $A1BASEDIR
```
Note

If running as a VM, a snapshot can be created and stored as a backup instead.

Start all processes.

service assure1-db start   # (If presentation or database server)
service assure1-bus start  # (If presentation server)
service assure1-web start  # (If presentation server)
service assure1-broker start

Full Offline Restore¶

Stop all processes, if this is not a new server.

$A1BASEDIR/bin/BrokerControl --batch stopall
service assure1-broker stop
service assure1-web stop  # (If presentation server)
service assure1-bus stop  # (If presentation server)
service assure1-db stop   # (If presentation or database server)

Restore the product.

tar xzf /mnt/backups/assure1.tgz -C $A1BASEDIR

Restore the startup scripts, if this is a new server.

# All servers
cp $A1BASEDIR/distrib/startup/assure1/init.d/assure1-broker.service.local /etc/systemd/system/assure1-broker.service
systemctl --quiet enable assure1-broker.service


# If a presentation server
cp $A1BASEDIR/distrib/startup/vendorRabbitMQ/init.d/assure1-bus.service.local /etc/systemd/system/assure1-bus.service
systemctl --quiet enable assure1-bus.service
cp $A1BASEDIR/distrib/startup/vendorApache/init.d/assure1-web.service.local /etc/systemd/system/assure1-web.service
systemctl --quiet enable assure1-web.service


# If presentation or database server
cp $A1BASEDIR/distrib/startup/vendorMariaDB/init.d/assure1-db.service.local /etc/systemd/system/assure1-db.service
systemctl --quiet enable assure1-db.service


# All servers
systemctl daemon-reload

Start all processes.

service assure1-db start   # (If presentation or database server)
service assure1-bus start  # (If presentation server)
service assure1-web start  # (If presentation server)
service assure1-broker start

Scheduled Online Backup and Restores¶

The following are the databases for backup and restore; note that the DB schema name is associated to the DB type, but they are not referenced together for the actual backup:

MariaDB for Assure1 and Events
InfluxDB for Metrics
Elastic for EventAnalytics (History)/Logs
OrientDB for Graph
Neo4j for Graph

Note

The following cron schedules are examples only. They should be updated to match your organization's policies.

Preparation¶

Directories for each of these scheduled backups need to be created ahead of time. Depending on whether the backup is on a presentation or database server, some of these directories may not be needed.

mkdir -p /mnt/backups/assure1/full  /mnt/backups/assure1/inc{1..6}
mkdir -p /mnt/backups/mariadb/full  /mnt/backups/mariadb/inc{1..6}
mkdir -p /mnt/backups/influxdb/full /mnt/backups/influxdb/inc{1..6}
mkdir -p /mnt/backups/elastic/
mkdir -p /mnt/backups/orientdb/
mkdir -p /mnt/backups/neo4j/

Assure1 Home¶

Backup¶

The Assure1 home directory backup should exclude data from running databases.

Warning

Crontab entries need to be created on Saturday afternoon to make sure the full backup is run first.

/etc/cron.d/assure1-home:

0 0 * * 0 assure1 rsync -a --delete --exclude={'var/mariadb_data','var/influxdb','var/elasticsearch',var/orientdb','var/neo4j'} $A1BASEDIR /mnt/backups/assure1/full && ln -s /mnt/backups/assure1/full /mnt/backups/assure1/latest
0 0 * * 1 assure1 rsync -a --delete --exclude={'var/mariadb_data','var/influxdb','var/elasticsearch',var/orientdb','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc1 && ln -s /mnt/backups/assure1/inc1 /mnt/backups/assure1/latest
0 0 * * 2 assure1 rsync -a --delete --exclude={'var/mariadb_data','var/influxdb','var/elasticsearch',var/orientdb','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc2 && ln -s /mnt/backups/assure1/inc2 /mnt/backups/assure1/latest
0 0 * * 3 assure1 rsync -a --delete --exclude={'var/mariadb_data','var/influxdb','var/elasticsearch',var/orientdb','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc3 && ln -s /mnt/backups/assure1/inc3 /mnt/backups/assure1/latest
0 0 * * 4 assure1 rsync -a --delete --exclude={'var/mariadb_data','var/influxdb','var/elasticsearch',var/orientdb','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc4 && ln -s /mnt/backups/assure1/inc4 /mnt/backups/assure1/latest
0 0 * * 5 assure1 rsync -a --delete --exclude={'var/mariadb_data','var/influxdb','var/elasticsearch',var/orientdb','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc5 && ln -s /mnt/backups/assure1/inc5 /mnt/backups/assure1/latest
0 0 * * 6 assure1 rsync -a --delete --exclude={'var/mariadb_data','var/influxdb','var/elasticsearch',var/orientdb','var/neo4j'} --link-dest /mnt/backups/assure1/latest $A1BASEDIR /mnt/backups/assure1/inc6 && ln -s /mnt/backups/assure1/inc6 /mnt/backups/assure1/latest

Restore¶

Stop all processes, if this is not a new server.

$A1BASEDIR/bin/BrokerControl --batch stopall
service assure1-broker stop
service assure1-web stop  # (If presentation server)
service assure1-bus stop  # (If presentation server)
service assure1-db stop   # (If presentation or database server)

Restore the product.

rsync -a --delete /mnt/backups/assure1/latest $A1BASEDIR

Restore the startup scripts, if this is a new server.

# All servers
cp $A1BASEDIR/distrib/startup/assure1/init.d/assure1-broker.service.local /etc/systemd/system/assure1-broker.service
systemctl --quiet enable assure1-broker.service


# If a presentation server
cp $A1BASEDIR/distrib/startup/vendorRabbitMQ/init.d/assure1-bus.service.local /etc/systemd/system/assure1-bus.service
systemctl --quiet enable assure1-bus.service
cp $A1BASEDIR/distrib/startup/vendorApache/init.d/assure1-web.service.local /etc/systemd/system/assure1-web.service
systemctl --quiet enable assure1-web.service


# If presentation or database server
cp $A1BASEDIR/distrib/startup/vendorMariaDB/init.d/assure1-db.service.local /etc/systemd/system/assure1-db.service
systemctl --quiet enable assure1-db.service


# All servers
systemctl daemon-reload

Start all processes.

service assure1-db start   # (If presentation or database server)
service assure1-bus start  # (If presentation server)
service assure1-web start  # (If presentation server)
service assure1-broker start

MariaDB¶

Read and understand MariaDB backup command usage: https://mariadb.com/kb/en/mariabackup-options/

Backup¶

Warning

Crontab entries need to be created on Saturday afternoon to make sure the full backup is run first.

/etc/cron.d/assure1-mariadb:

0 1 * * 0 assure1 $A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --target-dir=/mnt/backups/mariadb/full
0 1 * * 1 assure1 $A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mariadb/full --target-dir=/mnt/backups/mariadb/inc1
0 1 * * 2 assure1 $A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mariadb/inc1 --target-dir=/mnt/backups/mariadb/inc2
0 1 * * 3 assure1 $A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mariadb/inc2 --target-dir=/mnt/backups/mariadb/inc3
0 1 * * 4 assure1 $A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mariadb/inc3 --target-dir=/mnt/backups/mariadb/inc4
0 1 * * 5 assure1 $A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mariadb/inc4 --target-dir=/mnt/backups/mariadb/inc5
0 1 * * 6 assure1 $A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --backup --incremental-basedir=/mnt/backups/mariadb/inc5 --target-dir=/mnt/backups/mariadb/inc6

Restore¶

Stop the database, if this is not a new server.
```
service assure1-db stop
```

Prepare the database restoration. Stop running incremental restores when you reach the previous day (e.g. inc3 would be the last if today is Thursday).

$A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mariadb/full
$A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mariadb/full --incremental-dir=/mnt/backups/mariadb/inc1
$A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mariadb/full --incremental-dir=/mnt/backups/mariadb/inc2
$A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mariadb/full --incremental-dir=/mnt/backups/mariadb/inc3
$A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mariadb/full --incremental-dir=/mnt/backups/mariadb/inc4
$A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mariadb/full --incremental-dir=/mnt/backups/mariadb/inc5
$A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --prepare --apply-log-only --target-dir=/mnt/backups/mariadb/full --incremental-dir=/mnt/backups/mariadb/inc6

Restore the database.

$A1BASEDIR/vendor/mariadb/bin/mariabackup --defaults-file=$A1BASEDIR/etc/my-root.cnf --copy-back --force-non-empty-directories --target-dir=/mnt/backups/mariadb/full

Start the database.
```
service assure1-db start
```

InfluxDB¶

Read and understand InfluxDB backup command usage: https://docs.influxdata.com/influxdb/v1.8/administration/backup_and_restore/

Backup¶

Warning

Crontab entries need to be created on Saturday afternoon to make sure the full backup is run first.

/etc/cron.d/assure1-influxdb:

0 2 * * 0 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/full
0 2 * * 1 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc1
0 2 * * 2 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc2
0 2 * * 3 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc3
0 2 * * 4 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc4
0 2 * * 5 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc5
0 2 * * 6 assure1 $A1BASEDIR/vendor/influxdb/bin/influxd backup -portable -start \$(date --date='yesterday' '+\%Y-\%m-\%dT00:00:00Z') -end \$(date '+\%Y-\%m-\%dT00:00:00Z') /mnt/backups/influxdb/inc6

Restore¶

Stop the database, if this is not a new server.

$A1BASEDIR/bin/BrokerControl --batch listservices | grep InfluxDB
$A1BASEDIR/bin/BrokerControl --batch stopservice <ID>  # ID found from previous step

Restore the database. Stop running incremental restores when you reach the previous day (e.g. inc3 would be the last if today is Thursday).

$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/full
$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc1
$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc2
$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc3
$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc4
$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc5
$A1BASEDIR/vendor/influxdb/bin/influxd restore -portable /mnt/backups/influxdb/inc6

Start the database.

$A1BASEDIR/bin/BrokerControl --batch listservices | grep InfluxDB
$A1BASEDIR/bin/BrokerControl --batch startservice <ID>  # ID found from previous step

Elasticsearch¶

Read and understand Elasticsearch backup command usage:https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html

Note

Elasticsearch backup scheduling will be available in a future update of Assure1.

Preparation¶

Edit the "elasticsearch.yml" file in the "$A1BASEDIR/vendor/elasticsearch/config/" directory and add the following line:
```
path.repo: ["/mnt/backups/elastic"]
```

Restart the Elasticsearch service:

$A1BASEDIR/bin/BrokerControl --batch listservices | grep Elasticsearch
$A1BASEDIR/bin/BrokerControl --batch stopservice <ID>   # ID found from first step
$A1BASEDIR/bin/BrokerControl --batch startservice <ID>  # ID found from first step

In the UI, open the Elastic console by navigating to Analytics -> Events -> Administration -> Console.

Run the following in the console to create the backup repository:

PUT /_snapshot/backup
{
  "type": "fs",
  "settings": {
    "location": "/mnt/backups/elastic"
  }
}

Backup¶

In the UI, open the Elastic console by navigating to Analytics -> Events -> Administration -> Console.

Run the following in the console:

PUT /_snapshot/backup/<snapshot-{now%2fd}>

Restore¶

In the UI, open the Elastic console by navigating to Analytics -> Events -> Administration -> Console.
Run the following in the console to close the indices that are going to be restored:
```
POST /eventanalytics-YYYY.MM.DD/_close
--- AND/OR ---
POST /filebeat-7.3.1-YYYY.MM.DD/_close
```
Warning

System indices should not be closed while running this command.
Note
1. All indices can be displayed by running this command:
```
GET /_cat/indices
```
2. All event analytics and filebeat indices can be closed by running these commands:
```
POST /eventanalytics-*/_close
--- AND/OR ---
POST /filebeat-*/_close
```

Run the following in the console to restore indices from snapshot:

POST /_snapshot/backup/<NAME_OF_SNAPSHOT>/_restore
{
  "indices": "<NAME_OF_INDEX_1>,<NAME_OF_INDEX_2>,..."
}

OrientDB¶

Read and understand OrientDB backup command usage: http://orientdb.com/docs/3.0.x/admin/Export-to-and-Import-from-JSON.html

Backup¶

/etc/cron.d/assure1-orientdb:

0 1 * * * assure1 $A1BASEDIR/vendor/orientdb/bin/console.sh "CONNECT remote:$HOSTNAME/Graph root root; EXPORT DATABASE /mnt/backups/orientdb/full.gz"

Restore¶

$A1BASEDIR/vendor/orientdb/bin/console.sh "CONNECT remote:HOSTNAME/Graph root root; IMPORT DATABASE /mnt/backups/orientdb/full.gz  -preserveClusterIDs=true"

Neo4j¶

Read and understand Neo4j backup command usage: https://neo4j.com/docs/operations-manual/current/backup-restore/modes/

Backup¶

/etc/cron.d/assure1-neo4j:

0 2 * * * assure1 env HEAP_SIZE=2G NEO4J_CONF=$A1BASEDIR/etc/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin backup --backup-dir=/mnt/backups/neo4j --database=* --pagecache=4G

Restore¶

NEO4J_CONF=$A1BASEDIR/etc/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin restore --from=/mnt/backups/neo4j --database=neo4j --force
NEO4J_CONF=$A1BASEDIR/etc/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin restore --from=/mnt/backups/neo4j --database=Graph --force