Skip to content

v4 to v5 Upgrade Guide

Overview

Upgrading Assure1 v4 to Assure1 v5 allows an existing installation to gain the benefits of the latest functionality and enhancements available. Upgrades should be taken seriously and planned ahead of time. If you have access to a development instance, upgrades should be performed on there before planning your production upgrade.

Some of the major changes provided from upgrading include:

  • Unified service assurance without separate packages for each collection domain

  • Applications consolidated into core packages

  • Database changes

    • Event Historical database -> Event Analytics database

    • Metrics database -> InfluxDB Metric database

    • Monolith HSE database tables -> OrientDB Graph database

Note

Data will have to be migrated from the old databases to the new databases after upgrade. Depending on the sizes, it may take a long time. Plan for reporting if needed.

Note

If you do not have enough disk space for 2 copies of data for the Metric, Graph, and Historical (Event Analytics) databases, you should allocate new servers for those roles.

Upgrades can be performed in one of two ways supported

  1. In-place upgrades are performed by upgrading the existing Assure1 v4 servers to Assure1 v5. The new Assure1 v5 stack will completely migrate and replace the existing Assure1 v4 stack.

    • Additionally, to maintain the least amount of disruption, rolling upgrades can occur if only the primary side of the Assure1 v4 stack is upgraded first.

  2. Forklift upgrades are performed by taking a backup of some of the Assure1 v4 systems and restoring them on new hardware. A brand new Assure1 v5 stack is installed creating a dual environment for a short amount of time.

Pre-requisites

  1. Your Assure1 v4 installation must be updated to at least Update 80.

  2. If your Assure1 v4 installation is not on RedHat/CentOS 7, you must perform a forklift upgrade onto new servers.

  3. Run UpgradeAudit on your primary presentation server and correct any issues identified. Run again to validate corrections.

    $A1BASEDIR/bin/UpgradeAudit

  4. If you do not have a license for the full Assure1 v4 product, you will need to get an updated license.

    1. e.g. If you are only licensed for Fault Management, you will need a full license (including Performance, Topology, and SLM).

    2. At a minimum, you will need to install the SchemaDB packages listed by UpgradeAudit.

  5. (optional) Run UpgradeAudit on additional servers if you think there might be an uncommited rules files it would identify, and correct those issues. Run again to validate corrections.

Warning

If you do not correct issues identified with the UpgradeAudit, your upgrade may fail completely in some cases!

Preparation

  1. If you have a development environment, it is suggested you upgrade this environment first to get a feel for the process.

  2. Backup all systems prior to upgrade.

  3. Create a backup plan that includes this document, the prior upgrade experience notes, and rollback steps from backups if needed.

  4. Plan a maintenance window for the upgrade.

In-place Upgrade

  1. Starting with your primary presentation server, install Assure1 v5.

    1. Stop all v4 processes.

      $A1BASEDIR/bin/BrokerControl --batch stopall
systemctl stop monolith-broker
systemctl stop monolith-web
systemctl stop monolith-db
systemctl stop monolith-infinidb

    2. Verify all v4 processes are stopped.

    3. Download and install the Assure1 v5 RPM. The SetupWizard will detect the system is being upgraded and perform additional steps as needed.

    4. Repeat installation for each server, choosing the correct roles.

      Note

      If performing a rolling upgrade, do not upgrade any redundant servers at this time. Validate both v4 and v5 systems are working as expected.

    5. Once rolling upgrade validation has been verified, repeat the previous steps to upgrade the remaining redundant servers.

    6. On all redundant presentation and Event/Metric/Graph database servers, run the Redundancy Wizard

      $A1BASEDIR/bin/redundancy/RedundancyWizard

    7. On all redundant Historical database servers, run the Analytics Wizard

      $A1BASEDIR/bin/historical/AnalyticsWizard

  2. Migrate legacy databases as needed.

    1. Metric database migration

      1. From a server with the Core.Processing role, run the migration application.

        1. Basic, single threaded example:

          $A1BASEDIR/bin/core/processing/MetricDataMigration --MetricsDBHost v4-metrics-db.example.com

        2. Advanced, threaded, and ranged examples:

          $A1BASEDIR/bin/core/processing/MetricDataMigration --MetricsDBHost v4-metrics-db.example.com --Threads 4 --MetricIDs 1 10000
$A1BASEDIR/bin/core/processing/MetricDataMigration --MetricsDBHost v4-metrics-db.example.com --Threads 4 --MetricIDs 10001 20000
$A1BASEDIR/bin/core/processing/MetricDataMigration --MetricsDBHost v4-metrics-db.example.com --Threads 4 --MetricIDs 20001 30000
# continue as needed

    2. Graph database migration

      1. From a server with the Core.Processing role, run the migration application.

        1. Basic, single threaded example:

          $A1BASEDIR/bin/core/processing/GraphDataMigration --HSEDBHost v4-monolith-db.example.com

        2. Advanced, threaded, and ranged examples:

          $A1BASEDIR/bin/core/processing/GraphDataMigration --HSEDBHost v4-monolith-db.example.com --Threads 4 --LinkIDs 1 10000
$A1BASEDIR/bin/core/processing/GraphDataMigration --HSEDBHost v4-monolith-db.example.com --Threads 4 --LinkIDs 10001 20000
$A1BASEDIR/bin/core/processing/GraphDataMigration --HSEDBHost v4-monolith-db.example.com --Threads 4 --LinkIDs 20001 30000
# continue as needed

    3. Historical database migration

      1. From a server with the Historical.Processing role, run the migration application.

        1. Basic, single threaded example:

          $A1BASEDIR/bin/historical/HistoricalDataMigration --HistoricalDBHost v4-historical-db.example.com --From 2019-01-01 --To 2019-12-31

        2. Advanced, threaded, and ranged examples:

          $A1BASEDIR/bin/historical/HistoricalDataMigration  --HistoricalDBHost v4-historical-db.example.com --Threads 4 --From 2019-01-01 --To 2019-01-31
$A1BASEDIR/bin/historical/HistoricalDataMigration  --HistoricalDBHost v4-historical-db.example.com --Threads 4 --From 2019-02-01 --To 2019-02-31
$A1BASEDIR/bin/historical/HistoricalDataMigration  --HistoricalDBHost v4-historical-db.example.com --Threads 4 --From 2019-03-01 --To 2019-03-31
# continue as needed

  3. Review and validate configurations, data migrations, and functionality of systems.

    1. If you have custom applications, tools, or script (excluding rules), you will need to manually move them.

  4. Finalize upgrade by removing Assure1 v4 completely.

    1. On each system, run:

      $A1BASEDIR/bin/SetupWizard --Finalize-Upgrade

Forklift Upgrade

  1. On your existing v4 presentation server and any Event and/or Metric database (redundant server preferable if installed), backup Assure1 v4.

    1. Stop all v4 processes.

      $A1BASEDIR/bin/BrokerControl --batch stopall
systemctl stop monolith-broker
systemctl stop monolith-web
systemctl stop monolith-db
systemctl stop monolith-infinidb

    2. Backup Assure1 v4

      Note

      Replace /opt and monolith with your custom path if needed.

      cd /opt
tar czf Assure1v4.tgz monolith

    3. Start all v4 processes.

      systemctl start monolith-db
systemctl start monolith-web
systemctl start monolith-broker
systemctl start monolith-infinidb

  2. On your new v5 presentation server, install Assure1 v4 from backup.

    1. Install Assure1 v4 (e.g. using Update 80 or later)

      Note

      Replace the RPM filename with the correct version you are using.

      yum -y install Monolith-Broker-4.29.1-9500.x86_64.rpm

    2. Restore backup for the same type of server

      cd /opt
tar xzf /opt/upgrade/Assure1v4.tgz

    3. If the v4 backup was from a redundant server, disable redundant configurations before starting any services.

      Note

      Replace $A1BASEDIR with your v4 installation directory (e.g. /opt/monolith).

      rm -f $A1BASEDIR/var/mysql_data/master.info
rm -f $A1BASEDIR/etc/unison/*

  3. On the primary presentation server only, enable and start the database to make changes.

    1. Enable and start the database

      Note

      Replace $A1BASEDIR with your v4 installation directory (e.g. /opt/monolith).

      cp $A1BASEDIR/distrib/startup/vendorMySQL/init.d/monolith-db.service /etc/systemd/system/monolith-db.service
systemctl enable monolith-db.service
systemctl daemon-reload
systemctl start monolith-db
$A1BASEDIR/vendor/mysql/bin/mysql -u monolith -p Monolith

    2. Change host FQDNs for each server from the previous value to the new FQDNs

      1. Change "Servers" with the template UPDATE Servers SET ServerHostFQDN = "<NEW-HOST-FQDN>" WHERE ServerHostFQDN = "<OLD-HOST-FQDN>";

        UPDATE Servers SET ServerHostFQDN = "assure1-pres1.example.com", WebFQDN = "assure1-pres.example.com" WHERE ServerHostFQDN = "monolith-pres1.example.com";
UPDATE Servers SET ServerHostFQDN = "assure1-pres2.example.com", WebFQDN = "assure1-pres.example.com" WHERE ServerHostFQDN = "monolith-pres2.example.com";
UPDATE Servers SET ServerHostFQDN = "assure1-db1.example.com"  WHERE ServerHostFQDN = "monolith-db1.example.com";
UPDATE Servers SET ServerHostFQDN = "assure1-db2.example.com"  WHERE ServerHostFQDN = "monolith-db2.example.com";
UPDATE Servers SET ServerHostFQDN = "assure1-ea1.example.com"  WHERE ServerHostFQDN = "monolith-ea1.example.com";
UPDATE Servers SET ServerHostFQDN = "assure1-ea2.example.com"  WHERE ServerHostFQDN = "monolith-ea2.example.com";
UPDATE Servers SET ServerHostFQDN = "assure1-col1.example.com" WHERE ServerHostFQDN = "monolith-col1.example.com";
UPDATE Servers SET ServerHostFQDN = "assure1-col2.example.com" WHERE ServerHostFQDN = "monolith-col2.example.com";

      2. Change "Devices" with the template UPDATE Devices SET DNSName = "<NEW-HOST-FQDN>", IPAddress = INET_ATON("<NEW-HOST-IPv4>"), IPv6Address = INET6_ATON("<NEW-HOST-IPv6>") WHERE DNSName = "<OLD-HOST-FQDN>";

        UPDATE Devices SET DNSName = "assure1-pres1.example.com", IPAddress = INET_ATON('<A1_PRES1_IPv4_ADDRESS>'), IPv6Address = INET6_ATON('<A1_PRES1_IPv6_ADDRESS>') WHERE DNSName = "monolith-pres1.example.com";
UPDATE Devices SET DNSName = "assure1-pres2.example.com", IPAddress = INET_ATON('<A1_PRES2_IPv4_ADDRESS>'), IPv6Address = INET6_ATON('<A1_PRES2_IPv6_ADDRESS>') WHERE DNSName = "monolith-pres2.example.com";
UPDATE Devices SET DNSName = "assure1-db1.example.com",   IPAddress = INET_ATON('<A1_DB1_IPv4_ADDRESS>'),   IPv6Address = INET6_ATON('<A1_DB1_IPv6_ADDRESS>')   WHERE DNSName = "monolith-db1.example.com";
UPDATE Devices SET DNSName = "assure1-db2.example.com",   IPAddress = INET_ATON('<A1_DB2_IPv4_ADDRESS>'),   IPv6Address = INET6_ATON('<A1_DB2_IPv6_ADDRESS>')   WHERE DNSName = "monolith-db2.example.com";
UPDATE Devices SET DNSName = "assure1-ea1.example.com",   IPAddress = INET_ATON('<A1_EA1_IPv4_ADDRESS>'),   IPv6Address = INET6_ATON('<A1_EA1_IPv6_ADDRESS>')   WHERE DNSName = "monolith-ea1.example.com";
UPDATE Devices SET DNSName = "assure1-ea2.example.com",   IPAddress = INET_ATON('<A1_EA2_IPv4_ADDRESS>'),   IPv6Address = INET6_ATON('<A1_EA2_IPv6_ADDRESS>')   WHERE DNSName = "monolith-ea2.example.com";
UPDATE Devices SET DNSName = "assure1-col1.example.com",  IPAddress = INET_ATON('<A1_COL1_IPv4_ADDRESS>'),  IPv6Address = INET6_ATON('<A1_COL1_IPv6_ADDRESS>')  WHERE DNSName = "monolith-col1.example.com";
UPDATE Devices SET DNSName = "assure1-col2.example.com",  IPAddress = INET_ATON('<A1_COL2_IPv4_ADDRESS>'),  IPv6Address = INET6_ATON('<A1_COL2_IPv6_ADDRESS>')  WHERE DNSName = "monolith-col2.example.com";

      3. Change "CoreDatabases" with the template UPDATE CoreDatabases SET DBHost = "<NEW-HOST-FQDN>", DatabaseName = REPLACE(DatabaseName, "<OLD-HOST-FQDN>", "<NEW-HOST-FQDN>") WHERE DBHost = "<OLD-HOST-FQDN>";

        UPDATE CoreDatabases SET DBHost = "assure1-pres1.example.com", DatabaseName = REPLACE(DatabaseName, "monolith-pres1.example.com", "assure1-pres1.example.com") WHERE DBHost = "monolith-pres1.example.com";
UPDATE CoreDatabases SET DBHost = "assure1-pres2.example.com", DatabaseName = REPLACE(DatabaseName, "monolith-pres2.example.com", "assure1-pres2.example.com") WHERE DBHost = "monolith-pres2.example.com";
UPDATE CoreDatabases SET DBHost = "assure1-db1.example.com",   DatabaseName = REPLACE(DatabaseName, "monolith-db1.example.com",   "assure1-db1.example.com")   WHERE DBHost = "monolith-db1.example.com";
UPDATE CoreDatabases SET DBHost = "assure1-db2.example.com",   DatabaseName = REPLACE(DatabaseName, "monolith-db2.example.com",   "assure1-db2.example.com")   WHERE DBHost = "monolith-db2.example.com";

      4. Change "BrokerComponentConfig" with the template UPDATE BrokerComponentConfig SET BrokerComponentConfigValue = REPLACE(BrokerComponentConfigValue, "<OLD-HOST-FQDN>", "<NEW-HOST-FQDN>") WHERE BrokerComponentConfigTypeID IN (8, 19);

        UPDATE BrokerComponentConfig SET BrokerComponentConfigValue = REPLACE(BrokerComponentConfigValue, "monolith-pres1.example.com", "assure1-pres1.example.com") WHERE BrokerComponentConfigTypeID IN (8, 19);
UPDATE BrokerComponentConfig SET BrokerComponentConfigValue = REPLACE(BrokerComponentConfigValue, "monolith-pres2.example.com", "assure1-pres2.example.com") WHERE BrokerComponentConfigTypeID IN (8, 19);
UPDATE BrokerComponentConfig SET BrokerComponentConfigValue = REPLACE(BrokerComponentConfigValue, "monolith-db1.example.com",   "assure1-db1.example.com")   WHERE BrokerComponentConfigTypeID IN (8, 19);
UPDATE BrokerComponentConfig SET BrokerComponentConfigValue = REPLACE(BrokerComponentConfigValue, "monolith-db2.example.com",   "assure1-db2.example.com")   WHERE BrokerComponentConfigTypeID IN (8, 19);
UPDATE BrokerComponentConfig SET BrokerComponentConfigValue = REPLACE(BrokerComponentConfigValue, "monolith-ea1.example.com",   "assure1-ea1.example.com")   WHERE BrokerComponentConfigTypeID IN (8, 19);
UPDATE BrokerComponentConfig SET BrokerComponentConfigValue = REPLACE(BrokerComponentConfigValue, "monolith-ea2.example.com",   "assure1-ea2.example.com")   WHERE BrokerComponentConfigTypeID IN (8, 19);
UPDATE BrokerComponentConfig SET BrokerComponentConfigValue = REPLACE(BrokerComponentConfigValue, "monolith-col1.example.com",  "assure1-col1.example.com")  WHERE BrokerComponentConfigTypeID IN (8, 19);
UPDATE BrokerComponentConfig SET BrokerComponentConfigValue = REPLACE(BrokerComponentConfigValue, "monolith-col2.example.com",  "assure1-col2.example.com")  WHERE BrokerComponentConfigTypeID IN (8, 19);

    3. Remove "AuthVendor" configurations

      DELETE FROM AuthVendorSystems;
DELETE FROM AuthVendorUsers;

    4. Reset secondary replication

      RESET SLAVE;

    5. Stop the database

      systemctl stop monolith-db

  4. On the non-presentation Event and/or Metric servers only, enable and start the database to make changes.

    1. Enable and start the database

      Note

      Replace $A1BASEDIR with your v4 installation directory (e.g. /opt/monolith).

      cp $A1BASEDIR/distrib/startup/vendorMySQL/init.d/monolith-db.service /etc/systemd/system/monolith-db.service
systemctl enable monolith-db.service
systemctl daemon-reload
systemctl start monolith-db
$A1BASEDIR/vendor/mysql/bin/mysql -u monolith -p

    2. Reset secondary replication

      RESET SLAVE;

    3. Stop the database

      systemctl stop monolith-db

  5. On your new v5 presentation server, install Assure1 v5.

    1. Download and install the Assure1 v5 RPM. The SetupWizard will detect the system is being upgraded and perform additional steps as needed.

  6. Repeat installation for each new server (if applicable), choosing the correct roles.

    1. Download and install the Assure1 v5 RPM. The SetupWizard will detect the system is being upgraded and perform additional steps as needed.

    2. On all redundant presentation and Event/Metric/Graph database servers, run the Redundancy Wizard

      $A1BASEDIR/bin/redundancy/RedundancyWizard

    3. On all redundant Historical database servers, run the Analytics Wizard

      $A1BASEDIR/bin/historical/AnalyticsWizard

  7. Re-IP your fault collection server(s) (Syslog, Trap, etc.) to the previous IP address(es) and change associated DNS records, or reconfigure devices to send to the new IP address(es).

    1. If you re-IP your collection servers, you should additionally update the IP addresses of those devices from the Device Catalog -> Devices UI.

  8. Migrate legacy databases as needed.

    Note

    Depending on which servers these roles are installed on, you may need to manually copy the Monolith.conf from an existing Assure1 v4 server.

    1. Metric database migration

      1. From a server with the Core.Processing role, run the migration application.

        1. Basic, single threaded example:

          $A1BASEDIR/bin/core/processing/MetricDataMigration --V4Config /opt/monolith/etc/Monolith.conf --MetricsDBHost v4-metrics-db.example.com

        2. Advanced, threaded, and ranged examples:

          $A1BASEDIR/bin/core/processing/MetricDataMigration --V4Config /opt/monolith/etc/Monolith.conf --MetricsDBHost v4-metrics-db.example.com --Threads 4 --MetricIDs 1 10000
$A1BASEDIR/bin/core/processing/MetricDataMigration --V4Config /opt/monolith/etc/Monolith.conf --MetricsDBHost v4-metrics-db.example.com --Threads 4 --MetricIDs 10001 20000
$A1BASEDIR/bin/core/processing/MetricDataMigration --V4Config /opt/monolith/etc/Monolith.conf --MetricsDBHost v4-metrics-db.example.com --Threads 4 --MetricIDs 20001 30000
# continue as needed

    2. Graph database migration

      1. From a server with the Core.Processing role, run the migration application.

        1. Basic, single threaded example:

          $A1BASEDIR/bin/core/processing/GraphDataMigration --V4Config /opt/monolith/etc/Monolith.conf --HSEDBHost v4-monolith-db.example.com

        2. Advanced, threaded, and ranged examples:

          $A1BASEDIR/bin/core/processing/GraphDataMigration --V4Config /opt/monolith/etc/Monolith.conf --HSEDBHost v4-monolith-db.example.com --Threads 4 --LinkIDs 1 10000
$A1BASEDIR/bin/core/processing/GraphDataMigration --V4Config /opt/monolith/etc/Monolith.conf --HSEDBHost v4-monolith-db.example.com --Threads 4 --LinkIDs 10001 20000
$A1BASEDIR/bin/core/processing/GraphDataMigration --V4Config /opt/monolith/etc/Monolith.conf --HSEDBHost v4-monolith-db.example.com --Threads 4 --LinkIDs 20001 30000
# continue as needed

    3. Historical database migration

      1. From a server with the Historical.Processing role, run the migration application.

        1. Basic, single threaded example:

          $A1BASEDIR/bin/historical/HistoricalDataMigration --V4Config /opt/monolith/etc/Monolith.conf --HistoricalDBHost v4-historical-db.example.com --From 2019-01-01 --To 2019-12-31

        2. Advanced, threaded, and ranged examples:

          $A1BASEDIR/bin/historical/HistoricalDataMigration --V4Config /opt/monolith/etc/Monolith.conf --HistoricalDBHost v4-historical-db.example.com --Threads 4 --From 2019-01-01 --To 2019-01-31
$A1BASEDIR/bin/historical/HistoricalDataMigration --V4Config /opt/monolith/etc/Monolith.conf --HistoricalDBHost v4-historical-db.example.com --Threads 4 --From 2019-02-01 --To 2019-02-31
$A1BASEDIR/bin/historical/HistoricalDataMigration --V4Config /opt/monolith/etc/Monolith.conf --HistoricalDBHost v4-historical-db.example.com --Threads 4 --From 2019-03-01 --To 2019-03-31
# continue as needed

  9. Review and validate configurations, data migrations, and functionality of systems.

  10. Finalize upgrade by removing Assure1 v4 completely.

    1. On each system, run:

      $A1BASEDIR/bin/SetupWizard --Finalize-Upgrade

Uninstall Steps

If at any time an upgrade fails, you can run these commands to restore back to the Assure1 v4 installation.

Note

Package definitions may be updated in the Management database preventing re-running an upgrade. Contact support for further assistance at this point.

  1. Verify docker containers have been stopped and mount points removed:

    $A1BASEDIR/vendor/docker/bin/docker stop $($A1BASEDIR/vendor/docker/bin/docker ps -q)
for mount in $(mount | grep tmpfs | grep $A1BASEDIR/var/rke/var/lib | awk '{ print $3 }'); do umount $mount; done

  2. Stop all v5 processes:

    $A1BASEDIR/bin/BrokerControl --batch stopall
systemctl stop assure1-bus
systemctl stop assure1-web
systemctl stop assure1-db

  3. Look for any processes that were not stopped by the above:

    ps aux | grep assure1

    If any applications are listed (besides the "grep" process), run the following command with the process IDs to force the processes to stop:

    kill -9 (PROCESS ID)

  4. The following commands safely remove Assure1 v5 and delete the installation directory:

    rpm -e --noscripts Assure1
rm -rf $A1BASEDIR

  5. The following commands revert changes that were done to the user accounts for Assure1 v4:

    /usr/sbin/groupmod -n monolith assure1
/usr/sbin/usermod -l monolith -d /opt/monolith -g monolith assure1