Assure1 Graph Sink Microservice

Overview

The Assure1 Graph Sink microservice is part of the microservice topology pipeline. The Graph Sink is responsible for topology data that has been normalized. It and inserts the normalized topology data into the Neo4j database.

Prerequisites

1. A microservices cluster must be setup. Refer to Microservice Cluster Setup.

2. Apache Pulsar must be installed. Refer to Apache Pulsar microservice.

Setup

su - assure1
export NAMESPACE=a1-zone1-pri
export WEBFQDN=<Primary Presentation Web FQDN> 
a1helm install graph-sink assure1/graph-sink -n $NAMESPACE --set global.imageRegistry=$WEBFQDN

Default Configuration

Name	Value	Possible Values	Notes
LOG_LEVEL	INFO	FATAL, ERROR, WARN, INFO, DEBUG	Logging level used by application.
STREAM_INPUT	persistent://assure1/graph/sink	Text, 255 characters	Apache Pulsar topic path. Topic at end of path may be any text value.

Configurations can be changed by passing the values to the a1helm install prefixed with the configData parent key.

Example of setting the log level to DEBUG

a1helm install ... --set configData.LOG_LEVEL=DEBUG

Schema

Below is an example of JSON file formats with the accepted schema.

Single Vertex Schema

{
  "@timestamp": "2020-03-18T18:41:00.000Z",
  "_agents": [
    {
      "@timestamp": "2020-03-18T18:40:00.000Z",
      "host": "host.example.com",
      "id": "d5a346c8-d675-48a1-a42f-4a8db7c1552b",
      "app": "sample-integration",
      "version": "5.0.0",
      "input": "unknown://",
      "output": "pulsar+ssl://proxy.a1-messaging.svc.cluster.local/assure1/graph/sink"
    }
  ],
  "_domain": "topology",
  "_type": "vertex",
  "_version": "5.0.0",
  "vertex": {
    "@meta": false,
    "@method": "upsert",
    "@type": "Device",
    "keys": {
      "Name": "device-a.example.com",
      "ZoneID": 1
    },
    "properties": {
      "TimestampModified": "123456789000"
    }
  }
}

Edge Tuple Schema

{
  "@timestamp": "2021-03-30T15:32:21.690779663Z",
  "_agents": [
    {
      "@timestamp": "2021-03-30T15:32:21.690778749Z",
      "host": "pg-example.example.com",
      "id": "255c83bd-ff1e-2e10-6d92-dfecce7fba21",
      "app": "virtuora-connector",
      "version": "1.0.1",
      "input": "pg-example.example.com",
      "output": "persistent://assure1/graph/sink"
    }
  ],
  "_domain": "topology",
  "_type": "edgeTuple",
  "_version": "1.0.1",
  "edgeTuple": {
    "outVertex": {
      "@meta": false,
      "@method": "upsert",
      "@type": "Device",
      "keys": {
        "Name": "L100-107-TEST",
        "ZoneID": 1
      },
      "properties": {
        "TimestampModified": 1617118341
      }
    },
    "inVertex": {
      "@meta": false,
      "@method": "upsert",
      "@type": "Device",
      "keys": {
        "Name": "L100-107-TEST",
        "ZoneID": 0
      },
      "properties": {
        "TimestampModified": 1617118341
      }
    },
    "edge": {
      "@meta": false,
      "@method": "upsert",
      "@type": "ConnectsInterface",
      "keys": null,
      "properties": {
        "TimestampModified": 1617118341
      }
    }
  }
}

Deletion by single vertex example, type and at least one key is required:

{
  "@timestamp": "2020-03-18T18:41:00.000Z",
  "_agents": [
    {
      "@timestamp": "2020-03-18T18:40:00.000Z",
      "host": "host.example.com",
      "id": "d5a346c8-d675-48a1-a42f-4a8db7c1552b",
      "app": "sample-integration",
      "version": "5.0.0",
      "input": "unknown://",
      "output": "pulsar+ssl://proxy.a1-messaging.svc.cluster.local/assure1/graph/sink"
    }
  ],
  "_domain": "topology",
  "_type": "vertex",
  "_version": "5.0.0",
  "vertex": {
    "@meta": false,
    "@method": "delete",
    "@type": "Device",
    "keys": {
      "Name": "device-x.example.com",
      "ZoneID": 1
    }
  }
}

Deletion by single edge, type and at least one unique property is required:

{
  "@timestamp": "2020-03-18T18:41:00.000Z",
  "_agents": [
    {
      "@timestamp": "2020-03-18T18:40:00.000Z",
      "host": "host.example.com",
      "id": "d5a346c8-d675-48a1-a42f-4a8db7c1552b",
      "app": "sample-integration",
      "version": "5.0.0",
      "input": "unknown://",
      "output": "pulsar+ssl://proxy.a1-messaging.svc.cluster.local/assure1/graph/sink"
    }
  ],
  "_domain": "topology",
  "_type": "edge",
  "_version": "5.0.0",
  "edge": {
    "@meta": false,
    "@method": "delete",
    "@type": "HasInterface",
    "properties": {
      "connectionID": "1023"
    }
  }
}