2. OASE management

2.1. Introduction

This document aims to explain the OASE management function and how to use it.

2.3. Agent overview

This section explains the Exastro OASE Agent.

2.3.1. Agent

The Agent is independent from Exastro IT Automation (hereinafter written as ITA) and functions as a middleman between OASE and external services.
The Agent collects Event collection settings for external services from ITA and uses them to gather events from the external services. After that, the Agent will send the events to ITA.
OASE Agent overview

2.4. How to use the Agent

This section explains how to use the Exastro OASE Agent.

2.4.1. Workflow

A standard OASE workflow can be seen below (including using the Agent).
See the following sections for more detailed information regarding each of the steps.
OASE workflow

  • Workflow details and references

    1. Configure Event collection settings
      In the OASE management's Event collection menu, register settings related to the Event collection target services.
    2. Configure labels
      In the OASE Label creation/Labeling menu, configure settings for labeling events OASE.
      For more information, see Create label and Labeling.
    3. Install and start the Agent
      Start the Agent and collect Events.
      For more information, see Installation guide (Docker Compose).
      For more information, see Installation guide (Kubernetes).

2.5. Notification template (common) overview

The following figure illustrates the OASE notification function.
Notification template (common) overview

図 2.72 Notification template (common) overview

2.6. How to use Notification template (common)

A standard workflow containing operations needed to use the OASE Notification function can be seen below.
See the following sections for more detailed information regarding each of the steps.
Notification template (common) workflow

図 2.73 Notification template (common) workflow

  • Workflow details and references

    1. Maintaining (View/Edit)Notification template (common)
      From the OASE Management's Notification template (common) menu, users can maintain (view/edit) templates used by the OASE Notification function.
      For more information, see Notification template (common).
    2. Registering Notification settings
      Log in to the Exastro system as the Organization administrator and register notification settings from the Notification management menu.
      For more information, see Registering Notification destination settings.
    3. Configure Mail server(only for cases where the notification destination is set to a mail address).
      Log in to the Exastro system as the Organization administrator and register mail server settings from the Mail server settings menu.
      For more information, see Mail server settings.

注釈

The Notification template (common) can be used without changing it.

2.7. Function menu operation

This section explains how to operate the OASE Management function menus.

2.7.2. Event collection

  1. In OASE management ▶ Event collection, users can maintain (view/register/edit/discard) TIL, connection methods, authentication methods, event collection targets, etc.
Submenu (Event collection)

図 2.74 Submenu (Event collection)

  1. The Event collection※1 page's input items are as following.
※1 Configurations for Event collection targets
※2 The required combinations for Connection methods, Authentication information and Request methods are as following.

Connection method

Request method

Authenticatio information

IMAP password authentication

・IMAP: Plaintext
・Username
・Password

Bearer authentication
・GET
・POST

・Authentication token

Password authentication
・GET
・POST
・Username
・Password

Free authentication
・GET
・POST

・Input to Parameter
※3 For more information regarding Response keys, Response list flags and Event ID keys, see Response key and Event ID keyキー.

警告

If the collection destination is a mail, there might be cases where the collection events might be saved without characters that could not be decoded depending on the charachter code type.
For more information, see Agent decoding process.

2.7.3. Notification template (common)

  1. In the OASE management ▶ Notification template (common) menu, the user can maintain (view/update) template used by the OASE notification function.
Submenu (Notification template (common) )

図 2.75 Submenu (Notification template (common) )

  1. The items found in the Notification template (common) page are as following.

Item

Description

Input required

Input method

Restrictions

Template
Allows the user to edit the Notification template. The types are the following 4.
・New.j2
・Known(evaluated).j2
・Known(timeout).j2
・Undetected.j2

〇

Manual

Maximum size 2MB

Remarks

Free description field.

ー

Manual

Maximum length 4000 bytes
The template's default values are as following.
リスト 2.49 New.j2
 1 [TITLE]
 2 A new event has occured.
 3
 4 [BODY]
 5 Detailed information
 6 　Event ID                 : {{ _id }}
 7 　Event collection settings: {{ labels._exastro_event_collection_settings_id }}
 8 　Event fetch time         : {{ labels._exastro_fetched_time }}
 9 　Event end time           : {{ labels._exastro_end_time }}
10 　Event type               : {{ labels._exastro_type }}
11
12 　Re-evaluation
13 　　Evaluation rule name   : {{ labels._exastro_rule_name }}
14 　　Event                  : {{ exastro_events }}
15
16 　Label:
17 　  {% for key, value in labels.items() %}
18 　　・{{ key }}: {{ value }}
19 　　{% endfor %}
リスト 2.50 Known(evaluated).j2
 1 [TITLE]
 2 A known(evaluated) event has occured.
 3
 4 [BODY]
 5 Detailed information
 6 　Event ID                 : {{ _id }}
 7 　Event collection settings: {{ labels._exastro_event_collection_settings_id }}
 8 　Event fetch time         : {{ labels._exastro_fetched_time }}
 9 　Event end time           : {{ labels._exastro_end_time }}
10 　Event type               : {{ labels._exastro_type }}
11
12 　Re-evaluation
13 　　Evaluation rule name   : {{ labels._exastro_rule_name }}
14 　　Event                  : {{ exastro_events }}
15
16 　Label:
17 　  {% for key, value in labels.items() %}
18 　　・{{ key }}: {{ value }}
19 　　{% endfor %}
リスト 2.51 Known(timeout).j2
 1 [TITLE]
 2 A known(timeout) event has occured.
 3
 4 [BODY]
 5 Detailed information
 6 　Event ID                 : {{ _id }}
 7 　Event collection settings: {{ labels._exastro_event_collection_settings_id }}
 8 　Event fetch time         : {{ labels._exastro_fetched_time }}
 9 　Event end time           : {{ labels._exastro_end_time }}
10 　Event type               : {{ labels._exastro_type }}
11
12 　Re-evaluation
13 　　Evaluation rule name   : {{ labels._exastro_rule_name }}
14 　　Event                  : {{ exastro_events }}
15
16 　Label:
17 　  {% for key, value in labels.items() %}
18 　　・{{ key }}: {{ value }}
19 　　{% endfor %}
リスト 2.52 Undetected.j2
 1 [TITLE]
 2 An unknown event has occured.
 3
 4 [BODY]
 5 Detailed information
 6 　Event ID                 : {{ _id }}
 7 　Event collection settings: {{ labels._exastro_event_collection_settings_id }}
 8 　Event fetch time         : {{ labels._exastro_fetched_time }}
 9 　Event end time           : {{ labels._exastro_end_time }}
10 　Event type               : {{ labels._exastro_type }}
11
12 　Re-evaluation
13 　　Evaluation rule name   : {{ labels._exastro_rule_name }}
14 　　Event                  : {{ exastro_events }}
15
16 　Label:
17 　  {% for key, value in labels.items() %}
18 　　・{{ key }}: {{ value }}
19 　　{% endfor %}

2.8. Appendix

2.8.1. OASE Agent operation flow and ".env" setting Values

This section explains the following.
- OASE Agent's operation flow
- Parts of the .env file's setting values when OASE Agent is installed.
OASE Agent flow diagram

図 2.76 OASE Agent flow diagram

Parameter

Description

AGENT_NAME

Used as the database file name and the name of the OASE Agent.

EXASTRO_URL

Used as the request destination when API requests are sent to ITA.

EXASTRO_ORGANIZATION_ID

Used to distinguish Organizations when API requests are sent to ITA.

EXASTRO_WORKSPACE_ID
Used to distinguish Workspaces when API requests are sent to ITA.
The specified workspace must be linked to the organization specified to EXASTRO_ORGANIZATION_ID.

EXASTRO_REFRESH_TOKEN
Used as a authentication token for Bearer authentication when API requesting to ITA.
※The user's role must be able to edit the OASE - Event - Event history menu.

EXASTRO_USERNAME
Used as the Basic authentication username when API requests are sent to ITA.
※The user's role must be able to edit the OASE - Event - Event history menu.
※If not using EXASTRO_REFRESH_TOKEN (Not recommended)

EXASTRO_PASSWORD
Used as the Basic authentication password when API requests are sent to ITA.
※If not using EXASTRO_REFRESH_TOKEN (Not recommended)

EVENT_COLLECTION_SETTINGS_NAMES

The values set to this parameter fetches event collection settings from ITA and generates a setting file.

ITERATION

The number of how many the processes depicted with green arrows in the figure above will loop.

EXECUTE_INTERVAL

The interval time for the processes depicted with green arrows in the figure above.

2.8.2. Instant event collection settings display

This section explains how to instantly display the event collection settings when they are changed.
  1. Start the OASE Agent bash shell.
    docker exec -it <OASE Agent container name> bash
  2. Delete the setting file "event_collection_settings.json" within /tmp.
    rm /tmp/event_collection_settings.json

Tip

If the OASE Agent has no event_collection_settings.json setting file, fetch the Event collection settings from ITA and create a setting file.
The user can instantly have the settings take affect by deleting the settings file.
※If the users are not willing to delete the settings file, the user will have to wait until the proccess loops the amount of times specified by "ITERATION" from the previous chapter.

2.8.3. Agent decoding process

When Collection destination is set to Mail, information regarding the Agent's decoding proccess for the Collected events can be seen below.

Confirmed supported character codes

表 2.73 Confirmed supported character codes

Transmission method

Mail Header

Frormat

Language

Content-Transfer-Encoding

Content-Type

plaintext

English

7bit

text/plain; charset=US-ASCII

plaintext

Japanese

8bit

text/plain; charset=UTF-8

plaintext

English

8bit

text/plain; charset="ANSI_X3.4-1968"

html

English

・plaintext: 7bit

・html: quoted-printable

multipart/alternative

・plaintext: text/plain; charset=US-ASCII

・html: text/html; charset=UTF-8

html

Japanese

・plaintext: 8bit

・html: quoted-printable

multipart/alternative

・plaintext: text/plain; charset=UTF-8

・html: text/html; charset=UTF-8
The following is an example decpicts saved collection events with undecodable characters excluded.
Sending mail
Format：plaintext
Content-Transfer-Encoding：8bit
Content-Type：text/plain; charset="ANSI_X3.4-1968"
Subject：Failure
Contents：Failure：Server01\r\n
When seen on ITA
Subject：Failure
Contents：Server01\r\n
Collecting events without undecodable characters example/results (Event flow)

図 2.77 Collecting events without undecodable characters example/results (Event flow)

2.8.4. Response key and Event ID key

This section explains the following.
・ Response key
・ JMESPath
・ Response list flag
・ Event ID key

Response key

Response keys are keys that filters events from the response payload.

注釈

・Monitoring softwares have functions to fetch alerts and metrics (statuses) from the monitored machine through HTTP API,
・The Agent uses the HTTP API to fetch the alerts and metrics.
・However, the response payloads returned by the monitoring softwares can only be used by the Agent if they are in JSON format.
Items that can be specified as Response key are:
　・Response payloads that stores items handled as Events in child elements.
　・Child elements stored in array if there are multiple items that are handled as Events.
Response key is specified in JMESPath format.
For more information regarding JMESPath format, see the next section.

JMESPath

JMESPath is a query language for JSON targets.
Values corresponding to the specified JMESPath are filtered from the JSON.
The path is specified in a format where the JSON key is combined with periods (".").
If the JSON key values are in an array, put "[]" at the end of the JSON key.
　Example) The Parent value is an array and the array's child element key, children, is filtered.
parent[].children
　Specify the line above.
The following example uses the Sample response from the following URL to explain specifying JMESPaths.
Azure RESET-API Get Metric for data <https://learn.microsoft.com/ja-jp/rest/api/monitor/metrics/list?view=rest-monitor-2023-10-01&tabs=HTTP>.
 1{
 2  "cost": 598,
 3  "timespan": "2021-04-20T09:00:00Z/2021-04-20T14:00:00Z",
 4  "interval": "PT1H",
 5  "value": [
 6    {
 7      "id": "/subscriptions/1f3･･･c38/･･･/metrics/BlobCount",
 8      "type": "Microsoft.Insights/metrics",
 9      "name": {
10        "value": "BlobCount",
11        "localizedValue": "Blob Count"
12      },
13      "displayDescription": "The number of blob objects stored in the storage account.",
14      "unit": "Count",
15      "timeseries": [
16        {
17          "metadatavalues": [
18            {
19              "name": {
20                "value": "tier",
21                "localizedValue": "tier"
22              },
23              "value": "Hot"
24            }
25          ]
26        },
27        {
28          "metadatavalues": [
29            {
30              "name": {
31                "value": "tier",
32                "localizedValue": "tier"
33              },
34              "value": "Standard"
35            }
36          ]
37        },
38        {
39          "metadatavalues": [
40            {
41              "name": {
42                "value": "tier",
43                "localizedValue": "tier"
44              },
45              "value": "Cool"
46            }
47          ]
48        }
49      ],
50      "errorCode": "Success"
51    }
52  ],
53  "namespace": "microsoft.storage/storageaccounts/blobservices",
54  "resourceregion": "westus2"
55}

注釈

For more information regarding JMESPath, see the JMESPath Tutorial https://JMESPath.org/tutorial.html.
We also recommend that the user tries out JMESPath Try it Out! https://JMESPath.org/ where they can try out the JSON above.
  1. JMESPath fetching array values
For the JSON above, specifying the following for the JMESPath fetching the :program:`value`(array) from the 5th line
value
will fetch the following results.
(The line "//" : "･･･Sample Response line 14 ～line 49 shortened･･･" is an comment and have no effect on the results.)
[
  {
    "id": "/subscriptions/1f3･･･c38/･･･/metrics/BlobCount",
    "type": "Microsoft.Insights/metrics",
    "name": {
      "value": "BlobCount",
      "localizedValue": "Blob Count"
    },
    "displayDescription": "The number of blob objects stored in the storage account.",
    "//" : "･･･Sample Response line 14 ～line 49 shortened･･･",
    "errorCode": "Success"
  }
]
  1. JMESPath fetching objects
For the JSON above, specifying the following for the JMESPath fetching the value (array)'s name value
value[].name
will fetch the following results.
[
  {
    "value": "BlobCount",
    "localizedValue": "Blob Count"
  }
]
  1. JMESPath fetching multiple values from deep hierarchies
Specifying the following for the JMESPath fetching the value (array)'s timeseries (array)'s metadatavalues (array)'s' name
value[].timeseries[].metadatavalues[].name
will fetch the following results.
(The line "//" : " lines xx metadatavalues.name" is an comment and have no effect on the results.)
[
  "//" : "Line 19 metadatavalues.name"
  {
    "value": "tier",
    "localizedValue": "tier"
  },
  "//" : "Line 30 metadatavalues.name"
  {
    "value": "tier",
    "localizedValue": "tier"
  },
  "//" : "Line 41 metadatavalues.name"
  {
    "value": "tier",
    "localizedValue": "tier"
  }
]

Response list flag

Response list flag specifies whether the events filtered by the Response key are stored in arrays or not.
・True ： For when the events filtered by the Response key are stored in arrays
　　　　　　For cases like No 1. where JMESPath is used to fetch array values or for cases like No 3. where it is used to fetch multiple values from deep hierarchies.
・False ： For when the events filtered by the Response key are not stored in arrays (but in values and/or objects with child components
　　　　　　For cases like No 2. where the JMESPath is used to fetch objects.

Event ID key

Inventory ID key specifies keys used to fetch values that distinguishes events filtered by the Response key.
The Event ID key specifies a key existing in the results filtered by Response key.
Event ID key is also specified in JMESPath format.
If the Response key and Response list flag has the following specified for the Azure RESET-API's JSON above.

Item name

Setting value

Response key

value

Response list flag

True
The values filtered by Response key are as following
 1[
 2  {
 3    "id": "/subscriptions/1f3･･･c38/･･･/metrics/BlobCount",
 4    "type": "Microsoft.Insights/metrics",
 5    "name": {
 6      "value": "BlobCount",
 7      "localizedValue": "Blob Count"
 8    },
 9    "displayDescription": "The number of blob objects stored in the storage account.",
10    "unit": "Count",
11    "timeseries": [
12      {
13        "metadatavalues": [
14          {
15            "name": {
16              "value": "tier",
17              "localizedValue": "tier"
18            },
19            "value": "Hot"
20          }
21        ]
22      },
23      "//": "･･･ (metadatavalues are repeated and shortened) ･･･"
24    ],
25    "errorCode": "Success"
26  }
27]
From these results, the Event type values will have id values applied,
meaning that the OASE management ▶ Event collection setting values should be as following.

Item name

Setting value

Response key

value

Response list flag

True

Event ID key

id

警告

If Response key has a key specified to it that does not exist in one of the filtered values, the value will fetch blank values and will not function properly.
If a key that does not exist in the filtered values is specified to the Response key above, the system will not function properly.
表 2.74 Invalid Event ID key settings

Item name

Setting value

Response key

value

Response lsit flag

True

Event ID key

value[].id

2.8.5. Agent examples for the different monitoring softwares

This section contains setting examples in OASE management ▶ Event collection for using Zabbix and Grafana.
This section will first display examples using cURL commands to fetch alerts from the softwares.
Then, it will explain the order of configuring cURL parameters to OASE management ▶ Event collection.

警告

The HTTP API might differ depending on the monitoring software version.
Make sure to check how the HTTP API works for the version the user is using before configuring the settings in OASE management ▶ Event collection.

Zabbix

This section contains an example for configuring OASE management ▶ Event collection to fetch events from Zabbix.
The Zabbix used in the following section is running the following version.
・zabbix 6.4.12
  1. Fetch events from Zabbix with cURL command
The following example is an cURL command to fetch Faults(problems) from Zabbix.
curl --request POST \
--url http://<Zabbix IP Address or Domain>/zabbix/api_jsonrpc.php \
--header 'content-type: application/json-rpc' \
--data "{\"jsonrpc\": \"2.0\",\"method\": \"problem.get\",\"id\": 1,\"params\": {},\"auth\": \"<Zabbix API token>"}"
(More information regarding commands and <Zabbix API token> within parameters are written later in this section.)
The cURL command above returns a response similar to the one below.
 1{
 2   "jsonrpc": "2.0",
 3   "result": [
 4       {
 5           "eventid": "89",
 6           "source": "0",
 7           "object": "0",
 8           "objectid": "16046",
 9           "clock": "1709636653",
10           "ns": "906955445",
11           "r_eventid": "0",
12           "r_clock": "0",
13           "r_ns": "0",
14           "correlationid": "0",
15           "userid": "0",
16           "name": "High CPU utilization (over 90% for 5m)",
17           "acknowledged": "0",
18           "severity": "2",
19           "opdata": "Current utilization: 100 %",
20           "suppressed": "0",
21           "urls": []
22       }
23   ],
24   "id": 1
25}

注釈

For more information regarding Zabbix faults (problems), see the following.
https://www.zabbix.com/documentation/current/jp/manual/api/reference/problem/get
  1. Agent settings example for fetching events from Zabbix.
The following settings configures the OASE management ▶ Event collection values to fetch values similar to the cURL command listed above.
表 2.75 Zabbix setting value example

Item name

Setting value

Event collection name

<Name for fetching Zabbix faults>

Connection method

Free authentication

Requestion method

POST

Connection destination
http://<Zabbix IP Address or Domain>/zabbix/api_jsonrpc.php

Request header

 
{ "content-type" : "application/json-rpc" }

Parameter

 
{
  "jsonrpc":"2.0",
  "method":"problem.get",
  "id":1,
  "params":{},
  "auth":"<Zabbix API token>"
}

Response key

result

REsponse list flag

True

Event ID key

eventid

注釈

The <Zabbix API token> configured to Parameter is Zabbix user authentication information. The following command can be used to fetch user information.
Note that running the command below might cause already created <Zabbix API token> not being usable by existing users.
We recommend creating a new user and creating a new <Zabbix API token> with said user.
Create a new user
  1. Sign in to Zabbix as an admin through the browser.
    　The default admin login inforamtion are as following:
    　　・Username： Admin
    　　・Password： zabbix
  2. Select a user Side menu > User
  3. Click Create user at the top right part of the user page.
  4. Input username and password
  5. Click the Permission tab
  6. Go to the user role and click Select select.
    Select a "User role" or a role that has more API access.
  7. Input other required items
  8. Click Add.
Follow the following steps to create a user.
The following example is a command for creating an <API token>.
The Zabbix login information for the example below are as following.
　・Username: The user's <Username> from the previous operation.
　・Password: The user's <Password> from the previous operation.
curl --request POST \
--url ttp://<Zabbix IP Address or Domain>/zabbix/api_jsonrpc.php \
--header "Content-Type: application/json-rpc" \
--data "{\"auth\":null,\"method\":\"user.login\",\"id\":1,\"params\":{\"user\":\"<Username>\",\"password\":\"<Password>\"},\"jsonrpc\":\"2.0\"}" \
The cURL command above returns a response similar to the one below.
1{
2    "jsonrpc": "2.0",
3     "result": "<Zabbix API token>",
4     "id": 1
5}
Paste the <Zabbix API token> to the OASE management ▶ Event collection's Parameter's <Zabbix API token>.
※ <Zabbix API token> can be created from browsers.
After logging in through the browser, access Side menu > User settings > API token and create the token by pressing Create API token.

Grafan

This section contains an example for configuring OASE management ▶ Event collection to fetch events from Grafan.
The Grafan used in the following section is running the following version.
・Grafan 10.3
・Use Prometheus 2.49 for data source
  1. Fetch events from Grafan with cURL command.
The following example is an cURL command to fetch alerts from Grafan.
curl --request GET \
--url 'http://<GrafanaのIP addres か Domain>:3000/api/prometheus/grafana/api/v1/alerts' \
--header 'authorization: Bearer <Authentication token>' \
--header 'Content-Type: application/json'
(More information regarding commands and <Authentication token> within parameters are written later in this section.)
The cURL command above returns a response similar to the one below.
 1{
 2    "data": {
 3        "alerts": [
 4            {
 5                "activeAt": "2018-07-04T20:27:12.60602144+02:00",
 6                "annotations": {},
 7                "labels": {
 8                    "alertname": "my-alert"
 9                },
10                "state": "firing",
11                "value": "1e+00"
12            }
13        ]
14    },
15    "status": "success"
16}

注釈

For more information regarding Grafana alerts, see the following URL.
https://prometheus.io/docs/prometheus/latest/querying/api/#alerts
  1. Event collection settings example for fetching events from Grafana.
The following settings configures the OASE management ▶ Event collection values to fetch values similar to the cURL command listed above.
表 2.76 Grafana setting example

Item name

Setting value

Event collection name

<Name for fetching Grafana alerts>

Connection method

Bearer authentication

Request method

GET

Connection destination
http://<Grafana IP addres or Domain>:3000/api/prometheus/grafana/api/v1/rules

Request header

 
{ "Content-Type": "application/json" }

Authentication token

<Authentication token>

Response key

data.alerts

Response list flag

True

Event ID key

activeAt

注釈

<Authentication token> is Grafana authentication information.
It can be fetched by following the steps below.
  1. Log in to Grafan through the browser.
    　The default values are as following:
    　・username: admin
    　・Password: admin (Might have been changed if someone already has logged in)
  2. Select Home > Administration > Service accounts
  3. Click service account and create a service account.
  4. Input a name for the service name (uncapitalized alphanumeric characters only).
  5. Click Add service account token and create an Authentication token.
  6. For expiration,
    　Select either "No expiration" (no period　※recommended) or
    　"Set expiration" (with period).
  7. Click Generate token.
  8. Click Copy to clipboard and close
    and paste the Authentication token to the clipboard.
  9. Paste the Authentication token from the clipboard to OASE management ▶ Event collection's Authentication token.