5. Data backup/restore

5.1. Goal

This document aims to describe how to backup and restore persistent data used by the Exastro system.
If the user has access to databackup options from a public cloud, they can use that as well.

5.2. Pre-requisites

The backup/restore process described in this document requires that the following conditions are met,

5.2.1. Conditions

  • The user must be able to run the following commands.

    • tar

    • kubectl

  • The environment server must have enough disk space.

5.2.2. Encrypt key backup

Exastro システムのパスワードや認証情報といった機密情報はすべて暗号化されています。
必ず、下記で取得した暗号化キーをバックアップして、適切に保管してください。

危険

暗号化キーを紛失した場合、バックアップデータからシステムを復旧した際にデータの復号ができなくなります。
リスト 5.52 コマンド
# Exastro IT Automation ENCRYPT_KEY
kubectl get secret ita-secret-ita-global --namespace exastro -o jsonpath='{.data.ENCRYPT_KEY}' | base64 -d
リスト 5.53 出力結果
JnIoXzJtPic2MXFqRl1yI1chMj8hWzQrNypmVn41Pk8=
リスト 5.54 コマンド
# Exastro Platform ENCRYPT_KEY
kubectl get secret platform-secret-pf-global --namespace exastro -o jsonpath='{.data.ENCRYPT_KEY}' | base64 -d
リスト 5.55 出力結果
bHFZe2VEVVM2PmFeQDMqNG4oZT4lTlglLjJJekxBTHE=

5.3. Overview

The following data can be Backuped/Restored.
  • Backup/Restore target

    • Exastro Platform database

    • Exastro IT Automation database

    • Exastro IT Automation shared files

The user will use the kubectl command to backup/restore said data.
The process will pause the reverse proxy in order to prevent the user from writing data.
After that, the backyard process will be paused before the data will be backed up.
Lastly, the Pod number will be returned to the numbers between the backup process started.
The restore process will be the undergo the same process.

危険

This procedure will stop the service.

5.4. Backup

5.4.1. Stop service

  1. Pod 起動数の確認

    作業前の Pod 起動数の確認をし、状態を記録します。
    リスト 5.56 コマンド
    RS_AE=`kubectl get deploy ita-by-ansible-execute -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_ALRV=`kubectl get deploy ita-by-ansible-legacy-role-vars-listup -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_ATS=`kubectl get deploy ita-by-ansible-towermaster-sync -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_CS=`kubectl get deploy ita-by-conductor-synchronize -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_MC=`kubectl get deploy ita-by-menu-create -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_PA=`kubectl get deploy platform-auth -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
    以下はITA2.1.0以降の場合に実行してください。
    RS_ALV=`kubectl get deploy ita-by-ansible-legacy-vars-listup -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_APV=`kubectl get deploy ita-by-ansible-pioneer-vars-listup -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_CFI=`kubectl get deploy ita-by-cicd-for-iac -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_CR=`kubectl get deploy ita-by-conductor-regularly -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_COL=`kubectl get deploy ita-by-collector -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_EEI=`kubectl get deploy ita-by-excel-export-import -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_HS=`kubectl get deploy ita-by-hostgroup-split -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_MEI=`kubectl get deploy ita-by-menu-export-import -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_TCE=`kubectl get deploy ita-by-terraform-cli-execute -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_TCV=`kubectl get deploy ita-by-terraform-cli-vars-listup -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_TCEE=`kubectl get deploy ita-by-terraform-cloud-ep-execute -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_TCEV=`kubectl get deploy ita-by-terraform-cloud-ep-vars-listup -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
    以下はITA2.3.0以降の場合に実行してください。
    RS_OC=`kubectl get deploy ita-by-oase-conclusion -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`

  2. リバースプロキシの停止

    リバースプロキシ (platform-auth) の Pod 起動数を 0 に変更し、エンドユーザーからのアクセスを制限します。
    リスト 5.57 コマンド
    kubectl scale deployment platform-auth --namespace exastro --replicas=0

  3. バックヤード処理の停止

    バックヤード処理 (ita-by-***) の Pod 起動数を 0 に変更し、データベースの更新を停止します。
    リスト 5.58 コマンド
    kubectl scale deployment ita-by-ansible-execute --namespace exastro --replicas=0
kubectl scale deployment ita-by-ansible-legacy-role-vars-listup --namespace exastro --replicas=0
kubectl scale deployment ita-by-ansible-towermaster-sync --namespace exastro --replicas=0
kubectl scale deployment ita-by-conductor-synchronize --namespace exastro --replicas=0
kubectl scale deployment ita-by-menu-create --namespace exastro --replicas=0
    以下はITA2.1.0以降の場合に実行してください。
    kubectl scale deployment ita-by-ansible-legacy-vars-listup --namespace exastro --replicas=0
kubectl scale deployment ita-by-ansible-pioneer-vars-listup --namespace exastro --replicas=0
kubectl scale deployment ita-by-cicd-for-iac --namespace exastro --replicas=0
kubectl scale deployment ita-by-collector --namespace exastro --replicas=0
kubectl scale deployment ita-by-conductor-regularly --namespace exastro --replicas=0
kubectl scale deployment ita-by-excel-export-import --namespace exastro --replicas=0
kubectl scale deployment ita-by-hostgroup-split --namespace exastro --replicas=0
kubectl scale deployment ita-by-menu-export-import --namespace exastro --replicas=0
kubectl scale deployment ita-by-terraform-cli-execute --namespace exastro --replicas=0
kubectl scale deployment ita-by-terraform-cli-vars-listup --namespace exastro --replicas=0
kubectl scale deployment ita-by-terraform-cloud-ep-execute --namespace exastro --replicas=0
kubectl scale deployment ita-by-terraform-cloud-ep-vars-listup --namespace exastro --replicas=0
    以下はITA2.3.0以降の場合に実行してください。
    kubectl scale deployment ita-by-oase-conclusion --namespace exastro --replicas=0

  4. Pod 起動数の確認

    上記で停止した対象の Pod 数が 0 になっていることを確認
    リスト 5.59 コマンド
    kubectl get deployment --namespace exastro
    リスト 5.60 実行結果
    NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
mariadb                                  1/1     1            1           3h41m
ita-web-server                           1/1     1            1           3h41m
platform-web                             1/1     1            1           3h41m
ita-api-admin                            1/1     1            1           3h41m
ita-api-organization                     1/1     1            1           3h41m
platform-api                             1/1     1            1           3h41m
keycloak                                 1/1     1            1           3h41m
ita-by-menu-create                       0/0     0            0           3h41m
ita-by-ansible-execute                   0/0     0            0           3h41m
ita-by-ansible-legacy-role-vars-listup   0/0     0            0           3h41m
ita-by-ansible-towermaster-sync          0/0     0            0           3h41m
ita-by-conductor-synchronize             0/0     0            0           3h41m
platform-auth                            0/0     0            0           3h41m
    以下はITA2.1.0以降の場合に確認してください。
    NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
ita-by-ansible-legacy-vars-listup        0/0     0            0           3h41m
ita-by-ansible-pioneer-vars-listup       0/0     0            0           3h41m
ita-by-cicd-for-iac                      0/0     0            0           3h41m
ita-by-collector                         0/0     0            0           3h41m
ita-by-conductor-regularly               0/0     0            0           3h41m
ita-by-excel-export-import               0/0     0            0           3h41m
ita-by-hostgroup-split                   0/0     0            0           3h41m
ita-by-menu-export-import                0/0     0            0           3h41m
ita-by-terraform-cli-execute             0/0     0            0           3h41m
ita-by-terraform-cli-vars-listup         0/0     0            0           3h41m
ita-by-terraform-cloud-ep-execute        0/0     0            0           3h41m
ita-by-terraform-cloud-ep-vars-listup    0/0     0            0           3h41m
    以下はITA2.3.0以降の場合に確認してください。
    NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
ita-api-oase-receiver                    1/1     1            1           3h41m
ita-by-oase-conclusion                   0/0     0            0           3h41m

5.4.2. Backup

The procedure depends on if the Exastro Platform and Exastro IT Automation shares database or if they are divided.

  1. Creating container for maintenance

    Create a container for backups.
    リスト 5.61 Command
     1cat <<_EOF_ | kubectl apply -f - --namespace exastro
 2apiVersion: v1
 3kind: Pod
 4metadata:
 5  name: exastro-maintenance
 6  namespace: exastro
 7spec:
 8  containers:
 9  - command:
10    - sh
11    - -c
12    args:
13    - |
14      #!/bin/bash
15      sleep 3600
16    env:
17    - name: PF_DB_DATABASE
18      valueFrom:
19        configMapKeyRef:
20          key: DB_DATABASE
21          name: platform-params-pf-database
22    - name: PF_DB_HOST
23      valueFrom:
24        configMapKeyRef:
25          key: DB_HOST
26          name: platform-params-pf-database
27    - name: PF_DB_PORT
28      valueFrom:
29        configMapKeyRef:
30          key: DB_PORT
31          name: platform-params-pf-database
32    - name: PF_DB_ADMIN_PASSWORD
33      valueFrom:
34        secretKeyRef:
35          key: DB_ADMIN_PASSWORD
36          name: platform-secret-pf-database
37    - name: PF_DB_ADMIN_USER
38      valueFrom:
39        secretKeyRef:
40          key: DB_ADMIN_USER
41          name: platform-secret-pf-database
42    - name: ITA_DB_DATABASE
43      valueFrom:
44        configMapKeyRef:
45          key: DB_DATABASE
46          name: ita-params-ita-database
47    - name: ITA_DB_HOST
48      valueFrom:
49        configMapKeyRef:
50          key: DB_HOST
51          name: ita-params-ita-database
52    - name: ITA_DB_PORT
53      valueFrom:
54        configMapKeyRef:
55          key: DB_PORT
56          name: ita-params-ita-database
57    - name: ITA_STORAGEPATH
58      valueFrom:
59        configMapKeyRef:
60          key: STORAGEPATH
61          name: ita-params-ita-global
62    - name: ITA_DB_ADMIN_PASSWORD
63      valueFrom:
64        secretKeyRef:
65          key: DB_ADMIN_PASSWORD
66          name: ita-secret-ita-database
67    - name: ITA_DB_ADMIN_USER
68      valueFrom:
69        secretKeyRef:
70          key: DB_ADMIN_USER
71          name: ita-secret-ita-database
72    image: mariadb:10.9
73    imagePullPolicy: IfNotPresent
74    name: exastro-maintenance
75    resources: {}
76    securityContext:
77      allowPrivilegeEscalation: false
78      readOnlyRootFilesystem: false
79      runAsGroup: 1000
80      runAsNonRoot: true
81      runAsUser: 1000
82    volumeMounts:
83    - mountPath: /storage
84      name: volume-ita-backup-storage
85  volumes:
86  - name: volume-ita-backup-storage
87    persistentVolumeClaim:
88      claimName: pvc-ita-global
89  restartPolicy: Always
90  securityContext: {}
91  serviceAccount: default
92  serviceAccountName: default
93_EOF_

  2. Acquiring Database backup

    Acquire the mysqldump backup to the Database.
    The process differs depending on if the Exastro Platform and Exastro IT Automation database server is shared or divided.
    リスト 5.62 Database backup command for Exastro platform application
    kubectl exec -it exastro-maintenance --namespace exastro -- sh -c 'mysqldump -h ${PF_DB_HOST} -P ${PF_DB_PORT} -u ${PF_DB_ADMIN_USER} -p${PF_DB_ADMIN_PASSWORD} --all-databases --add-drop-table' | gzip > exastro_mysqldump_platform_db_`date +"%Y%m%d-%H%M%S"`.sql.gz
    リスト 5.63 Database backup command for Exastro platform user
    kubectl exec -it exastro-maintenance --namespace exastro -- sh -c 'mysqldump -h ${PF_DB_HOST} -P ${PF_DB_PORT} -u ${PF_DB_ADMIN_USER} -p${PF_DB_ADMIN_PASSWORD} --allow-keywords mysql' | gzip > exastro_mysqldump_platform_user_`date +"%Y%m%d-%H%M%S"`.sql.gz
    リスト 5.64 Database backup command for Exastro IT Automation application
    kubectl exec -it exastro-maintenance --namespace exastro -- sh -c 'mysqldump -h ${ITA_DB_HOST} -P ${ITA_DB_PORT} -u ${ITA_DB_ADMIN_USER} -p${ITA_DB_ADMIN_PASSWORD} --all-databases --add-drop-table' | gzip > exastro_mysqldump_ita_db_`date +"%Y%m%d-%H%M%S"`.sql.gz
    リスト 5.65 Database backup command for Exastro IT Automation user
    kubectl exec -it exastro-maintenance --namespace exastro -- sh -c 'mysqldump -h ${ITA_DB_HOST} -P ${ITA_DB_PORT} -u ${ITA_DB_ADMIN_USER} -p${ITA_DB_ADMIN_PASSWORD} --allow-keywords mysql' | gzip > exastro_mysqldump_ita_user_`date +"%Y%m%d-%H%M%S"`.sql.gz

  3. Acquire backup files

    Acquire Exastro IT Automation's file backup.
    リスト 5.66 command
    kubectl exec -i exastro-maintenance --namespace exastro -- sh -c 'tar zcvf - ${ITA_STORAGEPATH}' > exastro_storage_backup_ita_`date +"%Y%m%d-%H%M%S"`.tar.gz

  4. Delete container for maintenance

    Create container for Backup operations
    リスト 5.67 Command
    kubectl delete pod exastro-maintenance --namespace exastro

5.4.3. Restart service

  1. サービス再開

    サービス停止時に取得した各 Deployment の Pod 起動数を元に戻します。
    リスト 5.68 コマンド
    kubectl scale deployment ita-by-ansible-execute --namespace exastro --replicas=${RS_AE}
kubectl scale deployment ita-by-ansible-legacy-role-vars-listup --namespace exastro --replicas=${RS_ALRV}
kubectl scale deployment ita-by-ansible-towermaster-sync --namespace exastro --replicas=${RS_ATS}
kubectl scale deployment ita-by-conductor-synchronize --namespace exastro --replicas=${RS_CS}
kubectl scale deployment ita-by-menu-create --namespace exastro --replicas=${RS_MC}
kubectl scale deployment platform-auth --namespace exastro --replicas=${RS_PA}
    以下はITA2.1.1以降の場合に実行してください。
    kubectl scale deployment ita-by-ansible-legacy-vars-listup --namespace exastro --replicas=${RS_ALV}
kubectl scale deployment ita-by-ansible-pioneer-vars-listup --namespace exastro --replicas=${RS_APV}
kubectl scale deployment ita-by-cicd-for-iac --namespace exastro --replicas=${RS_CFI}
kubectl scale deployment ita-by-conductor-regularly --namespace exastro --replicas=${RS_CR}
kubectl scale deployment ita-by-collector --namespace exastro --replicas=${RS_COL}
kubectl scale deployment ita-by-excel-export-import --namespace exastro --replicas=${RS_EEI}
kubectl scale deployment ita-by-hostgroup-split --namespace exastro --replicas=${RS_HS}
kubectl scale deployment ita-by-menu-export-import --namespace exastro --replicas=${RS_MEI}
kubectl scale deployment ita-by-terraform-cli-execute --namespace exastro --replicas=${RS_TCE}
kubectl scale deployment ita-by-terraform-cli-vars-listup --namespace exastro --replicas=${RS_TCV}
kubectl scale deployment ita-by-terraform-cloud-ep-execute --namespace exastro --replicas=${RS_TCEE}
kubectl scale deployment ita-by-terraform-cloud-ep-vars-listup --namespace exastro --replicas=${RS_TCEV}
    以下はITA2.3.0以降の場合に実行してください。
    kubectl scale deployment ita-by-oase-conclusion --namespace exastro --replicas=${RS_OC}

  2. Pod 起動数の確認

    上記で起動した対象の Pod 数が元に戻りすべて READY になっていることを確認
    リスト 5.69 コマンド
    kubectl get deployment --namespace exastro
    リスト 5.70 実行結果
    NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
mariadb                                  1/1     1            1           7h46m
ita-web-server                           1/1     1            1           7h46m
platform-web                             1/1     1            1           7h46m
ita-api-admin                            1/1     1            1           7h46m
ita-api-organization                     1/1     1            1           7h46m
platform-api                             1/1     1            1           7h46m
keycloak                                 1/1     1            1           7h46m
ita-by-menu-create                       1/1     1            1           7h46m
ita-by-ansible-execute                   1/1     1            1           7h46m
ita-by-ansible-legacy-role-vars-listup   1/1     1            1           7h46m
ita-by-ansible-towermaster-sync          1/1     1            1           7h46m
ita-by-conductor-synchronize             1/1     1            1           7h46m
platform-auth                            1/1     1            1           7h46m
    以下はITA2.1.0以降の場合に確認してください。
    ita-by-ansible-legacy-vars-listup        1/1     1            1           7h46m
ita-by-ansible-pioneer-vars-listup       1/1     1            1           7h46m
ita-by-cicd-for-iac                      1/1     1            1           7h46m
ita-by-collector                         1/1     1            1           7h46m
ita-by-conductor-regularly               1/1     1            1           7h46m
ita-by-excel-export-import               1/1     1            1           7h46m
ita-by-hostgroup-split                   1/1     1            1           7h46m
ita-by-menu-export-import                1/1     1            1           7h46m
ita-by-terraform-cli-execute             1/1     1            1           7h46m
ita-by-terraform-cli-vars-listup         1/1     1            1           7h46m
ita-by-terraform-cloud-ep-execute        1/1     1            1           7h46m
ita-by-terraform-cloud-ep-vars-listup    1/1     1            1           7h46m
    以下はITA2.3.0以降の場合に確認してください。
    ita-api-oase-receiver                    1/1     1            1           7h46m
ita-by-oase-conclusion                   1/1     1            1           7h46m

5.5. Restore

5.5.1. Stop service

  1. Pod 起動数の確認

    作業前の Pod 起動数の確認をし、状態を記録します。
    リスト 5.71 コマンド
    RS_AE=`kubectl get deploy ita-by-ansible-execute -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_ALRV=`kubectl get deploy ita-by-ansible-legacy-role-vars-listup -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_ATS=`kubectl get deploy ita-by-ansible-towermaster-sync -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_CS=`kubectl get deploy ita-by-conductor-synchronize -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_MC=`kubectl get deploy ita-by-menu-create -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_PA=`kubectl get deploy platform-auth -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
    以下はITA2.1.0以降の場合に実行してください。
    RS_ALV=`kubectl get deploy ita-by-ansible-legacy-vars-listup -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_APV=`kubectl get deploy ita-by-ansible-pioneer-vars-listup -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_CFI=`kubectl get deploy ita-by-cicd-for-iac -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_CR=`kubectl get deploy ita-by-conductor-regularly -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_COL=`kubectl get deploy ita-by-collector -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_EEI=`kubectl get deploy ita-by-excel-export-import -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_HS=`kubectl get deploy ita-by-hostgroup-split -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_MEI=`kubectl get deploy ita-by-menu-export-import -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_TCE=`kubectl get deploy ita-by-terraform-cli-execute -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_TCV=`kubectl get deploy ita-by-terraform-cli-vars-listup -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_TCEE=`kubectl get deploy ita-by-terraform-cloud-ep-execute -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
RS_TCEV=`kubectl get deploy ita-by-terraform-cloud-ep-vars-listup -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`
    以下はITA2.3.0以降の場合に実行してください。
    RS_OC=`kubectl get deploy ita-by-oase-conclusion -o jsonpath='{@.spec.replicas}{"\n"}' --namespace exastro`

  2. リバースプロキシの停止

    リバースプロキシ (platform-auth) の Pod 起動数を 0 に変更し、エンドユーザーからのアクセスを制限します。
    リスト 5.72 コマンド
    kubectl scale deployment platform-auth --namespace exastro --replicas=0

  3. バックヤード処理の停止

    バックヤード処理 (ita-by-***) の Pod 起動数を 0 に変更し、データベースの更新を停止します。
    リスト 5.73 コマンド
    kubectl scale deployment ita-by-ansible-execute --namespace exastro --replicas=0
kubectl scale deployment ita-by-ansible-legacy-role-vars-listup --namespace exastro --replicas=0
kubectl scale deployment ita-by-ansible-towermaster-sync --namespace exastro --replicas=0
kubectl scale deployment ita-by-conductor-synchronize --namespace exastro --replicas=0
kubectl scale deployment ita-by-menu-create --namespace exastro --replicas=0
    以下はITA2.1.0以降の場合に実行してください。
    kubectl scale deployment ita-by-ansible-legacy-vars-listup --namespace exastro --replicas=0
kubectl scale deployment ita-by-ansible-pioneer-vars-listup --namespace exastro --replicas=0
kubectl scale deployment ita-by-cicd-for-iac --namespace exastro --replicas=0
kubectl scale deployment ita-by-collector --namespace exastro --replicas=0
kubectl scale deployment ita-by-conductor-regularly --namespace exastro --replicas=0
kubectl scale deployment ita-by-excel-export-import --namespace exastro --replicas=0
kubectl scale deployment ita-by-hostgroup-split --namespace exastro --replicas=0
kubectl scale deployment ita-by-menu-export-import --namespace exastro --replicas=0
kubectl scale deployment ita-by-terraform-cli-execute --namespace exastro --replicas=0
kubectl scale deployment ita-by-terraform-cli-vars-listup --namespace exastro --replicas=0
kubectl scale deployment ita-by-terraform-cloud-ep-execute --namespace exastro --replicas=0
kubectl scale deployment ita-by-terraform-cloud-ep-vars-listup --namespace exastro --replicas=0
    以下はITA2.3.0以降の場合に実行してください。
    kubectl scale deployment ita-by-oase-conclusion --namespace exastro --replicas=0

  4. Pod 起動数の確認

    上記で停止した対象の Pod 数が 0 になっていることを確認
    リスト 5.74 コマンド
    kubectl get deployment --namespace exastro
    リスト 5.75 実行結果
    NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
mariadb                                  1/1     1            1           3h41m
ita-web-server                           1/1     1            1           3h41m
platform-web                             1/1     1            1           3h41m
ita-api-admin                            1/1     1            1           3h41m
ita-api-organization                     1/1     1            1           3h41m
platform-api                             1/1     1            1           3h41m
keycloak                                 1/1     1            1           3h41m
ita-by-menu-create                       0/0     0            0           3h41m
ita-by-ansible-execute                   0/0     0            0           3h41m
ita-by-ansible-legacy-role-vars-listup   0/0     0            0           3h41m
ita-by-ansible-towermaster-sync          0/0     0            0           3h41m
ita-by-conductor-synchronize             0/0     0            0           3h41m
platform-auth                            0/0     0            0           3h41m
    以下はITA2.1.0以降の場合に確認してください。
    NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
ita-by-ansible-legacy-vars-listup        0/0     0            0           3h41m
ita-by-ansible-pioneer-vars-listup       0/0     0            0           3h41m
ita-by-cicd-for-iac                      0/0     0            0           3h41m
ita-by-collector                         0/0     0            0           3h41m
ita-by-conductor-regularly               0/0     0            0           3h41m
ita-by-excel-export-import               0/0     0            0           3h41m
ita-by-hostgroup-split                   0/0     0            0           3h41m
ita-by-menu-export-import                0/0     0            0           3h41m
ita-by-terraform-cli-execute             0/0     0            0           3h41m
ita-by-terraform-cli-vars-listup         0/0     0            0           3h41m
ita-by-terraform-cloud-ep-execute        0/0     0            0           3h41m
ita-by-terraform-cloud-ep-vars-listup    0/0     0            0           3h41m
    以下はITA2.3.0以降の場合に確認してください。
    NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
ita-api-oase-receiver                    1/1     1            1           3h41m
ita-by-oase-conclusion                   0/0     0            0           3h41m

5.5.2. Restore

The process differs depending on if the Exastro Platform and Exastro IT Automation database server is shared or divided.

  1. Create container for maintenance

    Create container for restoration operation
    リスト 5.76 Command
     1cat <<_EOF_ | kubectl apply -f - --namespace exastro
 2apiVersion: v1
 3kind: Pod
 4metadata:
 5  name: exastro-maintenance
 6  namespace: exastro
 7spec:
 8  containers:
 9  - command:
10    - sh
11    - -c
12    args:
13    - |
14      #!/bin/bash
15      sleep 3600
16    env:
17    - name: PF_DB_DATABASE
18      valueFrom:
19        configMapKeyRef:
20          key: DB_DATABASE
21          name: platform-params-pf-database
22    - name: PF_DB_HOST
23      valueFrom:
24        configMapKeyRef:
25          key: DB_HOST
26          name: platform-params-pf-database
27    - name: PF_DB_PORT
28      valueFrom:
29        configMapKeyRef:
30          key: DB_PORT
31          name: platform-params-pf-database
32    - name: PF_DB_ADMIN_PASSWORD
33      valueFrom:
34        secretKeyRef:
35          key: DB_ADMIN_PASSWORD
36          name: platform-secret-pf-database
37    - name: PF_DB_ADMIN_USER
38      valueFrom:
39        secretKeyRef:
40          key: DB_ADMIN_USER
41          name: platform-secret-pf-database
42    - name: ITA_DB_DATABASE
43      valueFrom:
44        configMapKeyRef:
45          key: DB_DATABASE
46          name: ita-params-ita-database
47    - name: ITA_DB_HOST
48      valueFrom:
49        configMapKeyRef:
50          key: DB_HOST
51          name: ita-params-ita-database
52    - name: ITA_DB_PORT
53      valueFrom:
54        configMapKeyRef:
55          key: DB_PORT
56          name: ita-params-ita-database
57    - name: ITA_STORAGEPATH
58      valueFrom:
59        configMapKeyRef:
60          key: STORAGEPATH
61          name: ita-params-ita-global
62    - name: ITA_DB_ADMIN_PASSWORD
63      valueFrom:
64        secretKeyRef:
65          key: DB_ADMIN_PASSWORD
66          name: ita-secret-ita-database
67    - name: ITA_DB_ADMIN_USER
68      valueFrom:
69        secretKeyRef:
70          key: DB_ADMIN_USER
71          name: ita-secret-ita-database
72    image: mariadb:10.9
73    imagePullPolicy: IfNotPresent
74    name: exastro-maintenance
75    resources: {}
76    securityContext:
77      allowPrivilegeEscalation: false
78      readOnlyRootFilesystem: false
79      runAsGroup: 1000
80      runAsNonRoot: true
81      runAsUser: 1000
82    volumeMounts:
83    - mountPath: /storage
84      name: volume-ita-backup-storage
85  volumes:
86  - name: volume-ita-backup-storage
87    persistentVolumeClaim:
88      claimName: pvc-ita-global
89  restartPolicy: Always
90  securityContext: {}
91  serviceAccount: default
92  serviceAccountName: default
93_EOF_

  2. Start Database restoration

    Restore mysqldump to database
    The process differs depending on if the Exastro Platform and Exastro IT Automation database server is shared or divided.
    リスト 5.77 Database restore command for user
    gzip -dc exastro_mysqldump_platform_user_YYYYMMDD-HHmmss.sql.gz | kubectl exec -i exastro-maintenance --namespace exastro -- sh -c 'mysql -h ${PF_DB_HOST} -P ${PF_DB_PORT} -u ${PF_DB_ADMIN_USER} -p${PF_DB_ADMIN_PASSWORD} mysql'
    リスト 5.78 Database restore command for application
    gzip -dc exastro_mysqldump_platform_db_YYYYMMDD-HHmmss.sql.gz | sed -e 's/DEFINER[ ]*=[ ]*[^*]*\*/\*/'| kubectl exec -i exastro-maintenance --namespace exastro -- sh -c 'mysql -h ${PF_DB_HOST} -P ${PF_DB_PORT} -u ${PF_DB_ADMIN_USER} -p${PF_DB_ADMIN_PASSWORD}'

  3. Start file restoration

    Acquire Exastro IT Automation file back up.
    リスト 5.83 Command
    kubectl exec -i exastro-maintenance --namespace exastro -- sh -c 'tar zxvf - -C ${ITA_STORAGEPATH}' < exastro_storage_backup_ita_YYYYMMDD-HHmmss.tar.gz

  4. Delete maintenance container

    Create container for backup operations.
    リスト 5.84 Command
    kubectl delete pod exastro-maintenance --namespace exastro

5.5.3. Restart service

  1. サービス再開

    サービス停止時に取得した各 Deployment の Pod 起動数を元に戻します。
    リスト 5.85 コマンド
    kubectl scale deployment ita-by-ansible-execute --namespace exastro --replicas=${RS_AE}
kubectl scale deployment ita-by-ansible-legacy-role-vars-listup --namespace exastro --replicas=${RS_ALRV}
kubectl scale deployment ita-by-ansible-towermaster-sync --namespace exastro --replicas=${RS_ATS}
kubectl scale deployment ita-by-conductor-synchronize --namespace exastro --replicas=${RS_CS}
kubectl scale deployment ita-by-menu-create --namespace exastro --replicas=${RS_MC}
kubectl scale deployment platform-auth --namespace exastro --replicas=${RS_PA}
    以下はITA2.1.1以降の場合に実行してください。
    kubectl scale deployment ita-by-ansible-legacy-vars-listup --namespace exastro --replicas=${RS_ALV}
kubectl scale deployment ita-by-ansible-pioneer-vars-listup --namespace exastro --replicas=${RS_APV}
kubectl scale deployment ita-by-cicd-for-iac --namespace exastro --replicas=${RS_CFI}
kubectl scale deployment ita-by-conductor-regularly --namespace exastro --replicas=${RS_CR}
kubectl scale deployment ita-by-collector --namespace exastro --replicas=${RS_COL}
kubectl scale deployment ita-by-excel-export-import --namespace exastro --replicas=${RS_EEI}
kubectl scale deployment ita-by-hostgroup-split --namespace exastro --replicas=${RS_HS}
kubectl scale deployment ita-by-menu-export-import --namespace exastro --replicas=${RS_MEI}
kubectl scale deployment ita-by-terraform-cli-execute --namespace exastro --replicas=${RS_TCE}
kubectl scale deployment ita-by-terraform-cli-vars-listup --namespace exastro --replicas=${RS_TCV}
kubectl scale deployment ita-by-terraform-cloud-ep-execute --namespace exastro --replicas=${RS_TCEE}
kubectl scale deployment ita-by-terraform-cloud-ep-vars-listup --namespace exastro --replicas=${RS_TCEV}
    以下はITA2.3.0以降の場合に実行してください。
    kubectl scale deployment ita-by-oase-conclusion --namespace exastro --replicas=${RS_OC}

  2. Pod 起動数の確認

    上記で起動した対象の Pod 数が元に戻りすべて READY になっていることを確認
    リスト 5.86 コマンド
    kubectl get deployment --namespace exastro
    リスト 5.87 実行結果
    NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
mariadb                                  1/1     1            1           7h46m
ita-web-server                           1/1     1            1           7h46m
platform-web                             1/1     1            1           7h46m
ita-api-admin                            1/1     1            1           7h46m
ita-api-organization                     1/1     1            1           7h46m
platform-api                             1/1     1            1           7h46m
keycloak                                 1/1     1            1           7h46m
ita-by-menu-create                       1/1     1            1           7h46m
ita-by-ansible-execute                   1/1     1            1           7h46m
ita-by-ansible-legacy-role-vars-listup   1/1     1            1           7h46m
ita-by-ansible-towermaster-sync          1/1     1            1           7h46m
ita-by-conductor-synchronize             1/1     1            1           7h46m
platform-auth                            1/1     1            1           7h46m
    以下はITA2.1.0以降の場合に確認してください。
    ita-by-ansible-legacy-vars-listup        1/1     1            1           7h46m
ita-by-ansible-pioneer-vars-listup       1/1     1            1           7h46m
ita-by-cicd-for-iac                      1/1     1            1           7h46m
ita-by-collector                         1/1     1            1           7h46m
ita-by-conductor-regularly               1/1     1            1           7h46m
ita-by-excel-export-import               1/1     1            1           7h46m
ita-by-hostgroup-split                   1/1     1            1           7h46m
ita-by-menu-export-import                1/1     1            1           7h46m
ita-by-terraform-cli-execute             1/1     1            1           7h46m
ita-by-terraform-cli-vars-listup         1/1     1            1           7h46m
ita-by-terraform-cloud-ep-execute        1/1     1            1           7h46m
ita-by-terraform-cloud-ep-vars-listup    1/1     1            1           7h46m
    以下はITA2.3.0以降の場合に確認してください。
    ita-api-oase-receiver                    1/1     1            1           7h46m
ita-by-oase-conclusion                   1/1     1            1           7h46m

5.6. Troubleshooting

5.6.1. Error 500 occurs after restore

  • Error

Error 500 occurs after the restore process has finished.

  • Solution

Delete all resources and rerun helm install command to re-install the system.