Exastro on Docker Compose - Offline

Introduction

This document aims to explain how to install Exastro Platform or Exastro IT Automation on Docker or Podman.

Features

This is the easiest and simplest way of installing Exastro IT Automation
For higher availability and service level, we recommend Kubernetes version

Pre-requisites

The server that is collecting files must be able to run docker. (When using Podman, the podman-docker must be installed.)
The construction status (OS version and installed pakcages) for both the file collecting environment and the installing environment must be the same.
  • Deploy environment

    The hardware requirements for the Container environment are as follows.
    Table 46 Hardware requirements(Minimum)

    Resource type

    Required resource

    CPU

    2 Cores (3.0 GHz, x86_64)

    Memory

    4GB

    Storage (Container image size)

    40GB

    Table 47 Hardware requirements(Recommended)

    Resource type

    Required resource

    CPU

    4 Cores (3.0 GHz, x86_64)

    Memory

    16GB

    Storage (Container image size)

    120GB

    Warning

    The required resources for the minimum configuration are for Exastro IT Automation’s core functions. Additional resources will be required if you are planning to deploy external systems, such as GitLab and Ansible Automation Platform.
    Users will have to prepare an additional storage area if they wish to persist databases or files.
    The storage space is only an estimate and varies based on the user’s needs. Make sure to take that into account when securing storage space.
  • Communication Protocols

    Table 48 Communication Protocols

    Use

    Description

    Source

    Destination

    Default

    For Exastro service

    For connecting to Exastro service

    Client

    Exastro system

    30080/tcp

    Exastro system (management)

    For Exastro system management function

    Client

    Exastro system

    30081/tcp

    GitLab service(options)

    For connecting to GitLab when linked with AAP

    Ansible Automation Platform

    Exastro system

    40080/tcp

    For GitLab service (option)

    For monitoring GitLab service

    Exastro system

    Exastro system

    40080/tcp

    File acquisition

    GitHub, Container images, Packages, etc.

    Exastro system

    Internet

    443/tcp

  • Confirmed compatible Operation systems and container platforms

    The following describes confirmed compatible operation systems as well as their versions.

    Table 49 Tested environments

    OS version

    podman version

    Docker Compose version

    Docker version

    Red Hat Enterprise Linux release 9.4 (Plow)

    podman version 4.9.4-rhel

    Docker Compose version v2.20.3

    Red Hat Enterprise Linux release 8.9 (Ootpa)

    podman version 4.9.4-rhel

    Docker Compose version v2.20.3

    AlmaLinux release 8.9 (Midnight Oncilla)

    Docker version 26.1.3, build b72abbb

  • Applications

    The user must be able to run curl and sudo commands.

Warning

The Exastro process must be able to be run with normal user permissions (it is not possible to install with root user).
Any normal users must be sudoer and have complete permissions.

Preparation

The user must prepare an URL for releasing the service.
Table 50 Example 1) Releasing service with IP Address

Service

URL

Exastro service

http://172.16.0.1:30080

Exastro management service

http://172.16.0.1:30081

GitLab service

http://172.16.0.1:40080

Table 51 Example 2) Releasing service with Domain

Service

URL

Exastro service

http://ita.example.com:30080

Exastro management service

http://ita.example.com:30081

GitLab service

http://ita.example.com:40080

Table 52 Example 3) Releasing service through LoadBalancer

Service

URL

Exastro service

https://ita.example.com

Exastro management service

https://ita-mng.example.com

GitLab service

https://gitlab.example.com

Tip

If the user is using HTTPS, they must use either LoadBalancer or Reverse proxy.
If the user plans to use LoadBalancer or Reverse proxy, they will have to prepare that themselves.

General flow

After preparing the online environment, the user can now install on to the offline system.
Flow image

Online environment

①Download Container image
②Download RPM packages
③Download docker-compose resources
④Download Exastro resources

Offline environment

⑤Install RPM packages
⑥Upload Container image
⑦Install docker-compose resources
⑧Install Exastro resources
⑨Boot Exastro ITA

Guide for Online environment(Environment that can connect to the internet)

First, fetch the files.
In the following example, the user is “test_user” and the home directory is /home/test_user.

①Upload container image

Create the container image list and the shell script that will download the container image in a partition that has over 25GB space.
These two must be created in the same directory.
The save.sh’s [“x.x.x”]=”x.x.x” contains both the Exastro IT Automation App Version and the Exastro Platform App Version.
Refer to the Component version and replace them with the newest versions.
Listing 188 Command
vi save.sh
Listing 189 Copy and paste the code below and rewrite the version
#!/bin/bash

ITA_VERSION=$1
declare -A PF_VERSION=(
  ["x.x.x"]="x.x.x"
)
if [ ! -d $1 ]; then
  mkdir $ITA_VERSION
fi

readarray -t image_list < "./image.list"
for image in ${image_list[@]}
do
  image_fullname=$(echo ${image} | sed -e "s/#__ITA_VERSION__#/${ITA_VERSION}/" -e "s/#__PF_VERSION__#/${PF_VERSION[$ITA_VERSION]}/")
  image_name=$(basename ${image_fullname} | sed -e "s/:/-/")
  if [ ! -e ${ITA_VERSION}/${image_name}.tar.gz ]; then
    echo $image_fullname $image_name
    docker pull ${image_fullname}
    if [ $? -eq 0 ]; then
      docker save ${image_fullname} | gzip -c > ${ITA_VERSION}/${image_name}.tar.gz
    fi
  fi
done
Listing 190 Command
vi image.list
Listing 191 Copy and paste the following code
docker.io/mariadb:10.9.8
docker.io/mariadb:10.11.4
docker.io/gitlab/gitlab-ce:15.11.13-ce.0
docker.io/mongo:6.0.7
docker.io/exastro/keycloak:#__PF_VERSION__#
docker.io/exastro/exastro-platform-auth:#__PF_VERSION__#
docker.io/exastro/exastro-platform-web:#__PF_VERSION__#
docker.io/exastro/exastro-platform-api:#__PF_VERSION__#
docker.io/exastro/exastro-platform-job:#__PF_VERSION__#
docker.io/exastro/exastro-platform-migration:#__PF_VERSION__#
docker.io/exastro/exastro-platform-migration:#__PF_VERSION__#
docker.io/exastro/exastro-it-automation-api-organization:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-api-admin:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-api-oase-receiver:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-web-server:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-agent:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-execute:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-execute-onpremises:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-legacy-role-vars-listup:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-legacy-vars-listup:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-pioneer-vars-listup:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-towermaster-sync:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-collector:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-conductor-synchronize:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-conductor-regularly:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-menu-create:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-menu-export-import:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-excel-export-import:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-terraform-cloud-ep-execute:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-terraform-cloud-ep-vars-listup:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-terraform-cli-execute:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-terraform-cli-vars-listup:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-hostgroup-split:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-cicd-for-iac:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-oase-conclusion:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-execinstance-dataautoclean:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-file-autoclean:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-migration:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-agent:#__ITA_VERSION__#

If the user is not added to a group, and permission error might occur. The following is not required if done in advanced.

Listing 192 Command
cat /etc/group | grep docker
#If the user is not added to any groups, run the following.
sudo usermod -aG docker ${USER}
cat /etc/group | grep docker
#Check that the user has been added to the group and reboot the server.
sudo reboot

After connecting to an online environment, run the following shell script and download the container image. The parameter specifies the version of ITA. This command can take several minutes before finishing(Depends on the server specs and the connection speeds).

Listing 193 Command
sudo systemctl start docker
sudo chmod a+x save.sh
sh ./save.sh x.x.x

②Download RPM packages

Download the RPM packages.
In the example below, the download destination directory is set to /tmp/docker-repo, and the installation destination directory is set to /tmp/docker-installroot.
Listing 195 Command
#Add repository
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
#Confirm current OS version
cat /etc/os-release
#Specify version fetched above for  --releasever=x.x.
sudo dnf install -y --downloadonly --downloaddir=/tmp/docker-repo --installroot=/tmp/docker-installroot --releasever=x.x docker-ce docker-ce-cli containerd.io git container-selinux
Install createrepo.
Listing 196 Command
sudo dnf install -y createrepo
Create local repository.
Since its not possible to access an online repository server from an offline environment, it is not possible to install packages with dnf.
It is only possible to install them with dnf if we add the packages to a local repository.
Listing 197 Command
sudo createrepo /tmp/docker-repo

③Download docker-compose resources

Change the URL to fit the user’s desired version.
The version used in the example below is version 2.28.0.
Listing 201 Command
curl -LO https://github.com/docker/compose/releases/download/v2.28.0/docker-compose-linux-x86_64

④Download Exastro resources

Download the docker-compose version’s Exastro resources.
The following example downloads to the /tmp directory.
Listing 202 Command
cd /tmp
curl -OL https://github.com/exastro-suite/exastro-docker-compose/archive/main.tar.gz

Transfer files

Transfer the files fetched in the online environment and to the offline environment with storage mediums such as FPT, SCP and SFTP.
Feel free to zip the files if they are too big.
The transfer files and their destinations are as following.
  • Container image:Free directoy

  • RPM Package: below /tmp

  • Exastro resource:Under normal user’s home directory

  • docker-compose-linux-x86_64:/usr/local/bin

Offline environment(Environment unable to connect to the internet)

After the steps for the online environment is finished, follow the steps below to install to the offline environment.

⑤Install RPM packages

Create Local repository Configuration file.
Listing 203 Command
sudo touch /etc/yum.repos.d/docker-repo.repo
Write the following information to the created configuration file (※add three slashes after file:)
Listing 204 Command
sudo vi /etc/yum.repos.d/docker-repo.repo

[docker-repo]
name=AlmaLinux-$releaserver - docker
baseurl=file:///tmp/docker-repo
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux
Install package
Listing 205 Command
sudo dnf -y --disablerepo=\* --enablerepo=docker-repo install docker-ce docker-ce-cli containerd.io git container-selinux
If an error mesage displays, search for the displayed modules and install all of them.
Listing 206 Message example
No available modular metadata for modular package 'perl-Mozilla-CA-20160104-7.module_el8.5.0+2812+ed912d05.noarch', it cannot be installed on the system
No available modular metadata for modular package 'perl-Net-SSLeay-1.88-2.module_el8.6.0+2811+fe6c84b0.x86_64', it cannot be installed on the system
Error: No available modular metadata for modular package
Listing 207 Check all the dispalyed modules and install all of them.
#If the target is perl-Mozilla-CA or perl-Net-SSLeay
cd /tmp/docker-repo
ls -l | grep -E "perl-Mozilla-CA|perl-Net-SSLeay"
sudo dnf -y --disablerepo=\* --enablerepo=docker-repo perl-Mozilla-CA-20160104-7.module_el8.5.0+2812+ed912d05.noarch.rmp perl-Net-SSLeay-1.88-2.module_el8.6.0+2811+fe6c84b0.x86_64.rpm
Reinstall the packages
Listing 208 Command
sudo dnf -y --disablerepo=\* --enablerepo=docker-repo install docker-ce docker-ce-cli containerd.io git container-selinux
Add the user to the docker group.
Listing 209 Command
sudo systemctl enable --now docker
cat /etc/group | grep docker
sudo usermod -aG docker ${USER}
#Confirm that the user name displays
cat /etc/group | grep docker
sudo  reboot
#Reconnect to the offline environment.

⑥Upload Container image

Create the container image list and the shell script that will download the container image in a partition that has over 25GB space.
These two must be created in the same directory. The image.list contains the same as the one created in the online environment.
The load.sh’s [“x.x.x”]=”x.x.x” must be the same version written in save.sh.
Listing 213 Command
vi load.sh
Listing 214 Copy and paste the code below and rewrite the version
ITA_VERSION=$1
declare -A PF_VERSION=(
  ["x.x.x"]="x.x.x"
)

readarray -t image_list < "./image.list"
for image in ${image_list[@]}
do
  image_fullname=$(echo ${image} | sed -e "s/#__ITA_VERSION__#/${ITA_VERSION}/" -e "s/#__PF_VERSION__#/${PF_VERSION[$ITA_VERSION]}/")
  image_name=$(basename ${image_fullname} | sed -e "s/:/-/")
  docker load < ${ITA_VERSION}/${image_name}.tar.gz
done

wait
Listing 215 Command
vi image.list
Listing 216 Copy and paste the following code
docker.io/mariadb:10.9.8
docker.io/mariadb:10.11.4
docker.io/gitlab/gitlab-ce:15.11.13-ce.0
docker.io/mongo:6.0.7
docker.io/exastro/keycloak:#__PF_VERSION__#
docker.io/exastro/exastro-platform-auth:#__PF_VERSION__#
docker.io/exastro/exastro-platform-web:#__PF_VERSION__#
docker.io/exastro/exastro-platform-api:#__PF_VERSION__#
docker.io/exastro/exastro-platform-job:#__PF_VERSION__#
docker.io/exastro/exastro-platform-migration:#__PF_VERSION__#
docker.io/exastro/exastro-platform-migration:#__PF_VERSION__#
docker.io/exastro/exastro-it-automation-api-organization:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-api-admin:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-api-oase-receiver:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-web-server:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-agent:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-execute:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-execute-onpremises:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-legacy-role-vars-listup:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-legacy-vars-listup:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-pioneer-vars-listup:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-towermaster-sync:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-collector:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-conductor-synchronize:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-conductor-regularly:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-menu-create:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-menu-export-import:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-excel-export-import:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-terraform-cloud-ep-execute:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-terraform-cloud-ep-vars-listup:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-terraform-cli-execute:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-terraform-cli-vars-listup:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-hostgroup-split:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-cicd-for-iac:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-oase-conclusion:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-execinstance-dataautoclean:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-file-autoclean:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-migration:#__ITA_VERSION__#
docker.io/exastro/exastro-it-automation-by-ansible-agent:#__ITA_VERSION__#
Execute Container image. The parameter specifies the version of ITA.
Listing 217 Command
sudo chmod a+x load.sh
sh ./load.sh x.x.x

⑦Install docker-compose resources

Add required permissions to the docker-compose-linux-x86_64 in /usr/local/bin.
Listing 219 Command
cd /usr/local/bin
sudo mv docker-compose-linux-x86_64 docker-compose
sudo chmod a+x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

⑧Install Exastro resources

Extract the docker-compose version’s Exastro resources under the home directory of a normal user. After that, change the directory name to exastro-docker-compose.
Listing 220 Command
tar -zxvf main.tar.gz
sudo mv exastro-docker-compose-main exastro-docker-compose
Change the SELinux mode to SELINUX=permissive.
Listing 221 Command
1sudo vi /etc/selinux/config
Listing 222 /etc/selinux/config example
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
# See also:
# https://docs.fedoraproject.org/en-US/quick-docs/getting-started-with-selinux/#getting-started-with-selinux-selinux-states-and-modes
#
# NOTE: In earlier Fedora kernel buil, SELINUX=disabled would also
# fully disable SELinux during boot. If you need a system with SELinux
# fully disabled instead of SELinux running with no policy loaded, you
# need to pass selinux=0 to the kernel command line. You can use grubby
# to persistently set the bootloader to boot with selinux=0:
#
#    grubby --update-kernel ALL --args selinux=0
#
# To revert back to SELinux enabled:
#
#    grubby --update-kernel ALL --remove-args selinux
#
SELINUX=permissive
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
Listing 223 Command
sudo reboot
#Reconnect to the offline environment
This setup.sh is the same used in Exastro on Docker Compose - Online. Follow the steps below to comment out the repository settings.
Listing 224 Command
sed -i 's/sudo dnf config-manager/#sudo dnf config-manager/' setup.sh
Install the Exastro Service packages and Exastro source file.
Listing 225 Command
cd ~/exastro-docker-compose && sh ./setup.sh install
When the required packages are installed, the user should be met with an interactible installation process where they can input setting values.
In order to edit detailed settings, input n or no and skip the next processes.
In order to boot the Exastro system container groups, input y or yes.
Deploying the Exastro system can take up to several minutes.(Depends on the server specs and the connection speeds)
Listing 226 OASE container deployment confirmation
Deploy OASE container ? (y/n) [default: y]:
Listing 227 Gitlab deployment confirmation
Deploy Gitlab containser? (y/n) [default: n]:
Listing 228 Password token automatic creation confirmation
Generate all password and token automatically? (y/n) [default: y]:
Listing 229 Exastro service URL
#Input 30800 for the port number if the OS is Red Hat Enterprise Linux. For everything else, input 80.
Input the Exastro service URL: https://ita.example.com:30080
Listing 230 Exastro management service URL
#Input 30801 for the port number if the OS is Red Hat Enterprise Linux. For everything else, input 81.
Input the Exastro management URL: https://ita.example.com:30081
Listing 231 Self-signed SSL/TSL certificate generation (If Exastro service URL/Exastro management service URL is set to https)
Generate self-signed SSL certificate? (y/n) [default: y]:
Listing 232 Server certificate/Secret key file path (If Self-signed SSL/TSL certificate generation is set to “n”)
#Specify the server certificate file path for the certificate file path and the secret key file's file path for the private-key file path.
Input path to your SSL certificate file.
certificate file path:
private-key file path:
Listing 235 Gitlab container URL(Required if deploying Gitlab container).
#Specify 40080 for the port number.
Input the external URL of Gitlab container  [default: (nothing)]:
Listing 236 Configuration file generation confirmation
System parametes are bellow.

System administrator password:    ********
Database password:                ********
OASE deployment                   true
MongoDB password                  ********
Service URL:                      http://ita.example.com:30080
Manegement URL:                   http://ita.example.com:30081
Docker GID:                       985
Docker Socket path:               /var/run/docker.sock
GitLab deployment:                false

Generate .env file with these settings? (y/n) [default: n]
Reboot server
Listing 237 Command
sudo reboot
Listing 238 Command
cd ~/exastro-docker-compose && sh ./setup.sh install
Listing 239 Reconfirm .env
#Press enter without inputting anything.
Regenerate .env file? (y/n) [default: n]:
Listing 240 Exastro container deploy confirmation
#Input y.
Deploy Exastro containers now? (y/n) [default: n]:
Check that the Container STATUS says UP.
Listing 241 Command
docker ps
Listing 256 Confirm generated settings file
System parametes are bellow.

System administrator password:    ********
Database password:                 ********
OASE deployment                   true
MongoDB password                  ********
Service URL:                      http://ita.example.com:30080
Manegement URL:                   http://ita.example.com:30081
Docker GID:                       1000
Docker Socket path:               /run/user/1000/podman/podman.sock
GitLab deployment:                false

      Generate .env file with these settings? (y/n) [default: n]


   | Reboot server

   .. code-block:: shell
      :caption: Command

      sudo reboot


   .. code-block:: shell
      :caption: Command

      cd ~/exastro-docker-compose && sh ./setup.sh install


   .. code-block:: shell
      :caption: Confirm that .env has been regenerated

      #Press enter without inputting anything.
      Regenerate .env file? (y/n) [default: n]:

   .. code-block:: shell
      :caption: Exastro container deploy confirmation

      #Input y
      Deploy Exastro containers now? (y/n) [default: n]:


   | Check that the Container STATUS says UP.
.. code-block:: shell
   :caption: Command

   podman ps

Login

Confirm the username and password information of the user.
Listing 257 command
1cd ~/exastro-docker-compose
2cat .env
Listing 258 Login information
1### Initial account information for creating system administrators
2#### Specify the username and password
3# SYSTEM_ADMIN=<Username>
4SYSTEM_ADMIN_PASSWORD=<Password>

Create Organization

After rebooting and logging in to the system, create an organization.

Tip

When linked to GitLab, the GitLab must be running in order to create Organizations.

Create Workspace

After creating an organization, log in to it and create a workspace.