Disconnected Install of RHOAI

In a disconnected installation there are two sides to the network, a low side security network that can talk to the world wide web, and a high side security network that can talk to the OpenShift cluster and the low side network but can not talk to the world wide web.

The rest of this document will refer to the server that can manage with the disconnected OpenShift cluster as the high side and the server that can fetch various resources as the low side.

In general, it is required to download all the required resources on the low side and then upload them to the high side for use with the OpenShift cluster.

Syncing resources with OpenShift

Red Hat delivers a tool called oc-mirror to help with syncing images from registries to a private disconnected registry.

This write up utilized oc-mirror v1. To learn more about it visit oc-mirror.

Currently there is a newer version (v2) which is GA since OCP 4.19.12 (prior to 4.19.12, this was a technology preview). The OpenShift documentation provides a migration from oc-mirror plugin v1 to v2 details.

Steps to install RHOAI in disconnected cluster

Step 1: Prepare the low side

Download the required resources on the low side and then rsync them to the high side. The following resources are required:

oc-mirror - The OpenShift mirroring tool that will be used to download the images from the Red Hat Quay registry and then upload them to the mirror registry

curl -L -o oc-mirror.tar.gz https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.16.9/oc-mirror.tar.gz # newer version available in a dedicated paths (e.g. 4.19.12 or newer)
tar -xzf oc-mirror.tar.gz
rm -f oc-mirror.tar.gz
chmod +x oc-mirror
sudo cp -v oc-mirror /bin/

mirror-registry - An image registry that serves container images to the OpenShift nodes. This is a smaller & streamlined version of the Red Hat Quay Image Registry
```
curl -L -o mirror-registry.tar.gz https://mirror.openshift.com/pub/cgw/mirror-registry/latest/mirror-registry-amd64.tar.gz
```
We will also download two additional tools that will be used later on the highside system.

openshift-install: The OpenShift Installer

curl -L -o openshift-install.tar.gz https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.20.4/openshift-install-linux.tar.gz
tar -xzf openshift-install.tar.gz openshift-install
rm -f openshift-install.tar.gz

oc: The OpenShift command line interface

curl -L -o oc.tar.gz https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.20.4/openshift-client-linux.tar.gz
tar -xzf oc.tar.gz oc
rm -f oc.tar.gz
sudo cp -v oc /bin

Create a ImageSetConfiguration file to download required images. This file will be used by the oc-mirror tool to download the images from the Red Hat registry.

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 4
storageConfig:
  registry:
    imageURL: registry.example.com:5000/mirror/oc-mirror-metadata
    skipTLS: false
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
    packages:
    - name: rhods-operator
      channels:
      - name: fast
        minVersion: 2.25.0
        maxVersion: 2.25.0
  additionalImages:
    - name: quay.io/modh/fms-hf-tuning@sha256:1ad46fe1a23f41f190c49ec2549c64f484c88fe220888a7a5700dd857ca243cc
    - name: quay.io/modh/ray@sha256:6d076aeb38ab3c34a6a2ef0f58dc667089aa15826fa08a73273c629333e12f1e
    - name: quay.io/modh/ray@sha256:23860dfe2e47bb69709b3883b08fd1a4d836ce02eaf8d0afeeafe6986d0fc8fb
    - name: quay.io/modh/ray@sha256:fb6f207de63e442c67bb48955cf0584f3704281faf17b90419cfa274fdec63c5
    - name: quay.io/modh/ray@sha256:6091617d45d5681058abecda57e0ee33f57b8855618e2509f1a354a20cc3403c
    - name: quay.io/modh/ray@sha256:f374130bf615a44d048fabbc75f4e6fb687446981383e3e815bda1dcda608964
    - name: registry.redhat.io/rhelai1/instructlab-nvidia-rhel9@sha256:b3dc9af0244aa6b84e6c3ef53e714a316daaefaae67e28de397cd71ee4b2ac7e
    - name: registry.redhat.io/openshift4/ose-oauth-proxy-rhel9@sha256:d3056b35d9a205b9f2c48d924f199c5ac23904eb18d526e4bff229e7c7181415
    - name: quay.io/modh/odh-minimal-notebook-container@sha256:2217d8a9cbf84c2bd3e6c6dc09089559e8a3905687ca3739e897c4b45e2b00b3
    - name: quay.io/modh/odh-minimal-notebook-container@sha256:e2296a1386e4d9756c386b4c7dc44bac6f61b99b3b894a10c9ff2d8d5602ca4e
    - name: quay.io/modh/odh-minimal-notebook-container@sha256:4ba72ae7f367a36030470fa4ac22eca0aab285c7c3f1c4cdcc33dc07aa522143
    - name: quay.io/modh/odh-minimal-notebook-container@sha256:eec50e5518176d5a31da739596a7ddae032d73851f9107846a587442ebd10a82
    - name: quay.io/modh/odh-minimal-notebook-container@sha256:39068767eebdf3a127fe8857fbdaca0832cdfef69eed6ec3ff6ed1858029420f
    - name: quay.io/modh/cuda-notebooks@sha256:55598c7de919afc6390cf59595549dc4554102481617ec42beaa3c47ef26d5e4
    - name: quay.io/modh/cuda-notebooks@sha256:81484fafe7012792ecdda28fef89287219c21b99c4e79a504aff0b265d94b429
    - name: quay.io/modh/cuda-notebooks@sha256:a484d344f6feab25e025ea75575d837f5725f819b50a6e3476cef1f9925c07a5
    - name: quay.io/modh/cuda-notebooks@sha256:f6cdc993b4d493ffaec876abb724ce44b3c6fc37560af974072b346e45ac1a3b
    - name: quay.io/modh/cuda-notebooks@sha256:00c53599f5085beedd0debb062652a1856b19921ccf59bd76134471d24c3fa7d
    - name: quay.io/modh/odh-pytorch-notebook@sha256:20f7ab8e7954106ea5e22f3ee0ba8bc7b03975e5735049a765e021aa7eb06861
    - name: quay.io/modh/odh-pytorch-notebook@sha256:2403b3dccc3daf5b45a973c49331fdac4ec66e2e020597975fcd9cb4a625099b
    - name: quay.io/modh/odh-pytorch-notebook@sha256:806e6524cb46bcbd228e37a92191c936bb4c117100fc731604e19df80286b19d
    - name: quay.io/modh/odh-pytorch-notebook@sha256:97b346197e6fc568c2eb52cb82e13a206277f27c21e299d1c211997f140f638b
    - name: quay.io/modh/odh-pytorch-notebook@sha256:b68e0192abf7d46c8c6876d0819b66c6a2d4a1e674f8893f8a71ffdcba96866c
    - name: quay.io/modh/odh-generic-data-science-notebook@sha256:d0ba5fc23e2b3846763f60e8ade8a0f561cdcd2bf6717df6e732f6f8b68b89c4
    - name: quay.io/modh/odh-generic-data-science-notebook@sha256:3e51c462fc03b5ccb080f006ced86d36480da036fa04b8685a3e4d6d51a817ba
    - name: quay.io/modh/odh-generic-data-science-notebook@sha256:39853fd63555ebba097483c5ac6a375d6039e5522c7294684efb7966ba4bc693
    - name: quay.io/modh/odh-generic-data-science-notebook@sha256:e2cab24ebe935d87f7596418772f5a97ce6a2e747ba0c1fd4cec08a728e99403
    - name: quay.io/modh/odh-generic-data-science-notebook@sha256:76e6af79c601a323f75a58e7005de0beac66b8cccc3d2b67efb6d11d85f0cfa1
    - name: quay.io/modh/cuda-notebooks@sha256:99d3fb964e635873214de4676c259a96c2ea25f3f79cc4bead5bc9f39aba34c0
    - name: quay.io/modh/cuda-notebooks@sha256:0e57a0b756872636489ccd713dc9f00ad69d0c481a66ee0de97860f13b4fedcd
    - name: quay.io/modh/cuda-notebooks@sha256:3da74d732d158b92eaada0a27fb7067fa18c8bde5033c672e23caed0f21d6481
    - name: quay.io/modh/cuda-notebooks@sha256:88d80821ff8c5d53526794261d519125d0763b621d824f8c3222127dab7b6cc8
    - name: quay.io/modh/cuda-notebooks@sha256:6fadedc5a10f5a914bb7b27cd41bc644392e5757ceaf07d930db884112054265
    - name: quay.io/modh/odh-trustyai-notebook@sha256:a1b863c2787ba2bca292e381561ed1d92cf5bc25705edfb1ded5e0720a12d102
    - name: quay.io/modh/odh-trustyai-notebook@sha256:70fe49cee6d5a231ddea7f94d7e21aefd3d8da71b69321f51c406a92173d3334
    - name: quay.io/modh/odh-trustyai-notebook@sha256:fe883d8513c5d133af1ee3f7bb0b7b37d3bada8ae73fc7209052591d4be681c0
    - name: quay.io/modh/odh-trustyai-notebook@sha256:8c5e653f6bc6a2050565cf92f397991fbec952dc05cdfea74b65b8fd3047c9d4
    - name: quay.io/modh/codeserver@sha256:92f2a10dde5c96b29324426b4325401e8f4a0d257e439927172d5fe909289c44
    - name: quay.io/modh/codeserver@sha256:1fd51b0e8a14995f1f7273a4b0b40f6e7e27e225ab179959747846e54079d61e
    - name: quay.io/modh/codeserver@sha256:b1a048f3711149e36a89e0eda1a5601130fb536ecc0aabae42ab6e4d26977354
    - name: quay.io/modh/rocm-notebooks@sha256:199367d2946fc8427611b4b96071cb411433ffbb5f0988279b10150020af22db
    - name: quay.io/modh/rocm-notebooks@sha256:1f0b19b7ae587d638e78697c67f1290d044e48bfecccfb72d7a16faeba13f980
    - name: quay.io/modh/rocm-notebooks@sha256:f94702219419e651327636b390d1872c58fd7b8f9f6b16a02c958ffb918eded3

RHOAI disonnected install helper will help you build the ImageSetConfiguration file. RHOAI Disonnected Install Helper

Mirror images to low side using oc-mirror tool. This will download the images from the Red Hat registry and then upload them to the mirror registry. A tar file will be created containing the images that will be used to load the mirror registry.
```
oc-mirror --config imageset-config.yaml file://low-side
```

Ensure you have access to Red Hat Registry to download the images. A pull secret is required to access the Red Hat registry. The pull secret is available in the Red Hat account (Red Hat Hybrid Cloud Console).

More information on disconnected mirroring can be found in the OpenShift documentation Managing OpenShift Container Platform clusters in a disconnected environment.

Step 2: Prepare the high side

The highside system needs to be setup to run an image registry inside the disconnected network. OpenShift provides a mirror-registry that is purpose-built for this task. The mirror-registry is a simplified, and portable, version of Red Hat’s Quay Image Registry.
You can use any image registry that supports the Docker v2 API, uses TLS encryption, and requires authenticated image pulls. Examples of alternative image registries include.
1. Harbor
2. JFrog Artifactory
3. Sonatype Nexus Repository
4. Red Hat Quay Registry (enterprise)

Transfering the installation content to the highside system using rsync

rsync -avP low-side-data/mirror-registry.tar.gz highside:high-side-data/

Uploading OpenShift’s installation images into mirror-registry

Upload OpenShift installation images into the mirror-registry

sudo mv -v high-side-data/oc /bin/
sudo mv -v high-side-data/oc-mirror /bin/
sudo mv -v high-side-data/openshift-install /bin/

Use oc-mirror to upload the images into the mirror-registry

oc-mirror --from=high-side-data/mirror_seq1_000000.tar docker://$(hostname):8443

Install Red Hat OpenShift on the high-side system.

Provide your disconnected pull secret
Identify your disconnected mirror-registry

Trust the mirror-registry’s TLS certificate

install-config.yaml

apiVersion: v1
metadata:
  name: disco
baseDomain: lab
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  replicas: 1
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.0.0.0/16
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
platform:
  aws:
    region: us-east-2
    subnets:
    - subnet-0b39c483c45355b10
publish: Internal
additionalTrustBundlePolicy: Always

The pull secret that gets added to a disconnected installation’s install-config.yaml is only required to provide credentials for the mirror-registry

Add ImageContentSourcePolicy to the install-config.yaml file. This is required to tell OpenShift where to find the images in the disconnected environment.
The imageContentSources: lines tell OpenShift that its installation images should be pulled from your mirror-registry instead of from quay.io
```
cat install-config.yaml
imageContentSources:
$(grep "mirrors:" -A 2 --no-group-separator high-side-data/oc-mirror-workspace/results-*/imageContentSourcePolicy.yaml)
```

Add the Root CA of your mirror-registry to the install-config.yaml file by running this command:

cat high-side-data/install-config.yaml
additionalTrustBundle: |
$(sed 's/^/  /' /home/quay-install/quay-rootCA/rootCA.pem)

Run the openshift-install command.

openshift-install create cluster --dir /mnt/high-side-data

Continue with day-2 activities

Disable the default OperaterHub locations since the cluster can’t reach out to them.

oc patch OperatorHub cluster --type json     -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'

oc-mirror will create catalog source.

Sample CatalogSource.yaml

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: cs-redhat-operator-index
  namespace: openshift-marketplace
spec:
  image: ip-10-0-54-104.us-east-2.compute.internal:8443/redhat/redhat-operator-index:v4.14
  sourceType: grpc

Configure OCP to pull from the mirror-registry

Red Hat OpenShift has a concept called Image Source Content Policy (ISCP), Image Digest Mirror Set (IDMS) and Image Tag Mirror Set (ITMS).

oc-mirror (v1) will generate an ISCP. However, ISCP is being deprecated in favor of IDMS and ITMS. The difference between an IDMS and ITMS is simply one will redirect only for digests (IDMS) and the other only for tags (ITMS). The deprecated ISCP will only do digests.

To convert the generated ISCP to IDMS.

This will create a new IDMS file.

oc adm migrate icsp <file_name>.yaml <file_name>.yaml <file_name>.yaml --dest-dir <path_to_the_directory>

More information about the ImageContentSourcePolicy and ImageDigestMirrorSet can be found in the OpenShift documentation. Convert ImageContentSourcePolicy for image registry repository mirroring

Sample Output

  apiVersion: config.openshift.io/v1
  kind: ImageDigestMirrorSet
  metadata:
    name: generic-0
  spec:
    imageDigestMirrors:
    - mirrorSourcePolicy: NeverContactSource
      mirrors:
      - ip-10-0-54-104.us-east-2.compute.internal:8443/openshift4
      source: registry.redhat.io/openshift4
    - mirrorSourcePolicy: NeverContactSource
      mirrors:
      - ip-10-0-54-104.us-east-2.compute.internal:8443/modh
      source: quay.io/modh
    - mirrorSourcePolicy: NeverContactSource
      mirrors:
      - ip-10-0-54-104.us-east-2.compute.internal:8443/rhoai
      source: quay.io/rhoai
    - mirrorSourcePolicy: NeverContactSource
      mirrors:
      - ip-10-0-54-104.us-east-2.compute.internal:8443/rhel8
      source: registry.redhat.io/rhel8
    - mirrorSourcePolicy: NeverContactSource
      mirrors:
      - ip-10-0-54-104.us-east-2.compute.internal:8443/integreatly
      source: quay.io/integreatly

You have to create ImageTagMirrorSet to pull images by Tag.

Sample ImageTagMirrorSet.yaml

apiVersion: config.openshift.io/v1
kind: ImageTagMirrorSet
metadata:
  name: generic-0
spec:
  imageTagMirrors:
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - ip-10-0-54-104.us-east-2.compute.internal:8443/openshift4
    source: registry.redhat.io/openshift4
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - ip-10-0-54-104.us-east-2.compute.internal:8443/gogs
    source: registry-1.docker.io/v2/gogs
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - ip-10-0-54-104.us-east-2.compute.internal:8443/rhpds
    source: quay.io/rhpds
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - ip-10-0-54-104.us-east-2.compute.internal:8443/kubebuilder
    source: gcr.io/kubebuilder
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - ip-10-0-54-104.us-east-2.compute.internal:8443/rhel8
    source: registry.redhat.io/rhel8
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - ip-10-0-54-104.us-east-2.compute.internal:8443/minio
    source: quay.io/minio

More information about ImageTagMirrorSet can be found in the official documentation: ImageTagMirrorSet

Common issues in disconnected installations

The disconnected installation requires a lot of resources. Make sure you have enough CPU and memory on the high side to run the OpenShift cluster.
The mirroring process utilizes a large amount of disk space during this process, we required 250GB of available disk space for downloading the images, creating the oc-mirror workspace and finally the tar containing the images for loading the mirror. On the high side, we required at least 450 GB.
The oc-mirror process is very slow, it took us 8 hours to download the images and create the tar file. Make sure you have enough time to complete the process.

Pre-requisites to install ai-accelerator in disconnected cluster

We need kustomize on high-side by downloading on low-side and then rsync it up. For example:

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash
rysnc -avP /mnt/low-side-data/ highside:/mnt/high-side-data/

On high side put kustomize on the path, for example:
```
sudo cp /mnt/high-side-data/kustomize /bin/
```

We need an instance of gitea to clone ai-accelerator and for ArgoCD. For POC, we are using gitea-operator to install gitea on the OpenShift cluster. The following is a sample Gitea CR file that can be used to install gitea on the OpenShift cluster.

apiVersion: pfe.rhpds.com/v1
kind: Gitea
metadata:
  name: simple-gitea
  namespace: gitea
spec:
  giteaImageTag: 1.20.0
  giteaVolumeSize: 4Gi
  giteaSsl: false
  postgresqlVolumeSize: 4Gi
  giteaAdminUser: user
  giteaAdminEmail: user@redhat.com
  giteaAdminPassword: usercredential

Known issue when installing ai-accelerator in disconnected cluster

Certificate error while downloading models from model storage or external storage. This error is seen when the certificates to be trusted are missing from cluster wide certificate authority bundle.

Error message:

Failed to pull model from storage
{"model_id": "fraud_", "error": "rc
error:
code = Unknown desc = Failed to pull model from storage due to error: unable to list objects in bucket
'my-storage': RequestError: send request failedincaused by: Get \"htts://xxxxxxx\": *509: certificate signed by unknown authority"}

Steps to resolve the certificate error

$ oc get secret -n openshift-ingress-operator router-ca -o jsonpath='{.data.tls\.crt}' | base64 -d > openshift-ca-bundle.pem
$ oc get configmap -n openshift-config openshift-service-ca.crt -o jsonpath='{.data.service-ca\.crt}' >> openshift-ca-bundle.pem
$ CA_BUNDLE_FILE=./openshift-ca-bundle.pem
$ oc patch dscinitialization default-dsci --type='json' -p='[{"op":"replace","path":"/spec/trustedCABundle/customCABundle","value":"'"$(awk '{printf "%s\\n", $0}' $CA_BUNDLE_FILE)"'"}]'

For More information on certificates: Working with certificates

Image Pull back off when starting a pipeline. This error is seen when the images are not available in the cluster registry.
Error message:
```
Pulling image "image-registry/rhoai/odh-data-science-pipelines-argo-workflowcontroller-rhel8@sha256:67de22353f12e748992dc4b762eb3818e63c55fc414537b2acdc9a536f9ad308"
Error: ImagePullBackOff
```
Steps to resolve the image pull back off
1. Identify the missing images
2. Pull the images from the Red Hat registry using oc-mirror.
3. Use oc-mirror to create a tar file containing the images and load them into the cluster registry.
4. Use oc-mirror to create a catalog source for the images.
5. Update ImageContentSourcePolicy and ImageTagMirrorSet or ImageDigestMirrorSet to point to the new catalog source.

Hands-on disconnected workshop

If you are interested in a hands-on workshop on how to install Red Hat OpenShift AI in a disconnected environment, you can provision Disconnected Environment on the Red Hat Demo Center. This environment is a fully disconnected environment. You can use this environment to test, modify IamgeSetConfiguration.yaml and learn how to install OpenShift AI in a disconnected environment.