Lab Setup and Prerequisites

Setting up GitOps: Configure automated deployment pipelines that manage our infrastructure and applications

Configuring Secret Management: Set up secure handling of API keys and credentials using HashiCorp Vault

We have created some OpenShift Quick Starts to make the initial configuration steps a bit easier to follow. They also allow us to standardize the environment being used (bash, linux command line, tools).

Figure 1. ETX Quick Starts

Select the first one ETX -1- The Basics and then Click Start > Step 1 - Set your shell

Next Select Run in Web Terminal play button. The Initialize Terminal screen should appear in the bottom half of your browser.

Select Start to run the terminal

Figure 2. ETX 1 The Basics

Continue with the ETX Quick Starts and follow the documentation at the same time. We will point out in the docs if a step or section is covered in a Quick Start using a TIP

Done ✅

You’ll likely either be working in teams of 2 people per cluster or groups of 2 where each person gets their own cluster (if resources are available)

Receive your cluster credentials 🔐

Set up your shared repository (choose one team member to do this):

Setup and run the Web Terminal Operator as described in the Quick Starts

Both team members: Later in the lab we will clone the forked repository to the OpenShift Web Terminal through steps in the Quick Starts, but if you’re running the lab locally you can go ahead and clone the repository locally to your laptop.

git clone git@github.com:your-gh-user/etx-agentic-ai.git
cd etx-agentic-ai

Figure 6. GitHub Repo Clone

Verify your setup ✅

We Recommend using the Quick Starts initially, then your laptop after the initial setup.

You may choose another client to bootstrap from (not recommended especially if you are on MacOSX which is not fully tested). It could be:

Setup env vars and login to OpenShift. You can either use the OpenShift Web Console to grab a login command or use the details below to construct the login command on your own.

export ADMIN_PASSWORD=password # replace with yours
export CLUSTER_NAME=ocp.4ldrd # replace with yours
export BASE_DOMAIN=sandbox2518.opentlc.com # replace with yours

oc login --server=https://api.${CLUSTER_NAME}.${BASE_DOMAIN}:6443 -u admin -p ${ADMIN_PASSWORD}

Done ✅

The following OpenShift ConsoleLinks should already exist in your cluster:

Bootstrap App-of-Apps

# We need to update our Argo CD Apps to point to your team fork
export YOUR_GITHUB_USER=your-gh-user  # the Team member who forked the GitHub Repo
cd etx-agentic-ai   # Navigate to root directory of code base if not already there

Replace the redhat-ai-services throughout the file with your GitHub username.

sed -i "s/redhat-ai-services/${YOUR_GITHUB_USER}/g" infra/app-of-apps/etx-app-of-apps.yaml

Update the redhat-ai-services to your GitHub username in the etx-app-of-apps.yaml file.

for x in $(ls infra/app-of-apps/sno); do
    sed -i "s/redhat-ai-services/${YOUR_GITHUB_USER}/g" infra/app-of-apps/sno/$x
done

Now we can save, commit, and push the changes to your GitHub fork.

# Its not real unless its in git
git add .; git commit -m "using my github fork"; git push

Finally, we can bootstrap the apps into our cluster.

# Bootstrap all our apps
oc apply -f infra/app-of-apps/etx-app-of-apps.yaml

Check the Install progress of the app-of-apps in the Agentic Argo CD

You will need to wait for the individual apps to be installed. This may take a few minutes. After a few minutes, you should see the following output to show that the apps have been installed.

Done ✅

Check what EC2 GPU enabled instance types we have running in our cluster

oc get machines.machine.openshift.io -A

NAMESPACE               NAME                                    PHASE     TYPE          REGION      ZONE         AGE
openshift-machine-api   ocp-kt5tz-master-0                      Running   c6a.2xlarge   us-east-2   us-east-2a   24h
openshift-machine-api   ocp-kt5tz-master-1                      Running   c6a.2xlarge   us-east-2   us-east-2b   24h
openshift-machine-api   ocp-kt5tz-master-2                      Running   c6a.2xlarge   us-east-2   us-east-2c   24h
openshift-machine-api   ocp-kt5tz-worker-gpu-us-east-2a-9vxzv   Running   g6e.2xlarge   us-east-2   us-east-2a   24h
openshift-machine-api   ocp-kt5tz-worker-us-east-2a-fcbcg       Running   m6a.4xlarge   us-east-2   us-east-2a   24h
openshift-machine-api   ocp-kt5tz-worker-us-east-2b-5zx84       Running   m6a.4xlarge   us-east-2   us-east-2b   24h
openshift-machine-api   ocp-kt5tz-worker-us-east-2c-z9xzs       Running   m6a.4xlarge   us-east-2   us-east-2c   24h

In our example, we have a g6e.2xlarge instance type, but your cluster may have a different instance type or GPU configuration. Take a moment to check with your classmates or instructor what instance types are present in your environment.

oc get $(oc get node -o name -l beta.kubernetes.io/instance-type=g6e.2xlarge) -o=jsonpath={.status.allocatable} | python3 -m json.tool

{
  "cpu": "7500m",
  "ephemeral-storage": "114345831029",
  "hugepages-1Gi": "0",
  "hugepages-2Mi": "0",
  "memory": "63801456Ki",
  "nvidia.com/gpu": "1",
  "pods": "250"
}

Label the node with the device-plugin.config that matches the GPU instance product e.g. NVIDIA-L40S for this instance type.

oc label --overwrite node \
    --selector=nvidia.com/gpu.product=NVIDIA-L40S \
    nvidia.com/device-plugin.config=NVIDIA-L40S

Now apply the GPU Cluster Policy and ConfigMap objects that setup Time Slicing - a method to share nvidia gpus.

oc apply -k infra/applications/gpu

After approx ~30sec check the number of allocatable GPUs

oc get $(oc get node -o name -l beta.kubernetes.io/instance-type=g6e.2xlarge) -o=jsonpath={.status.allocatable} | python3 -m json.tool

{
  "cpu": "7500m",
  "ephemeral-storage": "114345831029",
  "hugepages-1Gi": "0",
  "hugepages-2Mi": "0",
  "memory": "63801456Ki",
  "nvidia.com/gpu": "8",
  "pods": "250"
}

Done ✅