RHOAI Dashboard Configuration

RHOAI Dashboard Resources

Many of the objects in Red Hat OpenShift AI (RHOAI) are tied to an underlying Kubernetes resource. Because of this connection, many RHOAI objects can be created, updated, or deleted by performing the respective action on the underlying resource. Although some of these resources use a custom resource definition (CRD), many do not. For example, Data Science Projects are Red Hat OpenShift projects that include specific labels.

Idle Notebook Culling:

Notebooks and workbenches can consume many compute resources. Additionally, users might forget to shut them down when they are finished, which consumes cluster resources unnecessarily.

The Idle Notebook Culler helps reduce the number of inactive notebooks running on the cluster. It tracks the last time an action was taken in a notebook and shuts down the pods if the notebook has been inactive for a period of time. A notebook is considered inactive when no user has taken an action inside of the notebook. Actions include executing a cell, creating files, or interacting with the user interface.

The Notebook Culler timeout must be between 10 minutes and 1000 hours.

Idle Notebook Culling

To change or enable in the RHOAI Dashboard:

Go to the RHOAI Dashboard>Settings>Cluster Settings>Stop idle notebooks: When culler timeout is enabled and saved, the ConfigMap below will be created. You can enable culling outside of RHOAI Dashboard by applying this ConfigMap to redhat-ods-applications namespace.

Apply ConfigMap in redhat-ods-applications. CULL_IDLE_TIME and IDLENESS_CHECK_PERIOD is in minutes. ENABLE_CULLING is false by default.

kind: ConfigMap
apiVersion: v1
metadata:
  name: notebook-controller-culler-config
  namespace: redhat-ods-applications
  labels:
    opendatahub.io/dashboard: 'true'
data:
  CULL_IDLE_TIME: '60'
  ENABLE_CULLING: 'true'
  IDLENESS_CHECK_PERIOD: '1'

In RHOAI Dashboard>Settings>Cluster Settings>Stop idle notebooks If culling is enabled by ConfigMap, but then disabled in RHOAI Dashboard settings. The ConfigMap notebook-controller-culler-config will be deleted.

Optional: Update the Idle Notebook Culler in forked repository

Because our AI-accelerator project is using GitOps/ArgoCD, we’ll need to update the components/operators/openshift-ai/instance/components/idle-notebook-culling/culler-config.yaml for our changes to be applied. (GitOps is managing the notebook-controller-culler-config ConfigMap, so any changes made in the dashboard will be reverted by GitOps.)

Edit the components/operators/openshift-ai/instance/components/idle-notebook-culling/culler-config.yaml file and change the culling period to 90 minutes:

kind: ConfigMap
apiVersion: v1
metadata:
  name: notebook-controller-culler-config
  namespace: redhat-ods-applications
  labels:
    opendatahub.io/dashboard: 'true'
data:
  CULL_IDLE_TIME: '90'
  ENABLE_CULLING: 'true'
  IDLENESS_CHECK_PERIOD: '1'

After saving the changes and pushing to your forked repository, give the RHOAI Dashboard a few minutes to update. The changes should now be available.

To check your work please refer to This Branch

Workbench and Model Server Sizes

When launching workbenches or model servers from the RHOAI dashboard, users can select from several default sizes. These default options might not suit your organization’s needs. In such cases, you can use custom model server and notebook size configurations. To do this edit the odh-dashboard-config OdhDashboardConfig resource in namespace redhat-ods-applications. You can use the OpenShift Dashboard → Search or navigate to Home → API explorer and filter for OdhDashboardConfig:

apiVersion: opendatahub.io/v1alpha
kind: OdhDashboardConfig
metadata:
  annotations:
    internal.config.kubernetes.io/previousKinds: OdhDashboardConfig
    internal.config.kubernetes.io/previousNames: odh-dashboard-config
    internal.config.kubernetes.io/previousNamespaces: default
  name: odh-dashboard-config
  namespace: redhat-ods-applications
  labels:
    app.kubernetes.io/part-of: rhods-dashboard
    app.opendatahub.io/rhods-dashboard: 'true'
spec:
...output omitted...
  modelServerSizes:
    - name: Small
      resources:
        limits:
          cpu: '2'
          memory: 8Gi
        requests:
          cpu: '1'
          memory: 4Gi
  ...output omitted...
  notebookSizes:
    - name: Small
      resources:
        limits:
          cpu: '2'
          memory: 8Gi
        requests:
          cpu: '1'
          memory: 8Gi
    ...output omitted..

After making changes and saving, give the RHOAI Dashboard a few minutes to update. The changes should now be available.

Customize Model Server and Notebook Sizes in AI-accelerator

Since we are utilizing GitOps/ArgoCD in our cluster, we will need to update our forked project for our changes to be updated/synced on our cluster.

In the /components/operators/openshift-ai/instance/overlays/stable-2.##/kustomization.yaml, the components are referencing:

- ../../components/notebook-pod-sizes
- ../../components/model-server-pod-sizes

This allows GitOps/ArgoCD to enable these components/files in the cluster. So to make changes to the model server and notebook sizes in the cluster, we will need to update the files in the components/operators/openshift-ai/instance/components/model-server-pod-sizes and components/operators/openshift-ai/instance/components/notebook-pod-sizes directories. More specifically, the patch-rhoai-dashboard.yaml files.

Update the components/operators/openshift-ai/instance/components/model-server-pod-sizes/patch-rhoai-dashboard.yaml file to add a new size with the following specifications:
1. name: Small-Medium
2. resources.limits.cpu: 6
3. resources.limits.memory: 8Gi
4. resources.requests.cpu: 4
5. resources.requests.memory: 4Gi

Solution

apiVersion: opendatahub.io/v1alpha
kind: OdhDashboardConfig
metadata:
  name: odh-dashboard-config
spec:
  modelServerSizes:
    - name: Small
      resources:
        limits:
          cpu: '2'
          memory: 8Gi
        requests:
          cpu: '1'
          memory: 4Gi
    - name: Small-Medium
      resources:
        limits:
          cpu: '6'
          memory: 8Gi
        requests:
          cpu: '4'
          memory: 4Gi
    - name: Medium
      resources:
        limits:
          cpu: '8'
          memory: 10Gi
        requests:
          cpu: '4'
          memory: 8Gi
    - name: Large
      resources:
        limits:
          cpu: '10'
          memory: 20Gi
        requests:
          cpu: '6'
          memory: 16Gi

Similarly update the components/operators/openshift-ai/instance/components/notebook-pod-sizes/patch-rhoai-dashboard.yaml file to add a new size with the following specifications:
1. name: Small-Medium
2. resources.limits.cpu: 4
3. resources.limits.memory: 10Gi
4. resources.requests.cpu: 2
5. resources.requests.memory: 8Gi

Solution

apiVersion: opendatahub.io/v1alpha
kind: OdhDashboardConfig
metadata:
  name: odh-dashboard-config
spec:
  notebookSizes:
    - name: Small
      resources:
        limits:
          cpu: '2'
          memory: 8Gi
        requests:
          cpu: '1'
          memory: 8Gi
    - name: Small-Medium
      resources:
        limits:
          cpu: '4'
          memory: 10Gi
        requests:
          cpu: '2'
          memory: 8Gi
    - name: Medium
      resources:
        limits:
          cpu: '6'
          memory: 24Gi
        requests:
          cpu: '3'
          memory: 24Gi
    - name: Large
      resources:
        limits:
          cpu: '14'
          memory: 56Gi
        requests:
          cpu: '7'
          memory: 56Gi
    - name: X Large
      resources:
        limits:
          cpu: '30'
          memory: 120Gi
        requests:
          cpu: '15'
          memory: 120Gi

After making changes and saving, give the RHOAI Dashboard a few minutes to update (5-10 min). The changes should now be available.

Updated Configuration

To check your work please refer to This Branch