Skip to main content
  1. Take it in Bytes/
  2. Tech/OpenShift/
  3. OCP Engineering/

Leveraging OpenAI's LLM on OpenShift for Technical Q&A: A Case Study with Red Hat Documentation

·3 mins· 0
Technical Guide Demo OpenAI OpenShift Open Data Hub Kubeflow AI/ML LLM Red Hat Demo Seldon Model Training Data Science OpenShift
Table of Contents
OpenAI on OpenShift - This article is part of a series.
Part 3: This Article

Synopsis #

In this follow-up post, we’re going to explore how to leverage a locally hosted OpenAI Large Language Model (LLM) on an OpenShift cluster to answer specific technical questions based on a given documentation set. Basically, a locally-hosted and specially trained “ChatGPT”.

We’ll use Red Hat’s extensive documentation as our training set. We’ll also provide specific instructions and example configurations necessary to train the LLM via Kubeflow and Open Data Hub (ODH) on this set. Lastly, we’ll show you how to serve the model on the cluster and how it can be accessed and utilized.

Introduction #

Large Language Models like OpenAI’s GPT-4 have proven to be excellent at understanding and generating human-like text. This capability can be harnessed to create systems that can answer specific technical questions based on a provided documentation set. In this guide, we’ll train such a model on Red Hat’s documentation.

Training OpenAI LLM on Red Hat Documentation #

Training the LLM on Red Hat’s documentation involves two main steps: data preparation and model training.

Data Preparation #

  1. Collect the Documentation Data: Download the documentation from the Red Hat website. This documentation would be in HTML or PDF format, so you’ll need to convert it into plain text for training.

  2. Preprocessing: Clean the text data by removing unnecessary elements like footers, headers, and navigation links. Also, format the data as per the requirements of OpenAI’s training process.

Model Training using Kubeflow and ODH #

We’ll use Kubeflow’s TFJob for training the model and ODH for data ingestion and storage.

  1. Create a TFJob for Model Training:
apiVersion: ""
kind: "TFJob"
  name: "openai-llm-training"
      replicas: 2
      restartPolicy: OnFailure
            - name: tensorflow
              image: <training-image>
                - "--epochs=10"
                - "--data-dir=/mnt/data"
          - name: data
              claimName: <your-pvc>
          - mountPath: "/mnt/data"
            name: data

Replace <training-image> with the Docker image of your OpenAI LLM training script and <your-pvc> with your Persistent Volume Claim for data storage.

Serving the Model #

We’ll use Seldon, available in ODH, to serve our trained model.

  1. Create a SeldonDeployment:
# Create a SeldonDeployment configuration
cat <<EOF | oc apply -f -
kind: SeldonDeployment
  name: openai-llm
  - componentSpecs:
    - spec:
        - image: <openai/llm-trained-image>
          name: classifier
      children: []

      name: classifier
      type: MODEL
    name: default
    replicas: 1

Replace <openai/llm-trained-image> with the Docker image of your trained OpenAI LLM.

Accessing and Utilizing the Model #

Once your model is served, it can be accessed through the REST API endpoint that Seldon provides. This endpoint can be used to ask technical questions based on Red Hat’s documentation.

For example, you can send a POST request with a question as input to the model:

curl -X POST http://<seldon-endpoint>/api/v0.1/predictions -H 'Content-Type: application/json' -d '{"strData": "What is the command to create a new project in OpenShift?"}'

The model will return the answer based on its training on the Red Hat documentation.

Conclusion #

By training an OpenAI LLM on a specific documentation set like Red Hat’s and serving it on an OpenShift cluster, we can create a powerful tool for answering technical questions. This approach can greatly enhance productivity by providing quick, accurate answers to technical questions based on authoritative sources.

References #

  1. OpenShift Documentation
  2. Open Data Hub Documentation
  3. Kubeflow Documentation
  4. Red Hat Documentation
  5. OpenAI Large Language Models

OpenAI on OpenShift - This article is part of a series.
Part 3: This Article