Configure Text Generation Models¶
KubeAI supports the following engines for text generation models (LLMs, VLMs, ..):
- vLLM (Recommended for GPU)
- Ollama (Recommended for CPU)
- SGLang
- llama.cpp (GGUF models)
- Need something else? Please file an issue on GitHub.
There are 2 ways to install a text generation model in KubeAI:
- Use Helm with the kubeai/models chart.
- Use kubectl apply -f model.yaml to install a Model Custom Resource.
KubeAI comes with pre-validated and optimized Model configurations for popular text generation models. These models are available in the kubeai/models Helm chart and are also published as raw manifests in the manifests/model directory.
You can also easily define your own models using the Model Custom Resource directly or by using the kubeai/models Helm chart.
Install a Text Generation Model using Helm¶
You can take a look at all the pre-configured models in the chart's default values file.
You can get the default values for the models chart using the following command:
helm show values kubeai/models
Install Text Generation Model using L4 GPU¶
Enable the Llama 3.1 8B model using the Helm chart:
helm upgrade --install --reuse-values kubeai-models kubeai/models -f - <<EOF
catalog:
llama-3.1-8b-instruct-fp8-l4:
enabled: true
engine: VLLM
resourceProfile: nvidia-gpu-l4:1
minReplicas: 1 # by default this is 0
EOF
Install a Text Generation Model using kubectl¶
You can use the Model Custom Resource directly to install a model using kubectl apply -f model.yaml.
Install Text Generation Model using L4 GPU¶
Apply the following Model Custom Resource to install the Llama 3.1 8B model using vLLM on L4 GPU:
apiVersion: kubeai.org/v1
kind: Model
metadata:
name: llama-3.1-8b-instruct-fp8-l4
spec:
features: [TextGeneration]
url: hf://neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8
engine: VLLM
args:
- --max-model-len=16384
- --max-num-batched-token=16384
- --gpu-memory-utilization=0.9
- --disable-log-requests
resourceProfile: nvidia-gpu-l4:1
Small NVIDIA GPU examples¶
The model catalog includes small examples for testing SGLang, llama.cpp, and vLLM:
| Engine | Catalog entry | Model format |
|---|---|---|
| SGLang | qwen3-600m-sglang-gpu | Hugging Face weights |
| llama.cpp | qwen3-600m-llamacpp-gpu | GGUF, Q8_0 |
| SGLang | spark-x2.5-4b-sglang-gpu | Spark-X2.5-4B |
| vLLM | spark-x2.5-4b-vllm-gpu | Spark-X2.5-4B |
All examples use nvidia-gpu-t4:1, which requests one NVIDIA GPU with the default KubeAI profile. This profile does not select a T4 specifically; it can also be used on another compatible NVIDIA GPU. Check any profile overrides configured for your cluster. The NVIDIA driver, container runtime and device plugin must be configured before applying the models.
Enable one example with Helm:
helm upgrade --install --reuse-values kubeai-models kubeai/models -f - <<EOF
catalog:
qwen3-600m-sglang-gpu:
enabled: true
minReplicas: 1
EOF
For llama.cpp, replace the catalog entry with qwen3-600m-llamacpp-gpu. For Spark-X2.5-4B, use spark-x2.5-4b-sglang-gpu or spark-x2.5-4b-vllm-gpu. Alternatively, apply the corresponding raw manifest with kubectl apply -f. Raw manifests default to minReplicas: 0; the first inference request triggers model startup.
On a node with one GPU, run one GPU model at a time. Delete the previous Model and wait for its Pod to terminate before starting the next example. For Helm-managed models, disable the previous catalog entry in the same Helm release instead.
Use the Model's metadata.name as the model field in the chat request described below. A ready Pod confirms startup; a successful chat request through KubeAI verifies inference and routing.
These examples use the engine images configured in the KubeAI chart. To test a different image with a raw Model manifest, set spec.image; it takes precedence over the image selected by the resource profile.
For llama.cpp, pass argument names and values as separate entries in spec.args, as shown in the generated manifest.
Spark-X2.5 Special Configuration¶
Spark-X2.5-4B uses specialized parsers for tool calling and reasoning:
SGLang configuration:
- --dtype=half — Forces FP16 for T4 compatibility (T4 doesn't support BF16)
- --tool-call-parser=spark25 — Enables Spark's tool calling format
- --reasoning-parser=qwen3 — Enables reasoning mode with thinking/content separation
- --mem-fraction-static=0.7 — Reserves 70% of GPU memory for KV cache (below 0.8 default to leave headroom for prefill spikes on T4)
- --context-length=8192 — Limits context to 8K tokens on T4
vLLM configuration:
- --dtype=half — Forces FP16 for T4 compatibility
- --trust-remote-code — Allows custom model code execution
- --gpu-memory-utilization=0.85 — Uses 85% of GPU memory (below catalog-standard 0.9 for T4 safety margin)
- --enable-prefix-caching — Optimizes multi-turn conversations with shared prefixes
- --disable-log-requests — Suppresses per-request logging for production use
- --max-model-len=8192 — Limits context to 8K tokens on T4
T4 GPU considerations: Spark-X2.5-4B ships with BF16 weights (~8.2GB), but T4 GPUs (compute capability 7.5) don't support BF16. The catalog entries force FP16 and limit context to 8K tokens. For longer context or better performance, consider: - Using quantized weights (AWQ/GPTQ int4, ~2GB) which reduce memory footprint by 4× - Upgrading to A10/L4/A100 GPUs which support BF16 natively
All these parameters are pre-configured in the catalog entries.
Configure a Chat Template¶
Some models do not ship will chat templates and some engines such as vLLM do not provide a default one. In these cases, you can use .spec.files to inject a template at Pod runtime.
kind: Model
spec:
# ...
engine: VLLM
args:
- --chat-template=/config/chat-template.jinja
files:
- path: /config/chat-template.jinja
content: |
{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content']}}{% if (loop.last and add_generation_prompt) or not loop.last %}{{ '<|im_end|>' + '\n'}}{% endif %}{% endfor %}
{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}{{ '<|im_start|>assistant\n' }}{% endif %}
Interact with the Text Generation Model¶
The KubeAI service exposes an OpenAI compatible API that you can use to query the available models and interact with them.
The KubeAI service is available at http://kubeai/openai/v1 within the Kubernetes cluster.
You can also port-forward the KubeAI service to your local machine to interact with the models:
kubectl port-forward svc/kubeai 8000:80
You can now query the available models using curl:
curl http://localhost:8000/openai/v1/models
Using curl to interact with the model¶
Run the following curl command to interact with the model named llama-3.1-8b-instruct-fp8-l4:
curl "http://localhost:8000/openai/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
"model": "llama-3.1-8b-instruct-fp8-l4",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Write a haiku about recursion in programming."
}
]
}'
Using the OpenAI Python SDK to interact with the model¶
Once the pod is ready, you can use the OpenAI Python SDK to interact with the model: All OpenAI SDKs work with KubeAI since the KubeAI service is OpenAI API compatible.
See the below example code to interact with the model using the OpenAI Python SDK:
import os
from openai import OpenAI
# Assumes port-forward of kubeai service to localhost:8000.
kubeai_endpoint = "http://localhost:8000/openai/v1"
model_name = "llama-3.1-8b-instruct-fp8-l4"
# If you are running in a Kubernetes cluster, you can use the kubeai service endpoint.
if os.getenv("KUBERNETES_SERVICE_HOST"):
kubeai_endpoint = "http://kubeai/openai/v1"
client = OpenAI(api_key="ignored", base_url=kubeai_endpoint)
chat_completion = client.chat.completions.create(
messages=[
{
"role": "user",
"content": "Say this is a test",
}
],
model=model_name,
)
Ollama Configuration Notes¶
Insecure Model Pulling¶
To allow pulling Ollama models from insecure registries (e.g., HTTP-only or self-signed TLS), append the ?insecure=true query parameter to the model URL.
Warning: Use this only in trusted network environments.
Example Model spec using an insecure registry:
spec:
url: ollama://my-local-registry:5000/my-model?insecure=true
Disable Model Pulling¶
To disable model pulling from Ollama, set the pull field to false in the Model spec. This is useful when you want to use a model that is already available in the Ollama server.
Example Model spec to disable pulling:
spec:
url: ollama://my-local-registry:5000/my-model?pull=false