Deploying a Text Generation AI App with AWS EKS, ECR, GitOps, Kubernetes, Docker, and Terraform

Introduction

This guide walks through a real-world implementation of a text generation AI application powered by a Generative AI API (e.g., OpenAI, AWS Bedrock, or Hugging Face). The app will be containerized, deployed on AWS EKS, and managed via GitOps using Terraform, Kubernetes, and Helm.

What This Guide Covers:

• Setting up an AI API token securely
• Building a FastAPI-based AI app
• Containerizing the application with Docker
• Deploying on AWS EKS
• Using GitOps for automation
• Managing secrets securely
• Optimizing and troubleshooting deployments

1. Setting Up Generative AI API Token

Most AI APIs require authentication via API tokens. Example services include:

• OpenAI GPT (API key from OpenAI platform)
• AWS Bedrock (IAM role-based authentication)
• Hugging Face API (Access token from Hugging Face)

1.1. Storing API Token Securely in Kubernetes

Use Kubernetes Secrets to store the API token securely:

kubectl create secret generic ai-api-secret --from-literal=API_KEY='your-api-key-here'

Alternatively, use AWS Secrets Manager or HashiCorp Vault for better security.

2. Building the AI-Powered App

Create a FastAPI-based application that interacts with a Generative AI API.

2.1. Install Dependencies

pip install fastapi uvicorn requests python-dotenv

2.2. Implement API Integration

Create app.py to interact with the AI API:

from fastapi import FastAPI
import os
import requests

app = FastAPI()
API_KEY = os.getenv("API_KEY")
API_URL = "https://api.openai.com/v1/completions"

@app.post("/generate")
def generate_text(request: dict):
    headers = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}
    data = {"model": "text-davinci-003", "prompt": request["prompt"], "max_tokens": 100}
    response = requests.post(API_URL, json=data, headers=headers)
    return response.json()

3. Containerizing the Application

3.1. Create Dockerfile

FROM python:3.9
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]

3.2. Build and Push to AWS ECR

docker build -t text-gen-ai-app .
docker tag text-gen-ai-app <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/text-gen-ai-app:v1
docker push <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/text-gen-ai-app:v1

4. Deploying on AWS EKS with GitOps

4.1. Define Kubernetes Deployment

Create deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: text-gen-ai
spec:
  replicas: 2
  selector:
    matchLabels:
      app: text-gen-ai
  template:
    metadata:
      labels:
        app: text-gen-ai
    spec:
      containers:
      - name: text-gen-ai
        image: <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/text-gen-ai-app:v1
        ports:
        - containerPort: 8080
        env:
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: ai-api-secret
              key: API_KEY

4.2. Apply Kubernetes Manifest

kubectl apply -f deployment.yaml

4.3. Expose the Application

Create service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: text-gen-ai-service
spec:
  selector:
    app: text-gen-ai
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: LoadBalancer

Apply the service:

kubectl apply -f service.yaml

5. Automating with GitOps & Terraform

5.1. Define Infrastructure with Terraform

Create eks.tf:

resource "aws_eks_cluster" "main" {
  name     = "text-gen-eks"
  role_arn = aws_iam_role.eks_role.arn
  vpc_config {
    subnet_ids = module.vpc.public_subnets
  }
}

Deploy with:

terraform init && terraform apply -auto-approve

5.2. Enable GitOps with ArgoCD

Install ArgoCD:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Deploy ArgoCD app:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: text-gen-ai-app
spec:
  source:
    repoURL: "https://github.com/your-repo/text-gen-ai"
    path: "k8s"
  destination:
    server: "https://kubernetes.default.svc"
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Apply ArgoCD configuration:

kubectl apply -f argocd-app.yaml

6. Testing the Deployment

6.1. Find the External URL of the Service

kubectl get svc text-gen-ai-service

Copy the EXTERNAL-IP and use it in the next step.

6.2. Send a Test Request

curl -X POST "http://<your-lb-address>/generate" \
     -H "Content-Type: application/json" \
     -d '{"prompt": "What is the meaning of life?"}'

6.3. Check Logs for Debugging

kubectl logs -l app=text-gen-ai -f

6.4. Scale Up Deployment

kubectl scale deployment text-gen-ai --replicas=5

7. FAQs and Troubleshooting

❓ How to secure API tokens in production?

✅ Use AWS Secrets Manager, Kubernetes Secrets, or Vault instead of hardcoding.

❓ How to monitor performance?

✅ Use Prometheus + Grafana for tracking API response times and Kubernetes health.

❓ What if API calls fail?

✅ Implement retry logic and fallback mechanisms for better fault tolerance.

Conclusion

This guide covered deploying a text generation AI app using AWS EKS, ECR, GitOps, Kubernetes, Docker, and Terraform. We included:

• AI API token security
• Containerization
• GitOps automation
• Terraform-based EKS setup

Would you like to see a real-world example added with logging, monitoring, and multi-region deployments? 🚀

📢 Have questions or feedback? Drop a comment below or connect with me on Twitter/X@spysood!

Originally published on Medium.