Production-Ready DevSecOps - Part 1: Architecture Design

Most Kubernetes tutorials cover basic deployment, but production systems require more like: secrets management, cost optimization, high availability, and comprehensive security.

This guide presents a battle-tested architecture blueprint demonstrating how to build production-ready applications.

Using Easy Upload, a file processing application, as an example we’ll explore real-world DevSecOps patterns from local development through production deployment that you can adapt for your own applications.

Part 2 will provide the complete implementation guide with code, configurations, and deployment instructions.

What This Architecture Delivers

• Cost Optimization: 50-60% infrastructure cost reduction through GitOps and optimized resource management
• Security: 70-80% fewer vulnerabilities reaching production via shift-left security practices
• Speed: Faster delivery with automated pipelines and ephemeral preview environments

Why GitOps?

GitOps fundamentally transformed the workflow of teams I worked with, delivering greater efficiency, faster shipping, enhanced security, and significant cost savings.

Traditional vs. GitOps Transformation

Traditionally, CI/CD pipelines consist of four 24/7 environments (dev, test, staging, production).

GitOps replaces them with local dev, ephemeral preview environments per PR, and on-demand staging. Production remains the only 24/7 environment, using reserved instances for cost optimization.

Additionally, GitOps tools like ArgoCD guarantee automated deployments with Git as a source of truth, eliminating drift, instant rollbacks, and complete audit trail through Git history.

Results

• 50-60% cost reduction across teams
• Improved security through automated scanning and GitOps practices
• Faster code delivery with streamlined, automated processes
• Effortless scaling without operational overhead
• Enhanced reliability through automated rollbacks and drift prevention

Why DevSecOps?

Traditionally, security testing is only at the end, which creates bottlenecks and expensive fixes.

DevSecOps shifts security left with continuous security from development through production with SAST/DAST scan, and dependency checks integrated from the start.

Results & Benefits

• 70-80% reduction in vulnerabilities reaching production
• Early detection: Vulnerabilities caught during development, not testing
• Reduced costs: Early fixes cost significantly less than production remediation
• Faster time to market with security built-in

Application Architecture Overview

Easy Upload is a web application that handles file uploads and processing. This design demonstrates real-world complexity, perfect for showcasing DevSecOps architecture patterns.

Demo of App Working

How it Works

1. Frontend → API:
   • A static Astro frontend uploads files to the API Service.
2. API → S3 & Queue:
   • The API stores files in temporary S3 bucket (raw-uploads) and enqueues jobs in BullMQ for background processing
3. Worker → Processed Storage & Database:
   • Worker Service consumes BullMQ jobs, processes files, uploads results to processed-uploads S3 bucket, and updates PostgreSQL metadata
4. Real-time Updates → Frontend:
   • Changes pushed back via Server-Sent Events (SSE) using Redis Pub/Sub channels

Architecture Benefits

• BFF Pattern: Keeps frontend simple and cheap (S3 + CDN) while API handles all business logic.
• Async Processing: Heavy file processing runs separately from user-facing API, ensuring fast responses and independent scaling.
• Cost Optimization: S3 lifecycle policies automatically clean up temporary files to prevent storage bloat.
• Independent Scaling: Frontend, API, and worker services scale based on their specific demands.

Infrastructure Stack

The infrastructure is heavily automated, using managed cloud services balanced with Kubernetes workloads for simplicity, security, and flexibility.

Cloud Infrastructure Diagram

• 

Cloud & Infrastructure

Using vendor-agnostic managed services (like AWS RDS with PostgreSQL) for flexibility without lock-in:

• Cloud Provider: AWS
• Infrastructure as Code: Terraform
• Container Registry: AWS ECR
• Kubernetes Cluster: AWS EKS
• CI: GitHub Actions
• CD / GitOps: Argo CD
• Database: AWS RDS PostgreSQL (Multi-AZ)
• Cache / Queue: AWS ElastiCache Redis with BullMQ (Multi-AZ)
  • Note: BullMQ chosen over RabbitMQ to reduce infrastructure overhead and provide better Node.js integration for producer-consumer patterns
• DNS & TLS: AWS Route 53 & AWS Certificate Manager
• CDN / Entry Point: AWS CloudFront
• File Storage (Buckets): AWS S3
• Static Frontend & Assets: S3 storage with CloudFront delivery (private origin and cached)
• Load Balancing: AWS Network Load Balancer → Nginx Ingress

Scaling & High Availability

• Horizontal Pod Autoscaler: API & Worker pods scale based on CPU/memory
• Karpenter: Rapid EKS node provisioning with On-Demand + Spot instances
• Multi-AZ Deployments: RDS & ElastiCache
• Cost Management: Terraform resource tagging

Observability

• Logs: JSON logs → Fluent Bit → Loki
• Metrics: Prometheus collection for infrastructure (CPU/Memory), Kubernetes (Pod Status), and application-level metrics (error rates)
• Visualization: Grafana dashboards for API, Worker, and cluster health
• Alerting: Prometheus Alertmanager → Slack/PagerDuty with symptom-based alerts (e.g., high API errors)

Cloud & Infrastructure Security

• AWS WAF on CloudFront
• Network Policies: Kubernetes restricts pod ingress/egress
• Pod Identity: EKS Pod Identity grants fine-grained permissions (e.g., specific S3 access)
• Secrets Management: AWS Secrets Manager with External Secrets Operator (ESO)

Why This Infrastructure Design

• EKS + Managed Services: Perfect balance of control (Kubernetes workloads) and simplicity (AWS managed database, cache, secrets).
• Terraform: Ensures reproducible, auditable infrastructure with mandatory change reviews.
• Multi-AZ + Autoscaling: Guarantees high availability while optimizing costs through elastic scaling.
• Comprehensive Observability: Proactive monitoring and alerting prevents downtime and speeds up incident resolution.
• Security by Design: WAF, network policies, pod identity, and secret management enforce least privilege and limit breach impact.

Platform Infrastructure Pipeline (Terraform)

The infrastructure defined in Terraform has a dedicated pipeline ensuring all infrastructure changes are code-reviewed, security-scanned, and automatically applied.

Repository: platform-infra-repo

Workflow

1. Infrastructure Change: Developer modifies Terraform configuration
2. Pull Request: Triggers a GitHub Action for automated checks:
   • Linting: tflint
   • Security Scan: trivy config .
   • Plan Review: terraform plan output is posted as a PR comment.
3. Merge: On approval, a protected GitHub Action runs terraform apply.

Why This Approach

• Code-Reviewed Infrastructure: Eliminates risky manual console changes through mandatory PR reviews.
• Automated Security Scanning: Catches misconfigurations before they reach production.
• Plan + Apply Separation: Safer deployments with full visibility into changes before execution.
• Git as Source of Truth: Complete audit trail of every infrastructure modification.

Application CI/CD Pipeline (GitOps)

• Development Model: Trunk-Based Development
  • All developers work on short-lived feature branches off the main branch
  • Feature branches merged frequently to reduce integration conflicts
  • Keeps main branch always deployable
  • Encourages continuous integration and fast feedback
  • Paired with PR previews for testing new features in ephemeral environments
• Repositories:
  • app-repo (Monorepo)
    • Contains all application source code
    • Monorepo simplifies dependency management and ensures version consistency
    • Packages separated for easy future microservices migration
  • config-repo (GitOps repo)
    • Stores Kubernetes manifests for staging and production.
    • Separate from app code for independent deployments.
    • GitOps Approach: CI updates manifests; Argo CD deploys automatically.

CI/CD Pipeline Workflow Diagram

• 

CI/CD Security

• Static Code Analysis (SCA): Trivy for dependencies, IaC, and containers.
• Linting: ESLint, and Prettier
• Vulnerability Scanning: npm audit for known package vulnerabilities.
• Dependency Updates: Renovate for automated dependency PRs on weekly schedule.
• SAST: Semgrep for PRs and pre-commit hooks.
• DAST: OWASP ZAP for staging and preview environments.
• E2E Testing: Playwright.

Workflow

1. Feature Development
   • Developer creates a feature branch to work on a new feature.
2. Pre-commit (Optional)
   • Format code with ESLint + Prettier.
   • Run quick SAST scan using Semgrep.
3. Pull Request (CI)
   1. Merge request from feature branch to main triggers CI pipeline.
   2. Run Tests & Static Scans:
      • Unit & integration tests.
      • Full SAST scan with Semgrep.
      • SCA scan with Trivy on package.json.
      • npm audit for known package vulnerabilities.
4. Preview (Optional)
   • Deploy PR changes to ephemeral cloud environment using /deploy-preview comment
   • Features:
     • Fast & cheap builds via cached Docker layers and Kubernetes diffs
     • Realistic data using cloud services and “golden snapshot” database
     • Optional OWASP ZAP DAST scan against preview URL
5. Staging Pipeline
   1. Build production-ready assets & images:
      • Frontend → static S3 assets
      • API & Worker → Docker images
   2. Security Scans: Full Trivy scan of Docker images
   3. Push Artifacts:
      • Frontend assets to S3
      • API & Worker images to AWS ECR with unique tags (e.g., api:v1.2.3, worker:v1.2.3)
   4. Update Kubernetes manifests in config-repo with API & Worker image tags.
6. Staging Deployment
   • Argo CD deploys updated manifests to staging cluster.
   • E2E tests run via Playwright.
   • Full DAST scan using OWASP ZAP.
7. Production Promotion & Rollout
   1. Update production manifests in config-repo with validated image tags.
   2. Argo CD triggers Argo Rollouts for canary deployment:
      • Gradual rollout (e.g., 5% → 25% → 100%).
      • Monitored via Prometheus metrics (error rates, latency).
      • Automatic rollback on failure.

Why This Approach

• Trunk-Based Development: Keeps main branch always deployable and minimizes merge conflicts.
• Preview Environments: Enable rapid realistic feedback without disrupting staging environment.
• Separated Repositories: App code and deployment configs evolve independently, reducing coupling risks.
• Shift-Left Security: Automated scanning catches issues early without slowing down developers.
• Canary Deployments: Argo Rollouts provide safe, automated production rollouts with instant rollback capability.

Local Development Strategy

The local development environment mirrors production architecture while enabling rapid iteration. Developers can spin up the complete stack (API, worker services, PostgreSQL, Redis, and S3-compatible storage) with a single command.

Development Workflow

Development follows a streamlined process that maintains consistency across the team:

1. Initial Setup:
   • This launches all services (api, worker, Postgres, Redis, Minio) in Docker Compose, providing immediate access to a production-like environment.

   git clone https://github.com/andrecrafts/easyupload
   cd easyupload
   docker compose up --build
2. Create a feature branch for new work:

   git pull origin main
   git checkout -b feature/new-functionality
3. Start the environment with optional rebuild if dependencies changed:

   docker compose build 
   docker compose up
4. Develop with hot-reloading:

   # Example: Install new packages directly in running containers:
   docker compose exec api npm install <package>
5. Submit changes by committing and opening a pull request to main:
   • Pre-commit hooks automatically format code and run quick static analysis.

   git add .
   git commit -m "Add new functionality"
   git push origin feature/new-functionality

Features & Benefits

• Production Parity: Docker Compose mirrors production services with shared configuration, eliminating “works on my machine” issues
• Quick Onboarding: Standardized Docker Compose setup enables new developers to get productive quickly
• Fast Feedback Loop: Hot-reloading and pre-commit hooks provide immediate local validation before CI pipeline.
• Flexible Preview Options:
  • Full-stack previews: Deploy PRs to ephemeral cloud environments (frontend, backend, workers) for complete validation before staging/production.
  • Hybrid previews: Spin up ephemeral cloud dependencies (S3, RDS, Redis) and connect them to local Docker Compose for testing features without full stack costs.
• Development in Containers: VSCode’s Dev Containers extension enables development directly inside containers with full IntelliSense and debugging.

Complete Request Flow

The entire lifecycle of a user action demonstrates how all components work together:

1. User Request: User visits https://www.easyupload.com
2. Security Layer: CloudFront routes request, WAF inspects for threats
3. Frontend Delivery: Static frontend served from S3 via CloudFront Origin Access Control (OAC)
4. API Routing: API requests flow through Network Load Balancer → Ingress → API pod
5. File Processing:
   • API streams upload to S3 raw bucket
   • Records metadata in RDS
   • Enqueues job in Redis
6. Background Processing:
   • Worker processes file
   • Updates processed bucket
   • Updates database
   • Publishes Redis Pub/Sub updates
7. Real-time Updates: User receives progress updates and signed URLs for downloads

Complete Request Flow Diagram

• 

Conclusion

This architecture demonstrates how GitOps workflows, shift-left security, and cloud-native patterns work together to reduce costs, strengthen security, and accelerate delivery cycles.

Beyond the technical stack, this approach eliminates manual deployment risks and operational overhead allowing teams to focus on building features instead of managing infrastructure.

What’s Next?

In Part 2, I will publish the implementation guide, including:

• Dockerfiles and Docker Compose for local dev
• Terraform IaC for AWS resources
• GitHub Actions workflows
• Kubernetes manifests with ArgoCD and Argo Rollouts
• Screenshots of successful deployments and the app live on AWS