📚 BusinessExplain 🏠 Home 📊 Analytics
← Back to Skills

Ci Cd Mastery

⚙️ DevOps & Infrastructure 22 min read v1.0.0
📋 Contents

CI/CD Mastery

End-to-end guide for building, testing, securing, and deploying software with continuous integration and continuous delivery pipelines.

1. Pipeline Fundamentals

Core Concepts

| Concept | Definition |
|---|---|
| Continuous Integration (CI) | Automatically build and test every commit to detect regressions early. |
| Continuous Delivery (CD) | Every commit that passes CI is ready to deploy; deployment is manual gate. |
| Continuous Deployment | Every passing commit is deployed automatically — no human gate. |
| Pipeline | Ordered set of stages/jobs that transform source code into running software. |
| Stage | Logical group of jobs that run in sequence (build → test → deploy). |
| Job / Step | Atomic unit of work inside a stage — runs on a single agent/runner. |
| Artifact | Immutable file produced by a job and consumed downstream. |
| Cache | Mutable store to speed up dependency resolution across runs. |

Pipeline Anatomy

┌──────────┐   ┌──────────┐   ┌───────────┐   ┌──────────┐   ┌──────────┐
│  Source   │──▶│  Build   │──▶│   Test    │──▶│  Stage   │──▶│  Deploy  │
│ Checkout  │   │ Compile  │   │ Unit/Int  │   │ Artifact │   │  Prod    │
└──────────┘   └──────────┘   └───────────┘   └──────────┘   └──────────┘

Design Principles

  • Fail fast — run cheap/fast tests first; expensive E2E tests later.
  • Idempotent — re-running a job produces the same result.
  • Deterministic — pin dependencies & toolchains; no floating tags.
  • Minimal secrets — least-privilege tokens scoped to the job.
  • Observable — structured logs, metrics on duration/failure rates.
  • Immutable artifacts — build once, promote the same artifact through environments.

    • 2. GitHub Actions

    Workflow Syntax

    # .github/workflows/ci.yml
name: CI

    on:
      push:
        branches: [main]
      pull_request:
        branches: [main]
    

    concurrency:
      group: ${{ github.workflow }}-${{ github.ref }}
      cancel-in-progress: true
    

    permissions:
      contents: read
      id-token: write       # OIDC for cloud auth
      security-events: write # CodeQL
    

    env:
      REGISTRY: ghcr.io
      IMAGE_NAME: ${{ github.repository }}
    

    jobs:
      build:
        runs-on: ubuntu-latest
        outputs:
          version: ${{ steps.meta.outputs.version }}
        steps:
          - uses: actions/checkout@v4
    

    - uses: actions/setup-node@v4
            with:
              node-version: 20
              cache: npm
    

    - run: npm ci
          - run: npm run build
    

    - uses: actions/upload-artifact@v4
            with:
              name: dist
              path: dist/
              retention-days: 5
    

    test:
        needs: build
        runs-on: ubuntu-latest
        strategy:
          matrix:
            shard: [1, 2, 3]
        steps:
          - uses: actions/checkout@v4
          - uses: actions/setup-node@v4
            with:
              node-version: 20
              cache: npm
          - run: npm ci
          - run: npx vitest run --shard=${{ matrix.shard }}/3
    

    security:
        needs: build
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
          - uses: github/codeql-action/analyze@v3
            with:
              languages: javascript
    

    docker:
        needs: [build, test]
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
    

    - uses: actions/download-artifact@v4
            with:
              name: dist
              path: dist/
    

    - uses: docker/login-action@v3
            with:
              registry: ${{ env.REGISTRY }}
              username: ${{ github.actor }}
              password: ${{ secrets.GITHUB_TOKEN }}
    

    - uses: docker/metadata-action@v5
            id: meta
            with:
              images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
              tags: |
                type=ref,event=branch
                type=sha,prefix=
                type=semver,pattern={{version}}
    

    - uses: docker/build-push-action@v5
            with:
              context: .
              push: true
              tags: ${{ steps.meta.outputs.tags }}
              labels: ${{ steps.meta.outputs.labels }}
              cache-from: type=gha
              cache-to: type=gha,mode=max
    

    deploy-staging:
        needs: docker
        runs-on: ubuntu-latest
        environment: staging
        steps:
          - run: echo "Deploy ${{ needs.docker.outputs.version || 'latest' }} to staging"
    

    deploy-prod:
        needs: deploy-staging
        runs-on: ubuntu-latest
        environment: production
        steps:
          - run: echo "Deploy ${{ needs.docker.outputs.version || 'latest' }} to production"

    Actions Matrix

    jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node: [18, 20, 22]
        exclude:
          - os: windows-latest
            node: 18
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}

    Reusable Workflows

    # .github/workflows/_deploy.yml (reusable)
on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
      image_tag:
        required: true
        type: string
    secrets:
      DEPLOY_TOKEN:
        required: true

    jobs:
      deploy:
        runs-on: ubuntu-latest
        environment: ${{ inputs.environment }}
        steps:
          - run: |
              curl -X POST https://api.example.com/deploy \
                -H "Authorization: Bearer ${{ secrets.DEPLOY_TOKEN }}" \
                -d "{\"tag\": \"${{ inputs.image_tag }}\"}"

    # .github/workflows/ci.yml (caller)
jobs:
  deploy-staging:
    uses: ./.github/workflows/_deploy.yml
    with:
      environment: staging
      image_tag: ${{ needs.docker.outputs.version }}
    secrets:
      DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

    OIDC Authentication

    # AWS

  • uses: aws-actions/configure-aws-credentials@v4
    • 
  with:
    role-to-assume: arn:aws:iam::123456789012:role/GitHubOIDC
    aws-region: us-east-1

    GCP
    

  • uses: google-github-actions/auth@v2
    • 
  with:
    workload_identity_provider: projects/123/locations/global/workloadIdentityPools/github/providers/actions
    service_account: deploy@project.iam.gserviceaccount.com

    Azure
    

  • uses: azure/login@v2
    • 
  with:
    client-id: ${{ secrets.AZURE_CLIENT_ID }}
    tenant-id: ${{ secrets.AZURE_TENANT_ID }}
    subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

    Caching Strategies

    # Native cache via setup-* actions

  • uses: actions/setup-node@v4
    • 
  with:
    node-version: 20
    cache: npm          # auto-caches ~/.npm

    Manual cache
    

  • uses: actions/cache@v4
    • 
  with:
    path: |
      ~/.cache/pip
      ~/.local/bin/cog
    key: python-${{ runner.os }}-${{ hashFiles('requirements.txt') }}
    restore-keys: python-${{ runner.os }}-

    Docker layer cache via buildx
    

  • uses: docker/build-push-action@v5
    • 
  with:
    cache-from: type=gha
    cache-to: type=gha,mode=max

    3. GitLab CI

    .gitlab-ci.yml

    stages:
  - build
  - test
  - security
  - package
  - deploy

    variables:
      REGISTRY: $CI_REGISTRY
      IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
      DOCKER_DRIVER: overlay2
    

    default:
      image: node:20-alpine
      cache:
        key:
          files:
            - package-lock.json
        paths:
          - node_modules/
      before_script:
        - npm ci --ignore-scripts
    

    ---------- BUILD ----------
    
build:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour

    ---------- TEST ----------
    
test:unit:
  stage: test
  script:
    - npx vitest run --coverage
  coverage: '/All files[^|]\|[^|]\s+([\d.]+)/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

    test:integration:
      stage: test
      services:
        - postgres:16-alpine
      variables:
        POSTGRES_DB: testdb
        POSTGRES_USER: test
        POSTGRES_PASSWORD: test
        DATABASE_URL: postgres://test:test@postgres:5432/testdb
      script:
        - npx vitest run --config vitest.integration.config.ts
    

    ---------- SECURITY ----------
    
sast:
  stage: security
  trigger:
    include:
      - template: Security/SAST.gitlab-ci.yml

    container_scanning:
      stage: security
      trigger:
        include:
          - template: Security/Container-Scanning.gitlab-ci.yml
    

    ---------- PACKAGE ----------
    
docker:build:
  stage: package
  image: docker:24
  services:
    - docker:24-dind
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY
  script:
    - docker build -t $IMAGE .
    - docker push $IMAGE
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_COMMIT_TAG

    ---------- DEPLOY ----------
    
.deploy_template:
  stage: deploy
  image: bitnami/kubectl:latest
  script:
    - kubectl config use-context $KUBE_CONTEXT
    - kubectl set image deployment/$APP $APP=$IMAGE -n $NAMESPACE
    - kubectl rollout status deployment/$APP -n $NAMESPACE --timeout=120s

    deploy:staging:
      extends: .deploy_template
      variables:
        APP: myapp
        NAMESPACE: staging
      environment:
        name: staging
        url: https://staging.example.com
      rules:
        - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    

    deploy:production:
      extends: .deploy_template
      variables:
        APP: myapp
        NAMESPACE: production
      environment:
        name: production
        url: https://app.example.com
      rules:
        - if: $CI_COMMIT_TAG
      when: manual
    

    ---------- REVIEW APPS ----------
    
review:
  extends: .deploy_template
  variables:
    APP: review-${CI_MERGE_REQUEST_IID}
    NAMESPACE: review
  environment:
    name: review/$CI_MERGE_REQUEST_IID
    url: https://review-${CI_MERGE_REQUEST_IID}.example.com
    on_stop: review:stop
  rules:
    - if: $CI_MERGE_REQUEST_IID

    review:stop:
      extends: .deploy_template
      stage: deploy
      script:
        - kubectl delete namespace review-$CI_MERGE_REQUEST_IID
      environment:
        name: review/$CI_MERGE_REQUEST_IID
        action: stop
      rules:
        - if: $CI_MERGE_REQUEST_IID
          when: manual

    GitLab CI Key Features

    | Feature | Syntax | Purpose |
    |---|---|---|
    | extends | extends: .template | DRY job config via inheritance |
    | rules | rules: [{ if, when, ... }] | Conditional job inclusion |
    | trigger | trigger: { include: ... } | Multi-project / downstream pipelines |
    | needs | needs: [job] | DAG execution — run out of stage order |
    | services | services: [postgres] | Sidecar containers for tests |
    | artifacts:reports | reports: { cobertura, junit } | Native report ingestion |
    | environment | environment: { name, url } | Deployment tracking & protection |
    | cache | cache: { key, paths } | Dependency caching across pipelines |

    4. Jenkins

    Declarative Jenkinsfile

    // Jenkinsfile (Declarative Pipeline)
pipeline {
  agent none

    options {
        buildDiscarder(logRotator(numToKeepStr: '30'))
        timeout(time: 30, unit: 'MINUTES')
        disableConcurrentBuilds()
        timestamps()
      }
    

    environment {
        REGISTRY = 'ghcr.io'
        IMAGE    = "${REGISTRY}/myorg/myapp"
        DOCKER_CONFIG = credentials('docker-registry-auth')
      }
    

    triggers {
        pollSCM('H/5    ')
        upstream(upstreamProjects: 'infrastructure/deploy', threshold: hudson.model.Result.SUCCESS)
      }
    

    stages {
        stage('Build') {
          agent { label 'node20' }
          steps {
            sh 'npm ci && npm run build'
          }
          post {
            always {
              stash name: 'dist', includes: 'dist/**'
            }
          }
        }
    

    stage('Test') {
          parallel {
            stage('Unit') {
              agent { label 'node20' }
              steps {
                unstash 'dist'
                sh 'npx vitest run'
              }
              post {
                always { junit 'reports/junit.xml' }
              }
            }
            stage('Lint') {
              agent { label 'node20' }
              steps {
                sh 'npm run lint'
              }
            }
          }
        }
    

    stage('Security Scan') {
          agent { label 'node20' }
          steps {
            sh 'npx audit-ci --moderate'
            // Trivy FS scan
            sh 'trivy fs --exit-code 1 --severity HIGH,CRITICAL .'
          }
        }
    

    stage('Docker Build & Push') {
          agent { label 'docker' }
          steps {
            unstash 'dist'
            script {
              def tag = env.GIT_COMMIT?.take(8) ?: env.BUILD_NUMBER
              docker.withRegistry("https://${REGISTRY}", 'docker-registry-auth') {
                def img = docker.build("${IMAGE}:${tag}", '--build-arg VERSION=${tag} .')
                img.push()
                if (env.GIT_BRANCH == 'main') {
                  img.push('latest')
                }
              }
            }
          }
        }
    

    stage('Deploy Staging') {
          agent { label 'kubectl' }
          steps {
            sh "kubectl set image deployment/myapp myapp=${IMAGE}:${env.GIT_COMMIT.take(8)} -n staging"
            sh 'kubectl rollout status deployment/myapp -n staging --timeout=120s'
          }
        }
    

    stage('Deploy Production') {
          agent { label 'kubectl' }
          input {
            message 'Deploy to production?'
            ok 'Deploy'
          }
          steps {
            sh "kubectl set image deployment/myapp myapp=${IMAGE}:${env.GIT_COMMIT.take(8)} -n production"
            sh 'kubectl rollout status deployment/myapp -n production --timeout=120s'
          }
        }
      }
    

    post {
        failure { slackSend color: 'danger', message: "Build ${env.JOB_NAME} #${env.BUILD_NUMBER} failed!" }
        success { slackSend color: 'good', message: "Build ${env.JOB_NAME} #${env.BUILD_NUMBER} succeeded." }
      }
    }

    Shared Libraries

    // vars/deployK8s.groovy (shared library)
def call(Map config) {
  sh """
    kubectl set image deployment/${config.app} ${config.app}=${config.image} \
      -n ${config.namespace}
    kubectl rollout status deployment/${config.app} \
      -n ${config.namespace} --timeout=${config.timeout ?: '120s'}
  """
}

    // vars/dockerBuildPush.groovy
    def call(Map config) {
      docker.withRegistry("https://${config.registry}", config.credId) {
        def img = docker.build("${config.image}:${config.tag}", config.context ?: '.')
        img.push()
        img.push('latest')
      }
    }

    // Jenkinsfile using shared lib
@Library('my-pipeline-lib') _

    pipeline {
      agent any
      stages {
        stage('Deploy Staging') {
          steps {
            deployK8s app: 'myapp', image: 'ghcr.io/org/myapp:abc1234', namespace: 'staging'
          }
        }
      }
    }

    Multibranch Pipeline

    // Automatic branch discovery in Jenkins multibranch project
// Each branch with a Jenkinsfile gets its own pipeline
// PRs get pipelines via GitHub/Branch API integration
// Branch indexing triggers on push via webhook

    Jenkins best practices:

  • Pin plugin versions; use plugin.yaml for reprovisioning.

  • Use agent none at top level; label each stage.

  • Stash/unstash instead of checking out multiple times.

  • Use shared libraries for cross-team patterns.

  • Keep Jenkinsfiles in source control (pipeline-as-code).

    • 5. Deployment Strategies

    Blue-Green Deployment

    Two identical environments. Switch traffic atomically.

    # Kubernetes blue-green via Service selector swap
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
    slot: blue    # ← toggle this value to switch
  ports:
    - port: 80
      targetPort: 8080

    # Deploy new version to green
kubectl apply -f deployment-green.yaml

    Verify green is healthy
    
kubectl wait --for=condition=Available deploy/myapp-green --timeout=120s

    Run smoke tests against green
    
curl -sf http://green.internal/health

    Swap Service selector to green
    
kubectl patch svc myapp -p '{"spec":{"selector":{"slot":"green"}}}'

    Rollback: patch selector back to blue

    Canary Deployment

    Gradually shift traffic to the new version.

    # Istio VirtualService — 10% canary
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts: [myapp.example.com]
  http:
    - route:
        - destination:
            host: myapp
            subset: stable
          weight: 90
        - destination:
            host: myapp
            subset: canary
          weight: 10

    # Argo Rollouts Canary
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - setWeight: 30
        - pause: { duration: 5m }
        - setWeight: 60
        - pause: { duration: 5m }
        - setWeight: 100
      canaryService: myapp-canary
      stableService: myapp-stable

    Rolling Update (Kubernetes default)

    spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%       # pods above desired count during rollout
      maxUnavailable: 0   # never go below desired count

    A/B Testing (Feature Flags)

    # LaunchDarkly / Unleash / custom feature-flag evaluation
if feature_flags.is_enabled("new-checkout", user_id, default=False):
    return new_checkout(request)
return legacy_checkout(request)

    # Unleash feature toggle config
apiVersion: unleash.example.com/v1
kind: FeatureToggle
metadata:
  name: new-checkout
spec:
  strategy: gradualRollout
  parameters:
    percentage: 25
    groupId: checkout-experiment

    Feature Flag Deployment Flow

    ┌───────────┐   ┌──────────────┐   ┌──────────────┐
│  Deploy    │──▶│  Feature OFF │──▶│  Monitor     │
│  (anytime) │   │  in prod     │   │  no impact   │
└───────────┘   └──────┬───────┘   └──────────────┘
                       │
                       ▼  Toggle ON
                ┌──────────────┐   ┌──────────────┐
                │  Feature ON  │──▶│  Rollback:   │
                │  Gradual %   │   │  Toggle OFF  │
                └──────────────┘   └──────────────┘

    Strategy Comparison

    | Strategy | Downtime | Rollback Speed | Cost | Complexity |
    |---|---|---|---|---|
    | Blue-Green | Zero | Instant (selector swap) | 2× resources | Medium |
    | Canary | Zero | Fast (shift traffic) | 1.1× resources | High |
    | Rolling | Near-zero | Medium (re-rollout) | 1× resources | Low |
    | Feature Flags | Zero | Instant (toggle off) | 1× resources | Medium |

    6. Release Automation

    Semantic Versioning (SemVer)

    MAJOR.MINOR.PATCH
  │    │    └── bug fixes (backward compatible)
  │    └────── new features (backward compatible)
  └─────────── breaking changes

    Automated Versioning with Conventional Commits

    # .github/workflows/release.yml
name: Release

    on:
      push:
        branches: [main]
    

    permissions:
      contents: write
      pull-requests: write
    

    jobs:
      release:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
            with:
              fetch-depth: 0    # full history for versioning
    

    - uses: actions/setup-node@v4
            with:
              node-version: 20
    

    - name: Release
            env:
              GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
            run: |
              npx semantic-release

    .releaserc.json (semantic-release)

    {
  "branches": ["main", { "name": "next", "prerelease": true }],
  "plugins": [
    "@semantic-release/commit-analyzer",
    "@semantic-release/release-notes-generator",
    "@semantic-release/changelog",
    ["@semantic-release/exec", {
      "prepareCmd": "npm version ${nextRelease.version} --no-git-tag-version"
    }],
    ["@semantic-release/git", {
      "assets": ["package.json", "package-lock.json", "CHANGELOG.md"],
      "message": "chore(release): ${nextRelease.version}"
    }],
    "@semantic-release/github"
  ]
}

    Changelog Generation

    # git-cliff (Rust-based, fast)
git-cliff --output CHANGELOG.md

    cliff.toml
    
[changelog]
header = """# Changelog\n\nAll notable changes to this project will be documented.\n"""
body = """
{% if version %}## [{{ version | trim_start_matches(pat="v") }}] - {{ timestamp | date(format="%Y-%m-%d") }}
{% else %}## [Unreleased]
{% endif %}{% for commit in commits %}- {{ commit.message | trim_end_matches(pat="\\n") }}{% endfor %}"""
trim = true

    GitHub Releases in CI

    # .github/workflows/publish-release.yml
name: Publish Release

    on:
      push:
        tags: ['v*']
    

    jobs:
      release:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
    

    - uses: actions/setup-node@v4
            with:
              node-version: 20
              cache: npm
    

    - run: npm ci
          - run: npm run build
    

    - name: Create GitHub Release
            uses: softprops/action-gh-release@v2
            with:
              generate_release_notes: true
              files: |
                dist/*.tar.gz
                dist/*.sha256
    

    - name: Publish to npm
            run: npm publish --access public
            env:
              NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

    7. Docker / OCI Image Building in CI

    Multi-stage Dockerfile

    # Build stage
FROM node:20-alpine AS builder
WORKDIR /app
COPY package-lock.json package.json ./
RUN npm ci --ignore-scripts
COPY . .
RUN npm run build

    Production stage
    
FROM gcr.io/distroless/nodejs20-debian12:latest
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
USER nonroot:nonroot
EXPOSE 3000
CMD ["dist/server.js"]

    BuildKit + Cache in GitHub Actions

    - uses: docker/setup-buildx-action@v3

  • uses: docker/build-push-action@v5
    • 
  with:
    context: .
    push: true
    tags: ${{ steps.meta.outputs.tags }}
    cache-from: type=gha
    cache-to: type=gha,mode=max
    platforms: linux/amd64,linux/arm64
    provenance: true     # SLSA provenance
    sbom: true            # SBOM attestation

    Container Scanning

    # Trivy — run in CI

  • name: Run Trivy vulnerability scanner
    • 
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: '${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}'
    format: 'sarif'
    output: 'trivy-results.sarif'
    exit-code: '1'
    severity: 'CRITICAL,HIGH'

  • uses: github/codeql-action/upload-sarif@v3
    • 
  with:
    sarif_file: 'trivy-results.sarif'

    8. Testing in CI

    Test Pyramid in Pipelines

             ╱  E2E        ╲   — slow, flaky, few
        ╱  Integration   ╲  — medium speed, moderate count
       ╱    Unit Tests    ╲ — fast, many, deterministic

    Parallel Test Sharding

    # Vitest / Jest shards in GitHub Actions
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - run: npx vitest run --shard=${{ matrix.shard }}/4

    Integration Tests with Services

    # GitHub Actions — Postgres service
jobs:
  integration:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_DB: testdb
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
        ports: ['5432:5432']
        options: >-
          --health-cmd="pg_isready"
          --health-interval=10s
          --health-timeout=5s
          --health-retries=5
    steps:
      - run: npx vitest run --config vitest.integration.config.ts

    Security Scanning in CI

    | Tool | What it scans | CI Usage |
    |---|---|---|
    | CodeQL | SAST (C/C++, JS, Python, Java…) | github/codeql-action/analyze@v3 |
    | Trivy | Container images, FS, IaC | aquasecurity/trivy-action@master |
    | Snyk | Dependencies (SCA) | snyk/actions/test@master |
    | TruffleHog | Secrets in git history | trufflesecurity/trufflehog@main |
    | hadolint | Dockerfile lint | hadolint/hadolint-action@v3.1.0 |
    | Checkov | IaC (Terraform, K8s, CloudFormation) | bridgecrew/checkov-action@master |
    | OSV-scanner | Dependency vulnerabilities | google/osv-scanner-action@v1 |

    # Secret scanning pre-commit

  • name: TruffleHog
    • 
  uses: trufflesecurity/trufflehog@main
  with:
    extra_args: --only-verified

    9. Secret Management in Pipelines

    Principles

  • Never hardcode secrets in YAML, scripts, or Dockerfiles.
  • Use OIDC instead of long-lived cloud credentials where possible.
  • Scope secrets to the narrowest environment (staging vs production).
  • Rotate secrets regularly; automate rotation.
  • Audit secret access logs.

    • Platform Secret Stores

    | Platform | Mechanism | Notes |
    |---|---|---|
    | GitHub Actions | ${{ secrets.NAME }} | Org, repo, or environment level |
    | GitHub Environments | Protection rules + required reviewers | Gate deployments |
    | GitLab CI | $CI_VARIABLE (masked, protected) | Project or group level |
    | Jenkins | credentials() binding | Vault plugin recommended |
    | HashiCorp Vault | Dynamic secrets (TTL, auto-revoke) | hashicorp/vault-action@v3 |

    Vault Integration in GitHub Actions

    - name: Import Vault Secrets
  uses: hashicorp/vault-action@v3
  id: vault
  with:
    url: https://vault.example.com
    role: myapp-ci
    method: jwt
    path: github
    secrets: |
      secret/data/myapp/db DATABASE_URL | DB_URL;
      secret/data/myapp/api API_KEY    | API_KEY;

  • run: |
    • 
    echo "Connecting to ${{ steps.vault.outputs.DB_URL }}"
    npm run migrate

    GitHub Environment Protections

    # Settings → Environments → production

    - Required reviewers: @devops-lead
    

    - Wait timer: 5 minutes
    

    - Branch: main only
    

    - Deployment branch rules: main, release/*
    

    deploy:
      runs-on: ubuntu-latest
      environment: production   # enforces the above rules
      steps:
        - run: deploy-to-prod.sh

    10. Monitoring Pipelines

    Key Metrics

    | Metric | Description | Alert Threshold |
    |---|---|---|
    | Build Duration | Time from trigger to completion | > 2× median |
    | Flake Rate | % of passes that are intermittent | > 5% |
    | Mean Time to Recovery | Time from failure to green | > 30 min |
    | Deploy Frequency | Deploys per day/week | Trending down |
    | Change Failure Rate | % of deploys causing incidents | > 15% |
    | Pipeline Success Rate | % of runs that complete green | < 90% |

    GitHub Actions Monitoring

    # .github/workflows/pipeline-metrics.yml
name: Pipeline Metrics
on:
  workflow_run:
    workflows: ["CI"]
    types: [completed]

    jobs:
      metrics:
        runs-on: ubuntu-latest
        steps:
          - run: |
              echo "conclusion=${{ github.event.workflow_run.conclusion }}" >> "$GITHUB_STEP_SUMMARY"
              echo "run_time=$(( $(date +%s) - $(date -d '${{ github.event.workflow_run.run_started_at }}' +%s) ))" >> "$GITHUB_STEP_SUMMARY"
              curl -X POST https://metrics.example.com/api/pipeline \
                -d "workflow=${{ github.event.workflow_run.name }}" \
                -d "conclusion=${{ github.event.workflow_run.conclusion }}" \
                -d "duration=${run_time}"

    Datadog / Grafana Pipeline Integration

    # Send CI metrics to Datadog

  • uses: datadog/cli-pipeline-metrics@v1
    • 
  with:
    api-key: ${{ secrets.DATADOG_API_KEY }}
    pipeline-id: ${{ github.run_id }}

    Alerting on Failures

    # Slack alert on pipeline failure
post:
  failure:
    runs-on: ubuntu-latest
    steps:
      - uses: slackapi/slack-github-action@v1.27.0
        with:
          payload: |
            {
              "text": "❌ Pipeline failed",
              "blocks": [{
                "type": "section",
                "text": {
                  "type": "mrkdwn",
                  "text": "${{ github.workflow }} failed in run <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|${{ github.run_number }}>"
                }
              }]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

    11. Multi-Environment Promotion

    Promotion Flow

      dev  ──auto──▶  staging  ──manual──▶  production
   │                │                      │
   ▼                ▼                      ▼
 unit+lint     integration+smoke        canary+monitor
   ▼                ▼                      ▼
 dev cluster   staging cluster          prod cluster

    GitOps Promotion with ArgoCD

    # ArgoCD Application — dev
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-dev
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/org/myapp-manifests
    targetRevision: main
    path: overlays/dev
  destination:
    server: https://kubernetes.default.svc
    namespace: dev
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

    # Promote: update image tag in staging overlay

    automation/script or CI step:
    

  • run: |
    • 
    kustomize edit set image myapp=ghcr.io/org/myapp:${{ github.sha }}
    git commit -am "chore: promote ${{ github.sha }} to staging"
    git push

    Environment Matrix

    # Deploy to multiple environments via matrix
jobs:
  deploy:
    strategy:
      matrix:
        include:
          - environment: dev
            namespace: dev
            auto_deploy: true
          - environment: staging
            namespace: staging
            auto_deploy: true
          - environment: production
            namespace: production
            auto_deploy: false
    runs-on: ubuntu-latest
    environment: ${{ matrix.environment }}
    if: matrix.auto_deploy || github.event.inputs.deploy_prod == 'true'
    steps:
      - run: kubectl set image deployment/myapp myapp=$IMAGE -n ${{ matrix.namespace }}

    Flowcharts & Decision Diagrams

    Pipeline Architecture Decision Tree

    What CI/CD platform? ────────────────────────────┐
  │                                                  │
  ├─ GitHub-hosted project? ──Yes──→ GitHub Actions
  │                               (native, marketplace, OIDC)
  ├─ GitLab repo? ──Yes──→ GitLab CI
  │                         (built-in, .gitlab-ci.yml)
  ├─ Enterprise/legacy? ──Yes──→ Jenkins
  │                             (flexible, shared libraries)
  └─ Multi-cloud/hybrid ──→ Tekton + ArgoCD
                             (K8s-native, GitOps)

    Deployment Flow Decision Tree

    Deploy to prod? ─┬─ Risk appetite: low ──→ Blue-green (2 envs, instant rollback)
                  ├─ Risk appetite: medium ──→ Canary (5%→25%→100%, observe metrics)
                  ├─ Risk appetite: high ──→ Rolling (maxSurge=25%, gradual)
                  └─ Feature flag needed? ──Yes──→ Dark launch → flag ON → full rollout

    Pipeline Failure Debugging Flowchart

    Pipeline failed? ─┬─ Build stage ──→ Check dependency cache, Dockerfile, build logs
                   ├─ Test stage ──→ Check test report, flaky test history, service containers
                   ├─ Security scan ──→ Check vulnerability severity, allowlist, triage
                   ├─ Deploy stage ──→ Check credentials, namespace, health checks
                   └─ Integration test ──→ Check service mesh, network policies, timeouts

    Testing Strategy Enhancement for CI/CD

    Test Pyramid in CI

             ╱ E2E ╲           ← 5% — Cypress, Playwright (expensive, fragile)
        ╱────────╲
       ╱ Integration ╲     ← 15% — Service containers, API tests
      ╱────────────────╲
     ╱   Contract Tests ╲  ← 15% — Pact, Spring Cloud Contract
    ╱────────────────────╲
   ╱     Unit Tests       ╲← 65% — jest, pytest, go test (fast, deterministic)
  ╱──────────────────────────╲

    GitHub Actions Testing Matrix

    # Comprehensive test pipeline with matrix
jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node: [18, 20, 22]
        include:
          - os: ubuntu-latest
            node: 20
            coverage: true
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: "${{ matrix.node }}" }
      - run: npm ci
      - run: npm test -- --coverage=${{ matrix.coverage || false }}
      - uses: codecov/codecov-action@v4
        if: matrix.coverage
        with: { token: "${{ secrets.CODECOV_TOKEN }}" }

    security:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
          - uses: aquasecurity/trivy-action@master
            with: { scan-type: "fs", severity: "CRITICAL,HIGH" }
          - uses: github/codeql-action/analyze@v3
            with: { languages: "javascript" }
    

    e2e:
        runs-on: ubuntu-latest
        services:
          postgres:
            image: postgres:16
            env: { POSTGRES_PASSWORD: test, POSTGRES_DB: app_test }
            ports: ["5432:5432"]
            options: >-
              --health-cmd pg_isready
              --health-interval 10s
              --health-timeout 5s
              --health-retries 5
        steps:
          - uses: actions/checkout@v4
          - run: npm ci && npm run build
          - run: npx playwright install --with-deps
          - run: npx playwright test
          - uses: actions/upload-artifact@v4
            if: failure()
            with: { name: "playwright-report", path: "playwright-report/" }

    Performance Optimization for CI/CD

  • Caching: Cache dependency directories (node_modules, .m2, pip, Go module cache)
  • Parallelism: Run independent jobs in parallel; use matrix for multi-config
  • Incremental Builds: Use Turborepo/Nx monorepo caching for affected projects only
  • Docker Layer Caching: Multi-stage builds + BuildKit cache mount
  • Self-hosted Runners: Reduce queue time for large repos (6+ concurrent jobs)
  • Artifact Sharding: Split test suites across runners for faster feedback
  • Skip Unnecessary Work: paths filters to trigger only on relevant changes

    • 12. Best Practices

    Pipeline Design

  • Build once, promote everywhere — same artifact (Docker image, JAR) through all environments.
  • Pin all action/plugin versions — actions/checkout@eef61447b9 (full SHA) or at minimum @v4.
  • Fail fast — lint and unit tests before integration tests; timeout every job.
  • Concurrent groups — concurrency in GitHub Actions to cancel superseded runs.
  • Minimal permissions — permissions: contents: read by default; elevate only where needed.
  • Cache aggressively — dependency caches save 50–80% build time.
  • Artifact hygiene — set retention-days; don't accumulate forever.
  • Structured logs — JSON-formatted output for scraping and dashboards.

    • Security

  • OIDC over static secrets — ephemeral tokens via GitHub OIDC / GitLab JWT.
  • Pin base images — FROM node:20.11.0-alpine not FROM node:latest.
  • Scan continuously — SAST on every PR; SCA + container scan on every merge.
  • Sign artifacts — Cosign for container images; SLSA provenance.
  • Branch protection — require status checks before merge; no force push.
  • Least privilege — deploy tokens scoped to namespace; no cluster-admin.

    • Reliability

  • Retry flaky steps — retry in GitHub Actions; retry: 2 in GitLab.
  • Idempotent deployments — re-running a deploy job is safe.
  • Readiness probes — always wait for rollout/healthcheck before proceeding.
  • Rollback SOP — document and test rollback for every strategy.
  • Feature flags — decouple deploy from release.

    • 13. Common Pitfalls

    | Pitfall | Solution |
    |---|---|
    | Unpinned actions/tags | Pin to SHA: actions/checkout@b4ffde65 |
    | Secrets in logs | Use masked variables; never echo secrets |
    | Giant monolith pipeline | Split into reusable workflows / templates |
    | No caching | Add cache for language managers & Docker layers |
    | Flaky tests blocking deploys | Quarantine flaky tests; track flake rate |
    | Deploying different artifacts per env | Build once, promote the same image |
    | Manual deploy steps without audit | Use environment protections & required reviewers |
    | Running all tests sequentially | Parallelize with matrix / shard |
    | No timeout on jobs | Always set timeout-minutes |
    | Stale branches triggering CI | Use concurrency groups; auto-delete merged branches |
    | latest tag in production | Use SHA or semver tags; never floating tags |
    | Over-fetching secrets | Scope secrets to minimal needed; use OIDC |

    14. Quick Reference

    GitHub Actions — Useful Snippets

    # Conditional step

  • if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    • 
  run: echo "Main branch push"

    Reusable workflow output
    
outputs:
  version: ${{ steps.meta.outputs.version }}

    Manual dispatch with inputs
    
on:
  workflow_dispatch:
    inputs:
      environment:
        description: 'Target environment'
        required: true
        type: choice
        options: [staging, production]

    Matrix with include
    
strategy:
  matrix:
    include:
      - os: ubuntu-latest
        target: linux-x64
      - os: macos-latest
        target: darwin-arm64

    GitLab CI — Useful Snippets

    # DAG with needs
test:integration:
  needs: [build]
  stage: test   # can run before other stage-level jobs

    Only run on tags
    
rules:
  - if: $CI_COMMIT_TAG

    Manual job with environment
    
deploy:prod:
  when: manual
  environment:
    name: production

    Inherited variables
    
variables:
  APP_VERSION: $CI_COMMIT_TAG

    Jenkins — Useful Snippets

    // When conditions
when {
  branch 'main'
  expression { env.DEPLOY_ENV == 'production' }
}

    // Parallel stages
    stage('Test') {
      parallel {
        stage('Unit') { steps { sh 'npm test' } }
        stage('Lint') { steps { sh 'npm run lint' } }
      }
    }
    

    // Credentials binding
    withCredentials([string(credentialsId: 'api-key', variable: 'KEY')]) {
      sh "curl -H 'Authorization: Bearer ${KEY}' ..."
    }

    15. Tool Comparison Matrix

    | Feature | GitHub Actions | GitLab CI | Jenkins |
    |---|---|---|---|
    | Config location | .github/workflows/*.yml | .gitlab-ci.yml | Jenkinsfile |
    | Language | YAML | YAML | Groovy |
    | Hosted runners | ✅ Native | ✅ Native (SaaS) | ❌ Self-hosted |
    | Self-hosted runners | ✅ | ✅ | ✅ Native |
    | Reusability | Reusable workflows, composite actions | extends, include | Shared libraries |
    | Environments | ✅ Protection rules, reviewers | ✅ Environments, approvals | ✅ Input steps |
    | OIDC | ✅ AWS/GCP/Azure | ✅ JWT tokens | ✅ With plugins |
    | Container registry | ghcr.io | registry.gitlab.com | Any (ECR, ACR…) |
    | Caching | actions/cache, GHA cache | Built-in cache: | Plugins |
    | Secret scope | Org/Repo/Environment | Group/Project/Instance | Folder/Credentials |
    | Cost model | Free for public, minutes for private | Free tier then compute | Infrastructure cost |
    | Community | Marketplace (20k+ actions) | Templates | 1800+ plugins |

    This skill covers CI/CD fundamentals through production patterns. Adapt examples to your stack, cloud provider, and team size. Always test pipeline changes in a branch before merging to main.

    💬 Ask about Ci Cd Mastery