Ci Cd Mastery

CI/CD Mastery

End-to-end guide for building, testing, securing, and deploying software with continuous integration and continuous delivery pipelines.

1. Pipeline Fundamentals

Core Concepts

| Concept | Definition |
|---|---|
| Continuous Integration (CI) | Automatically build and test every commit to detect regressions early. |
| Continuous Delivery (CD) | Every commit that passes CI is ready to deploy; deployment is manual gate. |
| Continuous Deployment | Every passing commit is deployed automatically — no human gate. |
| Pipeline | Ordered set of stages/jobs that transform source code into running software. |
| Stage | Logical group of jobs that run in sequence (build → test → deploy). |
| Job / Step | Atomic unit of work inside a stage — runs on a single agent/runner. |
| Artifact | Immutable file produced by a job and consumed downstream. |
| Cache | Mutable store to speed up dependency resolution across runs. |

Pipeline Anatomy

┌──────────┐   ┌──────────┐   ┌───────────┐   ┌──────────┐   ┌──────────┐
│  Source   │──▶│  Build   │──▶│   Test    │──▶│  Stage   │──▶│  Deploy  │
│ Checkout  │   │ Compile  │   │ Unit/Int  │   │ Artifact │   │  Prod    │
└──────────┘   └──────────┘   └───────────┘   └──────────┘   └──────────┘

Design Principles

2. GitHub Actions

Workflow Syntax

# .github/workflows/ci.yml
name: CI
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
permissions:
  contents: read
  id-token: write       # OIDC for cloud auth
  security-events: write # CodeQL
env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}
jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      version: ${{ steps.meta.outputs.version }}
    steps:
      - uses: actions/checkout@v4
- uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
- run: npm ci
      - run: npm run build
- uses: actions/upload-artifact@v4
        with:
          name: dist
          path: dist/
          retention-days: 5
test:
    needs: build
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shard: [1, 2, 3]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npx vitest run --shard=${{ matrix.shard }}/3
security:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: github/codeql-action/analyze@v3
        with:
          languages: javascript
docker:
    needs: [build, test]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
- uses: actions/download-artifact@v4
        with:
          name: dist
          path: dist/
- uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/metadata-action@v5
        id: meta
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=ref,event=branch
            type=sha,prefix=
            type=semver,pattern={{version}}
- uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
deploy-staging:
    needs: docker
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - run: echo "Deploy ${{ needs.docker.outputs.version || 'latest' }} to staging"
deploy-prod:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production
    steps:
      - run: echo "Deploy ${{ needs.docker.outputs.version || 'latest' }} to production"

Actions Matrix

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node: [18, 20, 22]
        exclude:
          - os: windows-latest
            node: 18
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}

Reusable Workflows

# .github/workflows/_deploy.yml (reusable)
on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
      image_tag:
        required: true
        type: string
    secrets:
      DEPLOY_TOKEN:
        required: true
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment }}
    steps:
      - run: |
          curl -X POST https://api.example.com/deploy \
            -H "Authorization: Bearer ${{ secrets.DEPLOY_TOKEN }}" \
            -d "{\"tag\": \"${{ inputs.image_tag }}\"}"

# .github/workflows/ci.yml (caller)
jobs:
  deploy-staging:
    uses: ./.github/workflows/_deploy.yml
    with:
      environment: staging
      image_tag: ${{ needs.docker.outputs.version }}
    secrets:
      DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

OIDC Authentication

# AWS
uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/GitHubOIDC
    aws-region: us-east-1
GCP
uses: google-github-actions/auth@v2
  with:
    workload_identity_provider: projects/123/locations/global/workloadIdentityPools/github/providers/actions
    service_account: deploy@project.iam.gserviceaccount.com
Azure
uses: azure/login@v2
  with:
    client-id: ${{ secrets.AZURE_CLIENT_ID }}
    tenant-id: ${{ secrets.AZURE_TENANT_ID }}
    subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

Caching Strategies

# Native cache via setup-* actions
uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: npm          # auto-caches ~/.npm
Manual cache
uses: actions/cache@v4
  with:
    path: |
      ~/.cache/pip
      ~/.local/bin/cog
    key: python-${{ runner.os }}-${{ hashFiles('requirements.txt') }}
    restore-keys: python-${{ runner.os }}-
Docker layer cache via buildx
uses: docker/build-push-action@v5
  with:
    cache-from: type=gha
    cache-to: type=gha,mode=max

3. GitLab CI

.gitlab-ci.yml

stages:
  - build
  - test
  - security
  - package
  - deploy
variables:
  REGISTRY: $CI_REGISTRY
  IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
  DOCKER_DRIVER: overlay2
default:
  image: node:20-alpine
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
  before_script:
    - npm ci --ignore-scripts
---------- BUILD ----------
build:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour
---------- TEST ----------
test:unit:
  stage: test
  script:
    - npx vitest run --coverage
  coverage: '/All files[^|]\|[^|]\s+([\d.]+)/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
test:integration:
  stage: test
  services:
    - postgres:16-alpine
  variables:
    POSTGRES_DB: testdb
    POSTGRES_USER: test
    POSTGRES_PASSWORD: test
    DATABASE_URL: postgres://test:test@postgres:5432/testdb
  script:
    - npx vitest run --config vitest.integration.config.ts
---------- SECURITY ----------
sast:
  stage: security
  trigger:
    include:
      - template: Security/SAST.gitlab-ci.yml
container_scanning:
  stage: security
  trigger:
    include:
      - template: Security/Container-Scanning.gitlab-ci.yml
---------- PACKAGE ----------
docker:build:
  stage: package
  image: docker:24
  services:
    - docker:24-dind
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY
  script:
    - docker build -t $IMAGE .
    - docker push $IMAGE
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_COMMIT_TAG
---------- DEPLOY ----------
.deploy_template:
  stage: deploy
  image: bitnami/kubectl:latest
  script:
    - kubectl config use-context $KUBE_CONTEXT
    - kubectl set image deployment/$APP $APP=$IMAGE -n $NAMESPACE
    - kubectl rollout status deployment/$APP -n $NAMESPACE --timeout=120s
deploy:staging:
  extends: .deploy_template
  variables:
    APP: myapp
    NAMESPACE: staging
  environment:
    name: staging
    url: https://staging.example.com
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
deploy:production:
  extends: .deploy_template
  variables:
    APP: myapp
    NAMESPACE: production
  environment:
    name: production
    url: https://app.example.com
  rules:
    - if: $CI_COMMIT_TAG
  when: manual
---------- REVIEW APPS ----------
review:
  extends: .deploy_template
  variables:
    APP: review-${CI_MERGE_REQUEST_IID}
    NAMESPACE: review
  environment:
    name: review/$CI_MERGE_REQUEST_IID
    url: https://review-${CI_MERGE_REQUEST_IID}.example.com
    on_stop: review:stop
  rules:
    - if: $CI_MERGE_REQUEST_IID
review:stop:
  extends: .deploy_template
  stage: deploy
  script:
    - kubectl delete namespace review-$CI_MERGE_REQUEST_IID
  environment:
    name: review/$CI_MERGE_REQUEST_IID
    action: stop
  rules:
    - if: $CI_MERGE_REQUEST_IID
      when: manual

GitLab CI Key Features

| Feature | Syntax | Purpose |
|---|---|---|
| extends | extends: .template | DRY job config via inheritance |
| rules | rules: [{ if, when, ... }] | Conditional job inclusion |
| trigger | trigger: { include: ... } | Multi-project / downstream pipelines |
| needs | needs: [job] | DAG execution — run out of stage order |
| services | services: [postgres] | Sidecar containers for tests |
| artifacts:reports | reports: { cobertura, junit } | Native report ingestion |
| environment | environment: { name, url } | Deployment tracking & protection |
| cache | cache: { key, paths } | Dependency caching across pipelines |

4. Jenkins

Declarative Jenkinsfile

// Jenkinsfile (Declarative Pipeline)
pipeline {
  agent none
options {
    buildDiscarder(logRotator(numToKeepStr: '30'))
    timeout(time: 30, unit: 'MINUTES')
    disableConcurrentBuilds()
    timestamps()
  }
environment {
    REGISTRY = 'ghcr.io'
    IMAGE    = "${REGISTRY}/myorg/myapp"
    DOCKER_CONFIG = credentials('docker-registry-auth')
  }
triggers {
    pollSCM('H/5    ')
    upstream(upstreamProjects: 'infrastructure/deploy', threshold: hudson.model.Result.SUCCESS)
  }
stages {
    stage('Build') {
      agent { label 'node20' }
      steps {
        sh 'npm ci && npm run build'
      }
      post {
        always {
          stash name: 'dist', includes: 'dist/**'
        }
      }
    }
stage('Test') {
      parallel {
        stage('Unit') {
          agent { label 'node20' }
          steps {
            unstash 'dist'
            sh 'npx vitest run'
          }
          post {
            always { junit 'reports/junit.xml' }
          }
        }
        stage('Lint') {
          agent { label 'node20' }
          steps {
            sh 'npm run lint'
          }
        }
      }
    }
stage('Security Scan') {
      agent { label 'node20' }
      steps {
        sh 'npx audit-ci --moderate'
        // Trivy FS scan
        sh 'trivy fs --exit-code 1 --severity HIGH,CRITICAL .'
      }
    }
stage('Docker Build & Push') {
      agent { label 'docker' }
      steps {
        unstash 'dist'
        script {
          def tag = env.GIT_COMMIT?.take(8) ?: env.BUILD_NUMBER
          docker.withRegistry("https://${REGISTRY}", 'docker-registry-auth') {
            def img = docker.build("${IMAGE}:${tag}", '--build-arg VERSION=${tag} .')
            img.push()
            if (env.GIT_BRANCH == 'main') {
              img.push('latest')
            }
          }
        }
      }
    }
stage('Deploy Staging') {
      agent { label 'kubectl' }
      steps {
        sh "kubectl set image deployment/myapp myapp=${IMAGE}:${env.GIT_COMMIT.take(8)} -n staging"
        sh 'kubectl rollout status deployment/myapp -n staging --timeout=120s'
      }
    }
stage('Deploy Production') {
      agent { label 'kubectl' }
      input {
        message 'Deploy to production?'
        ok 'Deploy'
      }
      steps {
        sh "kubectl set image deployment/myapp myapp=${IMAGE}:${env.GIT_COMMIT.take(8)} -n production"
        sh 'kubectl rollout status deployment/myapp -n production --timeout=120s'
      }
    }
  }
post {
    failure { slackSend color: 'danger', message: "Build ${env.JOB_NAME} #${env.BUILD_NUMBER} failed!" }
    success { slackSend color: 'good', message: "Build ${env.JOB_NAME} #${env.BUILD_NUMBER} succeeded." }
  }
}

Shared Libraries

// vars/deployK8s.groovy (shared library)
def call(Map config) {
  sh """
    kubectl set image deployment/${config.app} ${config.app}=${config.image} \
      -n ${config.namespace}
    kubectl rollout status deployment/${config.app} \
      -n ${config.namespace} --timeout=${config.timeout ?: '120s'}
  """
}
// vars/dockerBuildPush.groovy
def call(Map config) {
  docker.withRegistry("https://${config.registry}", config.credId) {
    def img = docker.build("${config.image}:${config.tag}", config.context ?: '.')
    img.push()
    img.push('latest')
  }
}

// Jenkinsfile using shared lib
@Library('my-pipeline-lib') _
pipeline {
  agent any
  stages {
    stage('Deploy Staging') {
      steps {
        deployK8s app: 'myapp', image: 'ghcr.io/org/myapp:abc1234', namespace: 'staging'
      }
    }
  }
}

Multibranch Pipeline

// Automatic branch discovery in Jenkins multibranch project
// Each branch with a Jenkinsfile gets its own pipeline
// PRs get pipelines via GitHub/Branch API integration
// Branch indexing triggers on push via webhook

Jenkins best practices:

5. Deployment Strategies

Blue-Green Deployment

Two identical environments. Switch traffic atomically.

# Kubernetes blue-green via Service selector swap
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
    slot: blue    # ← toggle this value to switch
  ports:
    - port: 80
      targetPort: 8080

# Deploy new version to green
kubectl apply -f deployment-green.yaml
Verify green is healthy
kubectl wait --for=condition=Available deploy/myapp-green --timeout=120s
Run smoke tests against green
curl -sf http://green.internal/health
Swap Service selector to green
kubectl patch svc myapp -p '{"spec":{"selector":{"slot":"green"}}}'
Rollback: patch selector back to blue

Canary Deployment

Gradually shift traffic to the new version.

# Istio VirtualService — 10% canary
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts: [myapp.example.com]
  http:
    - route:
        - destination:
            host: myapp
            subset: stable
          weight: 90
        - destination:
            host: myapp
            subset: canary
          weight: 10

# Argo Rollouts Canary
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - setWeight: 30
        - pause: { duration: 5m }
        - setWeight: 60
        - pause: { duration: 5m }
        - setWeight: 100
      canaryService: myapp-canary
      stableService: myapp-stable

Rolling Update (Kubernetes default)

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%       # pods above desired count during rollout
      maxUnavailable: 0   # never go below desired count

A/B Testing (Feature Flags)

# LaunchDarkly / Unleash / custom feature-flag evaluation
if feature_flags.is_enabled("new-checkout", user_id, default=False):
    return new_checkout(request)
return legacy_checkout(request)

# Unleash feature toggle config
apiVersion: unleash.example.com/v1
kind: FeatureToggle
metadata:
  name: new-checkout
spec:
  strategy: gradualRollout
  parameters:
    percentage: 25
    groupId: checkout-experiment

Feature Flag Deployment Flow

┌───────────┐   ┌──────────────┐   ┌──────────────┐
│  Deploy    │──▶│  Feature OFF │──▶│  Monitor     │
│  (anytime) │   │  in prod     │   │  no impact   │
└───────────┘   └──────┬───────┘   └──────────────┘
                       │
                       ▼  Toggle ON
                ┌──────────────┐   ┌──────────────┐
                │  Feature ON  │──▶│  Rollback:   │
                │  Gradual %   │   │  Toggle OFF  │
                └──────────────┘   └──────────────┘

Strategy Comparison

| Strategy | Downtime | Rollback Speed | Cost | Complexity |
|---|---|---|---|---|
| Blue-Green | Zero | Instant (selector swap) | 2× resources | Medium |
| Canary | Zero | Fast (shift traffic) | 1.1× resources | High |
| Rolling | Near-zero | Medium (re-rollout) | 1× resources | Low |
| Feature Flags | Zero | Instant (toggle off) | 1× resources | Medium |

6. Release Automation

Semantic Versioning (SemVer)

MAJOR.MINOR.PATCH
  │    │    └── bug fixes (backward compatible)
  │    └────── new features (backward compatible)
  └─────────── breaking changes

Automated Versioning with Conventional Commits

# .github/workflows/release.yml
name: Release
on:
  push:
    branches: [main]
permissions:
  contents: write
  pull-requests: write
jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0    # full history for versioning
- uses: actions/setup-node@v4
        with:
          node-version: 20
- name: Release
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          npx semantic-release

.releaserc.json (semantic-release)

{
  "branches": ["main", { "name": "next", "prerelease": true }],
  "plugins": [
    "@semantic-release/commit-analyzer",
    "@semantic-release/release-notes-generator",
    "@semantic-release/changelog",
    ["@semantic-release/exec", {
      "prepareCmd": "npm version ${nextRelease.version} --no-git-tag-version"
    }],
    ["@semantic-release/git", {
      "assets": ["package.json", "package-lock.json", "CHANGELOG.md"],
      "message": "chore(release): ${nextRelease.version}"
    }],
    "@semantic-release/github"
  ]
}

Changelog Generation

# git-cliff (Rust-based, fast)
git-cliff --output CHANGELOG.md
cliff.toml
[changelog]
header = """# Changelog\n\nAll notable changes to this project will be documented.\n"""
body = """
{% if version %}## [{{ version | trim_start_matches(pat="v") }}] - {{ timestamp | date(format="%Y-%m-%d") }}
{% else %}## [Unreleased]
{% endif %}{% for commit in commits %}- {{ commit.message | trim_end_matches(pat="\\n") }}{% endfor %}"""
trim = true

GitHub Releases in CI

# .github/workflows/publish-release.yml
name: Publish Release
on:
  push:
    tags: ['v*']
jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
- uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
- run: npm ci
      - run: npm run build
- name: Create GitHub Release
        uses: softprops/action-gh-release@v2
        with:
          generate_release_notes: true
          files: |
            dist/*.tar.gz
            dist/*.sha256
- name: Publish to npm
        run: npm publish --access public
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

7. Docker / OCI Image Building in CI

Multi-stage Dockerfile

# Build stage
FROM node:20-alpine AS builder
WORKDIR /app
COPY package-lock.json package.json ./
RUN npm ci --ignore-scripts
COPY . .
RUN npm run build
Production stage
FROM gcr.io/distroless/nodejs20-debian12:latest
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
USER nonroot:nonroot
EXPOSE 3000
CMD ["dist/server.js"]

BuildKit + Cache in GitHub Actions

- uses: docker/setup-buildx-action@v3
uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: ${{ steps.meta.outputs.tags }}
    cache-from: type=gha
    cache-to: type=gha,mode=max
    platforms: linux/amd64,linux/arm64
    provenance: true     # SLSA provenance
    sbom: true            # SBOM attestation

Container Scanning

# Trivy — run in CI
name: Run Trivy vulnerability scanner
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: '${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}'
    format: 'sarif'
    output: 'trivy-results.sarif'
    exit-code: '1'
    severity: 'CRITICAL,HIGH'
uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: 'trivy-results.sarif'

8. Testing in CI

Test Pyramid in Pipelines

         ╱  E2E        ╲   — slow, flaky, few
        ╱  Integration   ╲  — medium speed, moderate count
       ╱    Unit Tests    ╲ — fast, many, deterministic

Parallel Test Sharding

# Vitest / Jest shards in GitHub Actions
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - run: npx vitest run --shard=${{ matrix.shard }}/4

Integration Tests with Services

# GitHub Actions — Postgres service
jobs:
  integration:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_DB: testdb
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
        ports: ['5432:5432']
        options: >-
          --health-cmd="pg_isready"
          --health-interval=10s
          --health-timeout=5s
          --health-retries=5
    steps:
      - run: npx vitest run --config vitest.integration.config.ts

Security Scanning in CI

| Tool | What it scans | CI Usage |
|---|---|---|
| CodeQL | SAST (C/C++, JS, Python, Java…) | github/codeql-action/analyze@v3 |
| Trivy | Container images, FS, IaC | aquasecurity/trivy-action@master |
| Snyk | Dependencies (SCA) | snyk/actions/test@master |
| TruffleHog | Secrets in git history | trufflesecurity/trufflehog@main |
| hadolint | Dockerfile lint | hadolint/hadolint-action@v3.1.0 |
| Checkov | IaC (Terraform, K8s, CloudFormation) | bridgecrew/checkov-action@master |
| OSV-scanner | Dependency vulnerabilities | google/osv-scanner-action@v1 |

# Secret scanning pre-commit
name: TruffleHog
  uses: trufflesecurity/trufflehog@main
  with:
    extra_args: --only-verified

9. Secret Management in Pipelines

Principles

Platform Secret Stores

| Platform | Mechanism | Notes |
|---|---|---|
| GitHub Actions | ${{ secrets.NAME }} | Org, repo, or environment level |
| GitHub Environments | Protection rules + required reviewers | Gate deployments |
| GitLab CI | $CI_VARIABLE (masked, protected) | Project or group level |
| Jenkins | credentials() binding | Vault plugin recommended |
| HashiCorp Vault | Dynamic secrets (TTL, auto-revoke) | hashicorp/vault-action@v3 |

Vault Integration in GitHub Actions

- name: Import Vault Secrets
  uses: hashicorp/vault-action@v3
  id: vault
  with:
    url: https://vault.example.com
    role: myapp-ci
    method: jwt
    path: github
    secrets: |
      secret/data/myapp/db DATABASE_URL | DB_URL;
      secret/data/myapp/api API_KEY    | API_KEY;
run: |
    echo "Connecting to ${{ steps.vault.outputs.DB_URL }}"
    npm run migrate

GitHub Environment Protections

# Settings → Environments → production
- Required reviewers: @devops-lead
- Wait timer: 5 minutes
- Branch: main only
- Deployment branch rules: main, release/*
deploy:
  runs-on: ubuntu-latest
  environment: production   # enforces the above rules
  steps:
    - run: deploy-to-prod.sh

10. Monitoring Pipelines

Key Metrics

| Metric | Description | Alert Threshold |
|---|---|---|
| Build Duration | Time from trigger to completion | > 2× median |
| Flake Rate | % of passes that are intermittent | > 5% |
| Mean Time to Recovery | Time from failure to green | > 30 min |
| Deploy Frequency | Deploys per day/week | Trending down |
| Change Failure Rate | % of deploys causing incidents | > 15% |
| Pipeline Success Rate | % of runs that complete green | < 90% |

GitHub Actions Monitoring

# .github/workflows/pipeline-metrics.yml
name: Pipeline Metrics
on:
  workflow_run:
    workflows: ["CI"]
    types: [completed]
jobs:
  metrics:
    runs-on: ubuntu-latest
    steps:
      - run: |
          echo "conclusion=${{ github.event.workflow_run.conclusion }}" >> "$GITHUB_STEP_SUMMARY"
          echo "run_time=$(( $(date +%s) - $(date -d '${{ github.event.workflow_run.run_started_at }}' +%s) ))" >> "$GITHUB_STEP_SUMMARY"
          curl -X POST https://metrics.example.com/api/pipeline \
            -d "workflow=${{ github.event.workflow_run.name }}" \
            -d "conclusion=${{ github.event.workflow_run.conclusion }}" \
            -d "duration=${run_time}"

Datadog / Grafana Pipeline Integration

# Send CI metrics to Datadog
uses: datadog/cli-pipeline-metrics@v1
  with:
    api-key: ${{ secrets.DATADOG_API_KEY }}
    pipeline-id: ${{ github.run_id }}

Alerting on Failures

# Slack alert on pipeline failure
post:
  failure:
    runs-on: ubuntu-latest
    steps:
      - uses: slackapi/slack-github-action@v1.27.0
        with:
          payload: |
            {
              "text": "❌ Pipeline failed",
              "blocks": [{
                "type": "section",
                "text": {
                  "type": "mrkdwn",
                  "text": "${{ github.workflow }} failed in run <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|${{ github.run_number }}>"
                }
              }]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

11. Multi-Environment Promotion

Promotion Flow

  dev  ──auto──▶  staging  ──manual──▶  production
   │                │                      │
   ▼                ▼                      ▼
 unit+lint     integration+smoke        canary+monitor
   ▼                ▼                      ▼
 dev cluster   staging cluster          prod cluster

GitOps Promotion with ArgoCD

# ArgoCD Application — dev
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-dev
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/org/myapp-manifests
    targetRevision: main
    path: overlays/dev
  destination:
    server: https://kubernetes.default.svc
    namespace: dev
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

# Promote: update image tag in staging overlay
automation/script or CI step:
run: |
    kustomize edit set image myapp=ghcr.io/org/myapp:${{ github.sha }}
    git commit -am "chore: promote ${{ github.sha }} to staging"
    git push

Environment Matrix

# Deploy to multiple environments via matrix
jobs:
  deploy:
    strategy:
      matrix:
        include:
          - environment: dev
            namespace: dev
            auto_deploy: true
          - environment: staging
            namespace: staging
            auto_deploy: true
          - environment: production
            namespace: production
            auto_deploy: false
    runs-on: ubuntu-latest
    environment: ${{ matrix.environment }}
    if: matrix.auto_deploy || github.event.inputs.deploy_prod == 'true'
    steps:
      - run: kubectl set image deployment/myapp myapp=$IMAGE -n ${{ matrix.namespace }}

Flowcharts & Decision Diagrams

Pipeline Architecture Decision Tree

What CI/CD platform? ────────────────────────────┐
  │                                                  │
  ├─ GitHub-hosted project? ──Yes──→ GitHub Actions
  │                               (native, marketplace, OIDC)
  ├─ GitLab repo? ──Yes──→ GitLab CI
  │                         (built-in, .gitlab-ci.yml)
  ├─ Enterprise/legacy? ──Yes──→ Jenkins
  │                             (flexible, shared libraries)
  └─ Multi-cloud/hybrid ──→ Tekton + ArgoCD
                             (K8s-native, GitOps)

Deployment Flow Decision Tree

Deploy to prod? ─┬─ Risk appetite: low ──→ Blue-green (2 envs, instant rollback)
                  ├─ Risk appetite: medium ──→ Canary (5%→25%→100%, observe metrics)
                  ├─ Risk appetite: high ──→ Rolling (maxSurge=25%, gradual)
                  └─ Feature flag needed? ──Yes──→ Dark launch → flag ON → full rollout

Pipeline Failure Debugging Flowchart

Pipeline failed? ─┬─ Build stage ──→ Check dependency cache, Dockerfile, build logs
                   ├─ Test stage ──→ Check test report, flaky test history, service containers
                   ├─ Security scan ──→ Check vulnerability severity, allowlist, triage
                   ├─ Deploy stage ──→ Check credentials, namespace, health checks
                   └─ Integration test ──→ Check service mesh, network policies, timeouts

Testing Strategy Enhancement for CI/CD

Test Pyramid in CI

         ╱ E2E ╲           ← 5% — Cypress, Playwright (expensive, fragile)
        ╱────────╲
       ╱ Integration ╲     ← 15% — Service containers, API tests
      ╱────────────────╲
     ╱   Contract Tests ╲  ← 15% — Pact, Spring Cloud Contract
    ╱────────────────────╲
   ╱     Unit Tests       ╲← 65% — jest, pytest, go test (fast, deterministic)
  ╱──────────────────────────╲

GitHub Actions Testing Matrix

# Comprehensive test pipeline with matrix
jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node: [18, 20, 22]
        include:
          - os: ubuntu-latest
            node: 20
            coverage: true
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: "${{ matrix.node }}" }
      - run: npm ci
      - run: npm test -- --coverage=${{ matrix.coverage || false }}
      - uses: codecov/codecov-action@v4
        if: matrix.coverage
        with: { token: "${{ secrets.CODECOV_TOKEN }}" }
security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aquasecurity/trivy-action@master
        with: { scan-type: "fs", severity: "CRITICAL,HIGH" }
      - uses: github/codeql-action/analyze@v3
        with: { languages: "javascript" }
e2e:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env: { POSTGRES_PASSWORD: test, POSTGRES_DB: app_test }
        ports: ["5432:5432"]
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - run: npx playwright install --with-deps
      - run: npx playwright test
      - uses: actions/upload-artifact@v4
        if: failure()
        with: { name: "playwright-report", path: "playwright-report/" }

Performance Optimization for CI/CD

12. Best Practices

Pipeline Design

Security

Reliability

13. Common Pitfalls

| Pitfall | Solution |
|---|---|
| Unpinned actions/tags | Pin to SHA: actions/checkout@b4ffde65 |
| Secrets in logs | Use masked variables; never echo secrets |
| Giant monolith pipeline | Split into reusable workflows / templates |
| No caching | Add cache for language managers & Docker layers |
| Flaky tests blocking deploys | Quarantine flaky tests; track flake rate |
| Deploying different artifacts per env | Build once, promote the same image |
| Manual deploy steps without audit | Use environment protections & required reviewers |
| Running all tests sequentially | Parallelize with matrix / shard |
| No timeout on jobs | Always set timeout-minutes |
| Stale branches triggering CI | Use concurrency groups; auto-delete merged branches |
| latest tag in production | Use SHA or semver tags; never floating tags |
| Over-fetching secrets | Scope secrets to minimal needed; use OIDC |

14. Quick Reference

GitHub Actions — Useful Snippets

# Conditional step
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
  run: echo "Main branch push"
Reusable workflow output
outputs:
  version: ${{ steps.meta.outputs.version }}
Manual dispatch with inputs
on:
  workflow_dispatch:
    inputs:
      environment:
        description: 'Target environment'
        required: true
        type: choice
        options: [staging, production]
Matrix with include
strategy:
  matrix:
    include:
      - os: ubuntu-latest
        target: linux-x64
      - os: macos-latest
        target: darwin-arm64

GitLab CI — Useful Snippets

# DAG with needs
test:integration:
  needs: [build]
  stage: test   # can run before other stage-level jobs
Only run on tags
rules:
  - if: $CI_COMMIT_TAG
Manual job with environment
deploy:prod:
  when: manual
  environment:
    name: production
Inherited variables
variables:
  APP_VERSION: $CI_COMMIT_TAG

Jenkins — Useful Snippets

// When conditions
when {
  branch 'main'
  expression { env.DEPLOY_ENV == 'production' }
}
// Parallel stages
stage('Test') {
  parallel {
    stage('Unit') { steps { sh 'npm test' } }
    stage('Lint') { steps { sh 'npm run lint' } }
  }
}
// Credentials binding
withCredentials([string(credentialsId: 'api-key', variable: 'KEY')]) {
  sh "curl -H 'Authorization: Bearer ${KEY}' ..."
}

15. Tool Comparison Matrix

| Feature | GitHub Actions | GitLab CI | Jenkins |
|---|---|---|---|
| Config location | .github/workflows/*.yml | .gitlab-ci.yml | Jenkinsfile |
| Language | YAML | YAML | Groovy |
| Hosted runners | ✅ Native | ✅ Native (SaaS) | ❌ Self-hosted |
| Self-hosted runners | ✅ | ✅ | ✅ Native |
| Reusability | Reusable workflows, composite actions | extends, include | Shared libraries |
| Environments | ✅ Protection rules, reviewers | ✅ Environments, approvals | ✅ Input steps |
| OIDC | ✅ AWS/GCP/Azure | ✅ JWT tokens | ✅ With plugins |
| Container registry | ghcr.io | registry.gitlab.com | Any (ECR, ACR…) |
| Caching | actions/cache, GHA cache | Built-in cache: | Plugins |
| Secret scope | Org/Repo/Environment | Group/Project/Instance | Folder/Credentials |
| Cost model | Free for public, minutes for private | Free tier then compute | Infrastructure cost |
| Community | Marketplace (20k+ actions) | Templates | 1800+ plugins |

This skill covers CI/CD fundamentals through production patterns. Adapt examples to your stack, cloud provider, and team size. Always test pipeline changes in a branch before merging to main.