CI/CD Pipelines That Don't Break at 2 AM

  • cicd
  • devops
  • java
  • gitlab
  • backend

A pipeline that fails randomly is worse than no pipeline. It trains developers to ignore failures. Once your team starts clicking "retry" as a reflex instead of reading the failure, you've lost the whole point.

Here's what makes the difference between a pipeline that adds confidence and one that adds noise.

Pipeline Stages That Actually Matter

Four stages cover most Java backend services:

  1. Build & Unit Tests — compile, run unit tests, check coverage thresholds. Must complete in under 3 minutes. If it takes longer, developers push without waiting.
  2. Integration Tests — spin up a PostgreSQL container, run @SpringBootTest slices. Slower, but this is where real regressions live.
  3. Docker Build & Push — build the image, run Trivy for vulnerability scanning, push to registry.
  4. Deploy — rolling update to staging; production needs a manual gate unless you've earned full CD trust.

Keep stages sequential. The point of the pipeline is to fail fast and fail informatively. Don't run Docker build if unit tests failed — you're wasting 4 minutes building an image you won't use.

Fast Feedback First

Unit tests should run first because they're fast and catch the most obvious regressions. Integration tests run second because they're slow and you don't want to wait on them if compilation failed.

In a Maven monorepo, run only what changed:

mvn test -pl order-service,shared-lib --also-make

-pl selects modules, --also-make builds their dependencies. If you change shared-lib, both shared-lib and order-service tests run. If you only change order-service, shared-lib tests are skipped.

For compilation and lint, Checkstyle and SpotBugs run in the same stage as mvn test. They add maybe 20 seconds and catch a class of bugs that tests won't:

<plugin>
  <groupId>com.github.spotbugs</groupId>
  <artifactId>spotbugs-maven-plugin</artifactId>
  <version>4.8.3.1</version>
  <executions>
    <execution>
      <goals><goal>check</goal></goals>
    </execution>
  </executions>
</plugin>

Flaky Tests — The Silent Pipeline Killer

A flaky test is one that passes most of the time and fails occasionally for reasons unrelated to code correctness. Timing issues, shared state between tests, external service calls in unit tests — these are the usual causes.

Flaky tests erode trust faster than anything else. When a build goes red, the first question becomes "is this real or is it the flaky Kafka test?" instead of "what broke?" Once you have two or three flaky tests, developers start ignoring red builds entirely.

My policy: when a test is identified as flaky, it gets quarantined immediately with a ticket reference:

@Test
@Disabled("flaky: INFRA-456 — timing race in Kafka consumer setup")
void shouldProcessOrderEventWithinTimeout() { ... }

A disabled test is honest. A flaky test in the suite is a lie. The ticket ensures it doesn't stay disabled forever.

Docker Layer Caching

A naive Dockerfile copies your entire project and runs mvn package. Every build downloads all dependencies from scratch — 3 to 8 minutes for a typical Spring Boot project. With layer caching:

FROM eclipse-temurin:21-jdk-alpine AS builder
WORKDIR /app

# Copy only the dependency manifests first
COPY pom.xml ./
COPY */pom.xml ./

# Download dependencies — this layer caches until pom.xml changes
RUN mvn dependency:go-offline -q

# Now copy source and build
COPY src/ ./src/
RUN mvn package -DskipTests -q

The dependency:go-offline layer is only invalidated when pom.xml changes. Most builds skip straight to the COPY src/ step. Build time drops from 6 minutes to under 90 seconds in a warm cache.

In GitLab CI, enable registry caching:

variables:
  DOCKER_BUILDKIT: "1"

docker-build:
  script:
    - docker build
        --cache-from $CI_REGISTRY_IMAGE:cache
        --build-arg BUILDKIT_INLINE_CACHE=1
        -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
        -t $CI_REGISTRY_IMAGE:cache .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:cache

Secrets in CI

The rules are simple but frequently violated:

  • Never echo a secret, even in a debug step. It ends up in build logs that anyone with CI access can read.
  • Use masked variables in GitLab CI. The UI enforces log scrubbing for masked variables.
  • Rotate tokens regularly. A CI token that's been the same for three years has probably been seen by too many people.
  • Prefer short-lived tokens. GitLab CI's CI_JOB_TOKEN scopes registry access to the current job and expires automatically. Use it instead of a long-lived personal access token for registry push.

What 2 AM Actually Looks Like

A deployment hangs. You push a new image, pods start rolling, and then you notice the rollout percentage stuck at 33%. You check and find pods in CrashLoopBackOff.

kubectl describe pod order-service-7d4f9b9c6-xkj2p

Events: Liveness probe failed: HTTP probe failed with statuscode: 404.

The new image has a refactored Actuator config. The health endpoint moved from /actuator/health to /health. The liveness probe is pointing at the old path. Kubernetes kills the pod, starts a new one, it fails the probe, gets killed — loop.

The fix takes two minutes: update the probe path in the Deployment manifest and redeploy. The pipeline took 20 minutes because nobody caught the endpoint change in code review.

The lesson: probe paths are a pipeline concern. Add a smoke test to your integration stage that hits /actuator/health/liveness and /actuator/health/readiness on the Docker image before pushing. If the container doesn't pass its own health checks, fail the pipeline.

A complete GitLab CI pipeline skeleton for a Spring Boot service:

stages:
  - test
  - integration
  - docker
  - deploy

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2"

.maven-cache: &maven-cache
  cache:
    key: "$CI_PROJECT_PATH-maven"
    paths:
      - .m2/

unit-test:
  stage: test
  <<: *maven-cache
  script:
    - mvn test -q
  artifacts:
    reports:
      junit: target/surefire-reports/*.xml

integration-test:
  stage: integration
  <<: *maven-cache
  services:
    - postgres:16-alpine
  variables:
    POSTGRES_DB: testdb
    POSTGRES_USER: test
    POSTGRES_PASSWORD: test
    SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/testdb
  script:
    - mvn verify -DskipUnitTests=true -q

docker-build:
  stage: docker
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - trivy image --exit-code 1 --severity HIGH,CRITICAL $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

deploy-staging:
  stage: deploy
  script:
    - kubectl set image deployment/order-service
        order-service=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - kubectl rollout status deployment/order-service --timeout=5m
  environment: staging

A pipeline is only trustworthy if a green build means the thing actually works. Every skipped check, every --force push, every disabled test is a bet that you personally won't be on call the night it matters.

← back to blog