CircleCI

CircleCI is a cloud-hosted continuous integration and continuous delivery (CI/CD) platform founded in 2011. It automates the build, test, and deployment pipeline for software teams. CircleCI's key differentiators are speed (Docker layer caching, test parallelism, powerful executor fleet), a rich orbs ecosystem for reusable configuration, and multi-platform support (Linux, macOS, Windows, ARM). It is platform-agnostic, integrating with GitHub, GitLab, and Bitbucket as source code providers.

All CircleCI pipeline configuration lives in .circleci/config.yml at the root of your repository. The config file defines the version, optional orbs, jobs (units of work), and workflows (orchestration of jobs).

Config structure

The top-level keys in a CircleCI config are:

• version — always 2.1 (current version, enables orbs, reusable config, pipelines)
• orbs — import reusable configuration packages from the Orb Registry
• jobs — define units of work (executor, steps)
• workflows — orchestrate jobs (ordering, parallelism, filters, approvals)
• commands — reusable step sequences (like functions)
• executors — reusable executor definitions
• parameters — pipeline-level parameters for dynamic configuration

Job anatomy

A job defines the executor (runtime environment), working directory, environment variables, and a sequence of steps to execute.

# Key step types
steps:
  - checkout                  # Clone the repo
  - run:                      # Execute a shell command
      name: Install deps
      command: npm ci
  - restore_cache:            # Restore a previously saved cache
      keys:
        - v1-deps-{{ checksum "package-lock.json" }}
        - v1-deps-
  - save_cache:               # Save files to cache
      key: v1-deps-{{ checksum "package-lock.json" }}
      paths:
        - node_modules
  - store_artifacts:          # Upload build artifacts
      path: dist
  - store_test_results:       # Upload JUnit XML test results
      path: test-results
  - persist_to_workspace:     # Share files with downstream jobs
      root: .
      paths:
        - dist
  - attach_workspace:         # Attach files from upstream jobs
      at: .

Environment variables in config

Environment variables can be set at the job level, step level, or inherited from project/context settings.

jobs:
  build:
    docker:
      - image: cimg/node:20.11
    environment:
      NODE_ENV: production          # Job-level env var
    steps:
      - checkout
      - run:
          name: Build
          environment:
            CI: "true"              # Step-level env var
          command: npm run build

Complete sample config.yml

version: 2.1

orbs:
  node: circleci/node@5.2
  docker: circleci/docker@2.6

executors:
  node-executor:
    docker:
      - image: cimg/node:20.11
    working_directory: ~/project
    resource_class: medium

jobs:
  build:
    executor: node-executor
    steps:
      - checkout
      - node/install-packages:
          pkg-manager: npm
      - run:
          name: Build application
          command: npm run build
      - persist_to_workspace:
          root: .
          paths:
            - dist
            - node_modules

  test:
    executor: node-executor
    parallelism: 4
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run:
          name: Run tests
          command: |
            TESTFILES=$(circleci tests glob "src/**/*.test.ts" | circleci tests split --split-by=timings)
            npm test -- $TESTFILES
      - store_test_results:
          path: test-results
      - store_artifacts:
          path: coverage

  deploy:
    docker:
      - image: cimg/base:current
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run:
          name: Deploy to production
          command: |
            echo "Deploying version ${CIRCLE_SHA1:0:7}..."
            ./scripts/deploy.sh

workflows:
  build-test-deploy:
    jobs:
      - build
      - test:
          requires:
            - build
      - hold-for-approval:
          type: approval
          requires:
            - test
          filters:
            branches:
              only: main
      - deploy:
          requires:
            - hold-for-approval
          context: production-secrets
          filters:
            branches:
              only: main

Dynamic configuration

Dynamic configuration uses setup workflows to generate or modify the pipeline config at runtime. This is essential for monorepos where you only want to build changed services.

# Enable dynamic config in .circleci/config.yml
setup: true

orbs:
  continuation: circleci/continuation@1.0

jobs:
  generate-config:
    docker:
      - image: cimg/base:current
    steps:
      - checkout
      - run:
          name: Generate pipeline config
          command: ./scripts/generate-config.sh > /tmp/generated-config.yml
      - continuation/continue:
          configuration_path: /tmp/generated-config.yml

Config validation

# Validate config locally using the CircleCI CLI
circleci config validate

# Process and expand the config (resolve orbs, anchors)
circleci config process .circleci/config.yml

# Validate a specific config file
circleci config validate path/to/config.yml

Executors define the runtime environment for jobs. CircleCI supports multiple executor types, each suited for different workloads. The executor determines the OS, available tools, isolation level, and resource class options.

Docker executor

The most common executor. Runs jobs inside Docker containers. You can specify multiple images — the first is the primary container, additional images run as service containers (databases, caches) accessible via localhost.

jobs:
  test:
    docker:
      - image: cimg/node:20.11        # Primary container
        environment:
          DATABASE_URL: postgres://postgres:password@localhost:5432/testdb
      - image: cimg/postgres:16.2      # Service container
        environment:
          POSTGRES_PASSWORD: password
          POSTGRES_DB: testdb
      - image: cimg/redis:7.2          # Another service container
    resource_class: medium
    steps:
      - checkout
      - run: npm ci
      - run: npm test

CircleCI provides convenience images (cimg/*) that are optimized for CI:

• cimg/base — minimal Ubuntu image with common CI tools
• cimg/node — Node.js with npm/yarn
• cimg/python — Python with pip
• cimg/go — Go with standard toolchain
• cimg/rust — Rust with cargo
• cimg/openjdk — Java with OpenJDK

Machine executor

Runs jobs in a full Linux VM. Required when you need Docker-in-Docker, privileged operations, or full kernel access.

jobs:
  integration:
    machine:
      image: ubuntu-2404:current
    resource_class: medium
    steps:
      - checkout
      - run:
          name: Build and test with Docker Compose
          command: |
            docker compose up -d
            docker compose run tests
            docker compose down

macOS executor

jobs:
  ios-build:
    macos:
      xcode: "16.2.0"
    resource_class: macos.m4pro.medium
    steps:
      - checkout
      - run: xcodebuild -scheme MyApp -sdk iphonesimulator -destination 'platform=iOS Simulator,name=iPhone 16' test

Windows executor

jobs:
  windows-build:
    machine:
      image: windows-server-2022-gui:current
    resource_class: windows.medium
    steps:
      - checkout
      - run:
          name: Build with MSBuild
          command: MSBuild.exe MyProject.sln /p:Configuration=Release
          shell: powershell.exe

Self-hosted runner

Run jobs on your own infrastructure. Install the CircleCI runner agent on any machine. Route jobs to runners using custom resource classes.

jobs:
  deploy:
    machine: true
    resource_class: myorg/my-private-runner
    steps:
      - checkout
      - run: ./deploy.sh

Executor comparison

Workflows orchestrate the execution of jobs. They define job ordering, parallelism, dependencies, filters, scheduled triggers, and manual approval gates. A single pipeline can contain multiple workflows.

Sequential and parallel execution

workflows:
  build-and-test:
    jobs:
      - build                    # Runs first
      - test-unit:               # Runs after build
          requires:
            - build
      - test-integration:        # Runs in parallel with test-unit
          requires:
            - build
      - deploy:                  # Runs after BOTH tests pass
          requires:
            - test-unit
            - test-integration

Workflow topology

Fan-in / fan-out

workflows:
  fan-out-fan-in:
    jobs:
      - checkout-code
      - lint:
          requires: [checkout-code]
      - test-api:
          requires: [checkout-code]
      - test-web:
          requires: [checkout-code]
      - test-worker:
          requires: [checkout-code]
      - build-image:
          requires: [lint, test-api, test-web, test-worker]

Workflow filters

Filters control which branches and tags trigger specific jobs.

workflows:
  deploy-pipeline:
    jobs:
      - build:
          filters:
            branches:
              only:
                - main
                - develop
      - deploy-staging:
          requires: [build]
          filters:
            branches:
              only: develop
      - deploy-production:
          requires: [build]
          filters:
            branches:
              only: main
      - release:
          filters:
            tags:
              only: /^v\d+\.\d+\.\d+$/
            branches:
              ignore: /.*/

Scheduled workflows

The legacy triggers syntax below still works but is deprecated. The recommended approach is scheduled pipelines, configured via the CircleCI UI or API under Project Settings → Triggers. Scheduled pipelines offer better actor control, dynamic config support, and single-schedule-to-multiple-workflows mapping.

# Legacy triggers syntax (still functional, but prefer scheduled pipelines)
workflows:
  nightly-security-scan:
    triggers:
      - schedule:
          cron: "0 2 * * *"       # Every night at 2 AM UTC
          filters:
            branches:
              only: main
    jobs:
      - security-scan
      - dependency-audit

Manual approval jobs

workflows:
  deploy-with-approval:
    jobs:
      - build
      - test:
          requires: [build]
      - hold:
          type: approval           # Pauses workflow until approved in UI
          requires: [test]
      - deploy-production:
          requires: [hold]
          context: production-secrets

Workspaces: sharing data between jobs

Workspaces allow jobs within the same workflow to share files. Use persist_to_workspace in an upstream job and attach_workspace in a downstream job.

jobs:
  build:
    docker:
      - image: cimg/node:20.11
    steps:
      - checkout
      - run: npm ci && npm run build
      - persist_to_workspace:
          root: .
          paths:
            - dist
            - node_modules

  deploy:
    docker:
      - image: cimg/base:current
    steps:
      - attach_workspace:
          at: ~/project
      - run: ls ~/project/dist    # Files from build job are here

Orbs are reusable, shareable packages of CircleCI configuration. They encapsulate commands, jobs, and executors into a single importable unit. Orbs dramatically reduce config boilerplate and promote best practices across projects.

What orbs contain

Using orbs

version: 2.1

orbs:
  node: circleci/node@5.2          # Import the Node.js orb
  aws-cli: circleci/aws-cli@4.1    # Import the AWS CLI orb
  docker: circleci/docker@2.6      # Import the Docker orb
  slack: circleci/slack@4.13       # Import the Slack orb

jobs:
  build:
    executor: node/default          # Use orb's executor
    steps:
      - checkout
      - node/install-packages       # Use orb's command
      - run: npm run build

workflows:
  main:
    jobs:
      - build
      - docker/publish:             # Use orb's job directly
          image: myorg/myapp
          tag: ${CIRCLE_SHA1:0:7}
          requires: [build]

Popular orbs

Creating custom orbs

# Initialize a new orb project
circleci orb init my-orb

# Validate the orb source
circleci orb validate src/@orb.yml

# Pack the orb from source directory
circleci orb pack src/ > orb.yml

# Publish a development version
circleci orb publish orb.yml myorg/my-orb@dev:alpha

# Promote to a semantic version
circleci orb publish promote myorg/my-orb@dev:alpha patch

Orb versioning

• Semantic versioning — myorg/my-orb@1.2.3. Pin to major (@1), minor (@1.2), or exact (@1.2.3).
• Volatile (dev) — myorg/my-orb@dev:alpha. Mutable, used during development. Expires after 90 days.
• Private orbs — organization-scoped orbs not visible in the public registry. Available on paid plans.

CircleCI provides three mechanisms for persisting data: caches (across builds), workspaces (within a workflow), and artifacts (build outputs). Understanding when to use each is key to fast, efficient pipelines.

Caching dependencies

Caches persist between pipeline runs. Use them for dependencies that rarely change (node_modules, pip packages, Go modules). Cache keys support template variables for intelligent invalidation.

steps:
  - restore_cache:
      keys:
        # Exact match first
        - v1-deps-{{ checksum "package-lock.json" }}
        # Fallback to most recent cache with this prefix
        - v1-deps-
  - run: npm ci
  - save_cache:
      key: v1-deps-{{ checksum "package-lock.json" }}
      paths:
        - node_modules
        - ~/.npm

Common cache key templates:

• {{ checksum "file" }} — SHA256 of a file (lockfile changes invalidate cache)
• {{ .Branch }} — current branch name
• {{ .Revision }} — current git SHA
• {{ epoch }} — Unix timestamp (always unique, use for forced invalidation)
• {{ .Environment.variableName }} — environment variable value

Docker layer caching (DLC)

Docker layer caching is a premium feature that caches Docker image layers between builds. When building images, unchanged layers are reused instead of rebuilt, saving significant time.

jobs:
  build-image:
    machine:
      image: ubuntu-2404:current
      docker_layer_caching: true     # Enable DLC
    steps:
      - checkout
      - run: docker build -t myapp:latest .

Artifacts

Artifacts are files produced by a job that you want to persist and access after the build. They are browsable in the CircleCI UI and downloadable via the API.

steps:
  - run: npm run build
  - run: npm test -- --coverage
  - store_artifacts:
      path: dist
      destination: build-output
  - store_artifacts:
      path: coverage
      destination: coverage-report

Test results

Use store_test_results to upload JUnit XML test results. CircleCI parses these for test insights, flaky test detection, and test splitting by timing data.

steps:
  - run:
      name: Run tests with JUnit reporter
      command: |
        JEST_JUNIT_OUTPUT_DIR=test-results \
        npm test -- --ci --reporters=jest-junit
  - store_test_results:
      path: test-results

Comparison: cache vs workspace vs artifact

CircleCI provides several mechanisms for managing secrets: project-level environment variables, organization-level contexts, OIDC tokens for cloud authentication, and integrations with external secret managers.

Project environment variables

Set in the CircleCI UI under Project Settings → Environment Variables. Available to all jobs in the project. Values are masked in job output logs.

• Set via UI or API (e.g., POST /api/v2/project/{project-slug}/envvar)
• Cannot be read back once set (write-only in UI)
• Available as standard environment variables in jobs
• Forked PRs do not receive project secrets (security measure)

Contexts

Contexts are organization-level groups of environment variables. They are shared across projects and assigned to jobs in workflows.

workflows:
  deploy:
    jobs:
      - deploy-staging:
          context: staging-secrets
      - deploy-production:
          context:
            - production-secrets
            - shared-aws-credentials

Context features:

• Security groups — restrict which teams/users can trigger jobs using a context
• Context expressions — conditional access based on branch, tag, or other attributes
• Audit trail — track which pipelines used which contexts
• Multiple contexts per job — compose secrets from multiple context groups

OIDC tokens

CircleCI OIDC provides short-lived tokens for authenticating with cloud providers (AWS, GCP, Azure) without storing long-lived credentials. The token contains claims about the pipeline (org_id, project_id, branch, etc.).

jobs:
  deploy-to-aws:
    docker:
      - image: cimg/aws:2025.01
    environment:
      AWS_ROLE_ARN: arn:aws:iam::123456789012:role/circleci-deploy
    steps:
      - checkout
      - run:
          name: Authenticate with AWS via OIDC
          command: |
            # CircleCI provides $CIRCLE_OIDC_TOKEN automatically
            aws sts assume-role-with-web-identity \
              --role-arn $AWS_ROLE_ARN \
              --role-session-name circleci-${CIRCLE_BUILD_NUM} \
              --web-identity-token $CIRCLE_OIDC_TOKEN \
              --duration-seconds 3600 > /tmp/creds.json
            export AWS_ACCESS_KEY_ID=$(jq -r '.Credentials.AccessKeyId' /tmp/creds.json)
            export AWS_SECRET_ACCESS_KEY=$(jq -r '.Credentials.SecretAccessKey' /tmp/creds.json)
            export AWS_SESSION_TOKEN=$(jq -r '.Credentials.SessionToken' /tmp/creds.json)
            aws s3 cp dist/ s3://my-bucket/ --recursive

HashiCorp Vault integration

jobs:
  deploy:
    docker:
      - image: cimg/base:current
    steps:
      - checkout
      - run:
          name: Fetch secrets from Vault
          command: |
            # Authenticate to Vault using AppRole
            VAULT_TOKEN=$(vault write -field=token auth/approle/login \
              role_id=$VAULT_ROLE_ID \
              secret_id=$VAULT_SECRET_ID)
            export VAULT_TOKEN

            # Read secrets
            DB_PASSWORD=$(vault kv get -field=password secret/myapp/db)
            API_KEY=$(vault kv get -field=key secret/myapp/api)

            # Use secrets (never echo them)
            ./deploy.sh

CircleCI provides several features to dramatically speed up your pipelines: test splitting with parallelism, resource classes, Docker layer caching, and pipeline optimization techniques.

Test splitting with parallelism

Set parallelism: N on a job to run N copies simultaneously. Use circleci tests split to distribute test files across containers.

jobs:
  test:
    docker:
      - image: cimg/node:20.11
    parallelism: 4                   # Run 4 copies of this job
    steps:
      - checkout
      - run: npm ci
      - run:
          name: Run tests (split by timing)
          command: |
            # Glob test files, split by historical timing data
            TESTFILES=$(circleci tests glob "src/**/*.test.ts" \
              | circleci tests split --split-by=timings)
            echo "Running tests: $TESTFILES"
            npx jest $TESTFILES --ci --reporters=jest-junit
      - store_test_results:
          path: test-results

Before and after: parallelism speedup

Total test time: 24 minutes
Container 1: 400 tests (24 min)

Total test time: ~6 minutes
Container 1: 100 tests (6 min)
Container 2: 100 tests (6 min)
Container 3: 100 tests (5.8 min)
Container 4: 100 tests (5.9 min)

Resource classes

Resource classes control the CPU and RAM allocated to a job. Larger resource classes run faster but consume more credits. The table below shows Docker executor classes; Linux VM (machine) and macOS classes have different specs and credit rates.

Docker layer caching

# Machine executor with DLC
jobs:
  build-image:
    machine:
      image: ubuntu-2404:current
      docker_layer_caching: true
    steps:
      - checkout
      - run: |
          docker build -t myapp:$CIRCLE_SHA1 .
          # Second build reuses cached layers
          # Typical savings: 2-10 minutes per build

Pipeline optimization

• Path filtering — use dynamic configuration to only run jobs for changed services in a monorepo
• Conditional steps — use when/unless conditions to skip unnecessary steps
• Skip CI — add [ci skip] or [skip ci] to commit messages to skip the pipeline entirely
• Cache aggressively — cache dependencies, build tools, and intermediate build artifacts
• Right-size resources — use small for linting, medium for builds, large for compilation-heavy jobs

# Conditional steps example
steps:
  - run:
      name: Deploy (only on main)
      command: ./deploy.sh
      when: on_success
  - when:
      condition:
        equal: [main, << pipeline.git.branch >>]
      steps:
        - run: echo "This only runs on main"

CircleCI integrates with a wide range of tools and platforms across the software delivery lifecycle. Most integrations are available as orbs, making setup straightforward.

Git platforms

Cloud providers

Container registries

# Build and push to multiple registries using the Docker orb
orbs:
  docker: circleci/docker@2.6

workflows:
  build-and-push:
    jobs:
      - docker/publish:
          image: myorg/myapp
          tag: ${CIRCLE_SHA1:0:7},latest
          registry: docker.io          # Docker Hub
      - docker/publish:
          image: myapp
          tag: ${CIRCLE_SHA1:0:7}
          registry: 123456789012.dkr.ecr.us-east-1.amazonaws.com  # ECR

Notifications

# Slack notifications using the Slack orb
orbs:
  slack: circleci/slack@4.13

jobs:
  deploy:
    docker:
      - image: cimg/base:current
    steps:
      - run: ./deploy.sh
      - slack/notify:
          event: pass
          template: success_tagged_deploy_1
      - slack/notify:
          event: fail
          mentions: "@oncall-team"
          template: basic_fail_1

Deployment targets

CircleCI Server is the self-hosted version of the CircleCI platform. It runs on Kubernetes and provides the same features as CircleCI Cloud, but within your own infrastructure. Use it when you need air-gapped environments, data residency, or strict compliance controls.

When to use CircleCI Server

• Air-gapped environments — code and build artifacts never leave your network
• Data residency — ensure all CI/CD data stays in a specific geographic region
• Compliance — FedRAMP, ITAR, HIPAA, or other regulations requiring self-hosted infrastructure
• Network restrictions — builds need access to internal services not exposed to the internet

Architecture

CircleCI Server is deployed as a Kubernetes application via Helm. The major components are:

Installation

# Add the CircleCI Helm repository
helm repo add circleci https://circleci-public.github.io/server-helm
helm repo update

# Create a values file with your configuration
# (license, GitHub/GitLab connection, storage, TLS, etc.)

# Install CircleCI Server
helm install circleci-server circleci/circleci-server \
  --namespace circleci-server \
  --create-namespace \
  --values values.yaml

# Check installation status
kubectl get pods -n circleci-server

Supported Kubernetes platforms

Differences from CircleCI Cloud

• You manage infrastructure — Kubernetes cluster, databases, object storage, TLS certificates
• You manage updates — Helm chart upgrades on your schedule (no automatic updates)
• License required — contact CircleCI sales for server licensing
• Feature parity gap — some cloud features may arrive on server with a delay
• Self-hosted runners — same runner agent works with both cloud and server

Securing your CI/CD pipeline is critical — it has access to source code, secrets, and production infrastructure. CircleCI provides multiple layers of security controls for pipelines, secrets, and organizational governance.

Pipeline security

Runner security

Self-hosted runners keep sensitive code on your infrastructure. Code is checked out, built, and tested on machines you control. Build artifacts never leave your network. Ideal for regulated industries.

Audit logs

CircleCI provides organization-level audit logs (Scale plan and above) that track:

• Context creation, deletion, and variable changes
• Project settings modifications
• User additions and removals
• Pipeline triggers and approval actions
• SSH access to build containers

# Download audit logs via API
curl -H "Circle-Token: $CIRCLECI_TOKEN" \
  "https://circleci.com/api/v2/organization/${ORG_ID}/audit-log?start-date=2026-03-01" \
  | jq '.items[]'

Compliance certifications

Supply chain security

# Sign container images with Cosign in your pipeline
jobs:
  sign-image:
    docker:
      - image: cimg/base:current
    steps:
      - run:
          name: Sign image with Cosign
          command: |
            cosign sign --key env://COSIGN_PRIVATE_KEY \
              myregistry.com/myapp:${CIRCLE_SHA1:0:7}
      - run:
          name: Generate SBOM
          command: |
            syft packages myregistry.com/myapp:${CIRCLE_SHA1:0:7} \
              -o spdx-json > sbom.json
      - store_artifacts:
          path: sbom.json

Config policies

Organization-level config policies allow administrators to enforce standards across all projects. Policies are written in OPA (Open Policy Agent) Rego language.

# Example policy: require approval for production deploys
package org

import future.keywords

deny[msg] {
  some job in input.workflows[_].jobs[_]
  job[_].context[_] == "production-secrets"
  not has_approval_before(input.workflows, job)
  msg := "Production context requires an approval job"
}

• Use config version 2.1 — enables orbs, reusable config, pipeline parameters, and all modern features. There is no reason to use older versions.
• Validate config before pushing — run circleci config validate locally. Add it to pre-commit hooks to catch YAML errors before they waste pipeline credits.
• Use orbs for common tasks — don't reinvent AWS CLI setup, Docker build/push, or Slack notifications. Use official orbs and pin them to a major version.
• Use contexts for shared secrets — prefer organization-level contexts over project environment variables. Restrict sensitive contexts to security groups.
• Enable OIDC for cloud access — use CircleCI OIDC tokens instead of stored AWS/GCP/Azure credentials. Eliminates credential rotation burden.
• Enable test splitting with parallelism — for test suites over 5 minutes, enable parallelism and circleci tests split --split-by=timings. Upload JUnit XML via store_test_results to feed timing data.
• Cache dependencies aggressively — use save_cache / restore_cache with checksum-based keys for all dependency directories. Always include a fallback key prefix.
• Right-size resource classes — use small for linting, medium for standard builds, large or xlarge for compilation-heavy jobs. Monitor credit usage and adjust.
• Use approval gates for production — add type: approval jobs before production deployments. Combine with restricted contexts for defense in depth.
• Use workflow filters — restrict production deploys to main branch. Use tag filters for release workflows. Never deploy from feature branches.
• Enable Docker layer caching — if you build Docker images, enable DLC to save 2-10 minutes per build. Worth the premium cost for frequent builders.
• Store test results — always upload JUnit XML with store_test_results. Enables test insights, flaky test detection, and timing-based test splitting.
• Use dynamic config for monorepos — set up dynamic configuration with path filtering to only build changed services. Saves significant credits and time.
• Set up notifications — use the Slack orb to notify on failures and deployment events. Ensure the team knows when builds break or deploys complete.
• Enable config policies — for organizations with multiple teams, use config policies to enforce standards (approval gates, context restrictions, resource class limits).
• Audit and rotate secrets — review context and project environment variables quarterly. Remove unused secrets. Rotate credentials on a regular schedule.