This tutorial demonstrates how to set up a multi-cloud architecture with Amazon Web Services (AWS) and Google Cloud. Working with a JavaScript monorepo, you will learn how to deploy a Node.js + MySQL server app to AWS Elastic Container Service (ECS) with the EC2 (Elastic Compute Cloud) launch type, and a React client to Google Cloud Run using CircleCI orbs.

By the end, you will have a fully functional, production-ready pipeline that delivers your app across two major clouds. You will also learn valuable tips to apply to real-world projects.

Prerequisites

To follow this tutorial, here is a checklist of what you will need:

• A CircleCI account.

• A Docker account to publish images to Docker Hub.

• An AWS account with the AWS CLI installed and configured. Follow the instructions on the Authenticating using IAM user credentials for the AWS CLI page to:

  • Create an IAM user with administrative access (you should avoid using the root user to spin up AWS resources).

  • Generate two sets of credentials for CLI and CircleCI.

  • Install the AWS CLI on your machine.

  • Add your credentials and configure the IAM user profile for CLI use.

• A Google Cloud account with billing enabled and the gcloud CLI configured.

• A terminal to run bash scripts, with OpenSSL and envsubst installed.

• Git installed on your local machine.

• A GitHub account.

• An IDE or code editor of your choice.

Preparing the cloud infrastructure

Before getting into the CI/CD pipeline, you need to set up the infrastructure where CircleCI will deploy the client and server apps. Once created, both services will be empty until the first CI/CD pipeline runs. After the pipeline config is ready, you will push your code to trigger CircleCI and automatically update the services. We will kick things off with AWS.

First, make sure you have the application code on your machine. Fork the repository on GitHub and clone it:

git clone <your-fork-url>

# navigate into the project directory
cd cara-store-catalog

Provisioning AWS resources

Inside the deployments/ directory, you will find two main folders, ecs/ and cloud-run/. Within them are scripts and configuration files to automate the process of provisioning resources on their respective cloud platforms.

One such file is ecs/scripts/export-env.template.sh, which exports the custom environment variables required for setup. Rename this file to export-env.sh and fill in the following values:

export AWS_PROFILE=
export DOCKERHUB_USERNAME=
export MYSQL_PASSWORD=
export MYSQL_ROOT_PASSWORD=

Next, rename .env.circleci.template to .env.circleci. This file contains a list of all the environment variables you’ll need to add to CircleCI. The setup script will auto-populate some of them (by executing set-circleci-env.sh), so all you have to do is copy and paste into CircleCI when needed.

Now, you can run the script to create the deployment resources. The script assumes your AWS account already has a default Virtual Private Cloud (VPC) configured with networking components like subnets and an internet gateway.

From the project’s root directory, enter these commands:

# add execution permissions
chmod +x deployments/ecs/scripts/export-env.sh deployments/ecs/scripts/set-circleci-env.sh deployments/ecs/scripts/setup-ecs.sh

# run the script
source deployments/ecs/scripts/setup-ecs.sh

Here’s an overview of what this does:

• Creates an ECS cluster.

• Sets up AWS Secrets Manager by:

  • Substituting the placeholders in templates/ecs-secrets.template.json using the values exported from export-env.sh.

  • Uploading the env variables in init/ecs-secrets.json.

• Creates IAM and Task Execution roles for the EC2 instance.

• Creates an application load balancer (ALB) and defines two security groups for the ALB and the ECS instance. This opens up the ALB for public access while it communicates requests to the server app running on the ECS instance.

• Generates and uploads a self-signed TLS certificate to allow https support for the ALB URL. This ensures the client app can communicate with the server securely.

• Creates a key pair to allow secure SSH access to the EC2 instance.

• Launches an EC2 instance and uploads the database initialization script server/db/init.sql.

• Runs set-circleci-env.sh to populate .env.circleci with the available values.

The setup script stops short of registering the task definition and creating the ECS service to keep infrastructure setup separate from deployment. If everything succeeds, you should see this message printed to the console:

[SUCCESS] ECS infrastructure with EC2 launch type successfully created
          Service will be launched and deployed via CircleCI

After the script completes, open the ECS and EC2 dashboards in your AWS account to confirm the creation of resources such as the load balancer, target group, and ECS cluster.

Inside the scripts/ folder are a few other scripts like pause-ecs.sh and resume-ecs.sh. Use these to suspend or restart the ECS service temporarily to save costs.

Provisioning Google Cloud resources

You might be wondering why the React app is Dockerized instead of hosted as a static website on Google Cloud Storage. While static hosting does offer simplicity, using Docker provides a unified way to manage the client app across environments. It allows you to spin up multiple instances of your app, enabling easy testing, scaling, and consistent behavior across deployments.

The Google Cloud Run setup files are in the deployments/cloud-run/ directory. Rename the cloud-run/scripts/export-env.template.sh file to export-env.sh and update these values from your Google Cloud account:

export GCP_REGION=
export BILLING_ACCOUNT_NAME=""

Just like with AWS, run the setup script for Google Cloud Run from the project’s root directory:

# add execution permissions
chmod +x deployments/cloud-run/scripts/export-env.sh deployments/cloud-run/scripts/set-circleci-env.sh deployments/cloud-run/scripts/setup-cloud-run.sh

# run the script
source deployments/cloud-run/scripts/setup-cloud-run.sh

What this script does:

• Creates a new project on Google Cloud.

• Links the project to the specified billing account.

• Enables all required APIs.

• Creates a runtime service account for the Cloud Run instance and a deployer service account for CircleCI.

• Creates a JSON key for CircleCI to authenticate with Google Cloud.

• Runs the Cloud Run set-circleci-env.sh to populate values into .env.circleci.

A successful setup should print a message like this to the console:

[SUCCESS] Cloud Run service environment prepared:
          Service: carastore-client
          App will be deployed via CircleCI

With both AWS and Google Cloud configured, this concludes the infrastructure setup portion of this tutorial. Next up is the CI/CD pipeline.

Configuring the CI/CD pipeline for deployment

Since we’re working with a monorepo, the best approach is to use CircleCI’s dynamic configuration. This allows the configuration of different workflows to run based on file changes. If you are unfamiliar with the concept, check out my tutorial on dynamic configuration with CircleCI.

In the project’s root directory, create the following folder and files:

mkdir .circleci

cd .circleci

touch config.yml continue_config.yml

The setup file

Add this code to config.yml:

version: 2.1

setup: true

orbs:
  path-filtering: circleci/path-filtering@2.0.1

workflows:
  filter-path:
    jobs:
      - path-filtering/filter:
          name: detect-modified-directories
          mapping: |
            server/.* run-server-jobs true
            client/.* run-client-jobs true
            .circleci/.* run-server-jobs true
            .circleci/.* run-client-jobs true
            deployments/ecs/.* run-task-job true
          base-revision: main

This configuration uses the filter job from the path-filtering orb to detect changes and set these pipeline parameters:

• run-server-jobs: triggers the server workflow when server/ files change.

• run-client-jobs: triggers the client workflow when client/ files change.

• run-task-job triggers the server deployment job within the server workflow when changes occur in the deployments/ecs/ folder. This ensures CircleCI updates the task definition and the ECS service accordingly.

Continuation configuration

After setting the pipeline parameters, CircleCI automatically executes the continuation config file and passes those pipeline values to it.

Open continue_config.yml and add this:

version: 2.1

parameters:
  run-server-jobs:
    type: boolean
    default: false
  run-client-jobs:
    type: boolean
    default: false
  run-task-job:
    type: boolean
    default: false

orbs:
  gcp-cli: circleci/gcp-cli@3.2.2
  aws-ecs: circleci/aws-ecs@7.1.0
  aws-cli: circleci/aws-cli@5.4.1

executors:
  node-exec:
    docker:
      - image: node:22.17-alpine3.22
  base-exec:
    docker:
      - image: cimg/base:current-24.04

commands:
  installdeps:
    description: "Install dependencies"
    parameters:
      directory:
        type: string
    steps:
      - checkout:
          path: ~/project
      - restore_cache:
          keys:
            - v1-<< parameters.directory >>-deps-{{ checksum "package.json" }}-{{ checksum "package-lock.json" }}
            - v1-<< parameters.directory >>-deps-{{ checksum "package.json" }}
            - v1-<< parameters.directory >>-deps-
      - run:
          name: Install dependencies
          command: npm ci
      - save_cache:
          key: v1-<< parameters.directory >>-deps-{{ checksum "package.json" }}-{{ checksum "package-lock.json" }}
          paths:
            - node_modules
  get-image-tag:
    description: "Retrieve tag for Docker image"
    parameters:
      directory:
        type: string
    steps:
      - run:
          name: Export version from package.json
          command: |
            IMAGE_TAG=$(jq -r '.version' ~/project/<< parameters.directory >>/package.json)
            echo "export IMAGE_TAG=$IMAGE_TAG" >> $BASH_ENV
            echo "IMAGE_TAG: $IMAGE_TAG"
            source $BASH_ENV

The bulk of this code declares reusable elements to avoid unnecessary code duplication:

• Default values for the pipeline parameters.

• CircleCI orbs to simplify the deployment setup.

• Executors to define the execution environment for the steps in each job.

• Commands to run across multiple jobs:

  • installdeps installs npm dependencies, stores them in a cache, and restores them in subsequent builds.

  • get-image-tag dynamically retrieves the tag number from each app’s package.json file.

Start defining the jobs for the pipeline by adding the test jobs:

jobs:
  test-client:
    executor: node-exec
    working_directory: ~/project/client
    steps:
      - installdeps:
          directory: client
      - run:
          name: Run client tests
          command: npm test

  test-server:
    executor: base-exec
    working_directory: ~/project/server
    steps:
      - checkout:
          path: ~/project
      - setup_remote_docker:
          docker_layer_caching: true
      - get-image-tag:
          directory: server
      - run:
          name: Spin up containers and run server tests
          command: |
            docker images
            docker compose -f compose.yaml -f compose.cicd.yaml up --build -d
            docker exec -it --user root nodejs-server-prod npm install
            docker exec -it nodejs-server-prod npm test
      - run:
          name: Stop and remove containers
          command: docker compose down -v

test-client uses the node-exec executor to maintain consistency with the Node.js version defined in the client app’s Dockerfile. It invokes the reusable installdeps command and then calls the test command specified in the client app’s package.json file. The tests must be run directly on the source code since they can’t run effectively on a static Dockerized build served by NGINX.

The test-server job’s structure is a little different. It uses the base-exec executor, which is CircleCI’s official Ubuntu Docker image. Here, tests can run on the server app’s Docker image, so the job spins up a container, installs dependencies as the root user, and runs tests inside it. The compose.cicd.yaml file is a Docker Compose merge file which specifies the production configuration. In contrast, the compose.override.yaml file in the server/ folder contains the development configuration.

Update continue_config.yml with the build job:

  build-docker-image:
    description: Build and publish << parameters.service >> Docker image
    parameters:
      service:
        type: string
      build_context:
        type: string
    executor: base-exec
    working_directory: ~/project/<< parameters.service >>
    steps:
      - checkout:
          path: ~/project
      - setup_remote_docker:
          docker_layer_caching: true
      - get-image-tag:
          directory: << parameters.service >>
      - run:
          name: Build << parameters.service >> image
          command: |
            docker build \
              -t $DOCKERHUB_USERNAME/carastore-<< parameters.service >>:$IMAGE_TAG \
              -t $DOCKERHUB_USERNAME/carastore-<< parameters.service >>:latest \
              << parameters.build_context >>
      - run:
          name: Authenticate and push image to Docker Hub
          command: |
            echo "$DOCKERHUB_PASSWORD" | docker login -u $DOCKERHUB_USERNAME --password-stdin
            docker push -a $DOCKERHUB_USERNAME/carastore-<< parameters.service >>

Both client and server apps share a similar build process, so the build-docker-image takes on a parameterized structure to make it reusable across workflows.

Add the client deployment job:

  deploy-client:
    executor: gcp-cli/default
    steps:
      - checkout
      - get-image-tag:
          directory: client
      - gcp-cli/setup:
          gcloud_service_key: GCLOUD_SERVICE_KEY
          google_compute_region: GCP_REGION
          google_project_id: GOOGLE_PROJECT_ID
      - run:
          name: Deploy to Google Cloud Run
          command: |
            IMAGE="docker.io/$DOCKERHUB_USERNAME/carastore-client:$IMAGE_TAG"

            gcloud run deploy "$SERVICE_NAME" \
              --image "$IMAGE" \
              --service-account "$RUNTIME_SA_EMAIL" \
              --allow-unauthenticated \
              --region "$GCP_REGION" \
              --platform managed \
              --port $CLIENT_APP_PORT \
              --cpu 1 \
              --memory 512Mi \
              --min-instances 0 \
              --max-instances 1 \
              --set-env-vars VITE_API_URL=$VITE_API_URL
      - run:
          name: Verify deployment success
          command: |
            URL=$(gcloud run services describe "$SERVICE_NAME" \
              --region "$GCP_REGION" \
              --format='value(status.url)')
            echo "[INFO] Service URL: $URL"
            for i in {1..30}; do
              STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$URL")
              if [ "$STATUS" -eq 200 ]; then
                echo "[SUCCESS] Deployment verified"
                exit 0
              fi
              echo "[WARN] Got $STATUS, retrying in 10s... ($i/30)"
              sleep 10
            done
            echo "[ERROR] Deployment verification timed out after 5 minutes"
            exit 1

Here, the previously defined gcp-cli orb installs and configures the gcloud CLI. You already set up the infrastructure by running the setup-cloud-run.sh script, so this job handles the client app deployment to Google Cloud Run. The final step confirms whether the deployment was successful.

Next, add the server deployment job:

  deploy-server:
    executor: base-exec
    working_directory: ~/project/deployments/ecs
    steps:
      - checkout:
          path: ~/project
      - get-image-tag:
          directory: server
      - run:
          name: Substitute env placeholders in task definition
          command: |
            envsubst < templates/task-definition.template.json > task-definition.json
      - aws-cli/setup:
          aws_access_key_id: $AWS_ACCESS_KEY_ID
          aws_secret_access_key: $AWS_SECRET_ACCESS_KEY
          region: $AWS_REGION
      - aws-ecs/update_task_definition_from_json:
          region: $AWS_REGION
          task_definition_json: task-definition.json
      - aws-ecs/update_service:
          region: $AWS_REGION
          family: "$MY_APP_PREFIX-server"
          service_name: "$MY_APP_PREFIX-service"
          cluster: "$MY_APP_PREFIX-cluster"
          create_service: true
          desired_count: "1"
          container_name: "nodejs-server"
          container_port: "5000"
          target_group: $TG_ARN
          skip_task_definition_registration: true
      - aws-ecs/verify_revision_is_deployed:
          region: $AWS_REGION
          family: "$MY_APP_PREFIX-server"
          service_name: "$MY_APP_PREFIX-service"
          cluster: "$MY_APP_PREFIX-cluster"
          task_definition_arn: $CCI_ORB_AWS_ECS_REGISTERED_TASK_DFN
          max_poll_attempts: 20

  deployment-coordinator:
    type: no-op

There are two CircleCI orbs working hand-in-hand here to execute the server deployment: aws-cli to install and configure the AWS CLI, and aws-ecs to manage the core deployment process.

This job can be summarized in four major steps:

1. envsubst replaces placeholders in task-definition.template.json with values from CircleCI project environment variables and outputs a valid JSON task definition.

2. The in-built aws-ecs/update_task_definition_from_json command registers a new task definition with this file.

3. aws-ecs/update_service handles the core service deployment with the specified parameters.

4. And finally, aws-ecs/verify_revision_is_deployed confirms if the rollout was successful.

The deployment-coordinator job is a no-op that performs no specific action or consumes credits. Its purpose is to ensure the server app gets re-deployed if only the task definition changes.

This will become clearer after adding the final piece of the config, workflows:

workflows:
  test-build-and-deploy-client:
    when: << pipeline.parameters.run-client-jobs >>
    jobs:
      - test-client
      - build-docker-image:
          name: build-client-image
          service: client
          build_context: .
          requires:
            - test-client
      - deploy-client:
          requires:
            - build-client-image

  build-test-and-deploy-server:
    when:
      or: [<< pipeline.parameters.run-server-jobs >>, << pipeline.parameters.run-task-job >>]
    jobs:
      - build-docker-image:
          name: build-server-image
          service: server
          build_context: --target prod .
          filters:
            pipeline.parameters.run-server-jobs
      - test-server:
          requires:
            - build-server-image
          filters:
            pipeline.parameters.run-server-jobs
      - deployment-coordinator
      - deploy-server:
          requires:
            - test-server
            - deployment-coordinator

Workflows determine how (and if) the outlined jobs should run. In both of these, CircleCI will only trigger the listed jobs “when” the specified pipeline parameters evaluate to true.

A brief explanation of what is going on in the build-test-and-deploy-server workflow:

• The logical or operator means the workflow will only run when one OR both pipeline parameters are true.

• If both or only the run-server-jobs pipeline parameter is true, it runs all jobs.

• The filters rule skips the build-docker-image and test-server jobs if only run-task-job is true. In that case, the no-op deployment-coordinator ensures deploy-server still runs, because it remains as a dependency, even when test-server gets filtered out. Without it, deploy-server would have no valid dependency, and CircleCI would skip it.

• The requires key stalls execution until the dependent job attains the default success status.

While the typical flow is build → test → deploy in most CI/CD pipelines, the client and server workflows diverge slightly in practice:

• Client workflow runs tests before the Docker build, since tests cannot run meaningfully on a static image.

• Server workflow runs the Docker build before tests, since tests depend on the built production image.

So, while technical constraints mean a disparity in order, the overall progression is the same: validate and package, then deploy.

Running your deployment pipeline on CircleCI

To connect your project to CircleCI, start by following the instructions on the Set up a project page. CircleCI will prompt you with a few options. For now, select Commit a starter CI pipeline to a new branch. This will immediately trigger a successful test pipeline on a new branch, confirming CircleCI has connected your project correctly.

Next, open the deployments/env.circleci file you renamed earlier. It contains a list of all the environment variables required for CircleCI and should have the following values pre-populated:

• ACCOUNT_ID

• AWS_REGION

• GCP_REGION

• RUNTIME_SA_EMAIL

• TG_ARN

• VITE_API_URL

Follow the instructions on the Set an environment variable page to add them all to your project:

ACCOUNT_ID= # auto-generated by set-circleci-env.sh
AWS_ACCESS_KEY_ID= # CircleCI credentials you generated earlier
AWS_REGION= # auto-generated by set-circleci-env.sh
AWS_SECRET_ACCESS_KEY= # CircleCI credentials you generated earlier
CLIENT_APP_PORT=8080
DOCKERHUB_USERNAME= # your docker hub username
DOCKERHUB_PASSWORD= # your docker hub password
GCLOUD_SERVICE_KEY= # copy the contents of deployments/cloud-run/keys/circleci-deployer-$GCP_PROJECT_ID.json
GCP_REGION= # auto-generated by set-circleci-env.sh
GOOGLE_PROJECT_ID=carastore-client-prod
MYSQL_DATABASE=carastore_catalog
MYSQL_HOST=mysql-db
MYSQL_PASSWORD= # same value you used in ecs/scripts/export-env.sh
MYSQL_ROOT_PASSWORD= # same value you used in ecs/scripts/export-env.sh
MYSQL_USER=carastore_admin
MY_APP_PREFIX=carastore
PORT=5000
RUNTIME_SA_EMAIL= # auto-generated by set-circleci-env.sh
SERVICE_NAME=carastore-client
SM_SECRET_NAME=/carastore/server/env
TG_ARN= # auto-generated by set-circleci-env.sh
VITE_API_URL= # auto-generated by set-circleci-env.sh

With the set, commit and push your config files to GitHub:

git add .

git commit -m "<add-a-commit-message-here>"

git push -u origin main

CircleCI will detect the branch automatically and run the defined workflows. Go to the CircleCI web app, select Pipelines in the sidebar, and you’ll see your workflows progress from build → test → deploy:

And with that, you have a working deployment pipeline live on CircleCI.

Testing your app in production

Test the server app by copying the value of VITE_API_URL from your env.circleci file and pasting it into a browser. You may see a warning screen like this telling you the connection is not secure:

This happens because the app uses a self-signed TLS certificate to ensure the secure client can communicate with the server. While this is fine for a practice project, in the real world, you would provision a valid ACM certificate and issue it for a domain you own.

Click the link to continue, and you should be directed to a screen with the message:

“Hello from the server!”

To test the React app:

1. Navigate to the Cloud Run section in the Google Cloud console.

2. Select the service named carastore-client, and you should see the URL displayed.

3. Click on it to see the app in production.

You will see a message saying no products are available.

Click Add Product in the top right corner, enter a test value in each text box, and submit. The app should redirect you to the home page, where you’ll see the product details. You can edit and or delete it for further testing.

This confirms your client app is successfully communicating with the server.

Clean up and best practices

Run the cleanup scripts to avoid incurring unnecessary charges.

For AWS:

# add execution permissions
chmod +x deployments/ecs/scripts/cleanup-ecs.sh

# run the script
source deployments/ecs/scripts/cleanup-ecs.sh

For Google Cloud Run:

# add execution permissions
chmod +x deployments/cloud-run/scripts/cleanup-cloud-run.sh

# run the script
source deployments/cloud-run/scripts/cleanup-cloud-run.sh

Conclusion

Maintaining a multi-cloud deployment strategy in the real world adds complexity, especially in terms of environmental drift. Over time, production and staging environments may diverge due to resource limits or networking behavior. Here are a couple of helpful tips to mitigate this:

• IaC (Infrastructure as Code) implementation (e.g., Terraform, Pulumi) to maintain parity.

• Careful monitoring and observability to catch inconsistencies.

As next steps, you should consider:

• Using CircleCI contexts for cloud credentials. E.g., staging vs production.

• Maintaining cloud-agnostic logic and avoiding hard-coding cloud-specific behavior into your app.

• Exporting logs to a centralized dashboard like Datadog or Grafana.

This CircleCI blog post, CI/CD for multi-cloud: Automate and unify deployments across providers, highlights key strategies for dealing with the increasing complexities of a robust multi-cloud architecture. I highly recommend it for a deeper dive into managing deployments across different providers.

I really enjoyed putting this together, and I hope it was just as fun for you to follow along. You can check out the full CircleCI configuration on the project’s multi_cloud branch.

With CircleCI’s dynamic configuration, you can ensure that only the jobs and workflows related to a modified service will run when you push your commits. Dynamic configuration is a mechanism that allows programmatic execution of your CI/CD pipeline based on pre-defined parameters and conditions, providing the benefit of efficiency, speed, and scalability.

In this tutorial, you will learn how to set up a CircleCI pipeline to automatically detect changes in your repo and trigger targeted workflows accordingly. You will define jobs and configure conditional workflows for a monorepo containing a React frontend and a Dockerized Node.js + MySQL backend.

Prerequisites

Before you begin, make sure you have the following:

• A CircleCI account.

• A Docker account.

• A GitHub account.

• Git installed on your computer.

• An IDE or code editor such as VSCode.

• Basic familiarity with CI/CD concepts and Docker.

Setting up dynamic configuration

With all of the prerequisites in place, it’s time to configure the monorepo. For this tutorial, you’ll be working with this simple JavaScript project.

Start by forking the repository on GitHub and cloning it to your local machine:

git clone <your-fork-url>

# navigate into the project directory
cd cara-store-catalog

From the project’s root directory, create a .circleci directory, along with the following files:

mkdir .circleci

cd .circleci

touch config.yml continue_config.yml

The config.yml file in the .circleci folder at the root of your project is where your pipeline’s main configuration lives. When you connect a repo to CircleCI, it automatically detects this file and begins orchestrating your workflows. When implementing dynamic configuration, this file also handles the initial setup phase.

The continue_config.yml contains the continuation configuration. It outlines the jobs to run and conditionally triggers workflows based on the pipeline parameters defined in config.yml.

Let us begin.

The setup phase

Open the config.yml file in your code editor and enter the code below:

version: 2.1

setup: true

orbs:
  path-filtering: circleci/path-filtering@2.0.1

workflows:
  filter-path:
    jobs:
      - path-filtering/filter:
          name: detect-modified-directories
          # <directory> <pipeline parameter> <value>
          mapping: |
            server/.* run-server-jobs true
            client/.* run-client-jobs true
            .circleci/.* run-server-jobs true
            .circleci/.* run-client-jobs true
          base-revision: main

The code above defines the initial pipeline configuration. The setup: true field tells CircleCI to enable dynamic configuration features for this file. The orbs key imports the path filtering orb to simplify the process of detecting modified paths and files. Behind the scenes, this orb also uses the continuation orb to trigger the next phase of the pipeline using updated parameter values.

The filter job, available by default on the path-filtering orb, maps each directory where you wish to detect changes to a pipeline parameter and sets an initial value for the parameter. What this means is, when the filter job detects file changes in, say, server/, it sets the run-server-jobs parameter to true and passes it on to the continuation config. You can map additional paths as you wish. Finally, base-revision indicates which branch to compare for changes.

That covers all the logic required for the config.yml file.

Continuation configuration

Next up is the continue_config.yml file. As mentioned earlier, this is where you will define the jobs and workflows you want to run.

Start by declaring the CircleCI version along with default values for the pipeline parameters from the setup phase:

version: 2.1

parameters:
  run-server-jobs:
    type: boolean
    default: false
  run-client-jobs:
    type: boolean
    default: false

To avoid needless code repetition across your config file, CircleCI provides the convenience of reusable configuration. Think of this as a feature similar to “functions” from traditional programming languages that you can call at various points in your code. You can learn more about this in the reusable configuration reference.

Go ahead and define a reusable executor and a command in your continuation_config.yml:

executors:
  node-exec:
    docker:
      - image: cimg/node:22.17

commands:
  installdeps:
    description: "Install dependencies"
    parameters:
      directory:
        type: string
    steps:
      - checkout:
          path: ~/project
      - restore_cache:
          keys:
            - v1-<< parameters.directory >>-deps-{{ checksum "package.json" }}-{{ checksum "package-lock.json" }}
            - v1-<< parameters.directory >>-deps-{{ checksum "package.json" }}
            - v1-<< parameters.directory >>-deps-
      - run:
          name: Install dependencies
          command: npm ci
      - save_cache:
          key: v1-<< parameters.directory >>-deps-{{ checksum "package.json" }}-{{ checksum "package-lock.json" }}
          paths:
            - node_modules

The node-exec executor sets a docker execution environment using the CircleCI Node.js convenience image. The installdeps command lays out a typical process for installing Node.js dependencies in CircleCI. The command takes a parameter called directory of type string, which each job will pass at the point of invocation. Instead of hardcoded keys like v1-server-deps to save and restore dependencies, the caching steps use this dynamic string with « parameters.directory », allowing multiple jobs to reuse the command.

Following this is an outline of a series of steps:

• checkout clones the repository. This step specifies a path because, as you will see later, each set of jobs that uses this step will do so from a different working directory. Doing this ensures CircleCI checks out the code to the right directory.

• restore_cache tells CircleCI to reuse a previously saved cache of installed dependencies, if any, to minimize redundancy. The keys attribute uses the directory parameter to read the saved cache. If it fails to find an exact match, it gracefully falls back to less specific keys.

• The npm ci command does a clean installation of the dependencies in your package.json file. It is typically used in CI environments and requires an existing package-lock.json file. See the npm-ci docs for more information about using this command.

• With save_cache, you are instructing CircleCI to store a cache of the dependencies based on the checksums of package.json and package-lock.json. You normally would not want CircleCI to install the same dependencies from scratch every time you push your code, and those files have not changed. That would slow down your execution while also using up precious compute resources. See the Caching dependencies page for more information about how it works.

Now it’s time to define the jobs your pipeline will run:

jobs:
  build-client:
    executor: node-exec
    working_directory: ~/project/client
    steps:
      - installdeps:
          directory: client
      - run:
          name: Build client app
          command: npm run build

  test-client:
    executor: node-exec
    working_directory: ~/project/client
    steps:
      - installdeps:
          directory: client
      - run:
          name: Run client tests
          command: npm test

The build-client and test-client jobs share a similar structure:

• Apply node-exec as the executor environment.

• Set the working directory to the client folder.

• Invoke the installdeps command and pass the client value for save_cache and restore_cache to use.

• Run the corresponding commands to build and test the code.

Add the test-server job to your continuation config:

test-server:
    docker:
      - image: cimg/node:22.17
      - image: cimg/mysql:8.0
        environment:
          MYSQL_ROOT_PASSWORD: djs0_32
    working_directory: ~/project/server
    steps:
      - installdeps:
          directory: server
      - run:
          name: Wait for MySQL
          command: dockerize -wait tcp://localhost:3306 -timeout 1m
      - run:
          name: Install MySQL client
          command: |
            sudo apt-get update
            sudo apt-get install default-mysql-client
      - run:
          name: Set up database
          command: |
            mysql -h 127.0.0.1 -u root -pdjs0_32 -e "CREATE USER '$MYSQL_USER'@'%' IDENTIFIED WITH mysql_native_password BY '$MYSQL_PASSWORD'"
            mysql -h 127.0.0.1 -u root -pdjs0_32 < db/init.sql
      - run:
          name: Run server tests
          command: npm test

The test-server job sets up the following:

• A primary container: cimg/node:22.17.

• A secondary/service container: cimg/mysql:8.0.

• A root password for the MySQL database for initial access.

• And, as mentioned earlier, the working directory to run the steps in.

It then invokes the installdeps command and passes the server value. To avoid race conditions, the job uses dockerize to wait for the container to start before attempting to use it. Once ready, it installs the MySQL client, sets up the database by creating a user and running the server/db/init.sql script, and then runs the server test suite.

Next, add the publish-server job:

publish-server:
    docker:
      - image: cimg/base:current
    working_directory: ~/project/server
    steps:
      - checkout:
          path: ~/project
      - setup_remote_docker:
          docker_layer_caching: true
      - run:
          name: Set image tag
          command: |
            IMAGE_TAG=$(jq -r '.version' package.json)
            echo "export IMAGE_TAG=$IMAGE_TAG" >> $BASH_ENV
            echo "IMAGE_TAG: $IMAGE_TAG"
            source $BASH_ENV
      - run:
          name: Build production image
          command: |
            docker build -t $DOCKERHUB_USERNAME/dynamic-config:$IMAGE_TAG --target prod .
      - run:
          name: Authenticate, tag, and push image to Docker Hub
          command: |
            echo "$DOCKERHUB_PASSWORD" | docker login -u $DOCKERHUB_USERNAME --password-stdin
            docker tag $DOCKERHUB_USERNAME/dynamic-config:$IMAGE_TAG $DOCKERHUB_USERNAME/dynamic-config:latest
            docker push $DOCKERHUB_USERNAME/dynamic-config:$IMAGE_TAG
            docker push $DOCKERHUB_USERNAME/dynamic-config:latest

Here is a rundown of what this block does:

• Sets setup_remote_docker: true to enable Docker layer caching for reuse in future builds.

• Extracts the version number from the server’s package.json file to tag the image.

• Builds a production version of the image. The FROM base AS prod line in the server/Dockerfile outlines the build instructions.

• Authenticates with Docker, sets the latest tag on the built image, and pushes both tags to Docker Hub.

And for the final piece of the continuation config:

workflows:
  test-and-publish-server:
    when: << pipeline.parameters.run-server-jobs >>
    jobs:
      - test-server
      - publish-server:
          requires:
            - test-server
  build-and-test-client:
    when: << not pipeline.parameters.run-client-jobs >>
    jobs:
      - build-client
      - test-client:
          requires:
            - build-client

Workflows orchestrate the jobs in your config. Here, the when clause directs CircleCI to run the specified jobs only when the pipeline parameter is true. And with requires, you are pausing the execution of a job until the previous one is successful.

That concludes the configuration setup.

Setting the environment variables

If you take a look at some of the project files, like index.js and compose.yaml, you will notice the use of a number of environment variables. These are necessary to avoid exposing sensitive credentials on GitHub.

To set your project variables, start by pushing the changes you have made so CircleCI can detect your .circleci/config.yml file. From your project’s root directory, enter the following commands:

git add .

git commit -m '<add-your-commit-message>'

git push -u origin main

Next, follow the instructions on the Set up a project page to connect your repo. After linking your project, CircleCI will kick off a pipeline and begin running all the workflows. The initial run will likely fail, but you can fix that right away.

Navigate to the Set an environment variable page and follow the instructions to add the following to your project:

MYSQL_USER=carastore_admin
MYSQL_PASSWORD=<assign-any-value>
MYSQL_DATABASE=carastore_catalog
DOCKERHUB_USERNAME=<your-dockerhub-username>
DOCKERHUB_PASSWORD=<your-dockerhub-password>

Please note that you may also assign any values you want to the MySQL user and database; however, you would have to update them accordingly in the server/db/init.sql file.

Testing your pipeline

With your configuration and environment variables in place, CircleCI can now automatically detect which folders you have modified and then run your workflows based on those modifications. To see this in full action, edit a file inside the client/ or server/ folder.

Commit and push your changes:

git add .

git commit -m '<add-your-commit-message>'

git push -u origin main

Open the project on your dashboard and watch as CircleCI triggers a new pipeline that runs only the corresponding workflow, demonstrating the efficiency of dynamic configuration.

Conclusion

Running jobs for unchanged services in a monorepo can result in wasted resources, slower pipelines, and increased costs in the long run. By implementing dynamic configuration, you can mitigate these downsides and optimize your pipelines for speed and efficiency.

I hope this tutorial has helped you understand how to improve your CI/CD workflows. If you wish to see the complete setup of the config files, check out the dynamic_config branch of the project repository. For more examples on how to further customise your workflows, see the CircleCI guide on Using dynamic configuration.