IMPORTANT: Developer documentation for the current development branch. This content is unreleased, may change without notice, and must not be treated as Buildish release documentation.
Distributed Multi-Job Builds
Why distributed mode exists
When multiple build jobs run in parallel and each publishes an independent full-cache generation, those generations diverge. Restoring only the newest one loses dependency downloads and other cache changes that exist solely in the other workers’ generations.
Distributed mode solves this with delta exchange:
- Each worker job uploads only the files that changed in the build tool’s cache directory during its build as a workflow artifact (the delta). It does not write the base cache directly.
- An aggregator job, which waits for all workers to complete, downloads every delta, merges them, and saves the merged result as the new base cache entry.
Every parallel job’s changes are captured. On the next run, all workers start from a cache that reflects the combined output of every job from the previous run.
sequenceDiagram
participant BC as Base cache
participant WA as Worker A
participant WB as Worker B
participant AG as Aggregator
BC-->>WA: restore
BC-->>WB: restore
WA->>WA: build + compute delta
WB->>WB: build + compute delta
WA-->>AG: upload Δ-A artifact
WB-->>AG: upload Δ-B artifact
AG->>AG: download Δ-A + Δ-B
AG->>AG: merge deltas
AG->>BC: save merged cache entry
How delta exchange works
- The worker’s prepare step restores the base cache and captures a snapshot of the build tool’s cache directory (the pre-build manifest).
- The build runs normally.
- The worker’s finalize step captures a second snapshot (the post-build manifest), computes the difference, packs only the changed and added files into a compressed artifact, and uploads it. The base cache is not written.
- The aggregator’s prepare step restores the base cache, downloads the selected worker envelopes, validates their identity and preconditions, and applies the merged delta.
- The aggregator’s finalize step saves the resulting immutable base-cache generation.
On pull-request events, read-only mode performs no exchange: workers upload nothing, and an
aggregator returns skipped-read-only without listing, downloading, validating, applying, or
deleting artifacts. This makes a static topology safe, but a workflow should avoid allocating the
aggregator runner when it can express the writable-event condition directly.
Gradle workflow example
1jobs:
2 worker-a:
3 runs-on: ubuntu-latest
4 permissions:
5 contents: read
6 steps:
7 - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd
8 - uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654
9 with:
10 distribution: temurin
11 java-version: '21'
12 - uses: buildish-tooling/buildish-mammoth-cache/actions/github/gradle@<commit-sha>
13 with:
14 job-mode: distributed-worker
15 - run: ./gradlew :module-a:build
16
17 worker-b:
18 runs-on: ubuntu-latest
19 permissions:
20 contents: read
21 steps:
22 - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd
23 - uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654
24 with:
25 distribution: temurin
26 java-version: '21'
27 - uses: buildish-tooling/buildish-mammoth-cache/actions/github/gradle@<commit-sha>
28 with:
29 job-mode: distributed-worker
30 - run: ./gradlew :module-b:build
31
32 aggregator:
33 needs: [worker-a, worker-b]
34 if: ${{ always() && github.event_name != 'pull_request' && github.event_name != 'pull_request_target' }}
35 runs-on: ubuntu-latest
36 permissions:
37 contents: read
38 steps:
39 - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd
40 - uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654
41 with:
42 distribution: temurin
43 java-version: '21'
44 - uses: buildish-tooling/buildish-mammoth-cache/actions/github/gradle@<commit-sha>
45 with:
46 job-mode: distributed-aggregator
47 dependent-jobs: worker-a, worker-b
Maven workflow example
1jobs:
2 worker-a:
3 runs-on: ubuntu-latest
4 permissions:
5 contents: read
6 steps:
7 - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd
8 - uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654
9 with:
10 distribution: temurin
11 java-version: '21'
12 - uses: buildish-tooling/buildish-mammoth-cache/actions/github/maven@<commit-sha>
13 with:
14 job-mode: distributed-worker
15 - run: mvn -pl module-a verify
16
17 worker-b:
18 runs-on: ubuntu-latest
19 permissions:
20 contents: read
21 steps:
22 - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd
23 - uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654
24 with:
25 distribution: temurin
26 java-version: '21'
27 - uses: buildish-tooling/buildish-mammoth-cache/actions/github/maven@<commit-sha>
28 with:
29 job-mode: distributed-worker
30 - run: mvn -pl module-b verify
31
32 aggregator:
33 needs: [worker-a, worker-b]
34 if: ${{ always() && github.event_name != 'pull_request' && github.event_name != 'pull_request_target' }}
35 runs-on: ubuntu-latest
36 permissions:
37 contents: read
38 steps:
39 - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd
40 - uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654
41 with:
42 distribution: temurin
43 java-version: '21'
44 - uses: buildish-tooling/buildish-mammoth-cache/actions/github/maven@<commit-sha>
45 with:
46 job-mode: distributed-aggregator
47 dependent-jobs: worker-a, worker-b
Requirements and constraints
Job names must be unique within the workflow run. Delta artifacts are identified by job name. Two worker jobs with the same name would produce the same artifact name and the aggregator would only see one of them.
The aggregator must declare needs for every worker. This ensures all deltas are uploaded
before the aggregator starts and consumes them during prepare.
A writable aggregator should use if: always() together with its trusted-event condition. The
aggregator must run after a failed worker so it can report the missing envelope instead of being
silently skipped. Pull-request aggregators may be omitted entirely; if they do run, their read-only
result is the explicit no-op described above.
Workers do not need the aggregator in their needs. Workers are independent of each other;
only the aggregator depends on all workers.
Re-runs work correctly. For each worker, the aggregator selects the highest available producer attempt that is not newer than its own attempt. A full rerun uses every new worker envelope; a failed-job rerun safely mixes rerun workers with retained earlier attempts; and an aggregator-only rerun reuses retained worker envelopes. Selection never crosses workflow run IDs.
A successful writable worker always uploads one envelope. This includes workers that changed no managed cache files: their explicit empty envelope is proof of participation and is distinct from a missing or failed worker.
Using a config file
Repeat inputs can be moved to a shared config file. The file format is identical for both build tools — use the config keys matching your tool’s action.
1# .github/buildish-mammoth-gradle.yml (Gradle example)
2cache-key-prefix: my-project-gradle-
3wrapper-properties-files: gradle/wrapper/gradle-wrapper.properties
1# Each worker job
2- uses: buildish-tooling/buildish-mammoth-cache/actions/github/gradle@<commit-sha>
3 with:
4 job-mode: distributed-worker
5 config-file: .github/buildish-mammoth-gradle.yml
6
7# Aggregator job
8- uses: buildish-tooling/buildish-mammoth-cache/actions/github/gradle@<commit-sha>
9 with:
10 job-mode: distributed-aggregator
11 dependent-jobs: worker-a, worker-b
12 config-file: .github/buildish-mammoth-gradle.yml