✨ (ci): Expand strategy.matrix into per-cell tasks
Changes
16 files changed, +516 -43
MODIFY
README.md
+2 -2
@@ -90,8 +90,8 @@
90
90
which a runner claims, executes, and streams logs for — visible on the repository's **Actions** tab;
91
91
a vanished runner's task is reclaimed after a timeout. Jobs are matched to runners by `runs-on`
92
92
labels, ordered by `needs` dependencies, and repository owners manage encrypted secrets and
93
- variables that are delivered to runners, and runs can be cancelled or re-run from the UI.
94
- Non-push events, `matrix`, and artifacts are follow-up phases. Guides: [for users](docs/users/ci-runners.md), [for admins](docs/admins/ci-runners.md),
93
+ variables that are delivered to runners, jobs support `needs` ordering and `strategy.matrix`, and
94
+ runs can be cancelled or re-run from the UI. Non-push events and artifacts are follow-up phases. Guides: [for users](docs/users/ci-runners.md), [for admins](docs/admins/ci-runners.md),
95
95
[architecture](docs/maintainers/ci-runners.md)
96
96
activities from; local users can in turn follow a remote repository — or a whole remote user, whose
97
97
public repositories are then followed and shown grouped — and read their pushes (see below)
MODIFY
docs/admins/ci-runners.md
+2 -2
@@ -89,7 +89,7 @@
89
89
| `ci_runner_registration_token` | Reusable registration tokens: `token_hash`, `created_by_id`, `created_at`, `last_used`. |
90
90
| `ci_runner` | Registered runners: `uuid` (the `x-runner-uuid` value), `token_hash`, `name`, `labels` (comma-joined), `version`, `status` (`IDLE`/`ACTIVE`/`OFFLINE`/`UNSPECIFIED`), `ephemeral`, `last_seen`, `created_at`. |
91
91
| `action_run` | One workflow run per repository: `number` (per-repo sequential), `workflow_name`, `workflow_file`, `event`, `ref`, `commit_sha`, `triggered_by_id`, `status` (`PENDING`/`RUNNING`/`SUCCESS`/`FAILURE`/`CANCELLED`), timestamps. Deleted with its repository. |
92
-| `action_task` | One job within a run: `seq` (surrogate int64 id handed to runners), `run_id`, `name`, `runs_on` (comma-joined labels for runner matching, empty = any), `needs` (comma-joined dependency job names), `outputs` (JSON of the job's reported outputs), `payload`, `runner_id` (the claiming runner, null while pending), `status`, `log_length` (durable log-row count = UpdateLog resume offset), `deadline` (zombie timeout), timestamps. Deleted with its run. |
92
+| `action_task` | One job within a run: `seq` (surrogate int64 id handed to runners), `run_id`, `name`, `runs_on` (comma-joined labels for runner matching, empty = any), `needs` (comma-joined dependency job names), `outputs` (JSON of the job's reported outputs), `job_id` (workflow job key; a matrix job's cells share it), `payload`, `runner_id` (the claiming runner, null while pending), `status`, `log_length` (durable log-row count = UpdateLog resume offset), `deadline` (zombie timeout), timestamps. Deleted with its run. |
93
93
| `action_log` | One log row of a task: `task_id`, `line_index` (0-based), `content`, `timestamp`. Deleted with its task. |
94
94
| `action_secret` | Per-repo CI secret: `repository_id`, `name`, `value_encrypted` (SecretCrypto envelope), decrypted only when delivered to a runner. Deleted with its repository. |
95
95
| `action_variable` | Per-repo CI variable: `repository_id`, `name`, `value` (plaintext config). Deleted with its repository. |
@@ -98,7 +98,7 @@
98
98
(`V24__action_task_seq.sql` adds `action_task.seq`, `V25__action_task_runs_on.sql` adds
99
99
`action_task.runs_on`, `V26__action_secrets_variables.sql` adds `action_secret`/`action_variable`,
100
100
`V27__action_task_needs.sql` adds `action_task.needs`, `V28__action_task_outputs.sql` adds
101
-`action_task.outputs`).
101
+`action_task.outputs`, `V29__action_task_job_id.sql` adds `action_task.job_id`).
102
102
Secrets are stored encrypted and require `GITSHARK_SECRET_KEY` to be set (same key as push mirrors);
103
103
without it, secrets cannot be decrypted and are omitted from what a runner receives.
104
104
The `ci_runner*` tables hold no repository data (losing them only forces re-registration); the
MODIFY
docs/maintainers/ci-runners.md
+13 -7
@@ -19,7 +19,7 @@
19
19
| Actions UI | `web/ActionResource.java` + `templates/ActionResource/` | Read-only per-repo run list + run detail (jobs and their log rows); sidebar `Actions` tab. |
20
20
| Secrets/variables UI | `web/ActionSettingsResource.java` + `ci/ActionSecretService.java` + `templates/ActionSettingsResource/` | Owner-only CRUD for CI secrets (write-only, encrypted) and variables at `settings/actions`. |
21
21
| Entities | `model/CiRunner.java`, `model/CiRunnerRegistrationToken.java` | Runner state (migration `V19`). |
22
-| Run entities | `model/ActionRun.java`, `model/ActionTask.java`, `model/ActionLog.java` | Run/job/log-row persistence (migrations `V23`–`V28`). `ActionTask.seq` (`bigserial`) is the surrogate int64 `Task.id`; `runs_on` = matching labels; `needs` = job dependencies; `outputs` = reported job outputs (JSON). |
22
+| Run entities | `model/ActionRun.java`, `model/ActionTask.java`, `model/ActionLog.java` | Run/job/log-row persistence (migrations `V23`–`V29`). `ActionTask.seq` (`bigserial`) = surrogate int64 `Task.id`; `runs_on` = matching labels; `needs` = job dependencies; `outputs` = reported job outputs (JSON); `job_id` = workflow job key (shared by a matrix job's cells). |
23
23
| Secret/variable entities | `model/ActionSecret.java`, `model/ActionVariable.java` | Per-repo CI secrets (encrypted) and variables (migration `V26`), delivered to runners in FetchTask. |
24
24
| Workflow ingest | `ci/WorkflowIngestService.java`, `ci/WorkflowRunFactory.java` | Post-receive hook: parse `.forgejo`/`.gitea` workflows at the pushed head, evaluate `on: push`, persist a run + its PENDING tasks (drained by FetchTask). |
25
25
| Admin UI | `ci/AdminRunnerResource.java` + `templates/AdminRunnerResource/` | Token generation, runner list, deletion. |
@@ -84,6 +84,12 @@
84
84
that fails to decrypt is dropped, never sent as ciphertext) in the FetchTask `Task.secrets`/`vars`
85
85
maps. Same trust model as GitHub self-hosted runners: secrets go to whatever runner claims the task
86
86
(over TLS). No repo/org scoping of secrets and no fork-PR guard yet (no PR triggers exist).
87
+- **Per-job payloads & `matrix`:** ingest builds each task a standalone single-job `workflow_payload`
88
+ (the original `name`/`on` plus just that job). A job with `strategy.matrix` expands into one task per
89
+ cross-product cell — display name `job (v1, v2)`, shared `action_task.job_id`, and a payload whose
90
+ `strategy.matrix` is reduced to that single cell so the runner resolves `matrix.*`. `github.job` is
91
+ the `job_id`; `needs` and cascade-cancel group a job's cells by `job_id` (a dependent waits for all
92
+ cells, and is cancelled if any cell fails). `include`/`exclude` not yet handled.
87
93
- **`needs` ordering:** each task records the jobs it depends on (`action_task.needs`, parsed at
88
94
ingest). Dispatch will not hand out a task until every needed job in the run has succeeded; when a
89
95
needed job ends FAILURE/CANCELLED, `rollUpRun` cancels the dependents (to a fixpoint, so the
@@ -125,7 +131,10 @@
125
131
`CancelRerunTest` (cancel settles run+unfinished tasks, re-run resets a finished run, a cancelled
126
132
task tells the runner to stop via UpdateTask) and `ActionControlUiTest` (owner cancels/re-runs over
127
133
HTTP, a non-writer is refused), `SupersededRunsTest` (a new push cancels the branch's earlier
128
- running run but leaves other branches alone).
134
+ running run but leaves other branches alone), `MatrixExpansionTest` (single- and two-dimension
135
+ matrices expand to one task per cell with a reduced payload; a non-matrix job stays single),
136
+ `MatrixNeedsTest` (a dependent waits for every cell of a needed matrix job, and one failed cell
137
+ cancels the dependent).
129
138
- **Zombie reclaim (`ZombieReclaimService`):** a scheduled sweep
130
139
(`gitshark.ci.zombie-reclaim-interval`, default 1m) fails any RUNNING task whose
131
140
`action_task.deadline` has passed — the runner is presumed gone — rolls its run up, and flags the
@@ -156,13 +165,10 @@
156
165
- **Long-poll & real `tasks_version`:** `FetchTask` returns immediately and `tasks_version` is a
157
166
coarse max-`seq` (bumps on creation, not state change), so with several simultaneous PENDING tasks a
158
167
runner may under-poll. Add server-side long-poll and a state-driven version counter.
159
-- **Per-job payload expansion:** `workflow_payload` is the raw workflow YAML (fine while a workflow
160
- has a single job, which the `github.job` context selects); a multi-job workflow needs each job
161
- isolated/expanded into its own payload.
162
168
- **Non-push events:** only `push` is evaluated; `pull_request`, scheduled and manual triggers are
163
169
not. (`!`-negation within a single pattern list is also not supported.)
164
-- **`matrix`:** expansion is not implemented — a job with `strategy.matrix` runs once, not once per
165
- cell (needs a per-job/per-cell payload expander).
170
+- **Matrix advanced options:** `include`/`exclude` and `fail-fast`/`max-parallel` are not honored
171
+ (plain dimension cross-product only).
166
172
- **Later phases:** artifacts (`ACTIONS_RESULTS_URL`), repo/org-scoped and ephemeral runners,
167
173
commit/MR status, non-push events.
168
174
MODIFY
docs/users/ci-runners.md
+22 -3
@@ -90,10 +90,29 @@
90
90
```
91
91
92
92
The result **and outputs** of each needed job are available to dependents — set outputs in the
93
-upstream job and read them with `${{ needs.build.outputs.* }}`. `matrix` expansion is not implemented
94
-yet.
93
+upstream job and read them with `${{ needs.build.outputs.* }}`.
94
+
95
+## Matrix builds
96
+
97
+A job with `strategy.matrix` runs once per combination of its values, each a separate entry on the
98
+Actions page:
99
+
100
+```yaml
101
+jobs:
102
+ test:
103
+ runs-on: ubuntu-latest
104
+ strategy:
105
+ matrix:
106
+ os: [linux, windows]
107
+ jdk: [17, 21]
108
+ steps:
109
+ - run: echo "${{ matrix.os }} / ${{ matrix.jdk }}"
110
+```
111
+
112
+This produces four runs (`test (linux, 17)`, `test (linux, 21)`, …). A job that `needs` a matrix job
113
+waits for all of its cells. `matrix.include` / `matrix.exclude` are not supported yet.
95
114
96
115
## What's coming
97
116
98
-- Non-push events (`pull_request`, scheduled, manual), `matrix`.
117
+- Non-push events (`pull_request`, scheduled, manual).
99
118
- Artifacts and commit/merge-request status integration.
MODIFY
src/main/java/de/workaround/ci/ConnectRunnerResource.java
+1 -1
@@ -237,7 +237,7 @@
237
237
putString(context, "run_id", String.valueOf(task.seq));
238
238
putString(context, "run_number", String.valueOf(run.number));
239
239
putString(context, "run_attempt", "1");
240
- putString(context, "job", task.name);
240
+ putString(context, "job", task.jobId);
241
241
putString(context, "ref", run.ref);
242
242
putString(context, "ref_name", refName);
243
243
putString(context, "ref_type", "branch");
MODIFY
src/main/java/de/workaround/ci/TaskDispatchService.java
+36 -13
@@ -2,6 +2,7 @@
2
2
3
3
import java.time.Duration;
4
4
import java.time.Instant;
5
+import java.util.ArrayList;
5
6
import java.util.Arrays;
6
7
import java.util.HashMap;
7
8
import java.util.List;
@@ -194,25 +195,27 @@
194
195
{
195
196
return true;
196
197
}
197
- Map<String, ActionTask> siblings = taskByJob(task.run);
198
- return needed.stream().allMatch(name ->
198
+ Map<String, List<ActionTask>> byJob = cellsByJob(task.run);
199
+ return needed.stream().allMatch(jobId ->
199
200
{
200
- ActionTask dep = siblings.get(name);
201
- return dep != null && dep.status == ActionRun.Status.SUCCESS;
201
+ List<ActionTask> cells = byJob.get(jobId);
202
+ return cells != null && !cells.isEmpty()
203
+ && cells.stream().allMatch(cell -> cell.status == ActionRun.Status.SUCCESS);
202
204
});
203
205
}
204
206
205
- private Map<String, ActionTask> taskByJob(ActionRun run)
207
+ /** Tasks of a run grouped by job id; a matrix job has several cells under one id. */
208
+ private Map<String, List<ActionTask>> cellsByJob(ActionRun run)
206
209
{
207
- Map<String, ActionTask> byJob = new HashMap<>();
210
+ Map<String, List<ActionTask>> byJob = new HashMap<>();
208
211
for (ActionTask sibling : tasks.findByRun(run))
209
212
{
210
- byJob.put(sibling.name, sibling);
213
+ byJob.computeIfAbsent(sibling.jobId, k -> new ArrayList<>()).add(sibling);
211
214
}
212
215
return byJob;
213
216
}
214
217
215
- /** The result and outputs of the jobs a task needs, for the runner's {@code needs} context. */
218
+ /** The aggregate result and merged outputs of the jobs a task needs, for its {@code needs} context. */
216
219
private Map<String, NeedInfo> needsResults(ActionTask task)
217
220
{
218
221
Set<String> needed = splitLabels(task.needs);
@@ -220,19 +223,39 @@
220
223
{
221
224
return Map.of();
222
225
}
223
- Map<String, ActionTask> siblings = taskByJob(task.run);
226
+ Map<String, List<ActionTask>> byJob = cellsByJob(task.run);
224
227
Map<String, NeedInfo> results = new HashMap<>();
225
- for (String name : needed)
228
+ for (String jobId : needed)
226
229
{
227
- ActionTask dep = siblings.get(name);
228
- if (dep != null)
230
+ List<ActionTask> cells = byJob.get(jobId);
231
+ if (cells != null && !cells.isEmpty())
229
232
{
230
- results.put(name, new NeedInfo(dep.status, ActionOutputs.parse(dep.outputs)));
233
+ Map<String, String> merged = new HashMap<>();
234
+ cells.forEach(cell -> merged.putAll(ActionOutputs.parse(cell.outputs)));
235
+ results.put(jobId, new NeedInfo(aggregate(cells), merged));
231
236
}
232
237
}
233
238
return results;
234
239
}
235
240
241
+ /** Worst-of over a job's cells: RUNNING if any unfinished, else FAILURE > CANCELLED > SUCCESS. */
242
+ private static ActionRun.Status aggregate(List<ActionTask> cells)
243
+ {
244
+ if (cells.stream().anyMatch(c -> !c.status.isTerminal()))
245
+ {
246
+ return ActionRun.Status.RUNNING;
247
+ }
248
+ if (cells.stream().anyMatch(c -> c.status == ActionRun.Status.FAILURE))
249
+ {
250
+ return ActionRun.Status.FAILURE;
251
+ }
252
+ if (cells.stream().anyMatch(c -> c.status == ActionRun.Status.CANCELLED))
253
+ {
254
+ return ActionRun.Status.CANCELLED;
255
+ }
256
+ return ActionRun.Status.SUCCESS;
257
+ }
258
+
236
259
/** Whether the runner advertises every label the task's {@code runs-on} requires (empty = any). */
237
260
private static boolean labelsSatisfied(String runsOn, Set<String> runnerLabels)
238
261
{
MODIFY
src/main/java/de/workaround/ci/TaskProgressService.java
+9 -6
@@ -1,6 +1,7 @@
1
1
package de.workaround.ci;
2
2
3
3
import java.time.Instant;
4
+import java.util.ArrayList;
4
5
import java.util.Arrays;
5
6
import java.util.HashMap;
6
7
import java.util.List;
@@ -152,10 +153,12 @@
152
153
*/
153
154
private static void cancelUnsatisfiableDependents(List<ActionTask> all)
154
155
{
155
- Map<String, ActionRun.Status> byJob = new HashMap<>();
156
+ // group tasks by job id (a matrix job has several cells sharing an id); references are live,
157
+ // so a task cancelled below is seen on the next pass without re-indexing
158
+ Map<String, List<ActionTask>> byJob = new HashMap<>();
156
159
for (ActionTask task : all)
157
160
{
158
- byJob.put(task.name, task.status);
161
+ byJob.computeIfAbsent(task.jobId, k -> new ArrayList<>()).add(task);
159
162
}
160
163
boolean changed = true;
161
164
while (changed)
@@ -167,16 +170,16 @@
167
170
{
168
171
continue;
169
172
}
170
- boolean blocked = needsOf(task).stream().anyMatch(name ->
173
+ boolean blocked = needsOf(task).stream().anyMatch(jobId ->
171
174
{
172
- ActionRun.Status dep = byJob.get(name);
173
- return dep == ActionRun.Status.FAILURE || dep == ActionRun.Status.CANCELLED;
175
+ List<ActionTask> cells = byJob.get(jobId);
176
+ return cells != null && cells.stream().anyMatch(cell ->
177
+ cell.status == ActionRun.Status.FAILURE || cell.status == ActionRun.Status.CANCELLED);
174
178
});
175
179
if (blocked)
176
180
{
177
181
task.status = ActionRun.Status.CANCELLED;
178
182
task.finishedAt = Instant.now();
179
- byJob.put(task.name, task.status);
180
183
changed = true;
181
184
}
182
185
}
MODIFY
src/main/java/de/workaround/ci/WorkflowIngestService.java
+90 -4
@@ -4,7 +4,9 @@
4
4
import java.util.ArrayList;
5
5
import java.util.HashSet;
6
6
import java.util.Iterator;
7
+import java.util.LinkedHashMap;
7
8
import java.util.List;
9
+import java.util.Map;
8
10
import java.util.Set;
9
11
import java.util.UUID;
10
12
@@ -21,6 +23,7 @@
21
23
import org.jboss.logging.Logger;
22
24
23
25
import com.fasterxml.jackson.databind.JsonNode;
26
+import com.fasterxml.jackson.databind.node.ObjectNode;
24
27
import com.fasterxml.jackson.dataformat.yaml.YAMLMapper;
25
28
26
29
import de.workaround.git.GitRepositoryService;
@@ -127,7 +130,7 @@
127
130
continue;
128
131
}
129
132
ActionRun run = factory.create(repo, pusherUserId, command.getRefName(), command.getNewId().name(),
130
- workflowName(root, workflow.path()), workflow.path(), jobs, workflow.content());
133
+ workflowName(root, workflow.path()), workflow.path(), jobs);
131
134
createdRuns.add(run.id);
132
135
}
133
136
if (!createdRuns.isEmpty())
@@ -413,14 +416,97 @@
413
416
{
414
417
for (Iterator<String> it = jobs.fieldNames(); it.hasNext();)
415
418
{
416
- String name = it.next();
417
- JsonNode job = jobs.get(name);
418
- specs.add(new WorkflowRunFactory.JobSpec(name, runsOn(job), needs(job)));
419
+ String jobId = it.next();
420
+ JsonNode job = jobs.get(jobId);
421
+ for (Map<String, String> cell : matrixCells(job))
422
+ {
423
+ String name = cell.isEmpty() ? jobId : jobId + " (" + String.join(", ", cell.values()) + ")";
424
+ String payload = singleJobPayload(root, jobId, job, cell);
425
+ specs.add(new WorkflowRunFactory.JobSpec(name, jobId, runsOn(job), needs(job), payload));
426
+ }
419
427
}
420
428
}
421
429
return specs;
422
430
}
423
431
432
+ /**
433
+ * The matrix combinations of a job: one empty map when the job has no {@code strategy.matrix},
434
+ * otherwise the cartesian product of its dimensions (insertion order preserved, {@code include}/
435
+ * {@code exclude} not yet supported). Each combination maps dimension name to its value.
436
+ */
437
+ private static List<Map<String, String>> matrixCells(JsonNode job)
438
+ {
439
+ JsonNode matrix = job.path("strategy").path("matrix");
440
+ if (!matrix.isObject())
441
+ {
442
+ return List.of(new LinkedHashMap<>());
443
+ }
444
+ List<Map<String, String>> cells = new ArrayList<>();
445
+ cells.add(new LinkedHashMap<>());
446
+ for (Iterator<String> it = matrix.fieldNames(); it.hasNext();)
447
+ {
448
+ String dim = it.next();
449
+ if (dim.equals("include") || dim.equals("exclude"))
450
+ {
451
+ continue;
452
+ }
453
+ JsonNode values = matrix.get(dim);
454
+ if (!values.isArray() || values.isEmpty())
455
+ {
456
+ continue;
457
+ }
458
+ List<Map<String, String>> expanded = new ArrayList<>();
459
+ for (Map<String, String> base : cells)
460
+ {
461
+ for (JsonNode value : values)
462
+ {
463
+ Map<String, String> next = new LinkedHashMap<>(base);
464
+ next.put(dim, value.asText());
465
+ expanded.add(next);
466
+ }
467
+ }
468
+ cells = expanded;
469
+ }
470
+ return cells;
471
+ }
472
+
473
+ /**
474
+ * A standalone single-job workflow for one task: the original {@code name}/{@code on} plus just
475
+ * this job, with its matrix (if any) reduced to the given cell so the runner resolves {@code
476
+ * matrix.*} to a single combination.
477
+ */
478
+ private static String singleJobPayload(JsonNode root, String jobId, JsonNode job, Map<String, String> cell)
479
+ {
480
+ ObjectNode out = YAML.createObjectNode();
481
+ if (root.has("name"))
482
+ {
483
+ out.set("name", root.get("name"));
484
+ }
485
+ JsonNode on = root.has("on") ? root.get("on") : root.get("true");
486
+ if (on != null)
487
+ {
488
+ out.set("on", on);
489
+ }
490
+ ObjectNode jobCopy = job.deepCopy();
491
+ if (!cell.isEmpty())
492
+ {
493
+ ObjectNode strategy = jobCopy.has("strategy") && jobCopy.get("strategy").isObject()
494
+ ? (ObjectNode) jobCopy.get("strategy")
495
+ : jobCopy.putObject("strategy");
496
+ ObjectNode matrix = strategy.putObject("matrix");
497
+ cell.forEach((dim, value) -> matrix.putArray(dim).add(value));
498
+ }
499
+ out.putObject("jobs").set(jobId, jobCopy);
500
+ try
501
+ {
502
+ return YAML.writeValueAsString(out);
503
+ }
504
+ catch (Exception e)
505
+ {
506
+ throw new IllegalStateException("Could not build single-job payload for " + jobId, e);
507
+ }
508
+ }
509
+
424
510
/** The job's {@code runs-on} as comma-joined labels; string or list, empty when absent. */
425
511
private static String runsOn(JsonNode job)
426
512
{
MODIFY
src/main/java/de/workaround/ci/WorkflowRunFactory.java
+9 -4
@@ -35,7 +35,7 @@
35
35
36
36
@Transactional
37
37
public ActionRun create(Repository repository, UUID pusherUserId, String ref, String commitSha,
38
- String workflowName, String workflowFile, List<JobSpec> jobs, String payload)
38
+ String workflowName, String workflowFile, List<JobSpec> jobs)
39
39
{
40
40
Repository repo = repositories.findById(repository.id);
41
41
@@ -55,16 +55,21 @@
55
55
ActionTask task = new ActionTask();
56
56
task.run = run;
57
57
task.name = job.name();
58
+ task.jobId = job.jobId();
58
59
task.runsOn = job.runsOn();
59
60
task.needs = job.needs();
60
- task.payload = payload;
61
+ task.payload = job.payload();
61
62
task.persist();
62
63
}
63
64
return run;
64
65
}
65
66
66
- /** A job discovered in a workflow: its id, comma-joined {@code runs-on} labels and {@code needs}. */
67
- public record JobSpec(String name, String runsOn, String needs)
67
+ /**
68
+ * A materialized task: its display {@code name} (a matrix cell like {@code build (linux)} or just
69
+ * the job id), the base {@code jobId}, {@code runs-on} labels, {@code needs}, and the standalone
70
+ * single-job {@code payload} handed to the runner.
71
+ */
72
+ public record JobSpec(String name, String jobId, String runsOn, String needs, String payload)
68
73
{
69
74
}
70
75
MODIFY
src/main/java/de/workaround/model/ActionTask.java
+4 -1
@@ -48,9 +48,12 @@
48
48
@Generated(event = EventType.INSERT)
49
49
public long seq;
50
50
51
- /** Job identifier from the workflow file, e.g. {@code build}. */
51
+ /** Display name of this task, e.g. {@code build} or a matrix cell {@code build (linux)}. */
52
52
public String name;
53
53
54
+ /** The workflow job key (e.g. {@code build}); shared by a matrix job's cells. Used for github.job and needs. */
55
+ public String jobId = "";
56
+
54
57
/** The job's {@code runs-on} labels, comma-joined; empty means no constraint (any runner). */
55
58
public String runsOn = "";
56
59
ADD
src/main/resources/db/migration/V29__action_task_job_id.sql
+13 -0
@@ -0,0 +1,13 @@
1
+-- Base job id for matrix expansion (issue #2, phase 2).
2
+--
3
+-- A matrix job expands into several tasks that share one job id but have distinct display names
4
+-- ("build (linux)", "build (windows)"). job_id is the workflow's job key — used for `github.job`,
5
+-- for `needs` resolution and for grouping a job's matrix cells. Backfilled to the display name for
6
+-- pre-matrix rows (one cell per job).
7
+
8
+alter table action_task
9
+ add column job_id text not null default '';
10
+
11
+update action_task
12
+set job_id = name
13
+where job_id = '';
MODIFY
src/test/java/de/workaround/ci/ForgejoRunnerRoundTripTest.java
+1 -0
@@ -161,6 +161,7 @@
161
161
ActionTask task = new ActionTask();
162
162
task.run = run;
163
163
task.name = "build";
164
+ task.jobId = "build";
164
165
task.payload = WORKFLOW;
165
166
task.persist();
166
167
return run.id;
ADD
src/test/java/de/workaround/ci/MatrixExpansionTest.java
+141 -0
@@ -0,0 +1,141 @@
1
+package de.workaround.ci;
2
+
3
+import java.nio.charset.StandardCharsets;
4
+import java.nio.file.Path;
5
+import java.util.List;
6
+import java.util.Map;
7
+import java.util.Set;
8
+import java.util.UUID;
9
+import java.util.stream.Collectors;
10
+
11
+import org.eclipse.jgit.lib.ObjectId;
12
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
13
+import org.eclipse.jgit.transport.ReceiveCommand;
14
+import org.junit.jupiter.api.Test;
15
+
16
+import de.workaround.git.GitRepositoryService;
17
+import de.workaround.git.GitTestSeeder;
18
+import de.workaround.model.ActionRun;
19
+import de.workaround.model.ActionTask;
20
+import de.workaround.model.Repository;
21
+import de.workaround.model.User;
22
+import io.quarkus.test.junit.QuarkusTest;
23
+import jakarta.inject.Inject;
24
+import jakarta.transaction.Transactional;
25
+
26
+import static org.junit.jupiter.api.Assertions.assertEquals;
27
+import static org.junit.jupiter.api.Assertions.assertTrue;
28
+
29
+/**
30
+ * Matrix expansion (issue #2, phase 2): a job with {@code strategy.matrix} becomes one task per
31
+ * combination, each with a single-cell payload and the shared job id.
32
+ */
33
+@QuarkusTest
34
+class MatrixExpansionTest
35
+{
36
+ @Inject
37
+ WorkflowIngestService ingest;
38
+
39
+ @Inject
40
+ GitRepositoryService repositories;
41
+
42
+ @Inject
43
+ ActionRun.Repo runs;
44
+
45
+ @Inject
46
+ ActionTask.Repo tasks;
47
+
48
+ @Test
49
+ void singleDimensionMatrixExpandsToOneTaskPerValue()
50
+ {
51
+ String yaml = """
52
+ on: push
53
+ jobs:
54
+ build:
55
+ runs-on: ubuntu-latest
56
+ strategy:
57
+ matrix:
58
+ os: [linux, windows]
59
+ steps:
60
+ - run: echo hi
61
+ """;
62
+ List<ActionTask> jobs = ingestAndList("mx-a", yaml);
63
+
64
+ assertEquals(2, jobs.size());
65
+ Set<String> names = jobs.stream().map(t -> t.name).collect(Collectors.toSet());
66
+ assertEquals(Set.of("build (linux)", "build (windows)"), names);
67
+ assertTrue(jobs.stream().allMatch(t -> t.jobId.equals("build")), "cells share the job id");
68
+
69
+ ActionTask linux = jobs.stream().filter(t -> t.name.equals("build (linux)")).findFirst().orElseThrow();
70
+ assertTrue(linux.payload.contains("linux"), "cell payload carries its matrix value");
71
+ assertTrue(!linux.payload.contains("windows"), "cell payload is reduced to its own value");
72
+ }
73
+
74
+ @Test
75
+ void twoDimensionMatrixExpandsToCartesianProduct()
76
+ {
77
+ String yaml = """
78
+ on: push
79
+ jobs:
80
+ test:
81
+ runs-on: ubuntu-latest
82
+ strategy:
83
+ matrix:
84
+ os: [linux, windows]
85
+ jdk: [17, 21]
86
+ steps:
87
+ - run: echo hi
88
+ """;
89
+ List<ActionTask> jobs = ingestAndList("mx-b", yaml);
90
+
91
+ assertEquals(4, jobs.size(), "2 x 2 = 4 cells");
92
+ assertTrue(jobs.stream().allMatch(t -> t.jobId.equals("test")));
93
+ assertTrue(jobs.stream().map(t -> t.name).allMatch(n -> n.startsWith("test (")));
94
+ }
95
+
96
+ @Test
97
+ void nonMatrixJobStaysSingle()
98
+ {
99
+ String yaml = "on: push\njobs:\n build:\n runs-on: ubuntu-latest\n steps:\n - run: echo hi\n";
100
+ List<ActionTask> jobs = ingestAndList("mx-c", yaml);
101
+
102
+ assertEquals(1, jobs.size());
103
+ assertEquals("build", jobs.get(0).name);
104
+ assertEquals("build", jobs.get(0).jobId);
105
+ }
106
+
107
+ private List<ActionTask> ingestAndList(String repoName, String yaml)
108
+ {
109
+ Repository repo = seedAndIngest(repoName, yaml);
110
+ return tasks.findByRun(runs.findByRepository(repo).get(0));
111
+ }
112
+
113
+ @Transactional
114
+ Repository seedAndIngest(String repoName, String yaml)
115
+ {
116
+ String username = repoName + "-" + UUID.randomUUID().toString().substring(0, 8);
117
+ User owner = new User();
118
+ owner.oidcSub = username;
119
+ owner.username = username;
120
+ owner.persist();
121
+ Repository repo = repositories.create(owner, repoName, Repository.Visibility.PUBLIC, null);
122
+
123
+ try
124
+ {
125
+ Path bare = repositories.repositoryPath(repo);
126
+ GitTestSeeder.seed(bare, Map.of(".forgejo/workflows/ci.yml", yaml.getBytes(StandardCharsets.UTF_8)));
127
+ try (org.eclipse.jgit.lib.Repository db = new FileRepositoryBuilder().setGitDir(bare.toFile()).build())
128
+ {
129
+ ObjectId head = db.resolve("refs/heads/main");
130
+ ReceiveCommand command = new ReceiveCommand(ObjectId.zeroId(), head, "refs/heads/main");
131
+ command.setResult(ReceiveCommand.Result.OK);
132
+ ingest.onPush(repo.ownerHandle(), repo.name, owner.id, db, List.of(command));
133
+ }
134
+ }
135
+ catch (Exception e)
136
+ {
137
+ throw new RuntimeException(e);
138
+ }
139
+ return repo;
140
+ }
141
+}
ADD
src/test/java/de/workaround/ci/MatrixNeedsTest.java
+171 -0
@@ -0,0 +1,171 @@
1
+package de.workaround.ci;
2
+
3
+import java.time.Instant;
4
+import java.time.temporal.ChronoUnit;
5
+import java.util.List;
6
+import java.util.UUID;
7
+
8
+import org.junit.jupiter.api.Test;
9
+
10
+import de.workaround.ci.proto.runner.v1.Result;
11
+import de.workaround.git.GitRepositoryService;
12
+import de.workaround.model.ActionRun;
13
+import de.workaround.model.ActionTask;
14
+import de.workaround.model.Repository;
15
+import de.workaround.model.User;
16
+import io.quarkus.test.junit.QuarkusTest;
17
+import jakarta.inject.Inject;
18
+import jakarta.transaction.Transactional;
19
+
20
+import static org.junit.jupiter.api.Assertions.assertEquals;
21
+import static org.junit.jupiter.api.Assertions.assertTrue;
22
+
23
+/**
24
+ * {@code needs} on a matrix job (issue #2, phase 2): a dependent waits for every cell of the needed
25
+ * job, and is cancelled if any cell fails.
26
+ */
27
+@QuarkusTest
28
+class MatrixNeedsTest
29
+{
30
+ @Inject
31
+ RunnerRegistrationService runnerService;
32
+
33
+ @Inject
34
+ TaskDispatchService dispatch;
35
+
36
+ @Inject
37
+ TaskProgressService progress;
38
+
39
+ @Inject
40
+ GitRepositoryService repositories;
41
+
42
+ @Inject
43
+ ActionRun.Repo runs;
44
+
45
+ @Inject
46
+ ActionTask.Repo tasks;
47
+
48
+ @Test
49
+ void dependentWaitsForEveryMatrixCell()
50
+ {
51
+ RunnerRegistrationService.RegisteredRunner reg = registerRunner();
52
+ UUID runId = seed("mn-a");
53
+
54
+ long cellA = claim(reg); // build (a)
55
+ assertTrue(dispatch.fetch(reg.runner().uuid, reg.plaintext()).task().isPresent(), "second build cell");
56
+ long cellB = otherPendingBuildCell(runId, cellA);
57
+
58
+ complete(reg, cellA, Result.RESULT_SUCCESS);
59
+ assertTrue(dispatch.fetch(reg.runner().uuid, reg.plaintext()).task().isEmpty(),
60
+ "deploy still blocked while one build cell is unfinished");
61
+
62
+ complete(reg, cellB, Result.RESULT_SUCCESS);
63
+ TaskDispatchService.Fetched deploy = dispatch.fetch(reg.runner().uuid, reg.plaintext());
64
+ assertEquals("deploy", deploy.task().orElseThrow().name);
65
+ assertEquals(ActionRun.Status.SUCCESS, deploy.needs().get("build").result(), "aggregate of all cells");
66
+ }
67
+
68
+ @Test
69
+ void oneFailedMatrixCellCancelsDependent()
70
+ {
71
+ RunnerRegistrationService.RegisteredRunner reg = registerRunner();
72
+ UUID runId = seed("mn-b");
73
+
74
+ long cellA = claim(reg);
75
+ dispatch.fetch(reg.runner().uuid, reg.plaintext()); // claim the other cell too
76
+ long cellB = otherPendingBuildCell(runId, cellA);
77
+
78
+ complete(reg, cellA, Result.RESULT_SUCCESS);
79
+ complete(reg, cellB, Result.RESULT_FAILURE);
80
+
81
+ assertEquals(ActionRun.Status.CANCELLED, deployStatus(runId), "a failed cell cancels the dependent");
82
+ assertEquals(ActionRun.Status.FAILURE, runStatus(runId));
83
+ }
84
+
85
+ private long claim(RunnerRegistrationService.RegisteredRunner reg)
86
+ {
87
+ return dispatch.fetch(reg.runner().uuid, reg.plaintext()).task().orElseThrow().seq;
88
+ }
89
+
90
+ private void complete(RunnerRegistrationService.RegisteredRunner reg, long seq, Result result)
91
+ {
92
+ progress.updateTask(reg.runner().uuid, reg.plaintext(), seq, result, null, java.util.Map.of());
93
+ }
94
+
95
+ @Transactional
96
+ long otherPendingBuildCell(UUID runId, long claimedSeq)
97
+ {
98
+ ActionRun run = runs.findById(runId);
99
+ return tasks.findByRun(run).stream()
100
+ .filter(t -> t.jobId.equals("build") && t.seq != claimedSeq)
101
+ .findFirst().orElseThrow().seq;
102
+ }
103
+
104
+ private RunnerRegistrationService.RegisteredRunner registerRunner()
105
+ {
106
+ String token = runnerService.createRegistrationToken(persistUser("mn-admin-" + shortId())).plaintext();
107
+ return runnerService.register(token, "mn-runner", List.of(), "v4.0.0", false);
108
+ }
109
+
110
+ @Transactional
111
+ UUID seed(String repoName)
112
+ {
113
+ User owner = persistUser(repoName + "-" + shortId());
114
+ Repository repo = repositories.create(owner, repoName, Repository.Visibility.PUBLIC, null);
115
+
116
+ ActionRun run = new ActionRun();
117
+ run.repository = repo;
118
+ run.number = runs.maxNumber(repo) + 1;
119
+ run.workflowName = "CI";
120
+ run.workflowFile = ".forgejo/workflows/ci.yml";
121
+ run.event = "push";
122
+ run.ref = "refs/heads/main";
123
+ run.commitSha = "0000000000000000000000000000000000000000";
124
+ run.persist();
125
+
126
+ cell(run, "build", "build (a)", "", 20);
127
+ cell(run, "build", "build (b)", "", 15);
128
+ cell(run, "deploy", "deploy", "build", 10);
129
+ return run.id;
130
+ }
131
+
132
+ private void cell(ActionRun run, String jobId, String name, String needs, int secondsAgo)
133
+ {
134
+ ActionTask task = new ActionTask();
135
+ task.run = run;
136
+ task.jobId = jobId;
137
+ task.name = name;
138
+ task.needs = needs;
139
+ task.payload = "on: push";
140
+ task.createdAt = Instant.now().minus(secondsAgo, ChronoUnit.SECONDS);
141
+ task.persist();
142
+ }
143
+
144
+ @Transactional
145
+ ActionRun.Status runStatus(UUID runId)
146
+ {
147
+ return runs.findById(runId).status;
148
+ }
149
+
150
+ @Transactional
151
+ ActionRun.Status deployStatus(UUID runId)
152
+ {
153
+ return tasks.findByRun(runs.findById(runId)).stream()
154
+ .filter(t -> t.jobId.equals("deploy")).findFirst().orElseThrow().status;
155
+ }
156
+
157
+ @Transactional
158
+ User persistUser(String name)
159
+ {
160
+ User user = new User();
161
+ user.oidcSub = name;
162
+ user.username = name;
163
+ user.persist();
164
+ return user;
165
+ }
166
+
167
+ private static String shortId()
168
+ {
169
+ return UUID.randomUUID().toString().substring(0, 8);
170
+ }
171
+}
MODIFY
src/test/java/de/workaround/ci/NeedsOrderingTest.java
+1 -0
@@ -116,6 +116,7 @@
116
116
ActionTask task = new ActionTask();
117
117
task.run = run;
118
118
task.name = name;
119
+ task.jobId = name;
119
120
task.needs = needs;
120
121
task.payload = "on: push";
121
122
task.createdAt = Instant.now().minusSeconds(secondsAgo);
MODIFY
src/test/java/de/workaround/ci/NeedsOutputsTest.java
+1 -0
@@ -114,6 +114,7 @@
114
114
ActionTask task = new ActionTask();
115
115
task.run = run;
116
116
task.name = name;
117
+ task.jobId = name;
117
118
task.needs = needs;
118
119
task.payload = "on: push";
119
120
task.createdAt = Instant.now().minusSeconds(secondsAgo);