✨ (federation): Auto re-sync followed users' new repositories
Changes
10 files changed, +134 -11
MODIFY
docs/admins/federation.md
+1 -0
@@ -21,6 +21,7 @@
21
21
| `GITSHARK_FEDERATION_BASE_URL` | — | Public HTTPS origin of this instance, e.g. `https://shark.example.com`. Actor IDs derive from it |
22
22
| `GITSHARK_FEDERATION_PEER_ALLOWLIST` | — (empty) | Comma-separated peer **hosts**, e.g. `shark.other.org,forge.example`. Empty denies every remote peer |
23
23
| `GITSHARK_FEDERATION_MAX_ATTEMPTS` | `8` | Delivery attempts before an outbound activity is dead-lettered |
24
+| `GITSHARK_FEDERATION_USER_RESYNC_INTERVAL` | `5m` | How often followed remote users are re-scanned for newly created public repositories (add-only) |
24
25
| `GITSHARK_FEDERATION_DEV_ALLOW_INSECURE` | `false` | **Dev only.** Permits `http://` and loopback/private targets. Never in production |
25
26
26
27
Minimal production setup:
MODIFY
docs/admins/getting-started.md
+1 -0
@@ -341,6 +341,7 @@
341
341
| `GITSHARK_FEDERATION_BASE_URL` | — | — | Public HTTPS origin; permanent actor-ID base |
342
342
| `GITSHARK_FEDERATION_PEER_ALLOWLIST` | — | — | Comma-separated peer hosts (empty denies all) |
343
343
| `GITSHARK_FEDERATION_MAX_ATTEMPTS` | — | `8` | Outbound delivery retry cap |
344
+| `GITSHARK_FEDERATION_USER_RESYNC_INTERVAL` | — | `5m` | Re-scan followed users for new public repos |
344
345
| `GITSHARK_FEDERATION_DEV_ALLOW_INSECURE` | — | `false` | Dev only: allow http/loopback peers |
345
346
| `GITSHARK_ADMIN_HANDLES` | — | — | Comma-separated handles allowed into `/admin/*` (CI runner management); empty means no admins (see [CI runners](ci-runners.md)) |
346
347
MODIFY
docs/maintainers/federation-roadmap.md
+7 -3
@@ -55,9 +55,13 @@
55
55
- ✅ Followed users persisted (`remote_user_follows`); the `/following` UI groups
56
56
repositories and their push activity under each followed user.
57
57
58
-Implemented as a **snapshot** at follow time (see the follow-a-user re-sync gap
59
-in [forgefed.md](forgefed.md)); repos the remote adds later need a re-follow to
60
-appear. A live re-sync is a possible follow-up.
58
+- ✅ A periodic `FederationResyncScheduler` re-scans each followed user's
59
+ repositories collection (add-only) so repos created after the follow are
60
+ picked up automatically — no re-follow needed.
61
+
62
+Remaining follow-ups are add-only reconcile limits (no unfollow on remote
63
+delete/private) and collection pagination — see the gap list in
64
+[forgefed.md](forgefed.md).
61
65
62
66
### Story 2 — Cross-instance fork with upstream tracking *(issue #12)*
63
67
MODIFY
docs/maintainers/forgefed.md
+10 -4
@@ -37,6 +37,7 @@
37
37
| `PushHandler` | Inbound: stores `Push` from repositories local users follow |
38
38
| `RemoteFollowService` | Outbound: follow/unfollow a remote repository or user, push feed query |
39
39
| `RemoteRepositoryDirectory` | Outbound: reads a remote `Person`'s `repositories` collection (fan-out source for follow-a-user) |
40
+| `FederationResyncScheduler` | Periodic add-only re-scan of followed users' repositories (picks up repos created after the follow) |
40
41
| `FederationPushService` | Outbound: fans out `Push` to followers from the git post-receive hook |
41
42
| `DeliveryService` | Persisted outbound queue with retry/backoff/dead-letter |
42
43
@@ -81,6 +82,10 @@
81
82
at follow time. `unfollowUser` undoes every tagged repository follow, then
82
83
removes the `RemoteUserFollow`. The `/following` page groups repositories under
83
84
their followed user; directly-followed repositories are listed separately.
85
+`FederationResyncScheduler` re-runs `RemoteRepositoryDirectory` for every
86
+followed user on a configurable interval
87
+(`gitshark.federation.user-resync-interval`, default 5m) and follows any new
88
+public repository — add-only, so repos that disappear remotely are left in place.
84
89
85
90
**Inbound follow** (`FollowHandler`): a remote actor follows one of our public
86
91
repositories → persist `RepositoryFollower` → enqueue a signed `Accept` back.
@@ -188,7 +193,8 @@
188
193
- Outbound follow/unfollow of a remote **user**: reads the `Person`'s
189
194
`repositories` collection and fans out to a repository follow per public repo,
190
195
shown grouped in the `/following` UI (federated-collaboration roadmap Story 1).
191
- Snapshot at follow time — new remote repositories are not auto-picked-up.
196
+ A periodic `FederationResyncScheduler` re-scans followed users (add-only), so
197
+ repositories created after the follow are picked up automatically.
192
198
- HTTP Signature signing/verification, per-actor keys, inbound dedup, peer
193
199
allowlist, SSRF guard, delivery queue with retry and dead-letter.
194
200
- Tested git-shark↔git-shark, including a scripted local two-host trial (see
@@ -235,9 +241,9 @@
235
241
federation.
236
242
- **Follower/feed UI depth** — repository pages don't show remote followers;
237
243
the push feed is a flat newest-50 with no pagination or per-repo filtering.
238
-- **Follow-a-user re-sync** — the fanned-out repository set is a snapshot at
239
- follow time; there is no re-fetch of a followed user's `repositories`
240
- collection, so repos they add later are never picked up. Also,
244
+- **Follow-a-user reconcile is add-only** — `FederationResyncScheduler` picks up
245
+ repositories added after the follow, but does not *unfollow* repositories that
246
+ the remote user made private or deleted; those stale follows linger. Also,
241
247
`RemoteRepositoryDirectory` reads only the first collection page — it does not
242
248
follow `next` pagination, fine for the git-shark↔git-shark scope but a gap for
243
249
users with large repository lists or broader ForgeFed peers.
MODIFY
docs/users/federation.md
+4 -3
@@ -35,9 +35,10 @@
35
35
grouped under the user, and their pushes flow into the **Recent pushes** feed
36
36
like any other follow.
37
37
38
-The repository set is a **snapshot taken when you follow** — repositories the
39
-remote user creates afterwards are not picked up automatically. Unfollowing the
40
-user unfollows every repository that was fanned out from it.
38
+Repositories the remote user creates *after* you follow them are picked up
39
+automatically: your instance re-scans each followed user periodically (every few
40
+minutes) and starts following any new public repository. Unfollowing the user
41
+unfollows every repository that was fanned out from it.
41
42
42
43
---
43
44
ADD
src/main/java/de/workaround/federation/FederationResyncScheduler.java
+38 -0
@@ -0,0 +1,38 @@
1
+package de.workaround.federation;
2
+
3
+import java.util.UUID;
4
+
5
+import io.quarkus.scheduler.Scheduled;
6
+import jakarta.enterprise.context.ApplicationScoped;
7
+import jakarta.inject.Inject;
8
+
9
+/**
10
+ * Periodically re-syncs every followed remote user's repository set, so repositories a user creates
11
+ * after being followed are eventually picked up (add-only — see {@link RemoteFollowService#resyncUser}).
12
+ * Interval is {@code gitshark.federation.user-resync-interval}; the pass is a no-op while federation
13
+ * is not operational.
14
+ */
15
+@ApplicationScoped
16
+public class FederationResyncScheduler
17
+{
18
+ @Inject
19
+ FederationConfig config;
20
+
21
+ @Inject
22
+ RemoteFollowService service;
23
+
24
+ @Scheduled(every = "{gitshark.federation.user-resync-interval}",
25
+ concurrentExecution = Scheduled.ConcurrentExecution.SKIP)
26
+ void resync()
27
+ {
28
+ if (!config.operational())
29
+ {
30
+ return;
31
+ }
32
+ for (UUID userFollowId : service.allUserFollowIds())
33
+ {
34
+ service.resyncUser(userFollowId);
35
+ }
36
+ }
37
+
38
+}
MODIFY
src/main/java/de/workaround/federation/RemoteFollowService.java
+38 -1
@@ -127,7 +127,8 @@
127
127
/**
128
128
* Follows a remote {@code Person} named by {@code input} (actor URL or {@code username@host}
129
129
* handle): fans out to a repository follow per public repository the remote advertises. Idempotent
130
- * per (user, remote person). The repository set is a snapshot at follow time.
130
+ * per (user, remote person). The repository set is kept in sync afterwards by the periodic
131
+ * add-only re-scan in {@link #resyncUser} (via {@code FederationResyncScheduler}).
131
132
*/
132
133
@Transactional
133
134
public RemoteUserFollow followUser(User user, String input)
@@ -182,6 +183,42 @@
182
183
userFollows.deleteById(userFollow.id);
183
184
}
184
185
186
+ /** Ids of every remote-user follow across all users, for the scheduled re-sync pass. */
187
+ @Transactional
188
+ public List<UUID> allUserFollowIds()
189
+ {
190
+ return userFollows.findAllIds();
191
+ }
192
+
193
+ /**
194
+ * Re-fetches a followed user's repositories collection and follows any public repository not yet
195
+ * followed (add-only — repositories that vanished remotely are left in place). Makes
196
+ * follow-then-create eventually consistent without a manual re-follow.
197
+ */
198
+ @Transactional
199
+ public void resyncUser(UUID userFollowId)
200
+ {
201
+ RemoteUserFollow userFollow = userFollows.findById(userFollowId);
202
+ if (userFollow == null)
203
+ {
204
+ return;
205
+ }
206
+ User user = userFollow.user;
207
+ for (String repoActorId : directory.repositoriesOf(userFollow.remoteUserActorId))
208
+ {
209
+ try
210
+ {
211
+ followRepositoryActor(user, repoActorId, userFollow.remoteUserActorId);
212
+ }
213
+ catch (RemoteFollowException e)
214
+ {
215
+ // followRepositoryActor is idempotent; only genuinely unresolvable repos land here.
216
+ LOG.debugf("Re-sync skipping unresolvable repository %s of %s: %s", repoActorId,
217
+ userFollow.remoteUserActorId, e.getMessage());
218
+ }
219
+ }
220
+ }
221
+
185
222
/** The handle to display: the entered {@code username@host} form, else the resolved actor id. */
186
223
private String displayHandle(String input, String actorId)
187
224
{
MODIFY
src/main/java/de/workaround/model/RemoteUserFollow.java
+3 -0
@@ -52,6 +52,9 @@
52
52
53
53
@HQL("select f from RemoteUserFollow f where f.user = :user order by f.createdAt")
54
54
List<RemoteUserFollow> findByUser(User user);
55
+
56
+ @HQL("select f.id from RemoteUserFollow f")
57
+ List<UUID> findAllIds();
55
58
}
56
59
57
60
}
MODIFY
src/main/resources/application.properties
+4 -0
@@ -131,8 +131,12 @@
131
131
gitshark.federation.base-url=${GITSHARK_FEDERATION_BASE_URL:}
132
132
gitshark.federation.peer-allowlist=${GITSHARK_FEDERATION_PEER_ALLOWLIST:}
133
133
gitshark.federation.delivery.max-attempts=${GITSHARK_FEDERATION_MAX_ATTEMPTS:8}
134
+# How often followed remote users are re-scanned for newly created repositories (add-only).
135
+gitshark.federation.user-resync-interval=${GITSHARK_FEDERATION_USER_RESYNC_INTERVAL:5m}
134
136
# Dev/local ONLY: allow http + loopback/private federation targets (allowlist still enforced).
135
137
gitshark.federation.dev-allow-insecure=${GITSHARK_FEDERATION_DEV_ALLOW_INSECURE:false}
136
138
%test.gitshark.federation.enabled=true
137
139
%test.gitshark.federation.base-url=https://shark.test
138
140
%test.gitshark.federation.peer-allowlist=peer.test,shark.test
141
+# Large interval in tests so the scheduler never fires mid-test; re-sync is driven directly.
142
+%test.gitshark.federation.user-resync-interval=1h
MODIFY
src/test/java/de/workaround/federation/RemoteUserFollowTest.java
+28 -0
@@ -104,6 +104,34 @@
104
104
}
105
105
106
106
@Test
107
+ void resyncPicksUpRepositoriesCreatedAfterFollow()
108
+ {
109
+ User user = persistUser("ruf-rick-" + unique());
110
+ String person = "https://peer.test/ap/users/bob-" + unique();
111
+ canned.put(person, List.of()); // no public repos at follow time
112
+ RemoteUserFollow uf = service.followUser(user, person);
113
+ assertTrue(service.repositoriesOfFollowedUser(user, person).isEmpty());
114
+
115
+ // The remote creates a public repository afterwards.
116
+ String repoOne = "https://peer.test/ap/repos/bob/late-" + unique();
117
+ seedRemoteActor(repoOne);
118
+ canned.put(person, List.of(repoOne));
119
+
120
+ service.resyncUser(uf.id);
121
+
122
+ List<RemoteFollow> repos = service.repositoriesOfFollowedUser(user, person);
123
+ assertEquals(1, repos.size());
124
+ assertEquals(repoOne, repos.get(0).remoteActorId);
125
+ assertEquals(person, repos.get(0).viaUserActorId);
126
+ assertTrue(deliveries.findDue(Instant.now()).stream()
127
+ .anyMatch(t -> t.targetInbox.equals(repoOne + "/inbox") && t.payload.contains("\"Follow\"")));
128
+
129
+ // Re-syncing again must not duplicate the follow.
130
+ service.resyncUser(uf.id);
131
+ assertEquals(1, service.repositoriesOfFollowedUser(user, person).size());
132
+ }
133
+
134
+ @Test
107
135
void unfollowUserUndoesEveryDerivedFollow()
108
136
{
109
137
User user = persistUser("ruf-frank-" + unique());