fix!: order privilege events by sequence number and merge snapshots by position (#333)
Unbound Release / Check Preconditions (push) Successful in 27s
Unbound Release / Create Tag (push) Skipped
Unbound Release / Create Release (push) Successful in 27s
authz_client / test (push) Successful in 1m10s
authz_client / vulnerabilities (push) Successful in 53s
Unbound Release / Generate Changelog and Handle PR (push) Successful in 41s
Release / release (push) Successful in 1m10s
pre-commit / pre-commit (push) Successful in 3m12s

## Why
The privilege cache could keep a grant authz-service had revoked:
- **Unordered keys:** each routing key has its own transient queue, so a late `Privilege.Added`/`User.Added` resurrected a revoked grant.
- **Startup gap:** services fetched `/authz` before binding their queues, so revocations published in between were lost until restart.

Design: ADR-0015 (docs PR, Proposed).

## What
- `Process` orders events by authz-service's global `sequenceNo` per (email, company). All four events come from the Company aggregate, so seq order equals commit order. An event only overrides older facts, and `User.Removed` stamps every privilege.
- Events without a sequence number fail closed: additions are dropped, and removals hold until the next snapshot. Negative or huge sequence numbers are dropped.
- `Fetch` checks the status and retries 503 (60×1 s, 30 s HTTP timeout). It reads `X-Authz-Sequence`, merges the snapshot as facts at that position, and raises a floor; snapshots older than the floor are ignored. A missing header merges at 0 with a warning (rollout window only).
- `CompaniesByUser` returns `[]`, and unknown privileges create no state. CLAUDE.md is rewritten.

**BREAKING:** `Process` without `SequenceNo` no longer grants, so service tests must set it. Services must call `Fetch()` after `conn.Start`.

## Verification
- `go test -race`: 98.3% coverage, including table-driven reorderings, snapshot-merge cases and a revocation-during-Fetch race test.
- 26 mutants on the ordering, merge and retry checks: all killed (each compiled and produced `--- FAIL`).
- prek passes.

**Expert review:** two rounds. Round 1: Security, Go Backend, Event Sourcing and Database experts reviewed both diffs. Round 2: Security and Event Sourcing re-reviewed the fixes. A final Event Sourcing review covered the committed-events wrapper. Fixed from the reviews: older snapshots merged after newer ones (floor check); a lagging read view serving snapshots that miss revocations (503 plus catch-up); a reset's TRUNCATE emptying a REPEATABLE READ snapshot (LOCK TABLE privileges, verified on PostgreSQL); seq-0 removals undone by older additions (pending stamp); late-committing events skipped by read view backfills (CommittedEventStore with LOCK TABLE events IN SHARE MODE, verified on PostgreSQL 18); an unbounded lock wait (lock_timeout plus retries); catch-up firing on ordinary lag (5 s stall, 10 s cooldown, 10 min deadline); plus smaller items (unknown privileges, invalid seqs, `require` in a goroutine, wrapped errors, `[]` not nil). **Deliberately deferred (tracked in Ambix):** stored-but-unpublished revocations (user decision: authz-service outbox); the readview library's commit-order gap for other services (upstream); removing the missing-header fallback and alerting on /authz 503s; moving Fetch after conn.Start in the 13 consumers (separate bump PRs).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01DVGsVQ8AMFR4NZoxyCoEqS
Reviewed-on: https://gitea.unbound.se/shiny/authz_client/pulls/333
This commit was merged in pull request #333.
This commit is contained in:
argoyle committed 2026-09-16 18:36:07 +00:00
1 parent 5100d53c76
commit e0d1ce3b31
4 files changed
+693 -142

No files matched your search

+10 -5
View File
@@ -23,10 +23,9 @@ import client "gitea.unbound.se/shiny/authz_client"
handler := client.New(client.WithBaseURL("http://authz-service"))
// Check user privileges
privileges := handler.Get(email, companyID)
if privileges.Invoicing {
// User has invoicing privileges
}
allowed := handler.IsAllowed(email, companyID, func(p client.CompanyPrivileges) bool {
return p.Invoicing
})
```
### Privileges
@@ -43,4 +42,10 @@ The `CompanyPrivileges` struct contains permission flags:
### Event Handling
Registers per-replica (transient) go-messaging-amqp consumers for privilege update events from the authz-service (`Setup()`), keeping the local privilege cache up-to-date. Each routing key gets its own queue, so events for one company arrive in any order — `Process` must never discard state it didn't itself record. `User.Added` therefore creates the company entry only when it is missing: a `Privilege.Added` processed ahead of it would otherwise be overwritten, and nothing re-reads the privilege until the next `Fetch()`. The converse doesn't hold yet — `setPrivileges` still creates an entry for a company it never saw a `User.Added` for, so a `Privilege.*` delivered after `User.Removed` resurrects an all-false membership. Don't combine `Setup()` with go-messaging-amqp's `WithReconnect`: a reconnect declares new per-replica queues, so revocations published during the outage are lost unless `Fetch()` runs again. Services exit on connection loss (`CloseListener`) and re-fetch on start.
Registers per-replica (transient) go-messaging-amqp consumers for privilege events from authz-service (`Setup()`). Each routing key gets its own queue, so events arrive in any order. `Process` orders them by the `sequenceNo` authz-service's event store stamps on every event: all four events come from authz-service's `Company` aggregate, so the global sequence number orders them within a company. Each (email, company) keeps the sequence number of the last fact about membership and about each privilege; an event only overrides older facts. `User.Removed` stamps every privilege, so a late grant can't come back. An event without `sequenceNo` fails closed: additions are dropped, and removals apply and block every later event for those facts until the next snapshot. An event with a negative or implausibly large `sequenceNo` is dropped. Tests that seed the handler must set `SequenceNo`.
### Startup
Call `Fetch()` **after** `conn.Start` (consumers bound), never before: events published between the snapshot and the binding would otherwise be lost. authz-service only serves a snapshot whose read view has applied every stored event. It sends that position in the `X-Authz-Sequence` header and answers 503 while the read view lags (backlog, backfill, reset); `Fetch` retries 503 for about a minute. `Fetch` merges the snapshot as facts at that sequence number (newer event facts win, pairs missing from the snapshot are removed) and drops later events at or below it. A snapshot older than one already merged is ignored, so concurrent or repeated fetches are safe. A snapshot without the header merges at 0 and logs a warning (rollout window only).
Don't combine `Setup()` with go-messaging-amqp's `WithReconnect`: a reconnect declares new per-replica queues, so revocations published during the outage are lost unless `Fetch()` runs again. Services exit on connection loss (`CloseListener`) and re-fetch on start.