fix!: order privilege events by sequence number and merge snapshots by position (#333)
Unbound Release / Check Preconditions (push) Successful in 27s
Unbound Release / Create Tag (push) Skipped
Unbound Release / Create Release (push) Successful in 27s
authz_client / test (push) Successful in 1m10s
authz_client / vulnerabilities (push) Successful in 53s
Unbound Release / Generate Changelog and Handle PR (push) Successful in 41s
Release / release (push) Successful in 1m10s
pre-commit / pre-commit (push) Successful in 3m12s

## Why
The privilege cache could keep a grant authz-service had revoked:
- **Unordered keys:** each routing key has its own transient queue, so a late `Privilege.Added`/`User.Added` resurrected a revoked grant.
- **Startup gap:** services fetched `/authz` before binding their queues, so revocations published in between were lost until restart.

Design: ADR-0015 (docs PR, Proposed).

## What
- `Process` orders events by authz-service's global `sequenceNo` per (email, company). All four events come from the Company aggregate, so seq order equals commit order. An event only overrides older facts, and `User.Removed` stamps every privilege.
- Events without a sequence number fail closed: additions are dropped, and removals hold until the next snapshot. Negative or huge sequence numbers are dropped.
- `Fetch` checks the status and retries 503 (60×1 s, 30 s HTTP timeout). It reads `X-Authz-Sequence`, merges the snapshot as facts at that position, and raises a floor; snapshots older than the floor are ignored. A missing header merges at 0 with a warning (rollout window only).
- `CompaniesByUser` returns `[]`, and unknown privileges create no state. CLAUDE.md is rewritten.

**BREAKING:** `Process` without `SequenceNo` no longer grants, so service tests must set it. Services must call `Fetch()` after `conn.Start`.

## Verification
- `go test -race`: 98.3% coverage, including table-driven reorderings, snapshot-merge cases and a revocation-during-Fetch race test.
- 26 mutants on the ordering, merge and retry checks: all killed (each compiled and produced `--- FAIL`).
- prek passes.

**Expert review:** two rounds. Round 1: Security, Go Backend, Event Sourcing and Database experts reviewed both diffs. Round 2: Security and Event Sourcing re-reviewed the fixes. A final Event Sourcing review covered the committed-events wrapper. Fixed from the reviews: older snapshots merged after newer ones (floor check); a lagging read view serving snapshots that miss revocations (503 plus catch-up); a reset's TRUNCATE emptying a REPEATABLE READ snapshot (LOCK TABLE privileges, verified on PostgreSQL); seq-0 removals undone by older additions (pending stamp); late-committing events skipped by read view backfills (CommittedEventStore with LOCK TABLE events IN SHARE MODE, verified on PostgreSQL 18); an unbounded lock wait (lock_timeout plus retries); catch-up firing on ordinary lag (5 s stall, 10 s cooldown, 10 min deadline); plus smaller items (unknown privileges, invalid seqs, `require` in a goroutine, wrapped errors, `[]` not nil). **Deliberately deferred (tracked in Ambix):** stored-but-unpublished revocations (user decision: authz-service outbox); the readview library's commit-order gap for other services (upstream); removing the missing-header fallback and alerting on /authz 503s; moving Fetch after conn.Start in the 13 consumers (separate bump PRs).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01DVGsVQ8AMFR4NZoxyCoEqS
Reviewed-on: https://gitea.unbound.se/shiny/authz_client/pulls/333
This commit was merged in pull request #333.
This commit is contained in:
argoyle committed 2026-09-16 18:36:07 +00:00
1 parent 5100d53c76
commit e0d1ce3b31
4 files changed
+693 -142

No files matched your search

+393 -61
View File
@@ -3,11 +3,14 @@ package client
import (
"context"
"fmt"
"math"
"net/http"
"net/http/httptest"
"sort"
"sync"
"sync/atomic"
"testing"
"time"
goamqp "codeberg.org/messaging/go-messaging-amqp"
spec "codeberg.org/messaging/messaging"
@@ -27,9 +30,10 @@ func TestPrivilegeHandler_Process_PrivilegeRemoved(t *testing.T) {
handler := New(WithBaseURL("base"))
err := handler.Process(&PrivilegeAdded{
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeAdmin,
SequenceNo: 1,
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeAdmin,
})
assert.NoError(t, err)
@@ -40,9 +44,10 @@ func TestPrivilegeHandler_Process_PrivilegeRemoved(t *testing.T) {
assert.Equal(t, []string{"abc-123"}, companies)
err = handler.Process(&PrivilegeRemoved{
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeAdmin,
SequenceNo: 2,
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeAdmin,
})
assert.NoError(t, err)
@@ -57,14 +62,16 @@ func TestPrivilegeHandler_Process_UserAdded_And_UserRemoved(t *testing.T) {
handler := New(WithBaseURL("base"))
err := handler.Process(&UserAdded{
Email: "jim@example.org",
CompanyID: "abc-123",
SequenceNo: 1,
Email: "jim@example.org",
CompanyID: "abc-123",
})
assert.NoError(t, err)
err = handler.Process(&UserAdded{
Email: "jim@example.org",
CompanyID: "abc-456",
SequenceNo: 2,
Email: "jim@example.org",
CompanyID: "abc-456",
})
assert.NoError(t, err)
@@ -75,14 +82,16 @@ func TestPrivilegeHandler_Process_UserAdded_And_UserRemoved(t *testing.T) {
assert.Equal(t, []string{"abc-123", "abc-456"}, companies)
err = handler.Process(&UserRemoved{
Email: "jim@example.org",
CompanyID: "abc-123",
SequenceNo: 3,
Email: "jim@example.org",
CompanyID: "abc-123",
})
assert.NoError(t, err)
err = handler.Process(&UserRemoved{
Email: "jim@example.org",
CompanyID: "abc-456",
SequenceNo: 4,
Email: "jim@example.org",
CompanyID: "abc-456",
})
assert.NoError(t, err)
@@ -96,8 +105,8 @@ func TestPrivilegeHandler_Process_UserAdded_Keeps_Existing_Privileges(t *testing
// Each routing key has its own transient queue, so Privilege.Added can be
// processed before the User.Added published before it. The privilege must
// survive either order: nothing re-reads it until the next Fetch at start.
userAdded := &UserAdded{Email: "jim@example.org", CompanyID: "abc-123"}
privilegeAdded := &PrivilegeAdded{Email: "jim@example.org", CompanyID: "abc-123", Privilege: PrivilegeAdmin}
userAdded := &UserAdded{SequenceNo: 1, Email: "jim@example.org", CompanyID: "abc-123"}
privilegeAdded := &PrivilegeAdded{SequenceNo: 2, Email: "jim@example.org", CompanyID: "abc-123", Privilege: PrivilegeAdmin}
for name, order := range map[string][]any{
"user added first": {userAdded, privilegeAdded},
"privilege added first": {privilegeAdded, userAdded},
@@ -118,14 +127,15 @@ func TestPrivilegeHandler_Process_UserAdded_Keeps_Existing_Privileges(t *testing
func TestPrivilegeHandler_Process_UserAdded_After_UserRemoved_Starts_Empty(t *testing.T) {
handler := New(WithBaseURL("base"))
assert.NoError(t, handler.Process(&UserAdded{Email: "jim@example.org", CompanyID: "abc-123"}))
assert.NoError(t, handler.Process(&UserAdded{SequenceNo: 1, Email: "jim@example.org", CompanyID: "abc-123"}))
assert.NoError(t, handler.Process(&PrivilegeAdded{
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeAdmin,
SequenceNo: 2,
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeAdmin,
}))
assert.NoError(t, handler.Process(&UserRemoved{Email: "jim@example.org", CompanyID: "abc-123"}))
assert.NoError(t, handler.Process(&UserAdded{Email: "jim@example.org", CompanyID: "abc-123"}))
assert.NoError(t, handler.Process(&UserRemoved{SequenceNo: 3, Email: "jim@example.org", CompanyID: "abc-123"}))
assert.NoError(t, handler.Process(&UserAdded{SequenceNo: 4, Email: "jim@example.org", CompanyID: "abc-123"}))
// Membership is back, without the privileges the removal took away.
assert.Equal(t, []string{"abc-123"}, handler.CompaniesByUser("jim@example.org", func(CompanyPrivileges) bool {
@@ -150,8 +160,9 @@ func TestPrivilegeHandler_GetCompanies_No_Companies_Found(t *testing.T) {
handler := New(WithBaseURL("base"))
err := handler.Process(&UserAdded{
Email: "jim@example.org",
CompanyID: "abc-123",
SequenceNo: 1,
Email: "jim@example.org",
CompanyID: "abc-123",
})
assert.NoError(t, err)
@@ -168,8 +179,9 @@ func TestPrivilegeHandler_GetCompanies_No_Companies_Found(t *testing.T) {
assert.Equal(t, []string{"abc-123"}, companies)
err = handler.Process(&UserRemoved{
Email: "jim@example.org",
CompanyID: "abc-123",
SequenceNo: 2,
Email: "jim@example.org",
CompanyID: "abc-123",
})
assert.NoError(t, err)
@@ -183,9 +195,10 @@ func TestPrivilegeHandler_GetCompanies_Company_With_Company_Access_Found(t *test
handler := New(WithBaseURL("base"))
err := handler.Process(&PrivilegeAdded{
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeCompany,
SequenceNo: 1,
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeCompany,
})
assert.NoError(t, err)
@@ -200,9 +213,10 @@ func TestPrivilegeHandler_GetCompanies_Company_With_Admin_Access_Found(t *testin
handler := New(WithBaseURL("base"))
err := handler.Process(&PrivilegeAdded{
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeConsumer,
SequenceNo: 1,
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeConsumer,
})
assert.NoError(t, err)
@@ -227,9 +241,10 @@ func TestPrivilegeHandler_IsAllowed_Return_True_If_Privilege_Exists(t *testing.T
handler := New(WithBaseURL("base"))
_ = handler.Process(&PrivilegeAdded{
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeTime,
SequenceNo: 1,
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeTime,
})
result := handler.IsAllowed("jim@example.org", "abc-123", func(privileges CompanyPrivileges) bool {
@@ -239,9 +254,10 @@ func TestPrivilegeHandler_IsAllowed_Return_True_If_Privilege_Exists(t *testing.T
assert.True(t, result)
_ = handler.Process(&PrivilegeAdded{
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeInvoicing,
SequenceNo: 2,
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeInvoicing,
})
result = handler.IsAllowed("jim@example.org", "abc-123", func(privileges CompanyPrivileges) bool {
@@ -251,9 +267,10 @@ func TestPrivilegeHandler_IsAllowed_Return_True_If_Privilege_Exists(t *testing.T
assert.True(t, result)
_ = handler.Process(&PrivilegeAdded{
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeAccounting,
SequenceNo: 3,
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeAccounting,
})
result = handler.IsAllowed("jim@example.org", "abc-123", func(privileges CompanyPrivileges) bool {
@@ -263,9 +280,10 @@ func TestPrivilegeHandler_IsAllowed_Return_True_If_Privilege_Exists(t *testing.T
assert.True(t, result)
_ = handler.Process(&PrivilegeAdded{
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeSupplier,
SequenceNo: 4,
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeSupplier,
})
result = handler.IsAllowed("jim@example.org", "abc-123", func(privileges CompanyPrivileges) bool {
@@ -275,9 +293,10 @@ func TestPrivilegeHandler_IsAllowed_Return_True_If_Privilege_Exists(t *testing.T
assert.True(t, result)
_ = handler.Process(&PrivilegeAdded{
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeSalary,
SequenceNo: 5,
Email: "jim@example.org",
CompanyID: "abc-123",
Privilege: PrivilegeSalary,
})
result = handler.IsAllowed("jim@example.org", "abc-123", func(privileges CompanyPrivileges) bool {
@@ -386,7 +405,7 @@ func TestPrivilegeHandler_Fetch_Valid(t *testing.T) {
err := handler.Fetch()
assert.NoError(t, err)
expectedPrivileges := map[string]map[string]*CompanyPrivileges{
expectedPrivileges := map[string]map[string]CompanyPrivileges{
"jim@example.org": {
"00010203-0405-4607-8809-0a0b0c0d0e0f": {
Admin: false,
@@ -400,7 +419,7 @@ func TestPrivilegeHandler_Fetch_Valid(t *testing.T) {
},
},
}
assert.Equal(t, expectedPrivileges, handler.privileges)
assert.Equal(t, expectedPrivileges, privilegesOf(handler))
}
func TestPrivilegeHandler_Fetch_Concurrent_Fetches(t *testing.T) {
@@ -450,7 +469,7 @@ func TestPrivilegeHandler_Fetch_Concurrent_Fetches(t *testing.T) {
}
// Verify privileges were set correctly
expectedPrivileges := map[string]map[string]*CompanyPrivileges{
expectedPrivileges := map[string]map[string]CompanyPrivileges{
"jim@example.org": {
"00010203-0405-4607-8809-0a0b0c0d0e0f": {
Admin: false,
@@ -464,7 +483,7 @@ func TestPrivilegeHandler_Fetch_Concurrent_Fetches(t *testing.T) {
},
},
}
assert.Equal(t, expectedPrivileges, handler.privileges)
assert.Equal(t, expectedPrivileges, privilegesOf(handler))
}
func TestPrivilegeHandler_Concurrent_Fetch_And_Read(t *testing.T) {
@@ -559,9 +578,10 @@ func TestPrivilegeHandler_Concurrent_Process_And_Read(t *testing.T) {
go func(id string) {
defer wg.Done()
_ = handler.Process(&PrivilegeAdded{
Email: "jim@example.org",
CompanyID: id,
Privilege: PrivilegeAdmin,
SequenceNo: 1,
Email: "jim@example.org",
CompanyID: id,
Privilege: PrivilegeAdmin,
})
}(companyID)
}
@@ -635,9 +655,10 @@ func TestPrivilegeHandler_Concurrent_Multiple_Operations(t *testing.T) {
go func(idx int) {
defer wg.Done()
_ = handler.Process(&PrivilegeAdded{
Email: "jane@example.org",
CompanyID: fmt.Sprintf("company-%d", idx%5),
Privilege: PrivilegeCompany,
SequenceNo: 1,
Email: "jane@example.org",
CompanyID: fmt.Sprintf("company-%d", idx%5),
Privilege: PrivilegeCompany,
})
}(i)
}
@@ -711,22 +732,333 @@ func TestPrivilegeHandler_process(t *testing.T) {
admin := func(p CompanyPrivileges) bool { return p.Admin }
assert.NoError(t, process[UserAdded](handler)(ctx, spec.ConsumableEvent[UserAdded]{
Payload: UserAdded{Email: "jim@example.org", CompanyID: "abc-123"},
Payload: UserAdded{SequenceNo: 1, Email: "jim@example.org", CompanyID: "abc-123"},
}))
assert.False(t, handler.IsAllowed("jim@example.org", "abc-123", admin))
assert.NoError(t, process[PrivilegeAdded](handler)(ctx, spec.ConsumableEvent[PrivilegeAdded]{
Payload: PrivilegeAdded{Email: "jim@example.org", CompanyID: "abc-123", Privilege: PrivilegeAdmin},
Payload: PrivilegeAdded{SequenceNo: 2, Email: "jim@example.org", CompanyID: "abc-123", Privilege: PrivilegeAdmin},
}))
assert.True(t, handler.IsAllowed("jim@example.org", "abc-123", admin))
assert.NoError(t, process[PrivilegeRemoved](handler)(ctx, spec.ConsumableEvent[PrivilegeRemoved]{
Payload: PrivilegeRemoved{Email: "jim@example.org", CompanyID: "abc-123", Privilege: PrivilegeAdmin},
Payload: PrivilegeRemoved{SequenceNo: 3, Email: "jim@example.org", CompanyID: "abc-123", Privilege: PrivilegeAdmin},
}))
assert.False(t, handler.IsAllowed("jim@example.org", "abc-123", admin))
assert.NoError(t, process[UserRemoved](handler)(ctx, spec.ConsumableEvent[UserRemoved]{
Payload: UserRemoved{Email: "jim@example.org", CompanyID: "abc-123"},
Payload: UserRemoved{SequenceNo: 4, Email: "jim@example.org", CompanyID: "abc-123"},
}))
assert.Empty(t, handler.CompaniesByUser("jim@example.org", func(CompanyPrivileges) bool { return true }))
}
// privilegesOf returns the privileges of every membership in the handler.
func privilegesOf(h *PrivilegeHandler) map[string]map[string]CompanyPrivileges {
h.RLock()
defer h.RUnlock()
result := map[string]map[string]CompanyPrivileges{}
for email, companies := range h.state {
for companyID, s := range companies {
if !s.member {
continue
}
if result[email] == nil {
result[email] = map[string]CompanyPrivileges{}
}
result[email][companyID] = s.privileges
}
}
return result
}
func TestPrivilegeHandler_Process_Orders_By_SequenceNo(t *testing.T) {
const email, company = "jim@example.org", "abc-123"
userAdded := func(seq int) any { return &UserAdded{Email: email, CompanyID: company, SequenceNo: seq} }
userRemoved := func(seq int) any { return &UserRemoved{Email: email, CompanyID: company, SequenceNo: seq} }
added := func(seq int) any {
return &PrivilegeAdded{Email: email, CompanyID: company, Privilege: PrivilegeAdmin, SequenceNo: seq}
}
removed := func(seq int) any {
return &PrivilegeRemoved{Email: email, CompanyID: company, Privilege: PrivilegeAdmin, SequenceNo: seq}
}
tests := []struct {
name string
delivered []any
member bool
admin bool
}{
{name: "in order", delivered: []any{userAdded(1), added(2)}, member: true, admin: true},
{name: "privilege before older user added keeps the grant", delivered: []any{added(2), userAdded(1)}, member: true, admin: true},
{name: "late privilege added after user removed", delivered: []any{userAdded(1), userRemoved(3), added(2)}, member: false, admin: false},
{name: "late privilege added after privilege removed", delivered: []any{userAdded(1), removed(3), added(2)}, member: true, admin: false},
{name: "late user added does not revive privileges removed with the user", delivered: []any{userRemoved(3), added(2), userAdded(1)}, member: false, admin: false},
{name: "re-added user starts without the removed privileges", delivered: []any{added(2), userRemoved(3), userAdded(4)}, member: true, admin: false},
{name: "privilege granted after re-add", delivered: []any{userAdded(4), userRemoved(3), added(5), added(2)}, member: true, admin: true},
{name: "late user removed does not remove a newer grant", delivered: []any{added(4), userRemoved(3)}, member: true, admin: true},
{name: "late user removed does not remove a newer membership", delivered: []any{userAdded(4), userRemoved(3)}, member: true, admin: false},
{name: "late privilege added before re-add stays revoked", delivered: []any{userAdded(1), userRemoved(3), added(2), userAdded(4)}, member: true, admin: false},
{name: "privilege removed after user removed keeps the user out", delivered: []any{userRemoved(2), removed(3)}, member: false, admin: false},
{name: "duplicate delivery is idempotent", delivered: []any{added(2), removed(3), added(2)}, member: true, admin: false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
handler := New()
for _, event := range tt.delivered {
require.NoError(t, handler.Process(event))
}
all := func(CompanyPrivileges) bool { return true }
assert.Equal(t, tt.member, handler.IsAllowed(email, company, all))
assert.Equal(t, tt.admin, handler.IsAllowed(email, company, func(p CompanyPrivileges) bool { return p.Admin }))
if tt.member {
assert.Equal(t, []string{company}, handler.CompaniesByUser(email, all))
} else {
assert.Equal(t, []string{}, handler.CompaniesByUser(email, all))
}
})
}
}
func TestPrivilegeHandler_Process_Without_SequenceNo_Fails_Closed(t *testing.T) {
const email, company = "jim@example.org", "abc-123"
admin := func(p CompanyPrivileges) bool { return p.Admin }
handler := New()
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: company, Privilege: PrivilegeAdmin}))
require.NoError(t, handler.Process(&UserAdded{Email: email, CompanyID: company}))
assert.False(t, handler.IsAllowed(email, company, func(CompanyPrivileges) bool { return true }), "an unordered addition is dropped")
require.NoError(t, handler.Process(&UserAdded{Email: email, CompanyID: company, SequenceNo: 1}))
assert.False(t, handler.IsAllowed(email, company, admin), "the dropped grant doesn't appear with a later membership")
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: company, Privilege: PrivilegeAdmin, SequenceNo: 5}))
require.NoError(t, handler.Process(&PrivilegeRemoved{Email: email, CompanyID: company, Privilege: PrivilegeAdmin}))
assert.False(t, handler.IsAllowed(email, company, admin), "an unordered privilege removal applies")
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: company, Privilege: PrivilegeTime, SequenceNo: 6}))
require.NoError(t, handler.Process(&UserRemoved{Email: email, CompanyID: company}))
assert.False(t, handler.IsAllowed(email, company, func(CompanyPrivileges) bool { return true }), "an unordered user removal applies")
}
// snapshotServer serves body as the /authz snapshot at seq; a negative seq sends no header.
func snapshotServer(t *testing.T, seq int, body string) *httptest.Server {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if seq >= 0 {
w.Header().Set(SequenceHeader, fmt.Sprint(seq))
}
_, _ = w.Write([]byte(body))
}))
t.Cleanup(server.Close)
return server
}
func TestPrivilegeHandler_Fetch_Merges_Snapshot_By_SequenceNo(t *testing.T) {
const email = "jim@example.org"
admin := func(p CompanyPrivileges) bool { return p.Admin }
all := func(CompanyPrivileges) bool { return true }
snapshot := `{"jim@example.org": {"in-snapshot": {"admin": true}, "stale-in-snapshot": {"admin": true}}}`
handler := New(WithBaseURL(snapshotServer(t, 10, snapshot).URL))
// Delivered while the snapshot was read.
require.NoError(t, handler.Process(&PrivilegeRemoved{Email: email, CompanyID: "in-snapshot", Privilege: PrivilegeAdmin, SequenceNo: 11}))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: "stale-in-snapshot", Privilege: PrivilegeTime, SequenceNo: 9}))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: "new", Privilege: PrivilegeAdmin, SequenceNo: 12}))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: "revoked", Privilege: PrivilegeAdmin, SequenceNo: 8}))
require.NoError(t, handler.Process(&UserAdded{Email: email, CompanyID: "revoked-later", SequenceNo: 7}))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: "revoked-later", Privilege: PrivilegeAdmin, SequenceNo: 13}))
require.NoError(t, handler.Fetch())
assert.True(t, handler.IsAllowed(email, "in-snapshot", all))
assert.False(t, handler.IsAllowed(email, "in-snapshot", admin), "a removal newer than the snapshot wins")
assert.False(t, handler.IsAllowed(email, "stale-in-snapshot", func(p CompanyPrivileges) bool { return p.Time }), "the snapshot replaces an older fact")
assert.True(t, handler.IsAllowed(email, "new", admin), "a grant newer than the snapshot is kept")
assert.False(t, handler.IsAllowed(email, "revoked", all), "a pair missing from the snapshot is removed")
assert.True(t, handler.IsAllowed(email, "revoked-later", admin), "a grant newer than the snapshot survives the removal")
// Events the snapshot already covers are stale.
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: "revoked", Privilege: PrivilegeAdmin, SequenceNo: 10}))
assert.False(t, handler.IsAllowed(email, "revoked", all), "an event at the snapshot sequence is stale")
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: "revoked", Privilege: PrivilegeAdmin, SequenceNo: 14}))
assert.True(t, handler.IsAllowed(email, "revoked", admin))
}
func TestPrivilegeHandler_Fetch_Replaces_Pairs_Missing_From_Snapshot(t *testing.T) {
handler := New(WithBaseURL(snapshotServer(t, 10, `{"jim@example.org": {"kept": {"company": true}}}`).URL))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: "jim@example.org", CompanyID: "gone", Privilege: PrivilegeAdmin, SequenceNo: 3}))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: "jane@example.org", CompanyID: "gone", Privilege: PrivilegeAdmin, SequenceNo: 4}))
require.NoError(t, handler.Fetch())
assert.Equal(t, map[string]map[string]CompanyPrivileges{"jim@example.org": {"kept": {Company: true}}}, privilegesOf(handler))
handler.RLock()
defer handler.RUnlock()
assert.NotContains(t, handler.state, "jane@example.org", "an email without companies is dropped")
}
func TestPrivilegeHandler_Fetch_Without_SequenceHeader_Keeps_Sequenced_Facts(t *testing.T) {
handler := New(WithBaseURL(snapshotServer(t, -1, `{"jim@example.org": {"abc-123": {"admin": true}}}`).URL))
require.NoError(t, handler.Process(&PrivilegeRemoved{Email: "jim@example.org", CompanyID: "abc-123", Privilege: PrivilegeAdmin, SequenceNo: 3}))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: "jim@example.org", CompanyID: "other", Privilege: PrivilegeAdmin, SequenceNo: 4}))
require.NoError(t, handler.Fetch())
assert.Equal(t, map[string]map[string]CompanyPrivileges{
"jim@example.org": {"abc-123": {}, "other": {Admin: true}},
}, privilegesOf(handler))
}
func TestPrivilegeHandler_Fetch_Rejects_Bad_Responses(t *testing.T) {
tests := []struct {
name string
status int
header string
err string
}{
{name: "status", status: http.StatusUnauthorized, err: "fetch privileges: unexpected status 401 Unauthorized"},
{name: "invalid header", status: http.StatusOK, header: "abc", err: `fetch privileges: invalid X-Authz-Sequence header "abc"`},
{name: "negative header", status: http.StatusOK, header: "-1", err: `fetch privileges: invalid X-Authz-Sequence header "-1"`},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if tt.header != "" {
w.Header().Set(SequenceHeader, tt.header)
}
w.WriteHeader(tt.status)
_, _ = w.Write([]byte(`{"jim@example.org": {"abc-123": {"admin": true}}}`))
}))
defer server.Close()
handler := New(WithBaseURL(server.URL))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: "jane@example.org", CompanyID: "abc-123", Privilege: PrivilegeTime, SequenceNo: 1}))
assert.EqualError(t, handler.Fetch(), tt.err)
assert.Equal(t, map[string]map[string]CompanyPrivileges{"jane@example.org": {"abc-123": {Time: true}}}, privilegesOf(handler), "state is untouched")
})
}
}
func TestPrivilegeHandler_Revocation_During_Fetch_Wins(t *testing.T) {
// The consumers are bound before Fetch, so a revocation published after authz-service
// read the snapshot is processed while the response is still on its way.
handler := New()
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// assert, not require: this runs on the server's goroutine.
assert.NoError(t, handler.Process(&UserRemoved{Email: "jim@example.org", CompanyID: "abc-123", SequenceNo: 21}))
w.Header().Set(SequenceHeader, "20")
_, _ = w.Write([]byte(`{"jim@example.org": {"abc-123": {"admin": true}}}`))
}))
defer server.Close()
handler.baseURL = server.URL
require.NoError(t, handler.Fetch())
assert.False(t, handler.IsAllowed("jim@example.org", "abc-123", func(CompanyPrivileges) bool { return true }))
}
func TestPrivilegeHandler_Fetch_Ignores_Snapshot_Older_Than_Merged_One(t *testing.T) {
const email = "jim@example.org"
all := func(CompanyPrivileges) bool { return true }
handler := New(WithBaseURL(snapshotServer(t, 12, `{}`).URL))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: "abc-123", Privilege: PrivilegeAdmin, SequenceNo: 5}))
require.NoError(t, handler.Process(&UserRemoved{Email: email, CompanyID: "abc-123", SequenceNo: 11}))
require.NoError(t, handler.Fetch())
// A concurrent Fetch read at seq 10, before the removal, merges last.
handler.baseURL = snapshotServer(t, 10, `{"jim@example.org": {"abc-123": {"admin": true}}}`).URL
require.NoError(t, handler.Fetch())
assert.False(t, handler.IsAllowed(email, "abc-123", all))
// So does a snapshot without a sequence number.
handler.baseURL = snapshotServer(t, -1, `{"jim@example.org": {"abc-123": {"admin": true}}}`).URL
require.NoError(t, handler.Fetch())
assert.False(t, handler.IsAllowed(email, "abc-123", all))
}
func TestPrivilegeHandler_Unordered_Removal_Holds_Until_Next_Snapshot(t *testing.T) {
const email, company = "jim@example.org", "abc-123"
all := func(CompanyPrivileges) bool { return true }
admin := func(p CompanyPrivileges) bool { return p.Admin }
handler := New(WithBaseURL(snapshotServer(t, 30, `{"jim@example.org": {"abc-123": {"admin": true}, "def-456": {"time": true}}}`).URL))
require.NoError(t, handler.Process(&UserAdded{Email: email, CompanyID: company, SequenceNo: 10}))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: company, Privilege: PrivilegeAdmin, SequenceNo: 11}))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: "def-456", Privilege: PrivilegeTime, SequenceNo: 12}))
require.NoError(t, handler.Process(&UserRemoved{Email: email, CompanyID: company}))
require.NoError(t, handler.Process(&PrivilegeRemoved{Email: email, CompanyID: "def-456", Privilege: PrivilegeTime}))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: company, Privilege: PrivilegeSalary, SequenceNo: 15}))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: company, Privilege: PrivilegeAdmin, SequenceNo: 16}))
require.NoError(t, handler.Process(&UserAdded{Email: email, CompanyID: company, SequenceNo: 17}))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: email, CompanyID: "def-456", Privilege: PrivilegeTime, SequenceNo: 18}))
assert.False(t, handler.IsAllowed(email, company, all), "no event overrides an unordered user removal")
assert.False(t, handler.IsAllowed(email, "def-456", func(p CompanyPrivileges) bool { return p.Time }), "no event overrides an unordered privilege removal")
require.NoError(t, handler.Fetch())
assert.True(t, handler.IsAllowed(email, company, admin), "the next snapshot replaces the unordered removal")
assert.True(t, handler.IsAllowed(email, "def-456", func(p CompanyPrivileges) bool { return p.Time }))
}
func TestPrivilegeHandler_Fetch_Deletes_Unordered_Removal_Missing_From_Snapshot(t *testing.T) {
handler := New(WithBaseURL(snapshotServer(t, 30, `{}`).URL))
require.NoError(t, handler.Process(&UserRemoved{Email: "jim@example.org", CompanyID: "abc-123"}))
require.NoError(t, handler.Fetch())
handler.RLock()
defer handler.RUnlock()
assert.Empty(t, handler.state)
}
func TestPrivilegeHandler_Process_Unknown_Privilege_Creates_No_State(t *testing.T) {
handler := New()
require.NoError(t, handler.Process(&PrivilegeRemoved{Email: "jim@example.org", CompanyID: "abc-123", Privilege: "UNKNOWN", SequenceNo: 1}))
require.NoError(t, handler.Process(&PrivilegeAdded{Email: "jim@example.org", CompanyID: "abc-123", Privilege: "UNKNOWN", SequenceNo: 2}))
handler.RLock()
defer handler.RUnlock()
assert.Empty(t, handler.state)
}
func TestPrivilegeHandler_Fetch_Retries_While_ReadView_Not_Ready(t *testing.T) {
var calls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if calls.Add(1) < 3 {
w.WriteHeader(http.StatusServiceUnavailable)
return
}
w.Header().Set(SequenceHeader, "7")
_, _ = w.Write([]byte(`{"jim@example.org": {"abc-123": {"admin": true}}}`))
}))
defer server.Close()
handler := New(WithBaseURL(server.URL))
handler.retryDelay = time.Millisecond
require.NoError(t, handler.Fetch())
assert.Equal(t, int32(3), calls.Load())
assert.True(t, handler.IsAllowed("jim@example.org", "abc-123", func(p CompanyPrivileges) bool { return p.Admin }))
}
func TestPrivilegeHandler_Fetch_Gives_Up_When_ReadView_Stays_Behind(t *testing.T) {
var calls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls.Add(1)
w.WriteHeader(http.StatusServiceUnavailable)
}))
defer server.Close()
handler := New(WithBaseURL(server.URL))
handler.retryDelay = time.Millisecond
handler.retries = 2
assert.EqualError(t, handler.Fetch(), "fetch privileges: authz-service read view not ready")
assert.Equal(t, int32(3), calls.Load())
}
func TestPrivilegeHandler_Process_Drops_Invalid_SequenceNo(t *testing.T) {
for _, seq := range []int{-1, math.MaxInt} {
handler := New()
require.NoError(t, handler.Process(&PrivilegeAdded{Email: "jim@example.org", CompanyID: "abc-123", Privilege: PrivilegeAdmin, SequenceNo: seq}))
require.NoError(t, handler.Process(&UserRemoved{Email: "jim@example.org", CompanyID: "abc-123", SequenceNo: seq}))
handler.RLock()
assert.Empty(t, handler.state, "seq %d", seq)
handler.RUnlock()
}
}