Files
authz_client/client.go
argoyle e0d1ce3b31
Unbound Release / Check Preconditions (push) Successful in 27s
Unbound Release / Create Tag (push) Skipped
Unbound Release / Create Release (push) Successful in 27s
authz_client / test (push) Successful in 1m10s
authz_client / vulnerabilities (push) Successful in 53s
Unbound Release / Generate Changelog and Handle PR (push) Successful in 41s
Release / release (push) Successful in 1m10s
pre-commit / pre-commit (push) Successful in 3m12s
fix!: order privilege events by sequence number and merge snapshots by position (#333)
## Why
The privilege cache could keep a grant authz-service had revoked:
- **Unordered keys:** each routing key has its own transient queue, so a late `Privilege.Added`/`User.Added` resurrected a revoked grant.
- **Startup gap:** services fetched `/authz` before binding their queues, so revocations published in between were lost until restart.

Design: ADR-0015 (docs PR, Proposed).

## What
- `Process` orders events by authz-service's global `sequenceNo` per (email, company). All four events come from the Company aggregate, so seq order equals commit order. An event only overrides older facts, and `User.Removed` stamps every privilege.
- Events without a sequence number fail closed: additions are dropped, and removals hold until the next snapshot. Negative or huge sequence numbers are dropped.
- `Fetch` checks the status and retries 503 (60×1 s, 30 s HTTP timeout). It reads `X-Authz-Sequence`, merges the snapshot as facts at that position, and raises a floor; snapshots older than the floor are ignored. A missing header merges at 0 with a warning (rollout window only).
- `CompaniesByUser` returns `[]`, and unknown privileges create no state. CLAUDE.md is rewritten.

**BREAKING:** `Process` without `SequenceNo` no longer grants, so service tests must set it. Services must call `Fetch()` after `conn.Start`.

## Verification
- `go test -race`: 98.3% coverage, including table-driven reorderings, snapshot-merge cases and a revocation-during-Fetch race test.
- 26 mutants on the ordering, merge and retry checks: all killed (each compiled and produced `--- FAIL`).
- prek passes.

**Expert review:** two rounds. Round 1: Security, Go Backend, Event Sourcing and Database experts reviewed both diffs. Round 2: Security and Event Sourcing re-reviewed the fixes. A final Event Sourcing review covered the committed-events wrapper. Fixed from the reviews: older snapshots merged after newer ones (floor check); a lagging read view serving snapshots that miss revocations (503 plus catch-up); a reset's TRUNCATE emptying a REPEATABLE READ snapshot (LOCK TABLE privileges, verified on PostgreSQL); seq-0 removals undone by older additions (pending stamp); late-committing events skipped by read view backfills (CommittedEventStore with LOCK TABLE events IN SHARE MODE, verified on PostgreSQL 18); an unbounded lock wait (lock_timeout plus retries); catch-up firing on ordinary lag (5 s stall, 10 s cooldown, 10 min deadline); plus smaller items (unknown privileges, invalid seqs, `require` in a goroutine, wrapped errors, `[]` not nil). **Deliberately deferred (tracked in Ambix):** stored-but-unpublished revocations (user decision: authz-service outbox); the readview library's commit-order gap for other services (upstream); removing the missing-header fallback and alerting on /authz 503s; moving Fetch after conn.Start in the 13 consumers (separate bump PRs).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01DVGsVQ8AMFR4NZoxyCoEqS
Reviewed-on: https://gitea.unbound.se/shiny/authz_client/pulls/333
2026-09-16 18:36:07 +00:00

423 lines
12 KiB
Go

package client
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"math"
"net/http"
"reflect"
"strconv"
"sync"
"time"
goamqp "codeberg.org/messaging/go-messaging-amqp"
spec "codeberg.org/messaging/messaging"
)
// SequenceHeader carries the global sequence number of the authz-service read view the
// /authz snapshot was read at.
const SequenceHeader = "X-Authz-Sequence"
// pending marks a fact cleared by an event without a sequence number. No event can
// override it; only a snapshot can.
const pending = math.MaxInt
// CompanyPrivileges contains the privileges for a combination of email address and company id
type CompanyPrivileges struct {
Admin bool `json:"admin"`
Company bool `json:"company"`
Consumer bool `json:"consumer"`
Time bool `json:"time"`
Invoicing bool `json:"invoicing"`
Accounting bool `json:"accounting"`
Supplier bool `json:"supplier"`
Salary bool `json:"salary"`
}
// field returns the flag for privilege, or nil for an unknown privilege.
func (c *CompanyPrivileges) field(privilege Privilege) *bool {
switch privilege {
case PrivilegeAdmin:
return &c.Admin
case PrivilegeCompany:
return &c.Company
case PrivilegeConsumer:
return &c.Consumer
case PrivilegeTime:
return &c.Time
case PrivilegeInvoicing:
return &c.Invoicing
case PrivilegeAccounting:
return &c.Accounting
case PrivilegeSupplier:
return &c.Supplier
case PrivilegeSalary:
return &c.Salary
}
return nil
}
// companyState is what the handler knows about one email in one company. Every
// privilege event for a company comes from authz-service's Company aggregate, so
// its global sequence number orders the events for the pair. The four routing
// keys arrive on separate queues in any order; a fact is only overwritten by a
// fact with a higher sequence number.
type companyState struct {
privileges CompanyPrivileges
// privilegeSeq is the sequence number of the last fact about each privilege.
privilegeSeq map[Privilege]int
// member is whether the user belongs to the company, as of memberSeq. A
// User.Removed stamps every privilege with its sequence number, so a grant
// older than the removal can't come back.
member bool
memberSeq int
}
func newCompanyState() *companyState {
return &companyState{privilegeSeq: map[Privilege]int{}}
}
// clearPending turns facts cleared without a sequence number into unordered facts, so
// the snapshot being merged replaces them.
func (s *companyState) clearPending() {
if s.memberSeq == pending {
s.memberSeq = 0
}
for p, seq := range s.privilegeSeq {
if seq == pending {
s.privilegeSeq[p] = 0
}
}
}
// maxSeq is the highest sequence number of any fact in the state.
func (s *companyState) maxSeq() int {
m := s.memberSeq
for _, seq := range s.privilegeSeq {
m = max(m, seq)
}
return m
}
// PrivilegeHandler processes PrivilegeAdded-events and fetches the initial set of privileges from an authz-service
type PrivilegeHandler struct {
*sync.RWMutex
client *http.Client
baseURL string
apiKey string
state map[string]map[string]*companyState
// floor is the sequence number of the newest snapshot. Its effects are all in
// the state, so an event at or below it is stale.
floor int
// retries is how many times Fetch retries a snapshot authz-service isn't ready to serve.
retries int
retryDelay time.Duration
}
// OptsFunc is used to configure the PrivilegeHandler
type OptsFunc func(handler *PrivilegeHandler)
// WithBaseURL sets the base URL to the authz-service
func WithBaseURL(url string) OptsFunc {
return func(handler *PrivilegeHandler) {
handler.baseURL = url
}
}
// WithAPIKey sets an API key used as a Bearer token when fetching privileges
func WithAPIKey(key string) OptsFunc {
return func(handler *PrivilegeHandler) {
handler.apiKey = key
}
}
// New creates a new PrivilegeHandler. Pass OptsFuncs to configure.
func New(opts ...OptsFunc) *PrivilegeHandler {
handler := &PrivilegeHandler{
RWMutex: &sync.RWMutex{},
client: &http.Client{Timeout: 30 * time.Second},
baseURL: "http://authz-service",
state: map[string]map[string]*companyState{},
retries: 60,
retryDelay: time.Second,
}
for _, opt := range opts {
opt(handler)
}
return handler
}
// Fetch a snapshot of all privileges from an authz-service and merge it into the state.
//
// Call it after the AMQP connection has started: authz-service only serves a snapshot
// that includes every stored event, so together with the bound queues no event is
// missed. While its read view is behind it answers 503, and Fetch retries. The snapshot
// is applied as facts at its sequence number: a fact from an event newer than the
// snapshot is kept, everything else is replaced, and a pair missing from the snapshot is
// removed. A snapshot older than one already merged is ignored.
func (h *PrivilegeHandler) Fetch() error {
for attempt := 0; ; attempt++ {
err := h.fetch()
if !errors.Is(err, errNotReady) || attempt >= h.retries {
return err
}
time.Sleep(h.retryDelay)
}
}
var errNotReady = errors.New("fetch privileges: authz-service read view not ready")
func (h *PrivilegeHandler) fetch() error {
req, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s/authz", h.baseURL), nil)
if err != nil {
return err
}
if h.apiKey != "" {
req.Header.Set("Authorization", "Bearer "+h.apiKey)
}
resp, err := h.client.Do(req)
if err != nil {
return err
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode == http.StatusServiceUnavailable {
return errNotReady
}
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("fetch privileges: unexpected status %s", resp.Status)
}
buff, err := io.ReadAll(resp.Body)
if err != nil {
return err
}
var snapshot map[string]map[string]CompanyPrivileges
if err := json.Unmarshal(buff, &snapshot); err != nil {
return err
}
seq := 0
if v := resp.Header.Get(SequenceHeader); v != "" {
seq, err = strconv.Atoi(v)
if err != nil || seq < 0 {
return fmt.Errorf("fetch privileges: invalid %s header %q", SequenceHeader, v)
}
} else {
// ponytail: only during the rollout window, before authz-service sends the header.
slog.Warn("authz snapshot has no sequence number; events older than it can revert it", "header", SequenceHeader)
}
h.Lock()
defer h.Unlock()
h.merge(snapshot, seq)
return nil
}
func (h *PrivilegeHandler) merge(snapshot map[string]map[string]CompanyPrivileges, seq int) {
if seq < h.floor {
// A newer snapshot is already merged; this one could only bring back removed state.
slog.Warn("ignoring authz snapshot older than the one already merged", "seq", seq, "floor", h.floor)
return
}
for email, companies := range h.state {
for companyID, s := range companies {
s.clearPending()
if _, exists := snapshot[email][companyID]; exists {
continue
}
if s.maxSeq() <= seq {
// Nothing newer than the snapshot; the floor keeps stale events out.
delete(companies, companyID)
continue
}
removeUser(s, seq, false)
}
if len(companies) == 0 {
delete(h.state, email)
}
}
for email, companies := range snapshot {
for companyID, privileges := range companies {
s := h.company(email, companyID)
if s.memberSeq <= seq {
s.member = true
s.memberSeq = seq
}
for _, p := range AllPrivilege {
if s.privilegeSeq[p] <= seq {
*s.privileges.field(p) = *privileges.field(p)
s.privilegeSeq[p] = seq
}
}
}
}
h.floor = max(h.floor, seq)
}
func (h *PrivilegeHandler) Setup() []goamqp.Setup {
return []goamqp.Setup{
goamqp.TransientEventStreamConsumer("User.Added", process[UserAdded](h)),
goamqp.TransientEventStreamConsumer("User.Removed", process[UserRemoved](h)),
goamqp.TransientEventStreamConsumer("Privilege.Added", process[PrivilegeAdded](h)),
goamqp.TransientEventStreamConsumer("Privilege.Removed", process[PrivilegeRemoved](h)),
}
}
// privilegeEvent is the set of events Process handles.
type privilegeEvent interface {
UserAdded | UserRemoved | PrivilegeAdded | PrivilegeRemoved
}
// process adapts Process to a typed go-messaging-amqp handler.
func process[T privilegeEvent](h *PrivilegeHandler) spec.EventHandler[T] {
return func(_ context.Context, event spec.ConsumableEvent[T]) error {
return h.Process(&event.Payload)
}
}
// Process privilege-related events and update the internal state.
//
// Events are applied by sequence number, not by arrival order. An event without a
// sequence number can't be ordered, so it fails closed: a removal is applied and an
// addition is dropped.
func (h *PrivilegeHandler) Process(msg any) error {
h.Lock()
defer h.Unlock()
switch ev := msg.(type) {
case *UserAdded:
if h.stale(ev.SequenceNo, true, ev) {
return nil
}
s := h.company(ev.Email, ev.CompanyID)
if ev.SequenceNo > s.memberSeq {
s.member = true
s.memberSeq = ev.SequenceNo
}
return nil
case *UserRemoved:
if h.stale(ev.SequenceNo, false, ev) {
return nil
}
removeUser(h.company(ev.Email, ev.CompanyID), ev.SequenceNo, ev.SequenceNo == 0)
return nil
case *PrivilegeAdded:
if h.stale(ev.SequenceNo, true, ev) {
return nil
}
h.setPrivilege(ev.Email, ev.CompanyID, ev.Privilege, ev.SequenceNo, true)
return nil
case *PrivilegeRemoved:
if h.stale(ev.SequenceNo, false, ev) {
return nil
}
h.setPrivilege(ev.Email, ev.CompanyID, ev.Privilege, ev.SequenceNo, false)
return nil
default:
slog.Error("unexpected privilege message type", "type", reflect.TypeOf(msg).String())
return fmt.Errorf("unexpected event type: '%s'", reflect.TypeOf(msg))
}
}
// stale reports whether an event must be skipped: an addition without a sequence
// number, or any event already covered by the latest snapshot.
func (h *PrivilegeHandler) stale(seq int, addition bool, ev any) bool {
if seq < 0 || seq >= pending {
slog.Error("dropping privilege event with invalid sequence number", "type", reflect.TypeOf(ev).String(), "seq", seq)
return true
}
if seq == 0 {
if addition {
slog.Warn("dropping privilege addition without sequence number", "type", reflect.TypeOf(ev).String())
}
return addition
}
return seq <= h.floor
}
func (h *PrivilegeHandler) company(email, companyID string) *companyState {
companies, exists := h.state[email]
if !exists {
companies = map[string]*companyState{}
h.state[email] = companies
}
s, exists := companies[companyID]
if !exists {
s = newCompanyState()
companies[companyID] = s
}
return s
}
// removeUser removes the user at seq, overriding every fact older than seq. An
// unordered removal (all) overrides every fact and marks them pending, so no event can
// bring them back before the next snapshot.
func removeUser(s *companyState, seq int, all bool) {
stamp := seq
if all {
stamp = pending
}
if all || s.memberSeq < seq {
s.member = false
s.memberSeq = max(s.memberSeq, stamp)
}
for _, p := range AllPrivilege {
if all || s.privilegeSeq[p] < seq {
*s.privileges.field(p) = false
s.privilegeSeq[p] = max(s.privilegeSeq[p], stamp)
}
}
}
func (h *PrivilegeHandler) setPrivilege(email, companyID string, privilege Privilege, seq int, set bool) {
if !privilege.IsValid() {
return
}
s := h.company(email, companyID)
if seq != 0 && seq <= s.privilegeSeq[privilege] {
return
}
*s.privileges.field(privilege) = set
if seq == 0 {
// Only an unordered removal gets here; keep it until the next snapshot.
seq = pending
}
s.privilegeSeq[privilege] = max(s.privilegeSeq[privilege], seq)
// authz-service's aggregate adds the user when a privilege is granted.
if set && seq > s.memberSeq {
s.member = true
s.memberSeq = seq
}
}
// CompaniesByUser return a slice of company ids matching the provided email and predicate func
func (h *PrivilegeHandler) CompaniesByUser(email string, predicate func(privileges CompanyPrivileges) bool) []string {
h.RLock()
defer h.RUnlock()
result := []string{}
for k, s := range h.state[email] {
if s.member && predicate(s.privileges) {
result = append(result, k)
}
}
return result
}
// IsAllowed return true if the provided predicate return true for the privileges matching the provided email and companyID, return false otherwise
func (h *PrivilegeHandler) IsAllowed(email, companyID string, predicate func(privileges CompanyPrivileges) bool) bool {
h.RLock()
defer h.RUnlock()
if s, exists := h.state[email][companyID]; exists && s.member {
return predicate(s.privileges)
}
return false
}