checkout-latency-cascade

completed

run-d4f359e3 · LuisRz1/battlegraf@main · mock mode

Started Sep 14, 02:23 AM UTC

Finished Sep 14, 02:27 AM UTC

Time to collapse

57.0s

collapse reproduced under chaos

p95 latency

4,632 ms → 268 ms

before → after verified fix

Error rate

38.4% → 0.5%

5xx / timeouts before → after

Diagnosis time

3.1 min

vs ~95 min manual SRE

Latency under identical chaos load

p95 latency (ms)

Error rate under identical chaos load

error rate (%)

Execution timeline

Chaos injection + collapse capture
k6 load at 200 vus against http://localhost:8080/api/v1/cart, latency fault on payment-service, collapse at 57.0s
chaos · done
Root cause classification
unresilient_dependency (confidence 0.93) — src/checkoutservice/main.go PlaceOrder
classification · done
Repository-aware diagnosis
rule-based-fallback analyzed LuisRz1/battlegraf with full repository context
diagnosis · done
PR generation + resilience verification
same chaos scenario re-executed on patched branch cdr/fix-checkout-latency-cascade — stable, p95 268.4 ms
verification · done

Root cause finding

Unresilient Dependency

Confidence 93% · src/checkoutservice/main.go PlaceOrder

The checkout path calls downstream services without a context deadline, retry budget or circuit breaker. Injected latency makes every request hold a worker until the transport timeout fires, so pools saturate and the checkout path collapses end to end.

p95 on checkout.path rose to 4632 ms under injected chaos
error rate peaked at 38.4% with 5xx responses
context deadline exceeded calling payment-service
goroutines blocked in grpc.Invoke waiting for payment-service
no deadline or circuit breaker found on the payment-service client during repository scan
collapse detected at t=57.0s under the same load profile

Resilience verification

STABLE AFTER FIX

p95

−94.2%

p99

−94.2%

collapse

−100.0%

error rate

−98.8%

Repository-aware diagnosis

rule-based-fallback

Analysis

The failing path fans out to downstream services over gRPC. The dependency leg uses a shared client with no per-call deadline and no failure isolation. Under injected latency every request holds a worker until the transport timeout fires, so throughput collapses long before errors surface.

Proposed refactor

1. Apply a 400 ms context deadline to the dependency call.

2. Add a bounded retry with jitter only for idempotent operations.

3. Wrap the client with a circuit breaker that opens after consecutive failures and half-opens after 5 s.

4. Reuse the resilience pattern already present in the shipping service for consistency with the repository.

Proposed change: Add context deadline, bounded retry with jitter and a circuit breaker around the dependency call.

Generated patch

cdr/fix-checkout-latency-cascade

src/checkoutservice/main.go

Template patch (Bob Shell unavailable)

# Proposed change: Add context deadline, bounded retry with jitter and a circuit breaker around the dependency call.
--- a/src/checkoutservice/main.go
+++ b/src/checkoutservice/main.go
@@ -184,9 +184,16 @@ func (cs *checkoutService) PlaceOrder(ctx context.Context, req *pb.PlaceOrderReq) (*pb.PlaceOrderResp, error) {
-	resp, err := cs.paymentSvc.Charge(ctx, &pb.ChargeRequest{
+	payCtx, cancel := context.WithTimeout(ctx, 400*time.Millisecond)
+	defer cancel()
+
+	resp, err := cs.paymentBreaker.Execute(func() (any, error) {
+		return cs.paymentSvc.Charge(payCtx, &pb.ChargeRequest{
 			Amount: &pb.Money{CurrencyCode: req.GetCurrencyCode(), Units: total.Units},
 			CreditCard: req.GetCreditCard(),
-	})
+		})
+	})
 	if err != nil {
-		return nil, status.Errorf(codes.Internal, "payment failed: %v", err)
+		return nil, status.Errorf(codes.Unavailable, "payment unavailable: %v", err)
 	}