checkout-latency-cascade
completedrun-1228bd92 · harrycas/localArtisans-backend@main · mock mode
Started Sep 14, 03:19 AM UTC
Finished Sep 14, 03:32 AM UTC
Time to collapse
57.0s
collapse reproduced under chaos
p95 latency
4,632 ms → 268 ms
before → after verified fix
Error rate
38.4% → 0.5%
5xx / timeouts before → after
Diagnosis time
3.1 min
vs ~95 min manual SRE
Latency under identical chaos load
p95 latency (ms)Error rate under identical chaos load
error rate (%)Execution timeline
Chaos injection + collapse capture
k6 load at 200 vus against http://localhost:8080/api/v1/cart, latency fault on payment-service, collapse at 57.0s
chaos · done
Root cause classification
unresilient_dependency (confidence 0.93) — src/checkoutservice/main.go PlaceOrder
classification · done
Repository-aware diagnosis
bob-shell analyzed harrycas/localArtisans-backend with full repository context
diagnosis · done
PR generation + resilience verification
same chaos scenario re-executed on patched branch cdr/fix-checkout-latency-cascade — stable, p95 268.4 ms
verification · done
Root cause finding
Unresilient DependencyConfidence 93% · src/checkoutservice/main.go PlaceOrder
The checkout path calls downstream services without a context deadline, retry budget or circuit breaker. Injected latency makes every request hold a worker until the transport timeout fires, so pools saturate and the checkout path collapses end to end.
- p95 on checkout.path rose to 4632 ms under injected chaos
- error rate peaked at 38.4% with 5xx responses
- context deadline exceeded calling payment-service
- goroutines blocked in grpc.Invoke waiting for payment-service
- no deadline or circuit breaker found on the payment-service client during repository scan
- collapse detected at t=57.0s under the same load profile
Resilience verification
STABLE AFTER FIXp95
−94.2%
p99
−94.2%
collapse
−100.0%
error rate
−98.8%
Repository-aware diagnosis
The plan is written to [`rca-refactoring-plan.md`](rca-refactoring-plan.md). Here's a summary of everything found and structured:
---
What Was Found
The repo is a **Java/Spring Boot microservices backend** (not Node.js) with two services: `svc-artisans` and `msvc-catalog`. Every finding below is grounded in actual file reads.
### Architecture
```mermaid
graph LR
Client --> UA[UserController :8080]
Client --> OA[OrderController :8080]
Client --> RA[ReviewController :8080]
Client --> PC[ProductController :8082]
Client --> CC[CategoryController :8082]
UA & OA & RA --> DB1[(PostgreSQL: artisans)]
PC & CC --> DB2[(PostgreSQL: artisansCatalog)]
UA -->|Feign| PC
```
---
7 Sub-Tasks in the Plan
| # | Sub-Task | Severity | Verified At |
|---|---|---|---|
| 1 | **Wrong response objects** — 3 endpoints return the input object instead of the persisted result (missing generated IDs) | 🔴 High | [`ReviewController.java:24,40`](svc-artisans/src/main/java/com/arquiproject/svc_artisans/controller/ReviewController.java), [`CategoryController.java:33`](msvc-catalog/src/main/java/com/arquiproject/msvc_catalog/controller/CategoryController.java) |
| 2 | **Delete ordering bug** — user is deleted from DB *before* Feign call to deactivate products; failure leaves orphaned active products | 🔴 High | [`UserService.java:62-63`](svc-artisans/src/main/java/com/arquiproject/svc_artisans/service/UserService.java) |
| 3 | **`System.out.println`** throughout all services — replace with SLF4J `log.error` | 🟡 Medium | 3 service files |
| 4 | **Plaintext password in `LoginResponse`** — the login endpoint returns the user's raw password in the JSON response | 🔴 Critical | [`LoginResponse.java`](svc-artisans/src/main/java/com/arquiproject/svc_artisans/views/LoginResponse.java), [`UserService.java:75`](svc-artisans/src/main/java/com/arquiproject/svc_artisans/service/UserService.java) |
| 5 | **Wrong HTTP 406 status** on create failures — semantically incorrect; should be 400 | 🟡 Medium | All controllers |
| 6 | **No global exception handler** — unhandled exceptions expose stack traces in responses | 🟡 Medium | Both services |
| 7 | **Hardcoded DB credentials** as property defaults — real usernames/passwords committed to source | 🔴 Critical | Both `application.properties` files |
---
Questions Before Implementation
1. **Sub-Task 4 (password exposure):** Should `LoginResponse` be extended to include a `userId` or `userType` field for the frontend, now that the password is removed — or should it only return email?
2. **Sub-Task 2 (delete ordering):** If `msvc-catalog` is unreachable and the delete is blocked, should the API return `503 Service Unavailable` or `500 Internal Server Error`?
3. **Sub-Task 6 (exception handler):** Is a simple catch-all for `500` sufficient, or should `DataIntegrityViolationException` (duplicate email on user create) also be surfaced as a distinct `409 CONFLICT`?
4. **Scope confirmation:** The plan intentionally excludes JWT auth, input validation, pagination, circuit breakers, and test coverage. Are any of those in scope for this refactoring pass?
Proposed change: Add context deadline, bounded retry with jitter and a circuit breaker around the dependency call.
Generated patch
cdr/fix-checkout-latency-cascadesrc/checkoutservice/main.go
Template patch (Bob Shell unavailable)
# Proposed change: Add context deadline, bounded retry with jitter and a circuit breaker around the dependency call.--- a/src/checkoutservice/main.go+++ b/src/checkoutservice/main.go@@ -184,9 +184,16 @@ func (cs *checkoutService) PlaceOrder(ctx context.Context, req *pb.PlaceOrderReq) (*pb.PlaceOrderResp, error) {- resp, err := cs.paymentSvc.Charge(ctx, &pb.ChargeRequest{+ payCtx, cancel := context.WithTimeout(ctx, 400*time.Millisecond)+ defer cancel()++ resp, err := cs.paymentBreaker.Execute(func() (any, error) {+ return cs.paymentSvc.Charge(payCtx, &pb.ChargeRequest{Amount: &pb.Money{CurrencyCode: req.GetCurrencyCode(), Units: total.Units},CreditCard: req.GetCreditCard(),- })+ })+ })if err != nil {- return nil, status.Errorf(codes.Internal, "payment failed: %v", err)+ return nil, status.Errorf(codes.Unavailable, "payment unavailable: %v", err)}