# ML inference cache & thundering-herd control

PoC design for reusing segmentation / angle / inflammation results across **page refresh**, **multiple browser tabs**, and **duplicate UI triggers** — without re-calling Modal Triton for the same profile.

## Problem

| Trigger | Before cache |
|---------|----------------|
| Open clinical workspace (3 frames) | 2 proxy HTTP + ~9 Triton calls |
| Refresh page | Full repeat (in-memory cache lost) |
| Second tab, same patient | Full repeat (separate JS heap) |
| React Strict Mode double-mount | Partially deduped in one tab only |
| Identical batch POST while first in flight | Proxy runs work again (queue only) |

## Cache identity (invariants)

A cached result is valid when all of these match:

```
patientMrn     — e.g. BN-2024-1847 (encounter / patient scope)
frameId        — e.g. med-lat-1
contentHash    — SHA-256 of frame image bytes (invalidates on re-scan)
pipelineVersion — model + postprocess fingerprint (invalidates on deploy)
```

**Cache key string:**

```
{patientMrn}|{frameId}|{contentHash}|{pipelineVersion}
```

**Batch coordination key** (cross-tab single-flight):

```
{patientMrn}|{pipelineVersion}|seg:{angleType}:{sortedFrameIds}
{patientMrn}|{pipelineVersion}|angle:{sortedFrameIds}
```

## CV pipeline (spec §7)

Frontend calls **`POST /api/test/analyze/batch`** — one orchestrated path per frame:

```
CLAHE → angle classification → (post-trans | sup-up-long only) inflammation
  → if inflammation: segmentation + thickness + severity + overlay
  → else: severity level 0, segmentation skipped
  → other angles: angle + severity 0 only
```

Pipeline version: **`poc-v2-spec-cv`** (`ML_INFERENCE_PIPELINE_VERSION` / `PROXY_PIPELINE_VERSION`).

Legacy split endpoints (`/segment/batch`, `/angle/batch`) remain for debugging but bypass spec gating.

## Two layers

### Layer 1 — IndexedDB (persist + refresh)

- Database: `lumina-msk-ml` / store `frame-inference-cache`
- Survives refresh and is **shared across tabs** on the same origin
- TTL: 24 hours (`cachedAt`); stale entries ignored
- Stores per frame:
  - segmentation: `overlaySrc`, `raw` backend payload subset
  - angle: `AngleClassificationResult`
  - inflammation is embedded in segmentation `raw.inflammation`

### Layer 2 — BroadcastChannel (thundering herd)

- Channel: `lumina-ml-inflight`
- Messages: `fetch-start` / `fetch-done` / `fetch-error` with `batchKey`
- Tab that starts a network batch announces `fetch-start`
- Other tabs **await** `fetch-done` for the same `batchKey`, then read IndexedDB
- Fixes race where two cold tabs both miss IDB before either writes

In-tab `inflightBatchByKey` Map remains as a third line of defense.

## Frontend call flow

```
getSegmentationResultsForProfile(frames, { patientMrn })
  1. For each frame: read IDB by cache key → hydrate memory cache
  2. If all frames hit → return (0 HTTP)
  3. batchKey = profile batch key
  4. await crossTab.waitIfInflight(batchKey)  // another tab may have finished
  5. Re-check IDB + memory
  6. in-flight Map single-flight → fetchSegmentationBatchForProfile
  7. Write IDB per frame; crossTab.notifyDone(batchKey)
```

Angle batch follows the same pattern via `getAngleClassificationResultsForProfile`.

`useSegmentationOverlay` passes `patientMrn` from `ClinicalWorkspacePage` → `DiagnosticCanvas`.

## Proxy server cache (`test_fast_api_proxy.py`)

In-memory (process lifetime):

| Map | Purpose |
|-----|---------|
| `_proxy_result_cache` | `cache_key → (expires_monotonic, json-serializable result)` |
| `_proxy_inflight` | `cache_key → asyncio.Future` — coalesce identical batch requests |

**Segment batch key:**

```
segment|{angle_type}|{pipeline_version}|{sorted(frame_id:sha256(image_bytes))}
```

**Angle batch key:**

```
angle|{pipeline_version}|{sorted(frame_id:sha256(image_bytes))}
```

TTL default: 1 hour (`PROXY_RESULT_CACHE_TTL_S` env).

On cache hit: return cached JSON immediately (0 Triton).
On concurrent miss: later request awaits the first request's Future.

## Pipeline version

Frontend: `ML_INFERENCE_PIPELINE_VERSION` in `mlInferencePipelineVersion.ts`  
Proxy: `PROXY_PIPELINE_VERSION` env (default `poc-v1`)

**Bump both** when Triton model names or overlay/postprocess logic change.

## Debugging

| Symptom | Check |
|---------|--------|
| Still 9 Triton calls every refresh | IDB miss — DevTools → Application → IndexedDB → `lumina-msk-ml` |
| Two tabs both fetch | BroadcastChannel not supported or different `patientMrn` / `pipelineVersion` |
| Stale overlay after model update | Bump `ML_INFERENCE_PIPELINE_VERSION` |
| Proxy logs "cache hit" | Working — no Triton for that batch |
| 502 herd | Separate issue — lock + retries; cache reduces frequency |

## Files

| File | Role |
|------|------|
| `docs/ml-inference-cache.md` | This document |
| `frontend/.../mlInferencePipelineVersion.ts` | Version constant |
| `frontend/.../mlInferenceCacheStore.ts` | IndexedDB read/write |
| `frontend/.../mlInferenceCrossTab.ts` | BroadcastChannel coordinator |
| `frontend/.../cvAnalyzeApi.ts` | Spec CV pipeline — single `/api/test/analyze/batch` call |
| `frontend/.../angleClassificationApi.ts` | IDB + cross-tab integration |
| `backend/tests/test_fast_api_proxy.py` | Server result cache + in-flight dedupe |
| `frontend/.../mlServiceError.ts` | Classify raw errors → ticket-friendly `MlServiceError` |
| `frontend/.../MlServiceErrorPanel.tsx` | Canvas overlay UI (support ref, copy, retry) |

## ML error UX (ticket-friendly)

When segmentation fails, the canvas shows **`MlServiceErrorPanel`** instead of raw browser messages (e.g. Safari’s *“The string did not match the expected pattern”*).

| Audience | What they see |
|----------|----------------|
| Clinician | Reassuring title + plain Vietnamese explanation; ultrasound image unchanged |
| IT / ticket | **Mã tham chiếu** (`LUM-ML-…`) + **Sao chép cho ticket** button |
| Developer | Collapsed **Chi tiết kỹ thuật** — operation, frame id, raw message, remediation hint |

**Error codes:** `BAD_RESPONSE` (proxy down / HTML not JSON), `NETWORK`, `SERVER_OVERLOAD` (502/503/Triton), `SERVER_UNAVAILABLE`, `CLIENT_ERROR`, `NO_RESULT`, `UNKNOWN`.

**Debugging:** Ask user for support reference → match browser console Network tab (`/api/test/segment/batch`) with proxy logs. Common root cause for `BAD_RESPONSE`: `test_fast_api_proxy.py` not running on port 8001.

## Worklist AI grade

Home-screen **Đề xuất AI** never uses `MOCK_PATIENTS.synovitisGrade`.

| State | Card shows |
|-------|------------|
| No cached inference | `Chưa phân tích` |
| Reading IndexedDB | `Đang tải…` |
| After workspace ML run | `Độ X` — max `severity.level` across cached profile frames for that `patientMrn` |

Grades refresh when the worklist tab becomes visible again (return from clinical workspace).