update the codebase poc ver1
This commit is contained in:
DatTT127
174
workspace/sprint_1_2/CODEBASE/docs/ml-inference-cache.md
Normal file
174
workspace/sprint_1_2/CODEBASE/docs/ml-inference-cache.md
Normal file
@@ -0,0 +1,174 @@
|
||||
# ML inference cache & thundering-herd control
|
||||
|
||||
PoC design for reusing segmentation / angle / inflammation results across **page refresh**, **multiple browser tabs**, and **duplicate UI triggers** — without re-calling Modal Triton for the same profile.
|
||||
|
||||
## Problem
|
||||
|
||||
| Trigger | Before cache |
|
||||
|---------|----------------|
|
||||
| Open clinical workspace (3 frames) | 2 proxy HTTP + ~9 Triton calls |
|
||||
| Refresh page | Full repeat (in-memory cache lost) |
|
||||
| Second tab, same patient | Full repeat (separate JS heap) |
|
||||
| React Strict Mode double-mount | Partially deduped in one tab only |
|
||||
| Identical batch POST while first in flight | Proxy runs work again (queue only) |
|
||||
|
||||
## Cache identity (invariants)
|
||||
|
||||
A cached result is valid when all of these match:
|
||||
|
||||
```
|
||||
patientMrn — e.g. BN-2024-1847 (encounter / patient scope)
|
||||
frameId — e.g. med-lat-1
|
||||
contentHash — SHA-256 of frame image bytes (invalidates on re-scan)
|
||||
pipelineVersion — model + postprocess fingerprint (invalidates on deploy)
|
||||
```
|
||||
|
||||
**Cache key string:**
|
||||
|
||||
```
|
||||
{patientMrn}|{frameId}|{contentHash}|{pipelineVersion}
|
||||
```
|
||||
|
||||
**Batch coordination key** (cross-tab single-flight):
|
||||
|
||||
```
|
||||
{patientMrn}|{pipelineVersion}|seg:{angleType}:{sortedFrameIds}
|
||||
{patientMrn}|{pipelineVersion}|angle:{sortedFrameIds}
|
||||
```
|
||||
|
||||
## CV pipeline (spec §7)
|
||||
|
||||
Frontend calls **`POST /api/test/analyze/batch`** — one orchestrated path per frame:
|
||||
|
||||
```
|
||||
CLAHE → angle classification → (post-trans | sup-up-long only) inflammation
|
||||
→ if inflammation: segmentation + thickness + severity + overlay
|
||||
→ else: severity level 0, segmentation skipped
|
||||
→ other angles: angle + severity 0 only
|
||||
```
|
||||
|
||||
Pipeline version: **`poc-v2-spec-cv`** (`ML_INFERENCE_PIPELINE_VERSION` / `PROXY_PIPELINE_VERSION`).
|
||||
|
||||
Legacy split endpoints (`/segment/batch`, `/angle/batch`) remain for debugging but bypass spec gating.
|
||||
|
||||
## Two layers
|
||||
|
||||
### Layer 1 — IndexedDB (persist + refresh)
|
||||
|
||||
- Database: `lumina-msk-ml` / store `frame-inference-cache`
|
||||
- Survives refresh and is **shared across tabs** on the same origin
|
||||
- TTL: 24 hours (`cachedAt`); stale entries ignored
|
||||
- Stores per frame:
|
||||
- segmentation: `overlaySrc`, `raw` backend payload subset
|
||||
- angle: `AngleClassificationResult`
|
||||
- inflammation is embedded in segmentation `raw.inflammation`
|
||||
|
||||
### Layer 2 — BroadcastChannel (thundering herd)
|
||||
|
||||
- Channel: `lumina-ml-inflight`
|
||||
- Messages: `fetch-start` / `fetch-done` / `fetch-error` with `batchKey`
|
||||
- Tab that starts a network batch announces `fetch-start`
|
||||
- Other tabs **await** `fetch-done` for the same `batchKey`, then read IndexedDB
|
||||
- Fixes race where two cold tabs both miss IDB before either writes
|
||||
|
||||
In-tab `inflightBatchByKey` Map remains as a third line of defense.
|
||||
|
||||
## Frontend call flow
|
||||
|
||||
```
|
||||
getSegmentationResultsForProfile(frames, { patientMrn })
|
||||
1. For each frame: read IDB by cache key → hydrate memory cache
|
||||
2. If all frames hit → return (0 HTTP)
|
||||
3. batchKey = profile batch key
|
||||
4. await crossTab.waitIfInflight(batchKey) // another tab may have finished
|
||||
5. Re-check IDB + memory
|
||||
6. in-flight Map single-flight → fetchSegmentationBatchForProfile
|
||||
7. Write IDB per frame; crossTab.notifyDone(batchKey)
|
||||
```
|
||||
|
||||
Angle batch follows the same pattern via `getAngleClassificationResultsForProfile`.
|
||||
|
||||
`useSegmentationOverlay` passes `patientMrn` from `ClinicalWorkspacePage` → `DiagnosticCanvas`.
|
||||
|
||||
## Proxy server cache (`test_fast_api_proxy.py`)
|
||||
|
||||
In-memory (process lifetime):
|
||||
|
||||
| Map | Purpose |
|
||||
|-----|---------|
|
||||
| `_proxy_result_cache` | `cache_key → (expires_monotonic, json-serializable result)` |
|
||||
| `_proxy_inflight` | `cache_key → asyncio.Future` — coalesce identical batch requests |
|
||||
|
||||
**Segment batch key:**
|
||||
|
||||
```
|
||||
segment|{angle_type}|{pipeline_version}|{sorted(frame_id:sha256(image_bytes))}
|
||||
```
|
||||
|
||||
**Angle batch key:**
|
||||
|
||||
```
|
||||
angle|{pipeline_version}|{sorted(frame_id:sha256(image_bytes))}
|
||||
```
|
||||
|
||||
TTL default: 1 hour (`PROXY_RESULT_CACHE_TTL_S` env).
|
||||
|
||||
On cache hit: return cached JSON immediately (0 Triton).
|
||||
On concurrent miss: later request awaits the first request's Future.
|
||||
|
||||
## Pipeline version
|
||||
|
||||
Frontend: `ML_INFERENCE_PIPELINE_VERSION` in `mlInferencePipelineVersion.ts`
|
||||
Proxy: `PROXY_PIPELINE_VERSION` env (default `poc-v1`)
|
||||
|
||||
**Bump both** when Triton model names or overlay/postprocess logic change.
|
||||
|
||||
## Debugging
|
||||
|
||||
| Symptom | Check |
|
||||
|---------|--------|
|
||||
| Still 9 Triton calls every refresh | IDB miss — DevTools → Application → IndexedDB → `lumina-msk-ml` |
|
||||
| Two tabs both fetch | BroadcastChannel not supported or different `patientMrn` / `pipelineVersion` |
|
||||
| Stale overlay after model update | Bump `ML_INFERENCE_PIPELINE_VERSION` |
|
||||
| Proxy logs "cache hit" | Working — no Triton for that batch |
|
||||
| 502 herd | Separate issue — lock + retries; cache reduces frequency |
|
||||
|
||||
## Files
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `docs/ml-inference-cache.md` | This document |
|
||||
| `frontend/.../mlInferencePipelineVersion.ts` | Version constant |
|
||||
| `frontend/.../mlInferenceCacheStore.ts` | IndexedDB read/write |
|
||||
| `frontend/.../mlInferenceCrossTab.ts` | BroadcastChannel coordinator |
|
||||
| `frontend/.../cvAnalyzeApi.ts` | Spec CV pipeline — single `/api/test/analyze/batch` call |
|
||||
| `frontend/.../angleClassificationApi.ts` | IDB + cross-tab integration |
|
||||
| `backend/tests/test_fast_api_proxy.py` | Server result cache + in-flight dedupe |
|
||||
| `frontend/.../mlServiceError.ts` | Classify raw errors → ticket-friendly `MlServiceError` |
|
||||
| `frontend/.../MlServiceErrorPanel.tsx` | Canvas overlay UI (support ref, copy, retry) |
|
||||
|
||||
## ML error UX (ticket-friendly)
|
||||
|
||||
When segmentation fails, the canvas shows **`MlServiceErrorPanel`** instead of raw browser messages (e.g. Safari’s *“The string did not match the expected pattern”*).
|
||||
|
||||
| Audience | What they see |
|
||||
|----------|----------------|
|
||||
| Clinician | Reassuring title + plain Vietnamese explanation; ultrasound image unchanged |
|
||||
| IT / ticket | **Mã tham chiếu** (`LUM-ML-…`) + **Sao chép cho ticket** button |
|
||||
| Developer | Collapsed **Chi tiết kỹ thuật** — operation, frame id, raw message, remediation hint |
|
||||
|
||||
**Error codes:** `BAD_RESPONSE` (proxy down / HTML not JSON), `NETWORK`, `SERVER_OVERLOAD` (502/503/Triton), `SERVER_UNAVAILABLE`, `CLIENT_ERROR`, `NO_RESULT`, `UNKNOWN`.
|
||||
|
||||
**Debugging:** Ask user for support reference → match browser console Network tab (`/api/test/segment/batch`) with proxy logs. Common root cause for `BAD_RESPONSE`: `test_fast_api_proxy.py` not running on port 8001.
|
||||
|
||||
## Worklist AI grade
|
||||
|
||||
Home-screen **Đề xuất AI** never uses `MOCK_PATIENTS.synovitisGrade`.
|
||||
|
||||
| State | Card shows |
|
||||
|-------|------------|
|
||||
| No cached inference | `Chưa phân tích` |
|
||||
| Reading IndexedDB | `Đang tải…` |
|
||||
| After workspace ML run | `Độ X` — max `severity.level` across cached profile frames for that `patientMrn` |
|
||||
|
||||
Grades refresh when the worklist tab becomes visible again (return from clinical workspace).
|
||||
Reference in New Issue
Block a user