Lumina-MSK/workspace/sprint_1_2/CODEBASE/docs/ml-inference-cache.md

ML inference cache & thundering-herd control

PoC design for reusing segmentation / angle / inflammation results across page refresh, multiple browser tabs, and duplicate UI triggers — without re-calling Modal Triton for the same profile.

Problem

Trigger	Before cache
Open clinical workspace (3 frames)	2 proxy HTTP + ~9 Triton calls
Refresh page	Full repeat (in-memory cache lost)
Second tab, same patient	Full repeat (separate JS heap)
React Strict Mode double-mount	Partially deduped in one tab only
Identical batch POST while first in flight	Proxy runs work again (queue only)

Cache identity (invariants)

A cached result is valid when all of these match:

patientMrn     — e.g. BN-2024-1847 (encounter / patient scope)
frameId        — e.g. med-lat-1
contentHash    — SHA-256 of frame image bytes (invalidates on re-scan)
pipelineVersion — model + postprocess fingerprint (invalidates on deploy)

Cache key string:

{patientMrn}|{frameId}|{contentHash}|{pipelineVersion}

Batch coordination key (cross-tab single-flight):

{patientMrn}|{pipelineVersion}|seg:{angleType}:{sortedFrameIds}
{patientMrn}|{pipelineVersion}|angle:{sortedFrameIds}

CV pipeline (spec §7)

Frontend calls POST /api/test/analyze/batch — one orchestrated path per frame:

CLAHE → angle classification → (post-trans | sup-up-long only) inflammation
  → if inflammation: segmentation + thickness + severity + overlay
  → else: severity level 0, segmentation skipped
  → other angles: angle + severity 0 only

Pipeline version: poc-v2-spec-cv (ML_INFERENCE_PIPELINE_VERSION / PROXY_PIPELINE_VERSION).

Legacy split endpoints (/segment/batch, /angle/batch) remain for debugging but bypass spec gating.

Two layers

Layer 1 — IndexedDB (persist + refresh)

• Database: lumina-msk-ml / store frame-inference-cache
• Survives refresh and is shared across tabs on the same origin
• TTL: 24 hours (cachedAt); stale entries ignored
• Stores per frame:
  • segmentation: overlaySrc, raw backend payload subset
  • angle: AngleClassificationResult
  • inflammation is embedded in segmentation raw.inflammation

Layer 2 — BroadcastChannel (thundering herd)

• Channel: lumina-ml-inflight
• Messages: fetch-start / fetch-done / fetch-error with batchKey
• Tab that starts a network batch announces fetch-start
• Other tabs await fetch-done for the same batchKey, then read IndexedDB
• Fixes race where two cold tabs both miss IDB before either writes

In-tab inflightBatchByKey Map remains as a third line of defense.

Frontend call flow

getSegmentationResultsForProfile(frames, { patientMrn })
  1. For each frame: read IDB by cache key → hydrate memory cache
  2. If all frames hit → return (0 HTTP)
  3. batchKey = profile batch key
  4. await crossTab.waitIfInflight(batchKey)  // another tab may have finished
  5. Re-check IDB + memory
  6. in-flight Map single-flight → fetchSegmentationBatchForProfile
  7. Write IDB per frame; crossTab.notifyDone(batchKey)

Angle batch follows the same pattern via getAngleClassificationResultsForProfile.

useSegmentationOverlay passes patientMrn from ClinicalWorkspacePage → DiagnosticCanvas.

Proxy server cache (test_fast_api_proxy.py)

In-memory (process lifetime):

Map	Purpose
_proxy_result_cache	cache_key → (expires_monotonic, json-serializable result)
_proxy_inflight	cache_key → asyncio.Future — coalesce identical batch requests

Segment batch key:

segment|{angle_type}|{pipeline_version}|{sorted(frame_id:sha256(image_bytes))}

Angle batch key:

angle|{pipeline_version}|{sorted(frame_id:sha256(image_bytes))}

TTL default: 1 hour (PROXY_RESULT_CACHE_TTL_S env).

On cache hit: return cached JSON immediately (0 Triton). On concurrent miss: later request awaits the first request's Future.

Pipeline version

Frontend: ML_INFERENCE_PIPELINE_VERSION in mlInferencePipelineVersion.ts
Proxy: PROXY_PIPELINE_VERSION env (default poc-v1)

Bump both when Triton model names or overlay/postprocess logic change.

Debugging

Symptom	Check
Still 9 Triton calls every refresh	IDB miss — DevTools → Application → IndexedDB → lumina-msk-ml
Two tabs both fetch	BroadcastChannel not supported or different patientMrn / pipelineVersion
Stale overlay after model update	Bump ML_INFERENCE_PIPELINE_VERSION
Proxy logs "cache hit"	Working — no Triton for that batch
502 herd	Separate issue — lock + retries; cache reduces frequency

Files

File	Role
docs/ml-inference-cache.md	This document
frontend/.../mlInferencePipelineVersion.ts	Version constant
frontend/.../mlInferenceCacheStore.ts	IndexedDB read/write
frontend/.../mlInferenceCrossTab.ts	BroadcastChannel coordinator
frontend/.../cvAnalyzeApi.ts	Spec CV pipeline — single /api/test/analyze/batch call
frontend/.../angleClassificationApi.ts	IDB + cross-tab integration
backend/tests/test_fast_api_proxy.py	Server result cache + in-flight dedupe
frontend/.../mlServiceError.ts	Classify raw errors → ticket-friendly MlServiceError
frontend/.../MlServiceErrorPanel.tsx	Canvas overlay UI (support ref, copy, retry)

ML error UX (ticket-friendly)

When segmentation fails, the canvas shows MlServiceErrorPanel instead of raw browser messages (e.g. Safari’s “The string did not match the expected pattern”).

Audience	What they see
Clinician	Reassuring title + plain Vietnamese explanation; ultrasound image unchanged
IT / ticket	Mã tham chiếu (LUM-ML-…) + Sao chép cho ticket button
Developer	Collapsed Chi tiết kỹ thuật — operation, frame id, raw message, remediation hint

Error codes: BAD_RESPONSE (proxy down / HTML not JSON), NETWORK, SERVER_OVERLOAD (502/503/Triton), SERVER_UNAVAILABLE, CLIENT_ERROR, NO_RESULT, UNKNOWN.

Debugging: Ask user for support reference → match browser console Network tab (/api/test/segment/batch) with proxy logs. Common root cause for BAD_RESPONSE: test_fast_api_proxy.py not running on port 8001.

Worklist AI grade

Home-screen Đề xuất AI never uses MOCK_PATIENTS.synovitisGrade.

State	Card shows
No cached inference	Chưa phân tích
Reading IndexedDB	Đang tải…
After workspace ML run	Độ X — max severity.level across cached profile frames for that patientMrn

Grades refresh when the worklist tab becomes visible again (return from clinical workspace).