update the codebase poc ver1
This commit is contained in:
DatTT127
@@ -73,14 +73,19 @@ Guided by the above requirements and the solution_architect_corpus, the architec
|
||||
### 4.1 High-Level Overview
|
||||
The system adopts a hybrid edge-cloud (on-premise) architecture with the following layers:
|
||||
1. **Edge Inference Layer**: Located on-premise at hospitals/clinics, runs lightweight ML models for real-time DICOM processing (NFR-5,6).
|
||||
2. **Micro-Services Layer**: Containerized services orchestrated by K3s (Kubernetes-certified, lightweight distribution designed for edge and resource-constrained production environments). K3s selected over alternatives (Docker Swarm, Nomad, ECS Fargate, Cloud Run) on the following criteria: NFR-16 requires on-premise deployment — eliminating ECS Fargate and Cloud Run (cloud-only platforms). Docker Swarm offers lowest PoC operational cost but is in maintenance mode with no viable scaling path; migration to production-grade orchestration requires complete rewrite of all deployment artifacts. Nomad is operationally viable but lacks ecosystem depth and has less documented migration path to multi-cluster federation. K3s is already production-grade at single-site scale, requires no migration to "production" — it IS the production platform. The scaling path to N hospitals is multi-cluster federation via K3s cluster-api, not platform replacement. All manifests and service definitions are forward-compatible with upstream Kubernetes.
|
||||
3. **Data Layer**: On-premise databases (PostgreSQL + pgvector for relational + vector search, Redis for caching, MinIO for S3-compatible object storage).
|
||||
3. **Embedding Layer**: EmbeddingGemma (768-dim) for RAG retrieval vectors; separate from BERT which is reserved for drift monitoring and RAG-Referee classification only.
|
||||
3. **Vector Layer (two-tier)**:
|
||||
2. **3-Tier LLM System**:
|
||||
- **Tier 1 (Edge Gemma)**: Browser/WebLLM (GemmaE2B local in WASM) — primary conversation, local inference, instant response, zero network cost. Orchestrates when to call cloud models via FastAPI middleware.
|
||||
- **Tier 2 (Gemini)**: GCP Vertex AI (Gemini) — orchestration, translation, UI planning, formatting. Governed by NFR-16a with redaction, consent, and audit logging.
|
||||
- **Tier 3 (MedGemma)**: Modal-deployed MedGemma (large-24b) — clinical deep-reasoning and report finalization. Governed by NFR-16a with redaction, consent, audit logging, and RAG-Referee validation.
|
||||
- **Tier 4 (Templates)**: MOH guideline template responses — rule-based, deterministic, safety floor when all inference tiers fail.
|
||||
3. **Micro-Services Layer**: Containerized services orchestrated by K3s (Kubernetes-certified, lightweight distribution designed for edge and resource-constrained production environments). K3s selected over alternatives (Docker Swarm, Nomad, ECS Fargate, Cloud Run) on the following criteria: NFR-16 requires on-premise deployment — eliminating ECS Fargate and Cloud Run (cloud-only platforms). Docker Swarm offers lowest PoC operational cost but is in maintenance mode with no viable scaling path; migration to production-grade orchestration requires complete rewrite of all deployment artifacts. Nomad is operationally viable but lacks ecosystem depth and has less documented migration path to multi-cluster federation. K3s is already production-grade at single-site scale, requires no migration to "production" — it IS the production platform. The scaling path to N hospitals is multi-cluster federation via K3s cluster-api, not platform replacement. All manifests and service definitions are forward-compatible with upstream Kubernetes.
|
||||
4. **Data Layer**: On-premise databases (PostgreSQL + pgvector for relational + vector search, Redis for caching, MinIO for S3-compatible object storage).
|
||||
5. **Embedding Layer**: EmbeddingGemma (768-dim) for RAG retrieval vectors; separate from BERT which is reserved for drift monitoring and RAG-Referee classification only.
|
||||
6. **Vector Layer (two-tier)**:
|
||||
- **Hot + Warm: pgvector** (Postgres disk-backed HNSW index, ~15K MOH guideline vectors plus session and case embeddings — serves both real-time RAG during consult and non-real-time analytical queries in a single store. NFR-18, NFR-12. NFR-16 compliant.)
|
||||
- **Cold: S3 Vectors** (AWS ap-southeast-1, billion-scale archival, cross-facility bootstrap source, disaster recovery rebuild origin — accessed under NFR-16a governance.)
|
||||
5. **Presentation Layer**: Progressive Web App (PWA) built with React, Web Workers, and service workers for offline capability and low memory footprint (NFR-4,14).
|
||||
6. **Observability & Security Layer**: Centralized logging (immutable append-only), monitoring, and audit trails (NFR-10,17).
|
||||
7. **Presentation Layer**: Progressive Web App (PWA) built with React, Web Workers, and service workers for offline capability and low memory footprint (NFR-4,14).
|
||||
8. **Observability & Security Layer**: Centralized logging (immutable append-only), monitoring, and audit trails (NFR-10,17).
|
||||
|
||||
### 4.2 Data Flow
|
||||
1. User captures DICOM image via PWA frontend.
|
||||
@@ -104,13 +109,12 @@ The system adopts a hybrid edge-cloud (on-premise) architecture with the followi
|
||||
[3] GCP CDN emergency fallback (asia-southeast1 = Singapore, nearest to Vietnam) — activated ONLY when hospital intranet CDN is confirmed unreachable. GCP backend bucket (`cdn-backend-models-vkist`) wraps a Cloud Storage bucket (`gs://vkist-models-{site_id}`) with Cloud CDN enabled; signed-URL access control (no `allUsers` public exposure); cache mode `CACHE_ALL_STATIC`, default TTL 24h, max TTL 7d (matches model version lifecycle). Service Worker receives a 307 redirect to signed CDN URL on intranet CDN failure; PWA caches the fetched weights locally via Cache Storage as normal. Because model weights contain zero patient data — they are static computational artifacts — the GCP CDN fetch event is committed to the on-prem audit log (case_hash, timestamp, consult_mode = "cdn_fallback") but does NOT trigger the full NFR-16a redaction pipeline. The redaction pipeline applies only to Tier 3a inference calls carrying clinical payloads, not to non-clinical model binary fetches.
|
||||
- **NFR-4 clarification**: NFR-4's 150MB ceiling covers the core application bundle (JS/TS code, UI assets, runtime state) at idle. Supra-150MB memory consumed by models, caches, and inference buffers is bounded separately under the WASM heap cap (1.5GB) and instrumented independently.
|
||||
- **Inference Fallback Chain** (LLM consult path only — CV pipeline is always on-prem Triton):
|
||||
- **Tier 0** → Browser WebLLM (GemmaE2B local in WASM, instant, zero network — preferred default when device has ≥3GB RAM and WebGPU available). Consult mode: `browser_local`. Triggered automatically on page load if model weights are cached and WebGPU is detected.
|
||||
- **Tier 1** → [REMOVED — Triton does NOT host LLM. Triton hosts CV models + EmbeddingGemma only.]
|
||||
- **Tier 1** → Browser WebLLM (Edge GemmaE2B local in WASM, instant, zero network — preferred default when device has ≥3GB RAM and WebGPU available). Consult mode: `tier_1`. Triggered automatically on page load if model weights are cached and WebGPU is detected. Responsible for primary conversation and orchestrating cloud calls.
|
||||
- **Tier 2** → On-prem Triton (hospital GPU node — CV pipeline ONLY: angle → inflammation → segmentation + EmbeddingGemma for RAG embeddings). The LLM consult does NOT fall back to Triton. If browser WebLLM is unavailable, the next tier is cloud.
|
||||
- **Tier 3a** → GCP Vertex AI (MedGemma via REST API — **PoC ONLY under NFR-16a**). Activated when Tier 0 (browser WebLLM) is unavailable (device unsupported, model not cached, memory exceeded) AND user explicitly consents per-session. All payloads MUST pass Decree 13 PII redaction gate (see §5.4) before leaving hospital boundary. GCP project configured with CSEK, 30-day storage lifecycle, 1-year project deletion timer. Consult mode: `cloud_vertex`.
|
||||
- **Tier 3b** → Cloud LLM Arbiter (GCP Vertex AI MedGemma — **PoC ONLY under NFR-16a, BERT-triggered correction**). Activated when BERT drift monitor (UC-74821) detects semantic impasse, logical contradiction, or contextual hallucination during Socratic dialogue, OR when radiologist explicitly requests "second opinion." Same Vertex AI endpoint as Tier 3a, but with arbiter system prompt + retrieved MOH chunks injected. ALL calls require prior audit-log commit of drift signal + user consent + Decree 13 redaction. Consult mode: `cloud_vertex_arbiter`.
|
||||
- **Tier 3c** → MOH guideline template responses (rule-based, no generative model, always available, deterministic, cited — safety floor when all inference tiers fail). Consult mode: `templates`.
|
||||
- **Circuit Breaker**: PWA Feature Flag Manager (`consult_mode` state machine) attempts tiers sequentially: `browser_local` → `cloud_vertex` → `cloud_vertex_arbiter` → `templates`. Emits tier-transition events to on-prem immutable audit log (NFR-17). Tier 3a/3b transitions require consent token + redaction manifest committed *before* the HTTP call. CDN fallback events (model weight distribution only) log lighter audit entry: no redaction required, no consent required (weights carry no PII).
|
||||
- **Tier 2** → GCP Vertex AI (Gemini — **PoC ONLY under NFR-16a**). Activated when Tier 1 (browser WebLLM) is unavailable AND Edge Gemma orchestrates a cloud call for task types: `orchestration`, `translation`, `ui_planning`, `formatting`. All payloads MUST pass Decree 13 PII redaction gate (see §5.4) before leaving hospital boundary. Consult mode: `tier_2`.
|
||||
- **Tier 3** → Modal MedGemma (clinical deep-reasoning & report finalization — **PoC ONLY under NFR-16a**). Activated when RAG confidence < threshold, radiologist explicitly requests "clinical depth", or Edge Gemma detects complex medical reasoning requiring clinical expertise. Same NFR-16a governance as Tier 2. Consult mode: `tier_3`. Output validated by RAG-Referee before reaching radiologist.
|
||||
- **Tier 4** → MOH guideline template responses (rule-based, no generative model, always available, deterministic, cited — safety floor when all inference tiers fail). Consult mode: `tier_4`.
|
||||
- **Circuit Breaker**: PWA Feature Flag Manager (`consult_mode` state machine) attempts tiers sequentially: `tier_1` → `tier_2` → `tier_3` → `tier_4`. Emits tier-transition events to on-prem immutable audit log (NFR-17). Tier 2/3 transitions require consent token + redaction manifest committed *before* the HTTP call. CDN fallback events (model weight distribution only) log lighter audit entry: no redaction required, no consent required (weights carry no PII).
|
||||
- **GCP CDN configuration** (emergency distribution path only):
|
||||
- Cloud Storage bucket: `gs://vkist-models-{site_id}` — Uniform bucket-level access, no `allUsers` public exposure.
|
||||
- Signed-URL generation: FastAPI generates time-limited signed URLs (signed by project service account private key, CSEK-managed) with a 15-minute expiry; returned to PWA as a `307` redirect on intranet CDN failure.
|
||||
@@ -133,7 +137,7 @@ The system adopts a hybrid edge-cloud (on-premise) architecture with the followi
|
||||
- **JWT session state** (TTL 1h, matches token expiry): validates active sessions without Postgres roundtrip; coalesced look-aside with single-flight mutex to prevent thundering herd on session-miss.
|
||||
- **MOH guideline chunks** (TTL 7d): version-key invalidation triggered by Postgres NOTIFY on guideline ingestion events; cache key = `guideline:{version}:{chunk_id}`, full key-space retired on version bump, no TTL races.
|
||||
- **DICOM metadata per session** (TTL 12h, per-session scope): avoids repeated Postgres lookups of DICOM headers during the 3-step ML pipeline; evicted after session completion.
|
||||
- **Circuit-breaker / consult_mode state** (TTL 2h, matches session): consult_mode enum (tier_1/tier_2/tier_3a/tier_3b) keyed by session_hash. Written atomically on tier transitions. Read by: FastAPI routing layer, PWA (via SSE status push), Prometheus metrics exporter. Justifies Redis over Postgres: sub-ms reads required, writes are frequent (per BERT token stream during Socratic dialogue), data is volatile (expires with session), and multiple FastAPI workers behind NGINX need shared visibility — in-process memory cannot cross worker boundaries.
|
||||
- **Circuit-breaker / consult_mode state** (TTL 2h, matches session): consult_mode enum (tier_1/tier_2/tier_3/tier_4) keyed by session_hash. Written atomically on tier transitions. Read by: FastAPI routing layer, PWA (via SSE status push), Prometheus metrics exporter. Justifies Redis over Postgres: sub-ms reads required, writes are frequent (per BERT token stream during Socratic dialogue), data is volatile (expires with session), and multiple FastAPI workers behind NGINX need shared visibility — in-process memory cannot cross worker boundaries.
|
||||
- **Rate-limit counters** (TTL 30s, sliding window): INCR on consult request entry. Self-clearing via EXPIRE. Prevents Triton GPU saturation under concurrent burst from multiple radiologists. Atomic Redis operation; Postgres row-lock alternative creates write amplification on hot path.
|
||||
- **Not cached in Redis** (served from primary store):
|
||||
- Audit log: append-only, NFR-17 immutability requirement — no intermediate cache layer.
|
||||
@@ -248,9 +252,9 @@ The system adopts a hybrid edge-cloud (on-premise) architecture with the followi
|
||||
- **Application**: NeMo Guardrails and GuardrailsAI require substantial runtime scaffolding, rule engines, and WASM-compatible bindings that do not exist in the browser ecosystem at the model inference layer. The client-side cannot run a NeMo Rail or GuardrailsAI pipeline inside a WebWorker alongside WebLLM. Therefore, the edge guardrail tier relies on two lightweight mechanisms:
|
||||
1. **Robust prompt-based rules**: Hard system-prompt constraints injected at every consult turn. Boundaries: no diagnosis outside MOH protocol scope; mandatory citation for every clinical assertion; no free-text echoing of PII even if user inputs it; immediate refusal to follow instructions that attempt prompt injection or jailbreak; tone and scope locked to clinical decision-support.
|
||||
2. **BERT hallucination / mal-intention detection**: Each candidate LLM token stream (or per-turn summary) is evaluated by a local Transformers.js BERT classifier running in a dedicated `guardrail.worker.ts` WebWorker. The classifier scores for: (a) hallucination probability (factual grounding vs retrieved MOH context); (b) mal-intention / prompt-injection indicators (user-context leaking, instruction override attempts, jailbreak markers); (c) scope violation (claims outside synovitis/MSK MOH guidelines).
|
||||
- **Session termination and cloud mitigation**: If BERT edge guardrail returns a violation score above threshold on any axis → FastAPI immediately terminates the WebLLM session (`stream.close()`), logs the trigger event to immutable audit, and opens a new mitigated consult session toward Tier 3a (cloud Vertex AI MedGemma). Mitigation path: the same user query + retrieved MOH chunks (already computed) + edge-detected-offense log are forwarded under NFR-16a governance (consent token confirmed, Decree 13 redaction re-applied, audit-log commit before egress). The radiologist sees: "Edge model restricted — escalating to cloud consult (redacted)."
|
||||
- **WebWorker isolation**: `guardrail.worker.ts` runs in a separate WebWorker from `cv.worker.ts` (LiteRT) and `llm.worker.ts` (WebLLM). No shared WASM memory between workers. IndexedDB caches artifacts but is never injected into LLM context. Unload order on memory pressure: `llm.worker.ts` → `guardrail.worker.ts` → `cv.worker.ts`.
|
||||
- **Cloud model guardrail**: Cloud Vertex AI MedGemma (Tier 3a/3b) uses natively supported Vertex AI Model Garden safety filters (tuned for healthcare) plus the same server-side FastAPI BERT RAG-Referee gate. No additional client-side framework required.
|
||||
- **Session termination and cloud mitigation**: If BERT edge guardrail returns a violation score above threshold on any axis → FastAPI immediately terminates the WebLLM session (`stream.close()`), logs the trigger event to immutable audit, and routes to the appropriate cloud tier based on `task_type`: orchestration/translation tasks route to Tier 2 (Gemini), clinical deep-reasoning/report finalization routes to Tier 3 (MedGemma). Mitigation path: the same user query + retrieved MOH chunks (already computed) + edge-detected-offense log are forwarded under NFR-16a governance (consent token confirmed, Decree 13 redaction re-applied, audit-log commit before egress). The radiologist sees: "Edge model restricted — escalating to cloud consult (redacted)."
|
||||
- **WebWorker isolation**: `guardrail.worker.ts` runs in a separate WebWorker from `cv.worker.ts` (LiteRT) and `llm.worker.ts` (WebLLM). No shared WASM memory between workers. IndexedDB caches artifacts but is never injected into LLM context. Unload order on memory pressure: `llm.worker.ts` → `guardrail.worker.ts` → `cv.worker.ts`.
|
||||
- **Cloud model guardrail**: Cloud models use the server-side FastAPI BERT RAG-Referee gate as the primary safety mechanism. Tier 2 (Gemini) uses Vertex AI Model Garden safety filters (tuned for healthcare). Tier 3 (MedGemma) output is additionally validated by RAG-Referee before reaching the radiologist. No additional client-side framework required.
|
||||
|
||||
### 5.8 Edge Data Hygiene: Anonymization, Redaction & Server-Side Ground-Check
|
||||
- **Pattern**: Edge-to-Server Redaction Pipeline with Ground-Truth Verification
|
||||
@@ -269,22 +273,23 @@ The system adopts a hybrid edge-cloud (on-premise) architecture with the followi
|
||||
### 5.9 RAG as Essential Pipeline Step (Non-Optional)
|
||||
- **Pattern**: Mandatory RAG Pre-Processing for All LLM Consult Paths
|
||||
- **Source**: NFR-18 (100% LLM text cites MOH protocol); NFR-12 (zero-friction explainability)
|
||||
- **Application**: RAG retrieval is NOT an optional tool the LLM may choose to invoke. It is a mandatory pipeline step that executes before any LLM generation, for both edge (browser WebLLM) and cloud (Vertex AI MedGemma) tiers.
|
||||
- **Application**: RAG retrieval is NOT an optional tool the LLM may choose to invoke. It is a mandatory pipeline step that executes before any LLM generation, for all three-tier systems: Edge Gemma (browser WebLLM), Gemini (GCP Vertex AI), and MedGemma (Modal).
|
||||
- **Pipeline order**:
|
||||
1. User query arrives (post-redaction)
|
||||
2. `rag.query()` retrieves top-k MOH guideline chunks from pgvector HNSW
|
||||
3. Retrieved chunks + strict system prompt ("You are a clinical decision-support assistant. Answer ONLY using the provided MOH guidelines. Cite chunk IDs. Do not speculate.") → injected into LLM context
|
||||
4. LLM generates grounded, cited response
|
||||
5. `referee.validate()` (BERT RAG-Referee) checks citation correctness, logical cohesion, factual contestant status
|
||||
5. `referee.validate()` (BERT RAG-Referee) checks citation correctness, logical cohesion, factual contestant status → mandatory for MedGemma (Tier 3) output; optional but recommended for Gemini (Tier 2); lightweight validation for Edge Gemma (Tier 1)
|
||||
6. If any axis fails → reject → fallback to MOH template
|
||||
- **Rationale**: If RAG were merely a tool call, a compromised or under-performing edge LLM could choose to skip retrieval and generate ungrounded text, violating NFR-18. Mandatory RAG pre-processing closes this vector.
|
||||
|
||||
### 5.10 Tool-Calling Semantics for Edge and Cloud LLMs
|
||||
- **Pattern**: Function-Calling as Convenience Layer Over Mandatory RAG
|
||||
- **Source**: Gemma Functions (Google DeepMind) + Vertex AI Function Calling
|
||||
- **Source**: Gemma Functions (Google DeepMind) + Vertex AI Function Calling + Modal MedGemma Function Calling
|
||||
- **Application**:
|
||||
- **Edge LLM (GemmaE2B via WebLLM)**: Supports Gemma Functions. The RAG retrieval is exposed as a function tool (`retrieve_moh_guidelines(query, top_k)`). HOWEVER, this tool is pre-invoked by the host pipeline per §5.9 before the first LLM token is generated. The function-calling interface is available for subsequent turns of Socratic dialogue where the model may need additional retrieval mid-conversation (e.g., user asks "what about contraindications?" → mid-turn retrieval). The host pipeline enforces the mandatory RAG call on turn 0; subsequent per-turn retrieval is model-initiated via function calling.
|
||||
- **Cloud LLM (MedGemma on Vertex AI)**: Native Vertex AI Function Calling is enabled. Same semantics: server-side FastAPI injects top-k MOH chunks before first generation; model may request further retrieval via function call during multi-turn consult.
|
||||
- **Edge LLM (Edge Gemma via WebLLM)**: Supports Gemma Functions. The RAG retrieval is exposed as a function tool (`retrieve_moh_guidelines(query, top_k)`). HOWEVER, this tool is pre-invoked by the host pipeline per §5.9 before the first LLM token is generated. The function-calling interface is available for subsequent turns of Socratic dialogue where the model may need additional retrieval mid-conversation (e.g., user asks "what about contraindications?" → mid-turn retrieval). The host pipeline enforces the mandatory RAG call on turn 0; subsequent per-turn retrieval is model-initiated via function calling.
|
||||
- **Cloud LLM Tier 2 (Gemini on Vertex AI)**: Native Vertex AI Function Calling is enabled for orchestration/translation tasks. Same semantics: server-side FastAPI injects top-k MOH chunks before first generation; model may request further retrieval via function call during multi-turn consult.
|
||||
- **Cloud LLM Tier 3 (MedGemma on Modal)**: Native function-calling interface for clinical reasoning tasks. Server-side FastAPI injects top-k MOH chunks and RAG-Referee validation results before first generation; model may request further retrieval via function call during multi-turn clinical consult.
|
||||
- **Why not pure tool-calling**: Pure tool-calling delegates the decision to retrieve to the model. NFR-18 requires 100% citation; delegating retrieval to model choice creates a non-zero probability of unsupported generation. Mandatory pre-injection + optional per-turn tool call is the safer contract.
|
||||
|
||||
### 5.11 Output Filtering
|
||||
@@ -304,6 +309,11 @@ The system adopts a hybrid edge-cloud (on-premise) architecture with the followi
|
||||
- `edge_guardrail_violation`: session_hash, violation_axis (hallucination|mal_intention|scope_breach), bert_score, user_query_hash, mitigation_target (cloud|template).
|
||||
- `rag_retrieval_event`: session_hash, query_hash, retrieved_chunk_ids, latency_ms.
|
||||
- `referee_decision`: session_hash, axis_results (attribution|cohesion|contestant), overall_pass, fallback_triggered.
|
||||
- `egress_consent_gemini`: session_hash, user_id, task_type, redaction_hash — before Gemini cloud call.
|
||||
- `egress_consent_medgemma`: session_hash, user_id, task_type, redaction_hash — before MedGemma cloud call.
|
||||
- `egress_response_gemini`: session_hash, task_type, status, ts — after Gemini response.
|
||||
- `egress_response_medgemma`: session_hash, task_type, status, ts — after MedGemma response.
|
||||
- `cloud_llm_escalation`: session_hash, from_tier, to_tier, reason (reroute, referee_fail, token_limit) — tracks Gemini→MedGemma or any cross-tier escalation.
|
||||
- **Audit events for this layer**: `redaction_edge_manifest` (client), `redaction_server_refine` (server-side cleaning pass), `redaction_failure` (server unable to clean → error), `egress_consent` + `egress_redact_manifest` (before cloud call, NFR-16a).
|
||||
- **Retention**: Audit events retained indefinitely (append-only, no UPDATE/DELETE). Raw LLM token streams retained for 90 days in MinIO (compressed, AES-256 at rest) for drift analysis. After 90 days, streams are deleted; only aggregated `referee_decision` and `rag_retrieval_event` summaries remain.
|
||||
- **BERT drift monitor telemetry**: edge BERT classifier score distributions exported to Prometheus; alert if false-positive rate on hallucination detection exceeds 5% or false-negative rate exceeds 0.1% (clinical safety threshold).
|
||||
|
||||
Reference in New Issue
Block a user