update the codebase poc ver1
This commit is contained in:
DatTT127
@@ -17,25 +17,30 @@ Build a reproducible, air-gapped-first musculoskeletal ultrasound analysis platf
|
||||
### Functional (Sprint 1-2)
|
||||
- [ ] FR-25: Load knee DICOM → segment joint structures → measure synovium thickness → grade synovitis 0-3
|
||||
- [ ] Grad-CAM overlay on primary viewport (zero extra clicks)
|
||||
- [ ] Circuit-breaker Socratic dialogue (radiologist challenges AI grade before finalizing)
|
||||
- [ ] Circuit-breaker Socratic dialogue (radiologist challenges AI grade before finalizing) across 3-tier LLM system (Edge Gemma → Gemini → MedGemma)
|
||||
- [ ] BERT drift monitor against baseline MOH corpus
|
||||
- [ ] RAG-Referee validates every LLM-generated explanation against top-k retrieved MOH guideline chunks
|
||||
- [ ] RAG-Referee validates every LLM-generated explanation against top-k retrieved MOH guideline chunks (mandatory for cloud tiers, lightweight for edge)
|
||||
- [ ] Decree 13 PII scrubbing on all outbound text (client-side + FastAPI middleware)
|
||||
- [ ] ladybugDB ontology traversal for anatomical entity disambiguation
|
||||
- [ ] GemmaE2B/MedGemma Vietnamese LLM consult (browser WebLLM local OR cloud Vertex AI) with MOH guideline citations
|
||||
- [ ] 3-model LLM consult system: Edge Gemma (browser WebLLM local), Gemini (GCP Vertex AI, orchestration/translation), MedGemma (Modal, clinical deep-reasoning) with MOH guideline citations and cost guarding (<20% MedGemma usage)
|
||||
- [ ] Circular 46/2018 PDF report generation
|
||||
- [ ] Immutable audit log append (NFR-17) and HITL digital signature gate (NFR-19)
|
||||
|
||||
### Non-Functional (Critical)
|
||||
- [ ] NFR-4: 150 MB idle app bundle
|
||||
- [ ] NFR-5: ≤1.5 s inference (on-prem Triton)
|
||||
- [ ] NFR-7: ≤200 ms TTFT token streaming
|
||||
- [ ] NFR-5: ≤1.5 s inference (on-prem Triton); cloud tiers targeted at <30s median
|
||||
- [ ] NFR-7: ≤200 ms TTFT token streaming; progressive streaming with 25s timeout for cloud responses
|
||||
- [ ] NFR-8: Fault-tolerant across Wi-Fi drops (state preserved)
|
||||
- [ ] NFR-10: Automated Generative Safety Guardrails: <90% verification prohibited, 100% of LLM-generated patient text explanations must pass verification. 3-tier guardrail: prompt rules + BERT edge detection (Tier 1) → Vertex AI safety filters (Tier 2) → RAG-Referee mandatory validation (Tier 3)
|
||||
- [ ] NFR-11: Onboarding ≤ 45 minutes
|
||||
- [ ] NFR-12: Zero-Friction Explainability Integration
|
||||
- [ ] NFR-13: Spatial Layer-Activation Mapping
|
||||
- [ ] NFR-14: No client-side GPU/neural accelerator required
|
||||
- [ ] NFR-15: Circular 46 EMR compliance
|
||||
- [ ] NFR-16: Air-gapped primary; NFR-16a PoC fallback with redaction/consent/audit
|
||||
- [ ] NFR-16: Air-gapped primary; NFR-16a PoC fallback with redaction/consent/audit for cloud LLM tiers (Gemini + MedGemma)
|
||||
- [ ] NFR-16a: Emergency Cloud Fallback with Redaction (PoC-SCOPE). Cloud LLM API (GCP Vertex AI Gemini + Modal MedGemma) invoked ONLY when browser-side WebLLM (Edge Gemma) is unavailable or orchestrates cloud task. FastAPI Redaction Middleware MUST strip all Decree 13 PII fields before egress. GCP CSEK + 30-day lifecycle + 1-year auto-delete. Audit-log commit + consent before egress. MedGemma usage <20% target with alerting. NFR-16a retired upon PoC sign-off.
|
||||
- [ ] NFR-17: Immutable audit log
|
||||
- [ ] NFR-18: 100% LLM text cites MOH protocol via RAG
|
||||
- [ ] NFR-18: 100% LLM text cites MOH protocol via RAG. Mandatory RAG pre-processing for all 3 tiers (not optional tool-calling). RAG-Referee validates MedGemma output.
|
||||
- [ ] NFR-19: HITL digital signature before FINALIZED/ARCHIVED
|
||||
|
||||
---
|
||||
@@ -79,7 +84,8 @@ System_Boundary(hospital_lan, "Hospital LAN (Air-Gapped, ≤10 Mbps)") {
|
||||
System_Boundary(k3s_cluster, "K3s Orchestration Cluster") {
|
||||
System_Boundary(edge_servers, "Application Server Cluster") {
|
||||
Container(edge_inference, "Edge Inference Service", "FastAPI (Python 3.12, Uvicorn)", "DICOM ingest, 3-step ML pipeline orchestration, Grad-CAM generation, report assembly, consult SSE streaming.")
|
||||
Container(rag_svc, "RAG & Knowledge Service", "FastAPI (Python 3.12, asyncpg)", "pgvector top-k retrieval, ladybugDB ontology wrapper, GemmaE2B/MedGemma consult route, RAG-Referee validation.")
|
||||
Container(cloud_llm_gateway, "Cloud LLM Gateway", "FastAPI (Python 3.12, Uvicorn)", "Routes orchestration/translation to Gemini (GCP) and clinical deep-reasoning to MedGemma (Modal); NFR-16a redaction, consent, audit enforcement; consult_mode state machine extension; cost guarding for MedGemma usage.")
|
||||
Container(rag_svc, "RAG & Knowledge Service", "FastAPI (Python 3.12, asyncpg)", "pgvector top-k retrieval, ladybugDB ontology wrapper, RAG-Referee validation for MedGemma output, mandatory RAG pre-processing for all tiers.")
|
||||
Container(audit_svc, "Audit & EMR Service", "Node.js / FastAPI Worker", "Immutable append-only audit events; HL7/FHIR EMR push with outbox retry.")
|
||||
Container(api_gw, "API Gateway", "Envoy", "TLS termination, rate limiting, routing, OIDC validation pass-through.")
|
||||
Container(auth_svc, "Identity & Access", "Keycloak", "OIDC, RBAC, realm vkist-msk.")
|
||||
@@ -94,11 +100,12 @@ System_Boundary(hospital_lan, "Hospital LAN (Air-Gapped, ≤10 Mbps)") {
|
||||
}
|
||||
}
|
||||
|
||||
System_Ext(triton, "Triton Inference Server", "NVIDIA Triton, ONNX/TensorRT, gRPC 8001", "3-step ML pipeline (angle → inflammation → segmentation) + embedding extraction.")
|
||||
System_Ext(triton, "Triton Inference Server", "NVIDIA Triton, ONNX/TensorRT, gRPC 8001", "3-step ML pipeline (angle → inflammation → segmentation) + EmbeddingGemma RAG embeddings.")
|
||||
System_Ext(emr, "Hospital EMR / HIS", "HL7/FHIR", "Finalized report storage, prescription sync, ground-truth records.")
|
||||
System_Ext(pacs, "PACS / Ultrasound Device", "DICOM/C-MOVE", "Image capture and retrieval.")
|
||||
System_Ext(gcp_cdn, "GCP CDN Emergency Fallback", "Signed-URL Cloud CDN (ap-southeast1)", "Non-clinical model weight distribution when intranet CDN unreachable.")
|
||||
System_Ext(vertex_ai, "GCP Vertex AI", "GemmaE2B via REST", "PoC-only NFR-16a inference tier; redacted payloads only.")
|
||||
System_Ext(vertex_ai, "GCP Vertex AI (Gemini)", "Gemini via REST", "PoC-only NFR-16a inference tier: orchestration, translation, UI planning; redacted payloads only.")
|
||||
System_Ext(modal_medgemma, "Modal MedGemma", "FastAPI + Transformers (T4 GPU)", "PoC-only NFR-16a clinical deep-reasoning endpoint; redacted payloads with RAG-Referee validation; keep-warm instances for latency budget.")
|
||||
|
||||
Rel(radiologist, pwa, "Loads scan, reviews grade, finalizes report, views explanations", "HTTPS 443")
|
||||
Rel(admin, nginx, "Deploys, monitors, configures", "HTTPS 443 / SSH")
|
||||
@@ -118,9 +125,16 @@ Rel(edge_inference, minio, "DICOM, overlays, reports", "S3 API")
|
||||
Rel(edge_inference, auth_svc, "Token introspection", "OIDC")
|
||||
Rel(edge_inference, obs_stack, "/metrics", "HTTP 9090")
|
||||
|
||||
Rel(edge_inference, cloud_llm_gateway, "Routes cloud consult", "HTTP 8000")
|
||||
Rel(cloud_llm_gateway, vertex_ai, "Gemini proxy (NFR-16a redacted)", "HTTPS 443 / REST")
|
||||
Rel(cloud_llm_gateway, modal_medgemma, "MedGemma proxy (NFR-16a redacted)", "HTTPS 443 / REST")
|
||||
Rel(cloud_llm_gateway, redis, "Consult mode, consent, rate-limit", "TCP 6379")
|
||||
Rel(cloud_llm_gateway, postgres, "Audit log append", "SQL 5432")
|
||||
|
||||
Rel(rag_svc, postgres, "pgvector HNSW queries", "SQL 5432")
|
||||
Rel(rag_svc, redis, "Guideline cache, pub/sub invalidation", "TCP 6379")
|
||||
Rel(rag_svc, minio, "Guideline PDF ingestion staging", "S3 API")
|
||||
Rel(rag_svc, edge_inference, "RAG + RAG-Referee results", "HTTP")
|
||||
|
||||
Rel(audit_svc, postgres, "Appends immutable audit events", "SQL 5432")
|
||||
Rel(audit_svc, emr, "Finalized report push", "HL7/FHIR")
|
||||
@@ -130,7 +144,6 @@ Rel(edge_inference, emr, "Report push (via audit-svc wrapper)", "HL7/FHIR")
|
||||
Rel(pwa, pacs, "Direct DICOM capture / C-MOVE", "DICOM")
|
||||
|
||||
Rel(k3s_cluster, gcp_cdn, "Model weight fetch fallback (non-clinical)", "HTTPS 443 (signed URL)")
|
||||
Rel(edge_inference, vertex_ai, "PoC-only cloud consult (NFR-16a redacted)", "HTTPS 443 / REST")
|
||||
Rel(postgres, minio, "Backup checkpoint", "S3 API")
|
||||
|
||||
@enduml
|
||||
@@ -143,8 +156,14 @@ Container communication summary:
|
||||
| PWA | NGINX | HTTPS 443 | All API requests |
|
||||
| NGINX | Envoy | HTTP 8000 | Route upstream |
|
||||
| Envoy | Edge Inference | HTTP 8000 | `/api/*` |
|
||||
| Envoy | Cloud LLM Gateway | HTTP 8000 | `/api/cloud-orchestrate`, `/api/cloud-consult` |
|
||||
| Envoy | RAG Service | HTTP 8001 | `/rag/*` |
|
||||
| Envoy | Keycloak | HTTP 8080 | OIDC validation |
|
||||
| Edge Inference | Cloud LLM Gateway | HTTP 8000 | Route cloud consult |
|
||||
| Cloud LLM Gateway | Gemini (Vertex AI) | HTTPS 443 / REST | Orchestration, translation, UI planning (NFR-16a) |
|
||||
| Cloud LLM Gateway | MedGemma (Modal) | HTTPS 443 / REST | Clinical deep-reasoning, report finalization (NFR-16a) |
|
||||
| Cloud LLM Gateway | Redis | TCP 6379 | Consult mode, consent, rate-limit |
|
||||
| Cloud LLM Gateway | Postgres | TCP 5432 | Audit log append |
|
||||
| Edge Inference | Triton | gRPC 8001 | 3-step ML pipeline + embeddings |
|
||||
| Edge Inference | Postgres | TCP 5432 | SQL + pgvector HNSW |
|
||||
| Edge Inference | Redis | TCP 6379 | Session, rate-limit, consult-mode |
|
||||
@@ -165,23 +184,24 @@ Container communication summary:
|
||||
!include <C4/C4_Component>
|
||||
|
||||
Container_Boundary(edge_svc, "edge-inference-svc (FastAPI)") {
|
||||
Component(api, "API Controller", "REST + SSE", "/api/analyze, /api/consult/stream")
|
||||
Component(api, "API Controller", "REST + SSE", "/api/analyze, /api/cloud-orchestrate redirect")
|
||||
Component(stream, "SSE Token Streamer", "FastAPI StreamingResponse", "200 ms TTFT, heart-beat every 30s")
|
||||
Component(preproc, "Image Preprocessor", "OpenCV + pydicom", "CLAHE, rescale, DICOM header scrub (client-side pre-check)")
|
||||
Component(router, "Inference Router", "consult_mode state", "Tier selection: WASM→Triton→Vertex→Templates")
|
||||
Component(breaker, "Circuit Breaker", "pybreaker", "Wrap Triton + EMR calls; fail-open to templates")
|
||||
Component(router, "Inference Router", "consult_mode state", "Tier selection: Edge Gemma (tier_1) → Gemini (tier_2, orchestration) → MedGemma (tier_3, clinical) → Templates (tier_4)")
|
||||
Component(breaker, "Circuit Breaker", "pybreaker", "Wrap Triton + EMR + Cloud LLM calls; fail-open to templates")
|
||||
Component(pipeline, "ML Pipeline", "gRPC client", "Angle→Inflammation→Segmentation→Measurement")
|
||||
Component(gradcam, "Grad-CAM Generator", "OpenCV", "Spatial activation overlay; base64 PNG to PWA")
|
||||
Component(report, "Report Builder", "WeasyPrint / ReportLab", "Circular 46 PDF; HITL signature gate before FINALIZED")
|
||||
Component(rag, "RAG Service", "pgvector SQL", "Retrieve top-5 MOH chunks; enforce NFR-18 citation")
|
||||
Component(referee, "RAG-Referee", "BERT classifier", "Reject LLM text if citation confidence < threshold")
|
||||
Component(rag, "RAG Service", "pgvector SQL", "Retrieve top-5 MOH chunks; mandatory pre-generation for all tiers; enforce NFR-18 citation")
|
||||
Component(referee, "RAG-Referee", "BERT classifier", "Reject MedGemma text if citation confidence < threshold; mandatory for tier_3")
|
||||
Component(nlp, "NLP Scrubber", "Microsoft Presidio", "Re-verify edge redaction; refine/clean residual PII; error if unresolvable")
|
||||
Component(audit, "Audit Logger", "Append-only writer", "Every tier transition, consent, finalize, override")
|
||||
Component(audit, "Audit Logger", "Append-only writer", "Every tier transition, consent, finalize, override; cloud_llm_escalation events")
|
||||
}
|
||||
Rel(api, stream, "delegates", "SSE")
|
||||
Rel(api, preproc, "validates image", "sync")
|
||||
Rel(api, router, "selects tier", "sync")
|
||||
Rel(router, pipeline, "invokes", "gRPC")
|
||||
Rel(router, api, "redirects cloud consult", "HTTP")
|
||||
Rel(router, stream, "fallback text", "SSE")
|
||||
Rel(api, gradcam, "requests overlay", "sync")
|
||||
Rel(api, report, "generates PDF", "sync")
|
||||
@@ -192,39 +212,27 @@ Rel(api, audit, "writes event", "sync")
|
||||
@enduml
|
||||
```
|
||||
|
||||
### 4.3 Component Diagrams (Tier 3)
|
||||
|
||||
**Edge Inference Service (edge-inference-svc)**
|
||||
**Cloud LLM Gateway**
|
||||
|
||||
```plantuml
|
||||
@startuml
|
||||
!include <C4/C4_Component>
|
||||
|
||||
Container_Boundary(edge_svc, "edge-inference-svc (FastAPI)") {
|
||||
Component(api, "API Controller", "REST + SSE", "/api/analyze, /api/consult/stream")
|
||||
Component(stream, "SSE Token Streamer", "FastAPI StreamingResponse", "200 ms TTFT, heart-beat every 30s")
|
||||
Component(preproc, "Image Preprocessor", "OpenCV + pydicom", "CLAHE, rescale, DICOM header scrub (client-side pre-check)")
|
||||
Component(router, "Inference Router", "consult_mode state", "Tier selection: WASM→Triton→Vertex→Templates")
|
||||
Component(breaker, "Circuit Breaker", "pybreaker", "Wrap Triton + EMR calls; fail-open to templates")
|
||||
Component(pipeline, "ML Pipeline", "gRPC client", "Angle→Inflammation→Segmentation→Measurement")
|
||||
Component(gradcam, "Grad-CAM Generator", "OpenCV", "Spatial activation overlay; base64 PNG to PWA")
|
||||
Component(report, "Report Builder", "WeasyPrint / ReportLab", "Circular 46 PDF; HITL signature gate before FINALIZED")
|
||||
Component(rag, "RAG Service", "pgvector SQL", "Retrieve top-5 MOH chunks; enforce NFR-18 citation")
|
||||
Component(referee, "RAG-Referee", "BERT classifier", "Reject LLM text if citation confidence < threshold")
|
||||
Component(nlp, "NLP Scrubber", "Microsoft Presidio", "Re-verify edge redaction; refine/clean residual PII; error if unresolvable")
|
||||
Component(audit, "Audit Logger", "Append-only writer", "Every tier transition, consent, finalize, override")
|
||||
Container_Boundary(cloud_gw, "cloud-llm-gateway (FastAPI)") {
|
||||
Component(api, "Cloud API Controller", "REST + SSE", "/api/cloud-orchestrate, /api/cloud-consult, /api/cloud-consult/stream")
|
||||
Component(gateway, "Cloud LLM Router", "task_type matcher", "Routes orchestration/translation → Gemini (tier_2); clinical deep-reasoning → MedGemma (tier_3)")
|
||||
Component(consent, "Consent Enforcer", "Redis + NFR-16a checklist", "Validates consent token + redaction manifest before egress")
|
||||
Component(audit, "Cloud Audit Emitter", "PostgreSQL append-only", "egress_consent, egress_response, cloud_llm_escalation events")
|
||||
Component(cost, "Cost Guard", "Redis counter + Prometheus", "Tracks MedGemma usage ratio; alerts if >20% over 24h window")
|
||||
Component(referee, "RAG-Referee Trigger", "BERT classifier", "Mandatory validation for MedGemma output before SSE to PWA")
|
||||
Component(rag, "RAG Pre-processing", "pgvector SQL", "Mandatory top-k injection before any cloud LLM generation")
|
||||
}
|
||||
Rel(api, stream, "delegates", "SSE")
|
||||
Rel(api, router, "selects tier", "sync")
|
||||
Rel(router, pipeline, "invokes", "gRPC")
|
||||
Rel(router, stream, "fallback text", "SSE")
|
||||
Rel(api, preproc, "validates image", "sync")
|
||||
Rel(api, gradcam, "requests overlay", "sync")
|
||||
Rel(api, report, "generates PDF", "sync")
|
||||
Rel(api, rag, "queries MOH", "SQL")
|
||||
Rel(rag, referee, "validates citations", "sync")
|
||||
Rel(api, nlp, "scrubs output", "sync")
|
||||
Rel(api, audit, "writes event", "sync")
|
||||
Rel(api, consent, "verifies", "sync")
|
||||
Rel(api, gateway, "routes", "sync")
|
||||
Rel(gateway, rag, "injects chunks", "sync")
|
||||
Rel(gateway, referee, "validates output", "sync")
|
||||
Rel(gateway, audit, "logs egress", "sync")
|
||||
Rel(gateway, cost, "increments counter", "sync")
|
||||
@enduml
|
||||
```
|
||||
|
||||
@@ -309,13 +317,14 @@ app/
|
||||
│ ├── auth.py # Keycloak OIDC validation
|
||||
│ ├── phi_scrub.py # Microsoft Presidio redaction gate (NFR-16a); refine edge output; error if unresolvable PII
|
||||
│ └── audit.py # Append-only event emitter
|
||||
├── routers/
|
||||
├── routers/ # FastAPI route handlers (existing + new cloud LLM)
|
||||
│ ├── analyze.py # POST /api/analyze (sync 3-step pipeline)
|
||||
│ ├── consult.py # SSE /api/consult/stream (GemmaE2B/MedGemma + RAG + guardrail session management)
|
||||
│ ├── pacs.py # C-MOVE proxy + DICOM upload
|
||||
│ ├── emr.py # HL7/FHIR push with outbox
|
||||
│ └── admin.py # Model update, drift review, cache invalidation
|
||||
├── services/
|
||||
│ ├── admin.py # Model update, drift review, cache invalidation
|
||||
│ ├── cloud_orchestrate.py # POST /api/cloud-orchestrate (Gemini proxy for orchestration/translation)
|
||||
│ └── cloud_consult.py # POST /api/cloud-consult + GET /api/cloud-consult/stream (MedGemma proxy)
|
||||
├── services/ # Business logic modules (existing + new cloud gateway)
|
||||
│ ├── inference_router.py # consult_mode state machine
|
||||
│ ├── triton_client.py # gRPC with retry decorator
|
||||
│ ├── rag.py # pgvector top-k + citation formatter
|
||||
@@ -324,11 +333,12 @@ app/
|
||||
│ ├── redaction.py # Presidio AnonymizerEngine; re-verify edge redaction; refine residual PII; error if unresolvable
|
||||
│ ├── report.py # Circular 46 PDF + HITL signature
|
||||
│ ├── ontology.py # ladybugDB C++ bindings wrapper
|
||||
│ └── audit_writer.py # WAL append
|
||||
│ ├── audit_writer.py # WAL append
|
||||
│ └── cloud_llm_gateway.py # Routes between Gemini (tier_2) and MedGemma (tier_3) based on task_type + consult_mode; NFR-16a consent/redaction/audit enforcement
|
||||
├── models/
|
||||
│ ├── dto.py # Request/response schemas (Pydantic v2)
|
||||
│ ├── domain.py # Case, Session, Grade, Embedding entities
|
||||
│ └── enums.py # ConsultMode, Grade, Tier
|
||||
│ └── enums.py # ConsultMode (tier_1..tier_4), Grade, Tier
|
||||
└── infra/
|
||||
├── cache.py # Redis client (5 data types only)
|
||||
├── db.py # SQLAlchemy async engine + pgvector
|
||||
@@ -443,22 +453,31 @@ Server-side FastAPI contracts:
|
||||
| Method | Path | Auth | Request | Response | Notes |
|
||||
|--------|------|------|---------|----------|-------|
|
||||
| POST | `/api/analyze` | JWT | multipart DICOM | JSON + Grad-CAM PNG | Sync, ≤1.5 s target |
|
||||
| GET | `/api/consult/stream` | JWT + consent | SSE text stream | NDJSON chunks | Token streaming ≤200 ms TTFT |
|
||||
| POST | `/api/cloud-orchestrate` | JWT + consent | JSON (task_type, prompt) | JSON text + tier | Gemini proxy; orchestration/translation/UI tasks |
|
||||
| POST | `/api/cloud-consult` | JWT + consent | JSON (task_type, prompt) | JSON text + tier | MedGemma proxy; clinical deep-reasoning |
|
||||
| GET | `/api/cloud-consult/stream` | JWT + consent | query params | SSE text stream | MedGemma streaming; 25s timeout, 30s budget |
|
||||
| POST | `/api/emr/push` | JWT | case_id | 202 Accepted | Outbox if offline |
|
||||
| POST | `/api/admin/models` | Admin | modelZip | 200 OK | K3s rolling restart |
|
||||
| GET | `/api/rag/citations?q=` | JWT | query string | JSON top-5 chunks | NFR-18 |
|
||||
| POST | `/api/safety/guardrail-check` | JWT | payload | GuardrailResult | NFR-10 edge guardrail |
|
||||
| GET | `/api/health` | — | — | HealthStatus | Liveness |
|
||||
| GET | `/api/model-registry` | JWT | — | ModelCatalog | Model registry |
|
||||
| POST | `/api/models/register` | Admin | modelZip | RegistrationResult | Model upload |
|
||||
|
||||
### 6.2 SSE Consult Stream Contract
|
||||
|
||||
```
|
||||
event: token
|
||||
data: {"text":"The","confidence":0.92}
|
||||
data: {"text":"The","confidence":0.92,"tier":"tier_2"}
|
||||
|
||||
event: citation
|
||||
data: {"source":"MOH guideline 2024 §3.2","page":12}
|
||||
data: {"source":"MOH guideline 2024 §3.2","page":12,"tier":"tier_2"}
|
||||
|
||||
event: tier_change
|
||||
data: {"from":"tier_1","to":"tier_2","reason":"edge_unavailable"}
|
||||
|
||||
event: done
|
||||
data: {"tier":"tier_2","latency_ms":340}
|
||||
data: {"tier":"tier_2","latency_ms":340,"model":"gemini-2.5-pro"}
|
||||
```
|
||||
|
||||
---
|
||||
@@ -475,6 +494,9 @@ data: {"tier":"tier_2","latency_ms":340}
|
||||
| Redis | — | OSS | **Use** | Scoped to 5 types; self-hosted |
|
||||
| MinIO | — | OSS | **Use** | S3-compatible; self-hosted |
|
||||
| Auth | Build (Keycloak) | Auth0/Okta SaaS | **Build** | NFR-16: Keycloak on-prem, no SaaS identity |
|
||||
| Cloud LLM Gateway | Build | — | **Build** | FastAPI service enforcing NFR-16a redaction/consent/audit; routes Gemini/MedGemma based on task_type |
|
||||
| Gemini (Vertex AI) | — | GCP PoC | **Use (NFR-16a)** | Orchestration, translation, UI planning; PoC only with redaction |
|
||||
| MedGemma (Modal) | — | Modal deploy | **Deploy (NFR-16a)** | Clinical deep-reasoning; PoC only with redaction + RAG-Referee |
|
||||
| EMR integration | Build | Mirth Connect | **Build** | Thin HL7 wrapper FastAPI → EMR; keeps surface area small |
|
||||
| CI/CD | Jenkins in K3s | SaaS GitHub Actions | **Build** | Jenkins inside LAN; cloud GitLab via SSH |
|
||||
| Issue tracking | Self-hosted Jira | Atlassian Cloud | **Build (NFR-16a)** | Cloud VM, compensating controls |
|
||||
@@ -520,15 +542,20 @@ data: {"tier":"tier_2","latency_ms":340}
|
||||
SNOMED-CT knee/hip subset; MSK entity relationships; C++ embedded bindings in FastAPI.
|
||||
|
||||
- [ ] **T3-C RAG service + RAG-Referee** (M)
|
||||
`/rag/query`; top-5 retrieval; BERT classifier to validate LLM citations; reject if threshold fail.
|
||||
`/rag/query`; top-5 retrieval; BERT classifier to validate LLM citations; reject if threshold fail. Mandatory for MedGemma (Tier 3) output; lightweight for Gemini (Tier 2) and Edge Gemma (Tier 1).
|
||||
|
||||
- [ ] **T3-D Browser WebLLM + Cloud MedGemma LLM endpoints** (L)
|
||||
Browser: WebLLM (GemmaE2B-Q4) loaded via Service Worker from intranet/GCP CDN; runs in separate WebWorker from CV pipeline.
|
||||
Cloud: MedGemma on GCP Vertex AI (NFR-16a governed); FastAPI wrapper with streaming + Decree 13 redaction middleware.
|
||||
Triton: EmbeddingGemma only (768-dim RAG embeddings), no LLM hosting.
|
||||
- [ ] **T3-D 3-Model LLM System: Edge Gemma + Gemini + MedGemma** (L)
|
||||
Tier 1 (Edge Gemma): Browser WebLLM (GemmaE2B-Q4) loaded via Service Worker from intranet/GCP CDN; runs in dedicated WebWorker.
|
||||
Tier 2 (Gemini): GCP Vertex AI (GCP Vertex AI Gemini) for orchestration, translation, UI planning; FastAPI wrapper with streaming + Decree 13 redaction middleware.
|
||||
Tier 3 (MedGemma): Modal-deployed MedGemma large-24b for clinical deep-reasoning and report finalization; FastAPI gateway with keep-warm instances, 30s timeout, RAG-Referee validation.
|
||||
Tier 4: MOH guideline templates (rule-based fallback).
|
||||
Consult mode state machine extended: `tier_1` → `tier_2` → `tier_3` → `tier_4`.
|
||||
|
||||
- [ ] **T3-E Decree 13 scrubber (client + server)** (M)
|
||||
Client: Dexie.js pre-egress regex. Server: FastAPI middleware for NFR-16a redaction; role-hash tokens.
|
||||
Client: Dexie.js pre-egress regex. Server: FastAPI middleware for NFR-16a redaction; role-hash tokens. Applied to all cloud tiers (Tier 2 + Tier 3).
|
||||
|
||||
- [ ] **T3-F Cloud LLM Gateway** (M)
|
||||
FastAPI service (`/api/cloud-orchestrate`, `/api/cloud-consult`, `/api/cloud-consult/stream`); routes based on `task_type`; enforces consent + redaction + audit before egress; tracks MedGemma usage for <20% cost guard target; extends consult_mode state machine.
|
||||
|
||||
### Phase 4: Compliance & HITL
|
||||
- [ ] **T4-A Immutable audit log** (M)
|
||||
@@ -561,13 +588,13 @@ Deliverables: K3s cluster up, DB VM ready, CI/CD pipeline green, PWA shell live.
|
||||
Deliverables: `/api/analyze` end-to-end; Grad-CAM overlay visible in PWA; circuit-breaker handles Triton failure.
|
||||
|
||||
### Week 5-6: Phase 3
|
||||
Deliverables: RAG queries return MOH citations; GemmaE2B/MedGemma streams Vietnamese explanations; RAG-Referee blocks unmapped LLM text.
|
||||
Deliverables: RAG queries return MOH citations; Edge Gemma (WebLLM) primary, Gemini (Vertex AI) for orchestration/translation, MedGemma (Modal) for clinical deep-reasoning; RAG-Referee validates MedGemma output; consult_mode state machine extended to tier_1→tier_2→tier_3→tier_4.
|
||||
|
||||
### Week 7: Phase 4
|
||||
Deliverables: Audit log immutable; HITL signature enforces finalization; Circular 46 PDF exports.
|
||||
Deliverables: Audit log immutable with new event types (egress_consent_gemini, egress_consent_medgemma, cloud_llm_escalation); HITL signature enforces finalization; Circular 46 PDF exports; cost guarding (<20% MedGemma usage) enforced.
|
||||
|
||||
### Week 8: Phase 5 + Sprint Review
|
||||
Deliverables: Dashboards, drift monitor, integration tests, demo-ready PWA.
|
||||
Deliverables: Dashboards (tier transitions, MedGemma usage count, latency p50/p99, consent events), drift monitor, integration tests, demo-ready PWA with 3-model routing status.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -73,14 +73,19 @@ Guided by the above requirements and the solution_architect_corpus, the architec
|
||||
### 4.1 High-Level Overview
|
||||
The system adopts a hybrid edge-cloud (on-premise) architecture with the following layers:
|
||||
1. **Edge Inference Layer**: Located on-premise at hospitals/clinics, runs lightweight ML models for real-time DICOM processing (NFR-5,6).
|
||||
2. **Micro-Services Layer**: Containerized services orchestrated by K3s (Kubernetes-certified, lightweight distribution designed for edge and resource-constrained production environments). K3s selected over alternatives (Docker Swarm, Nomad, ECS Fargate, Cloud Run) on the following criteria: NFR-16 requires on-premise deployment — eliminating ECS Fargate and Cloud Run (cloud-only platforms). Docker Swarm offers lowest PoC operational cost but is in maintenance mode with no viable scaling path; migration to production-grade orchestration requires complete rewrite of all deployment artifacts. Nomad is operationally viable but lacks ecosystem depth and has less documented migration path to multi-cluster federation. K3s is already production-grade at single-site scale, requires no migration to "production" — it IS the production platform. The scaling path to N hospitals is multi-cluster federation via K3s cluster-api, not platform replacement. All manifests and service definitions are forward-compatible with upstream Kubernetes.
|
||||
3. **Data Layer**: On-premise databases (PostgreSQL + pgvector for relational + vector search, Redis for caching, MinIO for S3-compatible object storage).
|
||||
3. **Embedding Layer**: EmbeddingGemma (768-dim) for RAG retrieval vectors; separate from BERT which is reserved for drift monitoring and RAG-Referee classification only.
|
||||
3. **Vector Layer (two-tier)**:
|
||||
2. **3-Tier LLM System**:
|
||||
- **Tier 1 (Edge Gemma)**: Browser/WebLLM (GemmaE2B local in WASM) — primary conversation, local inference, instant response, zero network cost. Orchestrates when to call cloud models via FastAPI middleware.
|
||||
- **Tier 2 (Gemini)**: GCP Vertex AI (Gemini) — orchestration, translation, UI planning, formatting. Governed by NFR-16a with redaction, consent, and audit logging.
|
||||
- **Tier 3 (MedGemma)**: Modal-deployed MedGemma (large-24b) — clinical deep-reasoning and report finalization. Governed by NFR-16a with redaction, consent, audit logging, and RAG-Referee validation.
|
||||
- **Tier 4 (Templates)**: MOH guideline template responses — rule-based, deterministic, safety floor when all inference tiers fail.
|
||||
3. **Micro-Services Layer**: Containerized services orchestrated by K3s (Kubernetes-certified, lightweight distribution designed for edge and resource-constrained production environments). K3s selected over alternatives (Docker Swarm, Nomad, ECS Fargate, Cloud Run) on the following criteria: NFR-16 requires on-premise deployment — eliminating ECS Fargate and Cloud Run (cloud-only platforms). Docker Swarm offers lowest PoC operational cost but is in maintenance mode with no viable scaling path; migration to production-grade orchestration requires complete rewrite of all deployment artifacts. Nomad is operationally viable but lacks ecosystem depth and has less documented migration path to multi-cluster federation. K3s is already production-grade at single-site scale, requires no migration to "production" — it IS the production platform. The scaling path to N hospitals is multi-cluster federation via K3s cluster-api, not platform replacement. All manifests and service definitions are forward-compatible with upstream Kubernetes.
|
||||
4. **Data Layer**: On-premise databases (PostgreSQL + pgvector for relational + vector search, Redis for caching, MinIO for S3-compatible object storage).
|
||||
5. **Embedding Layer**: EmbeddingGemma (768-dim) for RAG retrieval vectors; separate from BERT which is reserved for drift monitoring and RAG-Referee classification only.
|
||||
6. **Vector Layer (two-tier)**:
|
||||
- **Hot + Warm: pgvector** (Postgres disk-backed HNSW index, ~15K MOH guideline vectors plus session and case embeddings — serves both real-time RAG during consult and non-real-time analytical queries in a single store. NFR-18, NFR-12. NFR-16 compliant.)
|
||||
- **Cold: S3 Vectors** (AWS ap-southeast-1, billion-scale archival, cross-facility bootstrap source, disaster recovery rebuild origin — accessed under NFR-16a governance.)
|
||||
5. **Presentation Layer**: Progressive Web App (PWA) built with React, Web Workers, and service workers for offline capability and low memory footprint (NFR-4,14).
|
||||
6. **Observability & Security Layer**: Centralized logging (immutable append-only), monitoring, and audit trails (NFR-10,17).
|
||||
7. **Presentation Layer**: Progressive Web App (PWA) built with React, Web Workers, and service workers for offline capability and low memory footprint (NFR-4,14).
|
||||
8. **Observability & Security Layer**: Centralized logging (immutable append-only), monitoring, and audit trails (NFR-10,17).
|
||||
|
||||
### 4.2 Data Flow
|
||||
1. User captures DICOM image via PWA frontend.
|
||||
@@ -104,13 +109,12 @@ The system adopts a hybrid edge-cloud (on-premise) architecture with the followi
|
||||
[3] GCP CDN emergency fallback (asia-southeast1 = Singapore, nearest to Vietnam) — activated ONLY when hospital intranet CDN is confirmed unreachable. GCP backend bucket (`cdn-backend-models-vkist`) wraps a Cloud Storage bucket (`gs://vkist-models-{site_id}`) with Cloud CDN enabled; signed-URL access control (no `allUsers` public exposure); cache mode `CACHE_ALL_STATIC`, default TTL 24h, max TTL 7d (matches model version lifecycle). Service Worker receives a 307 redirect to signed CDN URL on intranet CDN failure; PWA caches the fetched weights locally via Cache Storage as normal. Because model weights contain zero patient data — they are static computational artifacts — the GCP CDN fetch event is committed to the on-prem audit log (case_hash, timestamp, consult_mode = "cdn_fallback") but does NOT trigger the full NFR-16a redaction pipeline. The redaction pipeline applies only to Tier 3a inference calls carrying clinical payloads, not to non-clinical model binary fetches.
|
||||
- **NFR-4 clarification**: NFR-4's 150MB ceiling covers the core application bundle (JS/TS code, UI assets, runtime state) at idle. Supra-150MB memory consumed by models, caches, and inference buffers is bounded separately under the WASM heap cap (1.5GB) and instrumented independently.
|
||||
- **Inference Fallback Chain** (LLM consult path only — CV pipeline is always on-prem Triton):
|
||||
- **Tier 0** → Browser WebLLM (GemmaE2B local in WASM, instant, zero network — preferred default when device has ≥3GB RAM and WebGPU available). Consult mode: `browser_local`. Triggered automatically on page load if model weights are cached and WebGPU is detected.
|
||||
- **Tier 1** → [REMOVED — Triton does NOT host LLM. Triton hosts CV models + EmbeddingGemma only.]
|
||||
- **Tier 1** → Browser WebLLM (Edge GemmaE2B local in WASM, instant, zero network — preferred default when device has ≥3GB RAM and WebGPU available). Consult mode: `tier_1`. Triggered automatically on page load if model weights are cached and WebGPU is detected. Responsible for primary conversation and orchestrating cloud calls.
|
||||
- **Tier 2** → On-prem Triton (hospital GPU node — CV pipeline ONLY: angle → inflammation → segmentation + EmbeddingGemma for RAG embeddings). The LLM consult does NOT fall back to Triton. If browser WebLLM is unavailable, the next tier is cloud.
|
||||
- **Tier 3a** → GCP Vertex AI (MedGemma via REST API — **PoC ONLY under NFR-16a**). Activated when Tier 0 (browser WebLLM) is unavailable (device unsupported, model not cached, memory exceeded) AND user explicitly consents per-session. All payloads MUST pass Decree 13 PII redaction gate (see §5.4) before leaving hospital boundary. GCP project configured with CSEK, 30-day storage lifecycle, 1-year project deletion timer. Consult mode: `cloud_vertex`.
|
||||
- **Tier 3b** → Cloud LLM Arbiter (GCP Vertex AI MedGemma — **PoC ONLY under NFR-16a, BERT-triggered correction**). Activated when BERT drift monitor (UC-74821) detects semantic impasse, logical contradiction, or contextual hallucination during Socratic dialogue, OR when radiologist explicitly requests "second opinion." Same Vertex AI endpoint as Tier 3a, but with arbiter system prompt + retrieved MOH chunks injected. ALL calls require prior audit-log commit of drift signal + user consent + Decree 13 redaction. Consult mode: `cloud_vertex_arbiter`.
|
||||
- **Tier 3c** → MOH guideline template responses (rule-based, no generative model, always available, deterministic, cited — safety floor when all inference tiers fail). Consult mode: `templates`.
|
||||
- **Circuit Breaker**: PWA Feature Flag Manager (`consult_mode` state machine) attempts tiers sequentially: `browser_local` → `cloud_vertex` → `cloud_vertex_arbiter` → `templates`. Emits tier-transition events to on-prem immutable audit log (NFR-17). Tier 3a/3b transitions require consent token + redaction manifest committed *before* the HTTP call. CDN fallback events (model weight distribution only) log lighter audit entry: no redaction required, no consent required (weights carry no PII).
|
||||
- **Tier 2** → GCP Vertex AI (Gemini — **PoC ONLY under NFR-16a**). Activated when Tier 1 (browser WebLLM) is unavailable AND Edge Gemma orchestrates a cloud call for task types: `orchestration`, `translation`, `ui_planning`, `formatting`. All payloads MUST pass Decree 13 PII redaction gate (see §5.4) before leaving hospital boundary. Consult mode: `tier_2`.
|
||||
- **Tier 3** → Modal MedGemma (clinical deep-reasoning & report finalization — **PoC ONLY under NFR-16a**). Activated when RAG confidence < threshold, radiologist explicitly requests "clinical depth", or Edge Gemma detects complex medical reasoning requiring clinical expertise. Same NFR-16a governance as Tier 2. Consult mode: `tier_3`. Output validated by RAG-Referee before reaching radiologist.
|
||||
- **Tier 4** → MOH guideline template responses (rule-based, no generative model, always available, deterministic, cited — safety floor when all inference tiers fail). Consult mode: `tier_4`.
|
||||
- **Circuit Breaker**: PWA Feature Flag Manager (`consult_mode` state machine) attempts tiers sequentially: `tier_1` → `tier_2` → `tier_3` → `tier_4`. Emits tier-transition events to on-prem immutable audit log (NFR-17). Tier 2/3 transitions require consent token + redaction manifest committed *before* the HTTP call. CDN fallback events (model weight distribution only) log lighter audit entry: no redaction required, no consent required (weights carry no PII).
|
||||
- **GCP CDN configuration** (emergency distribution path only):
|
||||
- Cloud Storage bucket: `gs://vkist-models-{site_id}` — Uniform bucket-level access, no `allUsers` public exposure.
|
||||
- Signed-URL generation: FastAPI generates time-limited signed URLs (signed by project service account private key, CSEK-managed) with a 15-minute expiry; returned to PWA as a `307` redirect on intranet CDN failure.
|
||||
@@ -133,7 +137,7 @@ The system adopts a hybrid edge-cloud (on-premise) architecture with the followi
|
||||
- **JWT session state** (TTL 1h, matches token expiry): validates active sessions without Postgres roundtrip; coalesced look-aside with single-flight mutex to prevent thundering herd on session-miss.
|
||||
- **MOH guideline chunks** (TTL 7d): version-key invalidation triggered by Postgres NOTIFY on guideline ingestion events; cache key = `guideline:{version}:{chunk_id}`, full key-space retired on version bump, no TTL races.
|
||||
- **DICOM metadata per session** (TTL 12h, per-session scope): avoids repeated Postgres lookups of DICOM headers during the 3-step ML pipeline; evicted after session completion.
|
||||
- **Circuit-breaker / consult_mode state** (TTL 2h, matches session): consult_mode enum (tier_1/tier_2/tier_3a/tier_3b) keyed by session_hash. Written atomically on tier transitions. Read by: FastAPI routing layer, PWA (via SSE status push), Prometheus metrics exporter. Justifies Redis over Postgres: sub-ms reads required, writes are frequent (per BERT token stream during Socratic dialogue), data is volatile (expires with session), and multiple FastAPI workers behind NGINX need shared visibility — in-process memory cannot cross worker boundaries.
|
||||
- **Circuit-breaker / consult_mode state** (TTL 2h, matches session): consult_mode enum (tier_1/tier_2/tier_3/tier_4) keyed by session_hash. Written atomically on tier transitions. Read by: FastAPI routing layer, PWA (via SSE status push), Prometheus metrics exporter. Justifies Redis over Postgres: sub-ms reads required, writes are frequent (per BERT token stream during Socratic dialogue), data is volatile (expires with session), and multiple FastAPI workers behind NGINX need shared visibility — in-process memory cannot cross worker boundaries.
|
||||
- **Rate-limit counters** (TTL 30s, sliding window): INCR on consult request entry. Self-clearing via EXPIRE. Prevents Triton GPU saturation under concurrent burst from multiple radiologists. Atomic Redis operation; Postgres row-lock alternative creates write amplification on hot path.
|
||||
- **Not cached in Redis** (served from primary store):
|
||||
- Audit log: append-only, NFR-17 immutability requirement — no intermediate cache layer.
|
||||
@@ -248,9 +252,9 @@ The system adopts a hybrid edge-cloud (on-premise) architecture with the followi
|
||||
- **Application**: NeMo Guardrails and GuardrailsAI require substantial runtime scaffolding, rule engines, and WASM-compatible bindings that do not exist in the browser ecosystem at the model inference layer. The client-side cannot run a NeMo Rail or GuardrailsAI pipeline inside a WebWorker alongside WebLLM. Therefore, the edge guardrail tier relies on two lightweight mechanisms:
|
||||
1. **Robust prompt-based rules**: Hard system-prompt constraints injected at every consult turn. Boundaries: no diagnosis outside MOH protocol scope; mandatory citation for every clinical assertion; no free-text echoing of PII even if user inputs it; immediate refusal to follow instructions that attempt prompt injection or jailbreak; tone and scope locked to clinical decision-support.
|
||||
2. **BERT hallucination / mal-intention detection**: Each candidate LLM token stream (or per-turn summary) is evaluated by a local Transformers.js BERT classifier running in a dedicated `guardrail.worker.ts` WebWorker. The classifier scores for: (a) hallucination probability (factual grounding vs retrieved MOH context); (b) mal-intention / prompt-injection indicators (user-context leaking, instruction override attempts, jailbreak markers); (c) scope violation (claims outside synovitis/MSK MOH guidelines).
|
||||
- **Session termination and cloud mitigation**: If BERT edge guardrail returns a violation score above threshold on any axis → FastAPI immediately terminates the WebLLM session (`stream.close()`), logs the trigger event to immutable audit, and opens a new mitigated consult session toward Tier 3a (cloud Vertex AI MedGemma). Mitigation path: the same user query + retrieved MOH chunks (already computed) + edge-detected-offense log are forwarded under NFR-16a governance (consent token confirmed, Decree 13 redaction re-applied, audit-log commit before egress). The radiologist sees: "Edge model restricted — escalating to cloud consult (redacted)."
|
||||
- **WebWorker isolation**: `guardrail.worker.ts` runs in a separate WebWorker from `cv.worker.ts` (LiteRT) and `llm.worker.ts` (WebLLM). No shared WASM memory between workers. IndexedDB caches artifacts but is never injected into LLM context. Unload order on memory pressure: `llm.worker.ts` → `guardrail.worker.ts` → `cv.worker.ts`.
|
||||
- **Cloud model guardrail**: Cloud Vertex AI MedGemma (Tier 3a/3b) uses natively supported Vertex AI Model Garden safety filters (tuned for healthcare) plus the same server-side FastAPI BERT RAG-Referee gate. No additional client-side framework required.
|
||||
- **Session termination and cloud mitigation**: If BERT edge guardrail returns a violation score above threshold on any axis → FastAPI immediately terminates the WebLLM session (`stream.close()`), logs the trigger event to immutable audit, and routes to the appropriate cloud tier based on `task_type`: orchestration/translation tasks route to Tier 2 (Gemini), clinical deep-reasoning/report finalization routes to Tier 3 (MedGemma). Mitigation path: the same user query + retrieved MOH chunks (already computed) + edge-detected-offense log are forwarded under NFR-16a governance (consent token confirmed, Decree 13 redaction re-applied, audit-log commit before egress). The radiologist sees: "Edge model restricted — escalating to cloud consult (redacted)."
|
||||
- **WebWorker isolation**: `guardrail.worker.ts` runs in a separate WebWorker from `cv.worker.ts` (LiteRT) and `llm.worker.ts` (WebLLM). No shared WASM memory between workers. IndexedDB caches artifacts but is never injected into LLM context. Unload order on memory pressure: `llm.worker.ts` → `guardrail.worker.ts` → `cv.worker.ts`.
|
||||
- **Cloud model guardrail**: Cloud models use the server-side FastAPI BERT RAG-Referee gate as the primary safety mechanism. Tier 2 (Gemini) uses Vertex AI Model Garden safety filters (tuned for healthcare). Tier 3 (MedGemma) output is additionally validated by RAG-Referee before reaching the radiologist. No additional client-side framework required.
|
||||
|
||||
### 5.8 Edge Data Hygiene: Anonymization, Redaction & Server-Side Ground-Check
|
||||
- **Pattern**: Edge-to-Server Redaction Pipeline with Ground-Truth Verification
|
||||
@@ -269,22 +273,23 @@ The system adopts a hybrid edge-cloud (on-premise) architecture with the followi
|
||||
### 5.9 RAG as Essential Pipeline Step (Non-Optional)
|
||||
- **Pattern**: Mandatory RAG Pre-Processing for All LLM Consult Paths
|
||||
- **Source**: NFR-18 (100% LLM text cites MOH protocol); NFR-12 (zero-friction explainability)
|
||||
- **Application**: RAG retrieval is NOT an optional tool the LLM may choose to invoke. It is a mandatory pipeline step that executes before any LLM generation, for both edge (browser WebLLM) and cloud (Vertex AI MedGemma) tiers.
|
||||
- **Application**: RAG retrieval is NOT an optional tool the LLM may choose to invoke. It is a mandatory pipeline step that executes before any LLM generation, for all three-tier systems: Edge Gemma (browser WebLLM), Gemini (GCP Vertex AI), and MedGemma (Modal).
|
||||
- **Pipeline order**:
|
||||
1. User query arrives (post-redaction)
|
||||
2. `rag.query()` retrieves top-k MOH guideline chunks from pgvector HNSW
|
||||
3. Retrieved chunks + strict system prompt ("You are a clinical decision-support assistant. Answer ONLY using the provided MOH guidelines. Cite chunk IDs. Do not speculate.") → injected into LLM context
|
||||
4. LLM generates grounded, cited response
|
||||
5. `referee.validate()` (BERT RAG-Referee) checks citation correctness, logical cohesion, factual contestant status
|
||||
5. `referee.validate()` (BERT RAG-Referee) checks citation correctness, logical cohesion, factual contestant status → mandatory for MedGemma (Tier 3) output; optional but recommended for Gemini (Tier 2); lightweight validation for Edge Gemma (Tier 1)
|
||||
6. If any axis fails → reject → fallback to MOH template
|
||||
- **Rationale**: If RAG were merely a tool call, a compromised or under-performing edge LLM could choose to skip retrieval and generate ungrounded text, violating NFR-18. Mandatory RAG pre-processing closes this vector.
|
||||
|
||||
### 5.10 Tool-Calling Semantics for Edge and Cloud LLMs
|
||||
- **Pattern**: Function-Calling as Convenience Layer Over Mandatory RAG
|
||||
- **Source**: Gemma Functions (Google DeepMind) + Vertex AI Function Calling
|
||||
- **Source**: Gemma Functions (Google DeepMind) + Vertex AI Function Calling + Modal MedGemma Function Calling
|
||||
- **Application**:
|
||||
- **Edge LLM (GemmaE2B via WebLLM)**: Supports Gemma Functions. The RAG retrieval is exposed as a function tool (`retrieve_moh_guidelines(query, top_k)`). HOWEVER, this tool is pre-invoked by the host pipeline per §5.9 before the first LLM token is generated. The function-calling interface is available for subsequent turns of Socratic dialogue where the model may need additional retrieval mid-conversation (e.g., user asks "what about contraindications?" → mid-turn retrieval). The host pipeline enforces the mandatory RAG call on turn 0; subsequent per-turn retrieval is model-initiated via function calling.
|
||||
- **Cloud LLM (MedGemma on Vertex AI)**: Native Vertex AI Function Calling is enabled. Same semantics: server-side FastAPI injects top-k MOH chunks before first generation; model may request further retrieval via function call during multi-turn consult.
|
||||
- **Edge LLM (Edge Gemma via WebLLM)**: Supports Gemma Functions. The RAG retrieval is exposed as a function tool (`retrieve_moh_guidelines(query, top_k)`). HOWEVER, this tool is pre-invoked by the host pipeline per §5.9 before the first LLM token is generated. The function-calling interface is available for subsequent turns of Socratic dialogue where the model may need additional retrieval mid-conversation (e.g., user asks "what about contraindications?" → mid-turn retrieval). The host pipeline enforces the mandatory RAG call on turn 0; subsequent per-turn retrieval is model-initiated via function calling.
|
||||
- **Cloud LLM Tier 2 (Gemini on Vertex AI)**: Native Vertex AI Function Calling is enabled for orchestration/translation tasks. Same semantics: server-side FastAPI injects top-k MOH chunks before first generation; model may request further retrieval via function call during multi-turn consult.
|
||||
- **Cloud LLM Tier 3 (MedGemma on Modal)**: Native function-calling interface for clinical reasoning tasks. Server-side FastAPI injects top-k MOH chunks and RAG-Referee validation results before first generation; model may request further retrieval via function call during multi-turn clinical consult.
|
||||
- **Why not pure tool-calling**: Pure tool-calling delegates the decision to retrieve to the model. NFR-18 requires 100% citation; delegating retrieval to model choice creates a non-zero probability of unsupported generation. Mandatory pre-injection + optional per-turn tool call is the safer contract.
|
||||
|
||||
### 5.11 Output Filtering
|
||||
@@ -304,6 +309,11 @@ The system adopts a hybrid edge-cloud (on-premise) architecture with the followi
|
||||
- `edge_guardrail_violation`: session_hash, violation_axis (hallucination|mal_intention|scope_breach), bert_score, user_query_hash, mitigation_target (cloud|template).
|
||||
- `rag_retrieval_event`: session_hash, query_hash, retrieved_chunk_ids, latency_ms.
|
||||
- `referee_decision`: session_hash, axis_results (attribution|cohesion|contestant), overall_pass, fallback_triggered.
|
||||
- `egress_consent_gemini`: session_hash, user_id, task_type, redaction_hash — before Gemini cloud call.
|
||||
- `egress_consent_medgemma`: session_hash, user_id, task_type, redaction_hash — before MedGemma cloud call.
|
||||
- `egress_response_gemini`: session_hash, task_type, status, ts — after Gemini response.
|
||||
- `egress_response_medgemma`: session_hash, task_type, status, ts — after MedGemma response.
|
||||
- `cloud_llm_escalation`: session_hash, from_tier, to_tier, reason (reroute, referee_fail, token_limit) — tracks Gemini→MedGemma or any cross-tier escalation.
|
||||
- **Audit events for this layer**: `redaction_edge_manifest` (client), `redaction_server_refine` (server-side cleaning pass), `redaction_failure` (server unable to clean → error), `egress_consent` + `egress_redact_manifest` (before cloud call, NFR-16a).
|
||||
- **Retention**: Audit events retained indefinitely (append-only, no UPDATE/DELETE). Raw LLM token streams retained for 90 days in MinIO (compressed, AES-256 at rest) for drift analysis. After 90 days, streams are deleted; only aggregated `referee_decision` and `rag_retrieval_event` summaries remain.
|
||||
- **BERT drift monitor telemetry**: edge BERT classifier score distributions exported to Prometheus; alert if false-positive rate on hallucination detection exceeds 5% or false-negative rate exceeds 0.1% (clinical safety threshold).
|
||||
|
||||
Reference in New Issue
Block a user