# VKIST MSK Ultrasound Stack — Investment Proposal & System Specification > **Audience:** CTO / Department Head / Executive Sponsor (technical-fluent, outcome-driven, risk-averse on healthcare investments) > **Author:** VKIST MSK Pilot Engineering Team > **Status:** Pilot proposal — active cycle June 2 – September 2, 2026 (strict 3-month window) > **Reading time:** Section 0 (Executive Summary) is a self-contained one-pager. Sections 1–8 provide the full technical and financial justification. --- ## Table of Contents 0. [Executive Summary (one page)](#0-executive-summary-one-page) 1. [Project Context — Who, What Pain, What Constraints](#1-project-context) 2. [How the System Addresses the Problem](#2-how-the-system-addresses-the-problem) 3. [Qualitative ROI — Behavioral & Workflow Change](#3-qualitative-roi--behavioral--workflow-change) 4. [Quantitative ROI — Projected Metrics & Measurement Plan](#4-quantitative-roi--projected-metrics--measurement-plan) 5. [Risk & Failure-Mode Handling — Why This Is Not Risky](#5-risk--failure-mode-handling--why-this-is-not-risky) 6. [Compliance & Data Governance](#6-compliance--data-governance) 7. [Roadmap & Milestones](#7-roadmap--milestones) 8. [Investment Ask & Scope Boundaries](#8-investment-ask--scope-boundaries) --- ## 0. Executive Summary (one page) **The problem, in one sentence.** Vietnamese public-hospital musculoskeletal (MSK) care runs on clinicians processing 100+ evaluations per shift with no time for objective double-checks, no shared visibility across the radiologist → surgeon → physiotherapist → patient chain, and patients so information-starved that they turn to dangerous folk remedies — while VKIST's proven computer-vision research models sit idle as static academic checkpoints. **What we are building.** The **VKIST MSK Ultrasound Stack** is an **air-gapped-first, single-hospital AI platform** that turns knee ultrasound scans into automated, explainable **synovitis grading (0–3)** with Vietnamese-language reasoning, every clinical statement **cited back to Ministry of Health (MOH) guidelines**, and a **human clinician signature required** before anything is finalized. It migrates VKIST's existing DeepLabv3/UNet/ConvNeXt vision models from research into a referenceable, production-grade medical software ecosystem. **Why it is a sound investment (the four pillars):** | Pillar | What it means for the leader | | :--- | :--- | | **Zero-friction adoption** | The AI is an *invisible background validation layer* — 0 extra clicks to see explanations (NFR-12/13). Clinicians keep their workflow; the system removes liability risk instead of adding work. | | **Non-negotiable safety** | Human-in-the-Loop signature gate (NFR-19), 100% MOH-cited explanations (NFR-18), immutable audit log (NFR-17), and a 4-quadrant safety framework that explicitly defends against automation bias *and* AI hallucination. | | **Regulatory sovereignty** | Runs inside the hospital LAN. Decree 13/2023 PII protection + Circular 46/2018 EMR compliance are built into the architecture, not bolted on. No patient data leaves the hospital in production. | | **Capital efficiency** | Built on proven open-source (K3s, Triton, Postgres/pgvector, Keycloak, MinIO) and existing VKIST models. No SaaS lock-in, no per-seat licensing, delivered in a fixed 3-month pilot. | **The ask.** Sponsorship of a fixed 3-month pilot on a single hospital's on-premise infrastructure (one Dell PowerEdge class node + one GPU node), a small cross-functional team, and a *governed, time-boxed* cloud allowance for the proof-of-concept LLM tiers (NFR-16a, retired at sign-off). See [Section 8](#8-investment-ask--scope-boundaries). **The expected return.** A validated, referenceable clinical AI product that (projected) removes tedious measurement work from overloaded specialists, gives physiotherapists diagnostic visibility they have never had, converts patient anxiety into institutional trust, and establishes VKIST's operational credibility in the local healthcare community — with every claim instrumented and measurable during the pilot itself (see [Section 4](#4-quantitative-roi--projected-metrics--measurement-plan)). > **Bottom line for the decision-maker:** This is a low-blast-radius, standards-compliant, single-site pilot that converts already-funded research into a safe, auditable clinical tool. The downside is bounded (one hospital, 3 months, open-source stack); the upside is a defensible product line and clinical-community trust. --- ## 1. Project Context ### 1.1 Who the System Is Designed For The platform coordinates a **four-persona clinical care chain**. The pilot (Sprint 1–2, FR-25) is centered on the **Diagnostic Radiologist (UP5)** as the primary human actor, with the other three personas as governance/future-sprint stakeholders whose needs shape the non-functional requirements. | Persona | Who they are | Environment | Core responsibility | What the system must respect | | :--- | :--- | :--- | :--- | :--- | | **UP5 — Diagnostic Radiologist** | Specialist MDs, age 28–55, from digitally native attendings to department heads | Low-light reading rooms; massive daily DICOM backlogs; asynchronous, detached from patients | Translate images into definitive reports that gate all downstream care | *Invisible workflow preservation* — no extra clicks, no DICOM render lag, no uncompensated patient chat | | **UP7 — Rheumatologist / Orthopedic Surgeon** | Treatment architects, age 30–60; seniors control procurement | Split across OR, chaotic outpatient clinics (>100 patients/shift), inpatient wards | Synthesize fragmented data into a treatment plan; bear full legal/clinical liability | *Clinical force multiplier* — rapid dashboards, automated patient-education visuals, strictly gated communication | | **UP6 — Physical Therapist** | Young (36–40% aged 20–29), highly stratified (≈54% vocational tier), digitally literate | Cramped (<20 m²), high-noise public units; 11–20+ patients/shift; shared/scarce workstations | Execute physician scripts into kinetic rehab programs | *Digital exoskeleton* — gel-proof knuckle-tap UI, zero-GPU mobile, localized language, objective progress data | | **UP8 — MSK Patient & Family Caregiver** | Elderly patients (45–80+) via younger caregiver proxies (20–45) | Home, pharmacy, crowded waiting rooms; saturated with health misinformation | Manage chronic pain; adhere to protocols; make family-centric decisions | *Empathy + accessibility* — jargon-free, high-contrast, large-font, dual-profile family access on legacy phones | *Source: [User_Research_Result.md](../Requirement_Analysis/USER_RESEARCH/User_Research_Result.md), [CONTEXT_VISION_SCOPE.md §3](../PLAN/CONTEXT_VISION_SCOPE.md).* ### 1.2 The Pain Points That Drive the Need These are not hypothetical — they are drawn from ethnographic user research (professional forums, patient forums, and published studies). - **Clinical exhaustion & volume:** Central referral hospitals (Bach Mai, Viet Duc) regularly exceed **100 evaluations per shift**, leaving no time for tedious measurements or objective double-checks. This creates the risk of *automation-bias blind-concurrence* — a clinician rubber-stamping an AI estimate under time pressure. - **The physiotherapist information bottleneck:** PTs receive only brief text prescriptions — no DICOM, no scan findings — forcing **semi-blind therapeutic execution** against estimated surface anatomy. - **The physiotherapist body & literacy crisis:** **76.4%** WMSD prevalence (career-shortening physical strain); **84%** report a foreign-language barrier (only 25% adequate English); 96%/93%/86% rely on personal experience/peer chat/old textbooks rather than formal evidence. - **Dangerous folk interventions:** Information-starved patients turn to leaf-wrapping or violent manual adjustment, causing tissue infection or permanent anatomical failure — because no one translated their scan into understandable terms. - **Stranded research value:** VKIST's CV models (DeepLabv3/UNet/ConvNeXt/MedViT) are proven in the lab but have no production pathway, so they generate zero operational or reputational value. ### 1.3 Constraints the System Must Meet **Functional Requirements (pilot scope — FR-25 pipeline):** | FR | Requirement | | :--- | :--- | | **FR-23/24/25** | Segment knee joint structures → measure synovium thickness (mm) → grade synovitis **0–3** | | **FR-25 / Grad-CAM** | Spatial activation overlay on the primary viewport with **zero extra clicks** | | **FR-25 / Safety loop** | Conversational Circuit Breaker + Socratic dialogue before finalization; BERT drift monitor; RAG-Referee arbitration | | **FR-25 / Reporting** | Circular 46/2018 compliant PDF report generation | | **FR-25 / Finalization** | Cryptographically signed, HITL-gated electronic record → EMR | **Non-Functional Requirements (the "why it won't fail" constraints):** | NFR | Constraint | Why the leader cares | | :--- | :--- | :--- | | **NFR-4** | ≤ 150 MB idle app bundle | Runs on cheap, legacy hospital/consumer hardware | | **NFR-5** | ≤ 1.5 s core inference (on-prem) | No workflow slowdown | | **NFR-7** | ≤ 200 ms token streaming (TTFT) | Feels instantaneous to the clinician | | **NFR-9** | ≥ 99.9% availability (≤ 45 s downtime/day) | Clinically dependable | | **NFR-10** | 100% of patient-facing LLM text passes safety verification | No unsafe generative output reaches a human | | **NFR-11** | Onboarding ≤ 45 minutes | Cheap to roll out to busy staff | | **NFR-12/13** | 0 extra clicks for explainability; native spatial activation maps | Anti-black-box; builds clinician trust | | **NFR-14** | No client-side GPU/accelerator required (Android 10+, 3 GB RAM) | Works on the fleet that actually exists | | **NFR-15** | Circular 46/2018 EMR compliance | Legally deployable | | **NFR-16** | Air-gapped primary (no PHI over public internet in production) | Data sovereignty (Decree 13) | | **NFR-17** | Immutable audit log | Accountability & liability defense | | **NFR-18** | 100% LLM clinical text cites MOH protocol via RAG | Every claim is traceable & defensible | | **NFR-19** | HITL digital signature before FINALIZED/ARCHIVED | A licensed human always owns the decision | *Source: [SOLUTION_ARCHITECTURE_SPEC.md §2](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md), [SOFTWARE_ARCHITECTURE_SPEC.md §2](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md).* ### 1.4 System Context — C4 Level 1 (PlantUML) The context diagram summarizes *who* and *what* the system boundary talks to. The pilot activates UP5 (and governance actors UP1/UP4); UP6/UP7/UP8 are shown as future-sprint boundaries. ```plantuml @startuml VKIST_MSK_System_Context !include title System Context Diagram — VKIST MSK Ultrasound Stack (Pilot: FR-25 Synovitis Grading) ' === PRIMARY HUMAN ACTOR (in pilot scope) === Person(radiologist, "Diagnostic Radiologist (UP5)", "Loads knee ultrasound, reviews AI grading (0-3), confirms/overrides, engages Socratic circuit breaker, finalizes & signs the report, reads GradCAM + RAG evidence") Person(admin, "System Administrator", "Deploys/monitors on-prem K3s, model updates, observability, NGINX failover") ' === GOVERNANCE / NFR-ALIGNMENT ACTORS === Person_Ext(senior_expert, "Healthcare Senior Expert (UP1)", "Clinical protocol validation, model-threshold approval, pilot sponsor") Person_Ext(support_staff, "Support Staff (UP4)", "Patient registration, case-queue management") ' === FUTURE-SPRINT PERSONAS (out of pilot scope) === Person_Ext(therapist, "Physical Therapist (UP6)", "Future: read-only diagnostic visibility, kinetic overlays, progress tracking") Person_Ext(surgeon, "Surgeon / Rheumatologist (UP7)", "Future: aggregated dashboards, treatment planning") Person_Ext(patient, "Patient & Caregiver (UP8)", "Future: patient portal, plain-language education, self-monitoring") ' === THE SYSTEM === System(mpps, "VKIST MSK Ultrasound Stack (FR-25)", "Air-gapped-first: knee ultrasound AI (angle -> inflammation -> segmentation), synovitis grading 0-3 with GradCAM, 4-quadrant clinical safety, MOH-cited Vietnamese explanations, HITL-signed Circular 46 reports") ' === EXTERNAL SYSTEMS === System_Ext(pacs, "Hospital PACS / Ultrasound Machine", "DICOM source (C-MOVE + direct capture)") System_Ext(emr, "Hospital EMR / HIS", "Signed report storage, prescription/lab sync (HL7/FHIR)") System_Ext(triton, "Triton Inference Server", "GPU node: 3-step ML pipeline + RAG embeddings (gRPC)") System_Ext(knowledge, "Clinical Knowledge Systems", "ladybugDB ontology (SNOMED-CT), pgvector MOH guideline index, Gemma/MedGemma LLM tiers") ' === RELATIONSHIPS === Rel(radiologist, mpps, "Reviews grade, overrides, finalizes & signs, reads explanations", "HTTPS") Rel(admin, mpps, "Deploys, monitors, updates models", "HTTPS / SSH") Rel(senior_expert, mpps, "Validates protocols, approves thresholds", "HTTPS (governance)") Rel(support_staff, mpps, "Registration, case queue", "HTTPS") Rel(therapist, mpps, "Future sprint scope", "—") Rel(surgeon, mpps, "Future sprint scope", "—") Rel(patient, mpps, "Future sprint scope", "—") Rel(mpps, pacs, "Imports DICOM", "DICOM / C-MOVE") Rel(mpps, emr, "Pushes signed reports + audit records", "HL7/FHIR") Rel(mpps, triton, "ML inference offload", "gRPC 8001") Rel(mpps, knowledge, "RAG retrieval, ontology traversal, LLM generation", "SQL (pgvector) + in-process") @enduml ``` *Adapted from [SOLUTION_ARCHITECTURE_SPEC.md §8.1](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md).* ### 1.5 Applicability — Use-Case View (PlantUML) The pilot's functional scope is organized around a **4-Quadrant Human-AI Interaction Framework** that deliberately defends against both *automation bias* (blindly trusting AI) and *clinician subservience* (AI overriding a correct human). This is the applicability model — what the radiologist can actually *do* with the system. ```plantuml @startuml VKIST_MSK_UseCases left to right direction skinparam packageStyle rectangle actor "Diagnostic Radiologist (UP5)" as Rad actor "MSK Vision Grader" as Grader <> actor "LLM Explainer" as LLM <> actor "RAG-Referee Arbitrator" as Referee <> actor "Hospital EMR" as EMR <> rectangle "VKIST MSK Workspace (FR-25)" { usecase "UC-48376\nLoad Patient Scan Session" as U1 usecase "UC-47988\nReview Suggested Synovitis Grade (0-3)" as U2 usecase "UC-92006\nFinalize & Sign Electronic Record" as U3 rectangle "Q1 — True Agreement (AI correct / Doctor correct)" { usecase "UC-25776\nGradCAM + CoT Explanation" as Q1a usecase "UC-02423\nLog High-Trust Concur Block" as Q1b } rectangle "Q2 — Automation Override (AI correct / Doctor oversight)" { usecase "UC-22159\nTrigger Circuit Breaker" as Q2a usecase "UC-55146\nSocratic Reasoning Dialogue" as Q2b usecase "UC-74821\nBERT Context-Drift Monitor" as Q2c usecase "UC-65473\nArbitrate via RAG-Referee" as Q2d } rectangle "Q3 — Clinician Subservience (AI hallucinates / Doctor correct)" { usecase "UC-25637\nExpose Pixel-Level Activation" as Q3a usecase "UC-60739\nIsolate Visual Noise/Artifacts" as Q3b usecase "UC-62864\nCommit Validated Ground-Truth" as Q3c } rectangle "Q4 — Double-Blind Failure (AI faulty / edge case)" { usecase "UC-35956\nActivate Clinical Investigation Mode" as Q4a usecase "UC-47796\nStructured Morphology Annotation" as Q4b usecase "UC-01580\nSerialize to Telemetry Queue" as Q4c } } Rad --> U1 Rad --> U2 Rad --> U3 Rad --> Q2b : Argue observations Rad --> Q3b : Tag artifacts Rad --> Q4b : Document manual findings Grader --> U1 : Feeds vision tensors + scores Grader --> U2 : Generates prediction LLM --> Q1a : Generates clinical CoT LLM --> Q2b : Generates Socratic checks Referee --> Q2d : Retrieves MOH guidelines U3 --> EMR : Syncs signed report Q1b --> EMR : Stores consensus log Q4c --> EMR : Routes anomaly to engineering telemetry U1 ..> Q1a : <> Q1a ..> Q1b : <> U2 <.. Q2a : <> (friction detected) Q2a ..> Q2b : <> Q2a ..> Q2c : <> Q2c ..> Q2d : <> (impasse/drift) U2 <.. Q3a : <> (clinician contests score) Q3a ..> Q3b : <> Q3b ..> Q3c : <> U2 <.. Q4a : <> (low confidence + empty RAG) Q4a ..> Q4b : <> Q4b ..> Q4c : <> @enduml ``` *Adapted from [full_usecase_planuml.md](../../workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/Use_Case/full_usecase_planuml.md) and [CONTEXT_FR_25_UC_ELIT.md](../Requirement_Analysis/UC_Design/FR_25_UC_DESIGN/CONTEXT_FR_25_UC_ELIT.md).* --- ## 2. How the System Addresses the Problem ### 2.1 Technology Choices — Problem → Solution → Why Optimal → Limitation Each technology is selected against a specific user problem or NFR, not for novelty. The table states the honest limitation of each choice so the leader sees a balanced picture. | Technology | User problem / NFR it solves | Why it is the optimal choice | Limitation / trade-off | | :--- | :--- | :--- | :--- | | **React PWA + LiteRT (MediaPipe) + WebLLM edge** | NFR-14 (no client GPU), NFR-4 (150 MB), zero-friction on legacy phones | Installable, offline-capable, runs a local angle pre-classifier + edge LLM with no server round-trip; degrades gracefully on weak hardware | Browser memory ceilings; edge LLM is small (GemmaE2B-Q4 ~1.3 GB) so deep reasoning must escalate | | **FastAPI edge-inference-svc (Python 3.12)** | FR-25 pipeline orchestration, Decree 13 middleware, SSE streaming | Async, lightweight, tight control over PII redaction + audit at the boundary; in-house domain logic | Python GIL for CPU-bound work — mitigated by offloading inference to Triton | | **Triton Inference Server (ONNX/TensorRT)** | NFR-5 (≤1.5 s inference), reuse of VKIST CV models | Industry-standard, GPU-efficient, self-hosted (no SaaS); serves the 3-step ensemble + embeddings | Requires a GPU node (capex); one more component to operate | | **Postgres + pgvector (HNSW)** | NFR-18 MOH-cited RAG, transactional consistency of clinical records | One store for records *and* vectors (~15K guideline vectors) → ~5–20 ms retrieval, no separate vector DB to run | Not optimal beyond ~100K vectors / 500 QPS (documented Phase-2 path to Qdrant) | | **RAG-Referee (BERT classifier)** | NFR-10/18 — reject unsupported LLM claims | Validates attribution + logical cohesion of every explanation before it reaches a human | Adds latency to the cloud tier; thresholds need clinical tuning | | **3-Tier LLM (Edge Gemma → Gemini → MedGemma)** | Vietnamese explanations, deep clinical reasoning, cost control | Cheapest-capable-tier routing: local first, cloud only when needed, <20% MedGemma usage cap | Cloud tiers are PoC-only (NFR-16a) and must be redacted — retired at sign-off | | **Keycloak (OIDC/RBAC) on-prem** | NFR-16 sovereignty, role separation (PT read-only) | No SaaS identity; realm-scoped RBAC inside the LAN | Operational overhead vs. managed Auth0/Okta (accepted for sovereignty) | | **K3s on Dell PowerEdge** | NFR-16 air-gap, NFR-9 availability | Lightweight Kubernetes for a single hospital; rolling updates, self-healing pods; NGINX+Keepalived VIP failover (≤2 s) | Single-site scale; not a multi-region HA design (by design for the pilot) | | **MinIO + Redis (5 scoped types)** | Object/blob store + session/cache | S3-compatible, self-hosted, no cloud egress; Redis scoped to prevent sprawl | Self-managed durability (mitigated by backup cron to MinIO) | **Honest, system-level limitations (stated up front for the leader):** - **Single-hospital scale by design.** The pilot targets one site; multi-site federation is a documented Phase-2 evolution, not a pilot promise. - **Cloud LLM tiers are proof-of-concept only** under NFR-16a, gated by redaction + consent + audit, and **formally retired at PoC sign-off** — production reverts to fully air-gapped NFR-16. - **Edge model is intentionally small** — deep reasoning escalates through the tier chain, which is why the cost guard (<20% MedGemma) and RAG-Referee exist. *Source: [SOFTWARE_ARCHITECTURE_SPEC.md §7 Build-vs-Buy](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md), [SOLUTION_ARCHITECTURE_SPEC.md §5](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md).* ### 2.2 Deployment Environment — C4 Container Diagram (PlantUML) The system runs entirely inside the hospital LAN. The only external touchpoints are the governed, PoC-scoped cloud fallbacks (NFR-16a). ```plantuml @startuml VKIST_MSK_Containers !include title C4 Container Diagram — VKIST MSK Ultrasound Stack Person(radiologist, "Diagnostic Radiologist (UP5)", "Loads DICOM, reviews grading, finalizes reports") Person(admin, "System Administrator", "K3s ops, model updates, observability, failover") Person_Ext(senior_expert, "Healthcare Senior Expert (UP1)", "Protocol validation, threshold approval") Person_Ext(support_staff, "Support Staff (UP4)", "Registration, case queue") System_Boundary(hospital_lan, "Hospital LAN (Air-Gapped, <=10 Mbps)") { Container(pwa, "React PWA Frontend", "React 18, TS, Zustand, LiteRT, MediaPipe, Dexie.js", "View-angle validation (WASM); DICOM preview; Grad-CAM overlay; Decree 13 scrubber; encrypted IndexedDB sessions") Container(nginx, "Active-Passive Gateway", "NGINX + Keepalived", "SSL termination, VIP load balancing, <=2 s failover") System_Boundary(k3s_cluster, "K3s Orchestration Cluster") { Container(edge_inference, "Edge Inference Service", "FastAPI (Python 3.12)", "DICOM ingest, 3-step ML orchestration, Grad-CAM, report assembly, SSE streaming") Container(cloud_llm_gateway, "Cloud LLM Gateway", "FastAPI (Python 3.12)", "Routes to Gemini (orchestration) / MedGemma (clinical); NFR-16a redaction, consent, audit, cost guard") Container(rag_svc, "RAG & Knowledge Service", "FastAPI (asyncpg)", "pgvector top-k, ladybugDB ontology, RAG-Referee validation, mandatory RAG pre-processing") Container(audit_svc, "Audit & EMR Service", "Node.js / FastAPI Worker", "Immutable append-only audit; HL7/FHIR EMR push with outbox retry") Container(api_gw, "API Gateway", "Envoy", "TLS, rate limiting, routing, OIDC pass-through") Container(auth_svc, "Identity & Access", "Keycloak", "OIDC, RBAC, realm vkist-msk") Container(obs_stack, "Observability Stack", "Prometheus + Grafana", "Metrics, dashboards, alerting") } System_Boundary(db_vm, "Database VM") { ContainerDb(postgres, "PostgreSQL + pgvector", "PostgreSQL 16 + pgvector", "EMR records, spatial markers, MOH HNSW index, embeddings, audit ledger") ContainerDb(redis, "Redis Cache", "Redis 7 (AOF+RDB)", "Sessions, guideline chunks, DICOM metadata, consult_mode state, rate-limit") ContainerDb(minio, "MinIO Object Store", "MinIO S3-compatible", "DICOM payloads, overlays, Grad-CAM heatmaps, report PDFs, model staging") } } System_Ext(triton, "Triton Inference Server", "NVIDIA Triton (ONNX/TensorRT)", "3-step ML pipeline + EmbeddingGemma RAG embeddings") System_Ext(emr, "Hospital EMR / HIS", "HL7/FHIR", "Signed report storage, prescription sync") System_Ext(pacs, "PACS / Ultrasound Device", "DICOM / C-MOVE", "Image capture & retrieval") System_Ext(vertex_ai, "GCP Vertex AI (Gemini)", "REST", "PoC-only NFR-16a: orchestration, translation; redacted payloads only") System_Ext(modal_medgemma, "Modal MedGemma", "T4 GPU", "PoC-only NFR-16a: clinical deep-reasoning; redacted + RAG-Referee validated") Rel(radiologist, pwa, "Loads scan, reviews grade, finalizes, reads explanations", "HTTPS 443") Rel(admin, nginx, "Deploys, monitors, configures", "HTTPS / SSH") Rel(senior_expert, pwa, "Validates protocols, approves thresholds", "HTTPS 443") Rel(support_staff, pwa, "Registration, case queue", "HTTPS 443") Rel(pwa, nginx, "API requests + pre-validated DICOM", "HTTPS 443") Rel(nginx, api_gw, "Routes upstream", "HTTP 8000") Rel(api_gw, edge_inference, "Forwards /api/*", "HTTP 8000") Rel(api_gw, rag_svc, "Forwards /rag/*", "HTTP 8001") Rel(api_gw, auth_svc, "Validates OIDC tokens", "HTTP 8080") Rel(edge_inference, triton, "3-step inference + embeddings", "gRPC 8001") Rel(edge_inference, postgres, "Reads guidelines; writes cases/embeddings/audit", "SQL 5432") Rel(edge_inference, redis, "Session, rate-limit, consult_mode", "TCP 6379") Rel(edge_inference, minio, "DICOM, overlays, reports", "S3 API") Rel(edge_inference, cloud_llm_gateway, "Routes cloud consult", "HTTP 8000") Rel(cloud_llm_gateway, vertex_ai, "Gemini proxy (NFR-16a redacted)", "HTTPS 443") Rel(cloud_llm_gateway, modal_medgemma, "MedGemma proxy (NFR-16a redacted)", "HTTPS 443") Rel(cloud_llm_gateway, postgres, "Audit log append", "SQL 5432") Rel(rag_svc, postgres, "pgvector HNSW queries", "SQL 5432") Rel(rag_svc, redis, "Guideline cache, pub/sub invalidation", "TCP 6379") Rel(audit_svc, postgres, "Appends immutable audit events", "SQL 5432") Rel(audit_svc, emr, "Finalized report push", "HL7/FHIR") Rel(pwa, pacs, "Direct DICOM capture / C-MOVE", "DICOM") @enduml ``` *Reused from [SOFTWARE_ARCHITECTURE_SPEC.md §4.2](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md).* ### 2.3 Deployment Topology — C4 Deployment Diagram (PlantUML) Physical placement: everything on hospital hardware; the only external node is the governed NFR-16a fallback. ```plantuml @startuml VKIST_MSK_Deployment !include LAYOUT_LEFT_RIGHT() Deployment_Node(hw, "Dell PowerEdge (Hospital)") { Deployment_Node(k3s, "K3s Cluster") { Deployment_Node(pod_edge, "Pod: edge-inference-svc") { Container(ci, "FastAPI", "Python 3.12, Uvicorn") } Deployment_Node(pod_rag, "Pod: rag-svc") { Container(cr, "FastAPI", "Python 3.12, asyncpg") } Deployment_Node(pod_audit, "Pod: audit-svc") { Container(ca, "Node.js", "WAL writer") } Deployment_Node(pod_gw, "Pod: api-gateway") { Container(cg, "Envoy", "TLS, rate limit") } Deployment_Node(pod_auth, "Pod: auth-svc") { Container(ck, "Keycloak", "OIDC, RBAC") } Deployment_Node(pod_obs, "Pod: observability") { Container(cp, "Prometheus") Container(cf, "Grafana") } } Deployment_Node(db_vm, "DB VM") { ContainerDb(pg, "PostgreSQL + pgvector HNSW") ContainerDb(rd, "Redis") ContainerDb(mn, "MinIO") } } Deployment_Node(triton_node, "GPU Node (Triton)", "NVIDIA A10/T4, 24 GB VRAM") { Container(tc, "Triton Inference Server", "ONNX/TensorRT, gRPC 8001") } Deployment_Node(ext, "External (NFR-16a governed, PoC only)") { ContainerDb(s3v, "GCP Vertex AI / Modal MedGemma / CDN fallback") } Rel(ci, pg, "SQL", "TCP 5432") Rel(ci, rd, "Cache", "TCP 6379") Rel(ci, mn, "Blobs", "S3 API") Rel(ci, tc, "Inference", "gRPC 8001") Rel(ci, ck, "Auth", "OIDC") Rel(ci, ca, "Audit", "HTTP") Rel(k3s, db_vm, "SQL / Cache / Blobs", "TCP") @enduml ``` *Reused from [SOFTWARE_ARCHITECTURE_SPEC.md §4.4](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md).* ### 2.4 Components & Interactions — C4 Component Diagram (PlantUML) Inside the core `edge-inference-svc`, each component maps to a specific functional requirement. The **Inference Router** implements the tiered LLM escalation; the **Circuit Breaker** guarantees the system fails safely to MOH templates. ```plantuml @startuml VKIST_MSK_Component_EdgeInference !include Container_Boundary(edge_svc, "edge-inference-svc (FastAPI)") { Component(api, "API Controller", "REST + SSE", "/api/analyze, cloud-orchestrate redirect") Component(stream, "SSE Token Streamer", "StreamingResponse", "200 ms TTFT, 30 s heartbeat") Component(preproc, "Image Preprocessor", "OpenCV + pydicom", "CLAHE, rescale, DICOM header scrub") Component(router, "Inference Router", "consult_mode state", "Tier selection: Edge Gemma -> Gemini -> MedGemma -> Templates") Component(breaker, "Circuit Breaker", "pybreaker", "Wraps Triton + EMR + Cloud LLM; fail-open to templates") Component(pipeline, "ML Pipeline", "gRPC client", "Angle -> Inflammation -> Segmentation -> Measurement") Component(gradcam, "Grad-CAM Generator", "OpenCV", "Spatial activation overlay -> base64 PNG") Component(report, "Report Builder", "WeasyPrint", "Circular 46 PDF; HITL signature gate") Component(rag, "RAG Service", "pgvector SQL", "Top-5 MOH chunks; mandatory pre-generation; NFR-18") Component(referee, "RAG-Referee", "BERT classifier", "Reject LLM text if citation confidence < threshold") Component(nlp, "NLP Scrubber", "Microsoft Presidio", "Re-verify edge redaction; clean residual PII") Component(audit, "Audit Logger", "Append-only writer", "Every tier transition, consent, finalize, override") } Rel(api, stream, "delegates", "SSE") Rel(api, preproc, "validates image", "sync") Rel(api, router, "selects tier", "sync") Rel(router, breaker, "guards calls", "sync") Rel(router, pipeline, "invokes", "gRPC") Rel(router, stream, "fallback text", "SSE") Rel(api, gradcam, "requests overlay", "sync") Rel(api, report, "generates PDF", "sync") Rel(api, rag, "queries MOH", "SQL") Rel(rag, referee, "validates citations", "sync") Rel(api, nlp, "scrubs output", "sync") Rel(api, audit, "writes event", "sync") @enduml ``` The **Cloud LLM Gateway** enforces the NFR-16a governance boundary — nothing egresses without consent, redaction, RAG grounding, and an audit record. ```plantuml @startuml VKIST_MSK_Component_CloudGateway !include Container_Boundary(cloud_gw, "cloud-llm-gateway (FastAPI)") { Component(capi, "Cloud API Controller", "REST + SSE", "/api/cloud-orchestrate, /api/cloud-consult") Component(gateway, "Cloud LLM Router", "task_type matcher", "Orchestration/translation -> Gemini; clinical -> MedGemma") Component(consent, "Consent Enforcer", "Redis + NFR-16a checklist", "Validates consent token + redaction manifest before egress") Component(caudit, "Cloud Audit Emitter", "PostgreSQL append-only", "egress_consent, egress_response, escalation events") Component(cost, "Cost Guard", "Redis + Prometheus", "Tracks MedGemma usage; alerts if >20% over 24h") Component(creferee, "RAG-Referee Trigger", "BERT classifier", "Mandatory validation of MedGemma output") Component(crag, "RAG Pre-processing", "pgvector SQL", "Mandatory top-k injection before generation") } Rel(capi, consent, "verifies", "sync") Rel(capi, gateway, "routes", "sync") Rel(gateway, crag, "injects chunks", "sync") Rel(gateway, creferee, "validates output", "sync") Rel(gateway, caudit, "logs egress", "sync") Rel(gateway, cost, "increments counter", "sync") @enduml ``` *Reused from [SOFTWARE_ARCHITECTURE_SPEC.md §4.3](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md).* --- ## 3. Qualitative ROI — Behavioral & Workflow Change The core value thesis: the system does not add a tool to a workflow — it **enriches the existing workflow and reconnects a fragmented care chain**. The following before/after captures the behavioral change per persona. | Persona | Before (current behavior) | After (with the system) | Behavioral shift | | :--- | :--- | :--- | :--- | | **UP5 Radiologist** | Manual, repetitive measurements under fatigue; risk of missing subtle findings; isolated in a reading room; liability anxiety | AI pre-measures + grades; Grad-CAM shows *why* with 0 extra clicks; a Socratic circuit breaker catches rushed sign-offs; every report is MOH-cited and audit-backed | From lone, fatigued gatekeeper → to a *validated* decision-maker shielded from liability | | **UP7 Surgeon** | Synthesizes fragmented silos by hand; no time for patient education; fears uncompensated patient messaging | Aggregated view of signed findings; auto-generated patient-friendly visuals; strictly gated (no open chat) communication | From data-assembler → to *clinical force multiplier* with protected time | | **UP6 Physiotherapist** | Semi-blind execution from a one-line prescription; no scan visibility; language barrier; manual charting | Read-only diagnostic visibility down the chain; objective progress metrics; gel-proof, localized, zero-GPU UI | From guesswork executor → to *data-backed* practitioner with professional standing | | **UP8 Patient/Caregiver** | Anxiety + confusion → folk remedies; crowdsources raw scans on social media | Scan translated into plain-language, high-tech visual validation of the doctor's diagnosis | From anxious misinformation-seeker → to a *trusting, informed* participant | **System-level qualitative returns:** - **Institutional trust & credibility:** A high-trust, objective, MOH-anchored tool displaces both dangerous folk interventions (patients) and unsecured external messaging (clinicians bypassing the system). - **Academic-to-production migration:** VKIST research models become a live, referenceable clinical asset — establishing the team's operational value in the local healthcare community. - **Care-chain cohesion:** The radiologist's sign-off *ripples* to the surgeon, PT, and patient — turning isolated actions into a connected clinical relationship (the "Synchronized Ritual" concept). *Source: [Motivation.md](../DEMO_EXP/Motivation.md), [User_Research_Result.md](../Requirement_Analysis/USER_RESEARCH/User_Research_Result.md).* --- ## 4. Quantitative ROI — Projected Metrics & Measurement Plan > **Important framing for the leader:** The pilot has not yet deployed (runs through Sep 2026), so the figures below are **modeled projections**, not measured outcomes. Each row states its **baseline source** and its **assumption**, and — critically — **how the pilot will actually measure it**. The value of the pilot is precisely to *convert these projections into evidence*. ### 4.1 Projected Before → After Metrics | Metric | Baseline (before) | Target (after) | Assumption / basis | How the pilot measures it | | :--- | :--- | :--- | :--- | :--- | | **Tedious measurement time per scan** | Manual synovium measurement + grading, several minutes/scan under load | Automated pre-measurement; radiologist verifies | FR-25 automates FR-23/24 (segment + measure) | Time-on-task instrumentation (session telemetry, NFR audit timestamps) | | **Core inference latency** | N/A (no automated pipeline today) | ≤ 1.5 s (on-prem) | NFR-5 target on Triton GPU node | Prometheus p50/p99 latency dashboards | | **Explainability access cost** | Not available / manual literature lookup | **0 extra clicks** (Grad-CAM + citation inline) | NFR-12/13 design mandate | UI event audit (clicks-to-explanation = 0) | | **Clinical claim traceability** | Ad-hoc, uncited | **100%** of LLM clinical text cites MOH protocol | NFR-18 mandatory RAG | RAG-Referee pass rate; citation coverage log | | **Unsafe generative output reaching a human** | N/A | **0%** (100% pass safety verification) | NFR-10 3-tier guardrail | Guardrail rejection log; audit events | | **Onboarding time per staff member** | Ad-hoc, hours | ≤ 45 minutes | NFR-11 target | Timed onboarding sessions during rollout | | **System availability** | Dependent on manual workflow | ≥ 99.9% (≤ 45 s downtime/day) | NFR-9 target | Uptime monitoring (Prometheus/Grafana) | | **Cloud (MedGemma) usage ratio** | N/A | < 20% of consults | Cost-guard target (capital efficiency) | Cost Guard Redis counter + alert | | **PT diagnostic visibility** | ~0 (text prescription only) | Read-only access to signed findings | RBAC role (future sprint) | Access-log coverage per PT-linked case | | **Auditability of AI accept/override** | None | 100% immutable capture | NFR-17 | Immutable audit ledger completeness check | ### 4.2 Behavioral-Change Indicators (Adoption Proxies) Beyond raw performance, the pilot measures whether behavior actually changed — the true ROI signal for a leader: - **Retention / anti-bypass:** % of target clinicians using the structured workspace instead of unsecured external messaging (a stated success criterion). - **Override rate & ground-truth capture:** frequency of clinician overrides (Q3) feeding the retraining corpus — proves the human-AI loop works both ways. - **Circuit-breaker engagement:** how often the Socratic breaker prevents a rushed sign-off (a direct automation-bias safety signal). - **Compliance events:** zero unencrypted PHI leak events (Decree 13) — a binary go/no-go for production. *Baselines from [User_Research_Result.md](../Requirement_Analysis/USER_RESEARCH/User_Research_Result.md); targets from NFRs in [SOLUTION_ARCHITECTURE_SPEC.md §2.2](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md); success criteria from [CONTEXT_VISION_SCOPE.md §6](../PLAN/CONTEXT_VISION_SCOPE.md).* --- ## 5. Risk & Failure-Mode Handling — Why This Is Not Risky The single most important message for an investment decision: **this system is engineered around the assumption that either the AI *or* the human can be wrong.** It handles all four combinations explicitly via the **4-Quadrant Human-AI Interaction Framework**. ```plantuml @startuml VKIST_MSK_Risk_Quadrants skinparam packageStyle rectangle title 4-Quadrant Human-AI Safety Framework (FR-25) rectangle "Q1 — True Agreement\n(AI correct / Doctor correct)" as Q1 rectangle "Q2 — Automation Override\n(AI correct / Doctor oversight)" as Q2 rectangle "Q3 — Clinician Subservience\n(AI hallucinates / Doctor correct)" as Q3 rectangle "Q4 — Double-Blind Failure\n(AI faulty / edge case)" as Q4 Q1 --> Q1 : GradCAM + CoT -> High-Trust Concur log Q2 --> Q2 : Circuit Breaker -> Socratic -> BERT drift -> RAG-Referee Q3 --> Q3 : Expose activations -> isolate artifacts -> commit ground-truth Q4 --> Q4 : Investigation mode -> manual annotation -> telemetry queue @enduml ``` | Risk quadrant | The danger | How the system mitigates it | | :--- | :--- | :--- | | **Q1 — True Agreement** | Even correct consensus needs a trace for liability | Grad-CAM + Chain-of-Thought explanation, then an immutable High-Trust Concur log block | | **Q2 — Automation Bias** | Overloaded doctor rubber-stamps AI | Conversational Circuit Breaker intercepts rushed finalization → Socratic dialogue → BERT drift monitor → RAG-Referee arbitration against MOH guidelines | | **Q3 — AI Hallucination** | AI is wrong, doctor is right, but AI "authority" pressures the human | Expose pixel-level activation logic; let the clinician mask artifacts and **commit the corrected ground-truth** (which also feeds retraining) | | **Q4 — Double-Blind Edge Case** | Both AI confidence and guideline coverage are low | Auto-switch to strict manual **Clinical Investigation Mode**; force structured morphology annotation; route the anomaly to an engineering telemetry queue (bypassing the EMR) | **Cross-cutting safety guarantees:** - **HITL gate (NFR-19):** No AI output becomes FINALIZED/ARCHIVED without a licensed clinician's cryptographic signature. *A human always owns the decision.* - **Fail-safe by design:** The Circuit Breaker fails **open to deterministic MOH templates** — if Triton, EMR, or a cloud LLM is down, the system degrades to a safe rule-based floor, not to an error. - **Immutable audit (NFR-17):** Every accept/override/tier-transition/consent event is append-only and tamper-proof — a complete liability defense record. - **RAG-Referee (NFR-18):** No uncited clinical claim reaches a human. *Source: [full_usecase_planuml.md](../../workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/Use_Case/full_usecase_planuml.md), and the Q1–Q4 scenario docs under [FR_25_UC_DESIGN/](../Requirement_Analysis/UC_Design/FR_25_UC_DESIGN/).* --- ## 6. Compliance & Data Governance Healthcare compliance is treated as a first-class architectural constraint, not an afterthought. | Regulation | Requirement | How the architecture satisfies it | | :--- | :--- | :--- | | **Decree 13/2023/ND-CP** (personal health data) | Sensitive PII must be protected; data sovereignty | Client-side WebCrypto scrubbing + FastAPI Presidio redaction middleware; role-hash tokens `[BN:7f3a2c]`; all production data resides in-hospital / sovereign VN data centers | | **Circular 46/2018/TT-BYT** (EMR) | National EMR compliance for stored records | Circular-46-compliant PDF reports; HL7/FHIR EMR push; NFR-15 (not relaxed even under NFR-16a) | | **Air-gap (NFR-16, permanent)** | No diagnostic/identifiable data over public internet in production | Everything runs on the hospital LAN; K3s network policies isolate workloads; local-only container registries | | **Governed cloud PoC (NFR-16a, temporary)** | Cloud LLM only as an emergency/PoC fallback | Invoked only when edge LLM unavailable **and** user consents; mandatory pre-egress redaction + audit commit; CSEK keys, 30-day lifecycle, 1-year auto-delete; **retired at PoC sign-off** | **The compliance headline for a leader:** in production, **no patient data leaves the hospital**, every clinical statement is cited, every action is audit-logged, and a human signature is legally required to finalize anything. *Source: [SOLUTION_ARCHITECTURE_SPEC.md §2.2, §5.8](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md), [CONTEXT_VISION_SCOPE.md §4](../PLAN/CONTEXT_VISION_SCOPE.md).* --- ## 7. Roadmap & Milestones A strict, time-boxed 3-month cycle (June 2 – September 2, 2026), executed as a hybrid of waterfall macro-phases and agile sprints. This bounds the investment and provides clear go/no-go checkpoints - However it can be tentative to change based on the actual resource of development team. ```mermaid gantt title VKIST MSK Pilot - 3-Month Delivery Roadmap dateFormat YYYY-MM-DD axisFormat %b %d section Foundation Sprint 0 Pitch and Architecture Spike :done, sp0, 2026-06-02, 2026-06-12 Win project pitch :milestone, m1, 2026-06-09 Sprint 1 Fast PoC Baseline :active, sp1, 2026-06-15, 2026-06-26 section Core Build Sprint 2 Multi-Modal and NLP Integration :sp2, 2026-06-29, 2026-07-10 Sprint 3 Collaborative Workspace :sp3, 2026-07-13, 2026-07-24 section Patient and Hardening Sprint 4 Patient-Facing PWA :sp4, 2026-07-27, 2026-08-07 Sprint 5 Feedback Pipeline and Hardening :sp5, 2026-08-10, 2026-08-21 Sprint 6 Beta Release and Evolution :sp6, 2026-08-24, 2026-09-02 Pilot cycle complete :milestone, m2, 2026-09-02 ``` | Sprint | Window | Focus & key deliverable | | :--- | :--- | :--- | | **0** | Jun 02–12 | Pitch & architecture clearance; VKIST ML model audit; CI/CD baseline | | **1** | Jun 15–26 | Fast end-to-end PoC: FastAPI pipeline + inference mask preview | | **2** | Jun 29–Jul 10 | Multi-modal NLP + Decree 13 client-side scrubbing | | **3** | Jul 13–24 | Collaborative canvas; RBAC (PT read-only); usability testing | | **4** | Jul 27–Aug 07 | Installable PWA; zero-GPU fallback for legacy phones | | **5** | Aug 10–21 | Feedback instrumentation; performance/caching; cleanup | | **6** | Aug 24–Sep 02 | Beta launch to MSK cohort; telemetry analysis; Phase-2 backlog | **Definition of success (pilot go/no-go):** zero workflow friction added; deterministic cross-device stability (incl. CPU-bound rendering); zero Decree 13 leak events; sustained clinician retention over external messaging; successful academic-to-production model migration. *Source: [CONTEXT_VISION_SCOPE.md §5–6](../PLAN/CONTEXT_VISION_SCOPE.md).* --- ## 8. Investment Ask & Scope Boundaries ### 8.1 The Resource Envelope (qualitative) | Category | Pilot ask | Rationale | | :--- | :--- | :--- | | **On-prem compute** | 1 × Dell PowerEdge-class node (K3s + DB VM) + 1 × GPU node (A10/T4, 24 GB VRAM) for Triton | Single-site air-gapped deployment; capex-only, no recurring SaaS | | **Team** | Small cross-functional squad (backend/ML, frontend, infra/DevOps) + clinical champion (UP1/UP5) for validation | Matches the 7-sprint scope; clinical sponsor is essential for governance & buy-in | | **Software** | Open-source stack (K3s, Triton, Postgres/pgvector, Redis, MinIO, Keycloak, Prometheus/Grafana) | Zero licensing cost; no vendor lock-in | | **Governed cloud (PoC only)** | Time-boxed GCP Vertex AI (Gemini) + Modal (MedGemma) budget, capped at <20% consult usage | NFR-16a fallback; **retired at sign-off** — not a recurring production cost | | **Data** | Access to de-identified MSK ultrasound + MOH guideline corpus | Needed to tune grading thresholds and RAG index | ### 8.2 In Scope vs. Out of Scope (Pilot) | In scope (pilot) | Out of scope (Phase 2+) | | :--- | :--- | | FR-25 knee synovitis grading (UP5 primary) | Multi-hospital federation / multi-region HA | | Grad-CAM explainability, Socratic safety loop, RAG-Referee | Full UP7 surgeon planning & UP8 patient portal at production scale | | Decree 13 + Circular 46 compliance, HITL sign-off, audit log | Predictive analytics (cartilage degeneration, hardware failure forecasting) | | Single-hospital air-gapped deployment | Qdrant hot-tier vector cache (only if >100K vectors / >500 QPS) | | PoC cloud LLM tiers (NFR-16a, retired at sign-off) | Permanent cloud dependency (explicitly prohibited by NFR-16) | ### 8.3 Go / No-Go Decision Criteria At the end of the 3-month cycle, the leader decides on Phase 2 based on objective evidence: 1. **Safety:** zero Decree 13 leak events; 100% HITL-gated finalizations; 100% MOH-cited clinical text. 2. **Performance:** inference ≤ 1.5 s; availability ≥ 99.9%; onboarding ≤ 45 min. 3. **Adoption:** measurable clinician retention within the structured workspace vs. external messaging. 4. **Strategic:** VKIST models successfully running in a referenceable production form. > **The investment is bounded and reversible:** one hospital, one quarter, open-source stack, no production cloud dependency. If the go/no-go criteria are not met, the blast radius is a single controlled pilot — not an enterprise commitment. --- ## Appendix — Source Documents - Vision, personas, constraints, milestones: [CONTEXT_VISION_SCOPE.md](../PLAN/CONTEXT_VISION_SCOPE.md) - Deep personas + quantitative baselines: [User_Research_Result.md](../Requirement_Analysis/USER_RESEARCH/User_Research_Result.md) - Architecture, FR/NFR, C4 diagrams, build-vs-buy: [SOFTWARE_ARCHITECTURE_SPEC.md](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md), [SOLUTION_ARCHITECTURE_SPEC.md](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md) - Use cases + 4-quadrant framework: [full_usecase_planuml.md](../../workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/Use_Case/full_usecase_planuml.md), [CONTEXT_FR_25_UC_ELIT.md](../Requirement_Analysis/UC_Design/FR_25_UC_DESIGN/CONTEXT_FR_25_UC_ELIT.md) - Risk scenarios: [FR_25_UC_DESIGN/](../Requirement_Analysis/UC_Design/FR_25_UC_DESIGN/) (Q1–Q4 scenario docs) - Impact narrative: [Motivation.md](../DEMO_EXP/Motivation.md) *End of proposal.*