12 Commits
poc1 ... main

Author SHA1 Message Date
David Tran
7af1553032 Merge pull request #7 from DTJ-Tran/poc_1_3
add the session memory
2026-07-15 23:30:27 +07:00
DatTT127
1f6b015815 add the session memory 2026-07-15 23:29:26 +07:00
David Tran
eed2d39c2d Merge pull request #6 from DTJ-Tran/poc_1_3
update the cv modal inference proxy server with optimization
2026-07-15 23:25:26 +07:00
DatTT127
1f5e71b46a update the cv modal inference proxy server with optimization 2026-07-15 23:24:34 +07:00
David Tran
3435cfefa0 Merge pull request #5 from DTJ-Tran/poc2
update session_memory - for refer to the passwork
2026-07-07 22:03:18 +07:00
DatTT127
3fbbca1eaa update session_memory - for refer to the passwork 2026-07-07 22:01:30 +07:00
David Tran
4d89e55d86 Merge pull request #4 from DTJ-Tran/poc2
update the requirements.txt
2026-07-07 16:17:22 +07:00
DatTT127
d862d1f7f8 update the requirements.txt 2026-07-07 16:14:50 +07:00
David Tran
2484597c7a Merge pull request #3 from DTJ-Tran/poc1
Poc1-Proof of Concept verison 1
2026-07-07 15:56:36 +07:00
HuyTa12112001
41e6a2a416 Refactor code predict imflammation 2026-06-27 16:25:43 +07:00
David Tran
41241489fb Merge pull request #2 from DTJ-Tran/update_
Update
2026-06-24 16:16:18 +07:00
David Tran
43f5c0f7da Merge pull request #1 from DTJ-Tran/update_
update the architecture & description
2026-06-24 10:35:20 +07:00
71 changed files with 5161 additions and 246 deletions

6
.gitignore vendored
View File

@@ -227,7 +227,6 @@ Policy_Analysis/
package-lock.json package-lock.json
package.json package.json
.env.development .env.development
dist/
run.sh run.sh
workspace/sprint_1_2/CODEBASE/knowledge/implementation/model/ workspace/sprint_1_2/CODEBASE/knowledge/implementation/model/
*.onnx_data *.onnx_data
@@ -236,3 +235,8 @@ workspace/sprint_1_2/CODEBASE/knowledge/implementation/model/
*.safetensors *.safetensors
*.pt *.pt
*.pth *.pth
*.pdf
*.xz
logs/
*.rdb
dist/

View File

@@ -27,8 +27,6 @@ PILOT_PROJECT/
│ │ ├── LEGACY # The legacy material - for archieved / pre-existed project │ │ ├── LEGACY # The legacy material - for archieved / pre-existed project
│ │ ├── CODEBASE # The project codebase │ │ ├── CODEBASE # The project codebase
│ │ ├── Design_Material/ # The system design - API - and UIX design material │ │ ├── Design_Material/ # The system design - API - and UIX design material
│ │ ├── SOFTWARE_SYSTEM_DESIGN_FR_25.md # Detailed software design specification
│ │ ├── SOLUTION_ARCHITECTURE_SPEC.md # Solution architecture overview
│ │ └── VISUALIZATION/ # Charts, diagrams, and visual assets │ │ └── VISUALIZATION/ # Charts, diagrams, and visual assets
│ └── ... # Additional sprint folders as needed │ └── ... # Additional sprint folders as needed
├── secrets/ # **Developermanaged secrets** (NOT tracked by git) ├── secrets/ # **Developermanaged secrets** (NOT tracked by git)

View File

@@ -0,0 +1,654 @@
# VKIST MSK Ultrasound Stack — Investment Proposal & System Specification
> **Audience:** CTO / Department Head / Executive Sponsor (technical-fluent, outcome-driven, risk-averse on healthcare investments)
> **Author:** VKIST MSK Pilot Engineering Team
> **Status:** Pilot proposal — active cycle June 2 September 2, 2026 (strict 3-month window)
> **Reading time:** Section 0 (Executive Summary) is a self-contained one-pager. Sections 18 provide the full technical and financial justification.
---
## Table of Contents
0. [Executive Summary (one page)](#0-executive-summary-one-page)
1. [Project Context — Who, What Pain, What Constraints](#1-project-context)
2. [How the System Addresses the Problem](#2-how-the-system-addresses-the-problem)
3. [Qualitative ROI — Behavioral & Workflow Change](#3-qualitative-roi--behavioral--workflow-change)
4. [Quantitative ROI — Projected Metrics & Measurement Plan](#4-quantitative-roi--projected-metrics--measurement-plan)
5. [Risk & Failure-Mode Handling — Why This Is Not Risky](#5-risk--failure-mode-handling--why-this-is-not-risky)
6. [Compliance & Data Governance](#6-compliance--data-governance)
7. [Roadmap & Milestones](#7-roadmap--milestones)
8. [Investment Ask & Scope Boundaries](#8-investment-ask--scope-boundaries)
---
## 0. Executive Summary (one page)
**The problem, in one sentence.** Vietnamese public-hospital musculoskeletal (MSK) care runs on clinicians processing 100+ evaluations per shift with no time for objective double-checks, no shared visibility across the radiologist → surgeon → physiotherapist → patient chain, and patients so information-starved that they turn to dangerous folk remedies — while VKIST's proven computer-vision research models sit idle as static academic checkpoints.
**What we are building.** The **VKIST MSK Ultrasound Stack** is an **air-gapped-first, single-hospital AI platform** that turns knee ultrasound scans into automated, explainable **synovitis grading (03)** with Vietnamese-language reasoning, every clinical statement **cited back to Ministry of Health (MOH) guidelines**, and a **human clinician signature required** before anything is finalized. It migrates VKIST's existing DeepLabv3/UNet/ConvNeXt vision models from research into a referenceable, production-grade medical software ecosystem.
**Why it is a sound investment (the four pillars):**
| Pillar | What it means for the leader |
| :--- | :--- |
| **Zero-friction adoption** | The AI is an *invisible background validation layer* — 0 extra clicks to see explanations (NFR-12/13). Clinicians keep their workflow; the system removes liability risk instead of adding work. |
| **Non-negotiable safety** | Human-in-the-Loop signature gate (NFR-19), 100% MOH-cited explanations (NFR-18), immutable audit log (NFR-17), and a 4-quadrant safety framework that explicitly defends against automation bias *and* AI hallucination. |
| **Regulatory sovereignty** | Runs inside the hospital LAN. Decree 13/2023 PII protection + Circular 46/2018 EMR compliance are built into the architecture, not bolted on. No patient data leaves the hospital in production. |
| **Capital efficiency** | Built on proven open-source (K3s, Triton, Postgres/pgvector, Keycloak, MinIO) and existing VKIST models. No SaaS lock-in, no per-seat licensing, delivered in a fixed 3-month pilot. |
**The ask.** Sponsorship of a fixed 3-month pilot on a single hospital's on-premise infrastructure (one Dell PowerEdge class node + one GPU node), a small cross-functional team, and a *governed, time-boxed* cloud allowance for the proof-of-concept LLM tiers (NFR-16a, retired at sign-off). See [Section 8](#8-investment-ask--scope-boundaries).
**The expected return.** A validated, referenceable clinical AI product that (projected) removes tedious measurement work from overloaded specialists, gives physiotherapists diagnostic visibility they have never had, converts patient anxiety into institutional trust, and establishes VKIST's operational credibility in the local healthcare community — with every claim instrumented and measurable during the pilot itself (see [Section 4](#4-quantitative-roi--projected-metrics--measurement-plan)).
> **Bottom line for the decision-maker:** This is a low-blast-radius, standards-compliant, single-site pilot that converts already-funded research into a safe, auditable clinical tool. The downside is bounded (one hospital, 3 months, open-source stack); the upside is a defensible product line and clinical-community trust.
---
## 1. Project Context
### 1.1 Who the System Is Designed For
The platform coordinates a **four-persona clinical care chain**. The pilot (Sprint 12, FR-25) is centered on the **Diagnostic Radiologist (UP5)** as the primary human actor, with the other three personas as governance/future-sprint stakeholders whose needs shape the non-functional requirements.
| Persona | Who they are | Environment | Core responsibility | What the system must respect |
| :--- | :--- | :--- | :--- | :--- |
| **UP5 — Diagnostic Radiologist** | Specialist MDs, age 2855, from digitally native attendings to department heads | Low-light reading rooms; massive daily DICOM backlogs; asynchronous, detached from patients | Translate images into definitive reports that gate all downstream care | *Invisible workflow preservation* — no extra clicks, no DICOM render lag, no uncompensated patient chat |
| **UP7 — Rheumatologist / Orthopedic Surgeon** | Treatment architects, age 3060; seniors control procurement | Split across OR, chaotic outpatient clinics (>100 patients/shift), inpatient wards | Synthesize fragmented data into a treatment plan; bear full legal/clinical liability | *Clinical force multiplier* — rapid dashboards, automated patient-education visuals, strictly gated communication |
| **UP6 — Physical Therapist** | Young (3640% aged 2029), highly stratified (≈54% vocational tier), digitally literate | Cramped (<20 m²), high-noise public units; 1120+ patients/shift; shared/scarce workstations | Execute physician scripts into kinetic rehab programs | *Digital exoskeleton* gel-proof knuckle-tap UI, zero-GPU mobile, localized language, objective progress data |
| **UP8 — MSK Patient & Family Caregiver** | Elderly patients (4580+) via younger caregiver proxies (2045) | Home, pharmacy, crowded waiting rooms; saturated with health misinformation | Manage chronic pain; adhere to protocols; make family-centric decisions | *Empathy + accessibility* jargon-free, high-contrast, large-font, dual-profile family access on legacy phones |
*Source: [User_Research_Result.md](../Requirement_Analysis/USER_RESEARCH/User_Research_Result.md), [CONTEXT_VISION_SCOPE.md §3](../PLAN/CONTEXT_VISION_SCOPE.md).*
### 1.2 The Pain Points That Drive the Need
These are not hypothetical they are drawn from ethnographic user research (professional forums, patient forums, and published studies).
- **Clinical exhaustion & volume:** Central referral hospitals (Bach Mai, Viet Duc) regularly exceed **100 evaluations per shift**, leaving no time for tedious measurements or objective double-checks. This creates the risk of *automation-bias blind-concurrence* a clinician rubber-stamping an AI estimate under time pressure.
- **The physiotherapist information bottleneck:** PTs receive only brief text prescriptions no DICOM, no scan findings forcing **semi-blind therapeutic execution** against estimated surface anatomy.
- **The physiotherapist body & literacy crisis:** **76.4%** WMSD prevalence (career-shortening physical strain); **84%** report a foreign-language barrier (only 25% adequate English); 96%/93%/86% rely on personal experience/peer chat/old textbooks rather than formal evidence.
- **Dangerous folk interventions:** Information-starved patients turn to leaf-wrapping or violent manual adjustment, causing tissue infection or permanent anatomical failure because no one translated their scan into understandable terms.
- **Stranded research value:** VKIST's CV models (DeepLabv3/UNet/ConvNeXt/MedViT) are proven in the lab but have no production pathway, so they generate zero operational or reputational value.
### 1.3 Constraints the System Must Meet
**Functional Requirements (pilot scope — FR-25 pipeline):**
| FR | Requirement |
| :--- | :--- |
| **FR-23/24/25** | Segment knee joint structures measure synovium thickness (mm) grade synovitis **03** |
| **FR-25 / Grad-CAM** | Spatial activation overlay on the primary viewport with **zero extra clicks** |
| **FR-25 / Safety loop** | Conversational Circuit Breaker + Socratic dialogue before finalization; BERT drift monitor; RAG-Referee arbitration |
| **FR-25 / Reporting** | Circular 46/2018 compliant PDF report generation |
| **FR-25 / Finalization** | Cryptographically signed, HITL-gated electronic record EMR |
**Non-Functional Requirements (the "why it won't fail" constraints):**
| NFR | Constraint | Why the leader cares |
| :--- | :--- | :--- |
| **NFR-4** | 150 MB idle app bundle | Runs on cheap, legacy hospital/consumer hardware |
| **NFR-5** | 1.5 s core inference (on-prem) | No workflow slowdown |
| **NFR-7** | 200 ms token streaming (TTFT) | Feels instantaneous to the clinician |
| **NFR-9** | 99.9% availability (≤ 45 s downtime/day) | Clinically dependable |
| **NFR-10** | 100% of patient-facing LLM text passes safety verification | No unsafe generative output reaches a human |
| **NFR-11** | Onboarding 45 minutes | Cheap to roll out to busy staff |
| **NFR-12/13** | 0 extra clicks for explainability; native spatial activation maps | Anti-black-box; builds clinician trust |
| **NFR-14** | No client-side GPU/accelerator required (Android 10+, 3 GB RAM) | Works on the fleet that actually exists |
| **NFR-15** | Circular 46/2018 EMR compliance | Legally deployable |
| **NFR-16** | Air-gapped primary (no PHI over public internet in production) | Data sovereignty (Decree 13) |
| **NFR-17** | Immutable audit log | Accountability & liability defense |
| **NFR-18** | 100% LLM clinical text cites MOH protocol via RAG | Every claim is traceable & defensible |
| **NFR-19** | HITL digital signature before FINALIZED/ARCHIVED | A licensed human always owns the decision |
*Source: [SOLUTION_ARCHITECTURE_SPEC.md §2](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md), [SOFTWARE_ARCHITECTURE_SPEC.md §2](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md).*
### 1.4 System Context — C4 Level 1 (PlantUML)
The context diagram summarizes *who* and *what* the system boundary talks to. The pilot activates UP5 (and governance actors UP1/UP4); UP6/UP7/UP8 are shown as future-sprint boundaries.
```plantuml
@startuml VKIST_MSK_System_Context
!include <C4/C4_Context>
title System Context Diagram — VKIST MSK Ultrasound Stack (Pilot: FR-25 Synovitis Grading)
' === PRIMARY HUMAN ACTOR (in pilot scope) ===
Person(radiologist, "Diagnostic Radiologist (UP5)", "Loads knee ultrasound, reviews AI grading (0-3), confirms/overrides, engages Socratic circuit breaker, finalizes & signs the report, reads GradCAM + RAG evidence")
Person(admin, "System Administrator", "Deploys/monitors on-prem K3s, model updates, observability, NGINX failover")
' === GOVERNANCE / NFR-ALIGNMENT ACTORS ===
Person_Ext(senior_expert, "Healthcare Senior Expert (UP1)", "Clinical protocol validation, model-threshold approval, pilot sponsor")
Person_Ext(support_staff, "Support Staff (UP4)", "Patient registration, case-queue management")
' === FUTURE-SPRINT PERSONAS (out of pilot scope) ===
Person_Ext(therapist, "Physical Therapist (UP6)", "Future: read-only diagnostic visibility, kinetic overlays, progress tracking")
Person_Ext(surgeon, "Surgeon / Rheumatologist (UP7)", "Future: aggregated dashboards, treatment planning")
Person_Ext(patient, "Patient & Caregiver (UP8)", "Future: patient portal, plain-language education, self-monitoring")
' === THE SYSTEM ===
System(mpps, "VKIST MSK Ultrasound Stack (FR-25)", "Air-gapped-first: knee ultrasound AI (angle -> inflammation -> segmentation), synovitis grading 0-3 with GradCAM, 4-quadrant clinical safety, MOH-cited Vietnamese explanations, HITL-signed Circular 46 reports")
' === EXTERNAL SYSTEMS ===
System_Ext(pacs, "Hospital PACS / Ultrasound Machine", "DICOM source (C-MOVE + direct capture)")
System_Ext(emr, "Hospital EMR / HIS", "Signed report storage, prescription/lab sync (HL7/FHIR)")
System_Ext(triton, "Triton Inference Server", "GPU node: 3-step ML pipeline + RAG embeddings (gRPC)")
System_Ext(knowledge, "Clinical Knowledge Systems", "ladybugDB ontology (SNOMED-CT), pgvector MOH guideline index, Gemma/MedGemma LLM tiers")
' === RELATIONSHIPS ===
Rel(radiologist, mpps, "Reviews grade, overrides, finalizes & signs, reads explanations", "HTTPS")
Rel(admin, mpps, "Deploys, monitors, updates models", "HTTPS / SSH")
Rel(senior_expert, mpps, "Validates protocols, approves thresholds", "HTTPS (governance)")
Rel(support_staff, mpps, "Registration, case queue", "HTTPS")
Rel(therapist, mpps, "Future sprint scope", "—")
Rel(surgeon, mpps, "Future sprint scope", "—")
Rel(patient, mpps, "Future sprint scope", "—")
Rel(mpps, pacs, "Imports DICOM", "DICOM / C-MOVE")
Rel(mpps, emr, "Pushes signed reports + audit records", "HL7/FHIR")
Rel(mpps, triton, "ML inference offload", "gRPC 8001")
Rel(mpps, knowledge, "RAG retrieval, ontology traversal, LLM generation", "SQL (pgvector) + in-process")
@enduml
```
*Adapted from [SOLUTION_ARCHITECTURE_SPEC.md §8.1](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md).*
### 1.5 Applicability — Use-Case View (PlantUML)
The pilot's functional scope is organized around a **4-Quadrant Human-AI Interaction Framework** that deliberately defends against both *automation bias* (blindly trusting AI) and *clinician subservience* (AI overriding a correct human). This is the applicability model what the radiologist can actually *do* with the system.
```plantuml
@startuml VKIST_MSK_UseCases
left to right direction
skinparam packageStyle rectangle
actor "Diagnostic Radiologist (UP5)" as Rad
actor "MSK Vision Grader" as Grader <<system>>
actor "LLM Explainer" as LLM <<system>>
actor "RAG-Referee Arbitrator" as Referee <<system>>
actor "Hospital EMR" as EMR <<system>>
rectangle "VKIST MSK Workspace (FR-25)" {
usecase "UC-48376\nLoad Patient Scan Session" as U1
usecase "UC-47988\nReview Suggested Synovitis Grade (0-3)" as U2
usecase "UC-92006\nFinalize & Sign Electronic Record" as U3
rectangle "Q1 — True Agreement (AI correct / Doctor correct)" {
usecase "UC-25776\nGradCAM + CoT Explanation" as Q1a
usecase "UC-02423\nLog High-Trust Concur Block" as Q1b
}
rectangle "Q2 — Automation Override (AI correct / Doctor oversight)" {
usecase "UC-22159\nTrigger Circuit Breaker" as Q2a
usecase "UC-55146\nSocratic Reasoning Dialogue" as Q2b
usecase "UC-74821\nBERT Context-Drift Monitor" as Q2c
usecase "UC-65473\nArbitrate via RAG-Referee" as Q2d
}
rectangle "Q3 — Clinician Subservience (AI hallucinates / Doctor correct)" {
usecase "UC-25637\nExpose Pixel-Level Activation" as Q3a
usecase "UC-60739\nIsolate Visual Noise/Artifacts" as Q3b
usecase "UC-62864\nCommit Validated Ground-Truth" as Q3c
}
rectangle "Q4 — Double-Blind Failure (AI faulty / edge case)" {
usecase "UC-35956\nActivate Clinical Investigation Mode" as Q4a
usecase "UC-47796\nStructured Morphology Annotation" as Q4b
usecase "UC-01580\nSerialize to Telemetry Queue" as Q4c
}
}
Rad --> U1
Rad --> U2
Rad --> U3
Rad --> Q2b : Argue observations
Rad --> Q3b : Tag artifacts
Rad --> Q4b : Document manual findings
Grader --> U1 : Feeds vision tensors + scores
Grader --> U2 : Generates prediction
LLM --> Q1a : Generates clinical CoT
LLM --> Q2b : Generates Socratic checks
Referee --> Q2d : Retrieves MOH guidelines
U3 --> EMR : Syncs signed report
Q1b --> EMR : Stores consensus log
Q4c --> EMR : Routes anomaly to engineering telemetry
U1 ..> Q1a : <<include>>
Q1a ..> Q1b : <<include>>
U2 <.. Q2a : <<extend>> (friction detected)
Q2a ..> Q2b : <<include>>
Q2a ..> Q2c : <<include>>
Q2c ..> Q2d : <<extend>> (impasse/drift)
U2 <.. Q3a : <<extend>> (clinician contests score)
Q3a ..> Q3b : <<include>>
Q3b ..> Q3c : <<include>>
U2 <.. Q4a : <<extend>> (low confidence + empty RAG)
Q4a ..> Q4b : <<include>>
Q4b ..> Q4c : <<include>>
@enduml
```
*Adapted from [full_usecase_planuml.md](../../workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/Use_Case/full_usecase_planuml.md) and [CONTEXT_FR_25_UC_ELIT.md](../Requirement_Analysis/UC_Design/FR_25_UC_DESIGN/CONTEXT_FR_25_UC_ELIT.md).*
---
## 2. How the System Addresses the Problem
### 2.1 Technology Choices — Problem → Solution → Why Optimal → Limitation
Each technology is selected against a specific user problem or NFR, not for novelty. The table states the honest limitation of each choice so the leader sees a balanced picture.
| Technology | User problem / NFR it solves | Why it is the optimal choice | Limitation / trade-off |
| :--- | :--- | :--- | :--- |
| **React PWA + LiteRT (MediaPipe) + WebLLM edge** | NFR-14 (no client GPU), NFR-4 (150 MB), zero-friction on legacy phones | Installable, offline-capable, runs a local angle pre-classifier + edge LLM with no server round-trip; degrades gracefully on weak hardware | Browser memory ceilings; edge LLM is small (GemmaE2B-Q4 ~1.3 GB) so deep reasoning must escalate |
| **FastAPI edge-inference-svc (Python 3.12)** | FR-25 pipeline orchestration, Decree 13 middleware, SSE streaming | Async, lightweight, tight control over PII redaction + audit at the boundary; in-house domain logic | Python GIL for CPU-bound work mitigated by offloading inference to Triton |
| **Triton Inference Server (ONNX/TensorRT)** | NFR-5 (≤1.5 s inference), reuse of VKIST CV models | Industry-standard, GPU-efficient, self-hosted (no SaaS); serves the 3-step ensemble + embeddings | Requires a GPU node (capex); one more component to operate |
| **Postgres + pgvector (HNSW)** | NFR-18 MOH-cited RAG, transactional consistency of clinical records | One store for records *and* vectors (~15K guideline vectors) ~520 ms retrieval, no separate vector DB to run | Not optimal beyond ~100K vectors / 500 QPS (documented Phase-2 path to Qdrant) |
| **RAG-Referee (BERT classifier)** | NFR-10/18 reject unsupported LLM claims | Validates attribution + logical cohesion of every explanation before it reaches a human | Adds latency to the cloud tier; thresholds need clinical tuning |
| **3-Tier LLM (Edge Gemma → Gemini → MedGemma)** | Vietnamese explanations, deep clinical reasoning, cost control | Cheapest-capable-tier routing: local first, cloud only when needed, <20% MedGemma usage cap | Cloud tiers are PoC-only (NFR-16a) and must be redacted retired at sign-off |
| **Keycloak (OIDC/RBAC) on-prem** | NFR-16 sovereignty, role separation (PT read-only) | No SaaS identity; realm-scoped RBAC inside the LAN | Operational overhead vs. managed Auth0/Okta (accepted for sovereignty) |
| **K3s on Dell PowerEdge** | NFR-16 air-gap, NFR-9 availability | Lightweight Kubernetes for a single hospital; rolling updates, self-healing pods; NGINX+Keepalived VIP failover (≤2 s) | Single-site scale; not a multi-region HA design (by design for the pilot) |
| **MinIO + Redis (5 scoped types)** | Object/blob store + session/cache | S3-compatible, self-hosted, no cloud egress; Redis scoped to prevent sprawl | Self-managed durability (mitigated by backup cron to MinIO) |
**Honest, system-level limitations (stated up front for the leader):**
- **Single-hospital scale by design.** The pilot targets one site; multi-site federation is a documented Phase-2 evolution, not a pilot promise.
- **Cloud LLM tiers are proof-of-concept only** under NFR-16a, gated by redaction + consent + audit, and **formally retired at PoC sign-off** production reverts to fully air-gapped NFR-16.
- **Edge model is intentionally small** deep reasoning escalates through the tier chain, which is why the cost guard (<20% MedGemma) and RAG-Referee exist.
*Source: [SOFTWARE_ARCHITECTURE_SPEC.md §7 Build-vs-Buy](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md), [SOLUTION_ARCHITECTURE_SPEC.md §5](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md).*
### 2.2 Deployment Environment — C4 Container Diagram (PlantUML)
The system runs entirely inside the hospital LAN. The only external touchpoints are the governed, PoC-scoped cloud fallbacks (NFR-16a).
```plantuml
@startuml VKIST_MSK_Containers
!include <C4/C4_Container>
title C4 Container Diagram — VKIST MSK Ultrasound Stack
Person(radiologist, "Diagnostic Radiologist (UP5)", "Loads DICOM, reviews grading, finalizes reports")
Person(admin, "System Administrator", "K3s ops, model updates, observability, failover")
Person_Ext(senior_expert, "Healthcare Senior Expert (UP1)", "Protocol validation, threshold approval")
Person_Ext(support_staff, "Support Staff (UP4)", "Registration, case queue")
System_Boundary(hospital_lan, "Hospital LAN (Air-Gapped, <=10 Mbps)") {
Container(pwa, "React PWA Frontend", "React 18, TS, Zustand, LiteRT, MediaPipe, Dexie.js", "View-angle validation (WASM); DICOM preview; Grad-CAM overlay; Decree 13 scrubber; encrypted IndexedDB sessions")
Container(nginx, "Active-Passive Gateway", "NGINX + Keepalived", "SSL termination, VIP load balancing, <=2 s failover")
System_Boundary(k3s_cluster, "K3s Orchestration Cluster") {
Container(edge_inference, "Edge Inference Service", "FastAPI (Python 3.12)", "DICOM ingest, 3-step ML orchestration, Grad-CAM, report assembly, SSE streaming")
Container(cloud_llm_gateway, "Cloud LLM Gateway", "FastAPI (Python 3.12)", "Routes to Gemini (orchestration) / MedGemma (clinical); NFR-16a redaction, consent, audit, cost guard")
Container(rag_svc, "RAG & Knowledge Service", "FastAPI (asyncpg)", "pgvector top-k, ladybugDB ontology, RAG-Referee validation, mandatory RAG pre-processing")
Container(audit_svc, "Audit & EMR Service", "Node.js / FastAPI Worker", "Immutable append-only audit; HL7/FHIR EMR push with outbox retry")
Container(api_gw, "API Gateway", "Envoy", "TLS, rate limiting, routing, OIDC pass-through")
Container(auth_svc, "Identity & Access", "Keycloak", "OIDC, RBAC, realm vkist-msk")
Container(obs_stack, "Observability Stack", "Prometheus + Grafana", "Metrics, dashboards, alerting")
}
System_Boundary(db_vm, "Database VM") {
ContainerDb(postgres, "PostgreSQL + pgvector", "PostgreSQL 16 + pgvector", "EMR records, spatial markers, MOH HNSW index, embeddings, audit ledger")
ContainerDb(redis, "Redis Cache", "Redis 7 (AOF+RDB)", "Sessions, guideline chunks, DICOM metadata, consult_mode state, rate-limit")
ContainerDb(minio, "MinIO Object Store", "MinIO S3-compatible", "DICOM payloads, overlays, Grad-CAM heatmaps, report PDFs, model staging")
}
}
System_Ext(triton, "Triton Inference Server", "NVIDIA Triton (ONNX/TensorRT)", "3-step ML pipeline + EmbeddingGemma RAG embeddings")
System_Ext(emr, "Hospital EMR / HIS", "HL7/FHIR", "Signed report storage, prescription sync")
System_Ext(pacs, "PACS / Ultrasound Device", "DICOM / C-MOVE", "Image capture & retrieval")
System_Ext(vertex_ai, "GCP Vertex AI (Gemini)", "REST", "PoC-only NFR-16a: orchestration, translation; redacted payloads only")
System_Ext(modal_medgemma, "Modal MedGemma", "T4 GPU", "PoC-only NFR-16a: clinical deep-reasoning; redacted + RAG-Referee validated")
Rel(radiologist, pwa, "Loads scan, reviews grade, finalizes, reads explanations", "HTTPS 443")
Rel(admin, nginx, "Deploys, monitors, configures", "HTTPS / SSH")
Rel(senior_expert, pwa, "Validates protocols, approves thresholds", "HTTPS 443")
Rel(support_staff, pwa, "Registration, case queue", "HTTPS 443")
Rel(pwa, nginx, "API requests + pre-validated DICOM", "HTTPS 443")
Rel(nginx, api_gw, "Routes upstream", "HTTP 8000")
Rel(api_gw, edge_inference, "Forwards /api/*", "HTTP 8000")
Rel(api_gw, rag_svc, "Forwards /rag/*", "HTTP 8001")
Rel(api_gw, auth_svc, "Validates OIDC tokens", "HTTP 8080")
Rel(edge_inference, triton, "3-step inference + embeddings", "gRPC 8001")
Rel(edge_inference, postgres, "Reads guidelines; writes cases/embeddings/audit", "SQL 5432")
Rel(edge_inference, redis, "Session, rate-limit, consult_mode", "TCP 6379")
Rel(edge_inference, minio, "DICOM, overlays, reports", "S3 API")
Rel(edge_inference, cloud_llm_gateway, "Routes cloud consult", "HTTP 8000")
Rel(cloud_llm_gateway, vertex_ai, "Gemini proxy (NFR-16a redacted)", "HTTPS 443")
Rel(cloud_llm_gateway, modal_medgemma, "MedGemma proxy (NFR-16a redacted)", "HTTPS 443")
Rel(cloud_llm_gateway, postgres, "Audit log append", "SQL 5432")
Rel(rag_svc, postgres, "pgvector HNSW queries", "SQL 5432")
Rel(rag_svc, redis, "Guideline cache, pub/sub invalidation", "TCP 6379")
Rel(audit_svc, postgres, "Appends immutable audit events", "SQL 5432")
Rel(audit_svc, emr, "Finalized report push", "HL7/FHIR")
Rel(pwa, pacs, "Direct DICOM capture / C-MOVE", "DICOM")
@enduml
```
*Reused from [SOFTWARE_ARCHITECTURE_SPEC.md §4.2](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md).*
### 2.3 Deployment Topology — C4 Deployment Diagram (PlantUML)
Physical placement: everything on hospital hardware; the only external node is the governed NFR-16a fallback.
```plantuml
@startuml VKIST_MSK_Deployment
!include <C4/C4_Deployment>
LAYOUT_LEFT_RIGHT()
Deployment_Node(hw, "Dell PowerEdge (Hospital)") {
Deployment_Node(k3s, "K3s Cluster") {
Deployment_Node(pod_edge, "Pod: edge-inference-svc") {
Container(ci, "FastAPI", "Python 3.12, Uvicorn")
}
Deployment_Node(pod_rag, "Pod: rag-svc") {
Container(cr, "FastAPI", "Python 3.12, asyncpg")
}
Deployment_Node(pod_audit, "Pod: audit-svc") {
Container(ca, "Node.js", "WAL writer")
}
Deployment_Node(pod_gw, "Pod: api-gateway") {
Container(cg, "Envoy", "TLS, rate limit")
}
Deployment_Node(pod_auth, "Pod: auth-svc") {
Container(ck, "Keycloak", "OIDC, RBAC")
}
Deployment_Node(pod_obs, "Pod: observability") {
Container(cp, "Prometheus")
Container(cf, "Grafana")
}
}
Deployment_Node(db_vm, "DB VM") {
ContainerDb(pg, "PostgreSQL + pgvector HNSW")
ContainerDb(rd, "Redis")
ContainerDb(mn, "MinIO")
}
}
Deployment_Node(triton_node, "GPU Node (Triton)", "NVIDIA A10/T4, 24 GB VRAM") {
Container(tc, "Triton Inference Server", "ONNX/TensorRT, gRPC 8001")
}
Deployment_Node(ext, "External (NFR-16a governed, PoC only)") {
ContainerDb(s3v, "GCP Vertex AI / Modal MedGemma / CDN fallback")
}
Rel(ci, pg, "SQL", "TCP 5432")
Rel(ci, rd, "Cache", "TCP 6379")
Rel(ci, mn, "Blobs", "S3 API")
Rel(ci, tc, "Inference", "gRPC 8001")
Rel(ci, ck, "Auth", "OIDC")
Rel(ci, ca, "Audit", "HTTP")
Rel(k3s, db_vm, "SQL / Cache / Blobs", "TCP")
@enduml
```
*Reused from [SOFTWARE_ARCHITECTURE_SPEC.md §4.4](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md).*
### 2.4 Components & Interactions — C4 Component Diagram (PlantUML)
Inside the core `edge-inference-svc`, each component maps to a specific functional requirement. The **Inference Router** implements the tiered LLM escalation; the **Circuit Breaker** guarantees the system fails safely to MOH templates.
```plantuml
@startuml VKIST_MSK_Component_EdgeInference
!include <C4/C4_Component>
Container_Boundary(edge_svc, "edge-inference-svc (FastAPI)") {
Component(api, "API Controller", "REST + SSE", "/api/analyze, cloud-orchestrate redirect")
Component(stream, "SSE Token Streamer", "StreamingResponse", "200 ms TTFT, 30 s heartbeat")
Component(preproc, "Image Preprocessor", "OpenCV + pydicom", "CLAHE, rescale, DICOM header scrub")
Component(router, "Inference Router", "consult_mode state", "Tier selection: Edge Gemma -> Gemini -> MedGemma -> Templates")
Component(breaker, "Circuit Breaker", "pybreaker", "Wraps Triton + EMR + Cloud LLM; fail-open to templates")
Component(pipeline, "ML Pipeline", "gRPC client", "Angle -> Inflammation -> Segmentation -> Measurement")
Component(gradcam, "Grad-CAM Generator", "OpenCV", "Spatial activation overlay -> base64 PNG")
Component(report, "Report Builder", "WeasyPrint", "Circular 46 PDF; HITL signature gate")
Component(rag, "RAG Service", "pgvector SQL", "Top-5 MOH chunks; mandatory pre-generation; NFR-18")
Component(referee, "RAG-Referee", "BERT classifier", "Reject LLM text if citation confidence < threshold")
Component(nlp, "NLP Scrubber", "Microsoft Presidio", "Re-verify edge redaction; clean residual PII")
Component(audit, "Audit Logger", "Append-only writer", "Every tier transition, consent, finalize, override")
}
Rel(api, stream, "delegates", "SSE")
Rel(api, preproc, "validates image", "sync")
Rel(api, router, "selects tier", "sync")
Rel(router, breaker, "guards calls", "sync")
Rel(router, pipeline, "invokes", "gRPC")
Rel(router, stream, "fallback text", "SSE")
Rel(api, gradcam, "requests overlay", "sync")
Rel(api, report, "generates PDF", "sync")
Rel(api, rag, "queries MOH", "SQL")
Rel(rag, referee, "validates citations", "sync")
Rel(api, nlp, "scrubs output", "sync")
Rel(api, audit, "writes event", "sync")
@enduml
```
The **Cloud LLM Gateway** enforces the NFR-16a governance boundary nothing egresses without consent, redaction, RAG grounding, and an audit record.
```plantuml
@startuml VKIST_MSK_Component_CloudGateway
!include <C4/C4_Component>
Container_Boundary(cloud_gw, "cloud-llm-gateway (FastAPI)") {
Component(capi, "Cloud API Controller", "REST + SSE", "/api/cloud-orchestrate, /api/cloud-consult")
Component(gateway, "Cloud LLM Router", "task_type matcher", "Orchestration/translation -> Gemini; clinical -> MedGemma")
Component(consent, "Consent Enforcer", "Redis + NFR-16a checklist", "Validates consent token + redaction manifest before egress")
Component(caudit, "Cloud Audit Emitter", "PostgreSQL append-only", "egress_consent, egress_response, escalation events")
Component(cost, "Cost Guard", "Redis + Prometheus", "Tracks MedGemma usage; alerts if >20% over 24h")
Component(creferee, "RAG-Referee Trigger", "BERT classifier", "Mandatory validation of MedGemma output")
Component(crag, "RAG Pre-processing", "pgvector SQL", "Mandatory top-k injection before generation")
}
Rel(capi, consent, "verifies", "sync")
Rel(capi, gateway, "routes", "sync")
Rel(gateway, crag, "injects chunks", "sync")
Rel(gateway, creferee, "validates output", "sync")
Rel(gateway, caudit, "logs egress", "sync")
Rel(gateway, cost, "increments counter", "sync")
@enduml
```
*Reused from [SOFTWARE_ARCHITECTURE_SPEC.md §4.3](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md).*
---
## 3. Qualitative ROI — Behavioral & Workflow Change
The core value thesis: the system does not add a tool to a workflow it **enriches the existing workflow and reconnects a fragmented care chain**. The following before/after captures the behavioral change per persona.
| Persona | Before (current behavior) | After (with the system) | Behavioral shift |
| :--- | :--- | :--- | :--- |
| **UP5 Radiologist** | Manual, repetitive measurements under fatigue; risk of missing subtle findings; isolated in a reading room; liability anxiety | AI pre-measures + grades; Grad-CAM shows *why* with 0 extra clicks; a Socratic circuit breaker catches rushed sign-offs; every report is MOH-cited and audit-backed | From lone, fatigued gatekeeper to a *validated* decision-maker shielded from liability |
| **UP7 Surgeon** | Synthesizes fragmented silos by hand; no time for patient education; fears uncompensated patient messaging | Aggregated view of signed findings; auto-generated patient-friendly visuals; strictly gated (no open chat) communication | From data-assembler to *clinical force multiplier* with protected time |
| **UP6 Physiotherapist** | Semi-blind execution from a one-line prescription; no scan visibility; language barrier; manual charting | Read-only diagnostic visibility down the chain; objective progress metrics; gel-proof, localized, zero-GPU UI | From guesswork executor to *data-backed* practitioner with professional standing |
| **UP8 Patient/Caregiver** | Anxiety + confusion folk remedies; crowdsources raw scans on social media | Scan translated into plain-language, high-tech visual validation of the doctor's diagnosis | From anxious misinformation-seeker to a *trusting, informed* participant |
**System-level qualitative returns:**
- **Institutional trust & credibility:** A high-trust, objective, MOH-anchored tool displaces both dangerous folk interventions (patients) and unsecured external messaging (clinicians bypassing the system).
- **Academic-to-production migration:** VKIST research models become a live, referenceable clinical asset establishing the team's operational value in the local healthcare community.
- **Care-chain cohesion:** The radiologist's sign-off *ripples* to the surgeon, PT, and patient turning isolated actions into a connected clinical relationship (the "Synchronized Ritual" concept).
*Source: [Motivation.md](../DEMO_EXP/Motivation.md), [User_Research_Result.md](../Requirement_Analysis/USER_RESEARCH/User_Research_Result.md).*
---
## 4. Quantitative ROI — Projected Metrics & Measurement Plan
> **Important framing for the leader:** The pilot has not yet deployed (runs through Sep 2026), so the figures below are **modeled projections**, not measured outcomes. Each row states its **baseline source** and its **assumption**, and — critically — **how the pilot will actually measure it**. The value of the pilot is precisely to *convert these projections into evidence*.
### 4.1 Projected Before → After Metrics
| Metric | Baseline (before) | Target (after) | Assumption / basis | How the pilot measures it |
| :--- | :--- | :--- | :--- | :--- |
| **Tedious measurement time per scan** | Manual synovium measurement + grading, several minutes/scan under load | Automated pre-measurement; radiologist verifies | FR-25 automates FR-23/24 (segment + measure) | Time-on-task instrumentation (session telemetry, NFR audit timestamps) |
| **Core inference latency** | N/A (no automated pipeline today) | 1.5 s (on-prem) | NFR-5 target on Triton GPU node | Prometheus p50/p99 latency dashboards |
| **Explainability access cost** | Not available / manual literature lookup | **0 extra clicks** (Grad-CAM + citation inline) | NFR-12/13 design mandate | UI event audit (clicks-to-explanation = 0) |
| **Clinical claim traceability** | Ad-hoc, uncited | **100%** of LLM clinical text cites MOH protocol | NFR-18 mandatory RAG | RAG-Referee pass rate; citation coverage log |
| **Unsafe generative output reaching a human** | N/A | **0%** (100% pass safety verification) | NFR-10 3-tier guardrail | Guardrail rejection log; audit events |
| **Onboarding time per staff member** | Ad-hoc, hours | 45 minutes | NFR-11 target | Timed onboarding sessions during rollout |
| **System availability** | Dependent on manual workflow | 99.9% (≤ 45 s downtime/day) | NFR-9 target | Uptime monitoring (Prometheus/Grafana) |
| **Cloud (MedGemma) usage ratio** | N/A | < 20% of consults | Cost-guard target (capital efficiency) | Cost Guard Redis counter + alert |
| **PT diagnostic visibility** | ~0 (text prescription only) | Read-only access to signed findings | RBAC role (future sprint) | Access-log coverage per PT-linked case |
| **Auditability of AI accept/override** | None | 100% immutable capture | NFR-17 | Immutable audit ledger completeness check |
### 4.2 Behavioral-Change Indicators (Adoption Proxies)
Beyond raw performance, the pilot measures whether behavior actually changed the true ROI signal for a leader:
- **Retention / anti-bypass:** % of target clinicians using the structured workspace instead of unsecured external messaging (a stated success criterion).
- **Override rate & ground-truth capture:** frequency of clinician overrides (Q3) feeding the retraining corpus proves the human-AI loop works both ways.
- **Circuit-breaker engagement:** how often the Socratic breaker prevents a rushed sign-off (a direct automation-bias safety signal).
- **Compliance events:** zero unencrypted PHI leak events (Decree 13) a binary go/no-go for production.
*Baselines from [User_Research_Result.md](../Requirement_Analysis/USER_RESEARCH/User_Research_Result.md); targets from NFRs in [SOLUTION_ARCHITECTURE_SPEC.md §2.2](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md); success criteria from [CONTEXT_VISION_SCOPE.md §6](../PLAN/CONTEXT_VISION_SCOPE.md).*
---
## 5. Risk & Failure-Mode Handling — Why This Is Not Risky
The single most important message for an investment decision: **this system is engineered around the assumption that either the AI *or* the human can be wrong.** It handles all four combinations explicitly via the **4-Quadrant Human-AI Interaction Framework**.
```plantuml
@startuml VKIST_MSK_Risk_Quadrants
skinparam packageStyle rectangle
title 4-Quadrant Human-AI Safety Framework (FR-25)
rectangle "Q1 — True Agreement\n(AI correct / Doctor correct)" as Q1
rectangle "Q2 — Automation Override\n(AI correct / Doctor oversight)" as Q2
rectangle "Q3 — Clinician Subservience\n(AI hallucinates / Doctor correct)" as Q3
rectangle "Q4 — Double-Blind Failure\n(AI faulty / edge case)" as Q4
Q1 --> Q1 : GradCAM + CoT -> High-Trust Concur log
Q2 --> Q2 : Circuit Breaker -> Socratic -> BERT drift -> RAG-Referee
Q3 --> Q3 : Expose activations -> isolate artifacts -> commit ground-truth
Q4 --> Q4 : Investigation mode -> manual annotation -> telemetry queue
@enduml
```
| Risk quadrant | The danger | How the system mitigates it |
| :--- | :--- | :--- |
| **Q1 — True Agreement** | Even correct consensus needs a trace for liability | Grad-CAM + Chain-of-Thought explanation, then an immutable High-Trust Concur log block |
| **Q2 — Automation Bias** | Overloaded doctor rubber-stamps AI | Conversational Circuit Breaker intercepts rushed finalization Socratic dialogue BERT drift monitor RAG-Referee arbitration against MOH guidelines |
| **Q3 — AI Hallucination** | AI is wrong, doctor is right, but AI "authority" pressures the human | Expose pixel-level activation logic; let the clinician mask artifacts and **commit the corrected ground-truth** (which also feeds retraining) |
| **Q4 — Double-Blind Edge Case** | Both AI confidence and guideline coverage are low | Auto-switch to strict manual **Clinical Investigation Mode**; force structured morphology annotation; route the anomaly to an engineering telemetry queue (bypassing the EMR) |
**Cross-cutting safety guarantees:**
- **HITL gate (NFR-19):** No AI output becomes FINALIZED/ARCHIVED without a licensed clinician's cryptographic signature. *A human always owns the decision.*
- **Fail-safe by design:** The Circuit Breaker fails **open to deterministic MOH templates** if Triton, EMR, or a cloud LLM is down, the system degrades to a safe rule-based floor, not to an error.
- **Immutable audit (NFR-17):** Every accept/override/tier-transition/consent event is append-only and tamper-proof a complete liability defense record.
- **RAG-Referee (NFR-18):** No uncited clinical claim reaches a human.
*Source: [full_usecase_planuml.md](../../workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/Use_Case/full_usecase_planuml.md), and the Q1Q4 scenario docs under [FR_25_UC_DESIGN/](../Requirement_Analysis/UC_Design/FR_25_UC_DESIGN/).*
---
## 6. Compliance & Data Governance
Healthcare compliance is treated as a first-class architectural constraint, not an afterthought.
| Regulation | Requirement | How the architecture satisfies it |
| :--- | :--- | :--- |
| **Decree 13/2023/ND-CP** (personal health data) | Sensitive PII must be protected; data sovereignty | Client-side WebCrypto scrubbing + FastAPI Presidio redaction middleware; role-hash tokens `[BN:7f3a2c]`; all production data resides in-hospital / sovereign VN data centers |
| **Circular 46/2018/TT-BYT** (EMR) | National EMR compliance for stored records | Circular-46-compliant PDF reports; HL7/FHIR EMR push; NFR-15 (not relaxed even under NFR-16a) |
| **Air-gap (NFR-16, permanent)** | No diagnostic/identifiable data over public internet in production | Everything runs on the hospital LAN; K3s network policies isolate workloads; local-only container registries |
| **Governed cloud PoC (NFR-16a, temporary)** | Cloud LLM only as an emergency/PoC fallback | Invoked only when edge LLM unavailable **and** user consents; mandatory pre-egress redaction + audit commit; CSEK keys, 30-day lifecycle, 1-year auto-delete; **retired at PoC sign-off** |
**The compliance headline for a leader:** in production, **no patient data leaves the hospital**, every clinical statement is cited, every action is audit-logged, and a human signature is legally required to finalize anything.
*Source: [SOLUTION_ARCHITECTURE_SPEC.md §2.2, §5.8](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md), [CONTEXT_VISION_SCOPE.md §4](../PLAN/CONTEXT_VISION_SCOPE.md).*
---
## 7. Roadmap & Milestones
A strict, time-boxed 3-month cycle (June 2 September 2, 2026), executed as a hybrid of waterfall macro-phases and agile sprints. This bounds the investment and provides clear go/no-go checkpoints - However it can be tentative to change based on the actual resource of development team.
```mermaid
gantt
title VKIST MSK Pilot - 3-Month Delivery Roadmap
dateFormat YYYY-MM-DD
axisFormat %b %d
section Foundation
Sprint 0 Pitch and Architecture Spike :done, sp0, 2026-06-02, 2026-06-12
Win project pitch :milestone, m1, 2026-06-09
Sprint 1 Fast PoC Baseline :active, sp1, 2026-06-15, 2026-06-26
section Core Build
Sprint 2 Multi-Modal and NLP Integration :sp2, 2026-06-29, 2026-07-10
Sprint 3 Collaborative Workspace :sp3, 2026-07-13, 2026-07-24
section Patient and Hardening
Sprint 4 Patient-Facing PWA :sp4, 2026-07-27, 2026-08-07
Sprint 5 Feedback Pipeline and Hardening :sp5, 2026-08-10, 2026-08-21
Sprint 6 Beta Release and Evolution :sp6, 2026-08-24, 2026-09-02
Pilot cycle complete :milestone, m2, 2026-09-02
```
| Sprint | Window | Focus & key deliverable |
| :--- | :--- | :--- |
| **0** | Jun 0212 | Pitch & architecture clearance; VKIST ML model audit; CI/CD baseline |
| **1** | Jun 1526 | Fast end-to-end PoC: FastAPI pipeline + inference mask preview |
| **2** | Jun 29Jul 10 | Multi-modal NLP + Decree 13 client-side scrubbing |
| **3** | Jul 1324 | Collaborative canvas; RBAC (PT read-only); usability testing |
| **4** | Jul 27Aug 07 | Installable PWA; zero-GPU fallback for legacy phones |
| **5** | Aug 1021 | Feedback instrumentation; performance/caching; cleanup |
| **6** | Aug 24Sep 02 | Beta launch to MSK cohort; telemetry analysis; Phase-2 backlog |
**Definition of success (pilot go/no-go):** zero workflow friction added; deterministic cross-device stability (incl. CPU-bound rendering); zero Decree 13 leak events; sustained clinician retention over external messaging; successful academic-to-production model migration.
*Source: [CONTEXT_VISION_SCOPE.md §56](../PLAN/CONTEXT_VISION_SCOPE.md).*
---
## 8. Investment Ask & Scope Boundaries
### 8.1 The Resource Envelope (qualitative)
| Category | Pilot ask | Rationale |
| :--- | :--- | :--- |
| **On-prem compute** | 1 × Dell PowerEdge-class node (K3s + DB VM) + 1 × GPU node (A10/T4, 24 GB VRAM) for Triton | Single-site air-gapped deployment; capex-only, no recurring SaaS |
| **Team** | Small cross-functional squad (backend/ML, frontend, infra/DevOps) + clinical champion (UP1/UP5) for validation | Matches the 7-sprint scope; clinical sponsor is essential for governance & buy-in |
| **Software** | Open-source stack (K3s, Triton, Postgres/pgvector, Redis, MinIO, Keycloak, Prometheus/Grafana) | Zero licensing cost; no vendor lock-in |
| **Governed cloud (PoC only)** | Time-boxed GCP Vertex AI (Gemini) + Modal (MedGemma) budget, capped at <20% consult usage | NFR-16a fallback; **retired at sign-off** not a recurring production cost |
| **Data** | Access to de-identified MSK ultrasound + MOH guideline corpus | Needed to tune grading thresholds and RAG index |
### 8.2 In Scope vs. Out of Scope (Pilot)
| In scope (pilot) | Out of scope (Phase 2+) |
| :--- | :--- |
| FR-25 knee synovitis grading (UP5 primary) | Multi-hospital federation / multi-region HA |
| Grad-CAM explainability, Socratic safety loop, RAG-Referee | Full UP7 surgeon planning & UP8 patient portal at production scale |
| Decree 13 + Circular 46 compliance, HITL sign-off, audit log | Predictive analytics (cartilage degeneration, hardware failure forecasting) |
| Single-hospital air-gapped deployment | Qdrant hot-tier vector cache (only if >100K vectors / >500 QPS) |
| PoC cloud LLM tiers (NFR-16a, retired at sign-off) | Permanent cloud dependency (explicitly prohibited by NFR-16) |
### 8.3 Go / No-Go Decision Criteria
At the end of the 3-month cycle, the leader decides on Phase 2 based on objective evidence:
1. **Safety:** zero Decree 13 leak events; 100% HITL-gated finalizations; 100% MOH-cited clinical text.
2. **Performance:** inference ≤ 1.5 s; availability ≥ 99.9%; onboarding ≤ 45 min.
3. **Adoption:** measurable clinician retention within the structured workspace vs. external messaging.
4. **Strategic:** VKIST models successfully running in a referenceable production form.
> **The investment is bounded and reversible:** one hospital, one quarter, open-source stack, no production cloud dependency. If the go/no-go criteria are not met, the blast radius is a single controlled pilot — not an enterprise commitment.
---
## Appendix — Source Documents
- Vision, personas, constraints, milestones: [CONTEXT_VISION_SCOPE.md](../PLAN/CONTEXT_VISION_SCOPE.md)
- Deep personas + quantitative baselines: [User_Research_Result.md](../Requirement_Analysis/USER_RESEARCH/User_Research_Result.md)
- Architecture, FR/NFR, C4 diagrams, build-vs-buy: [SOFTWARE_ARCHITECTURE_SPEC.md](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md), [SOLUTION_ARCHITECTURE_SPEC.md](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md)
- Use cases + 4-quadrant framework: [full_usecase_planuml.md](../../workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/Use_Case/full_usecase_planuml.md), [CONTEXT_FR_25_UC_ELIT.md](../Requirement_Analysis/UC_Design/FR_25_UC_DESIGN/CONTEXT_FR_25_UC_ELIT.md)
- Risk scenarios: [FR_25_UC_DESIGN/](../Requirement_Analysis/UC_Design/FR_25_UC_DESIGN/) (Q1Q4 scenario docs)
- Impact narrative: [Motivation.md](../DEMO_EXP/Motivation.md)
*End of proposal.*

View File

@@ -0,0 +1,37 @@
# Session Memory
## Current task
Created Kilo skill `cicd-architect-grill`.
## Created skill
- Skill name: `cicd-architect-grill`
- Skill file: `.kilocode/skills/cicd-architect-grill/SKILL.md`
- Skill line count: 49 lines after long draft compression.
## Skill purpose
Design, redesign, migrate, debate CI/CD pipelines for one software project. No cloud/language/runtime/platform assumptions.
## Skill behavior
- Inspect repo context before questions.
- Extract FRs, NFRs, constraints, success criteria, Definition of Done.
- Use C4 modeling only as deep as needed.
- Choose CI/CD pattern from situation, not vendor defaults.
- Cross-check: build reproducibility, tests, scans, artifacts, secrets, promotion, approvals, rollback, observability, cost, ownership.
- Grill one decision at a time when requested.
- Output draft/final plans as YAML.
## Repository context
- Workspace root: `/Users/davestran/Downloads/vkist_internship`
- Kilo config: `kilo.json`
- User-facing style: caveman style active.
- Existing skills inspected under `/Users/davestran/Downloads/vkist_internship/.kilocode/skills/`.
## Important constraints
- Do not assume GCP, AWS, Azure, Kubernetes, serverless, GitHub Actions, GitLab CI, Jenkins, Terraform unless user asks.
- Do not implement infrastructure until user approves design plan + implementation path.
- Do not ask many questions when repo evidence answers.
## Current status
- New skill directory exists.
- `SKILL.md` written and inspected.
- No tests run: Kilo skill definition, not executable code.

View File

@@ -0,0 +1,43 @@
# Session Memory
## Current task
Create session memory for 13 Jun 26.
## Work done
- Refined `API_CONTRACT_DRAFT.md` to v0.2.0 for FR-25 Synovitis Grading Workspace.
- Added public clinical routes + internal/local safety routes from API contract.
- Added async analysis job model, model registry, GradCAM, rationale, Socratic, BERT drift, RAG evidence, activations, artifact masks, ground truth, escalation, morphology, telemetry, report sign, EMR sync.
- Updated error codes, data types, OpenAPI skeleton, PlantUML sequence.
- Ran `git diff --check`; no whitespace errors.
## Architecture review
- Loaded `improve-codebase-architecture` skill.
- Read `LANGUAGE.md`; no repo `CONTEXT.md`; no `docs/adr`.
- Explore agent inspected `vkist-ultrasound/app.py`, `pdf_service.py`, `templates/js/script.js`, API docs, FR-25 use cases.
- Found 6 deepening candidates:
- Clinical Workflow State Machine Module — Strong.
- Model Registry + Runtime Adapter Module — Strong.
- Measurement + Severity Calibration Module — Strong.
- Privacy + Artifact + Audit Module — Strong.
- Finalization + Report + EMR State Module — Strong.
- Safety + Telemetry + Feedback Module — Worth exploring.
- HTML report write was attempted, but tool aborted before file landed. Report not completed.
## Repository context
- Workspace root: `/Users/davestran/Downloads/vkist_internship`
- User-facing style: caveman style active.
- Main code area: `PILOT_PROJECT/vkist-ultrasound`
- API contract target: `PILOT_PROJECT/workspace/sprint_1_2/Design_Material/API-docs/API_CONTRACT_DRAFT.md`
## Important constraints
- Do not propose interfaces before user picks architecture candidate.
- If user picks candidate, enter grilling loop.
- If naming new deepened module after concept missing from `CONTEXT.md`, update `CONTEXT.md`.
- If user rejects candidate with load-bearing reason, offer ADR only if future reviewers need it.
- Do not commit unless explicitly asked.
## Current status
- API contract file updated.
- Architecture candidates identified.
- HTML architecture report still pending due aborted write.
- No tests run for architecture review; no code changed during review.

View File

@@ -0,0 +1,60 @@
# Session Memory
## Current task
Create/update `grill_bento_codebase` architecture skill.
## Work done
- Loaded `grill-with-docs`, `grill-bento-codebase`, `skill-creator`.
- Inspected:
- `.kilocode/skills/grill_bento_codebase/SKILL.md`
- `.kilocode/skills/grill_bento_codebase/GEMINI.md`
- `.kilocode/skills/grill_bento_codebase/BENTO_DESIGN_GUIDELINE.md`
- `.kilocode/skills/grill_bento_codebase/bento_template.html`
- `.kilocode/rules/bento_template.html`
- `.kilocode/rules/BENTO_DESIGN_GUIDELINE.md`
- Updated `grill_bento_codebase` skill assets.
- Added Bento depth constraint:
- one Bento room may contain multiple sub-components/sub-projects
- max Bento depth = 3
- depth 1 = root Bento
- depth 2 = child sub-project
- depth 3 = grandchild sub-project
- depth 4+ invalid
- Added fallback rule: detail beyond depth 3 stays inside owning Bento as local `spec/`, `spec/docs/`, `implementation/`, or `tests/`.
- Moved skill-local source refs to:
- `BENTO_DESIGN_GUIDELINE.md`
- `bento_template.html`
- Bumped guideline version to `0.5.0`.
## Skill output contract
- Context lock from FRS, NFRS, user-interaction, Software_Arch, Solution_Arch.
- Bento-BEV HTML blueprint from `bento_template.html`.
- Markdown tree-view for dev-team flatout navigation.
- Room cards.
- Interface contracts.
- Dependency map.
- Traceability matrix.
- Open decisions, one at a time.
## Validation done
- Re-read full `.kilocode/skills/grill_bento_codebase/BENTO_DESIGN_GUIDELINE.md`.
- Confirmed depth rule appears in:
- `BENTO_DESIGN_GUIDELINE.md`
- `SKILL.md`
- `GEMINI.md`
- Grep checked no stale `.kilocode/rules` refs inside `.kilocode/skills/grill_bento_codebase`.
## Repository context
- Workspace root: `/Users/davestran/Downloads/vkist_internship`
- User-facing style: caveman style active.
- Skill area: `.kilocode/skills/grill_bento_codebase`
## Important constraints
- No Bento tree depth > 3.
- No nested Bento beyond grandchild depth.
- Extra detail stays local to owning Bento room.
- No commits unless explicitly asked.
## Current status
- `SKILL.md`, `GEMINI.md`, `BENTO_DESIGN_GUIDELINE.md` updated.
- No tests run: skill docs, not executable code.

View File

@@ -0,0 +1,71 @@
# Session Knowledge — 2026-07-15
## Goal
Debug why Celery batch API results are not displaying on the client UI and fix the full pipeline: server startup, worker task registration, Triton endpoint routing, polling storm, stale result re-submission.
## Constraints & Preferences
- Client should not hammer backend with polls; need request coalescing and caching.
- Already-completed batch results should not be re-submitted or re-queried on every page view.
- Worker must target Modal Triton endpoint, not localhost.
## Progress
### Done
- Fixed `GroupResult.restore()` returning `None` by adding `result.save()` after `apply_async()` in `submit_celery_batch()`.
- Fixed server startup `ModuleNotFoundError: No module named 'backend'` by correcting `CODEBASE_ROOT` path math in `backend/routers/run_cv_inference.sh` (changed `SCRIPT_DIR/..``SCRIPT_DIR/../..`).
- Added `celery_app.autodiscover_tasks(["backend.implementation.tasks"])` to `backend/services/celery_app.py` so worker registers `run_cv_chunk`.
- Added `export TRITON_ENDPOINT=...modal.run` to `backend/start_workers.sh` so worker targets Modal instead of `localhost:8000`.
- Added server-side TTL poll cache (`_batch_poll_cache`) in `backend/services/cv_celery_service.py` with default 2000ms TTL to coalesce rapid repeat polls.
- Increased client poll interval from 1000ms → 2000ms and removed unused `pollIntervalMs` parameter from `getCvAnalyzeResultsForProfileCelery()` in `frontend/implementation/src/lib/cvAnalyzeApi.ts`.
- Fixed poll endpoint HTTP status semantics: `GET /api/test/analyze/batch/celery/{job_id}` now returns `202 Accepted` for pending, `404 Not Found` for unknown, `200 OK` for completed/failed. This removes the misleading-200 problem for in-flight jobs.
- Added backend timing logs in `cv_celery_service.py`: submission logs chunk/image counts; completion/failure logs `duration_ms` from submission to terminal state.
- Added frontend timing logs in `getCvAnalyzeResultsForProfileCelery()` for completed/failed/unknown outcomes.
- Migrated frontend config from `.env`/`.env.development` to single YAML source of truth at `config/frontend.config.yaml`, loaded via `js-yaml` in `vite.config.ts` using Vite `define`. Removed old `.env` files. Added `VITE_MODAL_OLLAMA_TARGET` to YAML so proxy target is config-driven.
- Fixed TypeScript errors in `vite.config.ts` by adding `"types": ["node"]` and including `vite.config.ts` in `tsconfig.json`.
- Added client-side in-flight coalescing + completed-result cache in `getCvAnalyzeResultsForProfileCelery()` so repeat page views dont re-submit identical work.
### In Progress
- Worker needs force-restart to pick up `autodiscover_tasks` + `TRITON_ENDPOINT` changes.
- One submitted job reached `completed: 1/2` but never finished — need worker restart + fresh batch test.
- Client-side ETA estimation for pending batch jobs is planned but not yet implemented.
### Blocked
- Worker process restart — old PID/log still active without the new fixes applied.
## Key Decisions
- Used `result.save()` because Celery does not auto-persist `GroupResult` metadata to Redis.
- Added `autodiscover_tasks` instead of manual imports because worker is launched via `-A backend.services.celery_app` and never imports `cv_tasks.py` otherwise.
- Set worker `TRITON_ENDPOINT` in `start_workers.sh` because worker is separate process that doesnt run `cv_inference_server.py`s `os.environ.setdefault()`.
- Poll cache TTL set to 2000ms to match 2s client poll interval.
- Single YAML config replaces `.env` + `.env.development` to eliminate precedence confusion.
- `VITE_MODAL_OLLAMA_TARGET` lives in YAML so proxy target is editable without touching `vite.config.ts` code.
## Next Steps
- Force-restart Celery worker (`kill -9` stale PID, remove `celery.pid`, run `./start_workers.sh start`).
- Verify worker log shows `backend.implementation.tasks.cv_tasks.run_cv_chunk` under `[tasks]`.
- Submit fresh batch and confirm it transitions `pending``completed` with actual results.
- Implement ETA estimation in backend poll response based on observed chunk throughput.
- Address client behavior that submits new jobs while previous jobs are still `pending` (7+ concurrent jobs observed in logs).
## Critical Context
- Latest server logs show 7+ concurrent jobs all stuck at `pending` with `completed: 0/2` or `1/2`, plus new POST submissions every ~10s — indicating worker is not processing tasks.
- Worker log still shows stale startup at `16:22:18` with empty `[tasks]`; fixes in `celery_app.py` and `start_workers.sh` have not been loaded by running worker.
- One job (`13fd014e`) reached `completed: 1/2, progress: 0.5` and stalled — second chunk likely failed or was never picked up.
- Modal Triton endpoint: `https://dtj-tran--triton-s3-service-unified-triton-server.modal.run`.
- Redis broker/backend: `redis://localhost:6379/0`.
- Chunk size default: 4 frames per chunk (`CELERY_CHUNK_SIZE`).
- `cv_result_cache.py` already exists with in-flight coalescing pattern; new batch poll cache follows same design.
- Frontend not currently in Celery mode (`VITE_USE_CV_CELERY=false` in YAML).
- Direct batch route still shows noisy fallback logs when Triton returns 502 on batched inference — fallback works, but should log at `info` not `warning`.
## Relevant Files
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/services/cv_celery_service.py`: Contains `submit_celery_batch()` (fixed with `result.save()`), `get_celery_batch_result()` (now with 2000ms TTL poll cache + timing logs), and poll cache constants.
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/services/celery_app.py`: Added `autodiscover_tasks(["backend.implementation.tasks"])`; routes `cv_inference.run_chunk` to `cv-inference` queue.
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/start_workers.sh`: Added `TRITON_ENDPOINT` export so worker targets Modal.
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/routers/run_cv_inference.sh`: Fixed `CODEBASE_ROOT` path math (`SCRIPT_DIR/../..`) so `PYTHONPATH` resolves `backend` package.
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/implementation/tasks/cv_tasks.py`: Defines `run_cv_chunk` Celery task.
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/routers/cv_inference.py`: Contains `/api/test/analyze/batch/celery` endpoints. Poll endpoint now returns 202/404/200.
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/src/lib/cvAnalyzeApi.ts`: Poll interval 2000ms; client-side cache + in-flight coalescing; timing logs.
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/src/hooks/useSegmentationOverlay.ts`: Calls `getCvAnalyzeResultsForProfileCelery()`.
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/config/frontend.config.yaml`: Single source of truth for frontend feature flags and URLs.
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/vite.config.ts`: Loads YAML config and injects as `import.meta.env.*`.
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/tsconfig.json`: Added `"types": ["node"]` and included `vite.config.ts`.

View File

@@ -0,0 +1,45 @@
---
title: Session Memory 18 Jun 26
date: 2026-06-18
status: active
---
## Summary
Compressed 2026-06-18 session work: `cv_code` split, PlantUML Chen ER skill, session memory.
## cv_code module split
- Split legacy `legcode/app.py` into `ml/implementation/cv/code/` into three modules:
- `config.py`
- `framework.py`
- `server.py`
- Legacy file untouched until new modules pass import checks.
- `python -m py_compile` passed all files.
- `server.py` import blocked by missing `pdf_service`.
- `framework.py` path bug: `cv_root = Path(__file__).resolve().parents[2]` skips too far. Correct: `parents[1]`.
## skill-creator inspection
- Loaded `.config/kilo/.kilo/skills/skill-creator/SKILL.md`.
- Skill structure: `SKILL.md` required; optional `GEMINI.md`, `EXAMPLES.md`, `REFERENCE.md`, `scripts/`, `assets/`.
- Template description rules apply.
## er-chen-diagram skill
- Created `.kilocode/skills/er-chen-diagram/` scaffold.
- Files written: `SKILL.md`, `EXAMPLES.md`, `REFERENCE.md`.
- Skill contents:
- Mandatory `@startchen` / `@endchen`.
- Attributes only inside `{}`.
- Primary keys use `<<key>>`.
- Relationships empty blocks, then connect lines.
- Copy exact code shells verbatim.
## Session memory housekeeping
- Listed existing memory files: `12_Jun_26.md`, `13_Jun_26.md`, `14_Jun_26.md`.
- Recorded activity in this file.
## Relational DB Design
- Created relational DB design spec per plan: ER/schema PlantUML and Chen notation in RELATIONAL_DB_SCHEMA_SPEC.md.

View File

@@ -0,0 +1,159 @@
# Coding Agent Instruction: Resolve 17 Remaining Structural Gaps in Phase 0 & Phase 1
You have already fixed the TypeScript build failures. The 17 unchecked plan items below must be addressed structurally. Each item is a checklist entry in `.kilo/plans/35-figma-designs-implementation-plan.md`. Only mark an item `[x]` in that plan file after you have implemented it AND it compiles cleanly.
## Reference Files
- Implementation Plan: `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/.kilo/plans/35-figma-designs-implementation-plan.md`
- Frontend root: `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend`
- Severity rule: Do NOT mark `[x]` unless you have implemented the item AND verified it.
## Work Order: Exact Sequence with Acceptance Tests
### 0.1 Environment Tooling
1. Configure ESLint + Prettier
- Add `.eslintrc.cjs` and `.prettierrc` (or extend recommended configs).
- Add scripts to `package.json`: `"lint": "eslint src --ext .ts,.tsx"`.
- Acceptance: `npm run lint` runs without failing on the current src tree (warnings OK, but no auto-fixable errors left hanging).
2. Set up Husky
- Install `husky` + `lint-staged`.
- Add `prepare` script: `"prepare": "husky"`.
- Add `.husky/pre-commit` to run `lint-staged`.
- Acceptance: `git add src/ && git commit` triggers Husky and blocks on lint failure.
3. Configure Vitest
- Install: `vitest`, `@testing-library/react`, `@testing-library/jest-dom`, `jsdom`.
- Add `vitest.config.ts` with `test.environment = 'jsdom'` and `include: ['src/**/*.{test,spec}.{js,ts,jsx,tsx}']`.
- Add script: `"test": "vitest"`.
- Acceptance: `npx vitest run` executes (0 tests is OK for now; framework must be correct).
4. Set up Playwright
- Install `@playwright/test`.
- Add `playwright.config.ts` with baseURL `http://localhost:5173` and webServer using `vite preview` or same.
- Add script: `"test:e2e": "playwright test"`.
- Acceptance: `npx playwright test --list` does not error (no tests yet is OK).
5. Configure Storybook
- Install `@storybook/react-vite` and run `storybook init`.
- Add 1 story for `Button` atom and 1 story for `TopBar` organism.
- Acceptance: `npm run storybook` builds the UI and the 2 stories render.
### 0.3 Component Inventory
6. Analyze 35 Figma Files
- Inspect Figma paths under `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/Design_Material/` - section: `Figma Design Source Files`.
- Produce an inventory mapping file: `spec/figma-component-specs.md` is insufficient; append `spec/component-inventory.md` with:
- Unique screen count (should be 35).
- Reusable component list by atomic depth (atoms, molecules, organisms, templates).
- One dependency graph (Mermaid is fine).
- Acceptance: `spec/component-inventory.md` exists, has 35 distinct screens, and has a Mermaid dependency graph.
7. Build Storybook Stories for Primitives
- Add stories for `Button`, `Input`, `Icon`, `Badge`, `Label`, `Toggle`, `Tooltip`.
- Acceptance: All primitives render in Storybook.
### 0.4 API Contracts
8. Cross-Reference API Contracts
- Open `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/Design_Material/API_docs/API_CONTRACT_DRAFT.md` & the `API spec from the backend-file` in `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/spec/interface-contract.md` & the data-layer definition - check the reference from context in the dirrectory (a folder) at `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/data/spec`
- Cross-check each endpoint against `src/services/*.ts`.
- Acceptance: Each service file has JSDoc lists of endpoints it covers.
9. Define TypeScript Interfaces
- Add `src/types/api.ts` with request/response interfaces for all 7 services.
- Acceptance: All 7 services import and use their types.
10. Implement HTTP + MSW Adapters
- Add `src/services/adapters/httpAdapter.ts` and `src/services/adapters/mswAdapter.ts`.
- Each service file must use the adapter via `services/index.ts` seam.
- Acceptance: `tsc --noEmit` clean; `vitest run` can mock the adapter without touching prod code.
11. Implement Auth/Logging Interceptors
- In `httpAdapter.ts`, add request interceptor that:
- Attaches `Authorization` header from a session token (mocked OK).
- Logs request method + URL.
- Normalizes errors to a `ServiceError` class.
- Acceptance: Interceptor exists and is imported only by `httpAdapter.ts` (no component imports).
### 1.2 Zustand Slices
12. Co-locate Slice Tests
- Create `src/store/sessionSlice.test.ts`, `src/store/dicomSlice.test.ts`, etc. (6 files).
- Each test imports the slice factory (`createSessionSlice`) and invokes the interface.
- Acceptance: `vitest run --coverage` shows coverage files for each slice.
13. Create Domain Selectors
- In each slice file, export type-safe selectors via `create`.
- Acceptance: `TopBar` and `BottomBar` consume slice selectors instead of inline memoization.
### 1.3 WebWorkers
14. Create `src/workers/` Files
- `cv.worker.ts`: stub with `onmessage` and `postMessage` for `preprocess` msg type.
- `llm.worker.ts`: stub with `generate` msg type.
- `guardrail.worker.ts`: stub with `safetyCheck` msg type.
- Acceptance: `tsc --noEmit` clean; workers can be imported in a hook.
15. Implement Worker Pool + Hooks
- `src/hooks/useWebWorker.ts`: generic hook accepting worker class + message schema.
- `src/hooks/useCVWorker.ts`, `useLLMWorker.ts`, `useGuardrailWorker.ts`: domain wrappers.
- Acceptance: Hooks can be instantiated without runtime errors in a test render.
16. Transferable Message Protocols
- In each worker stub, explicitly show `Transferable` usage (pass `ImageBitmap` or `ArrayBuffer` in transfer list).
- Acceptance: TypeScript types enforce transferable contracts.
17. Fallback to Main Thread
- Each hook exports `isSupported: boolean`. If `Worker` constructor not available, use a mock implementation on main thread.
- Acceptance: In jsdom test, `isSupported` is false and hook still resolves.
### 1.4 Services Deepening
18. Move `encryption.ts` Out of the Store
- Remove `import { encryptStorage } from '../utils/encryption'` from `src/store/index.ts`.
- Create `src/services/storageAdapter.ts` that exports `createSessionStorage` and `createEncryptedStorage`.
- `store/index.ts` must import `createJSONStorage(() => storageAdapter)` from `services/storageAdapter`.
- Acceptance: `grep encryptStorage src/store/index.ts` returns nothing in src/store/.
### 1.5 Route-Aware Shell
19. Implement `protected.tsx`
- Add `src/routes/protected.tsx` with a `<RequireAuth>` wrapper that redirects to `/login` when `!sessionId`.
- Wrap `/workspace` in `<RequireAuth>`.
- Acceptance: Visit `/workspace` when unauthenticated; browser redirects to `/login`.
20. `WorkspaceShell` Owns 65/35 Split
- `WorkspaceShell.tsx` must render the 65/35 layout directly: `div.flex.h-screen` with `w-[65%]` ZoneA and `w-[35%]` ZoneB.
- `routes/index.tsx` `<DiagnosticWorkspace>` must only render `<WorkspaceShell />` without re-implementing the split.
- Acceptance: `grep "flex-1" src/components/templates/WorkspaceShell.tsx` finds exactly one instance; `grep "ZoneA" src/routes/index.tsx` finds zero (or just an import).
21. Lift `initApp` Out of `App.tsx`
- Add `src/hooks/useAppInit.ts`.
- Move the `initApp` effect into this hook.
- `App.tsx` becomes: `<AppRoutes />` plus `<AppInit />` or composes it.
- Acceptance: `grep "initApp" src/App.tsx` finds nothing.
22. `React.lazy` + `Suspense` in Routes
- Use `React.lazy(() => import('../components/organisms/ZoneA'))` and same for `ZoneB`.
- Wrap `<DiagnosticWorkspace>` children in `<Suspense fallback={<div>Loading...</div>}>`.
- Acceptance: `grep React.lazy src/routes/index.tsx` finds 2+ matches.
### 1.6 Layout Responsiveness
23. Responsive Breakpoints
- `WorkspaceShell.tsx` must use `@media` or Tailwind responsive classes so that on `< md` the 65/35 stack becomes a tabbed/toggle layout (Zone A | Zone B).
- Acceptance: Resize viewport to 375px; only one Zone visible at a time.
24. 60fps Rendering Hints
- Add `will-change: transform` to animated containers.
- Add `contain: layout style paint` to the workspace root.
- Acceptance: `grep -E "will-change|contain:" src/components/templates/WorkspaceShell.tsx` finds both.
## Strict Consequences
- If you cannot complete a specific item within this prompt, do not leave it half-done. Leave the plan checkbox `[ ]` and add a comment in code explaining what is blocking you.
- All code must keep `tsc --noEmit` clean after your changes.
- Do NOT touch backend files or any file outside `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend` unless an item explicitly requires it.
## Final Verification (YOU MUST RUN THESE)
```bash
cd /Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend
npx tsc --noEmit && npm run build
npm run lint
npx vitest run
npx playwright test --list
npm run storybook -- --headless 2>&1 | tail -20
```
Produce a summary of which commands passed/failed and why.

View File

@@ -0,0 +1,189 @@
# SESSION MEMORY — 26 Jun 26
## APP PURPOSE
VKIST MSK Platform — air-gapped musculoskeletal ultrasound analysis. FR-25 Synovitis Grading (knee). Vietnamese NLP. RAG citations. HITL finalization. ≤150 MB bundle. ≤1.5s inference.
## ARCHITECTURE (C4)
### Context
- **UP5** Radiologist: primary user — load scan, review grade, finalize, sign, view GradCAM, engage circuit breaker, review RAG evidence
- **UP1** Senior Expert: clinical protocol validation, MOH guideline approval, threshold sign-off
- **UP4** Support: registration, case queue
- External: PACS (DICOM), EMR/HIS (HL7/FHIR), Triton (GPU inference), ladybugDB (ontology), pgvector (MOH guidelines HNSW), GemmaE2B/MedGemma (LLM), EmbeddingGemma (RAG embeddings)
### Containers
- **PWA** React 18 + TS + Zustand + LiteRT + MediaPipe + Dexie.js. DICOM canvas, GradCAM overlay, edge guardrail (Transformers.js BERT + OpenRedaction + pii-filter) in WebWorkers.
- **NGINX + Keepalived** VIP failover ≤2s, SSL termination
- **FastAPI** edge-inference-svc: DICOM ingest, ML pipeline orchestration, Grad-CAM, report assembly, SSE streaming
- **FastAPI** rag-svc: pgvector top-k retrieval, ladybugDB ontology, LLM consult route, RAG-Referee
- **Postgres + pgvector** HNSW index for MOH guidelines
- **Redis** JWT sessions, rate-limit, consult_mode state
- **MinIO** S3: DICOM, overlays, GradCAM heatmaps, reports
- **Triton** gRPC :8001: 3-step ML pipeline + embeddings
### 3-STEP ML PIPELINE (Triton)
1. **Angle Classification** — ConvNeXt/DenseNet/ResNet/EfficientNet/Swin-V2. Classes: med-lat, post-trans, sup-trans-flex, sup-up-long
2. **Inflammation Detection** — EfficientNet-B0 binary (post-trans, sup-up-long only)
3. **Segmentation** — DeepLabV3-ResNet101 (post-trans) OR UNet3Plus-Attention/DeepLabV3 (sup-up-long). Classes: background, effusion, fat, femur, synovium, tendon
4. **Measurement** — ROI middle 1/3 of bounding box. PIXEL_TO_MM = 45/655
5. **Severity** — Combined: effusion 60% + synovium 40%. Grades 0-3: Rất nhẹ/Nhẹ/Trung bình/Nặng
### GRADCAM
- Zero extra clicks (auto on upload)
- Base64 PNG from FastAPI
- Canvas overlay, adjust opacity
### WEB WORKER TOPOLOGY
- `cv.worker.ts` — LiteRT WASM: CV inference (angle pre-classifier). Isolated WASM.
- `llm.worker.ts` — WebLLM WASM: GemmaE2B local generation. NFR-4 1.5GB heap.
- `guardrail.worker.ts` — Transformers.js WASM/WebGPU: BERT hallucination/mal-intention/scope-breach scoring.
**Isolation:** No SharedArrayBuffer/Atomics. postMessage with structured clone only.
## FRONTEND DESIGN (WS-25)
### Layout: 65/35 split (Zone A: DICOM | Zone B: Results + Chat)
```
┌─ Top Bar (patient ID, back, sign) ──────────────────────┐
├─ Zone A (65%) ────────┬─ Zone B (35%) ────────────────────┤
│ │ Phase Indicator (sticky) │
│ 5 Canvas Layers: │ Classification Card (grade,conf) │
│ 1. Background │ Explanation Panel (GradCAM thumb) │
│ 2. GradCAM │ Chat Panel (Socratic dialogue) │
│ 3. Segmentation │ Override Card (Phase 4) │
│ 4. Annotations (SVG) │ Quick Actions │
│ 5. Viewport HUD │ │
│ │ │
│ Toolbar: pan/zoom/ │ │
│ brush/eraser/caliper │ │
├───────────────────────┴───────────────────────────────────┤
└─ Bottom Bar (NFR status, EMR sync, offline indicator) ─────┘
```
### Canvas Layers (z-index)
- 0: Background DICOM (render once, never redraw on pan/zoom)
- 10: GradCAM (CSS transform, no redraw)
- 50: Segmentation overlay
- 100: SVG Annotations (brush/caliper)
- 200: Viewport HUD (text, mousemove redraw)
- 900-999: System chrome (top/bottom bar)
### Zustand Store Keys
- `sessionId`, `patientId`, `currentPhase`
- `dicomFrame: ImageBitmap | null` (GPU-friendly)
- `dicomMetadata`, `mlResults` (single atomic object)
- `clinicianGrade`, `isOverriding`, `overrideJustification`
- `maskedRegions: MaskedRegion[]` (SVG path data, not raster)
- `chatMessages` (pruned to 20, older → IndexedDB)
- `driftScore`, `refereeIntervened`
- `zoomLevel`, `panOffset` (not persisted, RAF updates)
- `activeTool: pan|zoom|brush|eraser|caliper|pin`
- `showGradCAM`, `showSegmentation`
### 6-PHASE WORKFLOW
1. **Classification** — ML grade, confidence bar
2. **Explanation** — GradCAM visible, RAG evidence, LLM rationale
3. **Negotiation** — Socratic dialogue, circuit breaker, BERT drift
4. **Reconfiguration** — Override card, pixel activation inspector
5. **Final Decision** — Summary, HITL signature
6. **EMR Sync** — HL7/FHIR push, audit hash
## KEY API CONTRACTS
### Core Endpoints
- `POST /api/v1/sessions` → create session
- `POST /api/v1/sessions/{id}/frames` → upload DICOM (multipart)
- `POST /api/v1/analysis-jobs` → trigger ML pipeline
- `GET /api/v1/analysis-jobs/{id}` → get result (angle/inflammation/segmentation/measurement/severity)
- `GET /api/v1/analysis-jobs/{id}/steps` → step-level status
- `PATCH /api/v1/sessions/{id}/review` → clinician review (approve/correct/reject)
- `POST /api/v1/reports` → generate report
- `POST /api/v1/reports/{id}/sign` → HITL digital signature
- `POST /api/v1/reports/{id}/emr-sync` → EMR push (outbox if offline)
### Safety Endpoints
- `POST /api/v1/sessions/{id}/explanations/gradcam` → GradCAM heatmap
- `POST /api/v1/sessions/{id}/safety/circuit-breaker` → trigger circuit breaker
- `POST /api/v1/sessions/{id}/chat/socratic` → Socratic dialogue (SSE streaming)
- `POST /api/v1/sessions/{id}/drift/check` → BERT semantic drift
- `POST /api/v1/sessions/{id}/rag/evidence` → RAG guideline arbitration
- `POST /api/v1/feedback` → ground-truth override telemetry
### ML Response Shape
```json
{
"result": {
"angle": { "class": "sup-up-long", "confidence": 0.9845 },
"inflammation": { "detected": true, "confidence": 0.942 },
"segmentation": { "mask_ref": "s3://...", "overlay_ref": "s3://...", "color_legend": {...} },
"measurement": { "thickness_mm": 6.87, "pixel_to_mm": 0.0687 },
"severity": { "level": 3, "label": "severe", "combined_score": 18.4 }
}
}
```
## NFR SUMMARY
| NFR | Target | Implementation |
|-----|--------|----------------|
| NFR-4 | ≤150 MB bundle | React PWA, tree-shake, split vendor chunk, Dexie.js |
| NFR-5 | ≤1.5s inference | Triton ONNX/TensorRT, client memory ≤150MB |
| NFR-7 | ≤200ms TTFT | SSE streaming, token chunking |
| NFR-8 | Fault tolerance | IndexedDB cache, SW offline mode, session recovery |
| NFR-13 | Grad-CAM zero-click | Auto on upload, no extra UI steps |
| NFR-14 | No client GPU | LiteRT WASM, CPU fallback |
| NFR-16 | Air-gapped primary | All inference + storage on-prem via K3s |
| NFR-17 | Immutable audit | PostgreSQL WAL, no UPDATE/DELETE triggers |
| NFR-18 | RAG citations | 100% LLM text cites MOH via pgvector HNSW |
| NFR-19 | HITL signature | `signer_id != NULL` constraint before FINALIZED |
## DECREE 13 / CIRCULAR 46
- Client: Decree 13 regex scrubber (OpenRedaction + pii-filter + js-data-anonymizer) before network
- Server: FastAPI `phi_scrub` middleware (Microsoft Presidio) as ground-check
- PDF: Circular 46/2018 format, bilingual VI/EN, audit hash, signer block
## SPRINT 1-2 DELIVERABLES
### Sprint 1 (Jun 1526) — "Fast PoC Baseline"
- [ ] PWA shell: React + Zustand + PWA manifest + Service Worker + Dexie.js
- [ ] DICOM canvas: Cornerstone.js + 5-layer canvas stack
- [ ] FastAPI `/api/analyze` endpoint (sync 3-step pipeline)
- [ ] Triton 3-step model ensemble + gRPC
- [ ] NGINX + Keepalived VIP
- [ ] Grad-CAM overlay on primary viewport
- [ ] Circuit-breaker skeleton
### Sprint 2 (Jun 29Jul 10) — "Multi-Modal & NLP Integration"
- [ ] GemmaE2B browser WebLLM + Cloud MedGemma Vertex AI
- [ ] BERT drift monitor + RAG-Referee
- [ ] Socratic dialogue (SSE streaming)
- [ ] ladybugDB ontology traversal
- [ ] Decree 13 scrubber (client + server)
- [ ] Circular 46 PDF report generator
- [ ] HITL signature gate
- [ ] Immutable audit log
- [ ] EMR/HIS HL7/FHIR push
## KEY FILES
- `/workspace/sprint_1_2/Design_Material/SPRINT_1_2_ARCHITECTURE_SPEC.md` — sprint-scoped architecture
- `/PROJ_LEVEL_READING/ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md` — full software architecture
- `/workspace/sprint_1_2/Design_Material/API_docs/API_CONTRACT_DRAFT.md` — API contract v0.2.1
- `/workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/UI_Screen_Spec/Screen_Analysis_WS-25.md` — WS-25 screen analysis
- `/workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/FR_25_UC_SPEC.md` — FR-25 use case spec
- `/workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/UC_CONTEXT.md` — use case context
## PHASE SEQUENCE (ML Pipeline)
```
DICOM upload → CLAHE → Angle Classification → (if post-trans) Inflammation → Segmentation
→ (if sup-up-long) Inflammation → Segmentation → Measurement → Severity (0-3)
→ GradCAM overlay → LLM rationale (GemmaE2B/MedGemma + RAG) → Clinician review
→ HITL sign → EMR push
```
## NEXT ACTIONS
- Implement PWA shell (T1-E)
- Set up FastAPI `/api/analyze` + `/api/consult/stream` (T2-B)
- Configure Triton ensemble (T2-A)
- Build WS-25 UI components (Zone A 5-layer canvas + Zone B scrollable panel)
- Wire up edge guardrail WebWorkers (cv.worker.ts, llm.worker.ts, guardrail.worker.ts)

View File

@@ -0,0 +1,43 @@
# SESSION MEMORY — 27 Jun 26
## BACKEND ARCHITECTURE & SPECIFICATION
### 1. Project Context (from Proj_context_26_Jun_26.md)
- **Purpose**: VKIST MSK Platform for air-gapped musculoskeletal ultrasound analysis (FR-25 Synovitis Grading).
- **Core Tech Stack**: FastAPI, PostgreSQL + pgvector, Redis, MinIO (S3), Triton (GPU inference), Local LLM/BERT.
- **ML Pipeline**: 3-step process: Angle Classification $\rightarrow$ Inflammation Detection $\rightarrow$ Segmentation/Measurement $\rightarrow$ Severity Grading (0-3).
- **Safety Stack**: GradCAM, Socratic Chat, BERT drift monitor, RAG-Referee.
- **Compliance**: Decree 13 (PII scrubbing), Circular 46 (PDF reports).
### 2. Backend Structural Specification (from backend-spec.md)
The backend follows a modular "Seams" architecture:
- **API Layer**: `backend/api/` contains FastAPI routers.
- **Implementation Layer**: `backend/implementation/` contains deep modules.
- **Adapters**: `backend/implementation/adapters/` handles low-level external service communication (S3, Triton, LLM, BERT).
**Core Modules:**
- **Auth**: JWT, session management.
- **Patient**: Patient CRUD, ingestion history.
- **Session**: Frame handling, S3 storage, review patching.
- **Analysis Jobs**: Async orchestration of Triton inference via KServe v2 HTTP.
- **Safety**: GradCAM, RAG evidence, Socratic chat, circuit breaker, drift check.
- **Notification/Settings/Telemetry**: Support modules for user preferences and system monitoring.
### 3. Interface Contract (from interface-contract.md)
**Key API Categories:**
- **Auth**: `/api/v1/auth/login`, `/api/v1/users/me`.
- **Patient**: `/api/v1/patients`, `/api/v1/patients/{id}/sessions`.
- **Clinical Workflow**:
- Sessions: `/api/v1/sessions`, `/api/v1/sessions/{id}/frames`, `/api/v1/sessions/{id}/review`.
- Analysis: `/api/v1/analysis-jobs` (async), `/api/v1/analysis` (sync).
- Reports: `/api/v1/reports`, `/api/v1/reports/{id}/sign`, `/api/v1/reports/{id}/emr-sync`.
- **Safety & Explanations**:
- GradCAM: `/api/v1/sessions/{id}/explanations/gradcam`.
- Rationale/Chat: `/api/v1/sessions/{id}/explanations/rationale`, `/api/v1/sessions/{id}/chat/socratic`.
- Guardrails: `/api/v1/sessions/{id}/drift/check`, `/api/v1/sessions/{id}/rag/evidence`.
### 4. Dependency Graph
- **Auth/Patient/Session/Safety/Telemetry** $\rightarrow$ PostgreSQL.
- **Session/AnalysisJobs/Safety** $\rightarrow$ S3.
- **AnalysisJobs** $\rightarrow$ Triton (Modal Serverless).
- **Safety** $\rightarrow$ Local LLM, BERT, RAG Knowledge Base.

View File

@@ -0,0 +1,37 @@
# Session Memory: 2026-06-27 Backend NLP/LLM Consult Implementation
## Task
Build Vietnamese Clinical LLM Consult endpoints (Rationale, Socratic Chat, Guardrail Check, SSE Stream). Follow NFR-16a governance.
## Implementation Details
### 1. LLM Adapter (`backend/implementation/adapters/llm_adapter.py`)
- Build `VertexAILangchainAdapter` via `langchain-google-vertexai`.
- Add `AuditCallbackHandler` for mandatory audit log commits before LLM call (NFR-16a).
- Use `run_in_executor` for `generate` to stop FastAPI block.
### 2. BERT Adapter (`backend/implementation/adapters/bert_adapter.py`)
- Build `BERTAdapter` stubs:
- `drift_check`: Find clinical drift.
- `referee_check`: Check RAG grounding.
- `guardrail_check`: Safety check tokens/chunks.
### 3. Safety Service (`backend/implementation/safety/service.py`)
- Link LLM + BERT adapters.
- Build `_verify_pre_egress` check:
- **Consent**: Check `consent:{session_id}` in Redis.
- **Redaction**: Check redaction manifest hash.
- Update services:
- `rationale`: Context explanation.
- `socratic_chat`: History chat + BERT referee grounding.
- `guardrail_check`: Link BERT guardrails.
- `chat_stream`: SSE generator + per-chunk filter.
- Add post-egress: set `consult_mode:{session_id}` $\rightarrow$ `cloud_vertex` in Redis.
### 4. Infrastructure & API
- **Redis Adapter**: Build `backend/implementation/adapters/redis_adapter.py` for state + consent.
- **Config**: Update `backend/implementation/config.py` with Vertex AI + Redis env vars.
- **API Router**: Update `backend/api/safety_api.py` for `redaction_hash` + SSE errors.
## Governance Flow (NFR-16a)
`Request` $\rightarrow$ `Consent Check (Redis)` $\rightarrow$ `Redaction Verify` $\rightarrow$ `Audit Commit (Langchain Callback)` $\rightarrow$ `LLM Call` $\rightarrow$ `Post-Egress Log` $\rightarrow$ `Response`

View File

@@ -0,0 +1,37 @@
# Session Memory: 2026-06-27 Backend NLP/LLM Consult Implementation
## Task
Implement Vietnamese Clinical LLM Consult endpoints (Rationale, Socratic Chat, Guardrail Check, SSE Stream) with strict NFR-16a governance.
## Implementation Details
### 1. LLM Adapter (`backend/implementation/adapters/llm_adapter.py`)
- Implemented `VertexAILangchainAdapter` using `langchain-google-vertexai`.
- Added `AuditCallbackHandler` to enforce mandatory audit log commits *before* LLM calls (NFR-16a).
- Wrapped synchronous `generate` calls in `run_in_executor` to avoid blocking FastAPI event loop.
### 2. BERT Adapter (`backend/implementation/adapters/bert_adapter.py`)
- Implemented `BERTAdapter` with stubs for:
- `drift_check`: Detects clinical domain drift.
- `referee_check`: Validates RAG grounding.
- `guardrail_check`: Token/chunk level safety verification.
### 3. Safety Service (`backend/implementation/safety/service.py`)
- Integrated LLM and BERT adapters.
- Implemented `_verify_pre_egress` checklist:
- **Consent**: Validates `consent:{session_id}` exists in Redis.
- **Redaction**: Verifies redaction manifest hash.
- Updated services:
- `rationale`: Context-aware explanation generation.
- `socratic_chat`: History-based conversation with BERT referee grounding.
- `guardrail_check`: Direct integration with BERT guardrails.
- `chat_stream`: SSE generator with per-chunk guardrail filtering.
- Added post-egress state update: sets `consult_mode:{session_id}` to `cloud_vertex` in Redis.
### 4. Infrastructure & API
- **Redis Adapter**: Created `backend/implementation/adapters/redis_adapter.py` for session state and consent management.
- **Config**: Updated `backend/implementation/config.py` with Vertex AI and Redis environment variables.
- **API Router**: Updated `backend/api/safety_api.py` to pass `redaction_hash` to services and handle SSE error events.
## Governance Flow (NFR-16a)
`Request` $\rightarrow$ `Consent Check (Redis)` $\rightarrow$ `Redaction Verify` $\rightarrow$ `Audit Commit (Langchain Callback)` $\rightarrow$ `LLM Call` $\rightarrow$ `Post-Egress Log` $\rightarrow$ `Response`

View File

@@ -0,0 +1,301 @@
# Frontend Development Progress - 27 Jun 26
## Session Summary
Date: 2026-06-27
Phase: 2 - Component Architecture & Deepening (35 Figma Designs Implementation Plan)
Mode: Caveman Lite
---
## 1. TopBar Refactor (Phase 2.0)
### Problem
TopBar reached into Zustand store for 4 fields (`patientId`, `clinicianGrade`, `isOverriding`, `sessionId`, `currentPhase`) and inlined sign button. Violated molecule abstraction layer.
### Solution
Refactored `TopBar` to consume `PatientHeader` and `ActionCluster` molecules via props instead of store selectors.
### Files Changed
- `src/components/organisms/TopBar.tsx`
- Removed direct `useDiagnosticWorkspaceStore` import
- Added props interface: `patientId`, `patientName`, `role`, `status`, `sessionId`, `currentPhase`, `canSign`, `isCircuitBroken`, `onSign`, `onCircuitBreak`, `onBack`, `onMore`
- Replaced inline patient info + sign button with `<PatientHeader />` and `<ActionCluster />` molecules
- Added `canSign` prop to `ActionCluster` for HITL gating
- `src/components/molecules/ActionCluster.tsx`
- Added `canSign?: boolean` prop
- Added `signDisabled` computed value: `isCircuitBroken || !canSign`
- Sign button now uses `disabled={signDisabled}` instead of just `isCircuitBroken`
- `src/components/templates/WorkspaceShell.tsx`
- Wired all TopBar props from store using selectors
- Passes `patientId`, `sessionId`, `currentPhase`, `canSign`, `isCircuitBroken`, `onSign`, `onCircuitBreak`
- `onCircuitBreak` now accepts `(active: boolean) => void` and triggers `window.confirm` modal for break/recovery
---
## 2. Canvas & Visualization Deepening (Phase 2.1)
### DiagnosticCanvas Enhancements
- Tool State: reads `activeTool`, `setActiveTool`, `zoomLevel`, `panOffset`, `showGradCAM`, `showSegmentation` from Zustand `uiSlice`
- 5-Layer Canvas Stack: 5 layers with dynamic rendering
- Layer 0: Cornerstone.js DICOM viewport
- Layer 10: GradCAM overlay (conditional via `showGradCAM`)
- Layer 50: Segmentation mask (conditional via `showSegmentation`)
- Layer 100: SVG Annotations + OffscreenCanvas
- Layer 200: Viewport HUD + live coords/zoom
- Performance:
- `requestAnimationFrame` dirty-flag loop for annotation canvas
- `ResizeObserver` syncs annotation canvas resolution
- Cornerstone lifecycle: init + viewport update (no reload on pan/zoom)
- `useCallback` for event handlers
- Touch:
- `onTouchStart`, `onTouchMove` for mobile/tablet coords
- Touch events mapped to same coordinate system
- Tool Palette:
- Extracted constant `TOOLS` array with icon mapping (`pin``map-pin`)
- Active tool state store-driven
- Live HUD:
- Replaced static HUD with live `mousePos` state
- Zoom level from store
### Canvas Performance
- Minimized Cornerstone redraws: separate image load from viewport updates
- `useRef` for OffscreenCanvas, RAF loop, dirty flag, image tracking
- Added `mousePos` for live coords
---
## 3. Reasoning & Chat Panel Deepening (Phase 2.2)
### ZoneB Composition
- Replaced placeholder with `PhaseIndicator` + `ReasoningPanel` + `ChatPanel`
- Layout: Sticky PhaseIndicator top, scrollable content below
- `ZoneBProps` interface covers all data flows
### ReasoningPanel Enhancements
- Added optional `PhaseIndicator` via `showPhaseIndicator` prop
- Classification Card:
- Color-coded grade badge (0-3) with tooltip
- Confidence score with `ConfidenceBar`
- Explanation Panel:
- GradCAM thumbnail with hover overlay
- Thickness measurement extraction/display
- Show Full GradCAM toggle via `showFullGradCam` / `onToggleFullGradCam` props
- Override Card:
- Rendered when `onSubmitOverride` provided
- Composes existing `OverrideCard` molecule
- Quick Actions Bar:
- `CircuitBreaker` molecule
- `NFRIndicator` molecule
- Report button
- EMR sync button
- Fullscreen toggle
### ChatPanel Enhancements
- Input:
- Replaced `<input>` with auto-resizing `<textarea>`
- `Enter` sends
- `Shift+Enter` inserts newline
- Auto-resize to 120px
- Streaming:
- Added `isStreaming` prop
- Streaming indicator in header
- Inline pulsing cursor on last assistant message when streaming
- Message Actions:
- Hover-revealed toolbar on non-system messages
- Copy button (`navigator.clipboard.writeText`)
- Regenerate button (assistant only)
- Feedback buttons: helpful (heart) / not helpful (alert-circle)
- All optional via callback props
- Virtualization:
- Placeholder: `react-window` `FixedSizeList` integration point
- Current: standard overflow-y-auto
---
## 4. Tool State & Annotations (Phase 2.3)
### uiSlice Additions
- `brushSize: number` (default 5, range 1-50)
- `brushColor: string` (default '#ff0000')
- `eraserSize: number` (default 10, range 1-100)
- `caliperUnit: 'mm' | 'px'` (default 'mm')
- `pressureSensitivity: boolean` (default false)
- Setters with validation
### annotationSlice Additions
- `selectedAnnotationId: string | null`
- `lockedLayerIds: string[]`
- Actions:
- `addAnnotation`
- `selectAnnotation`
- `deleteAnnotation`
- `updateAnnotation`
- `lockLayer`
- Derived selectors:
- `selectSelectedAnnotation`
- `selectLockedLayers`
- Undo/redo for all annotation mutations
### Store Updates
- `store/index.ts`:
- Exported new selectors
- Updated `resetSession` to clear new state fields
---
## 5. Safety & Compliance Wiring (Phase 2.4)
### Circuit Breaker
- `ActionCluster` passes `isCircuitBroken` state
- Sign button disabled when circuit broken
- `WorkspaceShell` applies `ring-2 ring-red-500 animate-pulse` when `isOverriding=true`
- Confirmation via `window.confirm` before break/recovery
### Defer 13 Scrubber (PIIScrubber)
- Fixed broken destructuring in `src/components/molecules/PIIScrubber.tsx`
- Added missing props: `text`, `onChange`, `className`
- Compiles and binds props correctly
### HITL Signature Gate
- Fixed missing `Input` import in `src/components/molecules/HITLGate.tsx`
- Added `import { Input } from '../atoms/Input';`
- Ready for credential entry modal
### NFR Status Indicator (BottomBar)
- Added `WorkerHealthEntry` interface
- Extended `BottomBarProps` with `workerHealth` and `isOffline`
- Integrated `NFRIndicator` for live status
- Removed hardcoded 'offline' when real values present
---
## 6. TypeScript & Lint Verification
### TypeScript
- `tsc --noEmit` passes clean
- All modified files compile clean
### ESLint
- All modified files pass lint
- Pre-existing warnings: stories/tests (.stories.tsx old Storybook imports, test `.set` destructuring)
- Not introduced this session
---
## 7. Plan File Updates
Updated `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/.kilo/plans/35-figma-designs-implementation-plan.md`:
- Tick 2.0 TopBar deepening (molecule consumption)
- Tick 2.1 Canvas & Visualization
- Tick 2.2 Reasoning & Chat Panel (Override Card + Quick Actions)
- Tick 2.3 Interaction & State Management (tool state, annotations, measurements)
- Tick 2.4 Safety & Compliance (Circuit Breaker, PII Scrubber, HITL Gate, NFR Indicator)
---
## 8. Known Pre-existing Issues (Not Introduced This Session)
- `src/services/safetyService.ts`: Vite build error (unrelated to Phase 2)
- Storybook stories: import `@storybook/react` directly instead of framework package
- Store slices tests: unused `set` destructuring warnings
---
## 9. Key Architectural Improvements
1. **Molecule Layer Strengthened**: TopBar no longer leaks store selectors; uses PatientHeader + ActionCluster props
2. **5-Layer Canvas Stack**: Dynamic store-driven visibility with OffscreenCanvas annotations and RAF rendering
3. **ZoneB Composed**: Clean separation of PhaseIndicator, ReasoningPanel, ChatPanel
4. **Tool State Centralized**: All tool preferences in uiSlice with validation
5. **Annotation System Ready**: Vector storage, undo/redo hooks, layer locking
6. **Safety Molecules Wired**: CircuitBreaker, PIIScrubber, HITLGate, NFRIndicator integrated into organisms
7. **Zero New Type Errors**: All changes type-check clean
---
## 10. Next Steps
- Wire Cornerstone DICOM loading (placeholder)
- Implement react-window virtualization in ChatPanel at message threshold
- Add Worker hooks (Phase 3.1)
- Implement Offline/Dexie sync (Phase 3.2)
- Build safety modals (CircuitBreaker confirm, HITL credential entry)
- Create 5-layer rendering tests
---
## 11. Phase 3: Worker Integration + Offline Sync (Appended 2026-06-28)
### Worker Hooks
- `src/hooks/workers/useCVWorker.ts`: DICOM frame → CLAHE resize → `cv.worker` → angle class + confidence. Real worker + mock adapter.
- `src/hooks/workers/useLLMWorker.ts`: prompt build + RAG context → `llm.worker` stream with token buffering for smooth display.
- `src/hooks/workers/useGuardrailWorker.ts`: pre-scan user input for hallucination/mal-intent → safety score → UI warning if threshold exceeded.
- `src/hooks/workers/useDriftScore.ts`: extract drift score from guardrail module.
- `src/hooks/workers/useSocraticChat.ts`: socratic dialogue via `llm.worker`.
- `src/hooks/workers/mocks.ts`: mock worker adapters for test environment using setTimeout.
- `src/workers/lifecycle.ts`: `WorkerPool` (lazy init, idle GC 5 min, error boundary).
- Barrel: `src/workers/index.ts` exports workerPool + message helpers.
### Offline Storage + Sync
- `src/services/db.ts` Dexie schema: `sessions`, `chatMessages`, `pendingReports`, `auditLog`.
- `src/services/sync.ts` SyncEngine: online/offline detect, queue when offline, exponential backoff retry (1s base, 30s cap, 5 max), LWW conflict resolution.
- `src/services/index.ts` exports `db`, `syncEngine`.
### PWA / Service Worker
- `vite.config.ts`: `vite-plugin-pwa` `generateSW` (Workbox). CacheFirst static assets, NetworkFirst API runtime, precache manifest.
- `npm i -D vite-plugin-pwa` added.
### Build Verification
- Pre-existing TS errors unchanged in `src/services/adapters/`, `src/*Service.ts`, store tests, stories. None introduced in new files.
---
## 12. Phase 4: Performance, NFR, a11y, i18n (Appended 2026-06-28)
### Bundle + Rendering
- `vite.config.ts`: `rollup-plugin-visualizer` (gzipSize, `dist/bundle-stats.html`). `manualChunks`:
- `vendor-media`: `cornerstone-*`, dicom parsers
- `vendor-store`: dexie, zustand
- `worker`: `src/workers/*`
- `app-core`: react, react-dom, i18next, react-router-dom, date-fns
- `DiagnosticCanvas.tsx`: `useMemo` on viewport transform.
- `ChatPanel.tsx`: `React.memo` `MessageBubble` with comparator `id`, `content`, `isStreaming`.
- `src/services/db.ts`: `saveChatMessagesWithEviction(messages, maxPerSession=20)` caps chat history via FIFO bulkPut + bulkDelete.
### i18n
- `src/i18n.ts`: i18next + `initReactI18next` + `i18next-http-backend`. Lng = `navigator.language` if starts with `en`, else `en`. `/locales/{{lng}}.json`.
- `src/locales/en.json`, `src/locales/vi.json`: 37 UI keys across TopBar, PatientHeader, BottomBar, ChatPanel, ReasoningPanel, WorkspaceShell.
- `src/App.tsx`: root wrapped in `<I18nextProvider i18n={i18n}>`.
- ReasoningPanel/BottomBar/TopBar/ChatPanel: hardcoded strings replaced with `t('Component.key')`.
### Accessibility
- `src/components/templates/WorkspaceShell.tsx`: root wrapper gets `role="main"`, `tabIndex={-1}`.
- `src/components/organisms/BottomBar.tsx`: NFR status container `aria-live="polite"`.
- `src/index.css`: global `:focus-visible` outline 2px solid var(--color-primary-500), offset 2px.
- Grad-CAM `<img alt="Grad-CAM">` retained; decorative icons already `aria-hidden="true"`.
### NFR / Lockdown
- `src/nfr/LockdownValidator.ts`: runtime DOM + fetch + XHR patch. Returns external URLs not matching `window.location.origin`.
- `src/services/adapters/httpAdapter.ts`: `VITE_API_BASE_URL` must start with `/` (relative) else throw. Same-origin only.
- `vite.config.ts`: `server.host='127.0.0.1'`, `server.https=false`. `preview.headers` CSP:
- `script-src 'self' 'wasm-unsafe-eval'`
- `style-src 'self' 'unsafe-inline'`
- `img-src 'self' data: https://fonts.gstatic.com`
- `font-src 'self' https://fonts.gstatic.com`
- `connect-src 'self' https://fonts.googleapis.com`
- Repo grep found zero external URL strings in `src/` and `public/`. Workbox retained Google Fonts runtime caching as sole allowed external origin.
### Bugfixes
- `ReasoningPanel.tsx`: missing `function ReasoningPanel(props)` caused syntax error; fixed + props destructured. `nfrStatus` type inline to avoid broken re-export path.
- `i18n.ts`: `useTranslation` export fixed; `HttpBackend` added as plugin.
- `ZoneB.tsx`: `import { ReasoningPanel }` → default import.
### Plan Updates
- `.kilo/plans/35-figma-designs-implementation-plan.md`:
- Phase 3.1 + 3.2 ticked during worker/offline work.
- Phase 4.1-4.3 items ticked where verification supports.
### Known Compile State
- `npm run build` fails on pre-existing unrelated files (dicomService typings, store tests, stories missing modules). New Phase 3/4 files compile clean.

View File

@@ -0,0 +1,225 @@
# Frontend Development Progress - 27 Jun 26
## Session Summary
Date: 2026-06-27
Phase: 2 - Component Architecture & Deepening (35 Figma Designs Implementation Plan)
Mode: Caveman Lite
---
## 1. TopBar Refactor (Phase 2.0)
### Problem
TopBar was directly reaching into the Zustand store for 4 fields (patientId, clinicianGrade, isOverriding, sessionId, currentPhase) and inlining the sign button. This violated the molecule abstraction layer.
### Solution
Refactored `TopBar` to consume `PatientHeader` and `ActionCluster` molecules via props instead of store selectors.
### Files Changed
- `src/components/organisms/TopBar.tsx`
- Removed direct `useDiagnosticWorkspaceStore` import
- Added props interface: `patientId`, `patientName`, `role`, `status`, `sessionId`, `currentPhase`, `canSign`, `isCircuitBroken`, `onSign`, `onCircuitBreak`, `onBack`, `onMore`
- Replaced inline patient info + sign button with `<PatientHeader />` and `<ActionCluster />` molecules
- Added `canSign` prop to `ActionCluster` for HITL gating
- `src/components/molecules/ActionCluster.tsx`
- Added `canSign?: boolean` prop
- Added `signDisabled` computed value: `isCircuitBroken || !canSign`
- Sign button now uses `disabled={signDisabled}` instead of just `isCircuitBroken`
- `src/components/templates/WorkspaceShell.tsx`
- Wired all TopBar props from store using selectors
- Passes `patientId`, `sessionId`, `currentPhase`, `canSign`, `isCircuitBroken`, `onSign`, `onCircuitBreak`
- `onCircuitBreak` now accepts `(active: boolean) => void` and triggers `window.confirm` modal for break/recovery
---
## 2. Canvas & Visualization Deepening (Phase 2.1)
### DiagnosticCanvas Enhancements
- Tool State Management: Component now reads `activeTool`, `setActiveTool`, `zoomLevel`, `panOffset`, `showGradCAM`, `showSegmentation` from Zustand `uiSlice`
- 5-Layer Canvas Stack: All 5 layers implemented with proper dynamic rendering
- Layer 0: Cornerstone.js DICOM viewport
- Layer 10: GradCAM overlay (conditionally rendered via `showGradCAM` store toggle)
- Layer 50: Segmentation mask (conditionally rendered via `showSegmentation` store toggle)
- Layer 100: SVG Annotations layer with OffscreenCanvas support
- Layer 200: Viewport HUD with live mouse coordinates and zoom level
- Performance Optimizations:
- `requestAnimationFrame` dirty-flag render loop for annotation canvas
- `ResizeObserver` to keep annotation canvas resolution in sync
- Separated Cornerstone lifecycle into initialization + viewport update effects (no reload on pan/zoom)
- `useCallback` for event handlers
- Touch Support:
- `onTouchStart`, `onTouchMove` handlers for mobile/tablet coordinate tracking
- Touch events mapped to same coordinate system as mouse
- Tool Palette:
- Extracted to constant `TOOLS` array with proper icon mapping (`pin` corrected to `map-pin`)
- Active tool state now store-driven, not prop-driven
- Live HUD:
- Replaced static "X: 124.5mm | Y: 88.2mm" with live `mousePos` state
- Zoom level pulled from store
### Canvas Performance
- Minimized Cornerstone redraws by separating image load from viewport updates
- Used `useRef` for OffscreenCanvas, RAF loop, dirty flag, image tracking
- Added `mousePos` state for live coordinate display
---
## 3. Reasoning & Chat Panel Deepening (Phase 2.2)
### ZoneB Composition
- Replaced placeholder with full composition: `PhaseIndicator` + `ReasoningPanel` + `ChatPanel`
- Layout: Sticky PhaseIndicator at top, scrollable content area below
- Comprehensive `ZoneBProps` interface covering all data flows
### ReasoningPanel Enhancements
- Added optional `PhaseIndicator` (controlled by `showPhaseIndicator` prop)
- Classification Card:
- Color-coded grade badge (0-3) with tooltip wrapper
- Confidence score with `ConfidenceBar`
- Explanation Panel:
- GradCAM thumbnail with hover overlay ("Expand View")
- Thickness measurement extraction and display
- "Show Full GradCAM" toggle via `showFullGradCam` / `onToggleFullGradCam` props
- Override Card:
- Conditionally rendered when `onSubmitOverride` callback provided
- Composes existing `OverrideCard` molecule
- Quick Actions Bar:
- `CircuitBreaker` molecule (toggle)
- `NFRIndicator` molecule
- Report generation button
- EMR sync button
- Fullscreen toggle
### ChatPanel Enhancements
- Input Handling:
- Replaced `<input>` with auto-resizing `<textarea>`
- `Enter` sends message
- `Shift+Enter` inserts newline
- Auto-resize up to 120px height
- Streaming Support:
- Added `isStreaming` prop
- Streaming indicator in header ("Streaming...")
- Inline pulsing cursor on last assistant message when streaming
- Message Actions:
- Hover-revealed action toolbar on non-system messages
- Copy button (uses `navigator.clipboard.writeText`)
- Regenerate button (assistant messages only)
- Feedback buttons: helpful (heart) / not helpful (alert-circle)
- All actions optional via callback props
- Virtualization:
- Placeholder comment block showing `react-window` `FixedSizeList` integration point
- Current implementation uses standard overflow-y-auto for simplicity
---
## 4. Tool State & Annotations (Phase 2.3)
### uiSlice Additions
- `brushSize: number` (default 5, range 1-50)
- `brushColor: string` (default '#ff0000')
- `eraserSize: number` (default 10, range 1-100)
- `caliperUnit: 'mm' | 'px'` (default 'mm')
- `pressureSensitivity: boolean` (default false)
- Setters for all above with validation
### annotationSlice Additions
- `selectedAnnotationId: string | null`
- `lockedLayerIds: string[]`
- Actions:
- `addAnnotation`
- `selectAnnotation`
- `deleteAnnotation`
- `updateAnnotation`
- `lockLayer`
- Derived selectors:
- `selectSelectedAnnotation`
- `selectLockedLayers`
- Undo/redo integration for all annotation mutations
### Store Updates
- `store/index.ts`:
- Exported new selectors
- Updated `resetSession` to clear new state fields
---
## 5. Safety & Compliance Wiring (Phase 2.4)
### Circuit Breaker
- `ActionCluster` updated to pass `isCircuitBroken` state
- Sign button disabled when circuit broken
- `WorkspaceShell` applies `ring-2 ring-red-500 animate-pulse` when `isOverriding=true`
- Confirmation modal via `window.confirm` before break/recovery
### Defer 13 Scrubber (PIIScrubber)
- Fixed broken destructuring in `src/components/molecules/PIIScrubber.tsx`
- Added missing props: `text`, `onChange`, `className`
- Component now compiles and correctly binds props
### HITL Signature Gate
- Fixed missing `Input` import in `src/components/molecules/HITLGate.tsx`
- Added `import { Input } from '../atoms/Input';`
- Component now ready for credential entry modal
### NFR Status Indicator (BottomBar)
- Added `WorkerHealthEntry` interface
- Extended `BottomBarProps` with `workerHealth` and `isOffline`
- Integrated `NFRIndicator` molecule for live status display
- Removed hardcoded 'offline' status when real values present
---
## 6. TypeScript & Lint Verification
### TypeScript
- `tsc --noEmit` passes clean
- All modified files compile without errors
### ESLint
- All modified files pass lint
- Pre-existing warnings in stories/tests (.stories.tsx using old Storybook imports, test `.set` destructuring)
- These are not introduced by this session's changes
---
## 7. Plan File Updates
Updated `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/.kilo/plans/35-figma-designs-implementation-plan.md`:
- Tick off 2.0 TopBar deepening (molecule consumption)
- Tick off 2.1 Canvas & Visualization (all sub-items)
- Tick off 2.2 Reasoning & Chat Panel (all sub-items including Override Card + Quick Actions)
- Tick off 2.3 Interaction & State Management (tool state, annotations, measurements)
- Tick off 2.4 Safety & Compliance Features (Circuit Breaker, PII Scrubber, HITL Gate, NFR Indicator)
---
## 8. Known Pre-existing Issues (Not Introduced This Session)
- `src/services/safetyService.ts`: Vite build error (unrelated to Phase 2 work)
- Storybook stories: import `@storybook/react` directly instead of framework package
- Store slices tests: unused `set` parameter destructuring warnings
---
## 9. Key Architectural Improvements
1. **Molecule Layer Strengthened**: TopBar no longer leaks store selectors; consumed via PatientHeader + ActionCluster props
2. **5-Layer Canvas Stack**: Dynamic, store-driven layer visibility with OffscreenCanvas annotation support and RAF-based rendering
3. **ZoneB Composed**: Clean separation of PhaseIndicator, ReasoningPanel, ChatPanel
4. **Tool State Centralized**: All tool preferences in uiSlice with validation
5. **Annotation System Ready**: Vector-based storage, undo/redo hooks, layer locking in place
6. **Safety Molecules Wired**: CircuitBreaker, PIIScrubber, HITLGate, NFRIndicator integrated into organisms
7. **Zero New Type Errors**: All changes type-check clean
---
## 10. Next Steps
- Wire actual Cornerstone DICOM loading (currently placeholder)
- Implement react-window virtualization in ChatPanel when message count exceeds threshold
- Add Worker hooks (Phase 3.1)
- Implement Offline/Dexie sync (Phase 3.2)
- Build safety modals (CircuitBreaker confirm, HITL credential entry)
- Create actual 5-layer rendering tests

159
session_memory/6_Jul_26.md Normal file
View File

@@ -0,0 +1,159 @@
---
title: Session Memory 6 Jul 26
date: 2026-07-06
status: active
---
## Summary
Sprint_1_2 clinical chat saw major LLM UX, streaming, and inference work (Jul 56). The highest-risk open issue is **frontend markdown rendering of generated assistant text** — it has caused tab freezes and catastrophic crashes during reasoning streams. Reasoning paths now use **plain text only** (`StreamingPlainText`) as a mitigation. **Beta functionality** (Planning mode + 🔥 high reasoning level) must be completed **by end of this week** (target: **Friday 10 Jul 2026**).
---
## Change log (what was updated)
### Clinical chat UI & model lifecycle
| Area | Change | Key files |
|------|--------|-----------|
| LLM loading bubble | Install vs load phases with distinct copy, progress bar, disabled composer | `ClinicalChatPanel.tsx`, `useClinicalChat.ts`, `modelLoadProgress.ts` |
| Install vs load semantics | First OPFS download (~1.9 GB) vs cached checkpoint init into worker/GPU | `useClinicalChat.ts` (`ModelLoadPhase: 'installing' \| 'loading'`) |
| Sidebar card switch | Both diagnosis + review layers stay mounted (CSS hide/show) so Gemma is not torn down on carousel switch | `SidebarLayerCarousel.tsx` |
| OPFS persistence | Completed install survives reload; interrupted download not resumable (manifest only written on success) | `opfsModelStore.ts` |
| Worker init progress | `init_progress` events wired through `LlmWorkerClient.init(onProgress)` | `llmWorkerClient.ts`, `llm.worker.ts` |
### Inference modes, reasoning levels & backends
| Area | Change | Key files |
|------|--------|-----------|
| Unified chat modes | `ask` merged into `chat`; inference modes: **Chat**, **Planning** (beta), **Agent** | `clinicalChatModes.ts`, `analyzePromptComplexity.ts` |
| Reasoning levels | 🧘 chill / 🤔 moderate / 🔥 high (beta); bar visibility persisted | `chatReasoningLevel.ts`, `ClinicalChatPanel.tsx` |
| Edge vs server toggle | 🤔 moderate: **Máy** (Gemma 4 E2B edge) vs **Server** (Gemma 4 E4B Modal Ollama `think:true`) | `reasoningModelBackend.ts`, `ollamaLlmClient.ts`, `inferenceBackend.ts` |
| Ollama dev proxy | `VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat`, model `gemma4:e4b` | `.env.development`, `vite.config.ts` |
| Agent mode | Uses Modal Ollama E4B when configured; tool loop unchanged | `clinicalChatModes.ts`, `runClinicalChatTurn.ts` |
| OOM mitigation | Bootstrap at 2048 tokens; `releaseInference()` before reload; reuse engine when `configured.maxTokens >= required` | `clinicalChatConfig.ts`, `llm.worker.ts`, `llmModelBootstrap.ts` |
| Qwen3 experiment | Dual-model (Qwen `.litertlm` for moderate) attempted then **disabled**`usesQwenReasoningLevel()` returns `false`; moderate uses Gemma CoT again | `qwenOpfsModelStore.ts`, `reasoningLlmClient.ts`, `chatReasoningLevel.ts` |
### Streaming, CoT split & markdown
| Area | Change | Key files |
|------|--------|-----------|
| CoT split | `splitGemmaThoughtOutput`, `isThoughtChannelComplete`; thought vs answer channels in message state | `prompts.ts`, `clinicalChat.ts`, `useClinicalChat.ts` |
| Collapsible reasoning | `ClinicalChatThought` — expand while streaming, auto-collapse when thought completes | `ClinicalChatThought.tsx` |
| Stream throttle | RAF-coalesced updates (~60/s) to reduce main-thread pressure | `streamUpdateThrottle.ts` |
| Token-by-token Ollama | Imperative DOM via `clinicalChatStreamRegistry` + `StreamingPlainText` (bypasses React 18 batching) | `clinicalChatStreamRegistry.ts`, `StreamingPlainText.tsx`, `ollamaLlmClient.ts` |
| Markdown renderer | Custom `ChatMarkdown` (bold, italic, code, lists, headings) — **deferred until stream ends** via `requestIdleCallback` + `startTransition` | `ChatMarkdown.tsx`, `ClinicalChatMessageBubble.tsx` |
| Reasoning = plain text | When `tracksThought`, thought + answer use `StreamingPlainText` only — **no `ChatMarkdown`** | `ClinicalChatThought.tsx`, `ClinicalChatMessageBubble.tsx` |
### Agent tools & Modal testing
| Area | Change | Key files |
|------|--------|-----------|
| Tool catalog doc | Walkthrough of Edge-LLM agent tools: `exa_search`, `supabase_query`, `escalate_medgemma` | session + `agent_tools_contract.md` |
| 3-layer smoke harness | Layer 0 Modal Ollama, Layer 1 BFF routes, Layer 2 browser `ToolExecutor` | `ml/tests/agent_tools/` |
| Python reference tests | Modal `/api/chat` streaming + thinking cases | `PILOT_PROJECT/tmp/test_endpoint.py`, `test_endpoint_img.py` |
| Gemma4 E4B deploy script | Modal Ollama serverless for Gemma 4 E4B | `PILOT_PROJECT/tmp/GemmaE4B_ollama_deploy.py` |
| gemma4_e2b lab | Tool smoke panel (no Gemma) for isolated tool calls | `ml/tests/gemma4_e2b/src/lib/toolSmoke.ts` |
---
## Critical issue: markdown rendering → catastrophic crash
### Symptom
Rendering **LLM-generated markdown** in the clinical chat UI (especially during or immediately after **reasoning / CoT streams**) has caused:
- Main-thread freezes (composer unresponsive while tokens still arrive)
- Tab crashes / OOM under combined **WebGPU model memory + large text buffers + markdown parse**
### Root cause (confirmed in debugging)
1. **Per-token full re-parse** — early implementation ran `ChatMarkdown` / `renderBlocks()` on the entire growing thought string every token → hundreds of parses per second.
2. **React re-render storm**`setMessages` on every token re-rendered the full message list.
3. **Post-stream markdown** — even with “parse after stream ends”, heavy `renderBlocks()` on long CoT + answer text still spikes CPU/memory.
4. **Dual large strings**`rawAccumulator`, `thoughtContent`, and `content` can all hold 2048-token traces simultaneously at 🤔 moderate.
### Mitigations applied (6 Jul)
- Stream updates throttled (`createStreamUpdateThrottle`)
- Plain text while `streaming === true`
- `React.memo` on `ClinicalChatMessageBubble`
- Reasoning paths (`tracksThought`) → **`StreamingPlainText` only, markdown disabled**
- Ollama path → imperative DOM updates per token
- `ChatMarkdown` uses idle callback + `startTransition` for post-stream formatting (chill / non-reasoning answers only)
### Current policy
> **Be wary of re-enabling markdown in reasoning mode until a safe renderer is proven.**
`ClinicalChatThought.tsx` comment: *"Reasoning panel — plain text only (markdown disabled while isolating stream crashes)."*
### Follow-up for later sprints
- [ ] Reproduce crash with a minimal `ChatMarkdown` + long fixture (no LLM) — isolate `renderBlocks` vs React
- [ ] Consider `react-markdown` with strict plugins OR server-side pre-render for final answer only
- [ ] Cap thought trace length in UI (truncate + “show more”)
- [ ] Virtualize message list for long sessions
- [ ] Re-test 🔥 high mode and Planning mode with any new markdown path before removing plain-text guard
- [ ] Audit whether chill-mode `ChatMarkdown` after stream is safe on 8 GB RAM devices
---
## Beta functionality — deadline end of week
**Target: complete by Friday 10 Jul 2026** (end of sprint week).
Features still marked `beta: true` (UI tab/button disabled until ready):
| Feature | ID | Location | What “done” means |
|---------|-----|----------|-------------------|
| **Planning mode** | `planning` | `clinicalChatModes.ts` | Checklist output stable; remove `beta` flag; selectable in mode tabs |
| **High reasoning** | `high` 🔥 | `chatReasoningLevel.ts` | 2048-token multi-turn works without crash; remove `beta` flag |
Related work not yet beta-flagged but in scope:
- Agent tool integration (Layer 3 after smoke tests pass)
- Live BFF tools (`VITE_CLINICAL_CHAT_MOCK_TOOLS=false` + credentials)
- Edge/server reasoning toggle polish (default server E4B when Modal up)
---
## Architecture snapshot (clinical LLM, 6 Jul EOD)
```
User message
├─ Mode: chat ─┬─ 🧘 chill → Gemma E2B edge (no CoT)
│ ├─ 🤔 moderate → Máy: Gemma E2B + CoT (plain text)
│ │ Server: Gemma E4B Ollama think:true (plain text)
│ └─ 🔥 high (BETA) → disabled in UI
├─ Mode: planning (BETA) → disabled in UI
└─ Mode: agent → Gemma E4B Ollama + tools (exa, supabase, medgemma)
Display:
tracksThought → StreamingPlainText (thought panel + answer)
else → StreamingPlainText while streaming → ChatMarkdown when done
```
---
## Key env / endpoints (dev)
```env
VITE_CLINICAL_CHAT_USE_LLM=true
VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat
VITE_OLLAMA_MODEL=gemma4:e4b
VITE_CLINICAL_CHAT_MOCK_TOOLS=true # flip false for live agent tools
```
Modal Gemma E4B (reference): `PILOT_PROJECT/tmp/test_endpoint.py``dtj-tran--ollama-gemma4-e4b-ollamaserver-web.modal.run`
---
## Session references
- Prior frontend memory: `session_memory/27_jun_26/27_jun_26_frontend.md`
- Agent tools smoke README: `CODEBASE/ml/tests/agent_tools/README.md`
- Today's active dev: `npm run dev` on frontend implementation

View File

@@ -0,0 +1,554 @@
# VKIST MSK Pilot — Intern Onboarding Index
**Date:** 7 Jul 2026
**Audience:** Phuc (ML / agentic), Quan (frontend / PWA), Dave Tran - Dat (lead)
**Repo:** `pilot_msk_ultrasound_stack` — clone already shared via GitHub
Read this file first, then open your role-specific guide:
| Track | Document |
| ---------------------------------------------- | ------------------------------------------------------------------ |
| **Phuc** — generative logic, agent, guardrails | [ONBOARDING_PHUC_ML_ENGINEER.md](./ONBOARDING_PHUC_ML_ENGINEER.md) |
| **Quan** — mobile UI, Figma, Stitch, PWA | [ONBOARDING_QUAN_FRONTEND.md](./ONBOARDING_QUAN_FRONTEND.md) |
> **Abbreviations:** See [§12 Glossary](./ONBOARDING_INDEX.md#12-glossary-of-abbreviations) — FR, NFR, RAG, CoT, PHI, PWA, etc.
---
## 1. What this project is
**VKIST MSK Pilot** moves academic musculoskeletal ultrasound ML (angle → inflammation → segmentation → synovitis grade 03) into a clinical web platform for Vietnamese public hospitals (Bach Mai, Viet Duc class facilities).
| Dimension | Summary |
| ---------- | ------------------------------------------------------------------------------------------------------- |
| **Cycle** | Jun 2 Sep 2, 2026 (strict 3-month window) |
| **Method** | Hybrid: waterfall phases + agile sprints |
| **Stack** | React PWA + FastAPI + Triton CV + 3-tier LLM (Edge Gemma → Gemini → MedGemma) + RAG over MOH guidelines |
| **Users** | Radiologists, surgeons, physiotherapists, patients/caregivers — each with different UI needs |
**Hard constraints (non-negotiable):**
- **Decree 13/2023** — no raw PHI leaves device without scrubbing
- **No open-ended patient chat** — structured, gated summaries only
- **Air-gapped hospital LAN** — cloud LLM only under NFR-16a (consent + audit + redaction)
- **Legacy smartphones** — zero-GPU fallback rendering for patient fleet
Full vision: `[PILOT_PROJECT/PROJ_LEVEL_READING/PLAN/CONTEXT_VISION_SCOPE.md](../../PILOT_PROJECT/PROJ_LEVEL_READING/PLAN/CONTEXT_VISION_SCOPE.md)`
**Tóm tắt (VI):** Dự án AI siêu âm khớp gối cho bệnh viện VN; 3 tháng; bảo mật Decree 13; các tinh năng hiện giờ đang optimize cho người dùng chính là bác sĩ cơ xương khớp - trong tương lai có thể cân nhắc `bệnh nhân` là đối tượng phát triển; làm tiếp phần Dave đã xây, không làm lại từ đầu.
---
## 2. How we work — continuation engineering
You are **not** here to read docs for three weeks. You inherit Dave's foundation and ship **incremental PRs** that close spec gaps toward sprint checkpoints.
```mermaid
flowchart LR
foundation[Dave_Foundation]
specs[Specs_and_UCs]
backlog[Open_Backlog]
checkpoint[Sprint_Checkpoint]
internPR[Intern_Incremental_PR]
foundation --> internPR
specs --> backlog
backlog --> internPR
internPR --> checkpoint
```
1. Dave assigns **one backlog ID** (P-n or Q-n).
2. You read **only** the spec + code files for that ticket.
3. Ship a small PR (≤300 lines preferred); demo something runnable weekly.
4. Dave assigns the next ticket.
**Tóm tắt (VI):** Dave giao ticket → đọc đúng spec → PR nhỏ → demo hàng tuần → ticket tiếp theo.
---
## 3. Foundation already built (do not rebuild)
Dave's latest handoff: `[session_memory/6_Jul_26.md](../6_Jul_26.md)`
| Area | Status | Key locations |
| --------------------- | ------------------------ | ------------------------------------------------------------------------- |
| FR-25 workspace shell | **Working PoC** | `ClinicalWorkspacePage.tsx`, `WorkspaceShell.tsx`, `DiagnosticCanvas.tsx` |
| CV inference | **Live** (FastAPI proxy) | `POST /api/test/analyze/batch`, `mlInferenceCacheStore.ts` |
| Clinical chat UI | **Working** | `ClinicalChatPanel.tsx`, `useClinicalChat.ts` |
| Edge Gemma 4 E2B | **Working** | `llm.worker.ts`, `clinicalLlmRuntime.ts`, OPFS |
| CoT streaming | **Working** (plain text) | `clinicalChatStreamRegistry.ts`, `StreamingPlainText.tsx` |
| Agent runtime | **Working** (mock tools) | `agent_runtime/`, `agentLoop.ts` |
| Cloud BFF | **Wired** | `cloud_orchestrate.py`, `cloud_consult.py`, `agent_tools.py` |
| 35 Figma routes | **Stub pages** | `figma-component-specs.md` |
| Safety / guardrails | **Spec + stubs** | `bert_adapter.py`; `guardrail.worker.ts` missing |
| Patient PWA | **Not started** | Sprint 4 |
**Tóm tắt (VI):** Workspace, chat, agent, inference đã chạy; guardrail và UI mobile/patient còn thiếu — đó là việc của bạn.
---
## 4. Sprint checkpoints (what your PRs advance)
| ID | Checkpoint | Sprint | Window | Owner |
| ---- | ----------------------------------------------- | ------ | -------------- | --------------------- |
| CP-1 | Beta clinical modes (Planning + high reasoning) | 2 | Jul 10 | Dave → Phuc polish |
| CP-2 | Live agent tools (mock → BFF) | 23 | Jul 1424 | **Phuc** |
| CP-3 | Guardrail slice v1 | 23 | Jul 21 Aug 7 | **Phuc** + Dave |
| CP-4 | Collaborative annotation UX | 3 | Jul 1324 | Dave; Quan UI support |
| CP-5 | Patient list/cards → Figma mobile | 34 | Jul 28 Aug 7 | **Quan** |
| CP-6 | Patient PWA (large text, 375px) | 4 | Jul 27 Aug 7 | **Quan** |
| CP-7 | Accessibility on 2 screens | 45 | Aug 421 | **Quan** |
| CP-8 | RAG retrieval gate / referee stub | 35 | Aug 121 | **Phuc** |
**Timeboxes:** Phuc until ~Aug 31; Quan ~2 months (ends ~early Aug).
**Tóm tắt (VI):** Mỗi PR phải đẩy một checkpoint CP-n, không chỉ viết tài liệu.
---
## 5. Open backlog
Dave maintains priority. Do not self-assign large items.
### Phuc — generative / agentic
| ID | Gap | Starting code | Done when |
| --- | -------------------------------- | ----------------------------------------------- | ----------------------------------------------------- |
| P-1 | Agent tools mock | `tools/registry.ts`, `agent_tools.py` | Live BFF; `VITE_CLINICAL_CHAT_MOCK_TOOLS=false` works |
| P-2 | `bert_adapter.py` stubs | `bert_adapter.py`, `safety/service.py` | One real PASS/FAIL check on outbound text |
| P-3 | `guardrail.worker.ts` missing | Create near `llm.worker.ts` | Worker hooks into chat before display |
| P-4 | RAG not enforced in chat mode | `runClinicalChatTurn.ts` | Clinical answers require retrieval context |
| P-5 | Planning + high reasoning `beta` | `clinicalChatModes.ts`, `chatReasoningLevel.ts` | Stable runs; beta flags removed |
| P-6 | `escalate_medgemma` E2E | `cloud_llm_gateway.py`, `tools/registry.ts` | Demo with audit log entry |
| P-7 | Conformal decoding eval | `ml/spec/conformal_decoding_*.md` | 3 VI clinical test cases (research) |
### Quan — frontend / PWA
| ID | Gap | Starting code | Done when |
| --- | ----------------------------- | ----------------------------- | ----------------------------------------------------------------- |
| Q-1 | 35 stub routes | `src/pages/*.tsx` | 2 priority screens match Figma |
| Q-2 | Safety overlay not routed | `SafetyEscalationOverlay.tsx` | Reachable from workspace |
| Q-3 | No mobile breakpoint | Patient pages | 375px pass, touch ≥44px |
| Q-4 | Accessibility open | Patient components | 1 view WCAG-oriented pass |
| Q-5 | No error boundaries | `App.tsx` | Friendly fallback on workspace route |
| Q-6 | Patient dashboard | `patient-card-*.tsx` | Jargon-free, large type, VI copy |
| Q-7 | Zero-GPU fallback UI | TBD banner | UI state when WebGL weak (Dave provides hook) |
| Q-8 | Stitch sync after ship (lite) | Stitch board | Optional: update frame when PR merges; no standalone proposal doc |
**Tóm tắt (VI):** Bảng backlog trên — chờ Dave giao ID cụ thể trước khi code.
---
## 6. Codebase map
**Active code:** `PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/`
**Do not develop in:** `PILOT_PROJECT/workspace/LEGACY/` (read-only reference)
> **Note:** Root `PILOT_PROJECT/README.md` references stale paths (`Reading_docs/`, `CONTEXT.md`). Use paths in this doc instead.
| Module | Path | Phuc | Quan |
| -------------- | --------------------- | ----------------------------------------- | ------------------------------------------------ |
| Frontend | `CODEBASE/frontend/` | `lib/llm/`, `workers/`, `useClinicalChat` | **Primary**`pages/`, `components/`, `styles/` |
| Backend | `CODEBASE/backend/` | **Primary** — routers, safety, adapters | Read-only |
| ML / NLP | `CODEBASE/ml/` | **Primary**`agent_runtime/`, tests | Do not touch |
| Knowledge | `CODEBASE/knowledge/` | When P-4/P-8 | Do not touch |
| Design | `Design_Material/` | Safety, API | UCs, screen specs |
| Session memory | `session_memory/` | `6_Jul_26.md` | `26_Jun_26/Proj_context` for layout |
### Frontend tree (Quan primary)
```
frontend/
├── spec/ ← uix_prototype_coverage, figma-component-specs, DESIGN
└── implementation/src/
├── pages/ ← Quan: screens
├── components/atoms|molecules|shells/
├── styles/design-tokens.ts
├── hooks/ ← Phuc: useClinicalChat only
├── lib/llm/ ← Phuc ONLY
└── workers/ ← Phuc ONLY
```
### ML + backend tree (Phuc primary)
```
ml/implementation/nlp/agent_runtime/src/
orchestrator/agentLoop.ts
tools/registry.ts
prompt/toolSystemPrompt.ts, toolOutputParser.ts
frontend/.../src/lib/llm/ ← edge inference, chat routing
backend/routers/ ← cloud_*, agent_tools
backend/services/ ← cloud_llm_gateway, agent_tools_service
backend/implementation/safety/, adapters/bert_adapter.py
knowledge/spec/ ← RAG when P-4/P-8
```
**Tóm tắt (VI):** Phuc = ml + backend + lib/llm; Quan = frontend pages/components; LEGACY không đụng.
---
## 7. Reading guide (summary)
Each role doc has three tiers. Quick reference:
| Tier | When |
| ---------------- | ---------------------------------- |
| **Must read** | Day 13, before first PR |
| **Should read** | When your ticket touches that area |
| **Learn deeper** | Keywords — grow over internship |
See [Phuc doc](./ONBOARDING_PHUC_ML_ENGINEER.md#3-what-to-read-what-to-learn) and [Quan doc](./ONBOARDING_QUAN_FRONTEND.md#3-what-to-read-what-to-learn) for full lists.
**Tóm tắt (VI):** Đọc bắt buộc trước PR đầu; đọc thêm theo ticket; học sâu theo từ khóa khi cần.
---
## 8. Dev toolkit
### Shared (both)
| Tool | Purpose |
| ----------------- | ----------------------------------------------------------- |
| **Cursor / Kilo** | Agentic coding — load `PILOT_PROJECT/AGENT_SKILL/` first |
| **Git + GitHub** | Branch per ticket; PR ≤300 lines |
| `secrets/` | Local only — see `PILOT_PROJECT/secrets_template/README.md` |
| `session_memory/` | Dave handoff — start with `6_Jul_26.md` |
### Phuc additions
```bash
# Frontend + clinical chat
cd PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation
npm install && npm run dev # :5173
# Gemma lab
cd ../../ml/tests/gemma4_e2b && npm install && npm run dev # :5174
# Agent smoke
cd ../agent_tools && npm install && npm run smoke
# Backend
cd ../../.. && conda activate vkist_ultra
PYTHONPATH=. python backend/tests/smoke_agent_tools_server.py
```
Key env: `VITE_CLINICAL_CHAT_USE_LLM`, `VITE_CLINICAL_CHAT_MOCK_TOOLS`, `VITE_API_BASE_URL`
### Quan additions
```bash
cd PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation
npm install && npm run dev # :5173
```
| Tool | Link |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| **Figma** (authority) | [MSK_Workspace_UIX-Design](https://www.figma.com/design/uakk3T4efS1AIckfTEFSZO/MSK_Workspace_UIX-Design?node-id=0-1) |
| **Stitch** (acceleration) | [Whiteboard 8188910238055386921](https://stitch.withgoogle.com/projects/8188910238055386921) |
| **Responsive test** | Chrome DevTools → iPhone SE 375px |
Stitch board has **web mocks** today — Quan creates **mobile variants** and exports references for the coding agent (see Quan doc).
**Tóm tắt (VI):** Cursor/Kilo + Git; Phuc thêm conda/smoke tests; Quan thêm Figma + Stitch + DevTools mobile.
---
## 9. Bonus — Feynman Technique with agents
When a concept or code path feels **hard**, don't just ask the agent to fix it — ask it to **teach it back to you** using the [Feynman Technique](https://en.wikipedia.org/wiki/Feynman_technique): explain simply, use an analogy, expose gaps, then verify your understanding.
### When to use
| Situation | Use Feynman prompt |
| ---------------------------------------------------- | -------------------------------- |
| You read code but can't explain it in your own words | ✅ |
| Agent generated a diff you don't fully understand | ✅ Before commit |
| Spec uses jargon (RAG, CoT, BFF, …) | ✅ |
| You're stuck >30 min on the same concept | ✅ Then ask Dave if still blocked |
### Universal template (copy into Cursor / Kilo)
```
Feynman explain — [TOPIC or FILE PATH]:
1. Explain like I'm a new intern with CS basics but new to this codebase.
2. Use one everyday analogy (non-medical if possible).
3. Walk the data/control flow in ≤8 numbered steps.
4. Define every abbreviation you use (or link to our glossary).
5. List 3 "common mistakes" a beginner would make here.
6. Ask me 2 short quiz questions — wait for my answers before continuing.
7. If I get them wrong, correct me simply.
Context: VKIST MSK Pilot, ticket [P-n or Q-n].
Do NOT write code yet — teach only.
```
### After the agent teaches you
1. **Rewrite** the idea in 35 bullets in your own words (PR description, standup note, or personal log).
2. If you still can't rewrite without copying — **you're not ready to commit** that code; ask one follow-up or ping Dave.
3. Optional: paste your summary to Dave at weekly demo — proves understanding, not just output.
Role-specific Feynman prompts: [Phuc §12](./ONBOARDING_PHUC_ML_ENGINEER.md#12-bonus--feynman-technique-prompts) · [Quan §14](./ONBOARDING_QUAN_FRONTEND.md#14-bonus--feynman-technique-prompts)
**Tóm tắt (VI):** Khi khó hiểu — bảo agent giải thích đơn giản (Feynman), trả lời quiz, tự viết lại bằng lời mình; chưa hiểu thì chưa commit.
---
## 10. Collaboration contract
| Rule | Detail |
| ------------- | ----------------------------------------------------------- |
| Assignment | Dave picks backlog ID — you don't self-select large scope |
| PR size | ≤300 lines; one backlog ID per PR |
| Weekly demo | Runnable change required |
| Agentic tools | OK after reading ticket spec; pass `AGENT_SKILL` checklists |
| Research | ≤40% time; only when tied to P-7 or approved milestone |
| Blocked >4h | Ping Dave with exact error + file path |
**Tóm tắt (VI):** PR nhỏ, demo mỗi tuần, hỏi Dave khi kẹt quá 4 giờ.
---
## 11. Current assignments (Dave fills in)
| Intern | Active ticket | Checkpoint | PR link | Status |
| ------ | ------------------------- | ---------- | ------- | ------ |
| Phuc | P-___ | CP-___ | | |
| Quan | Q-1 (`/` worklist mobile) | CP-5 | | |
---
## 12. Glossary of abbreviations
Terms you will see daily in specs, code comments, and standups. Bookmark this section.
### Project & process
| Abbr. | Full form | Meaning in this project |
| --------------- | --------------------------------- | -------------------------------------------------------------- |
| **VKIST** | Viện Khoa học Công nghệ (context) | Research institute partner; MSK ultrasound AI origin |
| **MSK** | Musculoskeletal | Joint/tendon/ligament domain — knee ultrasound focus |
| **PoC** | Proof of Concept | Current sprint build — not production-certified yet |
| **CP-n** | Checkpoint n | Sprint deliverable milestone (see §4) |
| **P-n / Q-n** | Phuc-ticket / Quan-ticket | Backlog IDs in §5 |
| **UC-n** | Use Case n | Clinical workflow spec (e.g. `UC-48376` load scan session) |
| **FR-n** | Functional Requirement n | Feature requirement (e.g. **FR-25** synovitis grading) |
| **NFR-n** | Non-Functional Requirement n | Quality constraint (latency, safety, privacy) |
| **WS-25** | Workspace 25 | Clinician diagnostic workspace UI (65/35 layout) |
| **HITL** | Human-in-the-Loop | Clinician must review/confirm before AI output is final |
| **UP5, UP1, …** | User Persona n | Stakeholder archetype (UP5 = radiologist, UP1 = senior expert) |
### Clinical & domain (Vietnamese context)
| Abbr. | Full form | Meaning |
| --------------- | ----------------------------------------------------------- | --------------------------------------------------------- |
| **MOH** | Ministry of Health (Vietnam) | Source of clinical guidelines for RAG corpus |
| **DICOM** | Digital Imaging and Communications in Medicine | Standard medical image format (ultrasound slices) |
| **PACS** | Picture Archiving and Communication System | Hospital image archive — DICOM source |
| **EMR / HIS** | Electronic Medical Record / Hospital Information System | Hospital patient record system |
| **HL7 / FHIR** | Health Level 7 / Fast Healthcare Interoperability Resources | Standards for EMR data exchange |
| **Synovitis** | — | Joint lining inflammation — graded **03** by this system |
| **GradCAM** | Gradient-weighted Class Activation Mapping | Heatmap showing which image regions drove AI grade |
| **ULTS** | Ultrasound | Modality used in FR-25 |
| **Circular 46** | Thông tư 46/2018/TT-BYT | Vietnam clinical report format regulation |
| **Decree 13** | Nghị định 13/2023/NĐ-CP | Vietnam personal data protection law (PHI rules) |
### Security & privacy
| Abbr. | Full form | Meaning |
| --------------- | ----------------------------------- | ----------------------------------------------------------------- |
| **PHI** | Protected Health Information | Patient-identifiable health data — never commit to git |
| **PII** | Personally Identifiable Information | Names, IDs, phone — scrubbed before cloud egress |
| **RBAC** | Role-Based Access Control | Permissions by clinician role (radiologist, therapist, …) |
| **JWT** | JSON Web Token | Session auth token (stored in Redis server-side) |
| **NFR-16a** | NFR exception 16a | PoC-only rule allowing cloud LLM with redaction + consent + audit |
| **AES-256-GCM** | Advanced Encryption Standard | Client-side encryption for local storage (target arch) |
### Architecture & infrastructure
| Abbr. | Full form | Meaning |
| ------------ | ----------------------------------- | -------------------------------------------------------------- |
| **PWA** | Progressive Web App | Installable web app — Sprint 4 patient target |
| **BFF** | Backend-for-Frontend | FastAPI layer that proxies agent tools + cloud LLM for browser |
| **API** | Application Programming Interface | HTTP endpoints (`/api/v1/...`) |
| **K3s** | Lightweight Kubernetes | On-prem hospital cluster (target deployment) |
| **Triton** | NVIDIA Triton Inference Server | GPU server for CV models + embeddings (`:8001` gRPC) |
| **FastAPI** | — | Python web framework for backend (`:8000`) |
| **NGINX** | — | Reverse proxy / SSL termination in front of FastAPI |
| **MinIO** | — | S3-compatible object store for DICOM blobs, reports |
| **pgvector** | PostgreSQL vector extension | Vector DB for MOH guideline semantic search (HNSW index) |
| **HNSW** | Hierarchical Navigable Small World | Approximate nearest-neighbor index for RAG retrieval |
| **GCP** | Google Cloud Platform | Hosts Gemini (Vertex AI) |
| **Modal** | Modal.com | Serverless GPU platform hosting MedGemma |
| **LAN** | Local Area Network | Hospital internal network — air-gapped primary mode |
| **CI/CD** | Continuous Integration / Deployment | Jenkins + GitLab pipeline |
| **SSE** | Server-Sent Events | One-way streaming from server to browser |
### ML, NLP & generative AI
| Abbr. | Full form | Meaning |
| --------------------- | ------------------------------------------------------- | ------------------------------------------------------------------- |
| **CV** | Computer Vision | Image ML pipeline (angle → inflammation → segmentation) |
| **NLP** | Natural Language Processing | Clinical text cleanup, explanations, chat |
| **LLM** | Large Language Model | Generative text models (Gemma, Gemini, MedGemma) |
| **RAG** | Retrieval-Augmented Generation | Fetch MOH guideline chunks before generating answer |
| **CoT** | Chain-of-Thought | Model "thinking" trace before final answer (thought channel) |
| **TTFT** | Time to First Token | Latency until first streamed word appears (NFR-7: ≤200 ms target) |
| **E2B / E4B** | Gemma 4 2B / 4B params | Edge (browser) vs server Ollama model sizes |
| **WebLLM** | Web Large Language Model | In-browser inference via WebGPU + WASM |
| **OPFS** | Origin Private File System | Browser storage for ~1.9 GB Gemma checkpoint |
| **WASM** | WebAssembly | Binary format for in-browser ML (MediaPipe, LiteRT) |
| **WebGPU** | — | Browser GPU API — required for edge Gemma |
| **WebGL** | — | Older browser 3D API — fallback path for weak devices |
| **BERT** | Bidirectional Encoder Representations from Transformers | Classifier for drift/guardrail/referee (partially stubbed) |
| **EmbeddingGemma** | — | 768-dim embedding model for RAG vector search |
| **MedGemma** | — | Tier-3 clinical deep-reasoning LLM (Modal, NFR-16a) |
| **Gemini** | — | Tier-2 orchestration/translation LLM (GCP Vertex AI) |
| **Gemma** | — | Tier-1 edge conversation LLM (browser MediaPipe) |
| **Agent** | — | Multi-step tool-calling loop (`<tool_call>` / `<final>` protocol) |
| **Guardrail** | — | Safety filter on generated text (hallucination, scope breach) |
| **RAG-Referee** | — | 3-axis citation validator for MedGemma output |
| **Circuit breaker** | — | Clinician challenges AI grade via Socratic dialogue before sign-off |
| **Socratic dialogue** | — | Structured Q&A between clinician and AI about a grade |
| **DeepLab / UNet** | — | Segmentation model architectures in CV pipeline |
| **ConvNeXt / MedViT** | — | Angle classification model families (legacy stack) |
| **LiteRT** | — | On-device inference runtime (TensorFlow Lite successor) |
| **TFLite / TFJS** | TensorFlow Lite / TensorFlow.js | Client-side ML runtimes |
| **ONNX** | Open Neural Network Exchange | Model interchange format for Triton deployment |
| **VRAM** | Video RAM | GPU memory — relevant for Triton/Modal sizing |
### Frontend & UI
| Abbr. | Full form | Meaning |
| ------------- | ------------------------------------------ | ------------------------------------------------------------- |
| **UI / UIX** | User Interface / User Interface eXperience | Screens, layout, interaction design |
| **SPA** | Single Page Application | React app — one HTML shell, client-side routing |
| **TS / TSX** | TypeScript / TypeScript + JSX | Frontend language + React syntax |
| **Zustand** | — | Lightweight React state store (target arch; PoC uses context) |
| **Dexie.js** | — | IndexedDB wrapper for offline cache |
| **IndexedDB** | — | Browser local database for inference cache, session replay |
| **MRN** | Medical Record Number | Patient hospital ID shown in TopBar |
| **FR-26** | Functional Requirement 26 | Future patient-facing dashboard scope |
| **a11y** | Accessibility | WCAG-oriented design (contrast, font scale, touch targets) |
### Tools & workflow
| Abbr. | Full form | Meaning |
| ------------ | ---------------------------------- | ----------------------------------------- |
| **PR** | Pull Request | Code review unit — keep ≤300 lines |
| **IDE** | Integrated Development Environment | VS Code / Cursor |
| **Dev Mode** | Figma Dev Mode | Inspect panel for spacing, tokens, CSS |
| **SDK** | Software Development Kit | e.g. `@google/stitch-sdk` for asset fetch |
**Tóm tắt (VI):** Bảng viết tắt trên dùng hàng ngày — FR/NFR = yêu cầu; RAG/LLM/CoT = AI sinh ngôn ngữ; PHI/Decree 13 = bảo mật; PWA/DICOM/PACS = lâm sàng & UI; hỏi Dave nếu gặp từ mới.
---
## 13. Quick links
| Resource | Path |
| -------------------- | ------------------------------------------------------------------------------------- |
| Project vision | `PILOT_PROJECT/PROJ_LEVEL_READING/PLAN/CONTEXT_VISION_SCOPE.md` |
| Sprint architecture | `PILOT_PROJECT/workspace/sprint_1_2/Design_Material/SPRINT_1_2_ARCHITECTURE_SPEC.md` |
| UI truth doc | `PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/spec/uix_prototype_coverage.md` |
| Agent skills | `PILOT_PROJECT/AGENT_SKILL/INTRODUCTION.md` |
| Dave's Jul 6 handoff | `session_memory/6_Jul_26.md` |
| Layout reference | `session_memory/26_Jun_26/Proj_context_26_Jun_26.md` |

View File

@@ -0,0 +1,396 @@
# Onboarding — Phuc (ML Engineer / Agentic & Generative Logic)
**Start here:** [ONBOARDING_INDEX.md](./ONBOARDING_INDEX.md) — [glossary](./ONBOARDING_INDEX.md#12-glossary-of-abbreviations) · [Feynman prompts (INDEX §9)](./ONBOARDING_INDEX.md#9-bonus--feynman-technique-with-agents)
**Date:** 7 Jul 2026
**Timebox:** ~Jul 7 Aug 31, 2026
**Lane:** Generative LLM, agent runtime, guardrails, RAG gates — co-designed with Dave
---
## 1. Your mission
You continue the **generative & agentic layer** Dave built. Your deliverable is **merged code** that advances checkpoints **CP-2, CP-3, CP-8** — not architecture slides.
You co-own agentic design and guardrail logic with Dave: you propose, Dave approves scope, you ship one vertical slice per PR.
**You are not expected to:** rewrite the CV/Triton pipeline, own full compliance sign-off, or refactor UI components.
**Tóm tắt (VI):** Tiếp tục xây LLM/agent/guardrail; mỗi tuần có PR chạy được; thiết kế guardrail cùng Dave; không làm lại pipeline những kết quả đã có - nếu có thì phải optimize & hạn chế redundancies overhead trong thiết kế.
---
## 2. Your codebase scope
### In scope — touch freely
| Path | What |
| --------------------------------------------------------------------------- | --------------------------------------- |
| `CODEBASE/ml/implementation/nlp/agent_runtime/` | Agent loop, tools, prompts |
| `CODEBASE/ml/tests/gemma4_e2b/` | Isolated Gemma + agent lab (`:5174`) |
| `CODEBASE/ml/tests/agent_tools/` | BFF / Modal / Exa smoke tests |
| `CODEBASE/ml/spec/conformal_decoding_llm_with_domain_stritcly_constrain.md` | Research spec |
| `CODEBASE/frontend/implementation/src/lib/llm/` | Edge inference, chat routing, prompts |
| `CODEBASE/frontend/implementation/src/hooks/useClinicalChat.ts` | Chat bootstrap + streaming |
| `CODEBASE/frontend/implementation/src/workers/llm.worker.ts` | MediaPipe Gemma worker |
| `CODEBASE/frontend/implementation/src/workers/` | Create `guardrail.worker.ts` here (P-3) |
| `CODEBASE/backend/routers/cloud_*.py`, `agent_tools.py` | Cloud + tool BFF |
| `CODEBASE/backend/services/cloud_llm_gateway.py`, `agent_tools_service.py` | Gateway + tools |
| `CODEBASE/backend/implementation/safety/` | Safety service |
| `CODEBASE/backend/implementation/adapters/bert_adapter.py` | Guardrail stubs |
| `CODEBASE/knowledge/spec/` | When working P-4 / P-8 |
### Read-only — understand, don't refactor
- `CODEBASE/backend/implementation/pipeline/` — CV inference (Dave)
- `CODEBASE/infra/` — unless P-6 Modal debugging
- `CODEBASE/frontend/implementation/src/components/` — Quan's lane
- `workspace/LEGACY/` — reference only
### Architecture mental model
```
User message
→ useClinicalChat / runClinicalChatTurn
→ [chat|planning] → Edge Gemma (llm.worker) OR Ollama E4B
→ [agent] → agentLoop → tools (exa, supabase, medgemma)
→ [YOUR GAPS] guardrail check, RAG gate, cloud escalation audit
→ display (StreamingPlainText)
```
Tier 1 = browser Gemma · Tier 2 = Gemini · Tier 3 = MedGemma (NFR-16a governed)
**Tóm tắt (VI):** Phạm vi = ml/agent_runtime, lib/llm, backend safety/routers; không sửa UI hay LEGACY.
---
## 3. What to read, what to learn
### Must read (Day 1 — before first PR)
| # | Document | Path |
| --- | ---------------------- | ---------------------------------------------------------------------------- |
| 1 | Dave's latest handoff | `session_memory/6_Jul_26.md` |
| 2 | Regulatory constraints | `PILOT_PROJECT/PROJ_LEVEL_READING/PLAN/CONTEXT_VISION_SCOPE.md` §4 |
| 3 | Agent protocol | `CODEBASE/ml/implementation/nlp/README.md` |
| 4 | PHI safety | `PILOT_PROJECT/AGENT_SKILL/secrets_and_phi_safety/SKILL.md` |
| 5 | Agent tools rules | `PILOT_PROJECT/AGENT_SKILL/agent_tool_execution/SKILL.md` |
| 6 | LLM + safety NFRs | `Design_Material/SPRINT_1_2_ARCHITECTURE_SPEC.md` — LLM tiers, NFR-10/16a/18 |
### Should read (when ticket assigned)
| Ticket | Read |
| -------- | --------------------------------------------------------------------------------------------------------------- |
| P-1, P-6 | `backend/spec/agent_tools_contract.md`, `agent_runtime/.../tools/registry.ts`, `backend/routers/agent_tools.py` |
| P-2, P-3 | `bert_adapter.py`, `safety/service.py`, `FR_25_UC_DESIGN/Q1_*.md``Q4_*.md` |
| P-4, P-8 | `knowledge/spec/knowledge_spec.md`, `agentLoop.ts`, `runClinicalChatTurn.ts` |
| P-5 | `clinicalChatModes.ts`, `chatReasoningLevel.ts`, `6_Jul_26.md` §beta |
| P-7 | `ml/spec/conformal_decoding_llm_with_domain_stritcly_constrain.md` |
| Any | Relevant `UC-*.md` in `Design_Material/FR_25_UC_DESIGN/Use_Case/` |
**Week 1 code trace (do once):**
`useClinicalChat.ts``runClinicalChatTurn.ts``agentLoop.ts` OR `llm.worker.ts``cloud_llm_gateway.py`
### Learn deeper (keywords)
> Abbreviations: [Glossary INDEX §12](./ONBOARDING_INDEX.md#12-glossary-of-abbreviations)
Study as tickets require — use papers, specs, and agent exploration:
| Keywords | Tickets |
| ------------------------------------------------------------------- | -------- |
| `prompt-constrained tool calling`, `agent orchestration loop` | P-1 |
| `RAG pre-retrieval`, `pgvector`, `semantic chunk`, `MOH corpus` | P-4, P-8 |
| `BERT guardrail`, `hallucination detection`, `scope-breach scoring` | P-2, P-3 |
| `conformal decoding`, `domain-constrained generation` | P-7 |
| `CoT streaming`, `thought/answer channel`, `WebLLM MediaPipe` | P-5 |
| `NFR-16a`, `consent audit`, `cloud egress redaction` | P-6 |
| `RAG-Referee`, `citation validation`, `3-axis contestant` | P-8 |
| `Transformers.js`, `Web Worker`, `OPFS model cache` | P-3 |
**Tóm tắt (VI):** Đọc 6 tài liệu bắt buộc ngày 1; đọc thêm theo ticket P-n; học sâu theo từ khóa khi làm.
---
## 4. Dev toolkit
| Tool | Command |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| Clinical chat UI | `cd CODEBASE/frontend/implementation && npm run dev``:5173` |
| Gemma lab | `cd CODEBASE/ml/tests/gemma4_e2b && npm run dev``:5174` |
| Agent smoke (test-the-agentic-functionality) | `cd CODEBASE/ml/tests/agent_tools && npm run smoke` |
| Backend smoke (test-the-agentic-functionality) | `conda activate vkist_ultra`; `cd CODEBASE`; `PYTHONPATH=. python backend/tests/smoke_agent_tools_server.py` |
| Agentic IDE | Cursor / Kilo — load `AGENT_SKILL/agent_tool_execution` |
**Environment variables (frontend** `.env.development`**):**
```env
VITE_CLINICAL_CHAT_USE_LLM=true
VITE_CLINICAL_CHAT_MOCK_TOOLS=true # flip false for P-1 done
VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat
VITE_API_BASE_URL=http://localhost:8001
```
Secrets: ask Dave — `PILOT_PROJECT/secrets_template/README.md`
**Tóm tắt (VI):** Chạy :5173 chat, :5174 lab, smoke tests; đổi MOCK_TOOLS=false khi P-1 xong.
---
## 5. Day 1 playbook (then ship)
**Morning**
1. Run `:5173` → open `/workspace/:patientId` → test chat + agent modes.
2. Run `:5174` gemma4_e2b → complete one tool smoke call.
3. Trace one chat turn and one agent turn (Section 3 code trace).
**Afternoon**
4. Hands-on verify backlog P-1P-7 (✅/❌ in your notes).
5. Sync with Dave → receive first ticket (likely **P-1** or **P-5**).
Do **not** spend Week 1 writing standalone architecture docs.
**Tóm tắt (VI):** Ngày 1: chạy app, trace code, sync Dave lấy ticket — không viết doc dài.
---
## 6. Spec vs implementation (your work surface)
| Component | Built today | Your tickets |
| ----------------------------------- | -------------- | ------------------ |
| Edge Gemma inference | ✅ | P-5 polish |
| Agent loop + `<tool_call>` protocol | ✅ (mock tools) | P-1 |
| Cloud gateway + audit hooks | ⚠️ partial | P-6 |
| Auto-retrieval in agent mode | ✅ | P-4 extend to chat |
| `bert_adapter.py` / safety service | ❌ stub | P-2 |
| `guardrail.worker.ts` | ❌ missing | P-3 |
| RAG-Referee 3-axis | ❌ spec only | P-8 |
| Planning + high reasoning beta | ⚠️ disabled | P-5 |
**Key facts:**
- Production chat = **browser Gemma 4 E2B** (MediaPipe), not server-side generation.
- Agent mode uses **prompt tags** (`<tool_call>`, `<final>`), not native function calling.
- Do **not** re-enable `ChatMarkdown` in reasoning mode — see `6_Jul_26.md` crash policy.
**Tóm tắt (VI):** Guardrail và RAG chat còn stub — đó là trọng tâm của bạn.
---
## 7. Your backlog (detail)
| ID | Gap | Spec | Starting files |
| ------- | ------------------------------ | --------------------------------- | ----------------------------------------------- |
| **P-1** | Mock agent tools | `agent_tools_contract.md` | `tools/registry.ts`, `agent_tools.py` |
| **P-2** | BERT adapter stubs | NFR-10, Q1Q4 | `bert_adapter.py`, `safety/service.py` |
| **P-3** | No edge guardrail worker | Web Workers spec | Create `guardrail.worker.ts` |
| **P-4** | No RAG gate in chat mode | NFR-18, `knowledge_spec.md` | `runClinicalChatTurn.ts` |
| **P-5** | Beta modes disabled | `6_Jul_26.md` | `clinicalChatModes.ts`, `chatReasoningLevel.ts` |
| **P-6** | MedGemma escalation incomplete | NFR-16a | `cloud_llm_gateway.py`, `tools/registry.ts` |
| **P-7** | Conformal decoding research | `ml/spec/conformal_decoding_*.md` | `gemma4_e2b/` lab |
**Tóm tắt (VI):** P-1 đến P-7 — chờ Dave giao ID / chủ động đề xuất giải pháp kỹ thuật & thiết kế; mỗi PR một ticket.
---
## 8. SMART milestones
### M1 — First merge (Jul 7 Jul 18)
| | |
| -------------- | --------------------------------------------- |
| **Ticket** | P-1 or P-5 (Dave assigns) |
| **Specific** | Live agent tools OR remove beta from one mode |
| **Measurable** | PR merged; live demo on `:5173` |
| **Achievable** | One backlog ID only |
| **Relevant** | CP-1 / CP-2 |
| **Time-bound** | Sprint 3 week 1 |
### M2 — Guardrail v1 (Jul 14 Aug 1)
| | |
| -------------- | ------------------------------------------------------- |
| **Ticket** | P-2 or P-3 (co-scoped with Dave) |
| **Specific** | RFC ≤1 page + one real PASS/FAIL check on outbound text |
| **Measurable** | Test/smoke on 3 fixtures from Q1Q4 scenarios |
| **Achievable** | Keyword gate OK for v1; full BERT optional |
| **Relevant** | CP-3, NFR-10 |
| **Time-bound** | Sprint 3 end |
### M3 — Retrieval or escalation (Jul 28 Aug 15)
| | |
| -------------- | -------------------------------------------------- |
| **Ticket** | P-4 or P-6 |
| **Specific** | RAG gate in chat OR MedGemma escalation with audit |
| **Measurable** | Screen recording + log snippet in PR |
| **Relevant** | CP-2 / CP-8, NFR-18 |
| **Time-bound** | Sprint 4 |
### M4 — Research or hardening (Aug 1 Aug 31)
| | |
| -------------- | ---------------------------------------------------- |
| **Ticket** | P-7 or extend P-2/P-3 |
| **Specific** | Conformal eval (3 VI cases) OR second guardrail axis |
| **Optional** | If M2M3 complete |
| **Time-bound** | Internship end |
**Tóm tắt (VI):** 4 mốc — PR đầu tiên tuần 1; guardrail tuần 34; RAG/escalation tháng 8.
---
## 9. Anti-patterns
- ❌ Refactor `agent_runtime` package structure without inform to members on update logic (no change-Log)
- ❌ Swap model families (Qwen experiment is closed)
- ❌ Touch `LEGACY/` or Quan's UI components
- ❌ Enable `ChatMarkdown` during reasoning streams (markdown rendering on LLM's outcome is problem - we shall put it under the rug)
- ❌ Commit secrets or PHI — ever
**Tóm tắt (VI):** Không refactor lớn, không đổi model, không markdown khi reasoning.
---
## 10. Definition of done
- [ ] ≥3 merged PRs across P-1P-6
- [ ] Can demo live agent tool call or guardrail check without Dave driving
- [ ] CP-2 or CP-3 marked advanced in team backlog
- [ ] Read `secrets_and_phi_safety` + `agent_tool_execution` skills
**Tóm tắt (VI):** 3 PR, demo được guardrail/agent, checkpoint CP-2 hoặc CP-3 tiến triển.
---
## 11. Agent prompt examples (optimize / change based on usecase)
**Explore generative flow:**
```
Trace runClinicalChatTurn.ts for agent mode: list each file called,
what tool protocol is used, and where mock tools are injected.
```
**Start P-1:**
```
Help me flip VITE_CLINICAL_CHAT_MOCK_TOOLS to false and fix any
BFF errors. Start from tools/registry.ts and agent_tools.py.
Follow AGENT_SKILL/agent_tool_execution/SKILL.md.
```
**Start P-2:**
```
Implement a minimal scope-breach keyword gate in bert_adapter.py
returning PASS/FAIL. Use Q1_True_Agreement.md as fixture context.
```
---
## 12. Bonus — Feynman Technique prompts
Use when a generative/agent concept feels opaque **before** you commit agent-generated code.
Master template: [INDEX §9](./ONBOARDING_INDEX.md#9-bonus--feynman-technique-with-agents).
### Hard concepts → ready-made prompts
**Agent loop & `<tool_call>` protocol (P-1):**
```
Feynman explain agentLoop.ts and the <tool_call> / <final> protocol in
@vkist/agent-runtime. Why prompt tags instead of OpenAI function calling?
Trace one turn with exa_search mocked. Quiz me after.
```
**Edge vs cloud 3-tier LLM (general):**
```
Feynman explain our 3-tier LLM: Edge Gemma → Gemini → MedGemma.
When does each tier run? What is NFR-16a in one sentence?
Use a "triage desk at a hospital" analogy. Quiz me.
```
**clinicalChatStreamRegistry + streaming (P-5 area):**
```
Feynman explain why clinicalChatStreamRegistry.ts paints tokens to the DOM
directly instead of React setState on every token. What breaks if we use
ChatMarkdown during CoT? ≤8 steps. Quiz me.
```
**RAG pre-retrieval (P-4 / P-8):**
```
Feynman explain mandatory RAG pre-generation in our system: what pgvector
retrieves, when agentLoop auto-calls supabase_query, and why chat mode
doesn't do this yet. MOH corpus in plain language. Quiz me.
```
**Guardrail vs RAG-Referee (P-2 / P-3):**
```
Feynman explain the difference between edge guardrail, bert_adapter
drift check, and RAG-Referee. Which are implemented vs stub today?
Use a "editor vs fact-checker" analogy. Quiz me.
```
**After each Feynman session:** write 3 bullets in your own words in the PR or ticket comment.
**Tóm tắt (VI):** Dùng prompt Feynman khi không hiểu agent/LLM; có sẵn prompt cho từng chủ đề khó; tự tóm tắt lại trước khi commit.

View File

@@ -0,0 +1,499 @@
# Onboarding — Quan (Frontend / Mobile PWA)
**Start here:** [ONBOARDING_INDEX.md](./ONBOARDING_INDEX.md) — [glossary](./ONBOARDING_INDEX.md#12-glossary-of-abbreviations) · [Feynman prompts (INDEX §9)](./ONBOARDING_INDEX.md#9-bonus--feynman-technique-with-agents)
**Date:** 7 Jul 2026
**Timebox:** ~2 months (ends ~early Aug 2026)
**Lane:** Mobile UI, Figma + Stitch design acceleration, patient PWA
---
## 1. Your mission
You continue the **UI layer** from stub routes toward Sprint 4 patient PWA checkpoints. You ship **visible screens in the browser** — not backend, not LLM internals.
Dave provides two design accelerators:
| Tool | Link | Role |
| ----------------- | -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| **Figma** | [MSK_Workspace_UIX-Design](https://www.figma.com/design/uakk3T4efS1AIckfTEFSZO/MSK_Workspace_UIX-Design?node-id=0-1) | Design authority — 35 screens, tokens, Vietnamese copy |
| **Google Stitch** | [Whiteboard](https://stitch.withgoogle.com/projects/8188910238055386921) | Acceleration — board has **web mocks**; you design **mobile variants** |
**Your agency:** Pick Figma-first, Stitch-first, or hybrid **per screen** — one line in the PR is enough. Dave assigns tickets; **Week 1 default ticket is Q-1 on the live `/` worklist** (highest yield).
**Week 1 principle — ship fast, correct, get feedback:**
| Priority | Rule |
| --------------- | ----------------------------------------------------------------------------------------------------- |
| **Speed** | Tools (Figma + Stitch) = **1 day max**, then code same day |
| **Yield** | Week 1 targets the **live route** `/` (`PatientWorklistPage`) mobile at 375px — not orphan stub pages |
| **Correctness** | `design-tokens.ts`, live navigation, no hardcoded colors, no pasted Stitch HTML |
| **Feedback** | Draft PR or screenshots to Dave **end of Day 1**; iterate Day 23 from review |
**Stitch (quick rules):**
1. Board has web mocks — use Stitch to sketch **mobile layout**, export screenshot for Cursor/Kilo
2. Agent writes React; you review every line
3. Figma wins on tokens and Vietnamese copy
**Tóm tắt (VI):** Tuần 1 = 1 ngày học tool, cùng ngày code mobile cho `/` worklist; PR/screenshot cuối ngày 1; sửa theo feedback Dave — không khảo sát tự do nhiều ngày.
---
## 2. Your codebase scope
### In scope — touch freely
```
CODEBASE/frontend/
├── spec/ ← read: uix_prototype_coverage, figma-component-specs, DESIGN
└── implementation/src/
├── pages/ ← your primary workspace
├── components/atoms/
├── components/molecules/
├── components/layout/
├── styles/design-tokens.ts ← token source of truth
├── App.tsx ← routes
└── public/assets/stitch/ ← exported Stitch references
```
### Read-only — understand, don't refactor
| File / folder | Why |
| ------------------------------------------ | -------------------------------- |
| `organisms/DiagnosticCanvas.tsx` | Dave owns DICOM canvas |
| `molecules/ClinicalChatPanel.tsx` | Read layout only — Phuc owns LLM |
| `lib/llm/`, `workers/` | Phuc's lane |
| `hooks/useClinicalChat.ts` | Know it exists |
| `data/mockData.ts`, `lib/patientStore.tsx` | How worklist gets data |
### Out of scope
`CODEBASE/backend/`, `CODEBASE/ml/`, `CODEBASE/knowledge/`, `CODEBASE/infra/`, `LEGACY/`
### Live routes today
```
/ → PatientWorklistPage.tsx ← Q-1, Q-6 priority
/workspace/:patientId → ClinicalWorkspacePage.tsx ← polish only; Q-2 overlay
+ 35 stub routes in src/pages/ ← Figma parity over time
```
**Tóm tắt (VI):** Sửa pages, atoms, molecules, tokens; không đụng canvas, chat LLM, backend.
---
## 3. What to read, what to learn
### Must read (Day 1 morning only — then code)
| # | Document | Time |
| --- | ----------------------------------------------------------------- | ---------------------------------- |
| 1 | `uix_prototype_coverage.md` | 20 min — **which routes are live** |
| 2 | `figma-component-specs.md` — Priority 12 + node for `/` worklist | 15 min |
| 3 | `design-tokens.ts` + skim `DESIGN.md` | 15 min |
| 4 | `AGENT_SKILL/secrets_and_phi_safety/SKILL.md` | 10 min |
Read `component-inventory.md`, `coding_convention`, and persona §3.4 **while coding**, not before Day 1 PR.
**Tóm tắt (VI):** Sáng ngày 1 chỉ đọc 4 tài liệu ngắn; chiều cùng ngày bắt đầu code.
### Should read (when ticket assigned)
| Ticket | Read |
| ---------- | ---------------------------------------------------------------------------------------- |
| Q-1, Q-6 | Figma node for screen; matching `src/pages/*.tsx` stub |
| Q-2 | `uix_prototype_coverage.md` §Safety; `SafetyEscalationOverlay.tsx` |
| Q-3, Q-4 | `CONTEXT_VISION_SCOPE.md` §3.4; `frontend_spec.md` PoC § |
| Q-5 | `figma-component-specs.md` — error boundaries item |
| Q-8 | [Stitch board](https://stitch.withgoogle.com/projects/8188910238055386921) + live routes |
| Any screen | `Design_Material/FR_25_UC_DESIGN/Use_Case/UC-*.md` |
| Layout | `session_memory/26_Jun_26/Proj_context_26_Jun_26.md` — 65/35 ASCII |
### Learn deeper (keywords)
> Abbreviations: [Glossary INDEX §12](./ONBOARDING_INDEX.md#12-glossary-of-abbreviations)
| Keywords | When |
| ---------------------------------------------------------- | ------------------- |
| `atomic design`, `atoms molecules organisms` | component-inventory |
| `React functional components`, `props`, `composition` | all Q tickets |
| `design tokens`, `CSS variables`, `responsive breakpoints` | Q-1, Q-3 |
| `mobile-first`, `375px viewport`, `touch target 44px` | Q-3, Q-6 |
| `accessibility`, `WCAG contrast`, `font scaling` | Q-4, Q-6 |
| `error boundaries` | Q-5 |
| `PWA`, `service worker` | Sprint 4 (later) |
| `Figma Dev Mode`, `Inspect` | Q-1 |
| `Stitch design-to-code`, `HTML export as reference` | Q-8 |
| `Vite`, `npm run dev` | daily |
| `Vietnamese clinical UX`, `jargon-free patient copy` | Q-6 |
**Tóm tắt (VI):** Đọc thêm theo ticket khi code; không ôm 7 tài liệu trước PR đầu.
---
## 4. Dev toolkit
| Category | Tool |
| ------------------- | ------------------------------------------------------------------------------------------------- |
| Dev server | `npm run dev``http://localhost:5173` |
| Design authority | [Figma](https://www.figma.com/design/uakk3T4efS1AIckfTEFSZO/MSK_Workspace_UIX-Design?node-id=0-1) |
| Design acceleration | [Stitch whiteboard](https://stitch.withgoogle.com/projects/8188910238055386921) |
| Coding agent | Cursor / Kilo |
| Mobile test | Chrome DevTools → iPhone SE 375×667 |
| PR evidence | Before/after screenshots desktop + 375px |
```bash
cd PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation
npm install
npm run dev
```
**Optional:** Stitch SDK — `public/assets/stitch/README.md` (OAuth via Dave)
### Agent prompt template
```
Implement [screen name] for mobile (375px).
Reference: Figma node [id] from figma-component-specs.md.
Layout hint: [Stitch mobile export screenshot or description].
Use existing atoms from component-inventory.md.
Use design-tokens.ts — no hardcoded colors.
Route: [path from figma-component-specs].
Match navigation in uix_prototype_coverage.md.
```
**Tóm tắt (VI):** npm run dev + Figma + Stitch + Cursor; test 375px; PR kèm screenshot.
---
## 5. Figma + Stitch workflow
### When to use which
| Source | Best for | Caveat |
| ------------- | --------------------------------------------- | --------------------------------------- |
| **Figma** | Tokens, spacing, VI copy, auth/patient cards | Slower from scratch — use Inspect |
| **Stitch** | Rapid mobile layout; agent reference material | Web mock ≠ mobile — create mobile frame |
| **Live code** | Navigation truth | `uix_prototype_coverage.md` |
**Legacy Stitch desktop** (`public/assets/stitch/README.md`, project `17628337625105266704`) — reference only.
**Primary Stitch:** [project 8188910238055386921](https://stitch.withgoogle.com/projects/8188910238055386921)
### Workflow choices (pick one per screen — state in PR)
| ID | When | Steps |
| -------------------- | -------------------------------------------------- | --------------------------------------------- |
| **A — Figma-first** | Screen in node map; tokens matter | Inspect → atoms → `src/pages/` |
| **B — Stitch-first** | Mobile patient screen; Stitch mobile variant ready | Stitch mobile design → export → agent → React |
| **C — Hybrid** | Stitch layout close; Figma has correct copy | Stitch structure + Figma polish → code |
**Rule:** Final PR must match live navigation (`/``/workspace/:id`), not a disconnected mock.
**Tóm tắt (VI):** Chọn A/B/C mỗi màn hình; Figma thắng khi khác Stitch; khớp route thật.
---
## 6. Week 1 sprint — tools in 1 day, ship mobile same week
> **No multi-day exploration.** Figma/Stitch are means to an end — the end is a **merged mobile PR** on a live screen.
### Day 1 — Tools (morning) + code (afternoon)
| Block | Time | Do |
| ------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Setup** | 30 min | `npm install && npm run dev` — open `/` on desktop + DevTools 375px |
| **Figma** | 45 min | Open [Figma](https://www.figma.com/design/uakk3T4efS1AIckfTEFSZO/MSK_Workspace_UIX-Design?node-id=0-1) → Inspect **Patient List** node `82-3155` (maps to live worklist) |
| **Stitch** | 45 min | Open [Stitch board](https://stitch.withgoogle.com/projects/8188910238055386921) → one mobile-frame sketch OR pick closest web mock → **screenshot for agent** |
| **Must-read** | 60 min | Section 3 four docs only |
| **Code** | Rest of day | **Q-1 on live `/`** — mobile layout for `PatientWorklistPage.tsx` (or `patient-list-main.tsx` if Dave assigns stub route). Cursor/Kilo + agent template (§4). **375px from the start.** |
| **EOD** | 15 min | Send Dave: draft PR **or** before/after screenshots at 375px + 1-line workflow choice (A/B/C) |
**Not acceptable as Week 1 deliverable:** gap lists, alignment essays, clicking all 35 routes, Storybook setup.
### Day 2 — Feedback loop
- Apply Dave's review on spacing, tokens, navigation, touch targets
- Push PR updates; aim for **merge-ready** by EOD if scope was ≤200 lines
### Day 35 — Second yield
- Merge Week 1 PR OR complete merge after small fixes
- Start **second screen** (patient card stub or Q-3 polish on same `/` route)
- Optional: update one Stitch frame to match what you shipped (Q-8 lite — no separate doc)
### Week 1 success criteria
| Check | Pass |
| ------------------------------ | ---------------------------------------------- |
| Live route `/` usable at 375px | No horizontal scroll |
| Uses `design-tokens.ts` | No raw hex in new code |
| Dave reviewed once | By end of Day 1 or 2 AM |
| Runnable demo | `npm run dev` — you walk Dave through tap flow |
**Tóm tắt (VI):** Ngày 1: tool buổi sáng, code buổi chiều, gửi Dave tối ngày 1; ngày 2 sửa feedback; ngày 35 merge + màn hình thứ 2.
---
## 7. What exists vs what you build
| Exists | You increment |
| ---------------------------------- | -------------------------------------------- |
| WS-25 workspace PoC (functional) | Mobile polish, not rewrite |
| 35 Figma stub routes | Q-1: 2 priority screens |
| Stitch web mocks | Q-8 lite: update frame after ship (optional) |
| `SafetyEscalationOverlay` (orphan) | Q-2: wire it |
| No error boundaries | Q-5 |
| No patient PWA | Q-6 |
### WS-25 layout (read-only context)
Clinician workspace is 65/35 split — your zone is **patient list, cards, patient dashboard**, not DICOM canvas:
```
┌─ TopBar ─────────────────────────────────────┐
├─ Zone A (65%) DICOM │ Zone B (35%) sidebar ┤ ← Dave/Phuc zone
└────────────────────────────────────────────────┘
Your zone: / worklist, /patients, patient-card-*, /home
```
See `session_memory/26_Jun_26/Proj_context_26_Jun_26.md` for full ASCII.
**Tóm tắt (VI):** Workspace clinician đã có; bạn làm patient list, card, dashboard mobile.
---
## 8. Your backlog (detail)
| ID | Gap | Priority screens / files |
| ------- | ----------------------- | --------------------------------------------------------------------------------------- |
| **Q-1** | Stub → Figma fidelity | Home, Login, Patient List, Patient Cards (see `figma-component-specs.md` §Priorities) |
| **Q-2** | Safety overlay unrouted | `SafetyEscalationOverlay.tsx`, `App.tsx` |
| **Q-3** | No 375px pass | Assigned patient screen |
| **Q-4** | Accessibility | Patient-facing view |
| **Q-5** | Error boundaries | `App.tsx` workspace route |
| **Q-6** | Patient dashboard | `patient-card-*.tsx`, FR-26 preview |
| **Q-7** | Zero-GPU UI banner | Dave provides detection hook |
| **Q-8** | Stitch sync (lite) | Update Stitch frame **after** you ship — optional in PR, not a standalone proposal gate |
**Figma priorities (from spec):**
1. Home, Login, Create Account
2. Patient List + Cards
3. Workspace shell (read-only for you)
4. Auth flows
5. AI View, Chat, Ultrasound
6. Remaining screens
**Tóm tắt (VI):** Ưu tiên Home, Patient List, Patient Card trước.
---
## 9. SMART milestones
### M1 — Week 1 ship (Jul 7 Jul 11) ← **start here**
| | |
| -------------- | ------------------------------------------------------------------------------------------------------- |
| **Ticket** | **Q-1 on live `/` worklist** (Dave confirms file: `PatientWorklistPage.tsx` or `patient-list-main.tsx`) |
| **Specific** | Mobile implementation at 375px on a **live** route — not a disconnected stub |
| **Measurable** | Draft PR Day 1 EOD; merged or merge-ready by Day 5; before/after screenshots |
| **Achievable** | ≤200 lines; tools only Day 1 morning |
| **Relevant** | CP-5 — fastest visible yield |
| **Feedback** | Dave review ≤24h after first push |
### M2 — Second screen (Jul 14 Jul 25)
| | |
| --------------- | ------------------------------------------------------------------- |
| **Ticket** | Q-1 #2 (patient card / home) or Q-3 deepen mobile on shipped screen |
| **Deliverable** | 2nd merged PR |
| **Relevant** | CP-5 / CP-6 |
### M3 — Mobile hardening (Jul 28 Aug 11)
| | |
| --------------- | -------------------------------------------------------------- |
| **Ticket** | Q-3 or Q-2 |
| **Deliverable** | 375px recording; taps ≥44px; optional Q-8 Stitch frame updated |
| **Relevant** | CP-6 |
### M4 — Accessibility (Aug 4 Aug 21) *optional*
| | |
| --------------- | ----------------------------------------- |
| **Ticket** | Q-4 |
| **Deliverable** | High contrast, large type on patient view |
| **Relevant** | CP-7 |
**Tóm tắt (VI):** M1 = tuần 1 merge mobile `/` worklist; không có M0 khảo sát riêng; M2M4 mở rộng sau khi có feedback.
---
## 10. Per-ticket workflow
```
1. Dave assigns Q-n
2. Pick workflow A / B / C — write in PR description
3. Capture Figma + Stitch reference screenshots
4. Cursor/Kilo: use agent prompt template (Section 4)
5. Review every line of the diff — understand before commit
6. npm run dev → screenshot desktop + 375px
7. PR: Q-n, CP-n, workflow choice, design links
```
**Tóm tắt (VI):** Mỗi ticket: chọn workflow → agent hỗ trợ → review diff → screenshot → PR.
---
## 11. Boundaries
**Touch:** `src/pages/`, `src/components/` (atoms/molecules/layout), `src/styles/`, `App.tsx`, Stitch whiteboard, `public/assets/stitch/`
**Do not touch:** `src/lib/llm/`, `src/workers/`, backend, ml
**Figma/Stitch edits:** OK to update Stitch frames and Figma comments; ask Dave before changing shared Figma components.
**Tóm tắt (VI):** Chỉ frontend UI; hỏi Dave trước khi sửa component Figma dùng chung.
---
## 12. Definition of done
- [ ] Week 1: live `/` worklist mobile at 375px shipped or merge-ready
- [ ] Dave feedback incorporated within 48h of first push
- [ ] ≥2 screens merged by end of Month 1 (M1 + M2)
- [ ] Each PR: workflow A/B/C (one line) + 375px screenshot
- [ ] Never committed `secrets/` or PHI
**Tóm tắt (VI):** Tuần 1 phải có mobile `/` chạy được + feedback Dave; không cần báo cáo alignment dài.
---
## 13. Agent prompt examples (optimize / change based on usecase)
**Learn the codebase:**
```
Explain PatientWorklistPage.tsx in simple terms: what components it uses,
where data comes from, and which Figma node it maps to.
```
**Start Q-1 (Figma-first):**
```
Implement /home per Figma node 33-143 in figma-component-specs.md.
Reuse Button and Badge from component-inventory. Use design-tokens.ts only.
Current stub: src/pages/home.tsx
```
**Week 1 — ship live worklist mobile (Day 1 afternoon):**
```
Mobile pass for PatientWorklistPage at 375px.
Figma node 82-3155. Stitch screenshot attached.
Use design-tokens.ts and existing Button, Badge, PatientCard from component-inventory.
No horizontal scroll. Touch targets >= 44px.
Do not change routing — stay on /.
Show me the minimal diff.
```
**Start Q-3 (mobile pass):**
```
Make patient-list-main.tsx usable at 375px: no horizontal scroll,
touch targets at least 44px. Show me which CSS to change.
```
---
## 14. Bonus — Feynman Technique prompts
Use when React layout, routing, or design workflow feels confusing **before** you commit.
Master template: [INDEX §9](./ONBOARDING_INDEX.md#9-bonus--feynman-technique-with-agents).
### Hard concepts → ready-made prompts
**Live route vs stub route (Week 1 critical):**
```
Feynman explain the difference between our LIVE routes (/ and /workspace/:id)
and the 35 Figma STUB routes in src/pages/. Why does Week 1 target
PatientWorklistPage on / ? Use a "demo kitchen vs blueprint room" analogy.
Quiz me.
```
**PatientWorklistPage data flow:**
```
Feynman explain PatientWorklistPage.tsx: where patient list data comes from
(mockData vs patientStore), what happens when I tap "Mở phiên chẩn đoán",
and which components re-render. ≤8 steps. No jargon without definition. Quiz me.
```
**design-tokens.ts vs hardcoded CSS:**
```
Feynman explain why we use design-tokens.ts instead of #hex in components.
Show one wrong example and one right example from our codebase.
Quiz me on how to add a new spacing value correctly.
```
**375px mobile layout:**
```
Feynman explain mobile-first pass at 375px for our worklist: flex vs grid,
overflow-x, and 44px touch targets. What is the smallest change to
PatientListShell or PatientWorklistPage to stop horizontal scroll?
Teach only — no code until I answer your quiz.
```
**Figma vs Stitch vs code (your workflow A/B/C):**
```
Feynman explain when I should use Figma-first vs Stitch-first vs hybrid
for a patient screen. What is "reference only" Stitch export vs production code?
Quiz me with one scenario: patient card on mobile.
```
**React atoms vs molecules (component-inventory):**
```
Feynman explain atomic design using OUR component-inventory.md: when is
something a Button (atom) vs PatientHeader (molecule)? Give 2 examples
from patient list screen. Quiz me.
```
### Feynman before commit rule
If the agent wrote code and you **cannot** Feynman-explain the diff back to a friend in 30 seconds → don't commit yet; run one Feynman prompt on the confusing file.
**Tóm tắt (VI):** Hỏi agent giải thích đơn giản khi khó hiểu React/route/mobile; có prompt mẫu; không commit code mình chưa giải thích lại được.

View File

@@ -31,7 +31,7 @@ os.environ.setdefault("TRITON_ENDPOINT", DEFAULT_TRITON_ENDPOINT)
import uvicorn import uvicorn
from fastapi import FastAPI from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.cors import CORSMiddleware
import asyncio
from backend.routers import cv_inference from backend.routers import cv_inference
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -44,10 +44,7 @@ async def lifespan(app: FastAPI):
logger.info("Starting CV inference service on Triton: %s", os.getenv("TRITON_ENDPOINT")) logger.info("Starting CV inference service on Triton: %s", os.getenv("TRITON_ENDPOINT"))
from backend.services.triton_warmup import warmup_triton_models from backend.services.triton_warmup import warmup_triton_models
try: warmup_task = asyncio.create_task(warmup_triton_models())
await warmup_triton_models()
except Exception as exc:
logger.warning("Triton warmup failed — first clinical request may be slow: %s", exc)
yield yield
logger.info("Shutting down CV inference service") logger.info("Shutting down CV inference service")

View File

@@ -3,16 +3,34 @@ import json
from typing import Any from typing import Any
import numpy as np import numpy as np
import requests import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
class TritonAdapter: class TritonAdapter:
def __init__(self, endpoint_url: str, timeout: float = 60.0): def __init__(self, endpoint_url: str, timeout: float = 60.0):
self.endpoint_url = endpoint_url.rstrip("/") self.endpoint_url = endpoint_url.rstrip("/")
self.timeout = timeout self.timeout = timeout
self._session = self._build_session()
@staticmethod
def _build_session() -> requests.Session:
session = requests.Session()
retry = Retry(
total=0,
connect=0,
read=0,
redirect=0,
status=0,
raise_on_status=False,
)
adapter = HTTPAdapter(max_retries=retry, pool_connections=20, pool_maxsize=50)
session.mount("http://", adapter)
session.mount("https://", adapter)
return session
async def close(self): async def close(self):
pass await asyncio.to_thread(self._session.close)
async def infer( async def infer(
@@ -61,7 +79,7 @@ class TritonAdapter:
} }
url = f"{self.endpoint_url}/v2/models/{model_name}/infer" url = f"{self.endpoint_url}/v2/models/{model_name}/infer"
response = requests.post(url, data=body, headers=headers, timeout=self.timeout) response = self._session.post(url, data=body, headers=headers, timeout=self.timeout)
response.raise_for_status() response.raise_for_status()
return self._parse_binary_response(response.headers, response.content) return self._parse_binary_response(response.headers, response.content)
@@ -91,7 +109,7 @@ class TritonAdapter:
def _model_ready_sync(self, model_name: str) -> bool: def _model_ready_sync(self, model_name: str) -> bool:
url = f"{self.endpoint_url}/v2/models/{model_name}" url = f"{self.endpoint_url}/v2/models/{model_name}"
response = requests.get(url, timeout=self.timeout) response = self._session.get(url, timeout=self.timeout)
if response.status_code == 404: if response.status_code == 404:
return False return False
response.raise_for_status() response.raise_for_status()
@@ -113,7 +131,7 @@ class TritonAdapter:
url = f"{self.endpoint_url}/v2/repository/index" url = f"{self.endpoint_url}/v2/repository/index"
# 2. Change requests.get to requests.post with an empty json payload {} # 2. Change requests.get to requests.post with an empty json payload {}
response = requests.post(url, json={}, timeout=self.timeout) response = self._session.post(url, json={}, timeout=self.timeout)
response.raise_for_status() response.raise_for_status()
data = response.json() data = response.json()

View File

@@ -0,0 +1,50 @@
import asyncio
import base64
import io
import logging
from typing import Any
from PIL import Image
from backend.implementation.postprocessing.calibration import calibration_config_from_params
from backend.services.cv_inference_service import CvInferenceOptions, run_single
from backend.services.celery_app import celery_app
logger = logging.getLogger(__name__)
def _decode_base64_to_pil(b64_str: str) -> Image.Image:
raw = base64.b64decode(b64_str)
return Image.open(io.BytesIO(raw)).convert("RGB")
@celery_app.task(bind=True, name="cv_inference.run_chunk", max_retries=2, default_retry_delay=10)
def run_cv_chunk(self, chunk_payload: dict) -> list[dict]:
images_b64 = chunk_payload.get("images_b64", [])
frame_ids = chunk_payload.get("frame_ids", [])
calibration_params = chunk_payload.get("calibration", {})
model_versions = chunk_payload.get("model_versions")
if not images_b64:
return []
images = [_decode_base64_to_pil(b64) for b64 in images_b64]
calibration = calibration_config_from_params(calibration_params)
options = CvInferenceOptions(
calibration=calibration,
model_versions=model_versions,
use_cache=False,
)
async def _run():
return await asyncio.gather(*[
run_single(img, frame_id=fid, options=options)
for img, fid in zip(images, frame_ids)
])
try:
results = asyncio.run(_run())
return results
except Exception as exc:
logger.exception("Celery chunk task failed: %s", exc)
raise self.retry(exc=exc, countdown=10, max_retries=2)

View File

@@ -0,0 +1,29 @@
import logging
import sys
from logging.handlers import RotatingFileHandler
from pathlib import Path
LOG_DIR = Path("logs")
LOG_DIR.mkdir(exist_ok=True)
LOG_FORMAT = "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s"
LOG_DATE_FORMAT = "%Y-%m-%d %H:%M:%S"
def setup_logging():
root = logging.getLogger()
root.setLevel(logging.DEBUG)
root.handlers.clear()
formatter = logging.Formatter(LOG_FORMAT, datefmt=LOG_DATE_FORMAT)
console = logging.StreamHandler(sys.stdout)
console.setLevel(logging.INFO)
console.setFormatter(formatter)
root.addHandler(console)
file_handler = RotatingFileHandler(
LOG_DIR / "app.log",
maxBytes=10 * 1024 * 1024,
backupCount=5,
encoding="utf-8",
)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)
root.addHandler(file_handler)
logging.getLogger("httpx").setLevel(logging.WARNING)
logging.getLogger("uvicorn.access").setLevel(logging.INFO)

View File

@@ -18,6 +18,10 @@ from backend.implementation.triton_batch import TRITON_MAX_BATCH_SIZE
from backend.services import cv_result_cache from backend.services import cv_result_cache
from backend.services import triton_runtime_service as triton_runtime from backend.services import triton_runtime_service as triton_runtime
from backend.services.cv_inference_service import CvBatchResult, CvInferenceOptions, run_batch, run_single from backend.services.cv_inference_service import CvBatchResult, CvInferenceOptions, run_batch, run_single
from backend.services import cv_celery_service
from backend.logging.logging_config import setup_logging
setup_logging()
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -135,14 +139,15 @@ async def cv_inference_health():
) )
@router.post("/analyze") @router.post("/analyze") # deprecated
async def analyze_upload( async def analyze_upload(
image: UploadFile = File(...), image: UploadFile = File(...),
calibration: str | None = Form(default=None), calibration: str | None = Form(default=None),
): ):
"""Spec-compliant CV pipeline: CLAHE → angle → inflammation → conditional segmentation.""" """Spec-compliant CV pipeline: CLAHE → angle → inflammation → conditional segmentation."""
image_pil = await _load_upload_image(image) image_pil = await _load_upload_image(image)
options = _build_options(_parse_calibration_form(calibration), use_cache=False)
options = _build_options(_parse_calibration_form(calibration), use_cache=True)
try: try:
result = await run_single(image_pil, frame_id=None, options=options) result = await run_single(image_pil, frame_id=None, options=options)
@@ -168,11 +173,13 @@ async def analyze_batch_upload(
calibration: str | None = Form(default=None), calibration: str | None = Form(default=None),
): ):
"""Spec-compliant CV batch — one full pipeline per frame (angle-first, gated segmentation).""" """Spec-compliant CV batch — one full pipeline per frame (angle-first, gated segmentation)."""
logger.info("Starting analyze batch upload")
if not images: if not images:
raise HTTPException(status_code=400, detail="At least one image is required") raise HTTPException(status_code=400, detail="At least one image is required")
try: try:
id_list = json.loads(frame_ids) id_list = json.loads(frame_ids)
# logger.info("Start to check the id_list")
except json.JSONDecodeError as exc: except json.JSONDecodeError as exc:
raise HTTPException(status_code=400, detail="frame_ids must be a JSON array of strings") from exc raise HTTPException(status_code=400, detail="frame_ids must be a JSON array of strings") from exc
@@ -213,6 +220,68 @@ async def analyze_batch_upload(
raise HTTPException(status_code=500, detail=str(exc)) from exc raise HTTPException(status_code=500, detail=str(exc)) from exc
@router.post("/analyze/batch/celery")
async def analyze_batch_celery(
images: list[UploadFile] = File(...),
frame_ids: str = Form(...),
calibration: str | None = Form(default=None),
):
"""Experiment: async chunk fan-out via Celery + Redis."""
if not images:
raise HTTPException(status_code=400, detail="At least one image is required")
try:
id_list = json.loads(frame_ids)
except json.JSONDecodeError as exc:
raise HTTPException(status_code=400, detail="frame_ids must be a JSON array of strings") from exc
if not isinstance(id_list, list) or not all(isinstance(x, str) for x in id_list):
raise HTTPException(status_code=400, detail="frame_ids must be a JSON array of strings")
if len(id_list) != len(images):
raise HTTPException(status_code=400, detail="frame_ids length must match images count")
image_pils: list[Image.Image] = []
for upload in images:
image_pils.append(await _load_upload_image(upload))
options = _build_options(_parse_calibration_form(calibration))
try:
job_id = cv_celery_service.submit_celery_batch(image_pils, id_list, options)
return JSONResponse({
"success": True,
"job_id": job_id,
"image_count": len(image_pils),
"mode": "celery-chunk-fanout",
"chunk_size": cv_celery_service.CELERY_CHUNK_SIZE,
})
except Exception as exc:
logger.exception("Celery batch submit failed")
raise HTTPException(status_code=500, detail=str(exc)) from exc
@router.get("/analyze/batch/celery/{job_id}")
async def analyze_batch_celery_result(job_id: str):
"""Poll result for a Celery chunk-fan-out batch job."""
try:
result = await cv_celery_service.get_celery_batch_result(job_id)
status = result.get("status")
if status == "pending":
return JSONResponse(
status_code=202,
content=result,
)
if status == "unknown":
return JSONResponse(
status_code=404,
content=result,
)
return JSONResponse(result)
except Exception as exc:
logger.exception("Celery batch result fetch failed")
raise HTTPException(status_code=500, detail=str(exc)) from exc
@router.post("/segment") @router.post("/segment")
@router.post("/segment/batch") @router.post("/segment/batch")
@router.post("/angle") @router.post("/angle")

View File

@@ -0,0 +1,27 @@
#!/usr/bin/env bash
#
# Launch the standalone CV inference FastAPI server (Modal Triton).
#
# Usage:
# ./backend/run_cv_inference.sh
#
# Works from any directory: it resolves the CODEBASE root relative to this
# script, sets PYTHONPATH so `import backend...` resolves, then runs the
# server as a module.
#
# Override defaults via env vars, e.g.:
# CV_INFERENCE_PORT=8080 ./backend/run_cv_inference.sh
#
set -euo pipefail
# CODEBASE root = grandparent dir of this script's directory (script lives in backend/routers/).
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CODEBASE_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
cd "${CODEBASE_ROOT}"
export PYTHONPATH="${CODEBASE_ROOT}:${PYTHONPATH:-}"
# exec python -m backend.cv_inference_server
exec uvicorn backend.cv_inference_server:app --host 0.0.0.0 --port ${CV_INFERENCE_PORT:-8001}

View File

@@ -0,0 +1,22 @@
from celery import Celery
from backend.implementation import config
celery_app = Celery(
"cv-inference",
broker=f"redis://{config.REDIS_HOST}:{config.REDIS_PORT}/{config.REDIS_DB}",
backend=f"redis://{config.REDIS_HOST}:{config.REDIS_PORT}/{config.REDIS_DB}",
# Explicit include — autodiscover looks for tasks.py, not cv_tasks.py
include=["backend.implementation.tasks.cv_tasks"],
)
# Must match the @task(name=...) value, not the Python module path
celery_app.conf.task_routes = {
"cv_inference.run_chunk": {"queue": "cv-inference"},
}
celery_app.conf.task_serializer = "json"
celery_app.conf.result_serializer = "json"
celery_app.conf.accept_content = ["json"]
celery_app.conf.result_expires = 3600
celery_app.conf.task_default_queue = "cv-inference"

View File

@@ -0,0 +1,130 @@
import base64
import dataclasses
import logging
import os
import time
from typing import Any
from backend.services.cv_inference_service import CvInferenceOptions, _encode_image_to_bytes
from backend.implementation.tasks.cv_tasks import run_cv_chunk
from backend.services.celery_app import celery_app
logger = logging.getLogger(__name__)
CELERY_CHUNK_SIZE = int(os.getenv("CELERY_CHUNK_SIZE", "4"))
CELERY_BATCH_POLL_CACHE_TTL_MS = float(os.getenv("CELERY_BATCH_POLL_CACHE_TTL_MS", "2000"))
_batch_poll_cache: dict[str, tuple[float, dict[str, Any]]] = {}
_batch_timings: dict[str, float] = {}
def submit_celery_batch(images, frame_ids, options):
if not images:
raise ValueError("images must not be empty")
if len(frame_ids) != len(images):
raise ValueError("frame_ids length must match images length")
from celery import group
chunks = []
for i in range(0, len(images), CELERY_CHUNK_SIZE):
chunk_imgs = images[i : i + CELERY_CHUNK_SIZE]
chunk_fids = frame_ids[i : i + CELERY_CHUNK_SIZE]
b64_images = [
base64.b64encode(_encode_image_to_bytes(img)).decode() for img in chunk_imgs
]
chunk_payload = {
"images_b64": b64_images,
"frame_ids": chunk_fids,
"calibration": dataclasses.asdict(options.calibration) if options.calibration else {},
"model_versions": options.model_versions,
}
chunks.append(chunk_payload)
job = group(run_cv_chunk.s(chunk) for chunk in chunks)
result = job.apply_async()
try:
result.save()
except Exception:
logger.exception("Failed to persist Celery GroupResult for job %s", result.id)
_batch_timings[result.id] = time.monotonic()
logger.info(
"Celery batch submitted job_id=%s chunks=%d images=%d",
result.id,
len(chunks),
len(images),
)
return result.id
async def get_celery_batch_result(job_id):
from celery.result import GroupResult
now = time.monotonic()
cached = _batch_poll_cache.get(job_id)
if cached and cached[0] > now:
return cached[1]
result = GroupResult.restore(job_id, app=celery_app)
if result is None:
payload = {
"status": "unknown",
"detail": "Job ID not found in result backend",
}
_batch_poll_cache[job_id] = (now + CELERY_BATCH_POLL_CACHE_TTL_MS / 1000, payload)
return payload
completed = result.completed_count()
total = len(result.results)
if not result.ready():
payload = {
"status": "pending",
"completed": completed,
"total": total,
"progress": round(completed / total, 2) if total > 0 else 0,
}
_batch_poll_cache[job_id] = (now + CELERY_BATCH_POLL_CACHE_TTL_MS / 1000, payload)
return payload
if result.failed():
errors = []
for r in result.results:
if r.failed():
errors.append(str(r.result))
payload = {
"status": "failed",
"errors": errors,
}
_batch_poll_cache[job_id] = (now + CELERY_BATCH_POLL_CACHE_TTL_MS / 1000, payload)
_log_batch_timing(job_id, "failed")
return payload
all_results = []
for r in result.results:
chunk_results = r.get()
all_results.extend(chunk_results)
payload = {
"status": "completed",
"image_count": len(all_results),
"results": all_results,
}
_batch_poll_cache[job_id] = (now + CELERY_BATCH_POLL_CACHE_TTL_MS / 1000, payload)
_log_batch_timing(job_id, "completed")
return payload
def _log_batch_timing(job_id: str, status: str) -> None:
start = _batch_timings.pop(job_id, None)
if start is not None:
duration_ms = (time.monotonic() - start) * 1000
logger.info(
"Celery batch %s job_id=%s duration_ms=%.0f",
status,
job_id,
duration_ms,
)
else:
logger.info("Celery batch %s job_id=%s (no start time recorded)", status, job_id)

View File

@@ -5,6 +5,7 @@ import asyncio
import base64 import base64
import io import io
import logging import logging
import os
from dataclasses import dataclass from dataclasses import dataclass
from typing import Any from typing import Any
@@ -52,8 +53,6 @@ SEGMENT_CLASSES_POST = {
6: "baker's cyst", 6: "baker's cyst",
} }
_triton_pipeline_lock = asyncio.Lock()
@dataclass @dataclass
class CvInferenceOptions: class CvInferenceOptions:
@@ -76,6 +75,12 @@ def _encode_image_to_data_url(image_pil: Image.Image) -> str:
return f"data:image/png;base64,{encoded}" return f"data:image/png;base64,{encoded}"
def _encode_image_to_bytes(image_pil: Image.Image) -> bytes:
buffered = io.BytesIO()
image_pil.save(buffered, format="PNG")
return buffered.getvalue()
def _build_inflammation_payload(logits_row: np.ndarray, config: CalibrationConfig) -> dict: def _build_inflammation_payload(logits_row: np.ndarray, config: CalibrationConfig) -> dict:
interpreted = interpret_inflammation_logits(logits_row, config) interpreted = interpret_inflammation_logits(logits_row, config)
return { return {
@@ -147,6 +152,9 @@ def _build_segmentation_result(
classes_detected = [name for name, mask in masks.items() if np.sum(mask) > 0 and name != "background"] classes_detected = [name for name, mask in masks.items() if np.sum(mask) > 0 and name != "background"]
color_legend = _build_color_legend(classes_detected, angle_type) color_legend = _build_color_legend(classes_detected, angle_type)
segmented_png_bytes = _encode_image_to_bytes(overlay)
segmented_b64 = base64.b64encode(segmented_png_bytes).decode()
result: dict[str, Any] = { result: dict[str, Any] = {
"success": True, "success": True,
"angle": angle_payload, "angle": angle_payload,
@@ -161,7 +169,7 @@ def _build_segmentation_result(
}, },
"images": { "images": {
"enhanced": enhanced_data_url, "enhanced": enhanced_data_url,
"segmented": _encode_image_to_data_url(overlay), "segmented": f"data:image/png;base64,{segmented_b64}",
}, },
"models_used": { "models_used": {
"angle": angle_model, "angle": angle_model,
@@ -281,16 +289,27 @@ async def _run_batch_uncached(
if len(frame_ids) != len(images): if len(frame_ids) != len(images):
raise ValueError("frame_ids length must match images length") raise ValueError("frame_ids length must match images length")
async with _triton_pipeline_lock: concurrency = int(os.getenv("CV_BATCH_CONCURRENCY", "2"))
results: list[dict[str, Any]] = [] semaphore = asyncio.Semaphore(concurrency)
infer_modes: list[str] = []
triton_call_count = 0 async def process_one(image_pil: Image.Image, fid: str, index: int):
for image_pil, fid in zip(images, frame_ids, strict=True): async with semaphore:
item, mode, calls = await _run_spec_cv_pipeline_single( if index > 0:
await asyncio.sleep(min(index * 0.15, 1.0))
return await _run_spec_cv_pipeline_single(
image_pil, image_pil,
frame_id=fid, frame_id=fid,
options=options, options=options,
) )
outcomes = await asyncio.gather(
*[process_one(img, fid, idx) for idx, (img, fid) in enumerate(zip(images, frame_ids))],
)
results: list[dict[str, Any]] = []
infer_modes: list[str] = []
triton_call_count = 0
for item, mode, calls in outcomes:
results.append(item) results.append(item)
infer_modes.append(mode) infer_modes.append(mode)
triton_call_count += calls triton_call_count += calls

View File

@@ -25,12 +25,11 @@ INPUT_NAME = "input_image"
OUTPUT_NAME = "logits" OUTPUT_NAME = "logits"
TRITON_INFER_TIMEOUT = float(os.getenv("TRITON_INFER_TIMEOUT", "120")) TRITON_INFER_TIMEOUT = float(os.getenv("TRITON_INFER_TIMEOUT", "120"))
TRITON_INFER_RETRIES = int(os.getenv("TRITON_INFER_RETRIES", "5")) TRITON_INFER_RETRIES = int(os.getenv("TRITON_INFER_RETRIES", "2"))
TRITON_RETRY_BASE_S = float(os.getenv("TRITON_RETRY_BASE_S", "4")) TRITON_RETRY_BASE_S = float(os.getenv("TRITON_RETRY_BASE_S", "2.0"))
TRITON_USE_BATCH_INFER = os.getenv("TRITON_USE_BATCH_INFER", "true").lower() TRITON_USE_BATCH_INFER = os.getenv("TRITON_USE_BATCH_INFER", "true").lower()
RETRYABLE_STATUS = {429, 502, 503, 504} RETRYABLE_STATUS = {429, 502, 503, 504}
_triton_infer_lock = asyncio.Lock()
_adapter: TritonAdapter | None = None _adapter: TritonAdapter | None = None
_adapter_endpoint: str | None = None _adapter_endpoint: str | None = None
@@ -43,6 +42,11 @@ def _get_adapter() -> TritonAdapter:
global _adapter, _adapter_endpoint global _adapter, _adapter_endpoint
endpoint = get_triton_endpoint() endpoint = get_triton_endpoint()
if _adapter is None or _adapter_endpoint != endpoint: if _adapter is None or _adapter_endpoint != endpoint:
if _adapter is not None:
try:
asyncio.get_event_loop().run_until_complete(_adapter.close())
except RuntimeError:
pass
_adapter = TritonAdapter(endpoint_url=endpoint, timeout=TRITON_INFER_TIMEOUT) _adapter = TritonAdapter(endpoint_url=endpoint, timeout=TRITON_INFER_TIMEOUT)
_adapter_endpoint = endpoint _adapter_endpoint = endpoint
return _adapter return _adapter
@@ -201,7 +205,7 @@ def _infer_angle_logits_chunk(images: list[Image.Image], model_name: str) -> tup
if not _should_try_batched_infer(len(images)): if not _should_try_batched_infer(len(images)):
return _infer_angle_logits_sequential(images, model_name), "sequential" return _infer_angle_logits_sequential(images, model_name), "sequential"
try: try:
return _infer_angle_logits_batch(images, model_name, max_retries=1), "batched" return _infer_angle_logits_batch(images, model_name), "batched"
except Exception as exc: except Exception as exc:
logger.warning( logger.warning(
"Batched angle infer×%s failed (%s); falling back to sequential single-image calls", "Batched angle infer×%s failed (%s); falling back to sequential single-image calls",
@@ -249,7 +253,7 @@ def _infer_inflammation_logits_chunk(
if not _should_try_batched_infer(len(images)): if not _should_try_batched_infer(len(images)):
return _infer_inflammation_logits_sequential(images, model_name), "sequential" return _infer_inflammation_logits_sequential(images, model_name), "sequential"
try: try:
return _infer_inflammation_logits_batch(images, model_name, max_retries=1), "batched" return _infer_inflammation_logits_batch(images, model_name), "batched"
except Exception as exc: except Exception as exc:
logger.warning( logger.warning(
"Batched inflammation infer×%s failed (%s); falling back to sequential", "Batched inflammation infer×%s failed (%s); falling back to sequential",
@@ -297,7 +301,7 @@ def _infer_segmentation_logits_chunk(
if not _should_try_batched_infer(len(images)): if not _should_try_batched_infer(len(images)):
return _infer_segmentation_logits_sequential(images, model_name), "sequential" return _infer_segmentation_logits_sequential(images, model_name), "sequential"
try: try:
return _infer_segmentation_logits_batch(images, model_name, max_retries=1), "batched" return _infer_segmentation_logits_batch(images, model_name), "batched"
except Exception as exc: except Exception as exc:
logger.warning( logger.warning(
"Batched segmentation infer×%s failed (%s); falling back to sequential", "Batched segmentation infer×%s failed (%s); falling back to sequential",
@@ -311,8 +315,6 @@ async def infer_angle_logits(
images: list[Image.Image], images: list[Image.Image],
model_name: str, model_name: str,
) -> tuple[np.ndarray, Literal["batched", "sequential"], int]: ) -> tuple[np.ndarray, Literal["batched", "sequential"], int]:
"""Run angle classification under the global Triton lock. Returns (logits, mode, call_count)."""
async with _triton_infer_lock:
all_logits: list[np.ndarray] = [] all_logits: list[np.ndarray] = []
modes: list[str] = [] modes: list[str] = []
call_count = 0 call_count = 0
@@ -332,7 +334,6 @@ async def infer_inflammation_logits(
images: list[Image.Image], images: list[Image.Image],
model_name: str, model_name: str,
) -> tuple[np.ndarray, Literal["batched", "sequential"], int]: ) -> tuple[np.ndarray, Literal["batched", "sequential"], int]:
async with _triton_infer_lock:
all_logits: list[np.ndarray] = [] all_logits: list[np.ndarray] = []
modes: list[str] = [] modes: list[str] = []
call_count = 0 call_count = 0
@@ -352,7 +353,6 @@ async def infer_segmentation_logits(
images: list[Image.Image], images: list[Image.Image],
model_name: str, model_name: str,
) -> tuple[np.ndarray, Literal["batched", "sequential"], int]: ) -> tuple[np.ndarray, Literal["batched", "sequential"], int]:
async with _triton_infer_lock:
all_logits: list[np.ndarray] = [] all_logits: list[np.ndarray] = []
modes: list[str] = [] modes: list[str] = []
call_count = 0 call_count = 0

View File

@@ -1,6 +1,7 @@
"""Warm up Modal Triton on CV service startup — reduces cold-start 502s and first-frame latency.""" """Warm up Modal Triton on CV service startup — reduces cold-start 502s and first-frame latency."""
from __future__ import annotations from __future__ import annotations
import asyncio
import logging import logging
import os import os
@@ -23,6 +24,20 @@ def _warmup_model_versions() -> dict[str, str]:
return versions return versions
async def _warmup_one(
name: str,
coro,
timeout: float,
) -> None:
try:
await asyncio.wait_for(coro, timeout=timeout)
logger.info("Triton warmup %s complete", name)
except asyncio.TimeoutError:
logger.warning("Triton warmup %s timed out after %.1fs", name, timeout)
except Exception as exc:
logger.warning("Triton warmup %s failed: %s", name, exc)
async def warmup_triton_models() -> None: async def warmup_triton_models() -> None:
if os.getenv("CV_SKIP_TRITON_WARMUP", "").lower() in {"1", "true", "yes"}: if os.getenv("CV_SKIP_TRITON_WARMUP", "").lower() in {"1", "true", "yes"}:
logger.info("Triton warmup skipped (CV_SKIP_TRITON_WARMUP)") logger.info("Triton warmup skipped (CV_SKIP_TRITON_WARMUP)")
@@ -36,13 +51,29 @@ async def warmup_triton_models() -> None:
img224 = Image.new("RGB", (224, 224), color=(128, 128, 128)) img224 = Image.new("RGB", (224, 224), color=(128, 128, 128))
img512 = Image.new("RGB", (512, 512), color=(128, 128, 128)) img512 = Image.new("RGB", (512, 512), color=(128, 128, 128))
warmup_timeout = float(os.getenv("TRITON_WARMUP_TIMEOUT", "15"))
logger.info( logger.info(
"Warming up Triton (angle=%s, inflammation=%s, segmentation=%s)…", "Warming up Triton (angle=%s, inflammation=%s, segmentation=%s, timeout=%.1fs)…",
angle_model, angle_model,
inflam_model, inflam_model,
seg_model, seg_model,
warmup_timeout,
)
await _warmup_one(
"angle",
triton_runtime.infer_angle_logits_single(img224, angle_model),
warmup_timeout,
)
await _warmup_one(
"inflammation",
triton_runtime.infer_inflammation_logits_single(img224, inflam_model),
warmup_timeout,
)
await _warmup_one(
"segmentation",
triton_runtime.infer_segmentation_logits_single(img512, seg_model),
warmup_timeout,
) )
await triton_runtime.infer_angle_logits_single(img224, angle_model)
await triton_runtime.infer_inflammation_logits_single(img224, inflam_model)
await triton_runtime.infer_segmentation_logits_single(img512, seg_model)
logger.info("Triton warmup complete") logger.info("Triton warmup complete")

View File

@@ -0,0 +1,179 @@
#!/usr/bin/env bash
#
# Start Redis + Celery worker for CV inference experiments.
#
# Usage:
# ./backend/start_celery_workers.sh # start both
# ./backend/start_celery_workers.sh stop # stop both
# ./backend/start_celery_workers.sh restart # restart both
# ./backend/start_celery_workers.sh status # show status
#
# Logs:
# logs/redis.log
# logs/celery.log
#
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CODEBASE_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
LOG_DIR="${CODEBASE_ROOT}/logs"
REDIS_PID_FILE="${LOG_DIR}/redis.pid"
CELERY_PID_FILE="${LOG_DIR}/celery.pid"
mkdir -p "${LOG_DIR}"
cd "${CODEBASE_ROOT}"
export PYTHONPATH="${CODEBASE_ROOT}:${PYTHONPATH:-}"
REDIS_PORT="${REDIS_PORT:-6379}"
REDIS_DB="${REDIS_DB:-0}"
CELERY_CHUNK_SIZE="${CELERY_CHUNK_SIZE:-4}"
export TRITON_ENDPOINT="${TRITON_ENDPOINT:-https://dtj-tran--triton-s3-service-unified-triton-server.modal.run}"
is_redis_running() {
if [ -f "${REDIS_PID_FILE}" ]; then
local pid
pid=$(cat "${REDIS_PID_FILE}")
if kill -0 "${pid}" 2>/dev/null; then
return 0
else
rm -f "${REDIS_PID_FILE}"
fi
fi
return 1
}
start_redis() {
if is_redis_running; then
echo "Redis already running (PID $(cat ${REDIS_PID_FILE}))"
return 0
fi
echo "Starting Redis on port ${REDIS_PORT}..."
redis-server \
--port "${REDIS_PORT}" \
--daemonize yes \
--pidfile "${REDIS_PID_FILE}" \
--logfile "${LOG_DIR}/redis.log" \
--save "" \
--appendonly no
sleep 0.5
if is_redis_running; then
echo "Redis started (PID $(cat ${REDIS_PID_FILE}))"
else
echo "Redis failed to start. Check ${LOG_DIR}/redis.log"
exit 1
fi
}
start_celery() {
if [ -f "${CELERY_PID_FILE}" ] && kill -0 "$(cat "${CELERY_PID_FILE}")" 2>/dev/null; then
echo "Celery already running (PID $(cat ${CELERY_PID_FILE}))"
return 0
fi
echo "Starting Celery worker..."
nohup celery \
-A backend.services.celery_app \
worker \
--loglevel=info \
-Q cv-inference \
--concurrency=2 \
> "${LOG_DIR}/celery.log" 2>&1 &
echo $! > "${CELERY_PID_FILE}"
sleep 1
if kill -0 "$(cat "${CELERY_PID_FILE}")" 2>/dev/null; then
echo "Celery started (PID $(cat ${CELERY_PID_FILE}))"
echo " Logs: ${LOG_DIR}/celery.log"
else
echo "Celery failed to start. Check ${LOG_DIR}/celery.log"
rm -f "${CELERY_PID_FILE}"
exit 1
fi
}
stop_redis() {
if is_redis_running; then
local pid
pid=$(cat "${REDIS_PID_FILE}")
echo "Stopping Redis (PID ${pid})..."
kill "${pid}" 2>/dev/null || true
sleep 0.5
rm -f "${REDIS_PID_FILE}"
echo "Redis stopped"
else
echo "Redis not running"
fi
}
stop_celery() {
if [ -f "${CELERY_PID_FILE}" ] && kill -0 "$(cat "${CELERY_PID_FILE}")" 2>/dev/null; then
local pid
pid=$(cat "${CELERY_PID_FILE}")
echo "Stopping Celery (PID ${pid})..."
kill "${pid}" 2>/dev/null || true
sleep 1
# Force kill if still running
if kill -0 "${pid}" 2>/dev/null; then
kill -9 "${pid}" 2>/dev/null || true
fi
rm -f "${CELERY_PID_FILE}"
echo "Celery stopped"
else
echo "Celery not running"
fi
}
status() {
echo "=== Worker Status ==="
if is_redis_running; then
echo "Redis: running (PID $(cat ${REDIS_PID_FILE}))"
else
echo "Redis: stopped"
fi
if [ -f "${CELERY_PID_FILE}" ] && kill -0 "$(cat "${CELERY_PID_FILE}")" 2>/dev/null; then
echo "Celery: running (PID $(cat ${CELERY_PID_FILE}))"
else
echo "Celery: stopped"
fi
}
case "${1:-start}" in
start)
start_redis
start_celery
echo ""
echo "Workers ready. Test with:"
echo " curl http://localhost:8001/api/test/analyze/batch/celery"
;;
stop)
stop_celery
stop_redis
;;
restart)
stop_celery
stop_redis
start_redis
start_celery
;;
status)
status
;;
*)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
;;
esac

View File

@@ -0,0 +1,34 @@
# Dependencies
node_modules/
# Build output
dist/
build/
# Vite
.vite/
# IDE
.idea/
.vscode/
*.swp
*.swo
# OS
.DS_Store
Thumbs.db
# Logs
*.log
npm-debug.log*
# Environment files (migrated to config/frontend.config.yaml)
.env
.env.*
!.env.example
# Test coverage
coverage/
# Misc
*.local

View File

@@ -0,0 +1,13 @@
# Frontend application configuration
# This is the single source of truth for frontend feature flags and URLs.
# Edit this file directly instead of using .env / .env.development.
VITE_USE_BACKEND_SEGMENTATION: "true"
VITE_SEGMENT_API_BASE: ""
VITE_USE_CV_CELERY: "false"
VITE_API_BASE_URL: ""
VITE_CLINICAL_CHAT_USE_LLM: "true"
VITE_CLINICAL_CHAT_MOCK_TOOLS: "true"
VITE_OLLAMA_CHAT_URL: "/api/ollama-chat/api/chat"
VITE_OLLAMA_MODEL: "gemma4:e4b"
VITE_MODAL_OLLAMA_TARGET: "https://dtj-tran--ollama-gemma4-e4b-ollamaserver-web.modal.run"

View File

@@ -27,7 +27,7 @@ export default function CalibrationControls({ config, onChange }: CalibrationCon
return ( return (
<div className="cal-ctrl"> <div className="cal-ctrl">
<div className="cal-ctrl__header"> <div className="cal-ctrl__header">
<span className="cal-ctrl__title">Điều chỉnh nhiệt đ (T)</span> <span className="cal-ctrl__title">Điều chỉnh đ chắc chắn (T)</span>
<span className="cal-ctrl__hint tnum">T = {config.temperature.toFixed(2)}</span> <span className="cal-ctrl__hint tnum">T = {config.temperature.toFixed(2)}</span>
</div> </div>
<CalibrationMetricHelp layout="block" /> <CalibrationMetricHelp layout="block" />

View File

@@ -32,10 +32,17 @@ const MODEL_LOAD_COPY: Record<
'installing-gemma': { 'installing-gemma': {
title: 'Đang cài đặt Gemma 4 E2B về máy…', title: 'Đang cài đặt Gemma 4 E2B về máy…',
subtitle: subtitle:
'Mô hình trò chuyện chính (~2 GB). Nếu bị gián đoạn, lần mở sau sẽ tiếp tục từ chỗ đã tải.', 'Mô hình trò chuyện chính (~1.87 GB). Nếu bị gián đoạn, lần mở sau sẽ tiếp tục từ chỗ đã tải.',
ariaLabel: 'Đang cài đặt Gemma 4 E2B', ariaLabel: 'Đang cài đặt Gemma 4 E2B',
composerPlaceholder: 'Đang cài đặt Gemma 4 E2B, vui lòng đợi…', composerPlaceholder: 'Đang cài đặt Gemma 4 E2B, vui lòng đợi…',
}, },
'resuming-gemma': {
title: 'Đang tiếp tục tải Gemma 4 E2B…',
subtitle:
'Lần tải trước bị gián đoạn — đang tiếp tục từ chỗ đã tải, không tải lại từ đầu.',
ariaLabel: 'Đang tiếp tục tải Gemma 4 E2B',
composerPlaceholder: 'Đang tiếp tục tải Gemma 4 E2B, vui lòng đợi…',
},
// 'installing-qwen': { ... }, // 'installing-qwen': { ... },
'loading-gemma': { 'loading-gemma': {
title: 'Đang nạp Gemma 4 E2B…', title: 'Đang nạp Gemma 4 E2B…',
@@ -78,6 +85,7 @@ export default function ClinicalChatPanel({
modelLoadPhase, modelLoadPhase,
modelLoadProgress, modelLoadProgress,
modelInstallTransferLabel, modelInstallTransferLabel,
modelLoadStalled,
modelLoadFading, modelLoadFading,
sendMessage, sendMessage,
stopGeneration, stopGeneration,
@@ -150,8 +158,17 @@ export default function ClinicalChatPanel({
const showLoadBubble = isModelLoading && modelLoadPhase !== null; const showLoadBubble = isModelLoading && modelLoadPhase !== null;
const loadCopy = modelLoadPhase ? MODEL_LOAD_COPY[modelLoadPhase] : null; const loadCopy = modelLoadPhase ? MODEL_LOAD_COPY[modelLoadPhase] : null;
const progressPercent = Math.min(100, Math.max(0, Math.round(modelLoadProgress))); const progressPercent = Math.min(100, Math.max(0, Math.round(modelLoadProgress)));
const installProgressLabel = const isInstallStalled =
modelLoadPhase && isModelInstallPhase(modelLoadPhase) && modelInstallTransferLabel modelLoadStalled && modelLoadPhase !== null && isModelInstallPhase(modelLoadPhase);
const loadTitle = isInstallStalled ? 'Tải Gemma bị gián đoạn' : loadCopy?.title;
const loadSubtitle = isInstallStalled
? `Mất kết nối — hiện không tải được. Đang thử lại tự động${
modelInstallTransferLabel ? ` · đã lưu ${modelInstallTransferLabel}` : ''
}. Có thể tải lại trang để tiếp tục từ chỗ đã tải.`
: loadCopy?.subtitle;
const installProgressLabel = isInstallStalled
? 'Gián đoạn'
: modelLoadPhase && isModelInstallPhase(modelLoadPhase) && modelInstallTransferLabel
? modelInstallTransferLabel ? modelInstallTransferLabel
: `${progressPercent}%`; : `${progressPercent}%`;
const activeModeMeta = getInferenceModeMeta(inferenceMode); const activeModeMeta = getInferenceModeMeta(inferenceMode);
@@ -340,20 +357,37 @@ export default function ClinicalChatPanel({
aria-hidden aria-hidden
/> />
<div <div
className={`clinical-chat__load-bubble clinical-chat__load-bubble--${modelLoadPhase} ${modelLoadFading ? 'clinical-chat__load-bubble--fade-out' : ''}`} className={`clinical-chat__load-bubble clinical-chat__load-bubble--${modelLoadPhase} ${isInstallStalled ? 'clinical-chat__load-bubble--stalled' : ''} ${modelLoadFading ? 'clinical-chat__load-bubble--fade-out' : ''}`}
role="status" role="status"
aria-live="polite" aria-live="polite"
aria-label={ aria-label={
modelLoadPhase && isModelInstallPhase(modelLoadPhase) && modelInstallTransferLabel isInstallStalled
? `${loadCopy.ariaLabel} bị gián đoạn, đang thử lại`
: modelLoadPhase && isModelInstallPhase(modelLoadPhase) && modelInstallTransferLabel
? `${loadCopy.ariaLabel}, ${modelInstallTransferLabel}` ? `${loadCopy.ariaLabel}, ${modelInstallTransferLabel}`
: `${loadCopy.ariaLabel}, ${progressPercent} phần trăm` : `${loadCopy.ariaLabel}, ${progressPercent} phần trăm`
} }
> >
<div <div
className={`clinical-chat__load-icon clinical-chat__load-icon--${modelLoadPhase}`} className={`clinical-chat__load-icon clinical-chat__load-icon--${modelLoadPhase} ${isInstallStalled ? 'clinical-chat__load-icon--stalled' : ''}`}
aria-hidden aria-hidden
> >
{modelLoadPhase && isModelInstallPhase(modelLoadPhase) ? ( {isInstallStalled ? (
<svg width="20" height="20" viewBox="0 0 24 24" fill="none">
<path
d="M12 8v5M12 16.5v.5"
stroke="currentColor"
strokeWidth="1.75"
strokeLinecap="round"
/>
<path
d="M10.3 3.9 2.4 18a2 2 0 0 0 1.7 3h15.8a2 2 0 0 0 1.7-3L13.7 3.9a2 2 0 0 0-3.4 0Z"
stroke="currentColor"
strokeWidth="1.6"
strokeLinejoin="round"
/>
</svg>
) : modelLoadPhase && isModelInstallPhase(modelLoadPhase) ? (
<svg width="20" height="20" viewBox="0 0 24 24" fill="none"> <svg width="20" height="20" viewBox="0 0 24 24" fill="none">
<path <path
d="M12 3v3M12 18v3M4.22 4.22l2.12 2.12M17.66 17.66l2.12 2.12M3 12h3M18 12h3M4.22 19.78l2.12-2.12M17.66 6.34l2.12-2.12" d="M12 3v3M12 18v3M4.22 4.22l2.12 2.12M17.66 17.66l2.12 2.12M3 12h3M18 12h3M4.22 19.78l2.12-2.12M17.66 6.34l2.12-2.12"
@@ -373,16 +407,16 @@ export default function ClinicalChatPanel({
</svg> </svg>
)} )}
</div> </div>
<p className="clinical-chat__load-title">{loadCopy.title}</p> <p className="clinical-chat__load-title">{loadTitle}</p>
<p className="clinical-chat__load-subtitle">{loadCopy.subtitle}</p> <p className="clinical-chat__load-subtitle">{loadSubtitle}</p>
<div className="clinical-chat__load-progress-row"> <div className="clinical-chat__load-progress-row">
<div className="clinical-chat__load-progress-track"> <div className="clinical-chat__load-progress-track">
<div <div
className={`clinical-chat__load-progress-fill clinical-chat__load-progress-fill--${modelLoadPhase}`} className={`clinical-chat__load-progress-fill clinical-chat__load-progress-fill--${modelLoadPhase} ${isInstallStalled ? 'clinical-chat__load-progress-fill--stalled' : ''}`}
style={{ width: `${progressPercent}%` }} style={{ width: `${progressPercent}%` }}
/> />
</div> </div>
<span className="clinical-chat__load-progress-label tnum">{installProgressLabel}</span> <span className={`clinical-chat__load-progress-label tnum ${isInstallStalled ? 'clinical-chat__load-progress-label--stalled' : ''}`}>{installProgressLabel}</span>
</div> </div>
</div> </div>
</> </>
@@ -781,12 +815,43 @@ const styles = `
color: var(--color-secondary); color: var(--color-secondary);
} }
.clinical-chat__load-icon--installing-gemma, .clinical-chat__load-icon--installing-gemma,
.clinical-chat__load-icon--resuming-gemma,
.clinical-chat__load-icon--installing-qwen { .clinical-chat__load-icon--installing-qwen {
animation: clinical-chat-spin 2.4s linear infinite; animation: clinical-chat-spin 2.4s linear infinite;
} }
.clinical-chat__load-icon--installing { .clinical-chat__load-icon--installing {
animation: clinical-chat-spin 2.4s linear infinite; animation: clinical-chat-spin 2.4s linear infinite;
} }
/* Stalled: stop the spin so a frozen download never looks like active progress. */
.clinical-chat__load-bubble--stalled {
border-color: rgba(180, 120, 40, 0.4);
}
.clinical-chat__load-icon--stalled {
animation: clinical-chat-stall-pulse 1.6s ease-in-out infinite;
background: rgba(180, 120, 40, 0.14);
color: #9a6b1f;
}
@keyframes clinical-chat-stall-pulse {
0%, 100% { opacity: 0.55; }
50% { opacity: 1; }
}
.clinical-chat__load-progress-fill--stalled {
background: repeating-linear-gradient(
45deg,
rgba(180, 120, 40, 0.55),
rgba(180, 120, 40, 0.55) 6px,
rgba(180, 120, 40, 0.3) 6px,
rgba(180, 120, 40, 0.3) 12px
);
animation: clinical-chat-stall-stripes 0.8s linear infinite;
}
@keyframes clinical-chat-stall-stripes {
from { background-position: 0 0; }
to { background-position: 17px 0; }
}
.clinical-chat__load-progress-label--stalled {
color: #9a6b1f;
}
@keyframes clinical-chat-pulse-icon { @keyframes clinical-chat-pulse-icon {
0%, 100% { opacity: 0.55; transform: scale(0.94); } 0%, 100% { opacity: 0.55; transform: scale(0.94); }
50% { opacity: 1; transform: scale(1); } 50% { opacity: 1; transform: scale(1); }

View File

@@ -1,4 +1,4 @@
import { memo, useEffect, useId, useLayoutEffect, useRef, useState } from 'react'; import { memo, useEffect, useId, useRef, useState } from 'react';
import StreamingPlainText from '../atoms/StreamingPlainText'; import StreamingPlainText from '../atoms/StreamingPlainText';
import { streamTargetKey } from '../../lib/llm/clinicalChatStreamRegistry'; import { streamTargetKey } from '../../lib/llm/clinicalChatStreamRegistry';
@@ -18,26 +18,14 @@ function ClinicalChatThought({
thoughtStreaming = false, thoughtStreaming = false,
}: ClinicalChatThoughtProps) { }: ClinicalChatThoughtProps) {
const panelId = useId(); const panelId = useId();
const wasThoughtStreamingRef = useRef(thoughtStreaming);
const userToggledRef = useRef(false); const userToggledRef = useRef(false);
const [expanded, setExpanded] = useState(true); // Collapsed by default — keeps the clinical answer prominent; click to expand.
const [expanded, setExpanded] = useState(false);
useEffect(() => { useEffect(() => {
setExpanded(true);
userToggledRef.current = false;
wasThoughtStreamingRef.current = false;
}, [messageId]);
useLayoutEffect(() => {
if (thoughtStreaming) {
if (!userToggledRef.current) {
setExpanded(true);
}
} else if (wasThoughtStreamingRef.current && !thoughtStreaming && !userToggledRef.current) {
setExpanded(false); setExpanded(false);
} userToggledRef.current = false;
wasThoughtStreamingRef.current = thoughtStreaming; }, [messageId]);
}, [thoughtStreaming]);
if (!content.trim() && !thoughtStreaming) { if (!content.trim() && !thoughtStreaming) {
return null; return null;
@@ -49,7 +37,8 @@ function ClinicalChatThought({
}; };
const label = thoughtStreaming ? 'Đang suy luận' : 'Suy luận'; const label = thoughtStreaming ? 'Đang suy luận' : 'Suy luận';
const preview = !thoughtStreaming && !expanded ? content.replace(/\s+/g, ' ').trim().slice(0, 100) : ''; const preview =
!expanded && !thoughtStreaming ? content.replace(/\s+/g, ' ').trim().slice(0, 100) : '';
return ( return (
<div <div
@@ -77,15 +66,15 @@ function ClinicalChatThought({
{!expanded && preview ? ( {!expanded && preview ? (
<p className="clinical-chat__thought-preview">{preview}</p> <p className="clinical-chat__thought-preview">{preview}</p>
) : null} ) : null}
{expanded ? ( {/* Body stays mounted even when collapsed so the imperative stream target keeps
<div id={panelId} className="clinical-chat__thought-body"> receiving tokens; visibility is toggled via the hidden attribute. */}
<div id={panelId} className="clinical-chat__thought-body" hidden={!expanded}>
<StreamingPlainText <StreamingPlainText
text={content} text={content}
streamTargetKey={streamTargetKey(messageId, 'thought')} streamTargetKey={streamTargetKey(messageId, 'thought')}
className="clinical-chat__thought-md chat-md__plain" className="clinical-chat__thought-md chat-md__plain"
/> />
</div> </div>
) : null}
</div> </div>
); );
} }

View File

@@ -48,32 +48,35 @@ export default function RecordingModeSelector({ value, onChange, disabled }: Rec
padding: 0; padding: 0;
} }
.recording-mode-selector legend { .recording-mode-selector legend {
font-size: 11px; font-size: 13px;
font-weight: 600; font-weight: 700;
color: #94a3b8; color: #e2e8f0;
margin-bottom: 6px; margin-bottom: 8px;
} }
.recording-mode-selector__options { .recording-mode-selector__options {
display: flex; display: flex;
flex-direction: column; flex-direction: column;
gap: 4px; gap: 8px;
} }
.recording-mode-selector__option { .recording-mode-selector__option {
display: flex; display: flex;
align-items: center; align-items: center;
gap: 8px; gap: 10px;
font-size: 12px; font-size: 14px;
color: #e2e8f0; font-weight: 500;
color: #f1f5f9;
cursor: pointer; cursor: pointer;
} }
.recording-mode-selector__option input { .recording-mode-selector__option input {
width: 16px;
height: 16px;
accent-color: #76c8b1; accent-color: #76c8b1;
} }
.recording-mode-selector__hint { .recording-mode-selector__hint {
margin: 6px 0 0; margin: 8px 0 0;
font-size: 11px; font-size: 12.5px;
line-height: 1.45; line-height: 1.5;
color: #64748b; color: #cbd5e1;
} }
.recording-mode-selector:disabled .recording-mode-selector__option { .recording-mode-selector:disabled .recording-mode-selector__option {
opacity: 0.55; opacity: 0.55;

View File

@@ -71,7 +71,7 @@ export default function SeverityBadge({
<div className="severity-panel"> <div className="severity-panel">
{severityLoading ? ( {severityLoading ? (
<div className="severity-panel__block glass"> <div className="severity-panel__block glass">
<span className="severity-panel__eyebrow">Viêm màng hoạt dịch đ nặng (từ phân đoạn)</span> <span className="severity-panel__eyebrow">Viêm màng hoạt dịch đ nặng (từ nh phân đoạn)</span>
<p className="severity-panel__pending">Đang chờ kết quả đ nặng từ phân đoạn</p> <p className="severity-panel__pending">Đang chờ kết quả đ nặng từ phân đoạn</p>
</div> </div>
) : grade != null && gradeLabel ? ( ) : grade != null && gradeLabel ? (
@@ -80,7 +80,7 @@ export default function SeverityBadge({
{grade} {grade}
</span> </span>
<div> <div>
<span className="severity-badge__label">Viêm màng hoạt dịch đ nặng (từ phân đoạn)</span> <span className="severity-badge__label">Viêm màng hoạt dịch đ nặng (từ nh phân đoạn)</span>
<strong>{gradeLabel}</strong> <strong>{gradeLabel}</strong>
{severity?.description && ( {severity?.description && (
<p className="severity-panel__desc">{severity.description}</p> <p className="severity-panel__desc">{severity.description}</p>

View File

@@ -34,6 +34,7 @@ interface DiagnosticCanvasProps {
patientMrn?: string; patientMrn?: string;
patientId?: string; patientId?: string;
scanFrames?: ScanFrame[]; scanFrames?: ScanFrame[];
useCelery?: boolean;
} }
export default function DiagnosticCanvas({ export default function DiagnosticCanvas({
@@ -54,6 +55,7 @@ export default function DiagnosticCanvas({
patientMrn, patientMrn,
patientId, patientId,
scanFrames: scanFramesProp, scanFrames: scanFramesProp,
useCelery,
}: DiagnosticCanvasProps) { }: DiagnosticCanvasProps) {
const activeFrames = const activeFrames =
(scanFramesProp && scanFramesProp.length > 0) (scanFramesProp && scanFramesProp.length > 0)
@@ -146,7 +148,7 @@ export default function DiagnosticCanvas({
const lockFrameNav = isSingleFrameNavLocked; const lockFrameNav = isSingleFrameNavLocked;
const { overlaySrc, interpretation, angleClassification, inflammationClassification, synovitisSeverity, isLoading, isSegmentationLoading, error, source, retry } = const { overlaySrc, interpretation, angleClassification, inflammationClassification, synovitisSeverity, isLoading, isSegmentationLoading, error, source, retry } =
useSegmentationOverlay(frame.id, frame.src, framesForCanvas, patientMrn); useSegmentationOverlay(frame.id, frame.src, framesForCanvas, patientMrn, useCelery);
useEffect(() => { useEffect(() => {
applyFrameIndex(0); applyFrameIndex(0);
@@ -360,13 +362,6 @@ export default function DiagnosticCanvas({
{showMask && ( {showMask && (
<SegmentationOverlay overlaySrc={overlaySrc} isLoading={isLoading} /> <SegmentationOverlay overlaySrc={overlaySrc} isLoading={isLoading} />
)} )}
{showMask && error && !isLoading && (
<MlServiceErrorPanel
error={error}
frameLabel={frameLabel}
onRetry={source === 'backend' ? retry : undefined}
/>
)}
<canvas <canvas
ref={canvasRef} ref={canvasRef}
className="diagnostic-canvas__annotation-canvas" className="diagnostic-canvas__annotation-canvas"
@@ -377,6 +372,13 @@ export default function DiagnosticCanvas({
{...drawHandlers} {...drawHandlers}
/> />
</div> </div>
{showMask && error && !isLoading && (
<MlServiceErrorPanel
error={error}
frameLabel={frameLabel}
onRetry={source === 'backend' ? retry : undefined}
/>
)}
{closedLoopPrompt && closedLoopPromptViewportAnchor && ( {closedLoopPrompt && closedLoopPromptViewportAnchor && (
<ClosedLoopPrompt <ClosedLoopPrompt
anchor={closedLoopPromptViewportAnchor} anchor={closedLoopPromptViewportAnchor}

View File

@@ -249,7 +249,7 @@ export default function ReviewDiagnosticSessionPanel({
return ( return (
<div className="review-session"> <div className="review-session">
<header className="review-session__header"> <header className="review-session__header">
<h2 className="review-session__title">Xem lại phiên chẩn đoán</h2> <h2 className="review-session__title">XEM LẠI PHIÊN CHUẨN ĐOÁN</h2>
<RecordingModeSelector <RecordingModeSelector
value={lifecycle.recordingMode} value={lifecycle.recordingMode}
onChange={lifecycle.setRecordingMode} onChange={lifecycle.setRecordingMode}
@@ -572,7 +572,7 @@ export default function ReviewDiagnosticSessionPanel({
padding-bottom: 8px; padding-bottom: 8px;
} }
.review-session__title { .review-session__title {
margin: 0; margin: 0 0 16px;
font-size: 15px; font-size: 15px;
font-weight: 700; font-weight: 700;
letter-spacing: 0.02em; letter-spacing: 0.02em;

View File

@@ -155,7 +155,7 @@ export default function SideNavBar({
<> <>
<aside className="side-nav glass-elevated"> <aside className="side-nav glass-elevated">
<section className="side-nav__section"> <section className="side-nav__section">
<h3>Đ xuất AI</h3> <h3>Đ xuất của AI</h3>
{hasCalibratableOutput && ( {hasCalibratableOutput && (
<CalibrationControls config={userConfig} onChange={setUserConfig} /> <CalibrationControls config={userConfig} onChange={setUserConfig} />

View File

@@ -153,9 +153,11 @@ export default function WorkspaceShell({ zoneA, zoneB }: WorkspaceShellProps) {
<style>{` <style>{`
.workspace-shell { .workspace-shell {
display: flex; display: flex;
flex: 1; flex: 0 0 calc(100dvh - var(--topbar-h) - var(--bottombar-h));
height: calc(100dvh - var(--topbar-h) - var(--bottombar-h));
min-height: 0; min-height: 0;
padding: var(--space-md); padding: var(--space-md);
overflow: hidden;
user-select: ${isDragging ? 'none' : 'auto'}; user-select: ${isDragging ? 'none' : 'auto'};
} }
.workspace-shell--dragging { .workspace-shell--dragging {
@@ -164,8 +166,10 @@ export default function WorkspaceShell({ zoneA, zoneB }: WorkspaceShellProps) {
.workspace-shell__zone-a { .workspace-shell__zone-a {
flex: 0 0 var(--workspace-zone-a-pct); flex: 0 0 var(--workspace-zone-a-pct);
min-width: 0; min-width: 0;
min-height: 0;
display: flex; display: flex;
flex-direction: column; flex-direction: column;
overflow: hidden;
transition: flex-basis 0.05s linear; transition: flex-basis 0.05s linear;
} }
.workspace-shell--dragging .workspace-shell__zone-a { .workspace-shell--dragging .workspace-shell__zone-a {
@@ -174,8 +178,10 @@ export default function WorkspaceShell({ zoneA, zoneB }: WorkspaceShellProps) {
.workspace-shell__zone-b { .workspace-shell__zone-b {
flex: 1 1 0; flex: 1 1 0;
min-width: 0; min-width: 0;
min-height: 0;
display: flex; display: flex;
flex-direction: column; flex-direction: column;
overflow: hidden;
font-size: calc(1rem * var(--workspace-panel-scale, 1)); font-size: calc(1rem * var(--workspace-panel-scale, 1));
} }
.workspace-shell__divider { .workspace-shell__divider {
@@ -246,13 +252,17 @@ export default function WorkspaceShell({ zoneA, zoneB }: WorkspaceShellProps) {
} }
@media (max-width: 1024px) { @media (max-width: 1024px) {
.workspace-shell { .workspace-shell {
flex: 1 1 auto;
height: auto;
flex-direction: column; flex-direction: column;
gap: var(--space-md); gap: var(--space-md);
overflow: visible;
} }
.workspace-shell__zone-a, .workspace-shell__zone-a,
.workspace-shell__zone-b { .workspace-shell__zone-b {
flex: 1 1 auto; flex: 1 1 auto;
font-size: 1rem; font-size: 1rem;
overflow: visible;
} }
} }
`}</style> `}</style>

View File

@@ -27,7 +27,7 @@ export const CALIBRATION_METRIC_HELP_KEY_POINTS: readonly CalibrationKeyPoint[]
{ {
title: 'Cơ chế bóp méo của AI hiện đại', title: 'Cơ chế bóp méo của AI hiện đại',
body: body:
'Các mạng càng sâu và thông minh thì xu hướng phân tách các điểm thô càng xa (ví dụ 20 điểm và 1 điểm). Khi quy đổi, khoảng cách quá lớn này bị ép thành xác suất cực đoan như 99,9% — hiện tượng “quá tự tin” (over-confidence).', 'Các mô hình AI càng phức tạp (kích thuớc lớn & sâu) và thông minh thì xu hướng phân tách các điểm thô càng xa (ví dụ 20 điểm và 1 điểm). Khi quy đổi, khoảng cách quá lớn này bị ép thành xác suất cực đoan như 99,9% — hiện tượng “quá tự tin” (over-confidence).',
}, },
{ {
title: 'Bác sĩ trưởng khoa hạ hỏa (tham số T)', title: 'Bác sĩ trưởng khoa hạ hỏa (tham số T)',

View File

@@ -22,19 +22,19 @@ export const CALIBRATION_TIERS: CalibrationTier[] = [
tier: 1, tier: 1,
labelVi: 'Nhạy cao / Sàng lọc', labelVi: 'Nhạy cao / Sàng lọc',
labelEn: 'Aggressive / Screening', labelEn: 'Aggressive / Screening',
triggerVi: 'Bác sĩ nghi ngờ viêm cao từ triệu chứng, chưa thấy trên ảnh.', triggerVi: 'Bác sĩ nghi ngờ mức độ viêm cao từ triệu chứng, chưa thấy trên ảnh.',
ruleVi: 'T = 0.7 (Sharpening)', ruleVi: 'T = 0.7',
recommendedT: 0.7, recommendedT: 0.7,
uiEffectVi: uiEffectVi:
'Đẩy xác suất biên lên — hạ ngưỡng cảnh báo viêm để không bỏ sót ca ranh giới.', 'Đẩy xác suất biên lên — hạ ngưỡng cảnh báo viêm để không bỏ sót ca hiếm gặp.',
}, },
{ {
id: 'standard', id: 'standard',
tier: 2, tier: 2,
labelVi: 'Chuẩn / Mặc định', labelVi: 'Chuẩn / Mặc định',
labelEn: 'Standard Baseline', labelEn: 'Standard Baseline',
triggerVi: 'Vận hành mặc định — chưa có prior lâm sàng từ người dùng.', triggerVi: 'Vận hành mặc định — chưa có dự đoán từ trước từ người dùng.',
ruleVi: 'T = 1.4 (Smoothing)', ruleVi: 'T = 1.4',
recommendedT: 1.4, recommendedT: 1.4,
uiEffectVi: uiEffectVi:
'Giảm overconfidence của mạng — phân bố thực tế, cân bằng toán học.', 'Giảm overconfidence của mạng — phân bố thực tế, cân bằng toán học.',
@@ -45,7 +45,7 @@ export const CALIBRATION_TIERS: CalibrationTier[] = [
labelVi: 'Bảo thủ / Hoài nghi', labelVi: 'Bảo thủ / Hoài nghi',
labelEn: 'Conservative / Skeptical', labelEn: 'Conservative / Skeptical',
triggerVi: 'Bác sĩ tin bệnh nhân khỏe; dùng AI chỉ để kiểm tra lại.', triggerVi: 'Bác sĩ tin bệnh nhân khỏe; dùng AI chỉ để kiểm tra lại.',
ruleVi: 'T = 2.2 (Heavy flattening)', ruleVi: 'T = 2.2',
recommendedT: 2.2, recommendedT: 2.2,
uiEffectVi: uiEffectVi:
'Làm phẳng phân bố — chỉ báo dương khi tín hiệu mô hình cực kỳ mạnh.', 'Làm phẳng phân bố — chỉ báo dương khi tín hiệu mô hình cực kỳ mạnh.',

View File

@@ -84,12 +84,17 @@ export type ClinicalChatRuntime = 'loading' | 'llm' | 'mock';
export type ModelLoadPhase = export type ModelLoadPhase =
| 'installing-gemma' | 'installing-gemma'
| 'resuming-gemma'
// | 'installing-qwen' // | 'installing-qwen'
| 'loading-gemma'; | 'loading-gemma';
// | 'loading-qwen'; // | 'loading-qwen';
/** No download progress for this long ⇒ treat the install as stalled (not "loading"). */
export const MODEL_INSTALL_STALL_MS = 15_000;
const MODEL_INSTALL_STALL_POLL_MS = 3_000;
export function isModelInstallPhase(phase: ModelLoadPhase): boolean { export function isModelInstallPhase(phase: ModelLoadPhase): boolean {
return phase === 'installing-gemma'; return phase === 'installing-gemma' || phase === 'resuming-gemma';
// || phase === 'installing-qwen'; // || phase === 'installing-qwen';
} }
@@ -137,6 +142,8 @@ export interface UseClinicalChatResult {
modelLoadProgress: number; modelLoadProgress: number;
/** Byte transfer label during OPFS install (e.g. "842.3 MB / 1.87 GB"). */ /** Byte transfer label during OPFS install (e.g. "842.3 MB / 1.87 GB"). */
modelInstallTransferLabel: string | null; modelInstallTransferLabel: string | null;
/** True when an install/resume has received no bytes for a while — not actually progressing. */
modelLoadStalled: boolean;
modelLoadFading: boolean; modelLoadFading: boolean;
sendMessage: () => void; sendMessage: () => void;
stopGeneration: () => void; stopGeneration: () => void;
@@ -212,6 +219,7 @@ export function useClinicalChat({
const [modelLoadPhase, setModelLoadPhase] = useState<ModelLoadPhase | null>(null); const [modelLoadPhase, setModelLoadPhase] = useState<ModelLoadPhase | null>(null);
const [modelLoadProgress, setModelLoadProgress] = useState(0); const [modelLoadProgress, setModelLoadProgress] = useState(0);
const [modelInstallTransferLabel, setModelInstallTransferLabel] = useState<string | null>(null); const [modelInstallTransferLabel, setModelInstallTransferLabel] = useState<string | null>(null);
const [modelLoadStalled, setModelLoadStalled] = useState(false);
const [modelLoadFading, setModelLoadFading] = useState(false); const [modelLoadFading, setModelLoadFading] = useState(false);
const [modeSuggestion, setModeSuggestion] = useState<ModeSuggestion | null>(null); const [modeSuggestion, setModeSuggestion] = useState<ModeSuggestion | null>(null);
@@ -365,11 +373,49 @@ export function useClinicalChat({
let cancelled = false; let cancelled = false;
let fadeTimer: number | undefined; let fadeTimer: number | undefined;
let stopLoadTicker: (() => void) | undefined; let stopLoadTicker: (() => void) | undefined;
let stallWatchdogId: number | undefined;
let lastInstallProgressAt = Date.now();
// Highest byte offset seen so far. Retry "heartbeats" re-emit the same offset;
// only a real increase counts as progress.
let lastInstallBytes = -1;
const clearStallWatchdog = () => {
if (stallWatchdogId !== undefined) {
window.clearInterval(stallWatchdogId);
stallWatchdogId = undefined;
}
};
// Flip the overlay from "downloading" to "stalled" when no *new bytes* arrive for a
// while, so a frozen bar (or a resume stuck retrying the same offset) never
// masquerades as active progress.
const startStallWatchdog = () => {
clearStallWatchdog();
lastInstallProgressAt = Date.now();
lastInstallBytes = -1;
setModelLoadStalled(false);
stallWatchdogId = window.setInterval(() => {
if (cancelled) {
return;
}
if (Date.now() - lastInstallProgressAt > MODEL_INSTALL_STALL_MS) {
setModelLoadStalled(true);
setStatusLabel('Tải Gemma bị gián đoạn — đang thử kết nối lại…');
}
}, MODEL_INSTALL_STALL_POLL_MS);
};
const noteInstallProgress = () => {
lastInstallProgressAt = Date.now();
setModelLoadStalled(false);
};
async function finishModelLoad(): Promise<void> { async function finishModelLoad(): Promise<void> {
if (cancelled) { if (cancelled) {
return; return;
} }
clearStallWatchdog();
setModelLoadStalled(false);
setModelLoadProgress(100); setModelLoadProgress(100);
setModelLoadFading(true); setModelLoadFading(true);
await new Promise<void>((resolve) => { await new Promise<void>((resolve) => {
@@ -469,9 +515,25 @@ export function useClinicalChat({
if (!cancelled) { if (!cancelled) {
setModelLoadProgress(8); setModelLoadProgress(8);
if (!initialGemma.loadable) { if (!initialGemma.loadable) {
setModelLoadPhase('installing-gemma'); // A partial checkpoint already on disk means we resume, not start fresh.
const isPartialResume = initialGemma.bytes > 0;
setModelLoadPhase(isPartialResume ? 'resuming-gemma' : 'installing-gemma');
setIsModelLoading(true); setIsModelLoading(true);
setStatusLabel('Đang cài đặt Gemma 4 E2B về máy…'); setStatusLabel(
isPartialResume
? 'Đang tiếp tục tải Gemma 4 E2B…'
: 'Đang cài đặt Gemma 4 E2B về máy…',
);
if (isPartialResume) {
setModelInstallTransferLabel(
formatInstallTransferLabel({
phase: 'resuming',
bytesLoaded: initialGemma.bytes,
bytesTotal: initialGemma.manifest?.bytes ?? null,
}),
);
}
startStallWatchdog();
} }
// else if (!initialQwen.loadable) { // else if (!initialQwen.loadable) {
// setModelLoadPhase('installing-qwen'); // setModelLoadPhase('installing-qwen');
@@ -486,11 +548,26 @@ export function useClinicalChat({
if (cancelled) { if (cancelled) {
return; return;
} }
setModelLoadPhase('installing-gemma');
setIsModelLoading(true); setIsModelLoading(true);
setStatusLabel('Đang cài đặt Gemma 4 E2B về máy…');
setModelInstallTransferLabel(formatInstallTransferLabel(progress)); setModelInstallTransferLabel(formatInstallTransferLabel(progress));
setModelLoadProgress(mapDownloadProgress(progress)); setModelLoadProgress(mapDownloadProgress(progress));
// Retry heartbeats re-emit the same offset — those must NOT reset the
// watchdog or clear the stalled state, otherwise a stuck resume keeps
// looking like active loading.
const advanced = progress.bytesLoaded > lastInstallBytes;
if (!advanced) {
return;
}
lastInstallBytes = progress.bytesLoaded;
noteInstallProgress();
const resuming = progress.phase === 'resuming';
setModelLoadPhase(resuming ? 'resuming-gemma' : 'installing-gemma');
setStatusLabel(
resuming
? 'Đang tiếp tục tải Gemma 4 E2B…'
: 'Đang cài đặt Gemma 4 E2B về máy…',
);
}, },
// onQwenDownloadProgress: (progress: QwenDownloadProgress) => { // onQwenDownloadProgress: (progress: QwenDownloadProgress) => {
// if (cancelled) { // if (cancelled) {
@@ -537,17 +614,20 @@ export function useClinicalChat({
); );
void preloadGemmaIntoMemory(); void preloadGemmaIntoMemory();
} catch (error) { } catch (error) {
clearStallWatchdog();
if (!cancelled) { if (!cancelled) {
setIsModelLoading(false); setIsModelLoading(false);
setModelLoadFading(false); setModelLoadFading(false);
setModelLoadPhase(null); setModelLoadPhase(null);
setModelInstallTransferLabel(null); setModelInstallTransferLabel(null);
setModelLoadStalled(false);
setRuntime('mock'); setRuntime('mock');
const message = error instanceof Error ? error.message : 'không tải được mô hình'; const message = error instanceof Error ? error.message : 'không tải được mô hình';
const isNetwork = const isNetwork =
error instanceof TypeError || error instanceof TypeError ||
(error instanceof DOMException && error.name === 'NetworkError') || (error instanceof DOMException &&
/network|failed to fetch|interrupted|gián đoạn|network_changed|err_network_changed/i.test( (error.name === 'NetworkError' || error.name === 'TimeoutError')) ||
/network|failed to fetch|interrupted|timed out|timeout|gián đoạn|network_changed|err_network_changed/i.test(
message, message,
); );
setStatusLabel( setStatusLabel(
@@ -563,6 +643,7 @@ export function useClinicalChat({
return () => { return () => {
cancelled = true; cancelled = true;
stopLoadTicker?.(); stopLoadTicker?.();
clearStallWatchdog();
if (fadeTimer !== undefined) { if (fadeTimer !== undefined) {
window.clearTimeout(fadeTimer); window.clearTimeout(fadeTimer);
} }
@@ -765,22 +846,26 @@ export function useClinicalChat({
let ollamaThoughtAcc = ''; let ollamaThoughtAcc = '';
let ollamaAnswerAcc = ''; let ollamaAnswerAcc = '';
const assistantId = createChatMessageId(); // A single generation may spawn continuation segments; each segment renders as
const assistantMessage: ClinicalChatMessage = { // its own bubble with its own reasoning, so continuation thinking never leaks
id: assistantId, // into a prior answer. currentAssistantId tracks the bubble being streamed.
let currentAssistantId = createChatMessageId();
let segmentCount = 1;
const spawnAssistantBubble = (id: string, pondering: boolean): ClinicalChatMessage => ({
id,
role: 'assistant', role: 'assistant',
content: '', content: '',
timestamp: new Date(), timestamp: new Date(),
streaming: true, streaming: true,
tracksThought: thoughtActive, tracksThought: thoughtActive,
pondering: useRemote, pondering,
ponderingVariant: mode === 'agent' ? 'agent' : 'chat', ponderingVariant: mode === 'agent' ? 'agent' : 'chat',
}; });
setMessages((prev) => [...prev, assistantMessage]); setMessages((prev) => [...prev, spawnAssistantBubble(currentAssistantId, useRemote)]);
setStatusLabel(generationStatusLabel(mode, activeLevel)); setStatusLabel(generationStatusLabel(mode, activeLevel));
let plainContentAccumulator = ''; let plainContentAccumulator = '';
const thoughtParser = let thoughtParser =
thoughtActive && !useOllamaThoughtStream ? createCotStreamParser() : null; thoughtActive && !useOllamaThoughtStream ? createCotStreamParser() : null;
let thoughtCompleteEmitted = false; let thoughtCompleteEmitted = false;
let ponderingCleared = false; let ponderingCleared = false;
@@ -790,7 +875,7 @@ export function useClinicalChat({
return; return;
} }
ponderingCleared = true; ponderingCleared = true;
updateMessage(assistantId, { pondering: false }); updateMessage(currentAssistantId, { pondering: false });
}; };
const useImperativeStreamPaint = useRemote || thoughtActive; const useImperativeStreamPaint = useRemote || thoughtActive;
@@ -801,8 +886,9 @@ export function useClinicalChat({
thoughtComplete?: boolean; thoughtComplete?: boolean;
tracksThought?: boolean; tracksThought?: boolean;
}>((patch) => { }>((patch) => {
const id = currentAssistantId;
setMessages((prev) => setMessages((prev) =>
prev.map((msg) => (msg.id === assistantId ? { ...msg, ...patch } : msg)), prev.map((msg) => (msg.id === id ? { ...msg, ...patch } : msg)),
); );
}); });
@@ -814,6 +900,7 @@ export function useClinicalChat({
tracksThought?: boolean; tracksThought?: boolean;
}, },
) => { ) => {
const id = currentAssistantId;
const hasStreamText = const hasStreamText =
patch.thoughtContent !== undefined || patch.content !== undefined; patch.thoughtContent !== undefined || patch.content !== undefined;
@@ -822,13 +909,13 @@ export function useClinicalChat({
if (patch.thoughtContent !== undefined) { if (patch.thoughtContent !== undefined) {
domHandled = domHandled =
setClinicalStreamText( setClinicalStreamText(
streamTargetKey(assistantId, 'thought'), streamTargetKey(id, 'thought'),
patch.thoughtContent, patch.thoughtContent,
) && domHandled; ) && domHandled;
} }
if (patch.content !== undefined) { if (patch.content !== undefined) {
domHandled = domHandled =
setClinicalStreamText(streamTargetKey(assistantId, 'answer'), patch.content) && setClinicalStreamText(streamTargetKey(id, 'answer'), patch.content) &&
domHandled; domHandled;
} }
@@ -840,7 +927,7 @@ export function useClinicalChat({
if (needsReact) { if (needsReact) {
flushSync(() => { flushSync(() => {
setMessages((prev) => setMessages((prev) =>
prev.map((msg) => (msg.id === assistantId ? { ...msg, ...patch } : msg)), prev.map((msg) => (msg.id === id ? { ...msg, ...patch } : msg)),
); );
}); });
} }
@@ -861,6 +948,45 @@ export function useClinicalChat({
} }
}; };
// Freeze the bubble being streamed using its OWN parser/accumulator, so its
// reasoning + answer slice stay self-contained.
const finalizeCurrentBubble = () => {
if (thoughtParser) {
const snapshot = thoughtParser.finalize();
updateMessage(currentAssistantId, {
content: snapshot.content,
thoughtContent: snapshot.thoughtContent || undefined,
tracksThought: true,
thoughtComplete: true,
streaming: false,
pondering: false,
});
} else {
updateMessage(currentAssistantId, {
content: plainContentAccumulator,
streaming: false,
pondering: false,
});
}
clearClinicalStreamTargetsForMessage(currentAssistantId);
};
// Continuation → close the current bubble and open a fresh one for the next segment.
const startNextSegmentBubble = () => {
finalizeCurrentBubble();
if (!useImperativeStreamPaint) {
edgeStreamRaf.cancel();
}
const nextId = createChatMessageId();
currentAssistantId = nextId;
segmentCount += 1;
thoughtParser =
thoughtActive && !useOllamaThoughtStream ? createCotStreamParser() : null;
plainContentAccumulator = '';
thoughtCompleteEmitted = false;
setMessages((prev) => [...prev, spawnAssistantBubble(nextId, false)]);
};
try { try {
const runTurn = () => const runTurn = () =>
runClinicalChatTurn( runClinicalChatTurn(
@@ -884,6 +1010,13 @@ export function useClinicalChat({
}, },
(event: ClinicalChatStreamEvent) => { (event: ClinicalChatStreamEvent) => {
try { try {
if (event.type === 'segment_boundary') {
// segment 1 is the bubble we already opened; only continuations spawn a new one.
if (event.segment >= 2 && !useOllamaThoughtStream) {
startNextSegmentBubble();
}
return;
}
if (event.type === 'thought_token') { if (event.type === 'thought_token') {
if (!event.partial || !useOllamaThoughtStream) { if (!event.partial || !useOllamaThoughtStream) {
return; return;
@@ -955,15 +1088,15 @@ export function useClinicalChat({
if (abortController.signal.aborted) { if (abortController.signal.aborted) {
disposeStreamThrottle(); disposeStreamThrottle();
updateMessage(assistantId, { streaming: false }); updateMessage(currentAssistantId, { streaming: false });
return; return;
} }
disposeStreamThrottle(); disposeStreamThrottle();
clearClinicalStreamTargetsForMessage(assistantId); clearClinicalStreamTargetsForMessage(currentAssistantId);
if (useOllamaThoughtStream) { if (useOllamaThoughtStream) {
updateMessage(assistantId, { updateMessage(currentAssistantId, {
content: ollamaAnswerAcc || result.finalAnswer, content: ollamaAnswerAcc || result.finalAnswer,
thoughtContent: ollamaThoughtAcc || undefined, thoughtContent: ollamaThoughtAcc || undefined,
tracksThought: true, tracksThought: true,
@@ -973,8 +1106,11 @@ export function useClinicalChat({
}); });
} else if (thoughtActive && thoughtParser) { } else if (thoughtActive && thoughtParser) {
const snapshot = thoughtParser.finalize(); const snapshot = thoughtParser.finalize();
const finalContent = snapshot.content || result.finalAnswer; // Only the sole segment may borrow the merged finalAnswer; continuation
updateMessage(assistantId, { // bubbles must keep just their own parsed slice to avoid duplication.
const finalContent =
snapshot.content || (segmentCount === 1 ? result.finalAnswer : '');
updateMessage(currentAssistantId, {
content: finalContent, content: finalContent,
thoughtContent: snapshot.thoughtContent || undefined, thoughtContent: snapshot.thoughtContent || undefined,
tracksThought: true, tracksThought: true,
@@ -983,8 +1119,9 @@ export function useClinicalChat({
pondering: false, pondering: false,
}); });
} else { } else {
updateMessage(assistantId, { updateMessage(currentAssistantId, {
content: result.finalAnswer, content:
plainContentAccumulator || (segmentCount === 1 ? result.finalAnswer : ''),
streaming: false, streaming: false,
pondering: false, pondering: false,
}); });
@@ -999,12 +1136,12 @@ export function useClinicalChat({
); );
} catch (error) { } catch (error) {
disposeStreamThrottle(); disposeStreamThrottle();
clearClinicalStreamTargetsForMessage(assistantId); clearClinicalStreamTargetsForMessage(currentAssistantId);
if (error instanceof DOMException && error.name === 'AbortError') { if (error instanceof DOMException && error.name === 'AbortError') {
updateMessage(assistantId, { streaming: false, pondering: false }); updateMessage(currentAssistantId, { streaming: false, pondering: false });
return; return;
} }
updateMessage(assistantId, { updateMessage(currentAssistantId, {
content: content:
error instanceof Error error instanceof Error
? `Không thể trả lời: ${error.message}` ? `Không thể trả lời: ${error.message}`
@@ -1132,6 +1269,7 @@ export function useClinicalChat({
modelLoadPhase, modelLoadPhase,
modelLoadProgress, modelLoadProgress,
modelInstallTransferLabel, modelInstallTransferLabel,
modelLoadStalled,
modelLoadFading, modelLoadFading,
sendMessage, sendMessage,
stopGeneration, stopGeneration,

View File

@@ -12,7 +12,10 @@ import {
normalizeBackendSeverity, normalizeBackendSeverity,
type SynovitisSeverityResult, type SynovitisSeverityResult,
} from '../data/synovitisSeverity'; } from '../data/synovitisSeverity';
import { getCachedCvAnalyzeResult, getCvAnalyzeResultsForProfile } from '../lib/cvAnalyzeApi'; import { getCachedCvAnalyzeResult, getCvAnalyzeResultsForProfile, getCvAnalyzeResultsForProfileCelery, clearCvAnalyzeResultCache, type CvFrameAnalyzeResult } from '../lib/cvAnalyzeApi';
import { clearSegmentationResultCache } from '../lib/segmentationApi';
import { clearAngleClassificationResultCache } from '../lib/angleClassificationApi';
import { clearMlInferenceCacheForPatient } from '../lib/mlInferenceCacheStore';
import type { ProfileMlContext } from '../lib/mlInferenceCacheKeys'; import type { ProfileMlContext } from '../lib/mlInferenceCacheKeys';
import type { ScanFrame } from '../data/scanFrames'; import type { ScanFrame } from '../data/scanFrames';
import { interpretSegmentationForDisplay } from '../lib/interpretSegmentationResult'; import { interpretSegmentationForDisplay } from '../lib/interpretSegmentationResult';
@@ -104,6 +107,7 @@ export function useSegmentationOverlay(
imageSrc: string, imageSrc: string,
profileFrames?: ScanFrame[], profileFrames?: ScanFrame[],
patientMrn?: string, patientMrn?: string,
useCelery?: boolean,
): UseSegmentationOverlayResult { ): UseSegmentationOverlayResult {
const [overlaySrc, setOverlaySrc] = useState<string | null>(null); const [overlaySrc, setOverlaySrc] = useState<string | null>(null);
const [interpretation, setInterpretation] = useState<SegmentationDisplayInterpretation | null>(null); const [interpretation, setInterpretation] = useState<SegmentationDisplayInterpretation | null>(null);
@@ -117,7 +121,15 @@ export function useSegmentationOverlay(
const [source, setSource] = useState<'backend' | null>(null); const [source, setSource] = useState<'backend' | null>(null);
const [retryNonce, setRetryNonce] = useState(0); const [retryNonce, setRetryNonce] = useState(0);
const retry = () => setRetryNonce((n) => n + 1); const retry = () => {
clearCvAnalyzeResultCache();
clearSegmentationResultCache();
clearAngleClassificationResultCache();
if (patientMrn) {
clearMlInferenceCacheForPatient(patientMrn);
}
setRetryNonce((n) => n + 1);
};
const latestSegmentationRef = useRef<SegmentationApiResult | undefined>(undefined); const latestSegmentationRef = useRef<SegmentationApiResult | undefined>(undefined);
@@ -161,7 +173,14 @@ export function useSegmentationOverlay(
const mlContext: ProfileMlContext | undefined = patientMrn const mlContext: ProfileMlContext | undefined = patientMrn
? { patientMrn } ? { patientMrn }
: undefined; : undefined;
const results = await getCvAnalyzeResultsForProfile(frameRefs, mlContext);
let results: Map<string, CvFrameAnalyzeResult>;
if (useCelery) {
results = await getCvAnalyzeResultsForProfileCelery(frameRefs, mlContext);
} else {
results = await getCvAnalyzeResultsForProfile(frameRefs, mlContext);
}
if (cancelled) return; if (cancelled) return;
const cvResult = results.get(frameId); const cvResult = results.get(frameId);
@@ -235,7 +254,7 @@ export function useSegmentationOverlay(
return () => { return () => {
cancelled = true; cancelled = true;
}; };
}, [frameId, imageSrc, profileFrames, patientMrn, retryNonce]); }, [frameId, imageSrc, profileFrames, patientMrn, retryNonce, useCelery]);
return { return {
overlaySrc, overlaySrc,

View File

@@ -31,6 +31,26 @@ export interface BackendCvAnalyzeBatchResponse {
detail?: string; detail?: string;
} }
export interface BackendCvCelerySubmitResponse {
success: boolean;
job_id: string;
image_count: number;
mode: string;
chunk_size: number;
detail?: string;
}
export interface BackendCvCeleryStatusResponse {
status: 'pending' | 'completed' | 'failed' | 'unknown';
completed?: number;
total?: number;
progress?: number;
image_count?: number;
results?: BackendSegmentationResponse[];
errors?: string[];
detail?: string;
}
export interface CvFrameAnalyzeResult { export interface CvFrameAnalyzeResult {
raw: BackendSegmentationResponse; raw: BackendSegmentationResponse;
segmentation: SegmentationApiResult; segmentation: SegmentationApiResult;
@@ -222,3 +242,166 @@ export async function getCvAnalyzeResultsForProfile(
} }
return resolved; return resolved;
} }
/**
* Submit CV batch for async Celery chunk fan-out.
* Returns job_id immediately — poll with pollCvAnalyzeBatchCelery().
*/
export async function submitCvAnalyzeBatchCelery(
frames: ProfileFrameRef[],
apiBase = getSegmentApiBase(),
): Promise<string> {
if (frames.length === 0) {
throw new Error('frames must not be empty');
}
const formData = new FormData();
const files = await Promise.all(
frames.map((frame, index) =>
imageUrlToFile(frame.src, `${frame.id || `frame-${index}`}.png`),
),
);
files.forEach((file) => formData.append('images', file));
formData.append('frame_ids', JSON.stringify(frames.map((frame) => frame.id)));
const response = await fetch(`${apiBase}/api/test/analyze/batch/celery`, {
method: 'POST',
body: formData,
});
const payload = await readMlApiJson<BackendCvCelerySubmitResponse>(response);
if (!response.ok) {
throw new Error(`Celery batch submit error (${response.status})`);
}
if (!payload.success || !payload.job_id) {
throw new Error('Celery batch submit returned success=false or missing job_id');
}
return payload.job_id;
}
/**
* Poll Celery batch job status.
* Returns status object — call again if status === 'pending'.
*/
export async function pollCvAnalyzeBatchCelery(
jobId: string,
apiBase = getSegmentApiBase(),
): Promise<BackendCvCeleryStatusResponse> {
const response = await fetch(`${apiBase}/api/test/analyze/batch/celery/${encodeURIComponent(jobId)}`);
const payload = await readMlApiJson<BackendCvCeleryStatusResponse>(response);
if (!response.ok) {
throw new Error(payload.detail ?? `Celery batch poll error (${response.status})`);
}
return payload;
}
/**
* Async CV pipeline via Celery chunk fan-out.
* Submits job, polls until completed/failed, maps results into cache.
* Returns cached results immediately if all frames are already available.
*/
export async function getCvAnalyzeResultsForProfileCelery(
frames: ProfileFrameRef[],
mlContext?: ProfileMlContext,
apiBase = getSegmentApiBase(),
signal?: AbortSignal,
): Promise<Map<string, CvFrameAnalyzeResult>> {
if (frames.length === 0) {
return new Map();
}
const cacheKey = buildBatchCacheKey(frames.map((f) => f.id));
// Return fully cached results without hitting the backend.
const cached = new Map<string, CvFrameAnalyzeResult>();
for (const frame of frames) {
const hit = cvAnalyzeResultCache.get(frame.id);
if (hit) {
cached.set(frame.id, hit);
}
}
if (cached.size === frames.length) {
return cached;
}
// Coalesce in-flight requests for the same batch of frames.
const existing = inflightBatchByKey.get(cacheKey);
if (existing) {
return existing;
}
const batchPromise = (async (): Promise<Map<string, CvFrameAnalyzeResult>> => {
const jobId = await submitCvAnalyzeBatchCelery(frames, apiBase);
const pollIntervalMs = 2000;
const submitTime = Date.now();
while (true) {
if (signal?.aborted) {
throw new Error('Celery batch poll aborted');
}
const status = await pollCvAnalyzeBatchCelery(jobId, apiBase);
if (status.status === 'completed') {
const byFrameId = new Map<string, CvFrameAnalyzeResult>();
if (status.results) {
for (const item of status.results) {
const frameId = item.frame_id;
if (!frameId) continue;
const cvResult = mapPayloadToCvResult(item);
byFrameId.set(frameId, cvResult);
cvAnalyzeResultCache.set(frameId, cvResult);
segmentationResultCache.set(frameId, cvResult.segmentation);
if (cvResult.angle) {
angleResultCache.set(frameId, cvResult.angle);
}
}
}
if (mlContext?.patientMrn && status.results) {
await persistCvBatch(
frames,
new Map([...byFrameId.entries()]),
mlContext,
);
}
const totalMs = Date.now() - submitTime;
console.log(
`[cvAnalyzeApi] Celery batch completed jobId=${jobId} frames=${frames.length} totalMs=${totalMs}`,
);
return byFrameId;
}
if (status.status === 'failed') {
const totalMs = Date.now() - submitTime;
console.error(
`[cvAnalyzeApi] Celery batch failed jobId=${jobId} frames=${frames.length} totalMs=${totalMs}`,
);
throw new Error(status.errors?.join('; ') ?? 'Celery batch job failed');
}
if (status.status === 'unknown') {
const totalMs = Date.now() - submitTime;
console.error(
`[cvAnalyzeApi] Celery batch unknown jobId=${jobId} frames=${frames.length} totalMs=${totalMs}`,
);
throw new Error(status.detail ?? `Celery job ${jobId} not found`);
}
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs));
}
})();
inflightBatchByKey.set(cacheKey, batchPromise);
try {
return await batchPromise;
} finally {
inflightBatchByKey.delete(cacheKey);
}
}

View File

@@ -24,12 +24,12 @@ function envFlag(name: string, defaultValue: boolean): boolean {
/** Try local Gemma worker when OPFS model is available. Falls back to mock replies otherwise. */ /** Try local Gemma worker when OPFS model is available. Falls back to mock replies otherwise. */
export function useLocalLlmWhenAvailable(): boolean { export function useLocalLlmWhenAvailable(): boolean {
return envFlag('VITE_CLINICAL_CHAT_USE_LLM', true); return import.meta.env.VITE_CLINICAL_CHAT_USE_LLM !== 'false';
} }
/** Use fixture tool results instead of BFF (default on in dev). */ /** Use fixture tool results instead of BFF (default on in dev). */
export function useMockAgentTools(): boolean { export function useMockAgentTools(): boolean {
return envFlag('VITE_CLINICAL_CHAT_MOCK_TOOLS', true); return import.meta.env.VITE_CLINICAL_CHAT_MOCK_TOOLS !== 'false';
} }
export function clinicalChatBffBaseUrl(): string { export function clinicalChatBffBaseUrl(): string {

View File

@@ -122,6 +122,7 @@ export class LlmWorkerClient {
promptOptions: PromptOptions, promptOptions: PromptOptions,
decode: DecodeParams, decode: DecodeParams,
onToken?: (partial: string) => void, onToken?: (partial: string) => void,
onSegmentStart?: (segment: number) => void,
): Promise<{ rawOutput: string; stats: GenerationStats }> { ): Promise<{ rawOutput: string; stats: GenerationStats }> {
const id = requestId(); const id = requestId();
this.activeGenerateRequestId = id; this.activeGenerateRequestId = id;
@@ -136,6 +137,7 @@ export class LlmWorkerClient {
}, },
reject, reject,
onToken: (partial) => onToken?.(partial), onToken: (partial) => onToken?.(partial),
onSegmentStart: (segment) => onSegmentStart?.(segment),
}); });
this.worker.postMessage({ this.worker.postMessage({
type: 'generate', type: 'generate',

View File

@@ -16,19 +16,23 @@ export function formatInstallTransferLabel(progress: DownloadProgress): string {
export function mapDownloadProgress(progress: DownloadProgress): number { export function mapDownloadProgress(progress: DownloadProgress): number {
switch (progress.phase) { switch (progress.phase) {
case 'downloading': case 'downloading':
case 'resuming': case 'resuming': {
if (!progress.bytesTotal || progress.bytesTotal <= 0) { if (!progress.bytesTotal || progress.bytesTotal <= 0) {
return progress.phase === 'resuming' ? 14 : 12; return progress.phase === 'resuming' ? 6 : 4;
}
// Track the real byte fraction so the bar matches the "X GB / Y GB" label,
// reserving the last few percent for verify/write before completion.
const fraction = Math.min(1, progress.bytesLoaded / progress.bytesTotal);
return 4 + Math.round(fraction * 92);
} }
return 10 + Math.round((progress.bytesLoaded / progress.bytesTotal) * 58);
case 'hashing': case 'hashing':
return 72; return 97;
case 'writing': case 'writing':
return 76; return 98;
case 'done': case 'done':
return 78; return 99;
default: default:
return 10; return 4;
} }
} }

View File

@@ -66,12 +66,26 @@ interface DownloadProbe {
supportsRange: boolean; supportsRange: boolean;
} }
/** HEAD/range probes should return fast; time out so a hung socket surfaces as a retriable error. */
const PROBE_TIMEOUT_MS = 12_000;
function probeTimeoutSignal(): AbortSignal | undefined {
return typeof AbortSignal !== 'undefined' && typeof AbortSignal.timeout === 'function'
? AbortSignal.timeout(PROBE_TIMEOUT_MS)
: undefined;
}
async function probeModelDownloadUrl(url: string): Promise<DownloadProbe> { async function probeModelDownloadUrl(url: string): Promise<DownloadProbe> {
let response = await fetch(url, { method: 'HEAD', redirect: 'follow' }); let response = await fetch(url, {
method: 'HEAD',
redirect: 'follow',
signal: probeTimeoutSignal(),
});
if (!response.ok) { if (!response.ok) {
response = await fetch(url, { response = await fetch(url, {
headers: { Range: 'bytes=0-0' }, headers: { Range: 'bytes=0-0' },
redirect: 'follow', redirect: 'follow',
signal: probeTimeoutSignal(),
}); });
} }
@@ -164,7 +178,7 @@ function isRetriableDownloadError(error: unknown): boolean {
if (error instanceof TypeError) { if (error instanceof TypeError) {
return true; return true;
} }
if (error instanceof DOMException && error.name === 'NetworkError') { if (error instanceof DOMException && (error.name === 'NetworkError' || error.name === 'TimeoutError')) {
return true; return true;
} }
const message = (error instanceof Error ? error.message : String(error)).toLowerCase(); const message = (error instanceof Error ? error.message : String(error)).toLowerCase();
@@ -175,6 +189,8 @@ function isRetriableDownloadError(error: unknown): boolean {
message.includes('failed to fetch') || message.includes('failed to fetch') ||
message.includes('load failed') || message.includes('load failed') ||
message.includes('interrupted') || message.includes('interrupted') ||
message.includes('timed out') ||
message.includes('timeout') ||
message.includes('gián đoạn') || message.includes('gián đoạn') ||
message.includes('http 502') || message.includes('http 502') ||
message.includes('http 503') || message.includes('http 503') ||
@@ -328,6 +344,17 @@ export async function checkOpfsModelLoadable(): Promise<OpfsModelLoadableStatus>
const file = await readOpfsModelFile(MODEL_TASK_FILENAME); const file = await readOpfsModelFile(MODEL_TASK_FILENAME);
if (file && file.size > 0 && (await isReadable(file))) { if (file && file.size > 0 && (await isReadable(file))) {
// An interrupted download never wrote a manifest (it is written last). Such a
// file can still clear the >500 MB header check while being a truncated,
// unloadable checkpoint — treat it as resumable, not loadable, so we resume the
// download instead of trying to init MediaPipe on a fragmented model.
if (file.size < EXPECTED_MODEL_TASK_BYTES) {
return invalidStatus(
`Partial download in OPFS (${formatBytes(file.size)} of ${formatBytes(EXPECTED_MODEL_TASK_BYTES)}). Will resume on next install.`,
manifest,
file,
);
}
const validationError = await validateTaskCandidate(file); const validationError = await validateTaskCandidate(file);
if (validationError) { if (validationError) {
return invalidStatus(validationError, manifest, file); return invalidStatus(validationError, manifest, file);

View File

@@ -30,7 +30,9 @@ export interface DirectChatTurnResult {
export type ClinicalChatStreamEvent = export type ClinicalChatStreamEvent =
| AgentEvent | AgentEvent
| { type: 'thought_token'; partial: string }; | { type: 'thought_token'; partial: string }
/** A new local-model generation segment started (continuation) → spawn a new bubble. */
| { type: 'segment_boundary'; segment: number };
export interface RunClinicalChatTurnInput { export interface RunClinicalChatTurnInput {
inferenceMode: InferenceMode; inferenceMode: InferenceMode;
@@ -56,10 +58,17 @@ function buildChatHistory(
if (message.streaming || !message.content.trim()) { if (message.streaming || !message.content.trim()) {
continue; continue;
} }
if (message.role === 'user') { if (message.role !== 'user' && message.role !== 'assistant') {
turns.push({ role: 'user', text: message.content }); continue;
} else if (message.role === 'assistant') { }
turns.push({ role: 'assistant', text: message.content }); const role: GemmaHistoryTurn['role'] = message.role === 'user' ? 'user' : 'assistant';
// Continuation segments render as separate assistant bubbles; coalesce consecutive
// same-role turns so Gemma sees one well-formed alternating turn.
const previous = turns[turns.length - 1];
if (previous && previous.role === role) {
previous.text = `${previous.text}${message.content}`;
} else {
turns.push({ role, text: message.content });
} }
} }
return turns.slice(-maxHistoryTurns * 2); return turns.slice(-maxHistoryTurns * 2);
@@ -126,6 +135,7 @@ export async function runDirectChatTurn(
historyMessages: ClinicalChatMessage[]; historyMessages: ClinicalChatMessage[];
onToken?: (partial: string) => void; onToken?: (partial: string) => void;
onThoughtToken?: (partial: string) => void; onThoughtToken?: (partial: string) => void;
onSegmentStart?: (segment: number) => void;
}, },
signal?: AbortSignal, signal?: AbortSignal,
): Promise<DirectChatTurnResult> { ): Promise<DirectChatTurnResult> {
@@ -206,6 +216,7 @@ export async function runDirectChatTurn(
promptOptions, promptOptions,
decode, decode,
input.onToken, input.onToken,
input.onSegmentStart,
); );
if (signal?.aborted) { if (signal?.aborted) {
@@ -252,6 +263,7 @@ export async function runClinicalChatTurn(
historyMessages: input.historyMessages, historyMessages: input.historyMessages,
onToken: (partial) => onEvent?.({ type: 'final_token', partial }), onToken: (partial) => onEvent?.({ type: 'final_token', partial }),
onThoughtToken: (partial) => onEvent?.({ type: 'thought_token', partial }), onThoughtToken: (partial) => onEvent?.({ type: 'thought_token', partial }),
onSegmentStart: (segment) => onEvent?.({ type: 'segment_boundary', segment }),
}, },
signal, signal,
); );

View File

@@ -227,6 +227,7 @@ function ClinicalWorkspaceContent({
onSegmentationLoadingChange={setIsSegmentationLoading} onSegmentationLoadingChange={setIsSegmentationLoading}
patientMrn={patient.mrn} patientMrn={patient.mrn}
patientId={patient.id} patientId={patient.id}
useCelery={import.meta.env.VITE_USE_CV_CELERY !== 'false'}
onRegisterSnapshotCapture={(capture) => { onRegisterSnapshotCapture={(capture) => {
captureSnapshotRef.current = capture; captureSnapshotRef.current = capture;
}} }}

View File

@@ -32,7 +32,9 @@ interface ImportMetaEnv {
readonly VITE_OLLAMA_CHAT_URL?: string; readonly VITE_OLLAMA_CHAT_URL?: string;
readonly VITE_OLLAMA_MODEL?: string; readonly VITE_OLLAMA_MODEL?: string;
readonly VITE_USE_BACKEND_SEGMENTATION?: string; readonly VITE_USE_BACKEND_SEGMENTATION?: string;
readonly VITE_USE_CV_CELERY?: string;
readonly VITE_SEGMENT_API_BASE?: string; readonly VITE_SEGMENT_API_BASE?: string;
readonly VITE_MODAL_OLLAMA_TARGET?: string;
} }
interface ImportMeta { interface ImportMeta {

View File

@@ -338,16 +338,16 @@ async function generateResponse(
} }
const baseBeforeSegment = combinedOutput; const baseBeforeSegment = combinedOutput;
let emittedLength = combinedOutput.length;
const tokenBatcher = createTokenBatcher(requestId, segmentNumber); const tokenBatcher = createTokenBatcher(requestId, segmentNumber);
let emittedSegmentLength = 0;
// Stream each segment's RAW tokens (its own thought channel + answer)
// independently. The client renders one bubble per segment and parses its own
// channel, so continuation thinking never leaks into a prior answer. The
// cross-segment merge below is only for the model's continuation prompt + stats.
const segmentRaw = await streamSegment(prompt, requestId, (_partial, segmentSoFar) => { const segmentRaw = await streamSegment(prompt, requestId, (_partial, segmentSoFar) => {
const merged = const delta = segmentSoFar.slice(emittedSegmentLength);
segments === 0 emittedSegmentLength = segmentSoFar.length;
? segmentSoFar
: mergeContinuationOutput(baseBeforeSegment, segmentSoFar, chainOfThought);
const delta = merged.slice(emittedLength);
emittedLength = merged.length;
if (delta.length > 0) { if (delta.length > 0) {
tokenBatcher.push(delta); tokenBatcher.push(delta);
} }

View File

@@ -15,7 +15,8 @@
"noUnusedLocals": true, "noUnusedLocals": true,
"noUnusedParameters": true, "noUnusedParameters": true,
"noFallthroughCasesInSwitch": true, "noFallthroughCasesInSwitch": true,
"noUncheckedSideEffectImports": true "noUncheckedSideEffectImports": true,
"types": ["node"]
}, },
"include": ["src"] "include": ["src", "vite.config.ts"]
} }

File diff suppressed because one or more lines are too long

View File

@@ -1,8 +1,27 @@
import { defineConfig } from 'vite'; import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react'; import react from '@vitejs/plugin-react';
import path from 'node:path'; import path from 'node:path';
import fs from 'fs';
import { load } from 'js-yaml';
function loadFrontendConfig(): Record<string, string> {
try {
const raw = fs.readFileSync('config/frontend.config.yaml', 'utf8');
return load(raw) as Record<string, string>;
} catch {
return {};
}
}
const frontendConfig = loadFrontendConfig();
const defineVars: Record<string, string> = {};
for (const [key, value] of Object.entries(frontendConfig)) {
defineVars[`import.meta.env.${key}`] = JSON.stringify(String(value));
}
const MODAL_OLLAMA_TARGET = const MODAL_OLLAMA_TARGET =
frontendConfig.VITE_MODAL_OLLAMA_TARGET ??
'https://dtj-tran--ollama-gemma4-e4b-ollamaserver-web.modal.run'; 'https://dtj-tran--ollama-gemma4-e4b-ollamaserver-web.modal.run';
export default defineConfig({ export default defineConfig({
@@ -52,4 +71,5 @@ export default defineConfig({
}, },
}, },
}, },
define: defineVars,
}); });

View File

@@ -1,31 +0,0 @@
# Install Docker image
docker network create jenkins
# install docker-integratable image
docker run --name jenkins-docker --rm --detach \
-p 8080:8080 -p 50000:50000 \
--restart=on-failure \
--privileged --network jenkins --network-alias docker \
--env DOCKER_TLS_CERTDIR=/certs \
--volume jenkins-docker-certs:/certs/client \
--volume jenkins-data:/var/jenkins_home \
--publish 2376:2376 \
docker:dind --storage-driver overlay2 \
-v jenkins_home:/var/jenkins_home jenkins/jenkins:lts
# install the docker images
docker run \
--name jenkins-blueocean \
--restart=on-failure \
--detach \
--network jenkins \
--env DOCKER_HOST=tcp://docker:2376 \
--env DOCKER_CERT_PATH=/certs/client \
--env DOCKER_TLS_VERIFY=1 \
--publish 8080:8080 \
--publish 50000:50000 \
--volume jenkins-data:/var/jenkins_home \
--volume jenkins-docker-certs:/certs/client:ro \
jenkins/jenkins:lts

View File

@@ -0,0 +1,42 @@
### 1. The Core Application Stack (Replacing the streaming engine)
* **Database:** **PostgreSQL** or **SQLite**
* *Why:* Extremely lightweight, universally supported, and standard for medical/relational data (like patient exercises, joint ranges of motion, and logs).
* **Backend:** **Node.js (Express)**, **Python (FastAPI)**, or **Go**
* *Why:* Fast to build for a PoC, easily containerized via Docker, and highly efficient. FastAPI is excellent if you plan to incorporate any musculoskeletal data analysis or ML later.
### 2. CI/CD & Infrastructure Components
* **Codebase:** **Gitea**
* *Why:* A lightweight, self-hosted Git platform that runs in a simple local Docker container using very little memory.
* **CI/CD Orchestrator & Build Server:** **Woodpecker CI** or **GitHub Actions** (if you're okay using GitHub until migrating to a private cloud).
* *Why:* Woodpecker is an open-source, container-first CI engine that runs locally with almost zero overhead.
* **Artifact Repository:** **Gitea Packages** or **Docker Registry**
* *Why:* You can store your built application images right inside Gitea or a tiny local Docker registry container.
### 3. Environments & Deployment
* **Deployment-Manager:** **Docker Compose** or **Portainer**
* *Why:* To deploy your musculoskeletal app, you just need Docker Compose to orchestrate your backend and database containers. Portainer gives you a simple web GUI to manage them locally.
* **Staging & Production Env:** Isolated local containers or separate virtual machines.
### 4. Feedback & Monitoring Loop
* **User Feedback Collector:** A custom-built form widget inside your app or an open-source tool like **Feedback Fish** / **Air Table**
* **Feedback-Resolve:** **Focalboard** or **Leantime** (Self-hosted, lightweight project boards to track bugs and user feature requests).
* **Monitoring & Logging:** **Prometheus + Grafana**
* *Why:* Perfect for tracking application uptime, API response times, and server health.

View File

@@ -0,0 +1,49 @@
```planUML
@startuml C4_Elements
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml
title Component Diagram for CI/CD Pipeline Architecture
Person(developer, "Developer(s)", "Writes code and pushes changes.")
Person(enduser, "End-User(s)", "Interacts with the production application.")
Container_Boundary(pipeline_boundary, "CI/CD Pipeline System") {
Component(codebase, "Codebase", "Git Repository", "Stores the source code and tracks version history.")
Component(orchestrator, "CI/CD Orchestrator", "Workflow Engine", "Triggers actions based on repository events.")
Component(build_server, "Build Server", "Compiler/Packager", "Compiles code and builds release packages.")
Component(artifact_repo, "Artifact Repository", "Storage", "Stores compiled binaries or container images.")
Component(test_pipeline, "Test-Pipeline", "Automation Suite", "Runs unit, integration, and security tests.")
Component(deploy_manager, "Deployment-Manager", "CD Engine", "Orchestrates deployment to various environments.")
Component(user_feedback, "User Feedback Collector", "In-App/Portal Widget", "Collects explicit feature requests and bug reports from users.")
Component(feedback_resolve, "Feedback-Resolve", "Tracking System", "Manages issues, bugs, and deployment logs.")
Component(monitoring, "Monitoring & Logging", "Observability", "Monitors application health and metrics.")
Container_Boundary(deploy_env, "Deployment Environments") {
Component(staging_env, "Staging/QA Env", "Environment", "Pre-production environment for testing.")
Component(prod_env, "Production Env", "Environment", "Live environment hosting the customer application.")
}
}
' Directional Flow Links
Rel(developer, codebase, "1. Pushes code")
Rel(codebase, orchestrator, "2. Triggers event webhook")
Rel(orchestrator, build_server, "3. Triggers build job")
Rel(build_server, artifact_repo, "4. Stores compiled artifact")
Rel(orchestrator, test_pipeline, "5. Runs automated tests")
Rel(orchestrator, deploy_manager, "6. Signals ready for deployment")
Rel(deploy_manager, artifact_repo, "7. Pulls latest artifact")
Rel(deploy_manager, staging_env, "8. Deploys to")
Rel(deploy_manager, prod_env, "9. Promotes approved changes to")
Rel(enduser, prod_env, "10. Interacts with application")
Rel(enduser, user_feedback, "11. Submits requests & feedback")
Rel(user_feedback, feedback_resolve, "12. Forwards user tickets")
Rel(monitoring, prod_env, "13. Reads metrics and errors")
Rel(monitoring, feedback_resolve, "14. Feeds performance/error logs")
Rel(feedback_resolve, developer, "15. Alerts for loop closure")
@enduml
```

View File

@@ -205,6 +205,9 @@ async def forward_list_models():
min_containers=1, # for keeping warm and prevention, min_containers=1, # for keeping warm and prevention,
buffer_containers=2, # Number of additional idle containers to maintain under active load. buffer_containers=2, # Number of additional idle containers to maintain under active load.
scaledown_window=30, # Max time (in seconds) a container can remain idle while scaling down. scaledown_window=30, # Max time (in seconds) a container can remain idle while scaling down.
volumes= {
'/mnt/vkist-ml-model' : modal.CloudBucketMount(bucket_name="vkist-ml-model", secret=modal.Secret.from_name("aws-secrets"))
},
secrets=[modal.Secret.from_name("aws-secrets")] secrets=[modal.Secret.from_name("aws-secrets")]
) )
@modal.asgi_app() @modal.asgi_app()
@@ -213,7 +216,11 @@ def unified_triton_server():
# Spawns Triton in the background. It will automatically read # Spawns Triton in the background. It will automatically read
# your "aws-secrets" environment keys to mount s3://vkist-ml-model/ # your "aws-secrets" environment keys to mount s3://vkist-ml-model/
cmd = ["tritonserver", "--model-repository=s3://vkist-ml-model/"] # cmd = ["tritonserver", "--model-repository=s3://vkist-ml-model/"] # bad pattern causing latency
cmd = ["tritonserver", "--model-repository=/mnt/vkist-ml-model/"]
# triton issue network connection -> AWS -> adding cold-off latency
# idea mounting the model to Modal Volume
subprocess.Popen(cmd) subprocess.Popen(cmd)
print("📋 Triton background process delegated. Handing routing control over to FastAPI.") print("📋 Triton background process delegated. Handing routing control over to FastAPI.")

View File

@@ -1,2 +1,2 @@
cd ../PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/infra/implementation/triton_run cd ../PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/infra/implementation/triton_run
MODAL_PROFILE=dtj-tran modal deploy PILOT_PROJECT/workspace/sprint_1_2/codebase/infra/implementation/triton_run/modal_triton.py MODAL_PROFILE=dtj-tran modal deploy modal_triton.py

View File

@@ -0,0 +1,138 @@
import json
import time
from pathlib import Path
import numpy as np
from PIL import Image
import requests
# Server configuration
TRITON_URL = "https://dtj-tran--triton-s3-service-unified-triton-server.modal.run"
TARGET_MODEL = "workspace/sprint_1_2/CODEBASE/infra/tests/models/efficientnet_b0_ultrasound_2_class.pth" # Target a single sub-model directly
IMFLAMMATION_CLASSES = ["Không viêm", "Có viêm"]
BASE_DIR = Path(__file__).resolve().parent
def load_image(path: Path) -> Image.Image:
if not path.exists():
raise FileNotFoundError(f"Test image not found at: {path}")
return Image.open(path).convert("RGB")
def preprocess_224(img: Image.Image) -> np.ndarray:
"""Preprocesses image to NCHW FP32 [1, 3, 224, 224] matching ResNet50 input requirements"""
img_resized = img.resize((224, 224), Image.Resampling.BILINEAR)
arr = np.asarray(img_resized).astype(np.float32) / 255.0
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
std = np.array([0.229, 0.224, 0.225], dtype=np.float32)
arr = (arr - mean) / std
arr = arr.transpose(2, 0, 1) # HWC -> CHW
arr = np.expand_dims(arr, axis=0) # Add batch dim -> NCHW
return arr
def softmax(x: np.ndarray) -> np.ndarray:
e = np.exp(x - np.max(x))
return e / np.sum(e)
def build_binary_request(input_tensor: np.ndarray, model_name: str) -> tuple[bytes, dict]:
"""Constructs a strict KServe v2 binary payload for a single input tensor"""
# NOTE: Check your individual model's config.pbtxt to verify if the
# expected input name is 'input_image', 'input_0', etc.
input_name = "input_image"
inputs = [
{
"name": input_name,
"shape": list(input_tensor.shape),
"datatype": "FP32",
"parameters": {"binary_data_size": input_tensor.nbytes},
}
]
# We request the standard 'logits' output tensor from this model
outputs = [{"name": "logits"}]
metadata = {
"model_name": model_name,
"model_version": "",
"inputs": inputs,
"outputs": outputs
}
metadata_bytes = json.dumps(metadata).encode("utf-8")
body = metadata_bytes + input_tensor.tobytes()
headers = {
"Inference-Header-Content-Length": str(len(metadata_bytes)),
"Content-Type": "application/octet-stream"
}
return body, headers
def parse_binary_response(resp: requests.Response) -> np.ndarray:
"""Parses Triton's binary response stream for a single output tensor"""
header_length = int(resp.headers.get("Inference-Header-Content-Length", "0"))
response_bytes = resp.content
metadata = json.loads(response_bytes[:header_length].decode("utf-8"))
binary_data = response_bytes[header_length:]
# Extract the first available output tensor
output_desc = metadata["outputs"][0]
shape = output_desc["shape"]
# Unpack raw bytes back into numpy array
arr = np.frombuffer(binary_data, dtype=np.float32).reshape(shape)
return arr
def main():
# Define a path to a real local image to test with
image_path = "test_images/sup-up-long_positive/58e7a7ef-de3e-11ee-97e2-0a580a5f5b60_11.png"
print(f"Targeting Individual Model: {TARGET_MODEL}")
print(f"Loading image from: {image_path}")
try:
# 1. Prepare image
img = load_image(image_path)
input_data = preprocess_224(img)
# 2. Build the payload
body, headers = build_binary_request(input_data, TARGET_MODEL)
# 3. Fire request to the targeted model endpoint
url = f"{TRITON_URL}/v2/models/{TARGET_MODEL}/infer"
print("Sending request to Triton server...")
t0 = time.time()
resp = requests.post(url, data=body, headers=headers, timeout=30)
resp.raise_for_status()
latency = time.time() - t0
# 4. Parse output logits
logits = parse_binary_response(resp)
logits = np.squeeze(logits) # Drop batch dimension
# 5. Decode probabilities
probs = softmax(logits)
predicted_idx = int(np.argmax(probs))
print("\n" + "=" * 40)
print("🎉 SUCCESSFUL INFERENCE")
print(f"Network Roundtrip Time: {latency:.4f}s")
print("-" * 40)
print("Class Probabilities:")
for cls_name, prob in zip(IMFLAMMATION_CLASSES, probs):
print(f" {cls_name:<15}: {prob * 100:.2f}%")
print("-" * 40)
print(f"Prediction: {IMFLAMMATION_CLASSES[predicted_idx]} ({probs[predicted_idx] * 100:.2f}%)")
print("=" * 40)
except Exception as e:
print(f"\n❌ Inference Failed: {e}")
if 'resp' in locals() and hasattr(resp, 'text'):
print(f"Server Error Message: {resp.text}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,103 @@
# Automatically generated by https://github.com/damnever/pigar.
aiosqlite==0.22.1
docling==2.70.0
fastapi==0.135.1
fastembed==0.8.0
gliner==0.2.27
grpcio==1.81.1
httpx==0.28.1
ingestion==0.0.42
langchain==1.3.7
langchain-text-splitters==1.1.2
modal==1.5.0
numpy==2.1.3
opencv-python==4.13.0.92
pgvector==0.4.2
pillow==12.3.0
psycopg2-binary==2.9.12
pycocotools==2.0.11
pydantic==2.13.4
pymupdf==1.28.0
pypdf==6.14.2
pytest==9.0.3
python-dotenv==1.2.2
redis==8.0.1
removeOaExtension==0.0.6
requests==2.34.2
starlette==1.2.1
supabase==2.31.0
timm==1.0.25
torch==2.12.1
torchinfo==1.8.0
torchvision==0.27.1
transformers==4.57.6
tritonclient==2.69.0
uvicorn==0.41.0
# WARNING(pigar): some manual fixes might be required as pigar has detected duplicate requirements for the same import name (possibly for different submodules).
# WARNING(pigar): the following duplicate requirements are for the import name: backend
agentbeats==1.2.6
AlgoVision-Quant-Research==0.0.2
alles-apin==0.0.1
asamba==1.0.8
backend==0.2.4.1
backendcatraca-tektek==0.0.7
BackupBackup==0.0.6
cinder-ml==1.8.5
cmbagent==0.0.1.post63
conting-researcher==0.14.0
cwyod-base==0.0.2
deepread==0.0.3
devshare==1.0.0
django-nextcloud-storage==2024.5.20
doccano==1.8.5
doccano-multi-label==1.8.4.2
docforge-ai==2.0.3
equation-phase-portrait-tool==0.1.0
excellxgene==2.9.6
flaskreactapp==1.0.21
flet-devtools==0.0.0
food-fighters==1.0.1
Fred-Frechet==1.14.5
gpt-researcher==0.15.1
hermes-revision-system==0.1.0
hotel-spider==1.1.5
latch-excellxgene==1.0.0
leap-backend==0.1.0
lemon-auto-saver==1.0.1
LPP0D==0.1.1
mcp-feedback-pipe==3.0.15
mmsbm==1.0.7
nia-cli==0.1.1
nmuwd==0.10.3
omnbot==2017.5
personal-site-msilvasy==1.0.5
pipebio==5.1.0
prastut-ai==1.0.0
productiongraph==0.2.0
pyapptest==1.0.0
python-backend==0.0.1
pyWikiCMS==0.5.0
quantum-finance==0.2.0
qutritium==1.5.2
rasa-storyteller==0.2.1
reapply-workflows==0.1.0
recall-core==0.1.1
Researcher==0.4.3
reusing-intent==0.1.2
sparseprop==0.1.14
terminal-note==0.10.0
traiNER-loop==0.1.0
typernexrad-cli==0.1.0
viasp==2.3.0.post3
vibe-reader==0.2.1
work-with-database==0.0.1
x-lib==0.0.27
# WARNING(pigar): the following duplicate requirements are for the import name: langchain_google_vertexai
langchain-google-vertexai==3.2.4
# WARNING(pigar): the following duplicate requirements are for the import name: optimum
optimum==2.1.0
optimum-onnx==0.1.0
Celery==5.6.3
redis==8.0.1

Binary file not shown.

After

Width:  |  Height:  |  Size: 722 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB