Compare commits
8 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1f6b015815 | ||
|
|
1f5e71b46a | ||
|
|
3fbbca1eaa | ||
|
|
d862d1f7f8 | ||
|
|
2484597c7a | ||
|
|
41e6a2a416 | ||
|
|
41241489fb | ||
|
|
43f5c0f7da |
6
.gitignore
vendored
6
.gitignore
vendored
@@ -227,7 +227,6 @@ Policy_Analysis/
|
|||||||
package-lock.json
|
package-lock.json
|
||||||
package.json
|
package.json
|
||||||
.env.development
|
.env.development
|
||||||
dist/
|
|
||||||
run.sh
|
run.sh
|
||||||
workspace/sprint_1_2/CODEBASE/knowledge/implementation/model/
|
workspace/sprint_1_2/CODEBASE/knowledge/implementation/model/
|
||||||
*.onnx_data
|
*.onnx_data
|
||||||
@@ -236,3 +235,8 @@ workspace/sprint_1_2/CODEBASE/knowledge/implementation/model/
|
|||||||
*.safetensors
|
*.safetensors
|
||||||
*.pt
|
*.pt
|
||||||
*.pth
|
*.pth
|
||||||
|
*.pdf
|
||||||
|
*.xz
|
||||||
|
logs/
|
||||||
|
*.rdb
|
||||||
|
dist/
|
||||||
@@ -27,8 +27,6 @@ PILOT_PROJECT/
|
|||||||
│ │ ├── LEGACY # The legacy material - for archieved / pre-existed project
|
│ │ ├── LEGACY # The legacy material - for archieved / pre-existed project
|
||||||
│ │ ├── CODEBASE # The project codebase
|
│ │ ├── CODEBASE # The project codebase
|
||||||
│ │ ├── Design_Material/ # The system design - API - and UIX design material
|
│ │ ├── Design_Material/ # The system design - API - and UIX design material
|
||||||
│ │ ├── SOFTWARE_SYSTEM_DESIGN_FR_25.md # Detailed software design specification
|
|
||||||
│ │ ├── SOLUTION_ARCHITECTURE_SPEC.md # Solution architecture overview
|
|
||||||
│ │ └── VISUALIZATION/ # Charts, diagrams, and visual assets
|
│ │ └── VISUALIZATION/ # Charts, diagrams, and visual assets
|
||||||
│ └── ... # Additional sprint folders as needed
|
│ └── ... # Additional sprint folders as needed
|
||||||
├── secrets/ # **Developer‑managed secrets** (NOT tracked by git)
|
├── secrets/ # **Developer‑managed secrets** (NOT tracked by git)
|
||||||
|
|||||||
Binary file not shown.
654
proj_level_reading/PROPOSAL/VKIST_MSK_PILOT_PROPOSAL.md
Normal file
654
proj_level_reading/PROPOSAL/VKIST_MSK_PILOT_PROPOSAL.md
Normal file
@@ -0,0 +1,654 @@
|
|||||||
|
# VKIST MSK Ultrasound Stack — Investment Proposal & System Specification
|
||||||
|
|
||||||
|
> **Audience:** CTO / Department Head / Executive Sponsor (technical-fluent, outcome-driven, risk-averse on healthcare investments)
|
||||||
|
> **Author:** VKIST MSK Pilot Engineering Team
|
||||||
|
> **Status:** Pilot proposal — active cycle June 2 – September 2, 2026 (strict 3-month window)
|
||||||
|
> **Reading time:** Section 0 (Executive Summary) is a self-contained one-pager. Sections 1–8 provide the full technical and financial justification.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
0. [Executive Summary (one page)](#0-executive-summary-one-page)
|
||||||
|
1. [Project Context — Who, What Pain, What Constraints](#1-project-context)
|
||||||
|
2. [How the System Addresses the Problem](#2-how-the-system-addresses-the-problem)
|
||||||
|
3. [Qualitative ROI — Behavioral & Workflow Change](#3-qualitative-roi--behavioral--workflow-change)
|
||||||
|
4. [Quantitative ROI — Projected Metrics & Measurement Plan](#4-quantitative-roi--projected-metrics--measurement-plan)
|
||||||
|
5. [Risk & Failure-Mode Handling — Why This Is Not Risky](#5-risk--failure-mode-handling--why-this-is-not-risky)
|
||||||
|
6. [Compliance & Data Governance](#6-compliance--data-governance)
|
||||||
|
7. [Roadmap & Milestones](#7-roadmap--milestones)
|
||||||
|
8. [Investment Ask & Scope Boundaries](#8-investment-ask--scope-boundaries)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. Executive Summary (one page)
|
||||||
|
|
||||||
|
**The problem, in one sentence.** Vietnamese public-hospital musculoskeletal (MSK) care runs on clinicians processing 100+ evaluations per shift with no time for objective double-checks, no shared visibility across the radiologist → surgeon → physiotherapist → patient chain, and patients so information-starved that they turn to dangerous folk remedies — while VKIST's proven computer-vision research models sit idle as static academic checkpoints.
|
||||||
|
|
||||||
|
**What we are building.** The **VKIST MSK Ultrasound Stack** is an **air-gapped-first, single-hospital AI platform** that turns knee ultrasound scans into automated, explainable **synovitis grading (0–3)** with Vietnamese-language reasoning, every clinical statement **cited back to Ministry of Health (MOH) guidelines**, and a **human clinician signature required** before anything is finalized. It migrates VKIST's existing DeepLabv3/UNet/ConvNeXt vision models from research into a referenceable, production-grade medical software ecosystem.
|
||||||
|
|
||||||
|
**Why it is a sound investment (the four pillars):**
|
||||||
|
|
||||||
|
| Pillar | What it means for the leader |
|
||||||
|
| :--- | :--- |
|
||||||
|
| **Zero-friction adoption** | The AI is an *invisible background validation layer* — 0 extra clicks to see explanations (NFR-12/13). Clinicians keep their workflow; the system removes liability risk instead of adding work. |
|
||||||
|
| **Non-negotiable safety** | Human-in-the-Loop signature gate (NFR-19), 100% MOH-cited explanations (NFR-18), immutable audit log (NFR-17), and a 4-quadrant safety framework that explicitly defends against automation bias *and* AI hallucination. |
|
||||||
|
| **Regulatory sovereignty** | Runs inside the hospital LAN. Decree 13/2023 PII protection + Circular 46/2018 EMR compliance are built into the architecture, not bolted on. No patient data leaves the hospital in production. |
|
||||||
|
| **Capital efficiency** | Built on proven open-source (K3s, Triton, Postgres/pgvector, Keycloak, MinIO) and existing VKIST models. No SaaS lock-in, no per-seat licensing, delivered in a fixed 3-month pilot. |
|
||||||
|
|
||||||
|
**The ask.** Sponsorship of a fixed 3-month pilot on a single hospital's on-premise infrastructure (one Dell PowerEdge class node + one GPU node), a small cross-functional team, and a *governed, time-boxed* cloud allowance for the proof-of-concept LLM tiers (NFR-16a, retired at sign-off). See [Section 8](#8-investment-ask--scope-boundaries).
|
||||||
|
|
||||||
|
**The expected return.** A validated, referenceable clinical AI product that (projected) removes tedious measurement work from overloaded specialists, gives physiotherapists diagnostic visibility they have never had, converts patient anxiety into institutional trust, and establishes VKIST's operational credibility in the local healthcare community — with every claim instrumented and measurable during the pilot itself (see [Section 4](#4-quantitative-roi--projected-metrics--measurement-plan)).
|
||||||
|
|
||||||
|
> **Bottom line for the decision-maker:** This is a low-blast-radius, standards-compliant, single-site pilot that converts already-funded research into a safe, auditable clinical tool. The downside is bounded (one hospital, 3 months, open-source stack); the upside is a defensible product line and clinical-community trust.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Project Context
|
||||||
|
|
||||||
|
### 1.1 Who the System Is Designed For
|
||||||
|
|
||||||
|
The platform coordinates a **four-persona clinical care chain**. The pilot (Sprint 1–2, FR-25) is centered on the **Diagnostic Radiologist (UP5)** as the primary human actor, with the other three personas as governance/future-sprint stakeholders whose needs shape the non-functional requirements.
|
||||||
|
|
||||||
|
| Persona | Who they are | Environment | Core responsibility | What the system must respect |
|
||||||
|
| :--- | :--- | :--- | :--- | :--- |
|
||||||
|
| **UP5 — Diagnostic Radiologist** | Specialist MDs, age 28–55, from digitally native attendings to department heads | Low-light reading rooms; massive daily DICOM backlogs; asynchronous, detached from patients | Translate images into definitive reports that gate all downstream care | *Invisible workflow preservation* — no extra clicks, no DICOM render lag, no uncompensated patient chat |
|
||||||
|
| **UP7 — Rheumatologist / Orthopedic Surgeon** | Treatment architects, age 30–60; seniors control procurement | Split across OR, chaotic outpatient clinics (>100 patients/shift), inpatient wards | Synthesize fragmented data into a treatment plan; bear full legal/clinical liability | *Clinical force multiplier* — rapid dashboards, automated patient-education visuals, strictly gated communication |
|
||||||
|
| **UP6 — Physical Therapist** | Young (36–40% aged 20–29), highly stratified (≈54% vocational tier), digitally literate | Cramped (<20 m²), high-noise public units; 11–20+ patients/shift; shared/scarce workstations | Execute physician scripts into kinetic rehab programs | *Digital exoskeleton* — gel-proof knuckle-tap UI, zero-GPU mobile, localized language, objective progress data |
|
||||||
|
| **UP8 — MSK Patient & Family Caregiver** | Elderly patients (45–80+) via younger caregiver proxies (20–45) | Home, pharmacy, crowded waiting rooms; saturated with health misinformation | Manage chronic pain; adhere to protocols; make family-centric decisions | *Empathy + accessibility* — jargon-free, high-contrast, large-font, dual-profile family access on legacy phones |
|
||||||
|
|
||||||
|
*Source: [User_Research_Result.md](../Requirement_Analysis/USER_RESEARCH/User_Research_Result.md), [CONTEXT_VISION_SCOPE.md §3](../PLAN/CONTEXT_VISION_SCOPE.md).*
|
||||||
|
|
||||||
|
### 1.2 The Pain Points That Drive the Need
|
||||||
|
|
||||||
|
These are not hypothetical — they are drawn from ethnographic user research (professional forums, patient forums, and published studies).
|
||||||
|
|
||||||
|
- **Clinical exhaustion & volume:** Central referral hospitals (Bach Mai, Viet Duc) regularly exceed **100 evaluations per shift**, leaving no time for tedious measurements or objective double-checks. This creates the risk of *automation-bias blind-concurrence* — a clinician rubber-stamping an AI estimate under time pressure.
|
||||||
|
- **The physiotherapist information bottleneck:** PTs receive only brief text prescriptions — no DICOM, no scan findings — forcing **semi-blind therapeutic execution** against estimated surface anatomy.
|
||||||
|
- **The physiotherapist body & literacy crisis:** **76.4%** WMSD prevalence (career-shortening physical strain); **84%** report a foreign-language barrier (only 25% adequate English); 96%/93%/86% rely on personal experience/peer chat/old textbooks rather than formal evidence.
|
||||||
|
- **Dangerous folk interventions:** Information-starved patients turn to leaf-wrapping or violent manual adjustment, causing tissue infection or permanent anatomical failure — because no one translated their scan into understandable terms.
|
||||||
|
- **Stranded research value:** VKIST's CV models (DeepLabv3/UNet/ConvNeXt/MedViT) are proven in the lab but have no production pathway, so they generate zero operational or reputational value.
|
||||||
|
|
||||||
|
### 1.3 Constraints the System Must Meet
|
||||||
|
|
||||||
|
**Functional Requirements (pilot scope — FR-25 pipeline):**
|
||||||
|
|
||||||
|
| FR | Requirement |
|
||||||
|
| :--- | :--- |
|
||||||
|
| **FR-23/24/25** | Segment knee joint structures → measure synovium thickness (mm) → grade synovitis **0–3** |
|
||||||
|
| **FR-25 / Grad-CAM** | Spatial activation overlay on the primary viewport with **zero extra clicks** |
|
||||||
|
| **FR-25 / Safety loop** | Conversational Circuit Breaker + Socratic dialogue before finalization; BERT drift monitor; RAG-Referee arbitration |
|
||||||
|
| **FR-25 / Reporting** | Circular 46/2018 compliant PDF report generation |
|
||||||
|
| **FR-25 / Finalization** | Cryptographically signed, HITL-gated electronic record → EMR |
|
||||||
|
|
||||||
|
**Non-Functional Requirements (the "why it won't fail" constraints):**
|
||||||
|
|
||||||
|
| NFR | Constraint | Why the leader cares |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| **NFR-4** | ≤ 150 MB idle app bundle | Runs on cheap, legacy hospital/consumer hardware |
|
||||||
|
| **NFR-5** | ≤ 1.5 s core inference (on-prem) | No workflow slowdown |
|
||||||
|
| **NFR-7** | ≤ 200 ms token streaming (TTFT) | Feels instantaneous to the clinician |
|
||||||
|
| **NFR-9** | ≥ 99.9% availability (≤ 45 s downtime/day) | Clinically dependable |
|
||||||
|
| **NFR-10** | 100% of patient-facing LLM text passes safety verification | No unsafe generative output reaches a human |
|
||||||
|
| **NFR-11** | Onboarding ≤ 45 minutes | Cheap to roll out to busy staff |
|
||||||
|
| **NFR-12/13** | 0 extra clicks for explainability; native spatial activation maps | Anti-black-box; builds clinician trust |
|
||||||
|
| **NFR-14** | No client-side GPU/accelerator required (Android 10+, 3 GB RAM) | Works on the fleet that actually exists |
|
||||||
|
| **NFR-15** | Circular 46/2018 EMR compliance | Legally deployable |
|
||||||
|
| **NFR-16** | Air-gapped primary (no PHI over public internet in production) | Data sovereignty (Decree 13) |
|
||||||
|
| **NFR-17** | Immutable audit log | Accountability & liability defense |
|
||||||
|
| **NFR-18** | 100% LLM clinical text cites MOH protocol via RAG | Every claim is traceable & defensible |
|
||||||
|
| **NFR-19** | HITL digital signature before FINALIZED/ARCHIVED | A licensed human always owns the decision |
|
||||||
|
|
||||||
|
*Source: [SOLUTION_ARCHITECTURE_SPEC.md §2](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md), [SOFTWARE_ARCHITECTURE_SPEC.md §2](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md).*
|
||||||
|
|
||||||
|
### 1.4 System Context — C4 Level 1 (PlantUML)
|
||||||
|
|
||||||
|
The context diagram summarizes *who* and *what* the system boundary talks to. The pilot activates UP5 (and governance actors UP1/UP4); UP6/UP7/UP8 are shown as future-sprint boundaries.
|
||||||
|
|
||||||
|
```plantuml
|
||||||
|
@startuml VKIST_MSK_System_Context
|
||||||
|
!include <C4/C4_Context>
|
||||||
|
|
||||||
|
title System Context Diagram — VKIST MSK Ultrasound Stack (Pilot: FR-25 Synovitis Grading)
|
||||||
|
|
||||||
|
' === PRIMARY HUMAN ACTOR (in pilot scope) ===
|
||||||
|
Person(radiologist, "Diagnostic Radiologist (UP5)", "Loads knee ultrasound, reviews AI grading (0-3), confirms/overrides, engages Socratic circuit breaker, finalizes & signs the report, reads GradCAM + RAG evidence")
|
||||||
|
Person(admin, "System Administrator", "Deploys/monitors on-prem K3s, model updates, observability, NGINX failover")
|
||||||
|
|
||||||
|
' === GOVERNANCE / NFR-ALIGNMENT ACTORS ===
|
||||||
|
Person_Ext(senior_expert, "Healthcare Senior Expert (UP1)", "Clinical protocol validation, model-threshold approval, pilot sponsor")
|
||||||
|
Person_Ext(support_staff, "Support Staff (UP4)", "Patient registration, case-queue management")
|
||||||
|
|
||||||
|
' === FUTURE-SPRINT PERSONAS (out of pilot scope) ===
|
||||||
|
Person_Ext(therapist, "Physical Therapist (UP6)", "Future: read-only diagnostic visibility, kinetic overlays, progress tracking")
|
||||||
|
Person_Ext(surgeon, "Surgeon / Rheumatologist (UP7)", "Future: aggregated dashboards, treatment planning")
|
||||||
|
Person_Ext(patient, "Patient & Caregiver (UP8)", "Future: patient portal, plain-language education, self-monitoring")
|
||||||
|
|
||||||
|
' === THE SYSTEM ===
|
||||||
|
System(mpps, "VKIST MSK Ultrasound Stack (FR-25)", "Air-gapped-first: knee ultrasound AI (angle -> inflammation -> segmentation), synovitis grading 0-3 with GradCAM, 4-quadrant clinical safety, MOH-cited Vietnamese explanations, HITL-signed Circular 46 reports")
|
||||||
|
|
||||||
|
' === EXTERNAL SYSTEMS ===
|
||||||
|
System_Ext(pacs, "Hospital PACS / Ultrasound Machine", "DICOM source (C-MOVE + direct capture)")
|
||||||
|
System_Ext(emr, "Hospital EMR / HIS", "Signed report storage, prescription/lab sync (HL7/FHIR)")
|
||||||
|
System_Ext(triton, "Triton Inference Server", "GPU node: 3-step ML pipeline + RAG embeddings (gRPC)")
|
||||||
|
System_Ext(knowledge, "Clinical Knowledge Systems", "ladybugDB ontology (SNOMED-CT), pgvector MOH guideline index, Gemma/MedGemma LLM tiers")
|
||||||
|
|
||||||
|
' === RELATIONSHIPS ===
|
||||||
|
Rel(radiologist, mpps, "Reviews grade, overrides, finalizes & signs, reads explanations", "HTTPS")
|
||||||
|
Rel(admin, mpps, "Deploys, monitors, updates models", "HTTPS / SSH")
|
||||||
|
Rel(senior_expert, mpps, "Validates protocols, approves thresholds", "HTTPS (governance)")
|
||||||
|
Rel(support_staff, mpps, "Registration, case queue", "HTTPS")
|
||||||
|
Rel(therapist, mpps, "Future sprint scope", "—")
|
||||||
|
Rel(surgeon, mpps, "Future sprint scope", "—")
|
||||||
|
Rel(patient, mpps, "Future sprint scope", "—")
|
||||||
|
|
||||||
|
Rel(mpps, pacs, "Imports DICOM", "DICOM / C-MOVE")
|
||||||
|
Rel(mpps, emr, "Pushes signed reports + audit records", "HL7/FHIR")
|
||||||
|
Rel(mpps, triton, "ML inference offload", "gRPC 8001")
|
||||||
|
Rel(mpps, knowledge, "RAG retrieval, ontology traversal, LLM generation", "SQL (pgvector) + in-process")
|
||||||
|
|
||||||
|
@enduml
|
||||||
|
```
|
||||||
|
|
||||||
|
*Adapted from [SOLUTION_ARCHITECTURE_SPEC.md §8.1](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md).*
|
||||||
|
|
||||||
|
### 1.5 Applicability — Use-Case View (PlantUML)
|
||||||
|
|
||||||
|
The pilot's functional scope is organized around a **4-Quadrant Human-AI Interaction Framework** that deliberately defends against both *automation bias* (blindly trusting AI) and *clinician subservience* (AI overriding a correct human). This is the applicability model — what the radiologist can actually *do* with the system.
|
||||||
|
|
||||||
|
```plantuml
|
||||||
|
@startuml VKIST_MSK_UseCases
|
||||||
|
left to right direction
|
||||||
|
skinparam packageStyle rectangle
|
||||||
|
|
||||||
|
actor "Diagnostic Radiologist (UP5)" as Rad
|
||||||
|
actor "MSK Vision Grader" as Grader <<system>>
|
||||||
|
actor "LLM Explainer" as LLM <<system>>
|
||||||
|
actor "RAG-Referee Arbitrator" as Referee <<system>>
|
||||||
|
actor "Hospital EMR" as EMR <<system>>
|
||||||
|
|
||||||
|
rectangle "VKIST MSK Workspace (FR-25)" {
|
||||||
|
usecase "UC-48376\nLoad Patient Scan Session" as U1
|
||||||
|
usecase "UC-47988\nReview Suggested Synovitis Grade (0-3)" as U2
|
||||||
|
usecase "UC-92006\nFinalize & Sign Electronic Record" as U3
|
||||||
|
|
||||||
|
rectangle "Q1 — True Agreement (AI correct / Doctor correct)" {
|
||||||
|
usecase "UC-25776\nGradCAM + CoT Explanation" as Q1a
|
||||||
|
usecase "UC-02423\nLog High-Trust Concur Block" as Q1b
|
||||||
|
}
|
||||||
|
rectangle "Q2 — Automation Override (AI correct / Doctor oversight)" {
|
||||||
|
usecase "UC-22159\nTrigger Circuit Breaker" as Q2a
|
||||||
|
usecase "UC-55146\nSocratic Reasoning Dialogue" as Q2b
|
||||||
|
usecase "UC-74821\nBERT Context-Drift Monitor" as Q2c
|
||||||
|
usecase "UC-65473\nArbitrate via RAG-Referee" as Q2d
|
||||||
|
}
|
||||||
|
rectangle "Q3 — Clinician Subservience (AI hallucinates / Doctor correct)" {
|
||||||
|
usecase "UC-25637\nExpose Pixel-Level Activation" as Q3a
|
||||||
|
usecase "UC-60739\nIsolate Visual Noise/Artifacts" as Q3b
|
||||||
|
usecase "UC-62864\nCommit Validated Ground-Truth" as Q3c
|
||||||
|
}
|
||||||
|
rectangle "Q4 — Double-Blind Failure (AI faulty / edge case)" {
|
||||||
|
usecase "UC-35956\nActivate Clinical Investigation Mode" as Q4a
|
||||||
|
usecase "UC-47796\nStructured Morphology Annotation" as Q4b
|
||||||
|
usecase "UC-01580\nSerialize to Telemetry Queue" as Q4c
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Rad --> U1
|
||||||
|
Rad --> U2
|
||||||
|
Rad --> U3
|
||||||
|
Rad --> Q2b : Argue observations
|
||||||
|
Rad --> Q3b : Tag artifacts
|
||||||
|
Rad --> Q4b : Document manual findings
|
||||||
|
|
||||||
|
Grader --> U1 : Feeds vision tensors + scores
|
||||||
|
Grader --> U2 : Generates prediction
|
||||||
|
LLM --> Q1a : Generates clinical CoT
|
||||||
|
LLM --> Q2b : Generates Socratic checks
|
||||||
|
Referee --> Q2d : Retrieves MOH guidelines
|
||||||
|
U3 --> EMR : Syncs signed report
|
||||||
|
Q1b --> EMR : Stores consensus log
|
||||||
|
Q4c --> EMR : Routes anomaly to engineering telemetry
|
||||||
|
|
||||||
|
U1 ..> Q1a : <<include>>
|
||||||
|
Q1a ..> Q1b : <<include>>
|
||||||
|
U2 <.. Q2a : <<extend>> (friction detected)
|
||||||
|
Q2a ..> Q2b : <<include>>
|
||||||
|
Q2a ..> Q2c : <<include>>
|
||||||
|
Q2c ..> Q2d : <<extend>> (impasse/drift)
|
||||||
|
U2 <.. Q3a : <<extend>> (clinician contests score)
|
||||||
|
Q3a ..> Q3b : <<include>>
|
||||||
|
Q3b ..> Q3c : <<include>>
|
||||||
|
U2 <.. Q4a : <<extend>> (low confidence + empty RAG)
|
||||||
|
Q4a ..> Q4b : <<include>>
|
||||||
|
Q4b ..> Q4c : <<include>>
|
||||||
|
|
||||||
|
@enduml
|
||||||
|
```
|
||||||
|
|
||||||
|
*Adapted from [full_usecase_planuml.md](../../workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/Use_Case/full_usecase_planuml.md) and [CONTEXT_FR_25_UC_ELIT.md](../Requirement_Analysis/UC_Design/FR_25_UC_DESIGN/CONTEXT_FR_25_UC_ELIT.md).*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. How the System Addresses the Problem
|
||||||
|
|
||||||
|
### 2.1 Technology Choices — Problem → Solution → Why Optimal → Limitation
|
||||||
|
|
||||||
|
Each technology is selected against a specific user problem or NFR, not for novelty. The table states the honest limitation of each choice so the leader sees a balanced picture.
|
||||||
|
|
||||||
|
| Technology | User problem / NFR it solves | Why it is the optimal choice | Limitation / trade-off |
|
||||||
|
| :--- | :--- | :--- | :--- |
|
||||||
|
| **React PWA + LiteRT (MediaPipe) + WebLLM edge** | NFR-14 (no client GPU), NFR-4 (150 MB), zero-friction on legacy phones | Installable, offline-capable, runs a local angle pre-classifier + edge LLM with no server round-trip; degrades gracefully on weak hardware | Browser memory ceilings; edge LLM is small (GemmaE2B-Q4 ~1.3 GB) so deep reasoning must escalate |
|
||||||
|
| **FastAPI edge-inference-svc (Python 3.12)** | FR-25 pipeline orchestration, Decree 13 middleware, SSE streaming | Async, lightweight, tight control over PII redaction + audit at the boundary; in-house domain logic | Python GIL for CPU-bound work — mitigated by offloading inference to Triton |
|
||||||
|
| **Triton Inference Server (ONNX/TensorRT)** | NFR-5 (≤1.5 s inference), reuse of VKIST CV models | Industry-standard, GPU-efficient, self-hosted (no SaaS); serves the 3-step ensemble + embeddings | Requires a GPU node (capex); one more component to operate |
|
||||||
|
| **Postgres + pgvector (HNSW)** | NFR-18 MOH-cited RAG, transactional consistency of clinical records | One store for records *and* vectors (~15K guideline vectors) → ~5–20 ms retrieval, no separate vector DB to run | Not optimal beyond ~100K vectors / 500 QPS (documented Phase-2 path to Qdrant) |
|
||||||
|
| **RAG-Referee (BERT classifier)** | NFR-10/18 — reject unsupported LLM claims | Validates attribution + logical cohesion of every explanation before it reaches a human | Adds latency to the cloud tier; thresholds need clinical tuning |
|
||||||
|
| **3-Tier LLM (Edge Gemma → Gemini → MedGemma)** | Vietnamese explanations, deep clinical reasoning, cost control | Cheapest-capable-tier routing: local first, cloud only when needed, <20% MedGemma usage cap | Cloud tiers are PoC-only (NFR-16a) and must be redacted — retired at sign-off |
|
||||||
|
| **Keycloak (OIDC/RBAC) on-prem** | NFR-16 sovereignty, role separation (PT read-only) | No SaaS identity; realm-scoped RBAC inside the LAN | Operational overhead vs. managed Auth0/Okta (accepted for sovereignty) |
|
||||||
|
| **K3s on Dell PowerEdge** | NFR-16 air-gap, NFR-9 availability | Lightweight Kubernetes for a single hospital; rolling updates, self-healing pods; NGINX+Keepalived VIP failover (≤2 s) | Single-site scale; not a multi-region HA design (by design for the pilot) |
|
||||||
|
| **MinIO + Redis (5 scoped types)** | Object/blob store + session/cache | S3-compatible, self-hosted, no cloud egress; Redis scoped to prevent sprawl | Self-managed durability (mitigated by backup cron to MinIO) |
|
||||||
|
|
||||||
|
**Honest, system-level limitations (stated up front for the leader):**
|
||||||
|
- **Single-hospital scale by design.** The pilot targets one site; multi-site federation is a documented Phase-2 evolution, not a pilot promise.
|
||||||
|
- **Cloud LLM tiers are proof-of-concept only** under NFR-16a, gated by redaction + consent + audit, and **formally retired at PoC sign-off** — production reverts to fully air-gapped NFR-16.
|
||||||
|
- **Edge model is intentionally small** — deep reasoning escalates through the tier chain, which is why the cost guard (<20% MedGemma) and RAG-Referee exist.
|
||||||
|
|
||||||
|
*Source: [SOFTWARE_ARCHITECTURE_SPEC.md §7 Build-vs-Buy](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md), [SOLUTION_ARCHITECTURE_SPEC.md §5](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md).*
|
||||||
|
|
||||||
|
### 2.2 Deployment Environment — C4 Container Diagram (PlantUML)
|
||||||
|
|
||||||
|
The system runs entirely inside the hospital LAN. The only external touchpoints are the governed, PoC-scoped cloud fallbacks (NFR-16a).
|
||||||
|
|
||||||
|
```plantuml
|
||||||
|
@startuml VKIST_MSK_Containers
|
||||||
|
!include <C4/C4_Container>
|
||||||
|
|
||||||
|
title C4 Container Diagram — VKIST MSK Ultrasound Stack
|
||||||
|
|
||||||
|
Person(radiologist, "Diagnostic Radiologist (UP5)", "Loads DICOM, reviews grading, finalizes reports")
|
||||||
|
Person(admin, "System Administrator", "K3s ops, model updates, observability, failover")
|
||||||
|
Person_Ext(senior_expert, "Healthcare Senior Expert (UP1)", "Protocol validation, threshold approval")
|
||||||
|
Person_Ext(support_staff, "Support Staff (UP4)", "Registration, case queue")
|
||||||
|
|
||||||
|
System_Boundary(hospital_lan, "Hospital LAN (Air-Gapped, <=10 Mbps)") {
|
||||||
|
Container(pwa, "React PWA Frontend", "React 18, TS, Zustand, LiteRT, MediaPipe, Dexie.js", "View-angle validation (WASM); DICOM preview; Grad-CAM overlay; Decree 13 scrubber; encrypted IndexedDB sessions")
|
||||||
|
Container(nginx, "Active-Passive Gateway", "NGINX + Keepalived", "SSL termination, VIP load balancing, <=2 s failover")
|
||||||
|
|
||||||
|
System_Boundary(k3s_cluster, "K3s Orchestration Cluster") {
|
||||||
|
Container(edge_inference, "Edge Inference Service", "FastAPI (Python 3.12)", "DICOM ingest, 3-step ML orchestration, Grad-CAM, report assembly, SSE streaming")
|
||||||
|
Container(cloud_llm_gateway, "Cloud LLM Gateway", "FastAPI (Python 3.12)", "Routes to Gemini (orchestration) / MedGemma (clinical); NFR-16a redaction, consent, audit, cost guard")
|
||||||
|
Container(rag_svc, "RAG & Knowledge Service", "FastAPI (asyncpg)", "pgvector top-k, ladybugDB ontology, RAG-Referee validation, mandatory RAG pre-processing")
|
||||||
|
Container(audit_svc, "Audit & EMR Service", "Node.js / FastAPI Worker", "Immutable append-only audit; HL7/FHIR EMR push with outbox retry")
|
||||||
|
Container(api_gw, "API Gateway", "Envoy", "TLS, rate limiting, routing, OIDC pass-through")
|
||||||
|
Container(auth_svc, "Identity & Access", "Keycloak", "OIDC, RBAC, realm vkist-msk")
|
||||||
|
Container(obs_stack, "Observability Stack", "Prometheus + Grafana", "Metrics, dashboards, alerting")
|
||||||
|
}
|
||||||
|
|
||||||
|
System_Boundary(db_vm, "Database VM") {
|
||||||
|
ContainerDb(postgres, "PostgreSQL + pgvector", "PostgreSQL 16 + pgvector", "EMR records, spatial markers, MOH HNSW index, embeddings, audit ledger")
|
||||||
|
ContainerDb(redis, "Redis Cache", "Redis 7 (AOF+RDB)", "Sessions, guideline chunks, DICOM metadata, consult_mode state, rate-limit")
|
||||||
|
ContainerDb(minio, "MinIO Object Store", "MinIO S3-compatible", "DICOM payloads, overlays, Grad-CAM heatmaps, report PDFs, model staging")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
System_Ext(triton, "Triton Inference Server", "NVIDIA Triton (ONNX/TensorRT)", "3-step ML pipeline + EmbeddingGemma RAG embeddings")
|
||||||
|
System_Ext(emr, "Hospital EMR / HIS", "HL7/FHIR", "Signed report storage, prescription sync")
|
||||||
|
System_Ext(pacs, "PACS / Ultrasound Device", "DICOM / C-MOVE", "Image capture & retrieval")
|
||||||
|
System_Ext(vertex_ai, "GCP Vertex AI (Gemini)", "REST", "PoC-only NFR-16a: orchestration, translation; redacted payloads only")
|
||||||
|
System_Ext(modal_medgemma, "Modal MedGemma", "T4 GPU", "PoC-only NFR-16a: clinical deep-reasoning; redacted + RAG-Referee validated")
|
||||||
|
|
||||||
|
Rel(radiologist, pwa, "Loads scan, reviews grade, finalizes, reads explanations", "HTTPS 443")
|
||||||
|
Rel(admin, nginx, "Deploys, monitors, configures", "HTTPS / SSH")
|
||||||
|
Rel(senior_expert, pwa, "Validates protocols, approves thresholds", "HTTPS 443")
|
||||||
|
Rel(support_staff, pwa, "Registration, case queue", "HTTPS 443")
|
||||||
|
|
||||||
|
Rel(pwa, nginx, "API requests + pre-validated DICOM", "HTTPS 443")
|
||||||
|
Rel(nginx, api_gw, "Routes upstream", "HTTP 8000")
|
||||||
|
Rel(api_gw, edge_inference, "Forwards /api/*", "HTTP 8000")
|
||||||
|
Rel(api_gw, rag_svc, "Forwards /rag/*", "HTTP 8001")
|
||||||
|
Rel(api_gw, auth_svc, "Validates OIDC tokens", "HTTP 8080")
|
||||||
|
|
||||||
|
Rel(edge_inference, triton, "3-step inference + embeddings", "gRPC 8001")
|
||||||
|
Rel(edge_inference, postgres, "Reads guidelines; writes cases/embeddings/audit", "SQL 5432")
|
||||||
|
Rel(edge_inference, redis, "Session, rate-limit, consult_mode", "TCP 6379")
|
||||||
|
Rel(edge_inference, minio, "DICOM, overlays, reports", "S3 API")
|
||||||
|
Rel(edge_inference, cloud_llm_gateway, "Routes cloud consult", "HTTP 8000")
|
||||||
|
|
||||||
|
Rel(cloud_llm_gateway, vertex_ai, "Gemini proxy (NFR-16a redacted)", "HTTPS 443")
|
||||||
|
Rel(cloud_llm_gateway, modal_medgemma, "MedGemma proxy (NFR-16a redacted)", "HTTPS 443")
|
||||||
|
Rel(cloud_llm_gateway, postgres, "Audit log append", "SQL 5432")
|
||||||
|
|
||||||
|
Rel(rag_svc, postgres, "pgvector HNSW queries", "SQL 5432")
|
||||||
|
Rel(rag_svc, redis, "Guideline cache, pub/sub invalidation", "TCP 6379")
|
||||||
|
|
||||||
|
Rel(audit_svc, postgres, "Appends immutable audit events", "SQL 5432")
|
||||||
|
Rel(audit_svc, emr, "Finalized report push", "HL7/FHIR")
|
||||||
|
|
||||||
|
Rel(pwa, pacs, "Direct DICOM capture / C-MOVE", "DICOM")
|
||||||
|
|
||||||
|
@enduml
|
||||||
|
```
|
||||||
|
|
||||||
|
*Reused from [SOFTWARE_ARCHITECTURE_SPEC.md §4.2](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md).*
|
||||||
|
|
||||||
|
### 2.3 Deployment Topology — C4 Deployment Diagram (PlantUML)
|
||||||
|
|
||||||
|
Physical placement: everything on hospital hardware; the only external node is the governed NFR-16a fallback.
|
||||||
|
|
||||||
|
```plantuml
|
||||||
|
@startuml VKIST_MSK_Deployment
|
||||||
|
!include <C4/C4_Deployment>
|
||||||
|
LAYOUT_LEFT_RIGHT()
|
||||||
|
|
||||||
|
Deployment_Node(hw, "Dell PowerEdge (Hospital)") {
|
||||||
|
Deployment_Node(k3s, "K3s Cluster") {
|
||||||
|
Deployment_Node(pod_edge, "Pod: edge-inference-svc") {
|
||||||
|
Container(ci, "FastAPI", "Python 3.12, Uvicorn")
|
||||||
|
}
|
||||||
|
Deployment_Node(pod_rag, "Pod: rag-svc") {
|
||||||
|
Container(cr, "FastAPI", "Python 3.12, asyncpg")
|
||||||
|
}
|
||||||
|
Deployment_Node(pod_audit, "Pod: audit-svc") {
|
||||||
|
Container(ca, "Node.js", "WAL writer")
|
||||||
|
}
|
||||||
|
Deployment_Node(pod_gw, "Pod: api-gateway") {
|
||||||
|
Container(cg, "Envoy", "TLS, rate limit")
|
||||||
|
}
|
||||||
|
Deployment_Node(pod_auth, "Pod: auth-svc") {
|
||||||
|
Container(ck, "Keycloak", "OIDC, RBAC")
|
||||||
|
}
|
||||||
|
Deployment_Node(pod_obs, "Pod: observability") {
|
||||||
|
Container(cp, "Prometheus")
|
||||||
|
Container(cf, "Grafana")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Deployment_Node(db_vm, "DB VM") {
|
||||||
|
ContainerDb(pg, "PostgreSQL + pgvector HNSW")
|
||||||
|
ContainerDb(rd, "Redis")
|
||||||
|
ContainerDb(mn, "MinIO")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Deployment_Node(triton_node, "GPU Node (Triton)", "NVIDIA A10/T4, 24 GB VRAM") {
|
||||||
|
Container(tc, "Triton Inference Server", "ONNX/TensorRT, gRPC 8001")
|
||||||
|
}
|
||||||
|
Deployment_Node(ext, "External (NFR-16a governed, PoC only)") {
|
||||||
|
ContainerDb(s3v, "GCP Vertex AI / Modal MedGemma / CDN fallback")
|
||||||
|
}
|
||||||
|
|
||||||
|
Rel(ci, pg, "SQL", "TCP 5432")
|
||||||
|
Rel(ci, rd, "Cache", "TCP 6379")
|
||||||
|
Rel(ci, mn, "Blobs", "S3 API")
|
||||||
|
Rel(ci, tc, "Inference", "gRPC 8001")
|
||||||
|
Rel(ci, ck, "Auth", "OIDC")
|
||||||
|
Rel(ci, ca, "Audit", "HTTP")
|
||||||
|
Rel(k3s, db_vm, "SQL / Cache / Blobs", "TCP")
|
||||||
|
|
||||||
|
@enduml
|
||||||
|
```
|
||||||
|
|
||||||
|
*Reused from [SOFTWARE_ARCHITECTURE_SPEC.md §4.4](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md).*
|
||||||
|
|
||||||
|
### 2.4 Components & Interactions — C4 Component Diagram (PlantUML)
|
||||||
|
|
||||||
|
Inside the core `edge-inference-svc`, each component maps to a specific functional requirement. The **Inference Router** implements the tiered LLM escalation; the **Circuit Breaker** guarantees the system fails safely to MOH templates.
|
||||||
|
|
||||||
|
```plantuml
|
||||||
|
@startuml VKIST_MSK_Component_EdgeInference
|
||||||
|
!include <C4/C4_Component>
|
||||||
|
|
||||||
|
Container_Boundary(edge_svc, "edge-inference-svc (FastAPI)") {
|
||||||
|
Component(api, "API Controller", "REST + SSE", "/api/analyze, cloud-orchestrate redirect")
|
||||||
|
Component(stream, "SSE Token Streamer", "StreamingResponse", "200 ms TTFT, 30 s heartbeat")
|
||||||
|
Component(preproc, "Image Preprocessor", "OpenCV + pydicom", "CLAHE, rescale, DICOM header scrub")
|
||||||
|
Component(router, "Inference Router", "consult_mode state", "Tier selection: Edge Gemma -> Gemini -> MedGemma -> Templates")
|
||||||
|
Component(breaker, "Circuit Breaker", "pybreaker", "Wraps Triton + EMR + Cloud LLM; fail-open to templates")
|
||||||
|
Component(pipeline, "ML Pipeline", "gRPC client", "Angle -> Inflammation -> Segmentation -> Measurement")
|
||||||
|
Component(gradcam, "Grad-CAM Generator", "OpenCV", "Spatial activation overlay -> base64 PNG")
|
||||||
|
Component(report, "Report Builder", "WeasyPrint", "Circular 46 PDF; HITL signature gate")
|
||||||
|
Component(rag, "RAG Service", "pgvector SQL", "Top-5 MOH chunks; mandatory pre-generation; NFR-18")
|
||||||
|
Component(referee, "RAG-Referee", "BERT classifier", "Reject LLM text if citation confidence < threshold")
|
||||||
|
Component(nlp, "NLP Scrubber", "Microsoft Presidio", "Re-verify edge redaction; clean residual PII")
|
||||||
|
Component(audit, "Audit Logger", "Append-only writer", "Every tier transition, consent, finalize, override")
|
||||||
|
}
|
||||||
|
|
||||||
|
Rel(api, stream, "delegates", "SSE")
|
||||||
|
Rel(api, preproc, "validates image", "sync")
|
||||||
|
Rel(api, router, "selects tier", "sync")
|
||||||
|
Rel(router, breaker, "guards calls", "sync")
|
||||||
|
Rel(router, pipeline, "invokes", "gRPC")
|
||||||
|
Rel(router, stream, "fallback text", "SSE")
|
||||||
|
Rel(api, gradcam, "requests overlay", "sync")
|
||||||
|
Rel(api, report, "generates PDF", "sync")
|
||||||
|
Rel(api, rag, "queries MOH", "SQL")
|
||||||
|
Rel(rag, referee, "validates citations", "sync")
|
||||||
|
Rel(api, nlp, "scrubs output", "sync")
|
||||||
|
Rel(api, audit, "writes event", "sync")
|
||||||
|
|
||||||
|
@enduml
|
||||||
|
```
|
||||||
|
|
||||||
|
The **Cloud LLM Gateway** enforces the NFR-16a governance boundary — nothing egresses without consent, redaction, RAG grounding, and an audit record.
|
||||||
|
|
||||||
|
```plantuml
|
||||||
|
@startuml VKIST_MSK_Component_CloudGateway
|
||||||
|
!include <C4/C4_Component>
|
||||||
|
|
||||||
|
Container_Boundary(cloud_gw, "cloud-llm-gateway (FastAPI)") {
|
||||||
|
Component(capi, "Cloud API Controller", "REST + SSE", "/api/cloud-orchestrate, /api/cloud-consult")
|
||||||
|
Component(gateway, "Cloud LLM Router", "task_type matcher", "Orchestration/translation -> Gemini; clinical -> MedGemma")
|
||||||
|
Component(consent, "Consent Enforcer", "Redis + NFR-16a checklist", "Validates consent token + redaction manifest before egress")
|
||||||
|
Component(caudit, "Cloud Audit Emitter", "PostgreSQL append-only", "egress_consent, egress_response, escalation events")
|
||||||
|
Component(cost, "Cost Guard", "Redis + Prometheus", "Tracks MedGemma usage; alerts if >20% over 24h")
|
||||||
|
Component(creferee, "RAG-Referee Trigger", "BERT classifier", "Mandatory validation of MedGemma output")
|
||||||
|
Component(crag, "RAG Pre-processing", "pgvector SQL", "Mandatory top-k injection before generation")
|
||||||
|
}
|
||||||
|
|
||||||
|
Rel(capi, consent, "verifies", "sync")
|
||||||
|
Rel(capi, gateway, "routes", "sync")
|
||||||
|
Rel(gateway, crag, "injects chunks", "sync")
|
||||||
|
Rel(gateway, creferee, "validates output", "sync")
|
||||||
|
Rel(gateway, caudit, "logs egress", "sync")
|
||||||
|
Rel(gateway, cost, "increments counter", "sync")
|
||||||
|
|
||||||
|
@enduml
|
||||||
|
```
|
||||||
|
|
||||||
|
*Reused from [SOFTWARE_ARCHITECTURE_SPEC.md §4.3](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md).*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Qualitative ROI — Behavioral & Workflow Change
|
||||||
|
|
||||||
|
The core value thesis: the system does not add a tool to a workflow — it **enriches the existing workflow and reconnects a fragmented care chain**. The following before/after captures the behavioral change per persona.
|
||||||
|
|
||||||
|
| Persona | Before (current behavior) | After (with the system) | Behavioral shift |
|
||||||
|
| :--- | :--- | :--- | :--- |
|
||||||
|
| **UP5 Radiologist** | Manual, repetitive measurements under fatigue; risk of missing subtle findings; isolated in a reading room; liability anxiety | AI pre-measures + grades; Grad-CAM shows *why* with 0 extra clicks; a Socratic circuit breaker catches rushed sign-offs; every report is MOH-cited and audit-backed | From lone, fatigued gatekeeper → to a *validated* decision-maker shielded from liability |
|
||||||
|
| **UP7 Surgeon** | Synthesizes fragmented silos by hand; no time for patient education; fears uncompensated patient messaging | Aggregated view of signed findings; auto-generated patient-friendly visuals; strictly gated (no open chat) communication | From data-assembler → to *clinical force multiplier* with protected time |
|
||||||
|
| **UP6 Physiotherapist** | Semi-blind execution from a one-line prescription; no scan visibility; language barrier; manual charting | Read-only diagnostic visibility down the chain; objective progress metrics; gel-proof, localized, zero-GPU UI | From guesswork executor → to *data-backed* practitioner with professional standing |
|
||||||
|
| **UP8 Patient/Caregiver** | Anxiety + confusion → folk remedies; crowdsources raw scans on social media | Scan translated into plain-language, high-tech visual validation of the doctor's diagnosis | From anxious misinformation-seeker → to a *trusting, informed* participant |
|
||||||
|
|
||||||
|
**System-level qualitative returns:**
|
||||||
|
- **Institutional trust & credibility:** A high-trust, objective, MOH-anchored tool displaces both dangerous folk interventions (patients) and unsecured external messaging (clinicians bypassing the system).
|
||||||
|
- **Academic-to-production migration:** VKIST research models become a live, referenceable clinical asset — establishing the team's operational value in the local healthcare community.
|
||||||
|
- **Care-chain cohesion:** The radiologist's sign-off *ripples* to the surgeon, PT, and patient — turning isolated actions into a connected clinical relationship (the "Synchronized Ritual" concept).
|
||||||
|
|
||||||
|
*Source: [Motivation.md](../DEMO_EXP/Motivation.md), [User_Research_Result.md](../Requirement_Analysis/USER_RESEARCH/User_Research_Result.md).*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Quantitative ROI — Projected Metrics & Measurement Plan
|
||||||
|
|
||||||
|
> **Important framing for the leader:** The pilot has not yet deployed (runs through Sep 2026), so the figures below are **modeled projections**, not measured outcomes. Each row states its **baseline source** and its **assumption**, and — critically — **how the pilot will actually measure it**. The value of the pilot is precisely to *convert these projections into evidence*.
|
||||||
|
|
||||||
|
### 4.1 Projected Before → After Metrics
|
||||||
|
|
||||||
|
| Metric | Baseline (before) | Target (after) | Assumption / basis | How the pilot measures it |
|
||||||
|
| :--- | :--- | :--- | :--- | :--- |
|
||||||
|
| **Tedious measurement time per scan** | Manual synovium measurement + grading, several minutes/scan under load | Automated pre-measurement; radiologist verifies | FR-25 automates FR-23/24 (segment + measure) | Time-on-task instrumentation (session telemetry, NFR audit timestamps) |
|
||||||
|
| **Core inference latency** | N/A (no automated pipeline today) | ≤ 1.5 s (on-prem) | NFR-5 target on Triton GPU node | Prometheus p50/p99 latency dashboards |
|
||||||
|
| **Explainability access cost** | Not available / manual literature lookup | **0 extra clicks** (Grad-CAM + citation inline) | NFR-12/13 design mandate | UI event audit (clicks-to-explanation = 0) |
|
||||||
|
| **Clinical claim traceability** | Ad-hoc, uncited | **100%** of LLM clinical text cites MOH protocol | NFR-18 mandatory RAG | RAG-Referee pass rate; citation coverage log |
|
||||||
|
| **Unsafe generative output reaching a human** | N/A | **0%** (100% pass safety verification) | NFR-10 3-tier guardrail | Guardrail rejection log; audit events |
|
||||||
|
| **Onboarding time per staff member** | Ad-hoc, hours | ≤ 45 minutes | NFR-11 target | Timed onboarding sessions during rollout |
|
||||||
|
| **System availability** | Dependent on manual workflow | ≥ 99.9% (≤ 45 s downtime/day) | NFR-9 target | Uptime monitoring (Prometheus/Grafana) |
|
||||||
|
| **Cloud (MedGemma) usage ratio** | N/A | < 20% of consults | Cost-guard target (capital efficiency) | Cost Guard Redis counter + alert |
|
||||||
|
| **PT diagnostic visibility** | ~0 (text prescription only) | Read-only access to signed findings | RBAC role (future sprint) | Access-log coverage per PT-linked case |
|
||||||
|
| **Auditability of AI accept/override** | None | 100% immutable capture | NFR-17 | Immutable audit ledger completeness check |
|
||||||
|
|
||||||
|
### 4.2 Behavioral-Change Indicators (Adoption Proxies)
|
||||||
|
|
||||||
|
Beyond raw performance, the pilot measures whether behavior actually changed — the true ROI signal for a leader:
|
||||||
|
|
||||||
|
- **Retention / anti-bypass:** % of target clinicians using the structured workspace instead of unsecured external messaging (a stated success criterion).
|
||||||
|
- **Override rate & ground-truth capture:** frequency of clinician overrides (Q3) feeding the retraining corpus — proves the human-AI loop works both ways.
|
||||||
|
- **Circuit-breaker engagement:** how often the Socratic breaker prevents a rushed sign-off (a direct automation-bias safety signal).
|
||||||
|
- **Compliance events:** zero unencrypted PHI leak events (Decree 13) — a binary go/no-go for production.
|
||||||
|
|
||||||
|
*Baselines from [User_Research_Result.md](../Requirement_Analysis/USER_RESEARCH/User_Research_Result.md); targets from NFRs in [SOLUTION_ARCHITECTURE_SPEC.md §2.2](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md); success criteria from [CONTEXT_VISION_SCOPE.md §6](../PLAN/CONTEXT_VISION_SCOPE.md).*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Risk & Failure-Mode Handling — Why This Is Not Risky
|
||||||
|
|
||||||
|
The single most important message for an investment decision: **this system is engineered around the assumption that either the AI *or* the human can be wrong.** It handles all four combinations explicitly via the **4-Quadrant Human-AI Interaction Framework**.
|
||||||
|
|
||||||
|
```plantuml
|
||||||
|
@startuml VKIST_MSK_Risk_Quadrants
|
||||||
|
skinparam packageStyle rectangle
|
||||||
|
title 4-Quadrant Human-AI Safety Framework (FR-25)
|
||||||
|
|
||||||
|
rectangle "Q1 — True Agreement\n(AI correct / Doctor correct)" as Q1
|
||||||
|
rectangle "Q2 — Automation Override\n(AI correct / Doctor oversight)" as Q2
|
||||||
|
rectangle "Q3 — Clinician Subservience\n(AI hallucinates / Doctor correct)" as Q3
|
||||||
|
rectangle "Q4 — Double-Blind Failure\n(AI faulty / edge case)" as Q4
|
||||||
|
|
||||||
|
Q1 --> Q1 : GradCAM + CoT -> High-Trust Concur log
|
||||||
|
Q2 --> Q2 : Circuit Breaker -> Socratic -> BERT drift -> RAG-Referee
|
||||||
|
Q3 --> Q3 : Expose activations -> isolate artifacts -> commit ground-truth
|
||||||
|
Q4 --> Q4 : Investigation mode -> manual annotation -> telemetry queue
|
||||||
|
|
||||||
|
@enduml
|
||||||
|
```
|
||||||
|
|
||||||
|
| Risk quadrant | The danger | How the system mitigates it |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| **Q1 — True Agreement** | Even correct consensus needs a trace for liability | Grad-CAM + Chain-of-Thought explanation, then an immutable High-Trust Concur log block |
|
||||||
|
| **Q2 — Automation Bias** | Overloaded doctor rubber-stamps AI | Conversational Circuit Breaker intercepts rushed finalization → Socratic dialogue → BERT drift monitor → RAG-Referee arbitration against MOH guidelines |
|
||||||
|
| **Q3 — AI Hallucination** | AI is wrong, doctor is right, but AI "authority" pressures the human | Expose pixel-level activation logic; let the clinician mask artifacts and **commit the corrected ground-truth** (which also feeds retraining) |
|
||||||
|
| **Q4 — Double-Blind Edge Case** | Both AI confidence and guideline coverage are low | Auto-switch to strict manual **Clinical Investigation Mode**; force structured morphology annotation; route the anomaly to an engineering telemetry queue (bypassing the EMR) |
|
||||||
|
|
||||||
|
**Cross-cutting safety guarantees:**
|
||||||
|
- **HITL gate (NFR-19):** No AI output becomes FINALIZED/ARCHIVED without a licensed clinician's cryptographic signature. *A human always owns the decision.*
|
||||||
|
- **Fail-safe by design:** The Circuit Breaker fails **open to deterministic MOH templates** — if Triton, EMR, or a cloud LLM is down, the system degrades to a safe rule-based floor, not to an error.
|
||||||
|
- **Immutable audit (NFR-17):** Every accept/override/tier-transition/consent event is append-only and tamper-proof — a complete liability defense record.
|
||||||
|
- **RAG-Referee (NFR-18):** No uncited clinical claim reaches a human.
|
||||||
|
|
||||||
|
*Source: [full_usecase_planuml.md](../../workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/Use_Case/full_usecase_planuml.md), and the Q1–Q4 scenario docs under [FR_25_UC_DESIGN/](../Requirement_Analysis/UC_Design/FR_25_UC_DESIGN/).*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Compliance & Data Governance
|
||||||
|
|
||||||
|
Healthcare compliance is treated as a first-class architectural constraint, not an afterthought.
|
||||||
|
|
||||||
|
| Regulation | Requirement | How the architecture satisfies it |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| **Decree 13/2023/ND-CP** (personal health data) | Sensitive PII must be protected; data sovereignty | Client-side WebCrypto scrubbing + FastAPI Presidio redaction middleware; role-hash tokens `[BN:7f3a2c]`; all production data resides in-hospital / sovereign VN data centers |
|
||||||
|
| **Circular 46/2018/TT-BYT** (EMR) | National EMR compliance for stored records | Circular-46-compliant PDF reports; HL7/FHIR EMR push; NFR-15 (not relaxed even under NFR-16a) |
|
||||||
|
| **Air-gap (NFR-16, permanent)** | No diagnostic/identifiable data over public internet in production | Everything runs on the hospital LAN; K3s network policies isolate workloads; local-only container registries |
|
||||||
|
| **Governed cloud PoC (NFR-16a, temporary)** | Cloud LLM only as an emergency/PoC fallback | Invoked only when edge LLM unavailable **and** user consents; mandatory pre-egress redaction + audit commit; CSEK keys, 30-day lifecycle, 1-year auto-delete; **retired at PoC sign-off** |
|
||||||
|
|
||||||
|
**The compliance headline for a leader:** in production, **no patient data leaves the hospital**, every clinical statement is cited, every action is audit-logged, and a human signature is legally required to finalize anything.
|
||||||
|
|
||||||
|
*Source: [SOLUTION_ARCHITECTURE_SPEC.md §2.2, §5.8](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md), [CONTEXT_VISION_SCOPE.md §4](../PLAN/CONTEXT_VISION_SCOPE.md).*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Roadmap & Milestones
|
||||||
|
|
||||||
|
A strict, time-boxed 3-month cycle (June 2 – September 2, 2026), executed as a hybrid of waterfall macro-phases and agile sprints. This bounds the investment and provides clear go/no-go checkpoints - However it can be tentative to change based on the actual resource of development team.
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
gantt
|
||||||
|
title VKIST MSK Pilot - 3-Month Delivery Roadmap
|
||||||
|
dateFormat YYYY-MM-DD
|
||||||
|
axisFormat %b %d
|
||||||
|
|
||||||
|
section Foundation
|
||||||
|
Sprint 0 Pitch and Architecture Spike :done, sp0, 2026-06-02, 2026-06-12
|
||||||
|
Win project pitch :milestone, m1, 2026-06-09
|
||||||
|
Sprint 1 Fast PoC Baseline :active, sp1, 2026-06-15, 2026-06-26
|
||||||
|
|
||||||
|
section Core Build
|
||||||
|
Sprint 2 Multi-Modal and NLP Integration :sp2, 2026-06-29, 2026-07-10
|
||||||
|
Sprint 3 Collaborative Workspace :sp3, 2026-07-13, 2026-07-24
|
||||||
|
|
||||||
|
section Patient and Hardening
|
||||||
|
Sprint 4 Patient-Facing PWA :sp4, 2026-07-27, 2026-08-07
|
||||||
|
Sprint 5 Feedback Pipeline and Hardening :sp5, 2026-08-10, 2026-08-21
|
||||||
|
Sprint 6 Beta Release and Evolution :sp6, 2026-08-24, 2026-09-02
|
||||||
|
Pilot cycle complete :milestone, m2, 2026-09-02
|
||||||
|
```
|
||||||
|
|
||||||
|
| Sprint | Window | Focus & key deliverable |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| **0** | Jun 02–12 | Pitch & architecture clearance; VKIST ML model audit; CI/CD baseline |
|
||||||
|
| **1** | Jun 15–26 | Fast end-to-end PoC: FastAPI pipeline + inference mask preview |
|
||||||
|
| **2** | Jun 29–Jul 10 | Multi-modal NLP + Decree 13 client-side scrubbing |
|
||||||
|
| **3** | Jul 13–24 | Collaborative canvas; RBAC (PT read-only); usability testing |
|
||||||
|
| **4** | Jul 27–Aug 07 | Installable PWA; zero-GPU fallback for legacy phones |
|
||||||
|
| **5** | Aug 10–21 | Feedback instrumentation; performance/caching; cleanup |
|
||||||
|
| **6** | Aug 24–Sep 02 | Beta launch to MSK cohort; telemetry analysis; Phase-2 backlog |
|
||||||
|
|
||||||
|
**Definition of success (pilot go/no-go):** zero workflow friction added; deterministic cross-device stability (incl. CPU-bound rendering); zero Decree 13 leak events; sustained clinician retention over external messaging; successful academic-to-production model migration.
|
||||||
|
|
||||||
|
*Source: [CONTEXT_VISION_SCOPE.md §5–6](../PLAN/CONTEXT_VISION_SCOPE.md).*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Investment Ask & Scope Boundaries
|
||||||
|
|
||||||
|
### 8.1 The Resource Envelope (qualitative)
|
||||||
|
|
||||||
|
| Category | Pilot ask | Rationale |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| **On-prem compute** | 1 × Dell PowerEdge-class node (K3s + DB VM) + 1 × GPU node (A10/T4, 24 GB VRAM) for Triton | Single-site air-gapped deployment; capex-only, no recurring SaaS |
|
||||||
|
| **Team** | Small cross-functional squad (backend/ML, frontend, infra/DevOps) + clinical champion (UP1/UP5) for validation | Matches the 7-sprint scope; clinical sponsor is essential for governance & buy-in |
|
||||||
|
| **Software** | Open-source stack (K3s, Triton, Postgres/pgvector, Redis, MinIO, Keycloak, Prometheus/Grafana) | Zero licensing cost; no vendor lock-in |
|
||||||
|
| **Governed cloud (PoC only)** | Time-boxed GCP Vertex AI (Gemini) + Modal (MedGemma) budget, capped at <20% consult usage | NFR-16a fallback; **retired at sign-off** — not a recurring production cost |
|
||||||
|
| **Data** | Access to de-identified MSK ultrasound + MOH guideline corpus | Needed to tune grading thresholds and RAG index |
|
||||||
|
|
||||||
|
### 8.2 In Scope vs. Out of Scope (Pilot)
|
||||||
|
|
||||||
|
| In scope (pilot) | Out of scope (Phase 2+) |
|
||||||
|
| :--- | :--- |
|
||||||
|
| FR-25 knee synovitis grading (UP5 primary) | Multi-hospital federation / multi-region HA |
|
||||||
|
| Grad-CAM explainability, Socratic safety loop, RAG-Referee | Full UP7 surgeon planning & UP8 patient portal at production scale |
|
||||||
|
| Decree 13 + Circular 46 compliance, HITL sign-off, audit log | Predictive analytics (cartilage degeneration, hardware failure forecasting) |
|
||||||
|
| Single-hospital air-gapped deployment | Qdrant hot-tier vector cache (only if >100K vectors / >500 QPS) |
|
||||||
|
| PoC cloud LLM tiers (NFR-16a, retired at sign-off) | Permanent cloud dependency (explicitly prohibited by NFR-16) |
|
||||||
|
|
||||||
|
### 8.3 Go / No-Go Decision Criteria
|
||||||
|
|
||||||
|
At the end of the 3-month cycle, the leader decides on Phase 2 based on objective evidence:
|
||||||
|
|
||||||
|
1. **Safety:** zero Decree 13 leak events; 100% HITL-gated finalizations; 100% MOH-cited clinical text.
|
||||||
|
2. **Performance:** inference ≤ 1.5 s; availability ≥ 99.9%; onboarding ≤ 45 min.
|
||||||
|
3. **Adoption:** measurable clinician retention within the structured workspace vs. external messaging.
|
||||||
|
4. **Strategic:** VKIST models successfully running in a referenceable production form.
|
||||||
|
|
||||||
|
> **The investment is bounded and reversible:** one hospital, one quarter, open-source stack, no production cloud dependency. If the go/no-go criteria are not met, the blast radius is a single controlled pilot — not an enterprise commitment.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix — Source Documents
|
||||||
|
|
||||||
|
- Vision, personas, constraints, milestones: [CONTEXT_VISION_SCOPE.md](../PLAN/CONTEXT_VISION_SCOPE.md)
|
||||||
|
- Deep personas + quantitative baselines: [User_Research_Result.md](../Requirement_Analysis/USER_RESEARCH/User_Research_Result.md)
|
||||||
|
- Architecture, FR/NFR, C4 diagrams, build-vs-buy: [SOFTWARE_ARCHITECTURE_SPEC.md](../ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md), [SOLUTION_ARCHITECTURE_SPEC.md](../ARCHITECT/SOLUTION_ARCHITECTURE_SPEC.md)
|
||||||
|
- Use cases + 4-quadrant framework: [full_usecase_planuml.md](../../workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/Use_Case/full_usecase_planuml.md), [CONTEXT_FR_25_UC_ELIT.md](../Requirement_Analysis/UC_Design/FR_25_UC_DESIGN/CONTEXT_FR_25_UC_ELIT.md)
|
||||||
|
- Risk scenarios: [FR_25_UC_DESIGN/](../Requirement_Analysis/UC_Design/FR_25_UC_DESIGN/) (Q1–Q4 scenario docs)
|
||||||
|
- Impact narrative: [Motivation.md](../DEMO_EXP/Motivation.md)
|
||||||
|
|
||||||
|
*End of proposal.*
|
||||||
37
session_memory/12_Jun_26.md
Normal file
37
session_memory/12_Jun_26.md
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
# Session Memory
|
||||||
|
|
||||||
|
## Current task
|
||||||
|
Created Kilo skill `cicd-architect-grill`.
|
||||||
|
|
||||||
|
## Created skill
|
||||||
|
- Skill name: `cicd-architect-grill`
|
||||||
|
- Skill file: `.kilocode/skills/cicd-architect-grill/SKILL.md`
|
||||||
|
- Skill line count: 49 lines after long draft compression.
|
||||||
|
|
||||||
|
## Skill purpose
|
||||||
|
Design, redesign, migrate, debate CI/CD pipelines for one software project. No cloud/language/runtime/platform assumptions.
|
||||||
|
|
||||||
|
## Skill behavior
|
||||||
|
- Inspect repo context before questions.
|
||||||
|
- Extract FRs, NFRs, constraints, success criteria, Definition of Done.
|
||||||
|
- Use C4 modeling only as deep as needed.
|
||||||
|
- Choose CI/CD pattern from situation, not vendor defaults.
|
||||||
|
- Cross-check: build reproducibility, tests, scans, artifacts, secrets, promotion, approvals, rollback, observability, cost, ownership.
|
||||||
|
- Grill one decision at a time when requested.
|
||||||
|
- Output draft/final plans as YAML.
|
||||||
|
|
||||||
|
## Repository context
|
||||||
|
- Workspace root: `/Users/davestran/Downloads/vkist_internship`
|
||||||
|
- Kilo config: `kilo.json`
|
||||||
|
- User-facing style: caveman style active.
|
||||||
|
- Existing skills inspected under `/Users/davestran/Downloads/vkist_internship/.kilocode/skills/`.
|
||||||
|
|
||||||
|
## Important constraints
|
||||||
|
- Do not assume GCP, AWS, Azure, Kubernetes, serverless, GitHub Actions, GitLab CI, Jenkins, Terraform unless user asks.
|
||||||
|
- Do not implement infrastructure until user approves design plan + implementation path.
|
||||||
|
- Do not ask many questions when repo evidence answers.
|
||||||
|
|
||||||
|
## Current status
|
||||||
|
- New skill directory exists.
|
||||||
|
- `SKILL.md` written and inspected.
|
||||||
|
- No tests run: Kilo skill definition, not executable code.
|
||||||
43
session_memory/13_Jun_26.md
Normal file
43
session_memory/13_Jun_26.md
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
# Session Memory
|
||||||
|
|
||||||
|
## Current task
|
||||||
|
Create session memory for 13 Jun 26.
|
||||||
|
|
||||||
|
## Work done
|
||||||
|
- Refined `API_CONTRACT_DRAFT.md` to v0.2.0 for FR-25 Synovitis Grading Workspace.
|
||||||
|
- Added public clinical routes + internal/local safety routes from API contract.
|
||||||
|
- Added async analysis job model, model registry, GradCAM, rationale, Socratic, BERT drift, RAG evidence, activations, artifact masks, ground truth, escalation, morphology, telemetry, report sign, EMR sync.
|
||||||
|
- Updated error codes, data types, OpenAPI skeleton, PlantUML sequence.
|
||||||
|
- Ran `git diff --check`; no whitespace errors.
|
||||||
|
|
||||||
|
## Architecture review
|
||||||
|
- Loaded `improve-codebase-architecture` skill.
|
||||||
|
- Read `LANGUAGE.md`; no repo `CONTEXT.md`; no `docs/adr`.
|
||||||
|
- Explore agent inspected `vkist-ultrasound/app.py`, `pdf_service.py`, `templates/js/script.js`, API docs, FR-25 use cases.
|
||||||
|
- Found 6 deepening candidates:
|
||||||
|
- Clinical Workflow State Machine Module — Strong.
|
||||||
|
- Model Registry + Runtime Adapter Module — Strong.
|
||||||
|
- Measurement + Severity Calibration Module — Strong.
|
||||||
|
- Privacy + Artifact + Audit Module — Strong.
|
||||||
|
- Finalization + Report + EMR State Module — Strong.
|
||||||
|
- Safety + Telemetry + Feedback Module — Worth exploring.
|
||||||
|
- HTML report write was attempted, but tool aborted before file landed. Report not completed.
|
||||||
|
|
||||||
|
## Repository context
|
||||||
|
- Workspace root: `/Users/davestran/Downloads/vkist_internship`
|
||||||
|
- User-facing style: caveman style active.
|
||||||
|
- Main code area: `PILOT_PROJECT/vkist-ultrasound`
|
||||||
|
- API contract target: `PILOT_PROJECT/workspace/sprint_1_2/Design_Material/API-docs/API_CONTRACT_DRAFT.md`
|
||||||
|
|
||||||
|
## Important constraints
|
||||||
|
- Do not propose interfaces before user picks architecture candidate.
|
||||||
|
- If user picks candidate, enter grilling loop.
|
||||||
|
- If naming new deepened module after concept missing from `CONTEXT.md`, update `CONTEXT.md`.
|
||||||
|
- If user rejects candidate with load-bearing reason, offer ADR only if future reviewers need it.
|
||||||
|
- Do not commit unless explicitly asked.
|
||||||
|
|
||||||
|
## Current status
|
||||||
|
- API contract file updated.
|
||||||
|
- Architecture candidates identified.
|
||||||
|
- HTML architecture report still pending due aborted write.
|
||||||
|
- No tests run for architecture review; no code changed during review.
|
||||||
60
session_memory/14_Jun_26.md
Normal file
60
session_memory/14_Jun_26.md
Normal file
@@ -0,0 +1,60 @@
|
|||||||
|
# Session Memory
|
||||||
|
|
||||||
|
## Current task
|
||||||
|
Create/update `grill_bento_codebase` architecture skill.
|
||||||
|
|
||||||
|
## Work done
|
||||||
|
- Loaded `grill-with-docs`, `grill-bento-codebase`, `skill-creator`.
|
||||||
|
- Inspected:
|
||||||
|
- `.kilocode/skills/grill_bento_codebase/SKILL.md`
|
||||||
|
- `.kilocode/skills/grill_bento_codebase/GEMINI.md`
|
||||||
|
- `.kilocode/skills/grill_bento_codebase/BENTO_DESIGN_GUIDELINE.md`
|
||||||
|
- `.kilocode/skills/grill_bento_codebase/bento_template.html`
|
||||||
|
- `.kilocode/rules/bento_template.html`
|
||||||
|
- `.kilocode/rules/BENTO_DESIGN_GUIDELINE.md`
|
||||||
|
- Updated `grill_bento_codebase` skill assets.
|
||||||
|
- Added Bento depth constraint:
|
||||||
|
- one Bento room may contain multiple sub-components/sub-projects
|
||||||
|
- max Bento depth = 3
|
||||||
|
- depth 1 = root Bento
|
||||||
|
- depth 2 = child sub-project
|
||||||
|
- depth 3 = grandchild sub-project
|
||||||
|
- depth 4+ invalid
|
||||||
|
- Added fallback rule: detail beyond depth 3 stays inside owning Bento as local `spec/`, `spec/docs/`, `implementation/`, or `tests/`.
|
||||||
|
- Moved skill-local source refs to:
|
||||||
|
- `BENTO_DESIGN_GUIDELINE.md`
|
||||||
|
- `bento_template.html`
|
||||||
|
- Bumped guideline version to `0.5.0`.
|
||||||
|
|
||||||
|
## Skill output contract
|
||||||
|
- Context lock from FRS, NFRS, user-interaction, Software_Arch, Solution_Arch.
|
||||||
|
- Bento-BEV HTML blueprint from `bento_template.html`.
|
||||||
|
- Markdown tree-view for dev-team flatout navigation.
|
||||||
|
- Room cards.
|
||||||
|
- Interface contracts.
|
||||||
|
- Dependency map.
|
||||||
|
- Traceability matrix.
|
||||||
|
- Open decisions, one at a time.
|
||||||
|
|
||||||
|
## Validation done
|
||||||
|
- Re-read full `.kilocode/skills/grill_bento_codebase/BENTO_DESIGN_GUIDELINE.md`.
|
||||||
|
- Confirmed depth rule appears in:
|
||||||
|
- `BENTO_DESIGN_GUIDELINE.md`
|
||||||
|
- `SKILL.md`
|
||||||
|
- `GEMINI.md`
|
||||||
|
- Grep checked no stale `.kilocode/rules` refs inside `.kilocode/skills/grill_bento_codebase`.
|
||||||
|
|
||||||
|
## Repository context
|
||||||
|
- Workspace root: `/Users/davestran/Downloads/vkist_internship`
|
||||||
|
- User-facing style: caveman style active.
|
||||||
|
- Skill area: `.kilocode/skills/grill_bento_codebase`
|
||||||
|
|
||||||
|
## Important constraints
|
||||||
|
- No Bento tree depth > 3.
|
||||||
|
- No nested Bento beyond grandchild depth.
|
||||||
|
- Extra detail stays local to owning Bento room.
|
||||||
|
- No commits unless explicitly asked.
|
||||||
|
|
||||||
|
## Current status
|
||||||
|
- `SKILL.md`, `GEMINI.md`, `BENTO_DESIGN_GUIDELINE.md` updated.
|
||||||
|
- No tests run: skill docs, not executable code.
|
||||||
71
session_memory/15_Jul_26.md
Normal file
71
session_memory/15_Jul_26.md
Normal file
@@ -0,0 +1,71 @@
|
|||||||
|
# Session Knowledge — 2026-07-15
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
Debug why Celery batch API results are not displaying on the client UI and fix the full pipeline: server startup, worker task registration, Triton endpoint routing, polling storm, stale result re-submission.
|
||||||
|
|
||||||
|
## Constraints & Preferences
|
||||||
|
- Client should not hammer backend with polls; need request coalescing and caching.
|
||||||
|
- Already-completed batch results should not be re-submitted or re-queried on every page view.
|
||||||
|
- Worker must target Modal Triton endpoint, not localhost.
|
||||||
|
|
||||||
|
## Progress
|
||||||
|
### Done
|
||||||
|
- Fixed `GroupResult.restore()` returning `None` by adding `result.save()` after `apply_async()` in `submit_celery_batch()`.
|
||||||
|
- Fixed server startup `ModuleNotFoundError: No module named 'backend'` by correcting `CODEBASE_ROOT` path math in `backend/routers/run_cv_inference.sh` (changed `SCRIPT_DIR/..` → `SCRIPT_DIR/../..`).
|
||||||
|
- Added `celery_app.autodiscover_tasks(["backend.implementation.tasks"])` to `backend/services/celery_app.py` so worker registers `run_cv_chunk`.
|
||||||
|
- Added `export TRITON_ENDPOINT=...modal.run` to `backend/start_workers.sh` so worker targets Modal instead of `localhost:8000`.
|
||||||
|
- Added server-side TTL poll cache (`_batch_poll_cache`) in `backend/services/cv_celery_service.py` with default 2000ms TTL to coalesce rapid repeat polls.
|
||||||
|
- Increased client poll interval from 1000ms → 2000ms and removed unused `pollIntervalMs` parameter from `getCvAnalyzeResultsForProfileCelery()` in `frontend/implementation/src/lib/cvAnalyzeApi.ts`.
|
||||||
|
- Fixed poll endpoint HTTP status semantics: `GET /api/test/analyze/batch/celery/{job_id}` now returns `202 Accepted` for pending, `404 Not Found` for unknown, `200 OK` for completed/failed. This removes the misleading-200 problem for in-flight jobs.
|
||||||
|
- Added backend timing logs in `cv_celery_service.py`: submission logs chunk/image counts; completion/failure logs `duration_ms` from submission to terminal state.
|
||||||
|
- Added frontend timing logs in `getCvAnalyzeResultsForProfileCelery()` for completed/failed/unknown outcomes.
|
||||||
|
- Migrated frontend config from `.env`/`.env.development` to single YAML source of truth at `config/frontend.config.yaml`, loaded via `js-yaml` in `vite.config.ts` using Vite `define`. Removed old `.env` files. Added `VITE_MODAL_OLLAMA_TARGET` to YAML so proxy target is config-driven.
|
||||||
|
- Fixed TypeScript errors in `vite.config.ts` by adding `"types": ["node"]` and including `vite.config.ts` in `tsconfig.json`.
|
||||||
|
- Added client-side in-flight coalescing + completed-result cache in `getCvAnalyzeResultsForProfileCelery()` so repeat page views don’t re-submit identical work.
|
||||||
|
|
||||||
|
### In Progress
|
||||||
|
- Worker needs force-restart to pick up `autodiscover_tasks` + `TRITON_ENDPOINT` changes.
|
||||||
|
- One submitted job reached `completed: 1/2` but never finished — need worker restart + fresh batch test.
|
||||||
|
- Client-side ETA estimation for pending batch jobs is planned but not yet implemented.
|
||||||
|
|
||||||
|
### Blocked
|
||||||
|
- Worker process restart — old PID/log still active without the new fixes applied.
|
||||||
|
|
||||||
|
## Key Decisions
|
||||||
|
- Used `result.save()` because Celery does not auto-persist `GroupResult` metadata to Redis.
|
||||||
|
- Added `autodiscover_tasks` instead of manual imports because worker is launched via `-A backend.services.celery_app` and never imports `cv_tasks.py` otherwise.
|
||||||
|
- Set worker `TRITON_ENDPOINT` in `start_workers.sh` because worker is separate process that doesn’t run `cv_inference_server.py`’s `os.environ.setdefault()`.
|
||||||
|
- Poll cache TTL set to 2000ms to match 2s client poll interval.
|
||||||
|
- Single YAML config replaces `.env` + `.env.development` to eliminate precedence confusion.
|
||||||
|
- `VITE_MODAL_OLLAMA_TARGET` lives in YAML so proxy target is editable without touching `vite.config.ts` code.
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
- Force-restart Celery worker (`kill -9` stale PID, remove `celery.pid`, run `./start_workers.sh start`).
|
||||||
|
- Verify worker log shows `backend.implementation.tasks.cv_tasks.run_cv_chunk` under `[tasks]`.
|
||||||
|
- Submit fresh batch and confirm it transitions `pending` → `completed` with actual results.
|
||||||
|
- Implement ETA estimation in backend poll response based on observed chunk throughput.
|
||||||
|
- Address client behavior that submits new jobs while previous jobs are still `pending` (7+ concurrent jobs observed in logs).
|
||||||
|
|
||||||
|
## Critical Context
|
||||||
|
- Latest server logs show 7+ concurrent jobs all stuck at `pending` with `completed: 0/2` or `1/2`, plus new POST submissions every ~10s — indicating worker is not processing tasks.
|
||||||
|
- Worker log still shows stale startup at `16:22:18` with empty `[tasks]`; fixes in `celery_app.py` and `start_workers.sh` have not been loaded by running worker.
|
||||||
|
- One job (`13fd014e`) reached `completed: 1/2, progress: 0.5` and stalled — second chunk likely failed or was never picked up.
|
||||||
|
- Modal Triton endpoint: `https://dtj-tran--triton-s3-service-unified-triton-server.modal.run`.
|
||||||
|
- Redis broker/backend: `redis://localhost:6379/0`.
|
||||||
|
- Chunk size default: 4 frames per chunk (`CELERY_CHUNK_SIZE`).
|
||||||
|
- `cv_result_cache.py` already exists with in-flight coalescing pattern; new batch poll cache follows same design.
|
||||||
|
- Frontend not currently in Celery mode (`VITE_USE_CV_CELERY=false` in YAML).
|
||||||
|
- Direct batch route still shows noisy fallback logs when Triton returns 502 on batched inference — fallback works, but should log at `info` not `warning`.
|
||||||
|
|
||||||
|
## Relevant Files
|
||||||
|
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/services/cv_celery_service.py`: Contains `submit_celery_batch()` (fixed with `result.save()`), `get_celery_batch_result()` (now with 2000ms TTL poll cache + timing logs), and poll cache constants.
|
||||||
|
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/services/celery_app.py`: Added `autodiscover_tasks(["backend.implementation.tasks"])`; routes `cv_inference.run_chunk` to `cv-inference` queue.
|
||||||
|
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/start_workers.sh`: Added `TRITON_ENDPOINT` export so worker targets Modal.
|
||||||
|
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/routers/run_cv_inference.sh`: Fixed `CODEBASE_ROOT` path math (`SCRIPT_DIR/../..`) so `PYTHONPATH` resolves `backend` package.
|
||||||
|
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/implementation/tasks/cv_tasks.py`: Defines `run_cv_chunk` Celery task.
|
||||||
|
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/routers/cv_inference.py`: Contains `/api/test/analyze/batch/celery` endpoints. Poll endpoint now returns 202/404/200.
|
||||||
|
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/src/lib/cvAnalyzeApi.ts`: Poll interval 2000ms; client-side cache + in-flight coalescing; timing logs.
|
||||||
|
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/src/hooks/useSegmentationOverlay.ts`: Calls `getCvAnalyzeResultsForProfileCelery()`.
|
||||||
|
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/config/frontend.config.yaml`: Single source of truth for frontend feature flags and URLs.
|
||||||
|
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/vite.config.ts`: Loads YAML config and injects as `import.meta.env.*`.
|
||||||
|
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/tsconfig.json`: Added `"types": ["node"]` and included `vite.config.ts`.
|
||||||
45
session_memory/18_Jun_26.md
Normal file
45
session_memory/18_Jun_26.md
Normal file
@@ -0,0 +1,45 @@
|
|||||||
|
---
|
||||||
|
title: Session Memory 18 Jun 26
|
||||||
|
date: 2026-06-18
|
||||||
|
status: active
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Compressed 2026-06-18 session work: `cv_code` split, PlantUML Chen ER skill, session memory.
|
||||||
|
|
||||||
|
## cv_code module split
|
||||||
|
|
||||||
|
- Split legacy `legcode/app.py` into `ml/implementation/cv/code/` into three modules:
|
||||||
|
- `config.py`
|
||||||
|
- `framework.py`
|
||||||
|
- `server.py`
|
||||||
|
- Legacy file untouched until new modules pass import checks.
|
||||||
|
- `python -m py_compile` passed all files.
|
||||||
|
- `server.py` import blocked by missing `pdf_service`.
|
||||||
|
- `framework.py` path bug: `cv_root = Path(__file__).resolve().parents[2]` skips too far. Correct: `parents[1]`.
|
||||||
|
|
||||||
|
## skill-creator inspection
|
||||||
|
|
||||||
|
- Loaded `.config/kilo/.kilo/skills/skill-creator/SKILL.md`.
|
||||||
|
- Skill structure: `SKILL.md` required; optional `GEMINI.md`, `EXAMPLES.md`, `REFERENCE.md`, `scripts/`, `assets/`.
|
||||||
|
- Template description rules apply.
|
||||||
|
|
||||||
|
## er-chen-diagram skill
|
||||||
|
|
||||||
|
- Created `.kilocode/skills/er-chen-diagram/` scaffold.
|
||||||
|
- Files written: `SKILL.md`, `EXAMPLES.md`, `REFERENCE.md`.
|
||||||
|
- Skill contents:
|
||||||
|
- Mandatory `@startchen` / `@endchen`.
|
||||||
|
- Attributes only inside `{}`.
|
||||||
|
- Primary keys use `<<key>>`.
|
||||||
|
- Relationships empty blocks, then connect lines.
|
||||||
|
- Copy exact code shells verbatim.
|
||||||
|
|
||||||
|
## Session memory housekeeping
|
||||||
|
|
||||||
|
- Listed existing memory files: `12_Jun_26.md`, `13_Jun_26.md`, `14_Jun_26.md`.
|
||||||
|
- Recorded activity in this file.
|
||||||
|
|
||||||
|
## Relational DB Design
|
||||||
|
- Created relational DB design spec per plan: ER/schema PlantUML and Chen notation in RELATIONAL_DB_SCHEMA_SPEC.md.
|
||||||
@@ -0,0 +1,159 @@
|
|||||||
|
# Coding Agent Instruction: Resolve 17 Remaining Structural Gaps in Phase 0 & Phase 1
|
||||||
|
|
||||||
|
You have already fixed the TypeScript build failures. The 17 unchecked plan items below must be addressed structurally. Each item is a checklist entry in `.kilo/plans/35-figma-designs-implementation-plan.md`. Only mark an item `[x]` in that plan file after you have implemented it AND it compiles cleanly.
|
||||||
|
|
||||||
|
## Reference Files
|
||||||
|
- Implementation Plan: `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/.kilo/plans/35-figma-designs-implementation-plan.md`
|
||||||
|
- Frontend root: `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend`
|
||||||
|
- Severity rule: Do NOT mark `[x]` unless you have implemented the item AND verified it.
|
||||||
|
|
||||||
|
## Work Order: Exact Sequence with Acceptance Tests
|
||||||
|
|
||||||
|
### 0.1 Environment Tooling
|
||||||
|
1. Configure ESLint + Prettier
|
||||||
|
- Add `.eslintrc.cjs` and `.prettierrc` (or extend recommended configs).
|
||||||
|
- Add scripts to `package.json`: `"lint": "eslint src --ext .ts,.tsx"`.
|
||||||
|
- Acceptance: `npm run lint` runs without failing on the current src tree (warnings OK, but no auto-fixable errors left hanging).
|
||||||
|
|
||||||
|
2. Set up Husky
|
||||||
|
- Install `husky` + `lint-staged`.
|
||||||
|
- Add `prepare` script: `"prepare": "husky"`.
|
||||||
|
- Add `.husky/pre-commit` to run `lint-staged`.
|
||||||
|
- Acceptance: `git add src/ && git commit` triggers Husky and blocks on lint failure.
|
||||||
|
|
||||||
|
3. Configure Vitest
|
||||||
|
- Install: `vitest`, `@testing-library/react`, `@testing-library/jest-dom`, `jsdom`.
|
||||||
|
- Add `vitest.config.ts` with `test.environment = 'jsdom'` and `include: ['src/**/*.{test,spec}.{js,ts,jsx,tsx}']`.
|
||||||
|
- Add script: `"test": "vitest"`.
|
||||||
|
- Acceptance: `npx vitest run` executes (0 tests is OK for now; framework must be correct).
|
||||||
|
|
||||||
|
4. Set up Playwright
|
||||||
|
- Install `@playwright/test`.
|
||||||
|
- Add `playwright.config.ts` with baseURL `http://localhost:5173` and webServer using `vite preview` or same.
|
||||||
|
- Add script: `"test:e2e": "playwright test"`.
|
||||||
|
- Acceptance: `npx playwright test --list` does not error (no tests yet is OK).
|
||||||
|
|
||||||
|
5. Configure Storybook
|
||||||
|
- Install `@storybook/react-vite` and run `storybook init`.
|
||||||
|
- Add 1 story for `Button` atom and 1 story for `TopBar` organism.
|
||||||
|
- Acceptance: `npm run storybook` builds the UI and the 2 stories render.
|
||||||
|
|
||||||
|
### 0.3 Component Inventory
|
||||||
|
6. Analyze 35 Figma Files
|
||||||
|
- Inspect Figma paths under `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/Design_Material/` - section: `Figma Design Source Files`.
|
||||||
|
- Produce an inventory mapping file: `spec/figma-component-specs.md` is insufficient; append `spec/component-inventory.md` with:
|
||||||
|
- Unique screen count (should be 35).
|
||||||
|
- Reusable component list by atomic depth (atoms, molecules, organisms, templates).
|
||||||
|
- One dependency graph (Mermaid is fine).
|
||||||
|
- Acceptance: `spec/component-inventory.md` exists, has 35 distinct screens, and has a Mermaid dependency graph.
|
||||||
|
|
||||||
|
7. Build Storybook Stories for Primitives
|
||||||
|
- Add stories for `Button`, `Input`, `Icon`, `Badge`, `Label`, `Toggle`, `Tooltip`.
|
||||||
|
- Acceptance: All primitives render in Storybook.
|
||||||
|
|
||||||
|
### 0.4 API Contracts
|
||||||
|
8. Cross-Reference API Contracts
|
||||||
|
- Open `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/Design_Material/API_docs/API_CONTRACT_DRAFT.md` & the `API spec from the backend-file` in `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/spec/interface-contract.md` & the data-layer definition - check the reference from context in the dirrectory (a folder) at `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/data/spec`
|
||||||
|
- Cross-check each endpoint against `src/services/*.ts`.
|
||||||
|
- Acceptance: Each service file has JSDoc lists of endpoints it covers.
|
||||||
|
|
||||||
|
9. Define TypeScript Interfaces
|
||||||
|
- Add `src/types/api.ts` with request/response interfaces for all 7 services.
|
||||||
|
- Acceptance: All 7 services import and use their types.
|
||||||
|
|
||||||
|
10. Implement HTTP + MSW Adapters
|
||||||
|
- Add `src/services/adapters/httpAdapter.ts` and `src/services/adapters/mswAdapter.ts`.
|
||||||
|
- Each service file must use the adapter via `services/index.ts` seam.
|
||||||
|
- Acceptance: `tsc --noEmit` clean; `vitest run` can mock the adapter without touching prod code.
|
||||||
|
|
||||||
|
11. Implement Auth/Logging Interceptors
|
||||||
|
- In `httpAdapter.ts`, add request interceptor that:
|
||||||
|
- Attaches `Authorization` header from a session token (mocked OK).
|
||||||
|
- Logs request method + URL.
|
||||||
|
- Normalizes errors to a `ServiceError` class.
|
||||||
|
- Acceptance: Interceptor exists and is imported only by `httpAdapter.ts` (no component imports).
|
||||||
|
|
||||||
|
### 1.2 Zustand Slices
|
||||||
|
12. Co-locate Slice Tests
|
||||||
|
- Create `src/store/sessionSlice.test.ts`, `src/store/dicomSlice.test.ts`, etc. (6 files).
|
||||||
|
- Each test imports the slice factory (`createSessionSlice`) and invokes the interface.
|
||||||
|
- Acceptance: `vitest run --coverage` shows coverage files for each slice.
|
||||||
|
|
||||||
|
13. Create Domain Selectors
|
||||||
|
- In each slice file, export type-safe selectors via `create`.
|
||||||
|
- Acceptance: `TopBar` and `BottomBar` consume slice selectors instead of inline memoization.
|
||||||
|
|
||||||
|
### 1.3 WebWorkers
|
||||||
|
14. Create `src/workers/` Files
|
||||||
|
- `cv.worker.ts`: stub with `onmessage` and `postMessage` for `preprocess` msg type.
|
||||||
|
- `llm.worker.ts`: stub with `generate` msg type.
|
||||||
|
- `guardrail.worker.ts`: stub with `safetyCheck` msg type.
|
||||||
|
- Acceptance: `tsc --noEmit` clean; workers can be imported in a hook.
|
||||||
|
|
||||||
|
15. Implement Worker Pool + Hooks
|
||||||
|
- `src/hooks/useWebWorker.ts`: generic hook accepting worker class + message schema.
|
||||||
|
- `src/hooks/useCVWorker.ts`, `useLLMWorker.ts`, `useGuardrailWorker.ts`: domain wrappers.
|
||||||
|
- Acceptance: Hooks can be instantiated without runtime errors in a test render.
|
||||||
|
|
||||||
|
16. Transferable Message Protocols
|
||||||
|
- In each worker stub, explicitly show `Transferable` usage (pass `ImageBitmap` or `ArrayBuffer` in transfer list).
|
||||||
|
- Acceptance: TypeScript types enforce transferable contracts.
|
||||||
|
|
||||||
|
17. Fallback to Main Thread
|
||||||
|
- Each hook exports `isSupported: boolean`. If `Worker` constructor not available, use a mock implementation on main thread.
|
||||||
|
- Acceptance: In jsdom test, `isSupported` is false and hook still resolves.
|
||||||
|
|
||||||
|
### 1.4 Services Deepening
|
||||||
|
18. Move `encryption.ts` Out of the Store
|
||||||
|
- Remove `import { encryptStorage } from '../utils/encryption'` from `src/store/index.ts`.
|
||||||
|
- Create `src/services/storageAdapter.ts` that exports `createSessionStorage` and `createEncryptedStorage`.
|
||||||
|
- `store/index.ts` must import `createJSONStorage(() => storageAdapter)` from `services/storageAdapter`.
|
||||||
|
- Acceptance: `grep encryptStorage src/store/index.ts` returns nothing in src/store/.
|
||||||
|
|
||||||
|
### 1.5 Route-Aware Shell
|
||||||
|
19. Implement `protected.tsx`
|
||||||
|
- Add `src/routes/protected.tsx` with a `<RequireAuth>` wrapper that redirects to `/login` when `!sessionId`.
|
||||||
|
- Wrap `/workspace` in `<RequireAuth>`.
|
||||||
|
- Acceptance: Visit `/workspace` when unauthenticated; browser redirects to `/login`.
|
||||||
|
|
||||||
|
20. `WorkspaceShell` Owns 65/35 Split
|
||||||
|
- `WorkspaceShell.tsx` must render the 65/35 layout directly: `div.flex.h-screen` with `w-[65%]` ZoneA and `w-[35%]` ZoneB.
|
||||||
|
- `routes/index.tsx` `<DiagnosticWorkspace>` must only render `<WorkspaceShell />` without re-implementing the split.
|
||||||
|
- Acceptance: `grep "flex-1" src/components/templates/WorkspaceShell.tsx` finds exactly one instance; `grep "ZoneA" src/routes/index.tsx` finds zero (or just an import).
|
||||||
|
|
||||||
|
21. Lift `initApp` Out of `App.tsx`
|
||||||
|
- Add `src/hooks/useAppInit.ts`.
|
||||||
|
- Move the `initApp` effect into this hook.
|
||||||
|
- `App.tsx` becomes: `<AppRoutes />` plus `<AppInit />` or composes it.
|
||||||
|
- Acceptance: `grep "initApp" src/App.tsx` finds nothing.
|
||||||
|
|
||||||
|
22. `React.lazy` + `Suspense` in Routes
|
||||||
|
- Use `React.lazy(() => import('../components/organisms/ZoneA'))` and same for `ZoneB`.
|
||||||
|
- Wrap `<DiagnosticWorkspace>` children in `<Suspense fallback={<div>Loading...</div>}>`.
|
||||||
|
- Acceptance: `grep React.lazy src/routes/index.tsx` finds 2+ matches.
|
||||||
|
|
||||||
|
### 1.6 Layout Responsiveness
|
||||||
|
23. Responsive Breakpoints
|
||||||
|
- `WorkspaceShell.tsx` must use `@media` or Tailwind responsive classes so that on `< md` the 65/35 stack becomes a tabbed/toggle layout (Zone A | Zone B).
|
||||||
|
- Acceptance: Resize viewport to 375px; only one Zone visible at a time.
|
||||||
|
|
||||||
|
24. 60fps Rendering Hints
|
||||||
|
- Add `will-change: transform` to animated containers.
|
||||||
|
- Add `contain: layout style paint` to the workspace root.
|
||||||
|
- Acceptance: `grep -E "will-change|contain:" src/components/templates/WorkspaceShell.tsx` finds both.
|
||||||
|
|
||||||
|
## Strict Consequences
|
||||||
|
- If you cannot complete a specific item within this prompt, do not leave it half-done. Leave the plan checkbox `[ ]` and add a comment in code explaining what is blocking you.
|
||||||
|
- All code must keep `tsc --noEmit` clean after your changes.
|
||||||
|
- Do NOT touch backend files or any file outside `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend` unless an item explicitly requires it.
|
||||||
|
|
||||||
|
## Final Verification (YOU MUST RUN THESE)
|
||||||
|
```bash
|
||||||
|
cd /Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend
|
||||||
|
npx tsc --noEmit && npm run build
|
||||||
|
npm run lint
|
||||||
|
npx vitest run
|
||||||
|
npx playwright test --list
|
||||||
|
npm run storybook -- --headless 2>&1 | tail -20
|
||||||
|
```
|
||||||
|
Produce a summary of which commands passed/failed and why.
|
||||||
189
session_memory/26_Jun_26/Proj_context_26_Jun_26.md
Normal file
189
session_memory/26_Jun_26/Proj_context_26_Jun_26.md
Normal file
@@ -0,0 +1,189 @@
|
|||||||
|
# SESSION MEMORY — 26 Jun 26
|
||||||
|
|
||||||
|
## APP PURPOSE
|
||||||
|
VKIST MSK Platform — air-gapped musculoskeletal ultrasound analysis. FR-25 Synovitis Grading (knee). Vietnamese NLP. RAG citations. HITL finalization. ≤150 MB bundle. ≤1.5s inference.
|
||||||
|
|
||||||
|
## ARCHITECTURE (C4)
|
||||||
|
|
||||||
|
### Context
|
||||||
|
- **UP5** Radiologist: primary user — load scan, review grade, finalize, sign, view GradCAM, engage circuit breaker, review RAG evidence
|
||||||
|
- **UP1** Senior Expert: clinical protocol validation, MOH guideline approval, threshold sign-off
|
||||||
|
- **UP4** Support: registration, case queue
|
||||||
|
- External: PACS (DICOM), EMR/HIS (HL7/FHIR), Triton (GPU inference), ladybugDB (ontology), pgvector (MOH guidelines HNSW), GemmaE2B/MedGemma (LLM), EmbeddingGemma (RAG embeddings)
|
||||||
|
|
||||||
|
### Containers
|
||||||
|
- **PWA** React 18 + TS + Zustand + LiteRT + MediaPipe + Dexie.js. DICOM canvas, GradCAM overlay, edge guardrail (Transformers.js BERT + OpenRedaction + pii-filter) in WebWorkers.
|
||||||
|
- **NGINX + Keepalived** VIP failover ≤2s, SSL termination
|
||||||
|
- **FastAPI** edge-inference-svc: DICOM ingest, ML pipeline orchestration, Grad-CAM, report assembly, SSE streaming
|
||||||
|
- **FastAPI** rag-svc: pgvector top-k retrieval, ladybugDB ontology, LLM consult route, RAG-Referee
|
||||||
|
- **Postgres + pgvector** HNSW index for MOH guidelines
|
||||||
|
- **Redis** JWT sessions, rate-limit, consult_mode state
|
||||||
|
- **MinIO** S3: DICOM, overlays, GradCAM heatmaps, reports
|
||||||
|
- **Triton** gRPC :8001: 3-step ML pipeline + embeddings
|
||||||
|
|
||||||
|
### 3-STEP ML PIPELINE (Triton)
|
||||||
|
1. **Angle Classification** — ConvNeXt/DenseNet/ResNet/EfficientNet/Swin-V2. Classes: med-lat, post-trans, sup-trans-flex, sup-up-long
|
||||||
|
2. **Inflammation Detection** — EfficientNet-B0 binary (post-trans, sup-up-long only)
|
||||||
|
3. **Segmentation** — DeepLabV3-ResNet101 (post-trans) OR UNet3Plus-Attention/DeepLabV3 (sup-up-long). Classes: background, effusion, fat, femur, synovium, tendon
|
||||||
|
4. **Measurement** — ROI middle 1/3 of bounding box. PIXEL_TO_MM = 45/655
|
||||||
|
5. **Severity** — Combined: effusion 60% + synovium 40%. Grades 0-3: Rất nhẹ/Nhẹ/Trung bình/Nặng
|
||||||
|
|
||||||
|
### GRADCAM
|
||||||
|
- Zero extra clicks (auto on upload)
|
||||||
|
- Base64 PNG from FastAPI
|
||||||
|
- Canvas overlay, adjust opacity
|
||||||
|
|
||||||
|
### WEB WORKER TOPOLOGY
|
||||||
|
- `cv.worker.ts` — LiteRT WASM: CV inference (angle pre-classifier). Isolated WASM.
|
||||||
|
- `llm.worker.ts` — WebLLM WASM: GemmaE2B local generation. NFR-4 1.5GB heap.
|
||||||
|
- `guardrail.worker.ts` — Transformers.js WASM/WebGPU: BERT hallucination/mal-intention/scope-breach scoring.
|
||||||
|
|
||||||
|
**Isolation:** No SharedArrayBuffer/Atomics. postMessage with structured clone only.
|
||||||
|
|
||||||
|
## FRONTEND DESIGN (WS-25)
|
||||||
|
|
||||||
|
### Layout: 65/35 split (Zone A: DICOM | Zone B: Results + Chat)
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─ Top Bar (patient ID, back, sign) ──────────────────────┐
|
||||||
|
├─ Zone A (65%) ────────┬─ Zone B (35%) ────────────────────┤
|
||||||
|
│ │ Phase Indicator (sticky) │
|
||||||
|
│ 5 Canvas Layers: │ Classification Card (grade,conf) │
|
||||||
|
│ 1. Background │ Explanation Panel (GradCAM thumb) │
|
||||||
|
│ 2. GradCAM │ Chat Panel (Socratic dialogue) │
|
||||||
|
│ 3. Segmentation │ Override Card (Phase 4) │
|
||||||
|
│ 4. Annotations (SVG) │ Quick Actions │
|
||||||
|
│ 5. Viewport HUD │ │
|
||||||
|
│ │ │
|
||||||
|
│ Toolbar: pan/zoom/ │ │
|
||||||
|
│ brush/eraser/caliper │ │
|
||||||
|
├───────────────────────┴───────────────────────────────────┤
|
||||||
|
└─ Bottom Bar (NFR status, EMR sync, offline indicator) ─────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Canvas Layers (z-index)
|
||||||
|
- 0: Background DICOM (render once, never redraw on pan/zoom)
|
||||||
|
- 10: GradCAM (CSS transform, no redraw)
|
||||||
|
- 50: Segmentation overlay
|
||||||
|
- 100: SVG Annotations (brush/caliper)
|
||||||
|
- 200: Viewport HUD (text, mousemove redraw)
|
||||||
|
- 900-999: System chrome (top/bottom bar)
|
||||||
|
|
||||||
|
### Zustand Store Keys
|
||||||
|
- `sessionId`, `patientId`, `currentPhase`
|
||||||
|
- `dicomFrame: ImageBitmap | null` (GPU-friendly)
|
||||||
|
- `dicomMetadata`, `mlResults` (single atomic object)
|
||||||
|
- `clinicianGrade`, `isOverriding`, `overrideJustification`
|
||||||
|
- `maskedRegions: MaskedRegion[]` (SVG path data, not raster)
|
||||||
|
- `chatMessages` (pruned to 20, older → IndexedDB)
|
||||||
|
- `driftScore`, `refereeIntervened`
|
||||||
|
- `zoomLevel`, `panOffset` (not persisted, RAF updates)
|
||||||
|
- `activeTool: pan|zoom|brush|eraser|caliper|pin`
|
||||||
|
- `showGradCAM`, `showSegmentation`
|
||||||
|
|
||||||
|
### 6-PHASE WORKFLOW
|
||||||
|
1. **Classification** — ML grade, confidence bar
|
||||||
|
2. **Explanation** — GradCAM visible, RAG evidence, LLM rationale
|
||||||
|
3. **Negotiation** — Socratic dialogue, circuit breaker, BERT drift
|
||||||
|
4. **Reconfiguration** — Override card, pixel activation inspector
|
||||||
|
5. **Final Decision** — Summary, HITL signature
|
||||||
|
6. **EMR Sync** — HL7/FHIR push, audit hash
|
||||||
|
|
||||||
|
## KEY API CONTRACTS
|
||||||
|
|
||||||
|
### Core Endpoints
|
||||||
|
- `POST /api/v1/sessions` → create session
|
||||||
|
- `POST /api/v1/sessions/{id}/frames` → upload DICOM (multipart)
|
||||||
|
- `POST /api/v1/analysis-jobs` → trigger ML pipeline
|
||||||
|
- `GET /api/v1/analysis-jobs/{id}` → get result (angle/inflammation/segmentation/measurement/severity)
|
||||||
|
- `GET /api/v1/analysis-jobs/{id}/steps` → step-level status
|
||||||
|
- `PATCH /api/v1/sessions/{id}/review` → clinician review (approve/correct/reject)
|
||||||
|
- `POST /api/v1/reports` → generate report
|
||||||
|
- `POST /api/v1/reports/{id}/sign` → HITL digital signature
|
||||||
|
- `POST /api/v1/reports/{id}/emr-sync` → EMR push (outbox if offline)
|
||||||
|
|
||||||
|
### Safety Endpoints
|
||||||
|
- `POST /api/v1/sessions/{id}/explanations/gradcam` → GradCAM heatmap
|
||||||
|
- `POST /api/v1/sessions/{id}/safety/circuit-breaker` → trigger circuit breaker
|
||||||
|
- `POST /api/v1/sessions/{id}/chat/socratic` → Socratic dialogue (SSE streaming)
|
||||||
|
- `POST /api/v1/sessions/{id}/drift/check` → BERT semantic drift
|
||||||
|
- `POST /api/v1/sessions/{id}/rag/evidence` → RAG guideline arbitration
|
||||||
|
- `POST /api/v1/feedback` → ground-truth override telemetry
|
||||||
|
|
||||||
|
### ML Response Shape
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"result": {
|
||||||
|
"angle": { "class": "sup-up-long", "confidence": 0.9845 },
|
||||||
|
"inflammation": { "detected": true, "confidence": 0.942 },
|
||||||
|
"segmentation": { "mask_ref": "s3://...", "overlay_ref": "s3://...", "color_legend": {...} },
|
||||||
|
"measurement": { "thickness_mm": 6.87, "pixel_to_mm": 0.0687 },
|
||||||
|
"severity": { "level": 3, "label": "severe", "combined_score": 18.4 }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## NFR SUMMARY
|
||||||
|
|
||||||
|
| NFR | Target | Implementation |
|
||||||
|
|-----|--------|----------------|
|
||||||
|
| NFR-4 | ≤150 MB bundle | React PWA, tree-shake, split vendor chunk, Dexie.js |
|
||||||
|
| NFR-5 | ≤1.5s inference | Triton ONNX/TensorRT, client memory ≤150MB |
|
||||||
|
| NFR-7 | ≤200ms TTFT | SSE streaming, token chunking |
|
||||||
|
| NFR-8 | Fault tolerance | IndexedDB cache, SW offline mode, session recovery |
|
||||||
|
| NFR-13 | Grad-CAM zero-click | Auto on upload, no extra UI steps |
|
||||||
|
| NFR-14 | No client GPU | LiteRT WASM, CPU fallback |
|
||||||
|
| NFR-16 | Air-gapped primary | All inference + storage on-prem via K3s |
|
||||||
|
| NFR-17 | Immutable audit | PostgreSQL WAL, no UPDATE/DELETE triggers |
|
||||||
|
| NFR-18 | RAG citations | 100% LLM text cites MOH via pgvector HNSW |
|
||||||
|
| NFR-19 | HITL signature | `signer_id != NULL` constraint before FINALIZED |
|
||||||
|
|
||||||
|
## DECREE 13 / CIRCULAR 46
|
||||||
|
- Client: Decree 13 regex scrubber (OpenRedaction + pii-filter + js-data-anonymizer) before network
|
||||||
|
- Server: FastAPI `phi_scrub` middleware (Microsoft Presidio) as ground-check
|
||||||
|
- PDF: Circular 46/2018 format, bilingual VI/EN, audit hash, signer block
|
||||||
|
|
||||||
|
## SPRINT 1-2 DELIVERABLES
|
||||||
|
|
||||||
|
### Sprint 1 (Jun 15–26) — "Fast PoC Baseline"
|
||||||
|
- [ ] PWA shell: React + Zustand + PWA manifest + Service Worker + Dexie.js
|
||||||
|
- [ ] DICOM canvas: Cornerstone.js + 5-layer canvas stack
|
||||||
|
- [ ] FastAPI `/api/analyze` endpoint (sync 3-step pipeline)
|
||||||
|
- [ ] Triton 3-step model ensemble + gRPC
|
||||||
|
- [ ] NGINX + Keepalived VIP
|
||||||
|
- [ ] Grad-CAM overlay on primary viewport
|
||||||
|
- [ ] Circuit-breaker skeleton
|
||||||
|
|
||||||
|
### Sprint 2 (Jun 29–Jul 10) — "Multi-Modal & NLP Integration"
|
||||||
|
- [ ] GemmaE2B browser WebLLM + Cloud MedGemma Vertex AI
|
||||||
|
- [ ] BERT drift monitor + RAG-Referee
|
||||||
|
- [ ] Socratic dialogue (SSE streaming)
|
||||||
|
- [ ] ladybugDB ontology traversal
|
||||||
|
- [ ] Decree 13 scrubber (client + server)
|
||||||
|
- [ ] Circular 46 PDF report generator
|
||||||
|
- [ ] HITL signature gate
|
||||||
|
- [ ] Immutable audit log
|
||||||
|
- [ ] EMR/HIS HL7/FHIR push
|
||||||
|
|
||||||
|
## KEY FILES
|
||||||
|
- `/workspace/sprint_1_2/Design_Material/SPRINT_1_2_ARCHITECTURE_SPEC.md` — sprint-scoped architecture
|
||||||
|
- `/PROJ_LEVEL_READING/ARCHITECT/SOFTWARE_ARCHITECTURE_SPEC.md` — full software architecture
|
||||||
|
- `/workspace/sprint_1_2/Design_Material/API_docs/API_CONTRACT_DRAFT.md` — API contract v0.2.1
|
||||||
|
- `/workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/UI_Screen_Spec/Screen_Analysis_WS-25.md` — WS-25 screen analysis
|
||||||
|
- `/workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/FR_25_UC_SPEC.md` — FR-25 use case spec
|
||||||
|
- `/workspace/sprint_1_2/Design_Material/FR_25_UC_DESIGN/UC_CONTEXT.md` — use case context
|
||||||
|
|
||||||
|
## PHASE SEQUENCE (ML Pipeline)
|
||||||
|
```
|
||||||
|
DICOM upload → CLAHE → Angle Classification → (if post-trans) Inflammation → Segmentation
|
||||||
|
→ (if sup-up-long) Inflammation → Segmentation → Measurement → Severity (0-3)
|
||||||
|
→ GradCAM overlay → LLM rationale (GemmaE2B/MedGemma + RAG) → Clinician review
|
||||||
|
→ HITL sign → EMR push
|
||||||
|
```
|
||||||
|
|
||||||
|
## NEXT ACTIONS
|
||||||
|
- Implement PWA shell (T1-E)
|
||||||
|
- Set up FastAPI `/api/analyze` + `/api/consult/stream` (T2-B)
|
||||||
|
- Configure Triton ensemble (T2-A)
|
||||||
|
- Build WS-25 UI components (Zone A 5-layer canvas + Zone B scrollable panel)
|
||||||
|
- Wire up edge guardrail WebWorkers (cv.worker.ts, llm.worker.ts, guardrail.worker.ts)
|
||||||
43
session_memory/27_jun_26/27_jun_26.md
Normal file
43
session_memory/27_jun_26/27_jun_26.md
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
# SESSION MEMORY — 27 Jun 26
|
||||||
|
|
||||||
|
## BACKEND ARCHITECTURE & SPECIFICATION
|
||||||
|
|
||||||
|
### 1. Project Context (from Proj_context_26_Jun_26.md)
|
||||||
|
- **Purpose**: VKIST MSK Platform for air-gapped musculoskeletal ultrasound analysis (FR-25 Synovitis Grading).
|
||||||
|
- **Core Tech Stack**: FastAPI, PostgreSQL + pgvector, Redis, MinIO (S3), Triton (GPU inference), Local LLM/BERT.
|
||||||
|
- **ML Pipeline**: 3-step process: Angle Classification $\rightarrow$ Inflammation Detection $\rightarrow$ Segmentation/Measurement $\rightarrow$ Severity Grading (0-3).
|
||||||
|
- **Safety Stack**: GradCAM, Socratic Chat, BERT drift monitor, RAG-Referee.
|
||||||
|
- **Compliance**: Decree 13 (PII scrubbing), Circular 46 (PDF reports).
|
||||||
|
|
||||||
|
### 2. Backend Structural Specification (from backend-spec.md)
|
||||||
|
The backend follows a modular "Seams" architecture:
|
||||||
|
- **API Layer**: `backend/api/` contains FastAPI routers.
|
||||||
|
- **Implementation Layer**: `backend/implementation/` contains deep modules.
|
||||||
|
- **Adapters**: `backend/implementation/adapters/` handles low-level external service communication (S3, Triton, LLM, BERT).
|
||||||
|
|
||||||
|
**Core Modules:**
|
||||||
|
- **Auth**: JWT, session management.
|
||||||
|
- **Patient**: Patient CRUD, ingestion history.
|
||||||
|
- **Session**: Frame handling, S3 storage, review patching.
|
||||||
|
- **Analysis Jobs**: Async orchestration of Triton inference via KServe v2 HTTP.
|
||||||
|
- **Safety**: GradCAM, RAG evidence, Socratic chat, circuit breaker, drift check.
|
||||||
|
- **Notification/Settings/Telemetry**: Support modules for user preferences and system monitoring.
|
||||||
|
|
||||||
|
### 3. Interface Contract (from interface-contract.md)
|
||||||
|
**Key API Categories:**
|
||||||
|
- **Auth**: `/api/v1/auth/login`, `/api/v1/users/me`.
|
||||||
|
- **Patient**: `/api/v1/patients`, `/api/v1/patients/{id}/sessions`.
|
||||||
|
- **Clinical Workflow**:
|
||||||
|
- Sessions: `/api/v1/sessions`, `/api/v1/sessions/{id}/frames`, `/api/v1/sessions/{id}/review`.
|
||||||
|
- Analysis: `/api/v1/analysis-jobs` (async), `/api/v1/analysis` (sync).
|
||||||
|
- Reports: `/api/v1/reports`, `/api/v1/reports/{id}/sign`, `/api/v1/reports/{id}/emr-sync`.
|
||||||
|
- **Safety & Explanations**:
|
||||||
|
- GradCAM: `/api/v1/sessions/{id}/explanations/gradcam`.
|
||||||
|
- Rationale/Chat: `/api/v1/sessions/{id}/explanations/rationale`, `/api/v1/sessions/{id}/chat/socratic`.
|
||||||
|
- Guardrails: `/api/v1/sessions/{id}/drift/check`, `/api/v1/sessions/{id}/rag/evidence`.
|
||||||
|
|
||||||
|
### 4. Dependency Graph
|
||||||
|
- **Auth/Patient/Session/Safety/Telemetry** $\rightarrow$ PostgreSQL.
|
||||||
|
- **Session/AnalysisJobs/Safety** $\rightarrow$ S3.
|
||||||
|
- **AnalysisJobs** $\rightarrow$ Triton (Modal Serverless).
|
||||||
|
- **Safety** $\rightarrow$ Local LLM, BERT, RAG Knowledge Base.
|
||||||
37
session_memory/27_jun_26/27_jun_26_backend.md
Normal file
37
session_memory/27_jun_26/27_jun_26_backend.md
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
# Session Memory: 2026-06-27 Backend NLP/LLM Consult Implementation
|
||||||
|
|
||||||
|
## Task
|
||||||
|
Build Vietnamese Clinical LLM Consult endpoints (Rationale, Socratic Chat, Guardrail Check, SSE Stream). Follow NFR-16a governance.
|
||||||
|
|
||||||
|
## Implementation Details
|
||||||
|
|
||||||
|
### 1. LLM Adapter (`backend/implementation/adapters/llm_adapter.py`)
|
||||||
|
- Build `VertexAILangchainAdapter` via `langchain-google-vertexai`.
|
||||||
|
- Add `AuditCallbackHandler` for mandatory audit log commits before LLM call (NFR-16a).
|
||||||
|
- Use `run_in_executor` for `generate` to stop FastAPI block.
|
||||||
|
|
||||||
|
### 2. BERT Adapter (`backend/implementation/adapters/bert_adapter.py`)
|
||||||
|
- Build `BERTAdapter` stubs:
|
||||||
|
- `drift_check`: Find clinical drift.
|
||||||
|
- `referee_check`: Check RAG grounding.
|
||||||
|
- `guardrail_check`: Safety check tokens/chunks.
|
||||||
|
|
||||||
|
### 3. Safety Service (`backend/implementation/safety/service.py`)
|
||||||
|
- Link LLM + BERT adapters.
|
||||||
|
- Build `_verify_pre_egress` check:
|
||||||
|
- **Consent**: Check `consent:{session_id}` in Redis.
|
||||||
|
- **Redaction**: Check redaction manifest hash.
|
||||||
|
- Update services:
|
||||||
|
- `rationale`: Context explanation.
|
||||||
|
- `socratic_chat`: History chat + BERT referee grounding.
|
||||||
|
- `guardrail_check`: Link BERT guardrails.
|
||||||
|
- `chat_stream`: SSE generator + per-chunk filter.
|
||||||
|
- Add post-egress: set `consult_mode:{session_id}` $\rightarrow$ `cloud_vertex` in Redis.
|
||||||
|
|
||||||
|
### 4. Infrastructure & API
|
||||||
|
- **Redis Adapter**: Build `backend/implementation/adapters/redis_adapter.py` for state + consent.
|
||||||
|
- **Config**: Update `backend/implementation/config.py` with Vertex AI + Redis env vars.
|
||||||
|
- **API Router**: Update `backend/api/safety_api.py` for `redaction_hash` + SSE errors.
|
||||||
|
|
||||||
|
## Governance Flow (NFR-16a)
|
||||||
|
`Request` $\rightarrow$ `Consent Check (Redis)` $\rightarrow$ `Redaction Verify` $\rightarrow$ `Audit Commit (Langchain Callback)` $\rightarrow$ `LLM Call` $\rightarrow$ `Post-Egress Log` $\rightarrow$ `Response`
|
||||||
37
session_memory/27_jun_26/27_jun_26_backend.original.md
Normal file
37
session_memory/27_jun_26/27_jun_26_backend.original.md
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
# Session Memory: 2026-06-27 Backend NLP/LLM Consult Implementation
|
||||||
|
|
||||||
|
## Task
|
||||||
|
Implement Vietnamese Clinical LLM Consult endpoints (Rationale, Socratic Chat, Guardrail Check, SSE Stream) with strict NFR-16a governance.
|
||||||
|
|
||||||
|
## Implementation Details
|
||||||
|
|
||||||
|
### 1. LLM Adapter (`backend/implementation/adapters/llm_adapter.py`)
|
||||||
|
- Implemented `VertexAILangchainAdapter` using `langchain-google-vertexai`.
|
||||||
|
- Added `AuditCallbackHandler` to enforce mandatory audit log commits *before* LLM calls (NFR-16a).
|
||||||
|
- Wrapped synchronous `generate` calls in `run_in_executor` to avoid blocking FastAPI event loop.
|
||||||
|
|
||||||
|
### 2. BERT Adapter (`backend/implementation/adapters/bert_adapter.py`)
|
||||||
|
- Implemented `BERTAdapter` with stubs for:
|
||||||
|
- `drift_check`: Detects clinical domain drift.
|
||||||
|
- `referee_check`: Validates RAG grounding.
|
||||||
|
- `guardrail_check`: Token/chunk level safety verification.
|
||||||
|
|
||||||
|
### 3. Safety Service (`backend/implementation/safety/service.py`)
|
||||||
|
- Integrated LLM and BERT adapters.
|
||||||
|
- Implemented `_verify_pre_egress` checklist:
|
||||||
|
- **Consent**: Validates `consent:{session_id}` exists in Redis.
|
||||||
|
- **Redaction**: Verifies redaction manifest hash.
|
||||||
|
- Updated services:
|
||||||
|
- `rationale`: Context-aware explanation generation.
|
||||||
|
- `socratic_chat`: History-based conversation with BERT referee grounding.
|
||||||
|
- `guardrail_check`: Direct integration with BERT guardrails.
|
||||||
|
- `chat_stream`: SSE generator with per-chunk guardrail filtering.
|
||||||
|
- Added post-egress state update: sets `consult_mode:{session_id}` to `cloud_vertex` in Redis.
|
||||||
|
|
||||||
|
### 4. Infrastructure & API
|
||||||
|
- **Redis Adapter**: Created `backend/implementation/adapters/redis_adapter.py` for session state and consent management.
|
||||||
|
- **Config**: Updated `backend/implementation/config.py` with Vertex AI and Redis environment variables.
|
||||||
|
- **API Router**: Updated `backend/api/safety_api.py` to pass `redaction_hash` to services and handle SSE error events.
|
||||||
|
|
||||||
|
## Governance Flow (NFR-16a)
|
||||||
|
`Request` $\rightarrow$ `Consent Check (Redis)` $\rightarrow$ `Redaction Verify` $\rightarrow$ `Audit Commit (Langchain Callback)` $\rightarrow$ `LLM Call` $\rightarrow$ `Post-Egress Log` $\rightarrow$ `Response`
|
||||||
301
session_memory/27_jun_26/27_jun_26_frontend.md
Normal file
301
session_memory/27_jun_26/27_jun_26_frontend.md
Normal file
@@ -0,0 +1,301 @@
|
|||||||
|
# Frontend Development Progress - 27 Jun 26
|
||||||
|
|
||||||
|
## Session Summary
|
||||||
|
Date: 2026-06-27
|
||||||
|
Phase: 2 - Component Architecture & Deepening (35 Figma Designs Implementation Plan)
|
||||||
|
Mode: Caveman Lite
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. TopBar Refactor (Phase 2.0)
|
||||||
|
|
||||||
|
### Problem
|
||||||
|
TopBar reached into Zustand store for 4 fields (`patientId`, `clinicianGrade`, `isOverriding`, `sessionId`, `currentPhase`) and inlined sign button. Violated molecule abstraction layer.
|
||||||
|
|
||||||
|
### Solution
|
||||||
|
Refactored `TopBar` to consume `PatientHeader` and `ActionCluster` molecules via props instead of store selectors.
|
||||||
|
|
||||||
|
### Files Changed
|
||||||
|
- `src/components/organisms/TopBar.tsx`
|
||||||
|
- Removed direct `useDiagnosticWorkspaceStore` import
|
||||||
|
- Added props interface: `patientId`, `patientName`, `role`, `status`, `sessionId`, `currentPhase`, `canSign`, `isCircuitBroken`, `onSign`, `onCircuitBreak`, `onBack`, `onMore`
|
||||||
|
- Replaced inline patient info + sign button with `<PatientHeader />` and `<ActionCluster />` molecules
|
||||||
|
- Added `canSign` prop to `ActionCluster` for HITL gating
|
||||||
|
|
||||||
|
- `src/components/molecules/ActionCluster.tsx`
|
||||||
|
- Added `canSign?: boolean` prop
|
||||||
|
- Added `signDisabled` computed value: `isCircuitBroken || !canSign`
|
||||||
|
- Sign button now uses `disabled={signDisabled}` instead of just `isCircuitBroken`
|
||||||
|
|
||||||
|
- `src/components/templates/WorkspaceShell.tsx`
|
||||||
|
- Wired all TopBar props from store using selectors
|
||||||
|
- Passes `patientId`, `sessionId`, `currentPhase`, `canSign`, `isCircuitBroken`, `onSign`, `onCircuitBreak`
|
||||||
|
- `onCircuitBreak` now accepts `(active: boolean) => void` and triggers `window.confirm` modal for break/recovery
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Canvas & Visualization Deepening (Phase 2.1)
|
||||||
|
|
||||||
|
### DiagnosticCanvas Enhancements
|
||||||
|
- Tool State: reads `activeTool`, `setActiveTool`, `zoomLevel`, `panOffset`, `showGradCAM`, `showSegmentation` from Zustand `uiSlice`
|
||||||
|
- 5-Layer Canvas Stack: 5 layers with dynamic rendering
|
||||||
|
- Layer 0: Cornerstone.js DICOM viewport
|
||||||
|
- Layer 10: GradCAM overlay (conditional via `showGradCAM`)
|
||||||
|
- Layer 50: Segmentation mask (conditional via `showSegmentation`)
|
||||||
|
- Layer 100: SVG Annotations + OffscreenCanvas
|
||||||
|
- Layer 200: Viewport HUD + live coords/zoom
|
||||||
|
- Performance:
|
||||||
|
- `requestAnimationFrame` dirty-flag loop for annotation canvas
|
||||||
|
- `ResizeObserver` syncs annotation canvas resolution
|
||||||
|
- Cornerstone lifecycle: init + viewport update (no reload on pan/zoom)
|
||||||
|
- `useCallback` for event handlers
|
||||||
|
- Touch:
|
||||||
|
- `onTouchStart`, `onTouchMove` for mobile/tablet coords
|
||||||
|
- Touch events mapped to same coordinate system
|
||||||
|
- Tool Palette:
|
||||||
|
- Extracted constant `TOOLS` array with icon mapping (`pin` → `map-pin`)
|
||||||
|
- Active tool state store-driven
|
||||||
|
- Live HUD:
|
||||||
|
- Replaced static HUD with live `mousePos` state
|
||||||
|
- Zoom level from store
|
||||||
|
|
||||||
|
### Canvas Performance
|
||||||
|
- Minimized Cornerstone redraws: separate image load from viewport updates
|
||||||
|
- `useRef` for OffscreenCanvas, RAF loop, dirty flag, image tracking
|
||||||
|
- Added `mousePos` for live coords
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Reasoning & Chat Panel Deepening (Phase 2.2)
|
||||||
|
|
||||||
|
### ZoneB Composition
|
||||||
|
- Replaced placeholder with `PhaseIndicator` + `ReasoningPanel` + `ChatPanel`
|
||||||
|
- Layout: Sticky PhaseIndicator top, scrollable content below
|
||||||
|
- `ZoneBProps` interface covers all data flows
|
||||||
|
|
||||||
|
### ReasoningPanel Enhancements
|
||||||
|
- Added optional `PhaseIndicator` via `showPhaseIndicator` prop
|
||||||
|
- Classification Card:
|
||||||
|
- Color-coded grade badge (0-3) with tooltip
|
||||||
|
- Confidence score with `ConfidenceBar`
|
||||||
|
- Explanation Panel:
|
||||||
|
- GradCAM thumbnail with hover overlay
|
||||||
|
- Thickness measurement extraction/display
|
||||||
|
- Show Full GradCAM toggle via `showFullGradCam` / `onToggleFullGradCam` props
|
||||||
|
- Override Card:
|
||||||
|
- Rendered when `onSubmitOverride` provided
|
||||||
|
- Composes existing `OverrideCard` molecule
|
||||||
|
- Quick Actions Bar:
|
||||||
|
- `CircuitBreaker` molecule
|
||||||
|
- `NFRIndicator` molecule
|
||||||
|
- Report button
|
||||||
|
- EMR sync button
|
||||||
|
- Fullscreen toggle
|
||||||
|
|
||||||
|
### ChatPanel Enhancements
|
||||||
|
- Input:
|
||||||
|
- Replaced `<input>` with auto-resizing `<textarea>`
|
||||||
|
- `Enter` sends
|
||||||
|
- `Shift+Enter` inserts newline
|
||||||
|
- Auto-resize to 120px
|
||||||
|
- Streaming:
|
||||||
|
- Added `isStreaming` prop
|
||||||
|
- Streaming indicator in header
|
||||||
|
- Inline pulsing cursor on last assistant message when streaming
|
||||||
|
- Message Actions:
|
||||||
|
- Hover-revealed toolbar on non-system messages
|
||||||
|
- Copy button (`navigator.clipboard.writeText`)
|
||||||
|
- Regenerate button (assistant only)
|
||||||
|
- Feedback buttons: helpful (heart) / not helpful (alert-circle)
|
||||||
|
- All optional via callback props
|
||||||
|
- Virtualization:
|
||||||
|
- Placeholder: `react-window` `FixedSizeList` integration point
|
||||||
|
- Current: standard overflow-y-auto
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Tool State & Annotations (Phase 2.3)
|
||||||
|
|
||||||
|
### uiSlice Additions
|
||||||
|
- `brushSize: number` (default 5, range 1-50)
|
||||||
|
- `brushColor: string` (default '#ff0000')
|
||||||
|
- `eraserSize: number` (default 10, range 1-100)
|
||||||
|
- `caliperUnit: 'mm' | 'px'` (default 'mm')
|
||||||
|
- `pressureSensitivity: boolean` (default false)
|
||||||
|
- Setters with validation
|
||||||
|
|
||||||
|
### annotationSlice Additions
|
||||||
|
- `selectedAnnotationId: string | null`
|
||||||
|
- `lockedLayerIds: string[]`
|
||||||
|
- Actions:
|
||||||
|
- `addAnnotation`
|
||||||
|
- `selectAnnotation`
|
||||||
|
- `deleteAnnotation`
|
||||||
|
- `updateAnnotation`
|
||||||
|
- `lockLayer`
|
||||||
|
- Derived selectors:
|
||||||
|
- `selectSelectedAnnotation`
|
||||||
|
- `selectLockedLayers`
|
||||||
|
- Undo/redo for all annotation mutations
|
||||||
|
|
||||||
|
### Store Updates
|
||||||
|
- `store/index.ts`:
|
||||||
|
- Exported new selectors
|
||||||
|
- Updated `resetSession` to clear new state fields
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Safety & Compliance Wiring (Phase 2.4)
|
||||||
|
|
||||||
|
### Circuit Breaker
|
||||||
|
- `ActionCluster` passes `isCircuitBroken` state
|
||||||
|
- Sign button disabled when circuit broken
|
||||||
|
- `WorkspaceShell` applies `ring-2 ring-red-500 animate-pulse` when `isOverriding=true`
|
||||||
|
- Confirmation via `window.confirm` before break/recovery
|
||||||
|
|
||||||
|
### Defer 13 Scrubber (PIIScrubber)
|
||||||
|
- Fixed broken destructuring in `src/components/molecules/PIIScrubber.tsx`
|
||||||
|
- Added missing props: `text`, `onChange`, `className`
|
||||||
|
- Compiles and binds props correctly
|
||||||
|
|
||||||
|
### HITL Signature Gate
|
||||||
|
- Fixed missing `Input` import in `src/components/molecules/HITLGate.tsx`
|
||||||
|
- Added `import { Input } from '../atoms/Input';`
|
||||||
|
- Ready for credential entry modal
|
||||||
|
|
||||||
|
### NFR Status Indicator (BottomBar)
|
||||||
|
- Added `WorkerHealthEntry` interface
|
||||||
|
- Extended `BottomBarProps` with `workerHealth` and `isOffline`
|
||||||
|
- Integrated `NFRIndicator` for live status
|
||||||
|
- Removed hardcoded 'offline' when real values present
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. TypeScript & Lint Verification
|
||||||
|
|
||||||
|
### TypeScript
|
||||||
|
- `tsc --noEmit` passes clean
|
||||||
|
- All modified files compile clean
|
||||||
|
|
||||||
|
### ESLint
|
||||||
|
- All modified files pass lint
|
||||||
|
- Pre-existing warnings: stories/tests (.stories.tsx old Storybook imports, test `.set` destructuring)
|
||||||
|
- Not introduced this session
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Plan File Updates
|
||||||
|
|
||||||
|
Updated `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/.kilo/plans/35-figma-designs-implementation-plan.md`:
|
||||||
|
- Tick 2.0 TopBar deepening (molecule consumption)
|
||||||
|
- Tick 2.1 Canvas & Visualization
|
||||||
|
- Tick 2.2 Reasoning & Chat Panel (Override Card + Quick Actions)
|
||||||
|
- Tick 2.3 Interaction & State Management (tool state, annotations, measurements)
|
||||||
|
- Tick 2.4 Safety & Compliance (Circuit Breaker, PII Scrubber, HITL Gate, NFR Indicator)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Known Pre-existing Issues (Not Introduced This Session)
|
||||||
|
|
||||||
|
- `src/services/safetyService.ts`: Vite build error (unrelated to Phase 2)
|
||||||
|
- Storybook stories: import `@storybook/react` directly instead of framework package
|
||||||
|
- Store slices tests: unused `set` destructuring warnings
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Key Architectural Improvements
|
||||||
|
|
||||||
|
1. **Molecule Layer Strengthened**: TopBar no longer leaks store selectors; uses PatientHeader + ActionCluster props
|
||||||
|
2. **5-Layer Canvas Stack**: Dynamic store-driven visibility with OffscreenCanvas annotations and RAF rendering
|
||||||
|
3. **ZoneB Composed**: Clean separation of PhaseIndicator, ReasoningPanel, ChatPanel
|
||||||
|
4. **Tool State Centralized**: All tool preferences in uiSlice with validation
|
||||||
|
5. **Annotation System Ready**: Vector storage, undo/redo hooks, layer locking
|
||||||
|
6. **Safety Molecules Wired**: CircuitBreaker, PIIScrubber, HITLGate, NFRIndicator integrated into organisms
|
||||||
|
7. **Zero New Type Errors**: All changes type-check clean
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Next Steps
|
||||||
|
|
||||||
|
- Wire Cornerstone DICOM loading (placeholder)
|
||||||
|
- Implement react-window virtualization in ChatPanel at message threshold
|
||||||
|
- Add Worker hooks (Phase 3.1)
|
||||||
|
- Implement Offline/Dexie sync (Phase 3.2)
|
||||||
|
- Build safety modals (CircuitBreaker confirm, HITL credential entry)
|
||||||
|
- Create 5-layer rendering tests
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Phase 3: Worker Integration + Offline Sync (Appended 2026-06-28)
|
||||||
|
|
||||||
|
### Worker Hooks
|
||||||
|
- `src/hooks/workers/useCVWorker.ts`: DICOM frame → CLAHE resize → `cv.worker` → angle class + confidence. Real worker + mock adapter.
|
||||||
|
- `src/hooks/workers/useLLMWorker.ts`: prompt build + RAG context → `llm.worker` stream with token buffering for smooth display.
|
||||||
|
- `src/hooks/workers/useGuardrailWorker.ts`: pre-scan user input for hallucination/mal-intent → safety score → UI warning if threshold exceeded.
|
||||||
|
- `src/hooks/workers/useDriftScore.ts`: extract drift score from guardrail module.
|
||||||
|
- `src/hooks/workers/useSocraticChat.ts`: socratic dialogue via `llm.worker`.
|
||||||
|
- `src/hooks/workers/mocks.ts`: mock worker adapters for test environment using setTimeout.
|
||||||
|
- `src/workers/lifecycle.ts`: `WorkerPool` (lazy init, idle GC 5 min, error boundary).
|
||||||
|
- Barrel: `src/workers/index.ts` exports workerPool + message helpers.
|
||||||
|
|
||||||
|
### Offline Storage + Sync
|
||||||
|
- `src/services/db.ts` Dexie schema: `sessions`, `chatMessages`, `pendingReports`, `auditLog`.
|
||||||
|
- `src/services/sync.ts` SyncEngine: online/offline detect, queue when offline, exponential backoff retry (1s base, 30s cap, 5 max), LWW conflict resolution.
|
||||||
|
- `src/services/index.ts` exports `db`, `syncEngine`.
|
||||||
|
|
||||||
|
### PWA / Service Worker
|
||||||
|
- `vite.config.ts`: `vite-plugin-pwa` `generateSW` (Workbox). CacheFirst static assets, NetworkFirst API runtime, precache manifest.
|
||||||
|
- `npm i -D vite-plugin-pwa` added.
|
||||||
|
|
||||||
|
### Build Verification
|
||||||
|
- Pre-existing TS errors unchanged in `src/services/adapters/`, `src/*Service.ts`, store tests, stories. None introduced in new files.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Phase 4: Performance, NFR, a11y, i18n (Appended 2026-06-28)
|
||||||
|
|
||||||
|
### Bundle + Rendering
|
||||||
|
- `vite.config.ts`: `rollup-plugin-visualizer` (gzipSize, `dist/bundle-stats.html`). `manualChunks`:
|
||||||
|
- `vendor-media`: `cornerstone-*`, dicom parsers
|
||||||
|
- `vendor-store`: dexie, zustand
|
||||||
|
- `worker`: `src/workers/*`
|
||||||
|
- `app-core`: react, react-dom, i18next, react-router-dom, date-fns
|
||||||
|
- `DiagnosticCanvas.tsx`: `useMemo` on viewport transform.
|
||||||
|
- `ChatPanel.tsx`: `React.memo` `MessageBubble` with comparator `id`, `content`, `isStreaming`.
|
||||||
|
- `src/services/db.ts`: `saveChatMessagesWithEviction(messages, maxPerSession=20)` caps chat history via FIFO bulkPut + bulkDelete.
|
||||||
|
|
||||||
|
### i18n
|
||||||
|
- `src/i18n.ts`: i18next + `initReactI18next` + `i18next-http-backend`. Lng = `navigator.language` if starts with `en`, else `en`. `/locales/{{lng}}.json`.
|
||||||
|
- `src/locales/en.json`, `src/locales/vi.json`: 37 UI keys across TopBar, PatientHeader, BottomBar, ChatPanel, ReasoningPanel, WorkspaceShell.
|
||||||
|
- `src/App.tsx`: root wrapped in `<I18nextProvider i18n={i18n}>`.
|
||||||
|
- ReasoningPanel/BottomBar/TopBar/ChatPanel: hardcoded strings replaced with `t('Component.key')`.
|
||||||
|
|
||||||
|
### Accessibility
|
||||||
|
- `src/components/templates/WorkspaceShell.tsx`: root wrapper gets `role="main"`, `tabIndex={-1}`.
|
||||||
|
- `src/components/organisms/BottomBar.tsx`: NFR status container `aria-live="polite"`.
|
||||||
|
- `src/index.css`: global `:focus-visible` outline 2px solid var(--color-primary-500), offset 2px.
|
||||||
|
- Grad-CAM `<img alt="Grad-CAM">` retained; decorative icons already `aria-hidden="true"`.
|
||||||
|
|
||||||
|
### NFR / Lockdown
|
||||||
|
- `src/nfr/LockdownValidator.ts`: runtime DOM + fetch + XHR patch. Returns external URLs not matching `window.location.origin`.
|
||||||
|
- `src/services/adapters/httpAdapter.ts`: `VITE_API_BASE_URL` must start with `/` (relative) else throw. Same-origin only.
|
||||||
|
- `vite.config.ts`: `server.host='127.0.0.1'`, `server.https=false`. `preview.headers` CSP:
|
||||||
|
- `script-src 'self' 'wasm-unsafe-eval'`
|
||||||
|
- `style-src 'self' 'unsafe-inline'`
|
||||||
|
- `img-src 'self' data: https://fonts.gstatic.com`
|
||||||
|
- `font-src 'self' https://fonts.gstatic.com`
|
||||||
|
- `connect-src 'self' https://fonts.googleapis.com`
|
||||||
|
- Repo grep found zero external URL strings in `src/` and `public/`. Workbox retained Google Fonts runtime caching as sole allowed external origin.
|
||||||
|
|
||||||
|
### Bugfixes
|
||||||
|
- `ReasoningPanel.tsx`: missing `function ReasoningPanel(props)` caused syntax error; fixed + props destructured. `nfrStatus` type inline to avoid broken re-export path.
|
||||||
|
- `i18n.ts`: `useTranslation` export fixed; `HttpBackend` added as plugin.
|
||||||
|
- `ZoneB.tsx`: `import { ReasoningPanel }` → default import.
|
||||||
|
|
||||||
|
### Plan Updates
|
||||||
|
- `.kilo/plans/35-figma-designs-implementation-plan.md`:
|
||||||
|
- Phase 3.1 + 3.2 ticked during worker/offline work.
|
||||||
|
- Phase 4.1-4.3 items ticked where verification supports.
|
||||||
|
|
||||||
|
### Known Compile State
|
||||||
|
- `npm run build` fails on pre-existing unrelated files (dicomService typings, store tests, stories missing modules). New Phase 3/4 files compile clean.
|
||||||
225
session_memory/27_jun_26/27_jun_26_frontend.original.md
Normal file
225
session_memory/27_jun_26/27_jun_26_frontend.original.md
Normal file
@@ -0,0 +1,225 @@
|
|||||||
|
# Frontend Development Progress - 27 Jun 26
|
||||||
|
|
||||||
|
## Session Summary
|
||||||
|
Date: 2026-06-27
|
||||||
|
Phase: 2 - Component Architecture & Deepening (35 Figma Designs Implementation Plan)
|
||||||
|
Mode: Caveman Lite
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. TopBar Refactor (Phase 2.0)
|
||||||
|
|
||||||
|
### Problem
|
||||||
|
TopBar was directly reaching into the Zustand store for 4 fields (patientId, clinicianGrade, isOverriding, sessionId, currentPhase) and inlining the sign button. This violated the molecule abstraction layer.
|
||||||
|
|
||||||
|
### Solution
|
||||||
|
Refactored `TopBar` to consume `PatientHeader` and `ActionCluster` molecules via props instead of store selectors.
|
||||||
|
|
||||||
|
### Files Changed
|
||||||
|
- `src/components/organisms/TopBar.tsx`
|
||||||
|
- Removed direct `useDiagnosticWorkspaceStore` import
|
||||||
|
- Added props interface: `patientId`, `patientName`, `role`, `status`, `sessionId`, `currentPhase`, `canSign`, `isCircuitBroken`, `onSign`, `onCircuitBreak`, `onBack`, `onMore`
|
||||||
|
- Replaced inline patient info + sign button with `<PatientHeader />` and `<ActionCluster />` molecules
|
||||||
|
- Added `canSign` prop to `ActionCluster` for HITL gating
|
||||||
|
|
||||||
|
- `src/components/molecules/ActionCluster.tsx`
|
||||||
|
- Added `canSign?: boolean` prop
|
||||||
|
- Added `signDisabled` computed value: `isCircuitBroken || !canSign`
|
||||||
|
- Sign button now uses `disabled={signDisabled}` instead of just `isCircuitBroken`
|
||||||
|
|
||||||
|
- `src/components/templates/WorkspaceShell.tsx`
|
||||||
|
- Wired all TopBar props from store using selectors
|
||||||
|
- Passes `patientId`, `sessionId`, `currentPhase`, `canSign`, `isCircuitBroken`, `onSign`, `onCircuitBreak`
|
||||||
|
- `onCircuitBreak` now accepts `(active: boolean) => void` and triggers `window.confirm` modal for break/recovery
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Canvas & Visualization Deepening (Phase 2.1)
|
||||||
|
|
||||||
|
### DiagnosticCanvas Enhancements
|
||||||
|
- Tool State Management: Component now reads `activeTool`, `setActiveTool`, `zoomLevel`, `panOffset`, `showGradCAM`, `showSegmentation` from Zustand `uiSlice`
|
||||||
|
- 5-Layer Canvas Stack: All 5 layers implemented with proper dynamic rendering
|
||||||
|
- Layer 0: Cornerstone.js DICOM viewport
|
||||||
|
- Layer 10: GradCAM overlay (conditionally rendered via `showGradCAM` store toggle)
|
||||||
|
- Layer 50: Segmentation mask (conditionally rendered via `showSegmentation` store toggle)
|
||||||
|
- Layer 100: SVG Annotations layer with OffscreenCanvas support
|
||||||
|
- Layer 200: Viewport HUD with live mouse coordinates and zoom level
|
||||||
|
- Performance Optimizations:
|
||||||
|
- `requestAnimationFrame` dirty-flag render loop for annotation canvas
|
||||||
|
- `ResizeObserver` to keep annotation canvas resolution in sync
|
||||||
|
- Separated Cornerstone lifecycle into initialization + viewport update effects (no reload on pan/zoom)
|
||||||
|
- `useCallback` for event handlers
|
||||||
|
- Touch Support:
|
||||||
|
- `onTouchStart`, `onTouchMove` handlers for mobile/tablet coordinate tracking
|
||||||
|
- Touch events mapped to same coordinate system as mouse
|
||||||
|
- Tool Palette:
|
||||||
|
- Extracted to constant `TOOLS` array with proper icon mapping (`pin` corrected to `map-pin`)
|
||||||
|
- Active tool state now store-driven, not prop-driven
|
||||||
|
- Live HUD:
|
||||||
|
- Replaced static "X: 124.5mm | Y: 88.2mm" with live `mousePos` state
|
||||||
|
- Zoom level pulled from store
|
||||||
|
|
||||||
|
### Canvas Performance
|
||||||
|
- Minimized Cornerstone redraws by separating image load from viewport updates
|
||||||
|
- Used `useRef` for OffscreenCanvas, RAF loop, dirty flag, image tracking
|
||||||
|
- Added `mousePos` state for live coordinate display
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Reasoning & Chat Panel Deepening (Phase 2.2)
|
||||||
|
|
||||||
|
### ZoneB Composition
|
||||||
|
- Replaced placeholder with full composition: `PhaseIndicator` + `ReasoningPanel` + `ChatPanel`
|
||||||
|
- Layout: Sticky PhaseIndicator at top, scrollable content area below
|
||||||
|
- Comprehensive `ZoneBProps` interface covering all data flows
|
||||||
|
|
||||||
|
### ReasoningPanel Enhancements
|
||||||
|
- Added optional `PhaseIndicator` (controlled by `showPhaseIndicator` prop)
|
||||||
|
- Classification Card:
|
||||||
|
- Color-coded grade badge (0-3) with tooltip wrapper
|
||||||
|
- Confidence score with `ConfidenceBar`
|
||||||
|
- Explanation Panel:
|
||||||
|
- GradCAM thumbnail with hover overlay ("Expand View")
|
||||||
|
- Thickness measurement extraction and display
|
||||||
|
- "Show Full GradCAM" toggle via `showFullGradCam` / `onToggleFullGradCam` props
|
||||||
|
- Override Card:
|
||||||
|
- Conditionally rendered when `onSubmitOverride` callback provided
|
||||||
|
- Composes existing `OverrideCard` molecule
|
||||||
|
- Quick Actions Bar:
|
||||||
|
- `CircuitBreaker` molecule (toggle)
|
||||||
|
- `NFRIndicator` molecule
|
||||||
|
- Report generation button
|
||||||
|
- EMR sync button
|
||||||
|
- Fullscreen toggle
|
||||||
|
|
||||||
|
### ChatPanel Enhancements
|
||||||
|
- Input Handling:
|
||||||
|
- Replaced `<input>` with auto-resizing `<textarea>`
|
||||||
|
- `Enter` sends message
|
||||||
|
- `Shift+Enter` inserts newline
|
||||||
|
- Auto-resize up to 120px height
|
||||||
|
- Streaming Support:
|
||||||
|
- Added `isStreaming` prop
|
||||||
|
- Streaming indicator in header ("Streaming...")
|
||||||
|
- Inline pulsing cursor on last assistant message when streaming
|
||||||
|
- Message Actions:
|
||||||
|
- Hover-revealed action toolbar on non-system messages
|
||||||
|
- Copy button (uses `navigator.clipboard.writeText`)
|
||||||
|
- Regenerate button (assistant messages only)
|
||||||
|
- Feedback buttons: helpful (heart) / not helpful (alert-circle)
|
||||||
|
- All actions optional via callback props
|
||||||
|
- Virtualization:
|
||||||
|
- Placeholder comment block showing `react-window` `FixedSizeList` integration point
|
||||||
|
- Current implementation uses standard overflow-y-auto for simplicity
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Tool State & Annotations (Phase 2.3)
|
||||||
|
|
||||||
|
### uiSlice Additions
|
||||||
|
- `brushSize: number` (default 5, range 1-50)
|
||||||
|
- `brushColor: string` (default '#ff0000')
|
||||||
|
- `eraserSize: number` (default 10, range 1-100)
|
||||||
|
- `caliperUnit: 'mm' | 'px'` (default 'mm')
|
||||||
|
- `pressureSensitivity: boolean` (default false)
|
||||||
|
- Setters for all above with validation
|
||||||
|
|
||||||
|
### annotationSlice Additions
|
||||||
|
- `selectedAnnotationId: string | null`
|
||||||
|
- `lockedLayerIds: string[]`
|
||||||
|
- Actions:
|
||||||
|
- `addAnnotation`
|
||||||
|
- `selectAnnotation`
|
||||||
|
- `deleteAnnotation`
|
||||||
|
- `updateAnnotation`
|
||||||
|
- `lockLayer`
|
||||||
|
- Derived selectors:
|
||||||
|
- `selectSelectedAnnotation`
|
||||||
|
- `selectLockedLayers`
|
||||||
|
- Undo/redo integration for all annotation mutations
|
||||||
|
|
||||||
|
### Store Updates
|
||||||
|
- `store/index.ts`:
|
||||||
|
- Exported new selectors
|
||||||
|
- Updated `resetSession` to clear new state fields
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Safety & Compliance Wiring (Phase 2.4)
|
||||||
|
|
||||||
|
### Circuit Breaker
|
||||||
|
- `ActionCluster` updated to pass `isCircuitBroken` state
|
||||||
|
- Sign button disabled when circuit broken
|
||||||
|
- `WorkspaceShell` applies `ring-2 ring-red-500 animate-pulse` when `isOverriding=true`
|
||||||
|
- Confirmation modal via `window.confirm` before break/recovery
|
||||||
|
|
||||||
|
### Defer 13 Scrubber (PIIScrubber)
|
||||||
|
- Fixed broken destructuring in `src/components/molecules/PIIScrubber.tsx`
|
||||||
|
- Added missing props: `text`, `onChange`, `className`
|
||||||
|
- Component now compiles and correctly binds props
|
||||||
|
|
||||||
|
### HITL Signature Gate
|
||||||
|
- Fixed missing `Input` import in `src/components/molecules/HITLGate.tsx`
|
||||||
|
- Added `import { Input } from '../atoms/Input';`
|
||||||
|
- Component now ready for credential entry modal
|
||||||
|
|
||||||
|
### NFR Status Indicator (BottomBar)
|
||||||
|
- Added `WorkerHealthEntry` interface
|
||||||
|
- Extended `BottomBarProps` with `workerHealth` and `isOffline`
|
||||||
|
- Integrated `NFRIndicator` molecule for live status display
|
||||||
|
- Removed hardcoded 'offline' status when real values present
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. TypeScript & Lint Verification
|
||||||
|
|
||||||
|
### TypeScript
|
||||||
|
- `tsc --noEmit` passes clean
|
||||||
|
- All modified files compile without errors
|
||||||
|
|
||||||
|
### ESLint
|
||||||
|
- All modified files pass lint
|
||||||
|
- Pre-existing warnings in stories/tests (.stories.tsx using old Storybook imports, test `.set` destructuring)
|
||||||
|
- These are not introduced by this session's changes
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Plan File Updates
|
||||||
|
|
||||||
|
Updated `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/.kilo/plans/35-figma-designs-implementation-plan.md`:
|
||||||
|
- Tick off 2.0 TopBar deepening (molecule consumption)
|
||||||
|
- Tick off 2.1 Canvas & Visualization (all sub-items)
|
||||||
|
- Tick off 2.2 Reasoning & Chat Panel (all sub-items including Override Card + Quick Actions)
|
||||||
|
- Tick off 2.3 Interaction & State Management (tool state, annotations, measurements)
|
||||||
|
- Tick off 2.4 Safety & Compliance Features (Circuit Breaker, PII Scrubber, HITL Gate, NFR Indicator)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Known Pre-existing Issues (Not Introduced This Session)
|
||||||
|
|
||||||
|
- `src/services/safetyService.ts`: Vite build error (unrelated to Phase 2 work)
|
||||||
|
- Storybook stories: import `@storybook/react` directly instead of framework package
|
||||||
|
- Store slices tests: unused `set` parameter destructuring warnings
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Key Architectural Improvements
|
||||||
|
|
||||||
|
1. **Molecule Layer Strengthened**: TopBar no longer leaks store selectors; consumed via PatientHeader + ActionCluster props
|
||||||
|
2. **5-Layer Canvas Stack**: Dynamic, store-driven layer visibility with OffscreenCanvas annotation support and RAF-based rendering
|
||||||
|
3. **ZoneB Composed**: Clean separation of PhaseIndicator, ReasoningPanel, ChatPanel
|
||||||
|
4. **Tool State Centralized**: All tool preferences in uiSlice with validation
|
||||||
|
5. **Annotation System Ready**: Vector-based storage, undo/redo hooks, layer locking in place
|
||||||
|
6. **Safety Molecules Wired**: CircuitBreaker, PIIScrubber, HITLGate, NFRIndicator integrated into organisms
|
||||||
|
7. **Zero New Type Errors**: All changes type-check clean
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Next Steps
|
||||||
|
|
||||||
|
- Wire actual Cornerstone DICOM loading (currently placeholder)
|
||||||
|
- Implement react-window virtualization in ChatPanel when message count exceeds threshold
|
||||||
|
- Add Worker hooks (Phase 3.1)
|
||||||
|
- Implement Offline/Dexie sync (Phase 3.2)
|
||||||
|
- Build safety modals (CircuitBreaker confirm, HITL credential entry)
|
||||||
|
- Create actual 5-layer rendering tests
|
||||||
0
session_memory/27_jun_26/27_jun_26_pipeline.md
Normal file
0
session_memory/27_jun_26/27_jun_26_pipeline.md
Normal file
159
session_memory/6_Jul_26.md
Normal file
159
session_memory/6_Jul_26.md
Normal file
@@ -0,0 +1,159 @@
|
|||||||
|
---
|
||||||
|
title: Session Memory 6 Jul 26
|
||||||
|
date: 2026-07-06
|
||||||
|
status: active
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Sprint_1_2 clinical chat saw major LLM UX, streaming, and inference work (Jul 5–6). The highest-risk open issue is **frontend markdown rendering of generated assistant text** — it has caused tab freezes and catastrophic crashes during reasoning streams. Reasoning paths now use **plain text only** (`StreamingPlainText`) as a mitigation. **Beta functionality** (Planning mode + 🔥 high reasoning level) must be completed **by end of this week** (target: **Friday 10 Jul 2026**).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Change log (what was updated)
|
||||||
|
|
||||||
|
### Clinical chat UI & model lifecycle
|
||||||
|
|
||||||
|
| Area | Change | Key files |
|
||||||
|
|------|--------|-----------|
|
||||||
|
| LLM loading bubble | Install vs load phases with distinct copy, progress bar, disabled composer | `ClinicalChatPanel.tsx`, `useClinicalChat.ts`, `modelLoadProgress.ts` |
|
||||||
|
| Install vs load semantics | First OPFS download (~1.9 GB) vs cached checkpoint init into worker/GPU | `useClinicalChat.ts` (`ModelLoadPhase: 'installing' \| 'loading'`) |
|
||||||
|
| Sidebar card switch | Both diagnosis + review layers stay mounted (CSS hide/show) so Gemma is not torn down on carousel switch | `SidebarLayerCarousel.tsx` |
|
||||||
|
| OPFS persistence | Completed install survives reload; interrupted download not resumable (manifest only written on success) | `opfsModelStore.ts` |
|
||||||
|
| Worker init progress | `init_progress` events wired through `LlmWorkerClient.init(onProgress)` | `llmWorkerClient.ts`, `llm.worker.ts` |
|
||||||
|
|
||||||
|
### Inference modes, reasoning levels & backends
|
||||||
|
|
||||||
|
| Area | Change | Key files |
|
||||||
|
|------|--------|-----------|
|
||||||
|
| Unified chat modes | `ask` merged into `chat`; inference modes: **Chat**, **Planning** (beta), **Agent** | `clinicalChatModes.ts`, `analyzePromptComplexity.ts` |
|
||||||
|
| Reasoning levels | 🧘 chill / 🤔 moderate / 🔥 high (beta); bar visibility persisted | `chatReasoningLevel.ts`, `ClinicalChatPanel.tsx` |
|
||||||
|
| Edge vs server toggle | 🤔 moderate: **Máy** (Gemma 4 E2B edge) vs **Server** (Gemma 4 E4B Modal Ollama `think:true`) | `reasoningModelBackend.ts`, `ollamaLlmClient.ts`, `inferenceBackend.ts` |
|
||||||
|
| Ollama dev proxy | `VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat`, model `gemma4:e4b` | `.env.development`, `vite.config.ts` |
|
||||||
|
| Agent mode | Uses Modal Ollama E4B when configured; tool loop unchanged | `clinicalChatModes.ts`, `runClinicalChatTurn.ts` |
|
||||||
|
| OOM mitigation | Bootstrap at 2048 tokens; `releaseInference()` before reload; reuse engine when `configured.maxTokens >= required` | `clinicalChatConfig.ts`, `llm.worker.ts`, `llmModelBootstrap.ts` |
|
||||||
|
| Qwen3 experiment | Dual-model (Qwen `.litertlm` for moderate) attempted then **disabled** — `usesQwenReasoningLevel()` returns `false`; moderate uses Gemma CoT again | `qwenOpfsModelStore.ts`, `reasoningLlmClient.ts`, `chatReasoningLevel.ts` |
|
||||||
|
|
||||||
|
### Streaming, CoT split & markdown
|
||||||
|
|
||||||
|
| Area | Change | Key files |
|
||||||
|
|------|--------|-----------|
|
||||||
|
| CoT split | `splitGemmaThoughtOutput`, `isThoughtChannelComplete`; thought vs answer channels in message state | `prompts.ts`, `clinicalChat.ts`, `useClinicalChat.ts` |
|
||||||
|
| Collapsible reasoning | `ClinicalChatThought` — expand while streaming, auto-collapse when thought completes | `ClinicalChatThought.tsx` |
|
||||||
|
| Stream throttle | RAF-coalesced updates (~60/s) to reduce main-thread pressure | `streamUpdateThrottle.ts` |
|
||||||
|
| Token-by-token Ollama | Imperative DOM via `clinicalChatStreamRegistry` + `StreamingPlainText` (bypasses React 18 batching) | `clinicalChatStreamRegistry.ts`, `StreamingPlainText.tsx`, `ollamaLlmClient.ts` |
|
||||||
|
| Markdown renderer | Custom `ChatMarkdown` (bold, italic, code, lists, headings) — **deferred until stream ends** via `requestIdleCallback` + `startTransition` | `ChatMarkdown.tsx`, `ClinicalChatMessageBubble.tsx` |
|
||||||
|
| Reasoning = plain text | When `tracksThought`, thought + answer use `StreamingPlainText` only — **no `ChatMarkdown`** | `ClinicalChatThought.tsx`, `ClinicalChatMessageBubble.tsx` |
|
||||||
|
|
||||||
|
### Agent tools & Modal testing
|
||||||
|
|
||||||
|
| Area | Change | Key files |
|
||||||
|
|------|--------|-----------|
|
||||||
|
| Tool catalog doc | Walkthrough of Edge-LLM agent tools: `exa_search`, `supabase_query`, `escalate_medgemma` | session + `agent_tools_contract.md` |
|
||||||
|
| 3-layer smoke harness | Layer 0 Modal Ollama, Layer 1 BFF routes, Layer 2 browser `ToolExecutor` | `ml/tests/agent_tools/` |
|
||||||
|
| Python reference tests | Modal `/api/chat` streaming + thinking cases | `PILOT_PROJECT/tmp/test_endpoint.py`, `test_endpoint_img.py` |
|
||||||
|
| Gemma4 E4B deploy script | Modal Ollama serverless for Gemma 4 E4B | `PILOT_PROJECT/tmp/GemmaE4B_ollama_deploy.py` |
|
||||||
|
| gemma4_e2b lab | Tool smoke panel (no Gemma) for isolated tool calls | `ml/tests/gemma4_e2b/src/lib/toolSmoke.ts` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Critical issue: markdown rendering → catastrophic crash
|
||||||
|
|
||||||
|
### Symptom
|
||||||
|
|
||||||
|
Rendering **LLM-generated markdown** in the clinical chat UI (especially during or immediately after **reasoning / CoT streams**) has caused:
|
||||||
|
|
||||||
|
- Main-thread freezes (composer unresponsive while tokens still arrive)
|
||||||
|
- Tab crashes / OOM under combined **WebGPU model memory + large text buffers + markdown parse**
|
||||||
|
|
||||||
|
### Root cause (confirmed in debugging)
|
||||||
|
|
||||||
|
1. **Per-token full re-parse** — early implementation ran `ChatMarkdown` / `renderBlocks()` on the entire growing thought string every token → hundreds of parses per second.
|
||||||
|
2. **React re-render storm** — `setMessages` on every token re-rendered the full message list.
|
||||||
|
3. **Post-stream markdown** — even with “parse after stream ends”, heavy `renderBlocks()` on long CoT + answer text still spikes CPU/memory.
|
||||||
|
4. **Dual large strings** — `rawAccumulator`, `thoughtContent`, and `content` can all hold 2048-token traces simultaneously at 🤔 moderate.
|
||||||
|
|
||||||
|
### Mitigations applied (6 Jul)
|
||||||
|
|
||||||
|
- Stream updates throttled (`createStreamUpdateThrottle`)
|
||||||
|
- Plain text while `streaming === true`
|
||||||
|
- `React.memo` on `ClinicalChatMessageBubble`
|
||||||
|
- Reasoning paths (`tracksThought`) → **`StreamingPlainText` only, markdown disabled**
|
||||||
|
- Ollama path → imperative DOM updates per token
|
||||||
|
- `ChatMarkdown` uses idle callback + `startTransition` for post-stream formatting (chill / non-reasoning answers only)
|
||||||
|
|
||||||
|
### Current policy
|
||||||
|
|
||||||
|
> **Be wary of re-enabling markdown in reasoning mode until a safe renderer is proven.**
|
||||||
|
|
||||||
|
`ClinicalChatThought.tsx` comment: *"Reasoning panel — plain text only (markdown disabled while isolating stream crashes)."*
|
||||||
|
|
||||||
|
### Follow-up for later sprints
|
||||||
|
|
||||||
|
- [ ] Reproduce crash with a minimal `ChatMarkdown` + long fixture (no LLM) — isolate `renderBlocks` vs React
|
||||||
|
- [ ] Consider `react-markdown` with strict plugins OR server-side pre-render for final answer only
|
||||||
|
- [ ] Cap thought trace length in UI (truncate + “show more”)
|
||||||
|
- [ ] Virtualize message list for long sessions
|
||||||
|
- [ ] Re-test 🔥 high mode and Planning mode with any new markdown path before removing plain-text guard
|
||||||
|
- [ ] Audit whether chill-mode `ChatMarkdown` after stream is safe on 8 GB RAM devices
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Beta functionality — deadline end of week
|
||||||
|
|
||||||
|
**Target: complete by Friday 10 Jul 2026** (end of sprint week).
|
||||||
|
|
||||||
|
Features still marked `beta: true` (UI tab/button disabled until ready):
|
||||||
|
|
||||||
|
| Feature | ID | Location | What “done” means |
|
||||||
|
|---------|-----|----------|-------------------|
|
||||||
|
| **Planning mode** | `planning` | `clinicalChatModes.ts` | Checklist output stable; remove `beta` flag; selectable in mode tabs |
|
||||||
|
| **High reasoning** | `high` 🔥 | `chatReasoningLevel.ts` | 2048-token multi-turn works without crash; remove `beta` flag |
|
||||||
|
|
||||||
|
Related work not yet beta-flagged but in scope:
|
||||||
|
|
||||||
|
- Agent tool integration (Layer 3 after smoke tests pass)
|
||||||
|
- Live BFF tools (`VITE_CLINICAL_CHAT_MOCK_TOOLS=false` + credentials)
|
||||||
|
- Edge/server reasoning toggle polish (default server E4B when Modal up)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture snapshot (clinical LLM, 6 Jul EOD)
|
||||||
|
|
||||||
|
```
|
||||||
|
User message
|
||||||
|
│
|
||||||
|
├─ Mode: chat ─┬─ 🧘 chill → Gemma E2B edge (no CoT)
|
||||||
|
│ ├─ 🤔 moderate → Máy: Gemma E2B + CoT (plain text)
|
||||||
|
│ │ Server: Gemma E4B Ollama think:true (plain text)
|
||||||
|
│ └─ 🔥 high (BETA) → disabled in UI
|
||||||
|
│
|
||||||
|
├─ Mode: planning (BETA) → disabled in UI
|
||||||
|
│
|
||||||
|
└─ Mode: agent → Gemma E4B Ollama + tools (exa, supabase, medgemma)
|
||||||
|
|
||||||
|
Display:
|
||||||
|
tracksThought → StreamingPlainText (thought panel + answer)
|
||||||
|
else → StreamingPlainText while streaming → ChatMarkdown when done
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Key env / endpoints (dev)
|
||||||
|
|
||||||
|
```env
|
||||||
|
VITE_CLINICAL_CHAT_USE_LLM=true
|
||||||
|
VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat
|
||||||
|
VITE_OLLAMA_MODEL=gemma4:e4b
|
||||||
|
VITE_CLINICAL_CHAT_MOCK_TOOLS=true # flip false for live agent tools
|
||||||
|
```
|
||||||
|
|
||||||
|
Modal Gemma E4B (reference): `PILOT_PROJECT/tmp/test_endpoint.py` → `dtj-tran--ollama-gemma4-e4b-ollamaserver-web.modal.run`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Session references
|
||||||
|
|
||||||
|
- Prior frontend memory: `session_memory/27_jun_26/27_jun_26_frontend.md`
|
||||||
|
- Agent tools smoke README: `CODEBASE/ml/tests/agent_tools/README.md`
|
||||||
|
- Today's active dev: `npm run dev` on frontend implementation
|
||||||
554
session_memory/7_Jun_26/ONBOARDING_INDEX.md
Normal file
554
session_memory/7_Jun_26/ONBOARDING_INDEX.md
Normal file
@@ -0,0 +1,554 @@
|
|||||||
|
# VKIST MSK Pilot — Intern Onboarding Index
|
||||||
|
|
||||||
|
**Date:** 7 Jul 2026
|
||||||
|
**Audience:** Phuc (ML / agentic), Quan (frontend / PWA), Dave Tran - Dat (lead)
|
||||||
|
**Repo:** `pilot_msk_ultrasound_stack` — clone already shared via GitHub
|
||||||
|
|
||||||
|
Read this file first, then open your role-specific guide:
|
||||||
|
|
||||||
|
|
||||||
|
| Track | Document |
|
||||||
|
| ---------------------------------------------- | ------------------------------------------------------------------ |
|
||||||
|
| **Phuc** — generative logic, agent, guardrails | [ONBOARDING_PHUC_ML_ENGINEER.md](./ONBOARDING_PHUC_ML_ENGINEER.md) |
|
||||||
|
| **Quan** — mobile UI, Figma, Stitch, PWA | [ONBOARDING_QUAN_FRONTEND.md](./ONBOARDING_QUAN_FRONTEND.md) |
|
||||||
|
|
||||||
|
|
||||||
|
> **Abbreviations:** See [§12 Glossary](./ONBOARDING_INDEX.md#12-glossary-of-abbreviations) — FR, NFR, RAG, CoT, PHI, PWA, etc.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 1. What this project is
|
||||||
|
|
||||||
|
**VKIST MSK Pilot** moves academic musculoskeletal ultrasound ML (angle → inflammation → segmentation → synovitis grade 0–3) into a clinical web platform for Vietnamese public hospitals (Bach Mai, Viet Duc class facilities).
|
||||||
|
|
||||||
|
|
||||||
|
| Dimension | Summary |
|
||||||
|
| ---------- | ------------------------------------------------------------------------------------------------------- |
|
||||||
|
| **Cycle** | Jun 2 – Sep 2, 2026 (strict 3-month window) |
|
||||||
|
| **Method** | Hybrid: waterfall phases + agile sprints |
|
||||||
|
| **Stack** | React PWA + FastAPI + Triton CV + 3-tier LLM (Edge Gemma → Gemini → MedGemma) + RAG over MOH guidelines |
|
||||||
|
| **Users** | Radiologists, surgeons, physiotherapists, patients/caregivers — each with different UI needs |
|
||||||
|
|
||||||
|
|
||||||
|
**Hard constraints (non-negotiable):**
|
||||||
|
|
||||||
|
- **Decree 13/2023** — no raw PHI leaves device without scrubbing
|
||||||
|
- **No open-ended patient chat** — structured, gated summaries only
|
||||||
|
- **Air-gapped hospital LAN** — cloud LLM only under NFR-16a (consent + audit + redaction)
|
||||||
|
- **Legacy smartphones** — zero-GPU fallback rendering for patient fleet
|
||||||
|
|
||||||
|
Full vision: `[PILOT_PROJECT/PROJ_LEVEL_READING/PLAN/CONTEXT_VISION_SCOPE.md](../../PILOT_PROJECT/PROJ_LEVEL_READING/PLAN/CONTEXT_VISION_SCOPE.md)`
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Dự án AI siêu âm khớp gối cho bệnh viện VN; 3 tháng; bảo mật Decree 13; các tinh năng hiện giờ đang optimize cho người dùng chính là bác sĩ cơ xương khớp - trong tương lai có thể cân nhắc `bệnh nhân` là đối tượng phát triển; làm tiếp phần Dave đã xây, không làm lại từ đầu.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 2. How we work — continuation engineering
|
||||||
|
|
||||||
|
You are **not** here to read docs for three weeks. You inherit Dave's foundation and ship **incremental PRs** that close spec gaps toward sprint checkpoints.
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
foundation[Dave_Foundation]
|
||||||
|
specs[Specs_and_UCs]
|
||||||
|
backlog[Open_Backlog]
|
||||||
|
checkpoint[Sprint_Checkpoint]
|
||||||
|
internPR[Intern_Incremental_PR]
|
||||||
|
foundation --> internPR
|
||||||
|
specs --> backlog
|
||||||
|
backlog --> internPR
|
||||||
|
internPR --> checkpoint
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
1. Dave assigns **one backlog ID** (P-n or Q-n).
|
||||||
|
2. You read **only** the spec + code files for that ticket.
|
||||||
|
3. Ship a small PR (≤300 lines preferred); demo something runnable weekly.
|
||||||
|
4. Dave assigns the next ticket.
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Dave giao ticket → đọc đúng spec → PR nhỏ → demo hàng tuần → ticket tiếp theo.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 3. Foundation already built (do not rebuild)
|
||||||
|
|
||||||
|
Dave's latest handoff: `[session_memory/6_Jul_26.md](../6_Jul_26.md)`
|
||||||
|
|
||||||
|
|
||||||
|
| Area | Status | Key locations |
|
||||||
|
| --------------------- | ------------------------ | ------------------------------------------------------------------------- |
|
||||||
|
| FR-25 workspace shell | **Working PoC** | `ClinicalWorkspacePage.tsx`, `WorkspaceShell.tsx`, `DiagnosticCanvas.tsx` |
|
||||||
|
| CV inference | **Live** (FastAPI proxy) | `POST /api/test/analyze/batch`, `mlInferenceCacheStore.ts` |
|
||||||
|
| Clinical chat UI | **Working** | `ClinicalChatPanel.tsx`, `useClinicalChat.ts` |
|
||||||
|
| Edge Gemma 4 E2B | **Working** | `llm.worker.ts`, `clinicalLlmRuntime.ts`, OPFS |
|
||||||
|
| CoT streaming | **Working** (plain text) | `clinicalChatStreamRegistry.ts`, `StreamingPlainText.tsx` |
|
||||||
|
| Agent runtime | **Working** (mock tools) | `agent_runtime/`, `agentLoop.ts` |
|
||||||
|
| Cloud BFF | **Wired** | `cloud_orchestrate.py`, `cloud_consult.py`, `agent_tools.py` |
|
||||||
|
| 35 Figma routes | **Stub pages** | `figma-component-specs.md` |
|
||||||
|
| Safety / guardrails | **Spec + stubs** | `bert_adapter.py`; `guardrail.worker.ts` missing |
|
||||||
|
| Patient PWA | **Not started** | Sprint 4 |
|
||||||
|
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Workspace, chat, agent, inference đã chạy; guardrail và UI mobile/patient còn thiếu — đó là việc của bạn.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 4. Sprint checkpoints (what your PRs advance)
|
||||||
|
|
||||||
|
|
||||||
|
| ID | Checkpoint | Sprint | Window | Owner |
|
||||||
|
| ---- | ----------------------------------------------- | ------ | -------------- | --------------------- |
|
||||||
|
| CP-1 | Beta clinical modes (Planning + high reasoning) | 2 | Jul 10 | Dave → Phuc polish |
|
||||||
|
| CP-2 | Live agent tools (mock → BFF) | 2–3 | Jul 14–24 | **Phuc** |
|
||||||
|
| CP-3 | Guardrail slice v1 | 2–3 | Jul 21 – Aug 7 | **Phuc** + Dave |
|
||||||
|
| CP-4 | Collaborative annotation UX | 3 | Jul 13–24 | Dave; Quan UI support |
|
||||||
|
| CP-5 | Patient list/cards → Figma mobile | 3–4 | Jul 28 – Aug 7 | **Quan** |
|
||||||
|
| CP-6 | Patient PWA (large text, 375px) | 4 | Jul 27 – Aug 7 | **Quan** |
|
||||||
|
| CP-7 | Accessibility on 2 screens | 4–5 | Aug 4–21 | **Quan** |
|
||||||
|
| CP-8 | RAG retrieval gate / referee stub | 3–5 | Aug 1–21 | **Phuc** |
|
||||||
|
|
||||||
|
|
||||||
|
**Timeboxes:** Phuc until ~Aug 31; Quan ~2 months (ends ~early Aug).
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Mỗi PR phải đẩy một checkpoint CP-n, không chỉ viết tài liệu.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 5. Open backlog
|
||||||
|
|
||||||
|
Dave maintains priority. Do not self-assign large items.
|
||||||
|
|
||||||
|
### Phuc — generative / agentic
|
||||||
|
|
||||||
|
|
||||||
|
| ID | Gap | Starting code | Done when |
|
||||||
|
| --- | -------------------------------- | ----------------------------------------------- | ----------------------------------------------------- |
|
||||||
|
| P-1 | Agent tools mock | `tools/registry.ts`, `agent_tools.py` | Live BFF; `VITE_CLINICAL_CHAT_MOCK_TOOLS=false` works |
|
||||||
|
| P-2 | `bert_adapter.py` stubs | `bert_adapter.py`, `safety/service.py` | One real PASS/FAIL check on outbound text |
|
||||||
|
| P-3 | `guardrail.worker.ts` missing | Create near `llm.worker.ts` | Worker hooks into chat before display |
|
||||||
|
| P-4 | RAG not enforced in chat mode | `runClinicalChatTurn.ts` | Clinical answers require retrieval context |
|
||||||
|
| P-5 | Planning + high reasoning `beta` | `clinicalChatModes.ts`, `chatReasoningLevel.ts` | Stable runs; beta flags removed |
|
||||||
|
| P-6 | `escalate_medgemma` E2E | `cloud_llm_gateway.py`, `tools/registry.ts` | Demo with audit log entry |
|
||||||
|
| P-7 | Conformal decoding eval | `ml/spec/conformal_decoding_*.md` | 3 VI clinical test cases (research) |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Quan — frontend / PWA
|
||||||
|
|
||||||
|
|
||||||
|
| ID | Gap | Starting code | Done when |
|
||||||
|
| --- | ----------------------------- | ----------------------------- | ----------------------------------------------------------------- |
|
||||||
|
| Q-1 | 35 stub routes | `src/pages/*.tsx` | 2 priority screens match Figma |
|
||||||
|
| Q-2 | Safety overlay not routed | `SafetyEscalationOverlay.tsx` | Reachable from workspace |
|
||||||
|
| Q-3 | No mobile breakpoint | Patient pages | 375px pass, touch ≥44px |
|
||||||
|
| Q-4 | Accessibility open | Patient components | 1 view WCAG-oriented pass |
|
||||||
|
| Q-5 | No error boundaries | `App.tsx` | Friendly fallback on workspace route |
|
||||||
|
| Q-6 | Patient dashboard | `patient-card-*.tsx` | Jargon-free, large type, VI copy |
|
||||||
|
| Q-7 | Zero-GPU fallback UI | TBD banner | UI state when WebGL weak (Dave provides hook) |
|
||||||
|
| Q-8 | Stitch sync after ship (lite) | Stitch board | Optional: update frame when PR merges; no standalone proposal doc |
|
||||||
|
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Bảng backlog trên — chờ Dave giao ID cụ thể trước khi code.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 6. Codebase map
|
||||||
|
|
||||||
|
**Active code:** `PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/`
|
||||||
|
**Do not develop in:** `PILOT_PROJECT/workspace/LEGACY/` (read-only reference)
|
||||||
|
|
||||||
|
> **Note:** Root `PILOT_PROJECT/README.md` references stale paths (`Reading_docs/`, `CONTEXT.md`). Use paths in this doc instead.
|
||||||
|
|
||||||
|
|
||||||
|
| Module | Path | Phuc | Quan |
|
||||||
|
| -------------- | --------------------- | ----------------------------------------- | ------------------------------------------------ |
|
||||||
|
| Frontend | `CODEBASE/frontend/` | `lib/llm/`, `workers/`, `useClinicalChat` | **Primary** — `pages/`, `components/`, `styles/` |
|
||||||
|
| Backend | `CODEBASE/backend/` | **Primary** — routers, safety, adapters | Read-only |
|
||||||
|
| ML / NLP | `CODEBASE/ml/` | **Primary** — `agent_runtime/`, tests | Do not touch |
|
||||||
|
| Knowledge | `CODEBASE/knowledge/` | When P-4/P-8 | Do not touch |
|
||||||
|
| Design | `Design_Material/` | Safety, API | UCs, screen specs |
|
||||||
|
| Session memory | `session_memory/` | `6_Jul_26.md` | `26_Jun_26/Proj_context` for layout |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Frontend tree (Quan primary)
|
||||||
|
|
||||||
|
```
|
||||||
|
frontend/
|
||||||
|
├── spec/ ← uix_prototype_coverage, figma-component-specs, DESIGN
|
||||||
|
└── implementation/src/
|
||||||
|
├── pages/ ← Quan: screens
|
||||||
|
├── components/atoms|molecules|shells/
|
||||||
|
├── styles/design-tokens.ts
|
||||||
|
├── hooks/ ← Phuc: useClinicalChat only
|
||||||
|
├── lib/llm/ ← Phuc ONLY
|
||||||
|
└── workers/ ← Phuc ONLY
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### ML + backend tree (Phuc primary)
|
||||||
|
|
||||||
|
```
|
||||||
|
ml/implementation/nlp/agent_runtime/src/
|
||||||
|
orchestrator/agentLoop.ts
|
||||||
|
tools/registry.ts
|
||||||
|
prompt/toolSystemPrompt.ts, toolOutputParser.ts
|
||||||
|
|
||||||
|
frontend/.../src/lib/llm/ ← edge inference, chat routing
|
||||||
|
backend/routers/ ← cloud_*, agent_tools
|
||||||
|
backend/services/ ← cloud_llm_gateway, agent_tools_service
|
||||||
|
backend/implementation/safety/, adapters/bert_adapter.py
|
||||||
|
knowledge/spec/ ← RAG when P-4/P-8
|
||||||
|
```
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Phuc = ml + backend + lib/llm; Quan = frontend pages/components; LEGACY không đụng.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 7. Reading guide (summary)
|
||||||
|
|
||||||
|
Each role doc has three tiers. Quick reference:
|
||||||
|
|
||||||
|
|
||||||
|
| Tier | When |
|
||||||
|
| ---------------- | ---------------------------------- |
|
||||||
|
| **Must read** | Day 1–3, before first PR |
|
||||||
|
| **Should read** | When your ticket touches that area |
|
||||||
|
| **Learn deeper** | Keywords — grow over internship |
|
||||||
|
|
||||||
|
|
||||||
|
See [Phuc doc](./ONBOARDING_PHUC_ML_ENGINEER.md#3-what-to-read-what-to-learn) and [Quan doc](./ONBOARDING_QUAN_FRONTEND.md#3-what-to-read-what-to-learn) for full lists.
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Đọc bắt buộc trước PR đầu; đọc thêm theo ticket; học sâu theo từ khóa khi cần.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 8. Dev toolkit
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Shared (both)
|
||||||
|
|
||||||
|
|
||||||
|
| Tool | Purpose |
|
||||||
|
| ----------------- | ----------------------------------------------------------- |
|
||||||
|
| **Cursor / Kilo** | Agentic coding — load `PILOT_PROJECT/AGENT_SKILL/` first |
|
||||||
|
| **Git + GitHub** | Branch per ticket; PR ≤300 lines |
|
||||||
|
| `secrets/` | Local only — see `PILOT_PROJECT/secrets_template/README.md` |
|
||||||
|
| `session_memory/` | Dave handoff — start with `6_Jul_26.md` |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Phuc additions
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Frontend + clinical chat
|
||||||
|
cd PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation
|
||||||
|
npm install && npm run dev # :5173
|
||||||
|
|
||||||
|
# Gemma lab
|
||||||
|
cd ../../ml/tests/gemma4_e2b && npm install && npm run dev # :5174
|
||||||
|
|
||||||
|
# Agent smoke
|
||||||
|
cd ../agent_tools && npm install && npm run smoke
|
||||||
|
|
||||||
|
# Backend
|
||||||
|
cd ../../.. && conda activate vkist_ultra
|
||||||
|
PYTHONPATH=. python backend/tests/smoke_agent_tools_server.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Key env: `VITE_CLINICAL_CHAT_USE_LLM`, `VITE_CLINICAL_CHAT_MOCK_TOOLS`, `VITE_API_BASE_URL`
|
||||||
|
|
||||||
|
### Quan additions
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation
|
||||||
|
npm install && npm run dev # :5173
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
| Tool | Link |
|
||||||
|
| ------------------------- | -------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
| **Figma** (authority) | [MSK_Workspace_UIX-Design](https://www.figma.com/design/uakk3T4efS1AIckfTEFSZO/MSK_Workspace_UIX-Design?node-id=0-1) |
|
||||||
|
| **Stitch** (acceleration) | [Whiteboard 8188910238055386921](https://stitch.withgoogle.com/projects/8188910238055386921) |
|
||||||
|
| **Responsive test** | Chrome DevTools → iPhone SE 375px |
|
||||||
|
|
||||||
|
|
||||||
|
Stitch board has **web mocks** today — Quan creates **mobile variants** and exports references for the coding agent (see Quan doc).
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Cursor/Kilo + Git; Phuc thêm conda/smoke tests; Quan thêm Figma + Stitch + DevTools mobile.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 9. Bonus — Feynman Technique with agents
|
||||||
|
|
||||||
|
When a concept or code path feels **hard**, don't just ask the agent to fix it — ask it to **teach it back to you** using the [Feynman Technique](https://en.wikipedia.org/wiki/Feynman_technique): explain simply, use an analogy, expose gaps, then verify your understanding.
|
||||||
|
|
||||||
|
### When to use
|
||||||
|
|
||||||
|
|
||||||
|
| Situation | Use Feynman prompt |
|
||||||
|
| ---------------------------------------------------- | -------------------------------- |
|
||||||
|
| You read code but can't explain it in your own words | ✅ |
|
||||||
|
| Agent generated a diff you don't fully understand | ✅ Before commit |
|
||||||
|
| Spec uses jargon (RAG, CoT, BFF, …) | ✅ |
|
||||||
|
| You're stuck >30 min on the same concept | ✅ Then ask Dave if still blocked |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Universal template (copy into Cursor / Kilo)
|
||||||
|
|
||||||
|
```
|
||||||
|
Feynman explain — [TOPIC or FILE PATH]:
|
||||||
|
|
||||||
|
1. Explain like I'm a new intern with CS basics but new to this codebase.
|
||||||
|
2. Use one everyday analogy (non-medical if possible).
|
||||||
|
3. Walk the data/control flow in ≤8 numbered steps.
|
||||||
|
4. Define every abbreviation you use (or link to our glossary).
|
||||||
|
5. List 3 "common mistakes" a beginner would make here.
|
||||||
|
6. Ask me 2 short quiz questions — wait for my answers before continuing.
|
||||||
|
7. If I get them wrong, correct me simply.
|
||||||
|
|
||||||
|
Context: VKIST MSK Pilot, ticket [P-n or Q-n].
|
||||||
|
Do NOT write code yet — teach only.
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### After the agent teaches you
|
||||||
|
|
||||||
|
1. **Rewrite** the idea in 3–5 bullets in your own words (PR description, standup note, or personal log).
|
||||||
|
2. If you still can't rewrite without copying — **you're not ready to commit** that code; ask one follow-up or ping Dave.
|
||||||
|
3. Optional: paste your summary to Dave at weekly demo — proves understanding, not just output.
|
||||||
|
|
||||||
|
Role-specific Feynman prompts: [Phuc §12](./ONBOARDING_PHUC_ML_ENGINEER.md#12-bonus--feynman-technique-prompts) · [Quan §14](./ONBOARDING_QUAN_FRONTEND.md#14-bonus--feynman-technique-prompts)
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Khi khó hiểu — bảo agent giải thích đơn giản (Feynman), trả lời quiz, tự viết lại bằng lời mình; chưa hiểu thì chưa commit.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 10. Collaboration contract
|
||||||
|
|
||||||
|
|
||||||
|
| Rule | Detail |
|
||||||
|
| ------------- | ----------------------------------------------------------- |
|
||||||
|
| Assignment | Dave picks backlog ID — you don't self-select large scope |
|
||||||
|
| PR size | ≤300 lines; one backlog ID per PR |
|
||||||
|
| Weekly demo | Runnable change required |
|
||||||
|
| Agentic tools | OK after reading ticket spec; pass `AGENT_SKILL` checklists |
|
||||||
|
| Research | ≤40% time; only when tied to P-7 or approved milestone |
|
||||||
|
| Blocked >4h | Ping Dave with exact error + file path |
|
||||||
|
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** PR nhỏ, demo mỗi tuần, hỏi Dave khi kẹt quá 4 giờ.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 11. Current assignments (Dave fills in)
|
||||||
|
|
||||||
|
|
||||||
|
| Intern | Active ticket | Checkpoint | PR link | Status |
|
||||||
|
| ------ | ------------------------- | ---------- | ------- | ------ |
|
||||||
|
| Phuc | P-___ | CP-___ | | |
|
||||||
|
| Quan | Q-1 (`/` worklist mobile) | CP-5 | | |
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 12. Glossary of abbreviations
|
||||||
|
|
||||||
|
Terms you will see daily in specs, code comments, and standups. Bookmark this section.
|
||||||
|
|
||||||
|
### Project & process
|
||||||
|
|
||||||
|
|
||||||
|
| Abbr. | Full form | Meaning in this project |
|
||||||
|
| --------------- | --------------------------------- | -------------------------------------------------------------- |
|
||||||
|
| **VKIST** | Viện Khoa học Công nghệ (context) | Research institute partner; MSK ultrasound AI origin |
|
||||||
|
| **MSK** | Musculoskeletal | Joint/tendon/ligament domain — knee ultrasound focus |
|
||||||
|
| **PoC** | Proof of Concept | Current sprint build — not production-certified yet |
|
||||||
|
| **CP-n** | Checkpoint n | Sprint deliverable milestone (see §4) |
|
||||||
|
| **P-n / Q-n** | Phuc-ticket / Quan-ticket | Backlog IDs in §5 |
|
||||||
|
| **UC-n** | Use Case n | Clinical workflow spec (e.g. `UC-48376` load scan session) |
|
||||||
|
| **FR-n** | Functional Requirement n | Feature requirement (e.g. **FR-25** synovitis grading) |
|
||||||
|
| **NFR-n** | Non-Functional Requirement n | Quality constraint (latency, safety, privacy) |
|
||||||
|
| **WS-25** | Workspace 25 | Clinician diagnostic workspace UI (65/35 layout) |
|
||||||
|
| **HITL** | Human-in-the-Loop | Clinician must review/confirm before AI output is final |
|
||||||
|
| **UP5, UP1, …** | User Persona n | Stakeholder archetype (UP5 = radiologist, UP1 = senior expert) |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Clinical & domain (Vietnamese context)
|
||||||
|
|
||||||
|
|
||||||
|
| Abbr. | Full form | Meaning |
|
||||||
|
| --------------- | ----------------------------------------------------------- | --------------------------------------------------------- |
|
||||||
|
| **MOH** | Ministry of Health (Vietnam) | Source of clinical guidelines for RAG corpus |
|
||||||
|
| **DICOM** | Digital Imaging and Communications in Medicine | Standard medical image format (ultrasound slices) |
|
||||||
|
| **PACS** | Picture Archiving and Communication System | Hospital image archive — DICOM source |
|
||||||
|
| **EMR / HIS** | Electronic Medical Record / Hospital Information System | Hospital patient record system |
|
||||||
|
| **HL7 / FHIR** | Health Level 7 / Fast Healthcare Interoperability Resources | Standards for EMR data exchange |
|
||||||
|
| **Synovitis** | — | Joint lining inflammation — graded **0–3** by this system |
|
||||||
|
| **GradCAM** | Gradient-weighted Class Activation Mapping | Heatmap showing which image regions drove AI grade |
|
||||||
|
| **ULTS** | Ultrasound | Modality used in FR-25 |
|
||||||
|
| **Circular 46** | Thông tư 46/2018/TT-BYT | Vietnam clinical report format regulation |
|
||||||
|
| **Decree 13** | Nghị định 13/2023/NĐ-CP | Vietnam personal data protection law (PHI rules) |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Security & privacy
|
||||||
|
|
||||||
|
|
||||||
|
| Abbr. | Full form | Meaning |
|
||||||
|
| --------------- | ----------------------------------- | ----------------------------------------------------------------- |
|
||||||
|
| **PHI** | Protected Health Information | Patient-identifiable health data — never commit to git |
|
||||||
|
| **PII** | Personally Identifiable Information | Names, IDs, phone — scrubbed before cloud egress |
|
||||||
|
| **RBAC** | Role-Based Access Control | Permissions by clinician role (radiologist, therapist, …) |
|
||||||
|
| **JWT** | JSON Web Token | Session auth token (stored in Redis server-side) |
|
||||||
|
| **NFR-16a** | NFR exception 16a | PoC-only rule allowing cloud LLM with redaction + consent + audit |
|
||||||
|
| **AES-256-GCM** | Advanced Encryption Standard | Client-side encryption for local storage (target arch) |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Architecture & infrastructure
|
||||||
|
|
||||||
|
|
||||||
|
| Abbr. | Full form | Meaning |
|
||||||
|
| ------------ | ----------------------------------- | -------------------------------------------------------------- |
|
||||||
|
| **PWA** | Progressive Web App | Installable web app — Sprint 4 patient target |
|
||||||
|
| **BFF** | Backend-for-Frontend | FastAPI layer that proxies agent tools + cloud LLM for browser |
|
||||||
|
| **API** | Application Programming Interface | HTTP endpoints (`/api/v1/...`) |
|
||||||
|
| **K3s** | Lightweight Kubernetes | On-prem hospital cluster (target deployment) |
|
||||||
|
| **Triton** | NVIDIA Triton Inference Server | GPU server for CV models + embeddings (`:8001` gRPC) |
|
||||||
|
| **FastAPI** | — | Python web framework for backend (`:8000`) |
|
||||||
|
| **NGINX** | — | Reverse proxy / SSL termination in front of FastAPI |
|
||||||
|
| **MinIO** | — | S3-compatible object store for DICOM blobs, reports |
|
||||||
|
| **pgvector** | PostgreSQL vector extension | Vector DB for MOH guideline semantic search (HNSW index) |
|
||||||
|
| **HNSW** | Hierarchical Navigable Small World | Approximate nearest-neighbor index for RAG retrieval |
|
||||||
|
| **GCP** | Google Cloud Platform | Hosts Gemini (Vertex AI) |
|
||||||
|
| **Modal** | Modal.com | Serverless GPU platform hosting MedGemma |
|
||||||
|
| **LAN** | Local Area Network | Hospital internal network — air-gapped primary mode |
|
||||||
|
| **CI/CD** | Continuous Integration / Deployment | Jenkins + GitLab pipeline |
|
||||||
|
| **SSE** | Server-Sent Events | One-way streaming from server to browser |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### ML, NLP & generative AI
|
||||||
|
|
||||||
|
|
||||||
|
| Abbr. | Full form | Meaning |
|
||||||
|
| --------------------- | ------------------------------------------------------- | ------------------------------------------------------------------- |
|
||||||
|
| **CV** | Computer Vision | Image ML pipeline (angle → inflammation → segmentation) |
|
||||||
|
| **NLP** | Natural Language Processing | Clinical text cleanup, explanations, chat |
|
||||||
|
| **LLM** | Large Language Model | Generative text models (Gemma, Gemini, MedGemma) |
|
||||||
|
| **RAG** | Retrieval-Augmented Generation | Fetch MOH guideline chunks before generating answer |
|
||||||
|
| **CoT** | Chain-of-Thought | Model "thinking" trace before final answer (thought channel) |
|
||||||
|
| **TTFT** | Time to First Token | Latency until first streamed word appears (NFR-7: ≤200 ms target) |
|
||||||
|
| **E2B / E4B** | Gemma 4 2B / 4B params | Edge (browser) vs server Ollama model sizes |
|
||||||
|
| **WebLLM** | Web Large Language Model | In-browser inference via WebGPU + WASM |
|
||||||
|
| **OPFS** | Origin Private File System | Browser storage for ~1.9 GB Gemma checkpoint |
|
||||||
|
| **WASM** | WebAssembly | Binary format for in-browser ML (MediaPipe, LiteRT) |
|
||||||
|
| **WebGPU** | — | Browser GPU API — required for edge Gemma |
|
||||||
|
| **WebGL** | — | Older browser 3D API — fallback path for weak devices |
|
||||||
|
| **BERT** | Bidirectional Encoder Representations from Transformers | Classifier for drift/guardrail/referee (partially stubbed) |
|
||||||
|
| **EmbeddingGemma** | — | 768-dim embedding model for RAG vector search |
|
||||||
|
| **MedGemma** | — | Tier-3 clinical deep-reasoning LLM (Modal, NFR-16a) |
|
||||||
|
| **Gemini** | — | Tier-2 orchestration/translation LLM (GCP Vertex AI) |
|
||||||
|
| **Gemma** | — | Tier-1 edge conversation LLM (browser MediaPipe) |
|
||||||
|
| **Agent** | — | Multi-step tool-calling loop (`<tool_call>` / `<final>` protocol) |
|
||||||
|
| **Guardrail** | — | Safety filter on generated text (hallucination, scope breach) |
|
||||||
|
| **RAG-Referee** | — | 3-axis citation validator for MedGemma output |
|
||||||
|
| **Circuit breaker** | — | Clinician challenges AI grade via Socratic dialogue before sign-off |
|
||||||
|
| **Socratic dialogue** | — | Structured Q&A between clinician and AI about a grade |
|
||||||
|
| **DeepLab / UNet** | — | Segmentation model architectures in CV pipeline |
|
||||||
|
| **ConvNeXt / MedViT** | — | Angle classification model families (legacy stack) |
|
||||||
|
| **LiteRT** | — | On-device inference runtime (TensorFlow Lite successor) |
|
||||||
|
| **TFLite / TFJS** | TensorFlow Lite / TensorFlow.js | Client-side ML runtimes |
|
||||||
|
| **ONNX** | Open Neural Network Exchange | Model interchange format for Triton deployment |
|
||||||
|
| **VRAM** | Video RAM | GPU memory — relevant for Triton/Modal sizing |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Frontend & UI
|
||||||
|
|
||||||
|
|
||||||
|
| Abbr. | Full form | Meaning |
|
||||||
|
| ------------- | ------------------------------------------ | ------------------------------------------------------------- |
|
||||||
|
| **UI / UIX** | User Interface / User Interface eXperience | Screens, layout, interaction design |
|
||||||
|
| **SPA** | Single Page Application | React app — one HTML shell, client-side routing |
|
||||||
|
| **TS / TSX** | TypeScript / TypeScript + JSX | Frontend language + React syntax |
|
||||||
|
| **Zustand** | — | Lightweight React state store (target arch; PoC uses context) |
|
||||||
|
| **Dexie.js** | — | IndexedDB wrapper for offline cache |
|
||||||
|
| **IndexedDB** | — | Browser local database for inference cache, session replay |
|
||||||
|
| **MRN** | Medical Record Number | Patient hospital ID shown in TopBar |
|
||||||
|
| **FR-26** | Functional Requirement 26 | Future patient-facing dashboard scope |
|
||||||
|
| **a11y** | Accessibility | WCAG-oriented design (contrast, font scale, touch targets) |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Tools & workflow
|
||||||
|
|
||||||
|
|
||||||
|
| Abbr. | Full form | Meaning |
|
||||||
|
| ------------ | ---------------------------------- | ----------------------------------------- |
|
||||||
|
| **PR** | Pull Request | Code review unit — keep ≤300 lines |
|
||||||
|
| **IDE** | Integrated Development Environment | VS Code / Cursor |
|
||||||
|
| **Dev Mode** | Figma Dev Mode | Inspect panel for spacing, tokens, CSS |
|
||||||
|
| **SDK** | Software Development Kit | e.g. `@google/stitch-sdk` for asset fetch |
|
||||||
|
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Bảng viết tắt trên dùng hàng ngày — FR/NFR = yêu cầu; RAG/LLM/CoT = AI sinh ngôn ngữ; PHI/Decree 13 = bảo mật; PWA/DICOM/PACS = lâm sàng & UI; hỏi Dave nếu gặp từ mới.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 13. Quick links
|
||||||
|
|
||||||
|
|
||||||
|
| Resource | Path |
|
||||||
|
| -------------------- | ------------------------------------------------------------------------------------- |
|
||||||
|
| Project vision | `PILOT_PROJECT/PROJ_LEVEL_READING/PLAN/CONTEXT_VISION_SCOPE.md` |
|
||||||
|
| Sprint architecture | `PILOT_PROJECT/workspace/sprint_1_2/Design_Material/SPRINT_1_2_ARCHITECTURE_SPEC.md` |
|
||||||
|
| UI truth doc | `PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/spec/uix_prototype_coverage.md` |
|
||||||
|
| Agent skills | `PILOT_PROJECT/AGENT_SKILL/INTRODUCTION.md` |
|
||||||
|
| Dave's Jul 6 handoff | `session_memory/6_Jul_26.md` |
|
||||||
|
| Layout reference | `session_memory/26_Jun_26/Proj_context_26_Jun_26.md` |
|
||||||
|
|
||||||
|
|
||||||
396
session_memory/7_Jun_26/ONBOARDING_PHUC_ML_ENGINEER.md
Normal file
396
session_memory/7_Jun_26/ONBOARDING_PHUC_ML_ENGINEER.md
Normal file
@@ -0,0 +1,396 @@
|
|||||||
|
# Onboarding — Phuc (ML Engineer / Agentic & Generative Logic)
|
||||||
|
|
||||||
|
**Start here:** [ONBOARDING_INDEX.md](./ONBOARDING_INDEX.md) — [glossary](./ONBOARDING_INDEX.md#12-glossary-of-abbreviations) · [Feynman prompts (INDEX §9)](./ONBOARDING_INDEX.md#9-bonus--feynman-technique-with-agents)
|
||||||
|
**Date:** 7 Jul 2026
|
||||||
|
**Timebox:** ~Jul 7 – Aug 31, 2026
|
||||||
|
**Lane:** Generative LLM, agent runtime, guardrails, RAG gates — co-designed with Dave
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Your mission
|
||||||
|
|
||||||
|
You continue the **generative & agentic layer** Dave built. Your deliverable is **merged code** that advances checkpoints **CP-2, CP-3, CP-8** — not architecture slides.
|
||||||
|
|
||||||
|
You co-own agentic design and guardrail logic with Dave: you propose, Dave approves scope, you ship one vertical slice per PR.
|
||||||
|
|
||||||
|
**You are not expected to:** rewrite the CV/Triton pipeline, own full compliance sign-off, or refactor UI components.
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Tiếp tục xây LLM/agent/guardrail; mỗi tuần có PR chạy được; thiết kế guardrail cùng Dave; không làm lại pipeline những kết quả đã có - nếu có thì phải optimize & hạn chế redundancies overhead trong thiết kế.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 2. Your codebase scope
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### In scope — touch freely
|
||||||
|
|
||||||
|
|
||||||
|
| Path | What |
|
||||||
|
| --------------------------------------------------------------------------- | --------------------------------------- |
|
||||||
|
| `CODEBASE/ml/implementation/nlp/agent_runtime/` | Agent loop, tools, prompts |
|
||||||
|
| `CODEBASE/ml/tests/gemma4_e2b/` | Isolated Gemma + agent lab (`:5174`) |
|
||||||
|
| `CODEBASE/ml/tests/agent_tools/` | BFF / Modal / Exa smoke tests |
|
||||||
|
| `CODEBASE/ml/spec/conformal_decoding_llm_with_domain_stritcly_constrain.md` | Research spec |
|
||||||
|
| `CODEBASE/frontend/implementation/src/lib/llm/` | Edge inference, chat routing, prompts |
|
||||||
|
| `CODEBASE/frontend/implementation/src/hooks/useClinicalChat.ts` | Chat bootstrap + streaming |
|
||||||
|
| `CODEBASE/frontend/implementation/src/workers/llm.worker.ts` | MediaPipe Gemma worker |
|
||||||
|
| `CODEBASE/frontend/implementation/src/workers/` | Create `guardrail.worker.ts` here (P-3) |
|
||||||
|
| `CODEBASE/backend/routers/cloud_*.py`, `agent_tools.py` | Cloud + tool BFF |
|
||||||
|
| `CODEBASE/backend/services/cloud_llm_gateway.py`, `agent_tools_service.py` | Gateway + tools |
|
||||||
|
| `CODEBASE/backend/implementation/safety/` | Safety service |
|
||||||
|
| `CODEBASE/backend/implementation/adapters/bert_adapter.py` | Guardrail stubs |
|
||||||
|
| `CODEBASE/knowledge/spec/` | When working P-4 / P-8 |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Read-only — understand, don't refactor
|
||||||
|
|
||||||
|
- `CODEBASE/backend/implementation/pipeline/` — CV inference (Dave)
|
||||||
|
- `CODEBASE/infra/` — unless P-6 Modal debugging
|
||||||
|
- `CODEBASE/frontend/implementation/src/components/` — Quan's lane
|
||||||
|
- `workspace/LEGACY/` — reference only
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Architecture mental model
|
||||||
|
|
||||||
|
```
|
||||||
|
User message
|
||||||
|
→ useClinicalChat / runClinicalChatTurn
|
||||||
|
→ [chat|planning] → Edge Gemma (llm.worker) OR Ollama E4B
|
||||||
|
→ [agent] → agentLoop → tools (exa, supabase, medgemma)
|
||||||
|
→ [YOUR GAPS] guardrail check, RAG gate, cloud escalation audit
|
||||||
|
→ display (StreamingPlainText)
|
||||||
|
```
|
||||||
|
|
||||||
|
Tier 1 = browser Gemma · Tier 2 = Gemini · Tier 3 = MedGemma (NFR-16a governed)
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Phạm vi = ml/agent_runtime, lib/llm, backend safety/routers; không sửa UI hay LEGACY.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 3. What to read, what to learn
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Must read (Day 1 — before first PR)
|
||||||
|
|
||||||
|
|
||||||
|
| # | Document | Path |
|
||||||
|
| --- | ---------------------- | ---------------------------------------------------------------------------- |
|
||||||
|
| 1 | Dave's latest handoff | `session_memory/6_Jul_26.md` |
|
||||||
|
| 2 | Regulatory constraints | `PILOT_PROJECT/PROJ_LEVEL_READING/PLAN/CONTEXT_VISION_SCOPE.md` §4 |
|
||||||
|
| 3 | Agent protocol | `CODEBASE/ml/implementation/nlp/README.md` |
|
||||||
|
| 4 | PHI safety | `PILOT_PROJECT/AGENT_SKILL/secrets_and_phi_safety/SKILL.md` |
|
||||||
|
| 5 | Agent tools rules | `PILOT_PROJECT/AGENT_SKILL/agent_tool_execution/SKILL.md` |
|
||||||
|
| 6 | LLM + safety NFRs | `Design_Material/SPRINT_1_2_ARCHITECTURE_SPEC.md` — LLM tiers, NFR-10/16a/18 |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Should read (when ticket assigned)
|
||||||
|
|
||||||
|
|
||||||
|
| Ticket | Read |
|
||||||
|
| -------- | --------------------------------------------------------------------------------------------------------------- |
|
||||||
|
| P-1, P-6 | `backend/spec/agent_tools_contract.md`, `agent_runtime/.../tools/registry.ts`, `backend/routers/agent_tools.py` |
|
||||||
|
| P-2, P-3 | `bert_adapter.py`, `safety/service.py`, `FR_25_UC_DESIGN/Q1_*.md` … `Q4_*.md` |
|
||||||
|
| P-4, P-8 | `knowledge/spec/knowledge_spec.md`, `agentLoop.ts`, `runClinicalChatTurn.ts` |
|
||||||
|
| P-5 | `clinicalChatModes.ts`, `chatReasoningLevel.ts`, `6_Jul_26.md` §beta |
|
||||||
|
| P-7 | `ml/spec/conformal_decoding_llm_with_domain_stritcly_constrain.md` |
|
||||||
|
| Any | Relevant `UC-*.md` in `Design_Material/FR_25_UC_DESIGN/Use_Case/` |
|
||||||
|
|
||||||
|
|
||||||
|
**Week 1 code trace (do once):**
|
||||||
|
`useClinicalChat.ts` → `runClinicalChatTurn.ts` → `agentLoop.ts` OR `llm.worker.ts` → `cloud_llm_gateway.py`
|
||||||
|
|
||||||
|
### Learn deeper (keywords)
|
||||||
|
|
||||||
|
> Abbreviations: [Glossary INDEX §12](./ONBOARDING_INDEX.md#12-glossary-of-abbreviations)
|
||||||
|
|
||||||
|
Study as tickets require — use papers, specs, and agent exploration:
|
||||||
|
|
||||||
|
|
||||||
|
| Keywords | Tickets |
|
||||||
|
| ------------------------------------------------------------------- | -------- |
|
||||||
|
| `prompt-constrained tool calling`, `agent orchestration loop` | P-1 |
|
||||||
|
| `RAG pre-retrieval`, `pgvector`, `semantic chunk`, `MOH corpus` | P-4, P-8 |
|
||||||
|
| `BERT guardrail`, `hallucination detection`, `scope-breach scoring` | P-2, P-3 |
|
||||||
|
| `conformal decoding`, `domain-constrained generation` | P-7 |
|
||||||
|
| `CoT streaming`, `thought/answer channel`, `WebLLM MediaPipe` | P-5 |
|
||||||
|
| `NFR-16a`, `consent audit`, `cloud egress redaction` | P-6 |
|
||||||
|
| `RAG-Referee`, `citation validation`, `3-axis contestant` | P-8 |
|
||||||
|
| `Transformers.js`, `Web Worker`, `OPFS model cache` | P-3 |
|
||||||
|
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Đọc 6 tài liệu bắt buộc ngày 1; đọc thêm theo ticket P-n; học sâu theo từ khóa khi làm.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 4. Dev toolkit
|
||||||
|
|
||||||
|
|
||||||
|
| Tool | Command |
|
||||||
|
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
||||||
|
| Clinical chat UI | `cd CODEBASE/frontend/implementation && npm run dev` → `:5173` |
|
||||||
|
| Gemma lab | `cd CODEBASE/ml/tests/gemma4_e2b && npm run dev` → `:5174` |
|
||||||
|
| Agent smoke (test-the-agentic-functionality) | `cd CODEBASE/ml/tests/agent_tools && npm run smoke` |
|
||||||
|
| Backend smoke (test-the-agentic-functionality) | `conda activate vkist_ultra`; `cd CODEBASE`; `PYTHONPATH=. python backend/tests/smoke_agent_tools_server.py` |
|
||||||
|
| Agentic IDE | Cursor / Kilo — load `AGENT_SKILL/agent_tool_execution` |
|
||||||
|
|
||||||
|
|
||||||
|
**Environment variables (frontend** `.env.development`**):**
|
||||||
|
|
||||||
|
```env
|
||||||
|
VITE_CLINICAL_CHAT_USE_LLM=true
|
||||||
|
VITE_CLINICAL_CHAT_MOCK_TOOLS=true # flip false for P-1 done
|
||||||
|
VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat
|
||||||
|
VITE_API_BASE_URL=http://localhost:8001
|
||||||
|
```
|
||||||
|
|
||||||
|
Secrets: ask Dave — `PILOT_PROJECT/secrets_template/README.md`
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Chạy :5173 chat, :5174 lab, smoke tests; đổi MOCK_TOOLS=false khi P-1 xong.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 5. Day 1 playbook (then ship)
|
||||||
|
|
||||||
|
**Morning**
|
||||||
|
|
||||||
|
1. Run `:5173` → open `/workspace/:patientId` → test chat + agent modes.
|
||||||
|
2. Run `:5174` gemma4_e2b → complete one tool smoke call.
|
||||||
|
3. Trace one chat turn and one agent turn (Section 3 code trace).
|
||||||
|
|
||||||
|
**Afternoon**
|
||||||
|
4. Hands-on verify backlog P-1–P-7 (✅/❌ in your notes).
|
||||||
|
5. Sync with Dave → receive first ticket (likely **P-1** or **P-5**).
|
||||||
|
|
||||||
|
Do **not** spend Week 1 writing standalone architecture docs.
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Ngày 1: chạy app, trace code, sync Dave lấy ticket — không viết doc dài.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 6. Spec vs implementation (your work surface)
|
||||||
|
|
||||||
|
|
||||||
|
| Component | Built today | Your tickets |
|
||||||
|
| ----------------------------------- | -------------- | ------------------ |
|
||||||
|
| Edge Gemma inference | ✅ | P-5 polish |
|
||||||
|
| Agent loop + `<tool_call>` protocol | ✅ (mock tools) | P-1 |
|
||||||
|
| Cloud gateway + audit hooks | ⚠️ partial | P-6 |
|
||||||
|
| Auto-retrieval in agent mode | ✅ | P-4 extend to chat |
|
||||||
|
| `bert_adapter.py` / safety service | ❌ stub | P-2 |
|
||||||
|
| `guardrail.worker.ts` | ❌ missing | P-3 |
|
||||||
|
| RAG-Referee 3-axis | ❌ spec only | P-8 |
|
||||||
|
| Planning + high reasoning beta | ⚠️ disabled | P-5 |
|
||||||
|
|
||||||
|
|
||||||
|
**Key facts:**
|
||||||
|
|
||||||
|
- Production chat = **browser Gemma 4 E2B** (MediaPipe), not server-side generation.
|
||||||
|
- Agent mode uses **prompt tags** (`<tool_call>`, `<final>`), not native function calling.
|
||||||
|
- Do **not** re-enable `ChatMarkdown` in reasoning mode — see `6_Jul_26.md` crash policy.
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Guardrail và RAG chat còn stub — đó là trọng tâm của bạn.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 7. Your backlog (detail)
|
||||||
|
|
||||||
|
|
||||||
|
| ID | Gap | Spec | Starting files |
|
||||||
|
| ------- | ------------------------------ | --------------------------------- | ----------------------------------------------- |
|
||||||
|
| **P-1** | Mock agent tools | `agent_tools_contract.md` | `tools/registry.ts`, `agent_tools.py` |
|
||||||
|
| **P-2** | BERT adapter stubs | NFR-10, Q1–Q4 | `bert_adapter.py`, `safety/service.py` |
|
||||||
|
| **P-3** | No edge guardrail worker | Web Workers spec | Create `guardrail.worker.ts` |
|
||||||
|
| **P-4** | No RAG gate in chat mode | NFR-18, `knowledge_spec.md` | `runClinicalChatTurn.ts` |
|
||||||
|
| **P-5** | Beta modes disabled | `6_Jul_26.md` | `clinicalChatModes.ts`, `chatReasoningLevel.ts` |
|
||||||
|
| **P-6** | MedGemma escalation incomplete | NFR-16a | `cloud_llm_gateway.py`, `tools/registry.ts` |
|
||||||
|
| **P-7** | Conformal decoding research | `ml/spec/conformal_decoding_*.md` | `gemma4_e2b/` lab |
|
||||||
|
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** P-1 đến P-7 — chờ Dave giao ID / chủ động đề xuất giải pháp kỹ thuật & thiết kế; mỗi PR một ticket.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 8. SMART milestones
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### M1 — First merge (Jul 7 – Jul 18)
|
||||||
|
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| -------------- | --------------------------------------------- |
|
||||||
|
| **Ticket** | P-1 or P-5 (Dave assigns) |
|
||||||
|
| **Specific** | Live agent tools OR remove beta from one mode |
|
||||||
|
| **Measurable** | PR merged; live demo on `:5173` |
|
||||||
|
| **Achievable** | One backlog ID only |
|
||||||
|
| **Relevant** | CP-1 / CP-2 |
|
||||||
|
| **Time-bound** | Sprint 3 week 1 |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### M2 — Guardrail v1 (Jul 14 – Aug 1)
|
||||||
|
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| -------------- | ------------------------------------------------------- |
|
||||||
|
| **Ticket** | P-2 or P-3 (co-scoped with Dave) |
|
||||||
|
| **Specific** | RFC ≤1 page + one real PASS/FAIL check on outbound text |
|
||||||
|
| **Measurable** | Test/smoke on 3 fixtures from Q1–Q4 scenarios |
|
||||||
|
| **Achievable** | Keyword gate OK for v1; full BERT optional |
|
||||||
|
| **Relevant** | CP-3, NFR-10 |
|
||||||
|
| **Time-bound** | Sprint 3 end |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### M3 — Retrieval or escalation (Jul 28 – Aug 15)
|
||||||
|
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| -------------- | -------------------------------------------------- |
|
||||||
|
| **Ticket** | P-4 or P-6 |
|
||||||
|
| **Specific** | RAG gate in chat OR MedGemma escalation with audit |
|
||||||
|
| **Measurable** | Screen recording + log snippet in PR |
|
||||||
|
| **Relevant** | CP-2 / CP-8, NFR-18 |
|
||||||
|
| **Time-bound** | Sprint 4 |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### M4 — Research or hardening (Aug 1 – Aug 31)
|
||||||
|
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| -------------- | ---------------------------------------------------- |
|
||||||
|
| **Ticket** | P-7 or extend P-2/P-3 |
|
||||||
|
| **Specific** | Conformal eval (3 VI cases) OR second guardrail axis |
|
||||||
|
| **Optional** | If M2–M3 complete |
|
||||||
|
| **Time-bound** | Internship end |
|
||||||
|
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** 4 mốc — PR đầu tiên tuần 1; guardrail tuần 3–4; RAG/escalation tháng 8.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 9. Anti-patterns
|
||||||
|
|
||||||
|
- ❌ Refactor `agent_runtime` package structure without inform to members on update logic (no change-Log)
|
||||||
|
- ❌ Swap model families (Qwen experiment is closed)
|
||||||
|
- ❌ Touch `LEGACY/` or Quan's UI components
|
||||||
|
- ❌ Enable `ChatMarkdown` during reasoning streams (markdown rendering on LLM's outcome is problem - we shall put it under the rug)
|
||||||
|
- ❌ Commit secrets or PHI — ever
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Không refactor lớn, không đổi model, không markdown khi reasoning.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 10. Definition of done
|
||||||
|
|
||||||
|
- [ ] ≥3 merged PRs across P-1–P-6
|
||||||
|
- [ ] Can demo live agent tool call or guardrail check without Dave driving
|
||||||
|
- [ ] CP-2 or CP-3 marked advanced in team backlog
|
||||||
|
- [ ] Read `secrets_and_phi_safety` + `agent_tool_execution` skills
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** 3 PR, demo được guardrail/agent, checkpoint CP-2 hoặc CP-3 tiến triển.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 11. Agent prompt examples (optimize / change based on usecase)
|
||||||
|
|
||||||
|
**Explore generative flow:**
|
||||||
|
|
||||||
|
```
|
||||||
|
Trace runClinicalChatTurn.ts for agent mode: list each file called,
|
||||||
|
what tool protocol is used, and where mock tools are injected.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Start P-1:**
|
||||||
|
|
||||||
|
```
|
||||||
|
Help me flip VITE_CLINICAL_CHAT_MOCK_TOOLS to false and fix any
|
||||||
|
BFF errors. Start from tools/registry.ts and agent_tools.py.
|
||||||
|
Follow AGENT_SKILL/agent_tool_execution/SKILL.md.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Start P-2:**
|
||||||
|
|
||||||
|
```
|
||||||
|
Implement a minimal scope-breach keyword gate in bert_adapter.py
|
||||||
|
returning PASS/FAIL. Use Q1_True_Agreement.md as fixture context.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Bonus — Feynman Technique prompts
|
||||||
|
|
||||||
|
Use when a generative/agent concept feels opaque **before** you commit agent-generated code.
|
||||||
|
Master template: [INDEX §9](./ONBOARDING_INDEX.md#9-bonus--feynman-technique-with-agents).
|
||||||
|
|
||||||
|
### Hard concepts → ready-made prompts
|
||||||
|
|
||||||
|
**Agent loop & `<tool_call>` protocol (P-1):**
|
||||||
|
```
|
||||||
|
Feynman explain agentLoop.ts and the <tool_call> / <final> protocol in
|
||||||
|
@vkist/agent-runtime. Why prompt tags instead of OpenAI function calling?
|
||||||
|
Trace one turn with exa_search mocked. Quiz me after.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Edge vs cloud 3-tier LLM (general):**
|
||||||
|
```
|
||||||
|
Feynman explain our 3-tier LLM: Edge Gemma → Gemini → MedGemma.
|
||||||
|
When does each tier run? What is NFR-16a in one sentence?
|
||||||
|
Use a "triage desk at a hospital" analogy. Quiz me.
|
||||||
|
```
|
||||||
|
|
||||||
|
**clinicalChatStreamRegistry + streaming (P-5 area):**
|
||||||
|
```
|
||||||
|
Feynman explain why clinicalChatStreamRegistry.ts paints tokens to the DOM
|
||||||
|
directly instead of React setState on every token. What breaks if we use
|
||||||
|
ChatMarkdown during CoT? ≤8 steps. Quiz me.
|
||||||
|
```
|
||||||
|
|
||||||
|
**RAG pre-retrieval (P-4 / P-8):**
|
||||||
|
```
|
||||||
|
Feynman explain mandatory RAG pre-generation in our system: what pgvector
|
||||||
|
retrieves, when agentLoop auto-calls supabase_query, and why chat mode
|
||||||
|
doesn't do this yet. MOH corpus in plain language. Quiz me.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Guardrail vs RAG-Referee (P-2 / P-3):**
|
||||||
|
```
|
||||||
|
Feynman explain the difference between edge guardrail, bert_adapter
|
||||||
|
drift check, and RAG-Referee. Which are implemented vs stub today?
|
||||||
|
Use a "editor vs fact-checker" analogy. Quiz me.
|
||||||
|
```
|
||||||
|
|
||||||
|
**After each Feynman session:** write 3 bullets in your own words in the PR or ticket comment.
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Dùng prompt Feynman khi không hiểu agent/LLM; có sẵn prompt cho từng chủ đề khó; tự tóm tắt lại trước khi commit.
|
||||||
499
session_memory/7_Jun_26/ONBOARDING_QUAN_FRONTEND.md
Normal file
499
session_memory/7_Jun_26/ONBOARDING_QUAN_FRONTEND.md
Normal file
@@ -0,0 +1,499 @@
|
|||||||
|
# Onboarding — Quan (Frontend / Mobile PWA)
|
||||||
|
|
||||||
|
**Start here:** [ONBOARDING_INDEX.md](./ONBOARDING_INDEX.md) — [glossary](./ONBOARDING_INDEX.md#12-glossary-of-abbreviations) · [Feynman prompts (INDEX §9)](./ONBOARDING_INDEX.md#9-bonus--feynman-technique-with-agents)
|
||||||
|
**Date:** 7 Jul 2026
|
||||||
|
**Timebox:** ~2 months (ends ~early Aug 2026)
|
||||||
|
**Lane:** Mobile UI, Figma + Stitch design acceleration, patient PWA
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Your mission
|
||||||
|
|
||||||
|
You continue the **UI layer** from stub routes toward Sprint 4 patient PWA checkpoints. You ship **visible screens in the browser** — not backend, not LLM internals.
|
||||||
|
|
||||||
|
Dave provides two design accelerators:
|
||||||
|
|
||||||
|
|
||||||
|
| Tool | Link | Role |
|
||||||
|
| ----------------- | -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
|
||||||
|
| **Figma** | [MSK_Workspace_UIX-Design](https://www.figma.com/design/uakk3T4efS1AIckfTEFSZO/MSK_Workspace_UIX-Design?node-id=0-1) | Design authority — 35 screens, tokens, Vietnamese copy |
|
||||||
|
| **Google Stitch** | [Whiteboard](https://stitch.withgoogle.com/projects/8188910238055386921) | Acceleration — board has **web mocks**; you design **mobile variants** |
|
||||||
|
|
||||||
|
|
||||||
|
**Your agency:** Pick Figma-first, Stitch-first, or hybrid **per screen** — one line in the PR is enough. Dave assigns tickets; **Week 1 default ticket is Q-1 on the live `/` worklist** (highest yield).
|
||||||
|
|
||||||
|
**Week 1 principle — ship fast, correct, get feedback:**
|
||||||
|
|
||||||
|
|
||||||
|
| Priority | Rule |
|
||||||
|
| --------------- | ----------------------------------------------------------------------------------------------------- |
|
||||||
|
| **Speed** | Tools (Figma + Stitch) = **1 day max**, then code same day |
|
||||||
|
| **Yield** | Week 1 targets the **live route** `/` (`PatientWorklistPage`) mobile at 375px — not orphan stub pages |
|
||||||
|
| **Correctness** | `design-tokens.ts`, live navigation, no hardcoded colors, no pasted Stitch HTML |
|
||||||
|
| **Feedback** | Draft PR or screenshots to Dave **end of Day 1**; iterate Day 2–3 from review |
|
||||||
|
|
||||||
|
|
||||||
|
**Stitch (quick rules):**
|
||||||
|
|
||||||
|
1. Board has web mocks — use Stitch to sketch **mobile layout**, export screenshot for Cursor/Kilo
|
||||||
|
2. Agent writes React; you review every line
|
||||||
|
3. Figma wins on tokens and Vietnamese copy
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Tuần 1 = 1 ngày học tool, cùng ngày code mobile cho `/` worklist; PR/screenshot cuối ngày 1; sửa theo feedback Dave — không khảo sát tự do nhiều ngày.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Your codebase scope
|
||||||
|
|
||||||
|
### In scope — touch freely
|
||||||
|
|
||||||
|
```
|
||||||
|
CODEBASE/frontend/
|
||||||
|
├── spec/ ← read: uix_prototype_coverage, figma-component-specs, DESIGN
|
||||||
|
└── implementation/src/
|
||||||
|
├── pages/ ← your primary workspace
|
||||||
|
├── components/atoms/
|
||||||
|
├── components/molecules/
|
||||||
|
├── components/layout/
|
||||||
|
├── styles/design-tokens.ts ← token source of truth
|
||||||
|
├── App.tsx ← routes
|
||||||
|
└── public/assets/stitch/ ← exported Stitch references
|
||||||
|
```
|
||||||
|
|
||||||
|
### Read-only — understand, don't refactor
|
||||||
|
|
||||||
|
|
||||||
|
| File / folder | Why |
|
||||||
|
| ------------------------------------------ | -------------------------------- |
|
||||||
|
| `organisms/DiagnosticCanvas.tsx` | Dave owns DICOM canvas |
|
||||||
|
| `molecules/ClinicalChatPanel.tsx` | Read layout only — Phuc owns LLM |
|
||||||
|
| `lib/llm/`, `workers/` | Phuc's lane |
|
||||||
|
| `hooks/useClinicalChat.ts` | Know it exists |
|
||||||
|
| `data/mockData.ts`, `lib/patientStore.tsx` | How worklist gets data |
|
||||||
|
|
||||||
|
|
||||||
|
### Out of scope
|
||||||
|
|
||||||
|
`CODEBASE/backend/`, `CODEBASE/ml/`, `CODEBASE/knowledge/`, `CODEBASE/infra/`, `LEGACY/`
|
||||||
|
|
||||||
|
### Live routes today
|
||||||
|
|
||||||
|
```
|
||||||
|
/ → PatientWorklistPage.tsx ← Q-1, Q-6 priority
|
||||||
|
/workspace/:patientId → ClinicalWorkspacePage.tsx ← polish only; Q-2 overlay
|
||||||
|
+ 35 stub routes in src/pages/ ← Figma parity over time
|
||||||
|
```
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Sửa pages, atoms, molecules, tokens; không đụng canvas, chat LLM, backend.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. What to read, what to learn
|
||||||
|
|
||||||
|
### Must read (Day 1 morning only — then code)
|
||||||
|
|
||||||
|
|
||||||
|
| # | Document | Time |
|
||||||
|
| --- | ----------------------------------------------------------------- | ---------------------------------- |
|
||||||
|
| 1 | `uix_prototype_coverage.md` | 20 min — **which routes are live** |
|
||||||
|
| 2 | `figma-component-specs.md` — Priority 1–2 + node for `/` worklist | 15 min |
|
||||||
|
| 3 | `design-tokens.ts` + skim `DESIGN.md` | 15 min |
|
||||||
|
| 4 | `AGENT_SKILL/secrets_and_phi_safety/SKILL.md` | 10 min |
|
||||||
|
|
||||||
|
|
||||||
|
Read `component-inventory.md`, `coding_convention`, and persona §3.4 **while coding**, not before Day 1 PR.
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Sáng ngày 1 chỉ đọc 4 tài liệu ngắn; chiều cùng ngày bắt đầu code.
|
||||||
|
|
||||||
|
### Should read (when ticket assigned)
|
||||||
|
|
||||||
|
|
||||||
|
| Ticket | Read |
|
||||||
|
| ---------- | ---------------------------------------------------------------------------------------- |
|
||||||
|
| Q-1, Q-6 | Figma node for screen; matching `src/pages/*.tsx` stub |
|
||||||
|
| Q-2 | `uix_prototype_coverage.md` §Safety; `SafetyEscalationOverlay.tsx` |
|
||||||
|
| Q-3, Q-4 | `CONTEXT_VISION_SCOPE.md` §3.4; `frontend_spec.md` PoC § |
|
||||||
|
| Q-5 | `figma-component-specs.md` — error boundaries item |
|
||||||
|
| Q-8 | [Stitch board](https://stitch.withgoogle.com/projects/8188910238055386921) + live routes |
|
||||||
|
| Any screen | `Design_Material/FR_25_UC_DESIGN/Use_Case/UC-*.md` |
|
||||||
|
| Layout | `session_memory/26_Jun_26/Proj_context_26_Jun_26.md` — 65/35 ASCII |
|
||||||
|
|
||||||
|
|
||||||
|
### Learn deeper (keywords)
|
||||||
|
|
||||||
|
> Abbreviations: [Glossary INDEX §12](./ONBOARDING_INDEX.md#12-glossary-of-abbreviations)
|
||||||
|
|
||||||
|
|
||||||
|
| Keywords | When |
|
||||||
|
| ---------------------------------------------------------- | ------------------- |
|
||||||
|
| `atomic design`, `atoms molecules organisms` | component-inventory |
|
||||||
|
| `React functional components`, `props`, `composition` | all Q tickets |
|
||||||
|
| `design tokens`, `CSS variables`, `responsive breakpoints` | Q-1, Q-3 |
|
||||||
|
| `mobile-first`, `375px viewport`, `touch target 44px` | Q-3, Q-6 |
|
||||||
|
| `accessibility`, `WCAG contrast`, `font scaling` | Q-4, Q-6 |
|
||||||
|
| `error boundaries` | Q-5 |
|
||||||
|
| `PWA`, `service worker` | Sprint 4 (later) |
|
||||||
|
| `Figma Dev Mode`, `Inspect` | Q-1 |
|
||||||
|
| `Stitch design-to-code`, `HTML export as reference` | Q-8 |
|
||||||
|
| `Vite`, `npm run dev` | daily |
|
||||||
|
| `Vietnamese clinical UX`, `jargon-free patient copy` | Q-6 |
|
||||||
|
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Đọc thêm theo ticket khi code; không ôm 7 tài liệu trước PR đầu.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Dev toolkit
|
||||||
|
|
||||||
|
|
||||||
|
| Category | Tool |
|
||||||
|
| ------------------- | ------------------------------------------------------------------------------------------------- |
|
||||||
|
| Dev server | `npm run dev` → `http://localhost:5173` |
|
||||||
|
| Design authority | [Figma](https://www.figma.com/design/uakk3T4efS1AIckfTEFSZO/MSK_Workspace_UIX-Design?node-id=0-1) |
|
||||||
|
| Design acceleration | [Stitch whiteboard](https://stitch.withgoogle.com/projects/8188910238055386921) |
|
||||||
|
| Coding agent | Cursor / Kilo |
|
||||||
|
| Mobile test | Chrome DevTools → iPhone SE 375×667 |
|
||||||
|
| PR evidence | Before/after screenshots desktop + 375px |
|
||||||
|
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation
|
||||||
|
npm install
|
||||||
|
npm run dev
|
||||||
|
```
|
||||||
|
|
||||||
|
**Optional:** Stitch SDK — `public/assets/stitch/README.md` (OAuth via Dave)
|
||||||
|
|
||||||
|
### Agent prompt template
|
||||||
|
|
||||||
|
```
|
||||||
|
Implement [screen name] for mobile (375px).
|
||||||
|
Reference: Figma node [id] from figma-component-specs.md.
|
||||||
|
Layout hint: [Stitch mobile export screenshot or description].
|
||||||
|
Use existing atoms from component-inventory.md.
|
||||||
|
Use design-tokens.ts — no hardcoded colors.
|
||||||
|
Route: [path from figma-component-specs].
|
||||||
|
Match navigation in uix_prototype_coverage.md.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** npm run dev + Figma + Stitch + Cursor; test 375px; PR kèm screenshot.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Figma + Stitch workflow
|
||||||
|
|
||||||
|
### When to use which
|
||||||
|
|
||||||
|
|
||||||
|
| Source | Best for | Caveat |
|
||||||
|
| ------------- | --------------------------------------------- | --------------------------------------- |
|
||||||
|
| **Figma** | Tokens, spacing, VI copy, auth/patient cards | Slower from scratch — use Inspect |
|
||||||
|
| **Stitch** | Rapid mobile layout; agent reference material | Web mock ≠ mobile — create mobile frame |
|
||||||
|
| **Live code** | Navigation truth | `uix_prototype_coverage.md` |
|
||||||
|
|
||||||
|
|
||||||
|
**Legacy Stitch desktop** (`public/assets/stitch/README.md`, project `17628337625105266704`) — reference only.
|
||||||
|
**Primary Stitch:** [project 8188910238055386921](https://stitch.withgoogle.com/projects/8188910238055386921)
|
||||||
|
|
||||||
|
### Workflow choices (pick one per screen — state in PR)
|
||||||
|
|
||||||
|
|
||||||
|
| ID | When | Steps |
|
||||||
|
| -------------------- | -------------------------------------------------- | --------------------------------------------- |
|
||||||
|
| **A — Figma-first** | Screen in node map; tokens matter | Inspect → atoms → `src/pages/` |
|
||||||
|
| **B — Stitch-first** | Mobile patient screen; Stitch mobile variant ready | Stitch mobile design → export → agent → React |
|
||||||
|
| **C — Hybrid** | Stitch layout close; Figma has correct copy | Stitch structure + Figma polish → code |
|
||||||
|
|
||||||
|
|
||||||
|
**Rule:** Final PR must match live navigation (`/` → `/workspace/:id`), not a disconnected mock.
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Chọn A/B/C mỗi màn hình; Figma thắng khi khác Stitch; khớp route thật.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Week 1 sprint — tools in 1 day, ship mobile same week
|
||||||
|
|
||||||
|
> **No multi-day exploration.** Figma/Stitch are means to an end — the end is a **merged mobile PR** on a live screen.
|
||||||
|
|
||||||
|
### Day 1 — Tools (morning) + code (afternoon)
|
||||||
|
|
||||||
|
|
||||||
|
| Block | Time | Do |
|
||||||
|
| ------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
| **Setup** | 30 min | `npm install && npm run dev` — open `/` on desktop + DevTools 375px |
|
||||||
|
| **Figma** | 45 min | Open [Figma](https://www.figma.com/design/uakk3T4efS1AIckfTEFSZO/MSK_Workspace_UIX-Design?node-id=0-1) → Inspect **Patient List** node `82-3155` (maps to live worklist) |
|
||||||
|
| **Stitch** | 45 min | Open [Stitch board](https://stitch.withgoogle.com/projects/8188910238055386921) → one mobile-frame sketch OR pick closest web mock → **screenshot for agent** |
|
||||||
|
| **Must-read** | 60 min | Section 3 four docs only |
|
||||||
|
| **Code** | Rest of day | **Q-1 on live `/`** — mobile layout for `PatientWorklistPage.tsx` (or `patient-list-main.tsx` if Dave assigns stub route). Cursor/Kilo + agent template (§4). **375px from the start.** |
|
||||||
|
| **EOD** | 15 min | Send Dave: draft PR **or** before/after screenshots at 375px + 1-line workflow choice (A/B/C) |
|
||||||
|
|
||||||
|
|
||||||
|
**Not acceptable as Week 1 deliverable:** gap lists, alignment essays, clicking all 35 routes, Storybook setup.
|
||||||
|
|
||||||
|
### Day 2 — Feedback loop
|
||||||
|
|
||||||
|
- Apply Dave's review on spacing, tokens, navigation, touch targets
|
||||||
|
- Push PR updates; aim for **merge-ready** by EOD if scope was ≤200 lines
|
||||||
|
|
||||||
|
### Day 3–5 — Second yield
|
||||||
|
|
||||||
|
- Merge Week 1 PR OR complete merge after small fixes
|
||||||
|
- Start **second screen** (patient card stub or Q-3 polish on same `/` route)
|
||||||
|
- Optional: update one Stitch frame to match what you shipped (Q-8 lite — no separate doc)
|
||||||
|
|
||||||
|
### Week 1 success criteria
|
||||||
|
|
||||||
|
|
||||||
|
| Check | Pass |
|
||||||
|
| ------------------------------ | ---------------------------------------------- |
|
||||||
|
| Live route `/` usable at 375px | No horizontal scroll |
|
||||||
|
| Uses `design-tokens.ts` | No raw hex in new code |
|
||||||
|
| Dave reviewed once | By end of Day 1 or 2 AM |
|
||||||
|
| Runnable demo | `npm run dev` — you walk Dave through tap flow |
|
||||||
|
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Ngày 1: tool buổi sáng, code buổi chiều, gửi Dave tối ngày 1; ngày 2 sửa feedback; ngày 3–5 merge + màn hình thứ 2.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. What exists vs what you build
|
||||||
|
|
||||||
|
|
||||||
|
| Exists | You increment |
|
||||||
|
| ---------------------------------- | -------------------------------------------- |
|
||||||
|
| WS-25 workspace PoC (functional) | Mobile polish, not rewrite |
|
||||||
|
| 35 Figma stub routes | Q-1: 2 priority screens |
|
||||||
|
| Stitch web mocks | Q-8 lite: update frame after ship (optional) |
|
||||||
|
| `SafetyEscalationOverlay` (orphan) | Q-2: wire it |
|
||||||
|
| No error boundaries | Q-5 |
|
||||||
|
| No patient PWA | Q-6 |
|
||||||
|
|
||||||
|
|
||||||
|
### WS-25 layout (read-only context)
|
||||||
|
|
||||||
|
Clinician workspace is 65/35 split — your zone is **patient list, cards, patient dashboard**, not DICOM canvas:
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─ TopBar ─────────────────────────────────────┐
|
||||||
|
├─ Zone A (65%) DICOM │ Zone B (35%) sidebar ┤ ← Dave/Phuc zone
|
||||||
|
└────────────────────────────────────────────────┘
|
||||||
|
|
||||||
|
Your zone: / worklist, /patients, patient-card-*, /home
|
||||||
|
```
|
||||||
|
|
||||||
|
See `session_memory/26_Jun_26/Proj_context_26_Jun_26.md` for full ASCII.
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Workspace clinician đã có; bạn làm patient list, card, dashboard mobile.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Your backlog (detail)
|
||||||
|
|
||||||
|
|
||||||
|
| ID | Gap | Priority screens / files |
|
||||||
|
| ------- | ----------------------- | --------------------------------------------------------------------------------------- |
|
||||||
|
| **Q-1** | Stub → Figma fidelity | Home, Login, Patient List, Patient Cards (see `figma-component-specs.md` §Priorities) |
|
||||||
|
| **Q-2** | Safety overlay unrouted | `SafetyEscalationOverlay.tsx`, `App.tsx` |
|
||||||
|
| **Q-3** | No 375px pass | Assigned patient screen |
|
||||||
|
| **Q-4** | Accessibility | Patient-facing view |
|
||||||
|
| **Q-5** | Error boundaries | `App.tsx` workspace route |
|
||||||
|
| **Q-6** | Patient dashboard | `patient-card-*.tsx`, FR-26 preview |
|
||||||
|
| **Q-7** | Zero-GPU UI banner | Dave provides detection hook |
|
||||||
|
| **Q-8** | Stitch sync (lite) | Update Stitch frame **after** you ship — optional in PR, not a standalone proposal gate |
|
||||||
|
|
||||||
|
|
||||||
|
**Figma priorities (from spec):**
|
||||||
|
|
||||||
|
1. Home, Login, Create Account
|
||||||
|
2. Patient List + Cards
|
||||||
|
3. Workspace shell (read-only for you)
|
||||||
|
4. Auth flows
|
||||||
|
5. AI View, Chat, Ultrasound
|
||||||
|
6. Remaining screens
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Ưu tiên Home, Patient List, Patient Card trước.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. SMART milestones
|
||||||
|
|
||||||
|
### M1 — Week 1 ship (Jul 7 – Jul 11) ← **start here**
|
||||||
|
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| -------------- | ------------------------------------------------------------------------------------------------------- |
|
||||||
|
| **Ticket** | **Q-1 on live `/` worklist** (Dave confirms file: `PatientWorklistPage.tsx` or `patient-list-main.tsx`) |
|
||||||
|
| **Specific** | Mobile implementation at 375px on a **live** route — not a disconnected stub |
|
||||||
|
| **Measurable** | Draft PR Day 1 EOD; merged or merge-ready by Day 5; before/after screenshots |
|
||||||
|
| **Achievable** | ≤200 lines; tools only Day 1 morning |
|
||||||
|
| **Relevant** | CP-5 — fastest visible yield |
|
||||||
|
| **Feedback** | Dave review ≤24h after first push |
|
||||||
|
|
||||||
|
|
||||||
|
### M2 — Second screen (Jul 14 – Jul 25)
|
||||||
|
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| --------------- | ------------------------------------------------------------------- |
|
||||||
|
| **Ticket** | Q-1 #2 (patient card / home) or Q-3 deepen mobile on shipped screen |
|
||||||
|
| **Deliverable** | 2nd merged PR |
|
||||||
|
| **Relevant** | CP-5 / CP-6 |
|
||||||
|
|
||||||
|
|
||||||
|
### M3 — Mobile hardening (Jul 28 – Aug 11)
|
||||||
|
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| --------------- | -------------------------------------------------------------- |
|
||||||
|
| **Ticket** | Q-3 or Q-2 |
|
||||||
|
| **Deliverable** | 375px recording; taps ≥44px; optional Q-8 Stitch frame updated |
|
||||||
|
| **Relevant** | CP-6 |
|
||||||
|
|
||||||
|
|
||||||
|
### M4 — Accessibility (Aug 4 – Aug 21) *optional*
|
||||||
|
|
||||||
|
|
||||||
|
| | |
|
||||||
|
| --------------- | ----------------------------------------- |
|
||||||
|
| **Ticket** | Q-4 |
|
||||||
|
| **Deliverable** | High contrast, large type on patient view |
|
||||||
|
| **Relevant** | CP-7 |
|
||||||
|
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** M1 = tuần 1 merge mobile `/` worklist; không có M0 khảo sát riêng; M2–M4 mở rộng sau khi có feedback.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Per-ticket workflow
|
||||||
|
|
||||||
|
```
|
||||||
|
1. Dave assigns Q-n
|
||||||
|
2. Pick workflow A / B / C — write in PR description
|
||||||
|
3. Capture Figma + Stitch reference screenshots
|
||||||
|
4. Cursor/Kilo: use agent prompt template (Section 4)
|
||||||
|
5. Review every line of the diff — understand before commit
|
||||||
|
6. npm run dev → screenshot desktop + 375px
|
||||||
|
7. PR: Q-n, CP-n, workflow choice, design links
|
||||||
|
```
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Mỗi ticket: chọn workflow → agent hỗ trợ → review diff → screenshot → PR.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Boundaries
|
||||||
|
|
||||||
|
**Touch:** `src/pages/`, `src/components/` (atoms/molecules/layout), `src/styles/`, `App.tsx`, Stitch whiteboard, `public/assets/stitch/`
|
||||||
|
|
||||||
|
**Do not touch:** `src/lib/llm/`, `src/workers/`, backend, ml
|
||||||
|
|
||||||
|
**Figma/Stitch edits:** OK to update Stitch frames and Figma comments; ask Dave before changing shared Figma components.
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Chỉ frontend UI; hỏi Dave trước khi sửa component Figma dùng chung.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Definition of done
|
||||||
|
|
||||||
|
- [ ] Week 1: live `/` worklist mobile at 375px shipped or merge-ready
|
||||||
|
- [ ] Dave feedback incorporated within 48h of first push
|
||||||
|
- [ ] ≥2 screens merged by end of Month 1 (M1 + M2)
|
||||||
|
- [ ] Each PR: workflow A/B/C (one line) + 375px screenshot
|
||||||
|
- [ ] Never committed `secrets/` or PHI
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Tuần 1 phải có mobile `/` chạy được + feedback Dave; không cần báo cáo alignment dài.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. Agent prompt examples (optimize / change based on usecase)
|
||||||
|
|
||||||
|
**Learn the codebase:**
|
||||||
|
|
||||||
|
```
|
||||||
|
Explain PatientWorklistPage.tsx in simple terms: what components it uses,
|
||||||
|
where data comes from, and which Figma node it maps to.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Start Q-1 (Figma-first):**
|
||||||
|
|
||||||
|
```
|
||||||
|
Implement /home per Figma node 33-143 in figma-component-specs.md.
|
||||||
|
Reuse Button and Badge from component-inventory. Use design-tokens.ts only.
|
||||||
|
Current stub: src/pages/home.tsx
|
||||||
|
```
|
||||||
|
|
||||||
|
**Week 1 — ship live worklist mobile (Day 1 afternoon):**
|
||||||
|
|
||||||
|
```
|
||||||
|
Mobile pass for PatientWorklistPage at 375px.
|
||||||
|
Figma node 82-3155. Stitch screenshot attached.
|
||||||
|
Use design-tokens.ts and existing Button, Badge, PatientCard from component-inventory.
|
||||||
|
No horizontal scroll. Touch targets >= 44px.
|
||||||
|
Do not change routing — stay on /.
|
||||||
|
Show me the minimal diff.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Start Q-3 (mobile pass):**
|
||||||
|
|
||||||
|
```
|
||||||
|
Make patient-list-main.tsx usable at 375px: no horizontal scroll,
|
||||||
|
touch targets at least 44px. Show me which CSS to change.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 14. Bonus — Feynman Technique prompts
|
||||||
|
|
||||||
|
Use when React layout, routing, or design workflow feels confusing **before** you commit.
|
||||||
|
Master template: [INDEX §9](./ONBOARDING_INDEX.md#9-bonus--feynman-technique-with-agents).
|
||||||
|
|
||||||
|
### Hard concepts → ready-made prompts
|
||||||
|
|
||||||
|
**Live route vs stub route (Week 1 critical):**
|
||||||
|
```
|
||||||
|
Feynman explain the difference between our LIVE routes (/ and /workspace/:id)
|
||||||
|
and the 35 Figma STUB routes in src/pages/. Why does Week 1 target
|
||||||
|
PatientWorklistPage on / ? Use a "demo kitchen vs blueprint room" analogy.
|
||||||
|
Quiz me.
|
||||||
|
```
|
||||||
|
|
||||||
|
**PatientWorklistPage data flow:**
|
||||||
|
```
|
||||||
|
Feynman explain PatientWorklistPage.tsx: where patient list data comes from
|
||||||
|
(mockData vs patientStore), what happens when I tap "Mở phiên chẩn đoán",
|
||||||
|
and which components re-render. ≤8 steps. No jargon without definition. Quiz me.
|
||||||
|
```
|
||||||
|
|
||||||
|
**design-tokens.ts vs hardcoded CSS:**
|
||||||
|
```
|
||||||
|
Feynman explain why we use design-tokens.ts instead of #hex in components.
|
||||||
|
Show one wrong example and one right example from our codebase.
|
||||||
|
Quiz me on how to add a new spacing value correctly.
|
||||||
|
```
|
||||||
|
|
||||||
|
**375px mobile layout:**
|
||||||
|
```
|
||||||
|
Feynman explain mobile-first pass at 375px for our worklist: flex vs grid,
|
||||||
|
overflow-x, and 44px touch targets. What is the smallest change to
|
||||||
|
PatientListShell or PatientWorklistPage to stop horizontal scroll?
|
||||||
|
Teach only — no code until I answer your quiz.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Figma vs Stitch vs code (your workflow A/B/C):**
|
||||||
|
```
|
||||||
|
Feynman explain when I should use Figma-first vs Stitch-first vs hybrid
|
||||||
|
for a patient screen. What is "reference only" Stitch export vs production code?
|
||||||
|
Quiz me with one scenario: patient card on mobile.
|
||||||
|
```
|
||||||
|
|
||||||
|
**React atoms vs molecules (component-inventory):**
|
||||||
|
```
|
||||||
|
Feynman explain atomic design using OUR component-inventory.md: when is
|
||||||
|
something a Button (atom) vs PatientHeader (molecule)? Give 2 examples
|
||||||
|
from patient list screen. Quiz me.
|
||||||
|
```
|
||||||
|
|
||||||
|
### Feynman before commit rule
|
||||||
|
|
||||||
|
If the agent wrote code and you **cannot** Feynman-explain the diff back to a friend in 30 seconds → don't commit yet; run one Feynman prompt on the confusing file.
|
||||||
|
|
||||||
|
**Tóm tắt (VI):** Hỏi agent giải thích đơn giản khi khó hiểu React/route/mobile; có prompt mẫu; không commit code mình chưa giải thích lại được.
|
||||||
@@ -31,7 +31,7 @@ os.environ.setdefault("TRITON_ENDPOINT", DEFAULT_TRITON_ENDPOINT)
|
|||||||
import uvicorn
|
import uvicorn
|
||||||
from fastapi import FastAPI
|
from fastapi import FastAPI
|
||||||
from fastapi.middleware.cors import CORSMiddleware
|
from fastapi.middleware.cors import CORSMiddleware
|
||||||
|
import asyncio
|
||||||
from backend.routers import cv_inference
|
from backend.routers import cv_inference
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
@@ -44,10 +44,7 @@ async def lifespan(app: FastAPI):
|
|||||||
logger.info("Starting CV inference service on Triton: %s", os.getenv("TRITON_ENDPOINT"))
|
logger.info("Starting CV inference service on Triton: %s", os.getenv("TRITON_ENDPOINT"))
|
||||||
from backend.services.triton_warmup import warmup_triton_models
|
from backend.services.triton_warmup import warmup_triton_models
|
||||||
|
|
||||||
try:
|
warmup_task = asyncio.create_task(warmup_triton_models())
|
||||||
await warmup_triton_models()
|
|
||||||
except Exception as exc:
|
|
||||||
logger.warning("Triton warmup failed — first clinical request may be slow: %s", exc)
|
|
||||||
yield
|
yield
|
||||||
logger.info("Shutting down CV inference service")
|
logger.info("Shutting down CV inference service")
|
||||||
|
|
||||||
|
|||||||
@@ -3,16 +3,34 @@ import json
|
|||||||
from typing import Any
|
from typing import Any
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import requests
|
import requests
|
||||||
|
from requests.adapters import HTTPAdapter
|
||||||
|
from urllib3.util.retry import Retry
|
||||||
|
|
||||||
|
|
||||||
class TritonAdapter:
|
class TritonAdapter:
|
||||||
def __init__(self, endpoint_url: str, timeout: float = 60.0):
|
def __init__(self, endpoint_url: str, timeout: float = 60.0):
|
||||||
self.endpoint_url = endpoint_url.rstrip("/")
|
self.endpoint_url = endpoint_url.rstrip("/")
|
||||||
self.timeout = timeout
|
self.timeout = timeout
|
||||||
|
self._session = self._build_session()
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _build_session() -> requests.Session:
|
||||||
|
session = requests.Session()
|
||||||
|
retry = Retry(
|
||||||
|
total=0,
|
||||||
|
connect=0,
|
||||||
|
read=0,
|
||||||
|
redirect=0,
|
||||||
|
status=0,
|
||||||
|
raise_on_status=False,
|
||||||
|
)
|
||||||
|
adapter = HTTPAdapter(max_retries=retry, pool_connections=20, pool_maxsize=50)
|
||||||
|
session.mount("http://", adapter)
|
||||||
|
session.mount("https://", adapter)
|
||||||
|
return session
|
||||||
|
|
||||||
async def close(self):
|
async def close(self):
|
||||||
pass
|
await asyncio.to_thread(self._session.close)
|
||||||
|
|
||||||
|
|
||||||
async def infer(
|
async def infer(
|
||||||
@@ -61,7 +79,7 @@ class TritonAdapter:
|
|||||||
}
|
}
|
||||||
|
|
||||||
url = f"{self.endpoint_url}/v2/models/{model_name}/infer"
|
url = f"{self.endpoint_url}/v2/models/{model_name}/infer"
|
||||||
response = requests.post(url, data=body, headers=headers, timeout=self.timeout)
|
response = self._session.post(url, data=body, headers=headers, timeout=self.timeout)
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
return self._parse_binary_response(response.headers, response.content)
|
return self._parse_binary_response(response.headers, response.content)
|
||||||
|
|
||||||
@@ -91,7 +109,7 @@ class TritonAdapter:
|
|||||||
|
|
||||||
def _model_ready_sync(self, model_name: str) -> bool:
|
def _model_ready_sync(self, model_name: str) -> bool:
|
||||||
url = f"{self.endpoint_url}/v2/models/{model_name}"
|
url = f"{self.endpoint_url}/v2/models/{model_name}"
|
||||||
response = requests.get(url, timeout=self.timeout)
|
response = self._session.get(url, timeout=self.timeout)
|
||||||
if response.status_code == 404:
|
if response.status_code == 404:
|
||||||
return False
|
return False
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
@@ -113,7 +131,7 @@ class TritonAdapter:
|
|||||||
url = f"{self.endpoint_url}/v2/repository/index"
|
url = f"{self.endpoint_url}/v2/repository/index"
|
||||||
|
|
||||||
# 2. Change requests.get to requests.post with an empty json payload {}
|
# 2. Change requests.get to requests.post with an empty json payload {}
|
||||||
response = requests.post(url, json={}, timeout=self.timeout)
|
response = self._session.post(url, json={}, timeout=self.timeout)
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
|
|
||||||
data = response.json()
|
data = response.json()
|
||||||
|
|||||||
@@ -0,0 +1,50 @@
|
|||||||
|
import asyncio
|
||||||
|
import base64
|
||||||
|
import io
|
||||||
|
import logging
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
from backend.implementation.postprocessing.calibration import calibration_config_from_params
|
||||||
|
from backend.services.cv_inference_service import CvInferenceOptions, run_single
|
||||||
|
from backend.services.celery_app import celery_app
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def _decode_base64_to_pil(b64_str: str) -> Image.Image:
|
||||||
|
raw = base64.b64decode(b64_str)
|
||||||
|
return Image.open(io.BytesIO(raw)).convert("RGB")
|
||||||
|
|
||||||
|
|
||||||
|
@celery_app.task(bind=True, name="cv_inference.run_chunk", max_retries=2, default_retry_delay=10)
|
||||||
|
def run_cv_chunk(self, chunk_payload: dict) -> list[dict]:
|
||||||
|
images_b64 = chunk_payload.get("images_b64", [])
|
||||||
|
frame_ids = chunk_payload.get("frame_ids", [])
|
||||||
|
calibration_params = chunk_payload.get("calibration", {})
|
||||||
|
model_versions = chunk_payload.get("model_versions")
|
||||||
|
|
||||||
|
if not images_b64:
|
||||||
|
return []
|
||||||
|
|
||||||
|
images = [_decode_base64_to_pil(b64) for b64 in images_b64]
|
||||||
|
calibration = calibration_config_from_params(calibration_params)
|
||||||
|
options = CvInferenceOptions(
|
||||||
|
calibration=calibration,
|
||||||
|
model_versions=model_versions,
|
||||||
|
use_cache=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def _run():
|
||||||
|
return await asyncio.gather(*[
|
||||||
|
run_single(img, frame_id=fid, options=options)
|
||||||
|
for img, fid in zip(images, frame_ids)
|
||||||
|
])
|
||||||
|
|
||||||
|
try:
|
||||||
|
results = asyncio.run(_run())
|
||||||
|
return results
|
||||||
|
except Exception as exc:
|
||||||
|
logger.exception("Celery chunk task failed: %s", exc)
|
||||||
|
raise self.retry(exc=exc, countdown=10, max_retries=2)
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
import logging
|
||||||
|
import sys
|
||||||
|
from logging.handlers import RotatingFileHandler
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
LOG_DIR = Path("logs")
|
||||||
|
LOG_DIR.mkdir(exist_ok=True)
|
||||||
|
LOG_FORMAT = "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s"
|
||||||
|
LOG_DATE_FORMAT = "%Y-%m-%d %H:%M:%S"
|
||||||
|
def setup_logging():
|
||||||
|
root = logging.getLogger()
|
||||||
|
root.setLevel(logging.DEBUG)
|
||||||
|
root.handlers.clear()
|
||||||
|
formatter = logging.Formatter(LOG_FORMAT, datefmt=LOG_DATE_FORMAT)
|
||||||
|
console = logging.StreamHandler(sys.stdout)
|
||||||
|
console.setLevel(logging.INFO)
|
||||||
|
console.setFormatter(formatter)
|
||||||
|
root.addHandler(console)
|
||||||
|
file_handler = RotatingFileHandler(
|
||||||
|
LOG_DIR / "app.log",
|
||||||
|
maxBytes=10 * 1024 * 1024,
|
||||||
|
backupCount=5,
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
file_handler.setLevel(logging.DEBUG)
|
||||||
|
file_handler.setFormatter(formatter)
|
||||||
|
root.addHandler(file_handler)
|
||||||
|
logging.getLogger("httpx").setLevel(logging.WARNING)
|
||||||
|
logging.getLogger("uvicorn.access").setLevel(logging.INFO)
|
||||||
@@ -18,6 +18,10 @@ from backend.implementation.triton_batch import TRITON_MAX_BATCH_SIZE
|
|||||||
from backend.services import cv_result_cache
|
from backend.services import cv_result_cache
|
||||||
from backend.services import triton_runtime_service as triton_runtime
|
from backend.services import triton_runtime_service as triton_runtime
|
||||||
from backend.services.cv_inference_service import CvBatchResult, CvInferenceOptions, run_batch, run_single
|
from backend.services.cv_inference_service import CvBatchResult, CvInferenceOptions, run_batch, run_single
|
||||||
|
from backend.services import cv_celery_service
|
||||||
|
from backend.logging.logging_config import setup_logging
|
||||||
|
|
||||||
|
setup_logging()
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -135,14 +139,15 @@ async def cv_inference_health():
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.post("/analyze")
|
@router.post("/analyze") # deprecated
|
||||||
async def analyze_upload(
|
async def analyze_upload(
|
||||||
image: UploadFile = File(...),
|
image: UploadFile = File(...),
|
||||||
calibration: str | None = Form(default=None),
|
calibration: str | None = Form(default=None),
|
||||||
):
|
):
|
||||||
"""Spec-compliant CV pipeline: CLAHE → angle → inflammation → conditional segmentation."""
|
"""Spec-compliant CV pipeline: CLAHE → angle → inflammation → conditional segmentation."""
|
||||||
image_pil = await _load_upload_image(image)
|
image_pil = await _load_upload_image(image)
|
||||||
options = _build_options(_parse_calibration_form(calibration), use_cache=False)
|
|
||||||
|
options = _build_options(_parse_calibration_form(calibration), use_cache=True)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
result = await run_single(image_pil, frame_id=None, options=options)
|
result = await run_single(image_pil, frame_id=None, options=options)
|
||||||
@@ -168,11 +173,13 @@ async def analyze_batch_upload(
|
|||||||
calibration: str | None = Form(default=None),
|
calibration: str | None = Form(default=None),
|
||||||
):
|
):
|
||||||
"""Spec-compliant CV batch — one full pipeline per frame (angle-first, gated segmentation)."""
|
"""Spec-compliant CV batch — one full pipeline per frame (angle-first, gated segmentation)."""
|
||||||
|
logger.info("Starting analyze batch upload")
|
||||||
if not images:
|
if not images:
|
||||||
raise HTTPException(status_code=400, detail="At least one image is required")
|
raise HTTPException(status_code=400, detail="At least one image is required")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
id_list = json.loads(frame_ids)
|
id_list = json.loads(frame_ids)
|
||||||
|
# logger.info("Start to check the id_list")
|
||||||
except json.JSONDecodeError as exc:
|
except json.JSONDecodeError as exc:
|
||||||
raise HTTPException(status_code=400, detail="frame_ids must be a JSON array of strings") from exc
|
raise HTTPException(status_code=400, detail="frame_ids must be a JSON array of strings") from exc
|
||||||
|
|
||||||
@@ -213,6 +220,68 @@ async def analyze_batch_upload(
|
|||||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/analyze/batch/celery")
|
||||||
|
async def analyze_batch_celery(
|
||||||
|
images: list[UploadFile] = File(...),
|
||||||
|
frame_ids: str = Form(...),
|
||||||
|
calibration: str | None = Form(default=None),
|
||||||
|
):
|
||||||
|
"""Experiment: async chunk fan-out via Celery + Redis."""
|
||||||
|
if not images:
|
||||||
|
raise HTTPException(status_code=400, detail="At least one image is required")
|
||||||
|
|
||||||
|
try:
|
||||||
|
id_list = json.loads(frame_ids)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise HTTPException(status_code=400, detail="frame_ids must be a JSON array of strings") from exc
|
||||||
|
|
||||||
|
if not isinstance(id_list, list) or not all(isinstance(x, str) for x in id_list):
|
||||||
|
raise HTTPException(status_code=400, detail="frame_ids must be a JSON array of strings")
|
||||||
|
if len(id_list) != len(images):
|
||||||
|
raise HTTPException(status_code=400, detail="frame_ids length must match images count")
|
||||||
|
|
||||||
|
image_pils: list[Image.Image] = []
|
||||||
|
for upload in images:
|
||||||
|
image_pils.append(await _load_upload_image(upload))
|
||||||
|
|
||||||
|
options = _build_options(_parse_calibration_form(calibration))
|
||||||
|
|
||||||
|
try:
|
||||||
|
job_id = cv_celery_service.submit_celery_batch(image_pils, id_list, options)
|
||||||
|
return JSONResponse({
|
||||||
|
"success": True,
|
||||||
|
"job_id": job_id,
|
||||||
|
"image_count": len(image_pils),
|
||||||
|
"mode": "celery-chunk-fanout",
|
||||||
|
"chunk_size": cv_celery_service.CELERY_CHUNK_SIZE,
|
||||||
|
})
|
||||||
|
except Exception as exc:
|
||||||
|
logger.exception("Celery batch submit failed")
|
||||||
|
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/analyze/batch/celery/{job_id}")
|
||||||
|
async def analyze_batch_celery_result(job_id: str):
|
||||||
|
"""Poll result for a Celery chunk-fan-out batch job."""
|
||||||
|
try:
|
||||||
|
result = await cv_celery_service.get_celery_batch_result(job_id)
|
||||||
|
status = result.get("status")
|
||||||
|
if status == "pending":
|
||||||
|
return JSONResponse(
|
||||||
|
status_code=202,
|
||||||
|
content=result,
|
||||||
|
)
|
||||||
|
if status == "unknown":
|
||||||
|
return JSONResponse(
|
||||||
|
status_code=404,
|
||||||
|
content=result,
|
||||||
|
)
|
||||||
|
return JSONResponse(result)
|
||||||
|
except Exception as exc:
|
||||||
|
logger.exception("Celery batch result fetch failed")
|
||||||
|
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||||
|
|
||||||
|
|
||||||
@router.post("/segment")
|
@router.post("/segment")
|
||||||
@router.post("/segment/batch")
|
@router.post("/segment/batch")
|
||||||
@router.post("/angle")
|
@router.post("/angle")
|
||||||
|
|||||||
27
workspace/sprint_1_2/CODEBASE/backend/routers/run_cv_inference.sh
Executable file
27
workspace/sprint_1_2/CODEBASE/backend/routers/run_cv_inference.sh
Executable file
@@ -0,0 +1,27 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
#
|
||||||
|
# Launch the standalone CV inference FastAPI server (Modal Triton).
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./backend/run_cv_inference.sh
|
||||||
|
#
|
||||||
|
# Works from any directory: it resolves the CODEBASE root relative to this
|
||||||
|
# script, sets PYTHONPATH so `import backend...` resolves, then runs the
|
||||||
|
# server as a module.
|
||||||
|
#
|
||||||
|
# Override defaults via env vars, e.g.:
|
||||||
|
# CV_INFERENCE_PORT=8080 ./backend/run_cv_inference.sh
|
||||||
|
#
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# CODEBASE root = grandparent dir of this script's directory (script lives in backend/routers/).
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
CODEBASE_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
||||||
|
|
||||||
|
cd "${CODEBASE_ROOT}"
|
||||||
|
|
||||||
|
export PYTHONPATH="${CODEBASE_ROOT}:${PYTHONPATH:-}"
|
||||||
|
|
||||||
|
# exec python -m backend.cv_inference_server
|
||||||
|
|
||||||
|
exec uvicorn backend.cv_inference_server:app --host 0.0.0.0 --port ${CV_INFERENCE_PORT:-8001}
|
||||||
22
workspace/sprint_1_2/CODEBASE/backend/services/celery_app.py
Normal file
22
workspace/sprint_1_2/CODEBASE/backend/services/celery_app.py
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
from celery import Celery
|
||||||
|
|
||||||
|
from backend.implementation import config
|
||||||
|
|
||||||
|
celery_app = Celery(
|
||||||
|
"cv-inference",
|
||||||
|
broker=f"redis://{config.REDIS_HOST}:{config.REDIS_PORT}/{config.REDIS_DB}",
|
||||||
|
backend=f"redis://{config.REDIS_HOST}:{config.REDIS_PORT}/{config.REDIS_DB}",
|
||||||
|
# Explicit include — autodiscover looks for tasks.py, not cv_tasks.py
|
||||||
|
include=["backend.implementation.tasks.cv_tasks"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# Must match the @task(name=...) value, not the Python module path
|
||||||
|
celery_app.conf.task_routes = {
|
||||||
|
"cv_inference.run_chunk": {"queue": "cv-inference"},
|
||||||
|
}
|
||||||
|
|
||||||
|
celery_app.conf.task_serializer = "json"
|
||||||
|
celery_app.conf.result_serializer = "json"
|
||||||
|
celery_app.conf.accept_content = ["json"]
|
||||||
|
celery_app.conf.result_expires = 3600
|
||||||
|
celery_app.conf.task_default_queue = "cv-inference"
|
||||||
@@ -0,0 +1,130 @@
|
|||||||
|
import base64
|
||||||
|
import dataclasses
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from backend.services.cv_inference_service import CvInferenceOptions, _encode_image_to_bytes
|
||||||
|
from backend.implementation.tasks.cv_tasks import run_cv_chunk
|
||||||
|
from backend.services.celery_app import celery_app
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
CELERY_CHUNK_SIZE = int(os.getenv("CELERY_CHUNK_SIZE", "4"))
|
||||||
|
CELERY_BATCH_POLL_CACHE_TTL_MS = float(os.getenv("CELERY_BATCH_POLL_CACHE_TTL_MS", "2000"))
|
||||||
|
|
||||||
|
_batch_poll_cache: dict[str, tuple[float, dict[str, Any]]] = {}
|
||||||
|
_batch_timings: dict[str, float] = {}
|
||||||
|
|
||||||
|
|
||||||
|
def submit_celery_batch(images, frame_ids, options):
|
||||||
|
if not images:
|
||||||
|
raise ValueError("images must not be empty")
|
||||||
|
if len(frame_ids) != len(images):
|
||||||
|
raise ValueError("frame_ids length must match images length")
|
||||||
|
|
||||||
|
from celery import group
|
||||||
|
|
||||||
|
chunks = []
|
||||||
|
for i in range(0, len(images), CELERY_CHUNK_SIZE):
|
||||||
|
chunk_imgs = images[i : i + CELERY_CHUNK_SIZE]
|
||||||
|
chunk_fids = frame_ids[i : i + CELERY_CHUNK_SIZE]
|
||||||
|
b64_images = [
|
||||||
|
base64.b64encode(_encode_image_to_bytes(img)).decode() for img in chunk_imgs
|
||||||
|
]
|
||||||
|
chunk_payload = {
|
||||||
|
"images_b64": b64_images,
|
||||||
|
"frame_ids": chunk_fids,
|
||||||
|
"calibration": dataclasses.asdict(options.calibration) if options.calibration else {},
|
||||||
|
"model_versions": options.model_versions,
|
||||||
|
}
|
||||||
|
chunks.append(chunk_payload)
|
||||||
|
|
||||||
|
job = group(run_cv_chunk.s(chunk) for chunk in chunks)
|
||||||
|
result = job.apply_async()
|
||||||
|
try:
|
||||||
|
result.save()
|
||||||
|
except Exception:
|
||||||
|
logger.exception("Failed to persist Celery GroupResult for job %s", result.id)
|
||||||
|
|
||||||
|
_batch_timings[result.id] = time.monotonic()
|
||||||
|
logger.info(
|
||||||
|
"Celery batch submitted job_id=%s chunks=%d images=%d",
|
||||||
|
result.id,
|
||||||
|
len(chunks),
|
||||||
|
len(images),
|
||||||
|
)
|
||||||
|
return result.id
|
||||||
|
|
||||||
|
|
||||||
|
async def get_celery_batch_result(job_id):
|
||||||
|
from celery.result import GroupResult
|
||||||
|
|
||||||
|
now = time.monotonic()
|
||||||
|
cached = _batch_poll_cache.get(job_id)
|
||||||
|
if cached and cached[0] > now:
|
||||||
|
return cached[1]
|
||||||
|
|
||||||
|
result = GroupResult.restore(job_id, app=celery_app)
|
||||||
|
if result is None:
|
||||||
|
payload = {
|
||||||
|
"status": "unknown",
|
||||||
|
"detail": "Job ID not found in result backend",
|
||||||
|
}
|
||||||
|
_batch_poll_cache[job_id] = (now + CELERY_BATCH_POLL_CACHE_TTL_MS / 1000, payload)
|
||||||
|
return payload
|
||||||
|
|
||||||
|
completed = result.completed_count()
|
||||||
|
total = len(result.results)
|
||||||
|
|
||||||
|
if not result.ready():
|
||||||
|
payload = {
|
||||||
|
"status": "pending",
|
||||||
|
"completed": completed,
|
||||||
|
"total": total,
|
||||||
|
"progress": round(completed / total, 2) if total > 0 else 0,
|
||||||
|
}
|
||||||
|
_batch_poll_cache[job_id] = (now + CELERY_BATCH_POLL_CACHE_TTL_MS / 1000, payload)
|
||||||
|
return payload
|
||||||
|
|
||||||
|
if result.failed():
|
||||||
|
errors = []
|
||||||
|
for r in result.results:
|
||||||
|
if r.failed():
|
||||||
|
errors.append(str(r.result))
|
||||||
|
payload = {
|
||||||
|
"status": "failed",
|
||||||
|
"errors": errors,
|
||||||
|
}
|
||||||
|
_batch_poll_cache[job_id] = (now + CELERY_BATCH_POLL_CACHE_TTL_MS / 1000, payload)
|
||||||
|
_log_batch_timing(job_id, "failed")
|
||||||
|
return payload
|
||||||
|
|
||||||
|
all_results = []
|
||||||
|
for r in result.results:
|
||||||
|
chunk_results = r.get()
|
||||||
|
all_results.extend(chunk_results)
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
"status": "completed",
|
||||||
|
"image_count": len(all_results),
|
||||||
|
"results": all_results,
|
||||||
|
}
|
||||||
|
_batch_poll_cache[job_id] = (now + CELERY_BATCH_POLL_CACHE_TTL_MS / 1000, payload)
|
||||||
|
_log_batch_timing(job_id, "completed")
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def _log_batch_timing(job_id: str, status: str) -> None:
|
||||||
|
start = _batch_timings.pop(job_id, None)
|
||||||
|
if start is not None:
|
||||||
|
duration_ms = (time.monotonic() - start) * 1000
|
||||||
|
logger.info(
|
||||||
|
"Celery batch %s job_id=%s duration_ms=%.0f",
|
||||||
|
status,
|
||||||
|
job_id,
|
||||||
|
duration_ms,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
logger.info("Celery batch %s job_id=%s (no start time recorded)", status, job_id)
|
||||||
@@ -5,6 +5,7 @@ import asyncio
|
|||||||
import base64
|
import base64
|
||||||
import io
|
import io
|
||||||
import logging
|
import logging
|
||||||
|
import os
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
@@ -52,8 +53,6 @@ SEGMENT_CLASSES_POST = {
|
|||||||
6: "baker's cyst",
|
6: "baker's cyst",
|
||||||
}
|
}
|
||||||
|
|
||||||
_triton_pipeline_lock = asyncio.Lock()
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class CvInferenceOptions:
|
class CvInferenceOptions:
|
||||||
@@ -76,6 +75,12 @@ def _encode_image_to_data_url(image_pil: Image.Image) -> str:
|
|||||||
return f"data:image/png;base64,{encoded}"
|
return f"data:image/png;base64,{encoded}"
|
||||||
|
|
||||||
|
|
||||||
|
def _encode_image_to_bytes(image_pil: Image.Image) -> bytes:
|
||||||
|
buffered = io.BytesIO()
|
||||||
|
image_pil.save(buffered, format="PNG")
|
||||||
|
return buffered.getvalue()
|
||||||
|
|
||||||
|
|
||||||
def _build_inflammation_payload(logits_row: np.ndarray, config: CalibrationConfig) -> dict:
|
def _build_inflammation_payload(logits_row: np.ndarray, config: CalibrationConfig) -> dict:
|
||||||
interpreted = interpret_inflammation_logits(logits_row, config)
|
interpreted = interpret_inflammation_logits(logits_row, config)
|
||||||
return {
|
return {
|
||||||
@@ -147,6 +152,9 @@ def _build_segmentation_result(
|
|||||||
classes_detected = [name for name, mask in masks.items() if np.sum(mask) > 0 and name != "background"]
|
classes_detected = [name for name, mask in masks.items() if np.sum(mask) > 0 and name != "background"]
|
||||||
color_legend = _build_color_legend(classes_detected, angle_type)
|
color_legend = _build_color_legend(classes_detected, angle_type)
|
||||||
|
|
||||||
|
segmented_png_bytes = _encode_image_to_bytes(overlay)
|
||||||
|
segmented_b64 = base64.b64encode(segmented_png_bytes).decode()
|
||||||
|
|
||||||
result: dict[str, Any] = {
|
result: dict[str, Any] = {
|
||||||
"success": True,
|
"success": True,
|
||||||
"angle": angle_payload,
|
"angle": angle_payload,
|
||||||
@@ -161,7 +169,7 @@ def _build_segmentation_result(
|
|||||||
},
|
},
|
||||||
"images": {
|
"images": {
|
||||||
"enhanced": enhanced_data_url,
|
"enhanced": enhanced_data_url,
|
||||||
"segmented": _encode_image_to_data_url(overlay),
|
"segmented": f"data:image/png;base64,{segmented_b64}",
|
||||||
},
|
},
|
||||||
"models_used": {
|
"models_used": {
|
||||||
"angle": angle_model,
|
"angle": angle_model,
|
||||||
@@ -281,19 +289,30 @@ async def _run_batch_uncached(
|
|||||||
if len(frame_ids) != len(images):
|
if len(frame_ids) != len(images):
|
||||||
raise ValueError("frame_ids length must match images length")
|
raise ValueError("frame_ids length must match images length")
|
||||||
|
|
||||||
async with _triton_pipeline_lock:
|
concurrency = int(os.getenv("CV_BATCH_CONCURRENCY", "2"))
|
||||||
results: list[dict[str, Any]] = []
|
semaphore = asyncio.Semaphore(concurrency)
|
||||||
infer_modes: list[str] = []
|
|
||||||
triton_call_count = 0
|
async def process_one(image_pil: Image.Image, fid: str, index: int):
|
||||||
for image_pil, fid in zip(images, frame_ids, strict=True):
|
async with semaphore:
|
||||||
item, mode, calls = await _run_spec_cv_pipeline_single(
|
if index > 0:
|
||||||
|
await asyncio.sleep(min(index * 0.15, 1.0))
|
||||||
|
return await _run_spec_cv_pipeline_single(
|
||||||
image_pil,
|
image_pil,
|
||||||
frame_id=fid,
|
frame_id=fid,
|
||||||
options=options,
|
options=options,
|
||||||
)
|
)
|
||||||
results.append(item)
|
|
||||||
infer_modes.append(mode)
|
outcomes = await asyncio.gather(
|
||||||
triton_call_count += calls
|
*[process_one(img, fid, idx) for idx, (img, fid) in enumerate(zip(images, frame_ids))],
|
||||||
|
)
|
||||||
|
|
||||||
|
results: list[dict[str, Any]] = []
|
||||||
|
infer_modes: list[str] = []
|
||||||
|
triton_call_count = 0
|
||||||
|
for item, mode, calls in outcomes:
|
||||||
|
results.append(item)
|
||||||
|
infer_modes.append(mode)
|
||||||
|
triton_call_count += calls
|
||||||
return CvBatchResult(
|
return CvBatchResult(
|
||||||
results=results,
|
results=results,
|
||||||
triton_infer_calls=triton_call_count,
|
triton_infer_calls=triton_call_count,
|
||||||
|
|||||||
@@ -25,12 +25,11 @@ INPUT_NAME = "input_image"
|
|||||||
OUTPUT_NAME = "logits"
|
OUTPUT_NAME = "logits"
|
||||||
|
|
||||||
TRITON_INFER_TIMEOUT = float(os.getenv("TRITON_INFER_TIMEOUT", "120"))
|
TRITON_INFER_TIMEOUT = float(os.getenv("TRITON_INFER_TIMEOUT", "120"))
|
||||||
TRITON_INFER_RETRIES = int(os.getenv("TRITON_INFER_RETRIES", "5"))
|
TRITON_INFER_RETRIES = int(os.getenv("TRITON_INFER_RETRIES", "2"))
|
||||||
TRITON_RETRY_BASE_S = float(os.getenv("TRITON_RETRY_BASE_S", "4"))
|
TRITON_RETRY_BASE_S = float(os.getenv("TRITON_RETRY_BASE_S", "2.0"))
|
||||||
TRITON_USE_BATCH_INFER = os.getenv("TRITON_USE_BATCH_INFER", "true").lower()
|
TRITON_USE_BATCH_INFER = os.getenv("TRITON_USE_BATCH_INFER", "true").lower()
|
||||||
RETRYABLE_STATUS = {429, 502, 503, 504}
|
RETRYABLE_STATUS = {429, 502, 503, 504}
|
||||||
|
|
||||||
_triton_infer_lock = asyncio.Lock()
|
|
||||||
_adapter: TritonAdapter | None = None
|
_adapter: TritonAdapter | None = None
|
||||||
_adapter_endpoint: str | None = None
|
_adapter_endpoint: str | None = None
|
||||||
|
|
||||||
@@ -43,6 +42,11 @@ def _get_adapter() -> TritonAdapter:
|
|||||||
global _adapter, _adapter_endpoint
|
global _adapter, _adapter_endpoint
|
||||||
endpoint = get_triton_endpoint()
|
endpoint = get_triton_endpoint()
|
||||||
if _adapter is None or _adapter_endpoint != endpoint:
|
if _adapter is None or _adapter_endpoint != endpoint:
|
||||||
|
if _adapter is not None:
|
||||||
|
try:
|
||||||
|
asyncio.get_event_loop().run_until_complete(_adapter.close())
|
||||||
|
except RuntimeError:
|
||||||
|
pass
|
||||||
_adapter = TritonAdapter(endpoint_url=endpoint, timeout=TRITON_INFER_TIMEOUT)
|
_adapter = TritonAdapter(endpoint_url=endpoint, timeout=TRITON_INFER_TIMEOUT)
|
||||||
_adapter_endpoint = endpoint
|
_adapter_endpoint = endpoint
|
||||||
return _adapter
|
return _adapter
|
||||||
@@ -201,7 +205,7 @@ def _infer_angle_logits_chunk(images: list[Image.Image], model_name: str) -> tup
|
|||||||
if not _should_try_batched_infer(len(images)):
|
if not _should_try_batched_infer(len(images)):
|
||||||
return _infer_angle_logits_sequential(images, model_name), "sequential"
|
return _infer_angle_logits_sequential(images, model_name), "sequential"
|
||||||
try:
|
try:
|
||||||
return _infer_angle_logits_batch(images, model_name, max_retries=1), "batched"
|
return _infer_angle_logits_batch(images, model_name), "batched"
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"Batched angle infer×%s failed (%s); falling back to sequential single-image calls",
|
"Batched angle infer×%s failed (%s); falling back to sequential single-image calls",
|
||||||
@@ -249,7 +253,7 @@ def _infer_inflammation_logits_chunk(
|
|||||||
if not _should_try_batched_infer(len(images)):
|
if not _should_try_batched_infer(len(images)):
|
||||||
return _infer_inflammation_logits_sequential(images, model_name), "sequential"
|
return _infer_inflammation_logits_sequential(images, model_name), "sequential"
|
||||||
try:
|
try:
|
||||||
return _infer_inflammation_logits_batch(images, model_name, max_retries=1), "batched"
|
return _infer_inflammation_logits_batch(images, model_name), "batched"
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"Batched inflammation infer×%s failed (%s); falling back to sequential",
|
"Batched inflammation infer×%s failed (%s); falling back to sequential",
|
||||||
@@ -297,7 +301,7 @@ def _infer_segmentation_logits_chunk(
|
|||||||
if not _should_try_batched_infer(len(images)):
|
if not _should_try_batched_infer(len(images)):
|
||||||
return _infer_segmentation_logits_sequential(images, model_name), "sequential"
|
return _infer_segmentation_logits_sequential(images, model_name), "sequential"
|
||||||
try:
|
try:
|
||||||
return _infer_segmentation_logits_batch(images, model_name, max_retries=1), "batched"
|
return _infer_segmentation_logits_batch(images, model_name), "batched"
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"Batched segmentation infer×%s failed (%s); falling back to sequential",
|
"Batched segmentation infer×%s failed (%s); falling back to sequential",
|
||||||
@@ -311,61 +315,57 @@ async def infer_angle_logits(
|
|||||||
images: list[Image.Image],
|
images: list[Image.Image],
|
||||||
model_name: str,
|
model_name: str,
|
||||||
) -> tuple[np.ndarray, Literal["batched", "sequential"], int]:
|
) -> tuple[np.ndarray, Literal["batched", "sequential"], int]:
|
||||||
"""Run angle classification under the global Triton lock. Returns (logits, mode, call_count)."""
|
all_logits: list[np.ndarray] = []
|
||||||
async with _triton_infer_lock:
|
modes: list[str] = []
|
||||||
all_logits: list[np.ndarray] = []
|
call_count = 0
|
||||||
modes: list[str] = []
|
for chunk in chunk_sequence(images):
|
||||||
call_count = 0
|
logits, mode = await asyncio.to_thread(_infer_angle_logits_chunk, chunk, model_name)
|
||||||
for chunk in chunk_sequence(images):
|
all_logits.append(logits)
|
||||||
logits, mode = await asyncio.to_thread(_infer_angle_logits_chunk, chunk, model_name)
|
modes.append(mode)
|
||||||
all_logits.append(logits)
|
call_count += 1 if mode == "batched" else len(chunk)
|
||||||
modes.append(mode)
|
combined = np.concatenate(all_logits, axis=0) if len(all_logits) > 1 else all_logits[0]
|
||||||
call_count += 1 if mode == "batched" else len(chunk)
|
infer_mode: Literal["batched", "sequential"] = (
|
||||||
combined = np.concatenate(all_logits, axis=0) if len(all_logits) > 1 else all_logits[0]
|
"sequential" if any(m == "sequential" for m in modes) else "batched"
|
||||||
infer_mode: Literal["batched", "sequential"] = (
|
)
|
||||||
"sequential" if any(m == "sequential" for m in modes) else "batched"
|
return combined, infer_mode, call_count
|
||||||
)
|
|
||||||
return combined, infer_mode, call_count
|
|
||||||
|
|
||||||
|
|
||||||
async def infer_inflammation_logits(
|
async def infer_inflammation_logits(
|
||||||
images: list[Image.Image],
|
images: list[Image.Image],
|
||||||
model_name: str,
|
model_name: str,
|
||||||
) -> tuple[np.ndarray, Literal["batched", "sequential"], int]:
|
) -> tuple[np.ndarray, Literal["batched", "sequential"], int]:
|
||||||
async with _triton_infer_lock:
|
all_logits: list[np.ndarray] = []
|
||||||
all_logits: list[np.ndarray] = []
|
modes: list[str] = []
|
||||||
modes: list[str] = []
|
call_count = 0
|
||||||
call_count = 0
|
for chunk in chunk_sequence(images):
|
||||||
for chunk in chunk_sequence(images):
|
logits, mode = await asyncio.to_thread(_infer_inflammation_logits_chunk, chunk, model_name)
|
||||||
logits, mode = await asyncio.to_thread(_infer_inflammation_logits_chunk, chunk, model_name)
|
all_logits.append(logits)
|
||||||
all_logits.append(logits)
|
modes.append(mode)
|
||||||
modes.append(mode)
|
call_count += 1 if mode == "batched" else len(chunk)
|
||||||
call_count += 1 if mode == "batched" else len(chunk)
|
combined = np.concatenate(all_logits, axis=0) if len(all_logits) > 1 else all_logits[0]
|
||||||
combined = np.concatenate(all_logits, axis=0) if len(all_logits) > 1 else all_logits[0]
|
infer_mode: Literal["batched", "sequential"] = (
|
||||||
infer_mode: Literal["batched", "sequential"] = (
|
"sequential" if any(m == "sequential" for m in modes) else "batched"
|
||||||
"sequential" if any(m == "sequential" for m in modes) else "batched"
|
)
|
||||||
)
|
return combined, infer_mode, call_count
|
||||||
return combined, infer_mode, call_count
|
|
||||||
|
|
||||||
|
|
||||||
async def infer_segmentation_logits(
|
async def infer_segmentation_logits(
|
||||||
images: list[Image.Image],
|
images: list[Image.Image],
|
||||||
model_name: str,
|
model_name: str,
|
||||||
) -> tuple[np.ndarray, Literal["batched", "sequential"], int]:
|
) -> tuple[np.ndarray, Literal["batched", "sequential"], int]:
|
||||||
async with _triton_infer_lock:
|
all_logits: list[np.ndarray] = []
|
||||||
all_logits: list[np.ndarray] = []
|
modes: list[str] = []
|
||||||
modes: list[str] = []
|
call_count = 0
|
||||||
call_count = 0
|
for chunk in chunk_sequence(images):
|
||||||
for chunk in chunk_sequence(images):
|
logits, mode = await asyncio.to_thread(_infer_segmentation_logits_chunk, chunk, model_name)
|
||||||
logits, mode = await asyncio.to_thread(_infer_segmentation_logits_chunk, chunk, model_name)
|
all_logits.append(logits)
|
||||||
all_logits.append(logits)
|
modes.append(mode)
|
||||||
modes.append(mode)
|
call_count += 1 if mode == "batched" else len(chunk)
|
||||||
call_count += 1 if mode == "batched" else len(chunk)
|
combined = np.concatenate(all_logits, axis=0) if len(all_logits) > 1 else all_logits[0]
|
||||||
combined = np.concatenate(all_logits, axis=0) if len(all_logits) > 1 else all_logits[0]
|
infer_mode: Literal["batched", "sequential"] = (
|
||||||
infer_mode: Literal["batched", "sequential"] = (
|
"sequential" if any(m == "sequential" for m in modes) else "batched"
|
||||||
"sequential" if any(m == "sequential" for m in modes) else "batched"
|
)
|
||||||
)
|
return combined, infer_mode, call_count
|
||||||
return combined, infer_mode, call_count
|
|
||||||
|
|
||||||
|
|
||||||
async def infer_angle_logits_single(image: Image.Image, model_name: str) -> tuple[np.ndarray, str, int]:
|
async def infer_angle_logits_single(image: Image.Image, model_name: str) -> tuple[np.ndarray, str, int]:
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
"""Warm up Modal Triton on CV service startup — reduces cold-start 502s and first-frame latency."""
|
"""Warm up Modal Triton on CV service startup — reduces cold-start 502s and first-frame latency."""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
|
|
||||||
@@ -23,6 +24,20 @@ def _warmup_model_versions() -> dict[str, str]:
|
|||||||
return versions
|
return versions
|
||||||
|
|
||||||
|
|
||||||
|
async def _warmup_one(
|
||||||
|
name: str,
|
||||||
|
coro,
|
||||||
|
timeout: float,
|
||||||
|
) -> None:
|
||||||
|
try:
|
||||||
|
await asyncio.wait_for(coro, timeout=timeout)
|
||||||
|
logger.info("Triton warmup %s complete", name)
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
logger.warning("Triton warmup %s timed out after %.1fs", name, timeout)
|
||||||
|
except Exception as exc:
|
||||||
|
logger.warning("Triton warmup %s failed: %s", name, exc)
|
||||||
|
|
||||||
|
|
||||||
async def warmup_triton_models() -> None:
|
async def warmup_triton_models() -> None:
|
||||||
if os.getenv("CV_SKIP_TRITON_WARMUP", "").lower() in {"1", "true", "yes"}:
|
if os.getenv("CV_SKIP_TRITON_WARMUP", "").lower() in {"1", "true", "yes"}:
|
||||||
logger.info("Triton warmup skipped (CV_SKIP_TRITON_WARMUP)")
|
logger.info("Triton warmup skipped (CV_SKIP_TRITON_WARMUP)")
|
||||||
@@ -36,13 +51,29 @@ async def warmup_triton_models() -> None:
|
|||||||
img224 = Image.new("RGB", (224, 224), color=(128, 128, 128))
|
img224 = Image.new("RGB", (224, 224), color=(128, 128, 128))
|
||||||
img512 = Image.new("RGB", (512, 512), color=(128, 128, 128))
|
img512 = Image.new("RGB", (512, 512), color=(128, 128, 128))
|
||||||
|
|
||||||
|
warmup_timeout = float(os.getenv("TRITON_WARMUP_TIMEOUT", "15"))
|
||||||
|
|
||||||
logger.info(
|
logger.info(
|
||||||
"Warming up Triton (angle=%s, inflammation=%s, segmentation=%s)…",
|
"Warming up Triton (angle=%s, inflammation=%s, segmentation=%s, timeout=%.1fs)…",
|
||||||
angle_model,
|
angle_model,
|
||||||
inflam_model,
|
inflam_model,
|
||||||
seg_model,
|
seg_model,
|
||||||
|
warmup_timeout,
|
||||||
|
)
|
||||||
|
|
||||||
|
await _warmup_one(
|
||||||
|
"angle",
|
||||||
|
triton_runtime.infer_angle_logits_single(img224, angle_model),
|
||||||
|
warmup_timeout,
|
||||||
|
)
|
||||||
|
await _warmup_one(
|
||||||
|
"inflammation",
|
||||||
|
triton_runtime.infer_inflammation_logits_single(img224, inflam_model),
|
||||||
|
warmup_timeout,
|
||||||
|
)
|
||||||
|
await _warmup_one(
|
||||||
|
"segmentation",
|
||||||
|
triton_runtime.infer_segmentation_logits_single(img512, seg_model),
|
||||||
|
warmup_timeout,
|
||||||
)
|
)
|
||||||
await triton_runtime.infer_angle_logits_single(img224, angle_model)
|
|
||||||
await triton_runtime.infer_inflammation_logits_single(img224, inflam_model)
|
|
||||||
await triton_runtime.infer_segmentation_logits_single(img512, seg_model)
|
|
||||||
logger.info("Triton warmup complete")
|
logger.info("Triton warmup complete")
|
||||||
|
|||||||
179
workspace/sprint_1_2/CODEBASE/backend/start_celery_workers.sh
Executable file
179
workspace/sprint_1_2/CODEBASE/backend/start_celery_workers.sh
Executable file
@@ -0,0 +1,179 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
#
|
||||||
|
# Start Redis + Celery worker for CV inference experiments.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./backend/start_celery_workers.sh # start both
|
||||||
|
# ./backend/start_celery_workers.sh stop # stop both
|
||||||
|
# ./backend/start_celery_workers.sh restart # restart both
|
||||||
|
# ./backend/start_celery_workers.sh status # show status
|
||||||
|
#
|
||||||
|
# Logs:
|
||||||
|
# logs/redis.log
|
||||||
|
# logs/celery.log
|
||||||
|
#
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
CODEBASE_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||||
|
LOG_DIR="${CODEBASE_ROOT}/logs"
|
||||||
|
REDIS_PID_FILE="${LOG_DIR}/redis.pid"
|
||||||
|
CELERY_PID_FILE="${LOG_DIR}/celery.pid"
|
||||||
|
|
||||||
|
mkdir -p "${LOG_DIR}"
|
||||||
|
|
||||||
|
cd "${CODEBASE_ROOT}"
|
||||||
|
export PYTHONPATH="${CODEBASE_ROOT}:${PYTHONPATH:-}"
|
||||||
|
|
||||||
|
REDIS_PORT="${REDIS_PORT:-6379}"
|
||||||
|
REDIS_DB="${REDIS_DB:-0}"
|
||||||
|
CELERY_CHUNK_SIZE="${CELERY_CHUNK_SIZE:-4}"
|
||||||
|
export TRITON_ENDPOINT="${TRITON_ENDPOINT:-https://dtj-tran--triton-s3-service-unified-triton-server.modal.run}"
|
||||||
|
|
||||||
|
|
||||||
|
is_redis_running() {
|
||||||
|
if [ -f "${REDIS_PID_FILE}" ]; then
|
||||||
|
local pid
|
||||||
|
pid=$(cat "${REDIS_PID_FILE}")
|
||||||
|
if kill -0 "${pid}" 2>/dev/null; then
|
||||||
|
return 0
|
||||||
|
else
|
||||||
|
rm -f "${REDIS_PID_FILE}"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
start_redis() {
|
||||||
|
if is_redis_running; then
|
||||||
|
echo "Redis already running (PID $(cat ${REDIS_PID_FILE}))"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Starting Redis on port ${REDIS_PORT}..."
|
||||||
|
redis-server \
|
||||||
|
--port "${REDIS_PORT}" \
|
||||||
|
--daemonize yes \
|
||||||
|
--pidfile "${REDIS_PID_FILE}" \
|
||||||
|
--logfile "${LOG_DIR}/redis.log" \
|
||||||
|
--save "" \
|
||||||
|
--appendonly no
|
||||||
|
|
||||||
|
sleep 0.5
|
||||||
|
|
||||||
|
if is_redis_running; then
|
||||||
|
echo "Redis started (PID $(cat ${REDIS_PID_FILE}))"
|
||||||
|
else
|
||||||
|
echo "Redis failed to start. Check ${LOG_DIR}/redis.log"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
start_celery() {
|
||||||
|
if [ -f "${CELERY_PID_FILE}" ] && kill -0 "$(cat "${CELERY_PID_FILE}")" 2>/dev/null; then
|
||||||
|
echo "Celery already running (PID $(cat ${CELERY_PID_FILE}))"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Starting Celery worker..."
|
||||||
|
nohup celery \
|
||||||
|
-A backend.services.celery_app \
|
||||||
|
worker \
|
||||||
|
--loglevel=info \
|
||||||
|
-Q cv-inference \
|
||||||
|
--concurrency=2 \
|
||||||
|
> "${LOG_DIR}/celery.log" 2>&1 &
|
||||||
|
|
||||||
|
echo $! > "${CELERY_PID_FILE}"
|
||||||
|
|
||||||
|
sleep 1
|
||||||
|
|
||||||
|
if kill -0 "$(cat "${CELERY_PID_FILE}")" 2>/dev/null; then
|
||||||
|
echo "Celery started (PID $(cat ${CELERY_PID_FILE}))"
|
||||||
|
echo " Logs: ${LOG_DIR}/celery.log"
|
||||||
|
else
|
||||||
|
echo "Celery failed to start. Check ${LOG_DIR}/celery.log"
|
||||||
|
rm -f "${CELERY_PID_FILE}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
stop_redis() {
|
||||||
|
if is_redis_running; then
|
||||||
|
local pid
|
||||||
|
pid=$(cat "${REDIS_PID_FILE}")
|
||||||
|
echo "Stopping Redis (PID ${pid})..."
|
||||||
|
kill "${pid}" 2>/dev/null || true
|
||||||
|
sleep 0.5
|
||||||
|
rm -f "${REDIS_PID_FILE}"
|
||||||
|
echo "Redis stopped"
|
||||||
|
else
|
||||||
|
echo "Redis not running"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
stop_celery() {
|
||||||
|
if [ -f "${CELERY_PID_FILE}" ] && kill -0 "$(cat "${CELERY_PID_FILE}")" 2>/dev/null; then
|
||||||
|
local pid
|
||||||
|
pid=$(cat "${CELERY_PID_FILE}")
|
||||||
|
echo "Stopping Celery (PID ${pid})..."
|
||||||
|
kill "${pid}" 2>/dev/null || true
|
||||||
|
sleep 1
|
||||||
|
# Force kill if still running
|
||||||
|
if kill -0 "${pid}" 2>/dev/null; then
|
||||||
|
kill -9 "${pid}" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
rm -f "${CELERY_PID_FILE}"
|
||||||
|
echo "Celery stopped"
|
||||||
|
else
|
||||||
|
echo "Celery not running"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
status() {
|
||||||
|
echo "=== Worker Status ==="
|
||||||
|
if is_redis_running; then
|
||||||
|
echo "Redis: running (PID $(cat ${REDIS_PID_FILE}))"
|
||||||
|
else
|
||||||
|
echo "Redis: stopped"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -f "${CELERY_PID_FILE}" ] && kill -0 "$(cat "${CELERY_PID_FILE}")" 2>/dev/null; then
|
||||||
|
echo "Celery: running (PID $(cat ${CELERY_PID_FILE}))"
|
||||||
|
else
|
||||||
|
echo "Celery: stopped"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
case "${1:-start}" in
|
||||||
|
start)
|
||||||
|
start_redis
|
||||||
|
start_celery
|
||||||
|
echo ""
|
||||||
|
echo "Workers ready. Test with:"
|
||||||
|
echo " curl http://localhost:8001/api/test/analyze/batch/celery"
|
||||||
|
;;
|
||||||
|
stop)
|
||||||
|
stop_celery
|
||||||
|
stop_redis
|
||||||
|
;;
|
||||||
|
restart)
|
||||||
|
stop_celery
|
||||||
|
stop_redis
|
||||||
|
start_redis
|
||||||
|
start_celery
|
||||||
|
;;
|
||||||
|
status)
|
||||||
|
status
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Usage: $0 {start|stop|restart|status}"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
34
workspace/sprint_1_2/CODEBASE/frontend/implementation/.gitignore
vendored
Normal file
34
workspace/sprint_1_2/CODEBASE/frontend/implementation/.gitignore
vendored
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
# Dependencies
|
||||||
|
node_modules/
|
||||||
|
|
||||||
|
# Build output
|
||||||
|
dist/
|
||||||
|
build/
|
||||||
|
|
||||||
|
# Vite
|
||||||
|
.vite/
|
||||||
|
|
||||||
|
# IDE
|
||||||
|
.idea/
|
||||||
|
.vscode/
|
||||||
|
*.swp
|
||||||
|
*.swo
|
||||||
|
|
||||||
|
# OS
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
|
|
||||||
|
# Logs
|
||||||
|
*.log
|
||||||
|
npm-debug.log*
|
||||||
|
|
||||||
|
# Environment files (migrated to config/frontend.config.yaml)
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
!.env.example
|
||||||
|
|
||||||
|
# Test coverage
|
||||||
|
coverage/
|
||||||
|
|
||||||
|
# Misc
|
||||||
|
*.local
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
# Frontend application configuration
|
||||||
|
# This is the single source of truth for frontend feature flags and URLs.
|
||||||
|
# Edit this file directly instead of using .env / .env.development.
|
||||||
|
|
||||||
|
VITE_USE_BACKEND_SEGMENTATION: "true"
|
||||||
|
VITE_SEGMENT_API_BASE: ""
|
||||||
|
VITE_USE_CV_CELERY: "false"
|
||||||
|
VITE_API_BASE_URL: ""
|
||||||
|
VITE_CLINICAL_CHAT_USE_LLM: "true"
|
||||||
|
VITE_CLINICAL_CHAT_MOCK_TOOLS: "true"
|
||||||
|
VITE_OLLAMA_CHAT_URL: "/api/ollama-chat/api/chat"
|
||||||
|
VITE_OLLAMA_MODEL: "gemma4:e4b"
|
||||||
|
VITE_MODAL_OLLAMA_TARGET: "https://dtj-tran--ollama-gemma4-e4b-ollamaserver-web.modal.run"
|
||||||
@@ -27,7 +27,7 @@ export default function CalibrationControls({ config, onChange }: CalibrationCon
|
|||||||
return (
|
return (
|
||||||
<div className="cal-ctrl">
|
<div className="cal-ctrl">
|
||||||
<div className="cal-ctrl__header">
|
<div className="cal-ctrl__header">
|
||||||
<span className="cal-ctrl__title">Điều chỉnh nhiệt độ (T)</span>
|
<span className="cal-ctrl__title">Điều chỉnh độ chắc chắn (T)</span>
|
||||||
<span className="cal-ctrl__hint tnum">T = {config.temperature.toFixed(2)}</span>
|
<span className="cal-ctrl__hint tnum">T = {config.temperature.toFixed(2)}</span>
|
||||||
</div>
|
</div>
|
||||||
<CalibrationMetricHelp layout="block" />
|
<CalibrationMetricHelp layout="block" />
|
||||||
|
|||||||
@@ -32,10 +32,17 @@ const MODEL_LOAD_COPY: Record<
|
|||||||
'installing-gemma': {
|
'installing-gemma': {
|
||||||
title: 'Đang cài đặt Gemma 4 E2B về máy…',
|
title: 'Đang cài đặt Gemma 4 E2B về máy…',
|
||||||
subtitle:
|
subtitle:
|
||||||
'Mô hình trò chuyện chính (~2 GB). Nếu bị gián đoạn, lần mở sau sẽ tiếp tục từ chỗ đã tải.',
|
'Mô hình trò chuyện chính (~1.87 GB). Nếu bị gián đoạn, lần mở sau sẽ tiếp tục từ chỗ đã tải.',
|
||||||
ariaLabel: 'Đang cài đặt Gemma 4 E2B',
|
ariaLabel: 'Đang cài đặt Gemma 4 E2B',
|
||||||
composerPlaceholder: 'Đang cài đặt Gemma 4 E2B, vui lòng đợi…',
|
composerPlaceholder: 'Đang cài đặt Gemma 4 E2B, vui lòng đợi…',
|
||||||
},
|
},
|
||||||
|
'resuming-gemma': {
|
||||||
|
title: 'Đang tiếp tục tải Gemma 4 E2B…',
|
||||||
|
subtitle:
|
||||||
|
'Lần tải trước bị gián đoạn — đang tiếp tục từ chỗ đã tải, không tải lại từ đầu.',
|
||||||
|
ariaLabel: 'Đang tiếp tục tải Gemma 4 E2B',
|
||||||
|
composerPlaceholder: 'Đang tiếp tục tải Gemma 4 E2B, vui lòng đợi…',
|
||||||
|
},
|
||||||
// 'installing-qwen': { ... },
|
// 'installing-qwen': { ... },
|
||||||
'loading-gemma': {
|
'loading-gemma': {
|
||||||
title: 'Đang nạp Gemma 4 E2B…',
|
title: 'Đang nạp Gemma 4 E2B…',
|
||||||
@@ -78,6 +85,7 @@ export default function ClinicalChatPanel({
|
|||||||
modelLoadPhase,
|
modelLoadPhase,
|
||||||
modelLoadProgress,
|
modelLoadProgress,
|
||||||
modelInstallTransferLabel,
|
modelInstallTransferLabel,
|
||||||
|
modelLoadStalled,
|
||||||
modelLoadFading,
|
modelLoadFading,
|
||||||
sendMessage,
|
sendMessage,
|
||||||
stopGeneration,
|
stopGeneration,
|
||||||
@@ -150,8 +158,17 @@ export default function ClinicalChatPanel({
|
|||||||
const showLoadBubble = isModelLoading && modelLoadPhase !== null;
|
const showLoadBubble = isModelLoading && modelLoadPhase !== null;
|
||||||
const loadCopy = modelLoadPhase ? MODEL_LOAD_COPY[modelLoadPhase] : null;
|
const loadCopy = modelLoadPhase ? MODEL_LOAD_COPY[modelLoadPhase] : null;
|
||||||
const progressPercent = Math.min(100, Math.max(0, Math.round(modelLoadProgress)));
|
const progressPercent = Math.min(100, Math.max(0, Math.round(modelLoadProgress)));
|
||||||
const installProgressLabel =
|
const isInstallStalled =
|
||||||
modelLoadPhase && isModelInstallPhase(modelLoadPhase) && modelInstallTransferLabel
|
modelLoadStalled && modelLoadPhase !== null && isModelInstallPhase(modelLoadPhase);
|
||||||
|
const loadTitle = isInstallStalled ? 'Tải Gemma bị gián đoạn' : loadCopy?.title;
|
||||||
|
const loadSubtitle = isInstallStalled
|
||||||
|
? `Mất kết nối — hiện không tải được. Đang thử lại tự động${
|
||||||
|
modelInstallTransferLabel ? ` · đã lưu ${modelInstallTransferLabel}` : ''
|
||||||
|
}. Có thể tải lại trang để tiếp tục từ chỗ đã tải.`
|
||||||
|
: loadCopy?.subtitle;
|
||||||
|
const installProgressLabel = isInstallStalled
|
||||||
|
? 'Gián đoạn'
|
||||||
|
: modelLoadPhase && isModelInstallPhase(modelLoadPhase) && modelInstallTransferLabel
|
||||||
? modelInstallTransferLabel
|
? modelInstallTransferLabel
|
||||||
: `${progressPercent}%`;
|
: `${progressPercent}%`;
|
||||||
const activeModeMeta = getInferenceModeMeta(inferenceMode);
|
const activeModeMeta = getInferenceModeMeta(inferenceMode);
|
||||||
@@ -340,20 +357,37 @@ export default function ClinicalChatPanel({
|
|||||||
aria-hidden
|
aria-hidden
|
||||||
/>
|
/>
|
||||||
<div
|
<div
|
||||||
className={`clinical-chat__load-bubble clinical-chat__load-bubble--${modelLoadPhase} ${modelLoadFading ? 'clinical-chat__load-bubble--fade-out' : ''}`}
|
className={`clinical-chat__load-bubble clinical-chat__load-bubble--${modelLoadPhase} ${isInstallStalled ? 'clinical-chat__load-bubble--stalled' : ''} ${modelLoadFading ? 'clinical-chat__load-bubble--fade-out' : ''}`}
|
||||||
role="status"
|
role="status"
|
||||||
aria-live="polite"
|
aria-live="polite"
|
||||||
aria-label={
|
aria-label={
|
||||||
modelLoadPhase && isModelInstallPhase(modelLoadPhase) && modelInstallTransferLabel
|
isInstallStalled
|
||||||
? `${loadCopy.ariaLabel}, ${modelInstallTransferLabel}`
|
? `${loadCopy.ariaLabel} bị gián đoạn, đang thử lại`
|
||||||
: `${loadCopy.ariaLabel}, ${progressPercent} phần trăm`
|
: modelLoadPhase && isModelInstallPhase(modelLoadPhase) && modelInstallTransferLabel
|
||||||
|
? `${loadCopy.ariaLabel}, ${modelInstallTransferLabel}`
|
||||||
|
: `${loadCopy.ariaLabel}, ${progressPercent} phần trăm`
|
||||||
}
|
}
|
||||||
>
|
>
|
||||||
<div
|
<div
|
||||||
className={`clinical-chat__load-icon clinical-chat__load-icon--${modelLoadPhase}`}
|
className={`clinical-chat__load-icon clinical-chat__load-icon--${modelLoadPhase} ${isInstallStalled ? 'clinical-chat__load-icon--stalled' : ''}`}
|
||||||
aria-hidden
|
aria-hidden
|
||||||
>
|
>
|
||||||
{modelLoadPhase && isModelInstallPhase(modelLoadPhase) ? (
|
{isInstallStalled ? (
|
||||||
|
<svg width="20" height="20" viewBox="0 0 24 24" fill="none">
|
||||||
|
<path
|
||||||
|
d="M12 8v5M12 16.5v.5"
|
||||||
|
stroke="currentColor"
|
||||||
|
strokeWidth="1.75"
|
||||||
|
strokeLinecap="round"
|
||||||
|
/>
|
||||||
|
<path
|
||||||
|
d="M10.3 3.9 2.4 18a2 2 0 0 0 1.7 3h15.8a2 2 0 0 0 1.7-3L13.7 3.9a2 2 0 0 0-3.4 0Z"
|
||||||
|
stroke="currentColor"
|
||||||
|
strokeWidth="1.6"
|
||||||
|
strokeLinejoin="round"
|
||||||
|
/>
|
||||||
|
</svg>
|
||||||
|
) : modelLoadPhase && isModelInstallPhase(modelLoadPhase) ? (
|
||||||
<svg width="20" height="20" viewBox="0 0 24 24" fill="none">
|
<svg width="20" height="20" viewBox="0 0 24 24" fill="none">
|
||||||
<path
|
<path
|
||||||
d="M12 3v3M12 18v3M4.22 4.22l2.12 2.12M17.66 17.66l2.12 2.12M3 12h3M18 12h3M4.22 19.78l2.12-2.12M17.66 6.34l2.12-2.12"
|
d="M12 3v3M12 18v3M4.22 4.22l2.12 2.12M17.66 17.66l2.12 2.12M3 12h3M18 12h3M4.22 19.78l2.12-2.12M17.66 6.34l2.12-2.12"
|
||||||
@@ -373,16 +407,16 @@ export default function ClinicalChatPanel({
|
|||||||
</svg>
|
</svg>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
<p className="clinical-chat__load-title">{loadCopy.title}</p>
|
<p className="clinical-chat__load-title">{loadTitle}</p>
|
||||||
<p className="clinical-chat__load-subtitle">{loadCopy.subtitle}</p>
|
<p className="clinical-chat__load-subtitle">{loadSubtitle}</p>
|
||||||
<div className="clinical-chat__load-progress-row">
|
<div className="clinical-chat__load-progress-row">
|
||||||
<div className="clinical-chat__load-progress-track">
|
<div className="clinical-chat__load-progress-track">
|
||||||
<div
|
<div
|
||||||
className={`clinical-chat__load-progress-fill clinical-chat__load-progress-fill--${modelLoadPhase}`}
|
className={`clinical-chat__load-progress-fill clinical-chat__load-progress-fill--${modelLoadPhase} ${isInstallStalled ? 'clinical-chat__load-progress-fill--stalled' : ''}`}
|
||||||
style={{ width: `${progressPercent}%` }}
|
style={{ width: `${progressPercent}%` }}
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
<span className="clinical-chat__load-progress-label tnum">{installProgressLabel}</span>
|
<span className={`clinical-chat__load-progress-label tnum ${isInstallStalled ? 'clinical-chat__load-progress-label--stalled' : ''}`}>{installProgressLabel}</span>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</>
|
</>
|
||||||
@@ -781,12 +815,43 @@ const styles = `
|
|||||||
color: var(--color-secondary);
|
color: var(--color-secondary);
|
||||||
}
|
}
|
||||||
.clinical-chat__load-icon--installing-gemma,
|
.clinical-chat__load-icon--installing-gemma,
|
||||||
|
.clinical-chat__load-icon--resuming-gemma,
|
||||||
.clinical-chat__load-icon--installing-qwen {
|
.clinical-chat__load-icon--installing-qwen {
|
||||||
animation: clinical-chat-spin 2.4s linear infinite;
|
animation: clinical-chat-spin 2.4s linear infinite;
|
||||||
}
|
}
|
||||||
.clinical-chat__load-icon--installing {
|
.clinical-chat__load-icon--installing {
|
||||||
animation: clinical-chat-spin 2.4s linear infinite;
|
animation: clinical-chat-spin 2.4s linear infinite;
|
||||||
}
|
}
|
||||||
|
/* Stalled: stop the spin so a frozen download never looks like active progress. */
|
||||||
|
.clinical-chat__load-bubble--stalled {
|
||||||
|
border-color: rgba(180, 120, 40, 0.4);
|
||||||
|
}
|
||||||
|
.clinical-chat__load-icon--stalled {
|
||||||
|
animation: clinical-chat-stall-pulse 1.6s ease-in-out infinite;
|
||||||
|
background: rgba(180, 120, 40, 0.14);
|
||||||
|
color: #9a6b1f;
|
||||||
|
}
|
||||||
|
@keyframes clinical-chat-stall-pulse {
|
||||||
|
0%, 100% { opacity: 0.55; }
|
||||||
|
50% { opacity: 1; }
|
||||||
|
}
|
||||||
|
.clinical-chat__load-progress-fill--stalled {
|
||||||
|
background: repeating-linear-gradient(
|
||||||
|
45deg,
|
||||||
|
rgba(180, 120, 40, 0.55),
|
||||||
|
rgba(180, 120, 40, 0.55) 6px,
|
||||||
|
rgba(180, 120, 40, 0.3) 6px,
|
||||||
|
rgba(180, 120, 40, 0.3) 12px
|
||||||
|
);
|
||||||
|
animation: clinical-chat-stall-stripes 0.8s linear infinite;
|
||||||
|
}
|
||||||
|
@keyframes clinical-chat-stall-stripes {
|
||||||
|
from { background-position: 0 0; }
|
||||||
|
to { background-position: 17px 0; }
|
||||||
|
}
|
||||||
|
.clinical-chat__load-progress-label--stalled {
|
||||||
|
color: #9a6b1f;
|
||||||
|
}
|
||||||
@keyframes clinical-chat-pulse-icon {
|
@keyframes clinical-chat-pulse-icon {
|
||||||
0%, 100% { opacity: 0.55; transform: scale(0.94); }
|
0%, 100% { opacity: 0.55; transform: scale(0.94); }
|
||||||
50% { opacity: 1; transform: scale(1); }
|
50% { opacity: 1; transform: scale(1); }
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
import { memo, useEffect, useId, useLayoutEffect, useRef, useState } from 'react';
|
import { memo, useEffect, useId, useRef, useState } from 'react';
|
||||||
import StreamingPlainText from '../atoms/StreamingPlainText';
|
import StreamingPlainText from '../atoms/StreamingPlainText';
|
||||||
import { streamTargetKey } from '../../lib/llm/clinicalChatStreamRegistry';
|
import { streamTargetKey } from '../../lib/llm/clinicalChatStreamRegistry';
|
||||||
|
|
||||||
@@ -18,27 +18,15 @@ function ClinicalChatThought({
|
|||||||
thoughtStreaming = false,
|
thoughtStreaming = false,
|
||||||
}: ClinicalChatThoughtProps) {
|
}: ClinicalChatThoughtProps) {
|
||||||
const panelId = useId();
|
const panelId = useId();
|
||||||
const wasThoughtStreamingRef = useRef(thoughtStreaming);
|
|
||||||
const userToggledRef = useRef(false);
|
const userToggledRef = useRef(false);
|
||||||
const [expanded, setExpanded] = useState(true);
|
// Collapsed by default — keeps the clinical answer prominent; click to expand.
|
||||||
|
const [expanded, setExpanded] = useState(false);
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
setExpanded(true);
|
setExpanded(false);
|
||||||
userToggledRef.current = false;
|
userToggledRef.current = false;
|
||||||
wasThoughtStreamingRef.current = false;
|
|
||||||
}, [messageId]);
|
}, [messageId]);
|
||||||
|
|
||||||
useLayoutEffect(() => {
|
|
||||||
if (thoughtStreaming) {
|
|
||||||
if (!userToggledRef.current) {
|
|
||||||
setExpanded(true);
|
|
||||||
}
|
|
||||||
} else if (wasThoughtStreamingRef.current && !thoughtStreaming && !userToggledRef.current) {
|
|
||||||
setExpanded(false);
|
|
||||||
}
|
|
||||||
wasThoughtStreamingRef.current = thoughtStreaming;
|
|
||||||
}, [thoughtStreaming]);
|
|
||||||
|
|
||||||
if (!content.trim() && !thoughtStreaming) {
|
if (!content.trim() && !thoughtStreaming) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
@@ -49,7 +37,8 @@ function ClinicalChatThought({
|
|||||||
};
|
};
|
||||||
|
|
||||||
const label = thoughtStreaming ? 'Đang suy luận' : 'Suy luận';
|
const label = thoughtStreaming ? 'Đang suy luận' : 'Suy luận';
|
||||||
const preview = !thoughtStreaming && !expanded ? content.replace(/\s+/g, ' ').trim().slice(0, 100) : '';
|
const preview =
|
||||||
|
!expanded && !thoughtStreaming ? content.replace(/\s+/g, ' ').trim().slice(0, 100) : '';
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div
|
<div
|
||||||
@@ -77,15 +66,15 @@ function ClinicalChatThought({
|
|||||||
{!expanded && preview ? (
|
{!expanded && preview ? (
|
||||||
<p className="clinical-chat__thought-preview">{preview}…</p>
|
<p className="clinical-chat__thought-preview">{preview}…</p>
|
||||||
) : null}
|
) : null}
|
||||||
{expanded ? (
|
{/* Body stays mounted even when collapsed so the imperative stream target keeps
|
||||||
<div id={panelId} className="clinical-chat__thought-body">
|
receiving tokens; visibility is toggled via the hidden attribute. */}
|
||||||
<StreamingPlainText
|
<div id={panelId} className="clinical-chat__thought-body" hidden={!expanded}>
|
||||||
text={content}
|
<StreamingPlainText
|
||||||
streamTargetKey={streamTargetKey(messageId, 'thought')}
|
text={content}
|
||||||
className="clinical-chat__thought-md chat-md__plain"
|
streamTargetKey={streamTargetKey(messageId, 'thought')}
|
||||||
/>
|
className="clinical-chat__thought-md chat-md__plain"
|
||||||
</div>
|
/>
|
||||||
) : null}
|
</div>
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -48,32 +48,35 @@ export default function RecordingModeSelector({ value, onChange, disabled }: Rec
|
|||||||
padding: 0;
|
padding: 0;
|
||||||
}
|
}
|
||||||
.recording-mode-selector legend {
|
.recording-mode-selector legend {
|
||||||
font-size: 11px;
|
font-size: 13px;
|
||||||
font-weight: 600;
|
font-weight: 700;
|
||||||
color: #94a3b8;
|
color: #e2e8f0;
|
||||||
margin-bottom: 6px;
|
margin-bottom: 8px;
|
||||||
}
|
}
|
||||||
.recording-mode-selector__options {
|
.recording-mode-selector__options {
|
||||||
display: flex;
|
display: flex;
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
gap: 4px;
|
gap: 8px;
|
||||||
}
|
}
|
||||||
.recording-mode-selector__option {
|
.recording-mode-selector__option {
|
||||||
display: flex;
|
display: flex;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
gap: 8px;
|
gap: 10px;
|
||||||
font-size: 12px;
|
font-size: 14px;
|
||||||
color: #e2e8f0;
|
font-weight: 500;
|
||||||
|
color: #f1f5f9;
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
}
|
}
|
||||||
.recording-mode-selector__option input {
|
.recording-mode-selector__option input {
|
||||||
|
width: 16px;
|
||||||
|
height: 16px;
|
||||||
accent-color: #76c8b1;
|
accent-color: #76c8b1;
|
||||||
}
|
}
|
||||||
.recording-mode-selector__hint {
|
.recording-mode-selector__hint {
|
||||||
margin: 6px 0 0;
|
margin: 8px 0 0;
|
||||||
font-size: 11px;
|
font-size: 12.5px;
|
||||||
line-height: 1.45;
|
line-height: 1.5;
|
||||||
color: #64748b;
|
color: #cbd5e1;
|
||||||
}
|
}
|
||||||
.recording-mode-selector:disabled .recording-mode-selector__option {
|
.recording-mode-selector:disabled .recording-mode-selector__option {
|
||||||
opacity: 0.55;
|
opacity: 0.55;
|
||||||
|
|||||||
@@ -71,7 +71,7 @@ export default function SeverityBadge({
|
|||||||
<div className="severity-panel">
|
<div className="severity-panel">
|
||||||
{severityLoading ? (
|
{severityLoading ? (
|
||||||
<div className="severity-panel__block glass">
|
<div className="severity-panel__block glass">
|
||||||
<span className="severity-panel__eyebrow">Viêm màng hoạt dịch — độ nặng (từ phân đoạn)</span>
|
<span className="severity-panel__eyebrow">Viêm màng hoạt dịch — độ nặng (từ ảnh phân đoạn)</span>
|
||||||
<p className="severity-panel__pending">Đang chờ kết quả độ nặng từ phân đoạn…</p>
|
<p className="severity-panel__pending">Đang chờ kết quả độ nặng từ phân đoạn…</p>
|
||||||
</div>
|
</div>
|
||||||
) : grade != null && gradeLabel ? (
|
) : grade != null && gradeLabel ? (
|
||||||
@@ -80,7 +80,7 @@ export default function SeverityBadge({
|
|||||||
{grade}
|
{grade}
|
||||||
</span>
|
</span>
|
||||||
<div>
|
<div>
|
||||||
<span className="severity-badge__label">Viêm màng hoạt dịch — độ nặng (từ phân đoạn)</span>
|
<span className="severity-badge__label">Viêm màng hoạt dịch — độ nặng (từ ảnh phân đoạn)</span>
|
||||||
<strong>{gradeLabel}</strong>
|
<strong>{gradeLabel}</strong>
|
||||||
{severity?.description && (
|
{severity?.description && (
|
||||||
<p className="severity-panel__desc">{severity.description}</p>
|
<p className="severity-panel__desc">{severity.description}</p>
|
||||||
|
|||||||
@@ -34,6 +34,7 @@ interface DiagnosticCanvasProps {
|
|||||||
patientMrn?: string;
|
patientMrn?: string;
|
||||||
patientId?: string;
|
patientId?: string;
|
||||||
scanFrames?: ScanFrame[];
|
scanFrames?: ScanFrame[];
|
||||||
|
useCelery?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
export default function DiagnosticCanvas({
|
export default function DiagnosticCanvas({
|
||||||
@@ -54,6 +55,7 @@ export default function DiagnosticCanvas({
|
|||||||
patientMrn,
|
patientMrn,
|
||||||
patientId,
|
patientId,
|
||||||
scanFrames: scanFramesProp,
|
scanFrames: scanFramesProp,
|
||||||
|
useCelery,
|
||||||
}: DiagnosticCanvasProps) {
|
}: DiagnosticCanvasProps) {
|
||||||
const activeFrames =
|
const activeFrames =
|
||||||
(scanFramesProp && scanFramesProp.length > 0)
|
(scanFramesProp && scanFramesProp.length > 0)
|
||||||
@@ -146,7 +148,7 @@ export default function DiagnosticCanvas({
|
|||||||
const lockFrameNav = isSingleFrameNavLocked;
|
const lockFrameNav = isSingleFrameNavLocked;
|
||||||
|
|
||||||
const { overlaySrc, interpretation, angleClassification, inflammationClassification, synovitisSeverity, isLoading, isSegmentationLoading, error, source, retry } =
|
const { overlaySrc, interpretation, angleClassification, inflammationClassification, synovitisSeverity, isLoading, isSegmentationLoading, error, source, retry } =
|
||||||
useSegmentationOverlay(frame.id, frame.src, framesForCanvas, patientMrn);
|
useSegmentationOverlay(frame.id, frame.src, framesForCanvas, patientMrn, useCelery);
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
applyFrameIndex(0);
|
applyFrameIndex(0);
|
||||||
@@ -360,13 +362,6 @@ export default function DiagnosticCanvas({
|
|||||||
{showMask && (
|
{showMask && (
|
||||||
<SegmentationOverlay overlaySrc={overlaySrc} isLoading={isLoading} />
|
<SegmentationOverlay overlaySrc={overlaySrc} isLoading={isLoading} />
|
||||||
)}
|
)}
|
||||||
{showMask && error && !isLoading && (
|
|
||||||
<MlServiceErrorPanel
|
|
||||||
error={error}
|
|
||||||
frameLabel={frameLabel}
|
|
||||||
onRetry={source === 'backend' ? retry : undefined}
|
|
||||||
/>
|
|
||||||
)}
|
|
||||||
<canvas
|
<canvas
|
||||||
ref={canvasRef}
|
ref={canvasRef}
|
||||||
className="diagnostic-canvas__annotation-canvas"
|
className="diagnostic-canvas__annotation-canvas"
|
||||||
@@ -377,6 +372,13 @@ export default function DiagnosticCanvas({
|
|||||||
{...drawHandlers}
|
{...drawHandlers}
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
|
{showMask && error && !isLoading && (
|
||||||
|
<MlServiceErrorPanel
|
||||||
|
error={error}
|
||||||
|
frameLabel={frameLabel}
|
||||||
|
onRetry={source === 'backend' ? retry : undefined}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
{closedLoopPrompt && closedLoopPromptViewportAnchor && (
|
{closedLoopPrompt && closedLoopPromptViewportAnchor && (
|
||||||
<ClosedLoopPrompt
|
<ClosedLoopPrompt
|
||||||
anchor={closedLoopPromptViewportAnchor}
|
anchor={closedLoopPromptViewportAnchor}
|
||||||
|
|||||||
@@ -249,7 +249,7 @@ export default function ReviewDiagnosticSessionPanel({
|
|||||||
return (
|
return (
|
||||||
<div className="review-session">
|
<div className="review-session">
|
||||||
<header className="review-session__header">
|
<header className="review-session__header">
|
||||||
<h2 className="review-session__title">Xem lại phiên chẩn đoán</h2>
|
<h2 className="review-session__title">XEM LẠI PHIÊN CHUẨN ĐOÁN</h2>
|
||||||
<RecordingModeSelector
|
<RecordingModeSelector
|
||||||
value={lifecycle.recordingMode}
|
value={lifecycle.recordingMode}
|
||||||
onChange={lifecycle.setRecordingMode}
|
onChange={lifecycle.setRecordingMode}
|
||||||
@@ -572,7 +572,7 @@ export default function ReviewDiagnosticSessionPanel({
|
|||||||
padding-bottom: 8px;
|
padding-bottom: 8px;
|
||||||
}
|
}
|
||||||
.review-session__title {
|
.review-session__title {
|
||||||
margin: 0;
|
margin: 0 0 16px;
|
||||||
font-size: 15px;
|
font-size: 15px;
|
||||||
font-weight: 700;
|
font-weight: 700;
|
||||||
letter-spacing: 0.02em;
|
letter-spacing: 0.02em;
|
||||||
|
|||||||
@@ -155,7 +155,7 @@ export default function SideNavBar({
|
|||||||
<>
|
<>
|
||||||
<aside className="side-nav glass-elevated">
|
<aside className="side-nav glass-elevated">
|
||||||
<section className="side-nav__section">
|
<section className="side-nav__section">
|
||||||
<h3>Đề xuất AI</h3>
|
<h3>Đề xuất của AI</h3>
|
||||||
|
|
||||||
{hasCalibratableOutput && (
|
{hasCalibratableOutput && (
|
||||||
<CalibrationControls config={userConfig} onChange={setUserConfig} />
|
<CalibrationControls config={userConfig} onChange={setUserConfig} />
|
||||||
|
|||||||
@@ -153,9 +153,11 @@ export default function WorkspaceShell({ zoneA, zoneB }: WorkspaceShellProps) {
|
|||||||
<style>{`
|
<style>{`
|
||||||
.workspace-shell {
|
.workspace-shell {
|
||||||
display: flex;
|
display: flex;
|
||||||
flex: 1;
|
flex: 0 0 calc(100dvh - var(--topbar-h) - var(--bottombar-h));
|
||||||
|
height: calc(100dvh - var(--topbar-h) - var(--bottombar-h));
|
||||||
min-height: 0;
|
min-height: 0;
|
||||||
padding: var(--space-md);
|
padding: var(--space-md);
|
||||||
|
overflow: hidden;
|
||||||
user-select: ${isDragging ? 'none' : 'auto'};
|
user-select: ${isDragging ? 'none' : 'auto'};
|
||||||
}
|
}
|
||||||
.workspace-shell--dragging {
|
.workspace-shell--dragging {
|
||||||
@@ -164,8 +166,10 @@ export default function WorkspaceShell({ zoneA, zoneB }: WorkspaceShellProps) {
|
|||||||
.workspace-shell__zone-a {
|
.workspace-shell__zone-a {
|
||||||
flex: 0 0 var(--workspace-zone-a-pct);
|
flex: 0 0 var(--workspace-zone-a-pct);
|
||||||
min-width: 0;
|
min-width: 0;
|
||||||
|
min-height: 0;
|
||||||
display: flex;
|
display: flex;
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
|
overflow: hidden;
|
||||||
transition: flex-basis 0.05s linear;
|
transition: flex-basis 0.05s linear;
|
||||||
}
|
}
|
||||||
.workspace-shell--dragging .workspace-shell__zone-a {
|
.workspace-shell--dragging .workspace-shell__zone-a {
|
||||||
@@ -174,8 +178,10 @@ export default function WorkspaceShell({ zoneA, zoneB }: WorkspaceShellProps) {
|
|||||||
.workspace-shell__zone-b {
|
.workspace-shell__zone-b {
|
||||||
flex: 1 1 0;
|
flex: 1 1 0;
|
||||||
min-width: 0;
|
min-width: 0;
|
||||||
|
min-height: 0;
|
||||||
display: flex;
|
display: flex;
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
|
overflow: hidden;
|
||||||
font-size: calc(1rem * var(--workspace-panel-scale, 1));
|
font-size: calc(1rem * var(--workspace-panel-scale, 1));
|
||||||
}
|
}
|
||||||
.workspace-shell__divider {
|
.workspace-shell__divider {
|
||||||
@@ -246,13 +252,17 @@ export default function WorkspaceShell({ zoneA, zoneB }: WorkspaceShellProps) {
|
|||||||
}
|
}
|
||||||
@media (max-width: 1024px) {
|
@media (max-width: 1024px) {
|
||||||
.workspace-shell {
|
.workspace-shell {
|
||||||
|
flex: 1 1 auto;
|
||||||
|
height: auto;
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
gap: var(--space-md);
|
gap: var(--space-md);
|
||||||
|
overflow: visible;
|
||||||
}
|
}
|
||||||
.workspace-shell__zone-a,
|
.workspace-shell__zone-a,
|
||||||
.workspace-shell__zone-b {
|
.workspace-shell__zone-b {
|
||||||
flex: 1 1 auto;
|
flex: 1 1 auto;
|
||||||
font-size: 1rem;
|
font-size: 1rem;
|
||||||
|
overflow: visible;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
`}</style>
|
`}</style>
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ export const CALIBRATION_METRIC_HELP_KEY_POINTS: readonly CalibrationKeyPoint[]
|
|||||||
{
|
{
|
||||||
title: 'Cơ chế bóp méo của AI hiện đại',
|
title: 'Cơ chế bóp méo của AI hiện đại',
|
||||||
body:
|
body:
|
||||||
'Các mạng càng sâu và thông minh thì xu hướng phân tách các điểm thô càng xa (ví dụ 20 điểm và 1 điểm). Khi quy đổi, khoảng cách quá lớn này bị ép thành xác suất cực đoan như 99,9% — hiện tượng “quá tự tin” (over-confidence).',
|
'Các mô hình AI càng phức tạp (kích thuớc lớn & sâu) và thông minh thì xu hướng phân tách các điểm thô càng xa (ví dụ 20 điểm và 1 điểm). Khi quy đổi, khoảng cách quá lớn này bị ép thành xác suất cực đoan như 99,9% — hiện tượng “quá tự tin” (over-confidence).',
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
title: 'Bác sĩ trưởng khoa hạ hỏa (tham số T)',
|
title: 'Bác sĩ trưởng khoa hạ hỏa (tham số T)',
|
||||||
|
|||||||
@@ -22,19 +22,19 @@ export const CALIBRATION_TIERS: CalibrationTier[] = [
|
|||||||
tier: 1,
|
tier: 1,
|
||||||
labelVi: 'Nhạy cao / Sàng lọc',
|
labelVi: 'Nhạy cao / Sàng lọc',
|
||||||
labelEn: 'Aggressive / Screening',
|
labelEn: 'Aggressive / Screening',
|
||||||
triggerVi: 'Bác sĩ nghi ngờ viêm cao từ triệu chứng, chưa thấy trên ảnh.',
|
triggerVi: 'Bác sĩ nghi ngờ mức độ viêm cao từ triệu chứng, chưa thấy trên ảnh.',
|
||||||
ruleVi: 'T = 0.7 (Sharpening)',
|
ruleVi: 'T = 0.7',
|
||||||
recommendedT: 0.7,
|
recommendedT: 0.7,
|
||||||
uiEffectVi:
|
uiEffectVi:
|
||||||
'Đẩy xác suất biên lên — hạ ngưỡng cảnh báo viêm để không bỏ sót ca ranh giới.',
|
'Đẩy xác suất biên lên — hạ ngưỡng cảnh báo viêm để không bỏ sót ca hiếm gặp.',
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
id: 'standard',
|
id: 'standard',
|
||||||
tier: 2,
|
tier: 2,
|
||||||
labelVi: 'Chuẩn / Mặc định',
|
labelVi: 'Chuẩn / Mặc định',
|
||||||
labelEn: 'Standard Baseline',
|
labelEn: 'Standard Baseline',
|
||||||
triggerVi: 'Vận hành mặc định — chưa có prior lâm sàng từ người dùng.',
|
triggerVi: 'Vận hành mặc định — chưa có dự đoán từ trước từ người dùng.',
|
||||||
ruleVi: 'T = 1.4 (Smoothing)',
|
ruleVi: 'T = 1.4',
|
||||||
recommendedT: 1.4,
|
recommendedT: 1.4,
|
||||||
uiEffectVi:
|
uiEffectVi:
|
||||||
'Giảm overconfidence của mạng — phân bố thực tế, cân bằng toán học.',
|
'Giảm overconfidence của mạng — phân bố thực tế, cân bằng toán học.',
|
||||||
@@ -45,7 +45,7 @@ export const CALIBRATION_TIERS: CalibrationTier[] = [
|
|||||||
labelVi: 'Bảo thủ / Hoài nghi',
|
labelVi: 'Bảo thủ / Hoài nghi',
|
||||||
labelEn: 'Conservative / Skeptical',
|
labelEn: 'Conservative / Skeptical',
|
||||||
triggerVi: 'Bác sĩ tin bệnh nhân khỏe; dùng AI chỉ để kiểm tra lại.',
|
triggerVi: 'Bác sĩ tin bệnh nhân khỏe; dùng AI chỉ để kiểm tra lại.',
|
||||||
ruleVi: 'T = 2.2 (Heavy flattening)',
|
ruleVi: 'T = 2.2',
|
||||||
recommendedT: 2.2,
|
recommendedT: 2.2,
|
||||||
uiEffectVi:
|
uiEffectVi:
|
||||||
'Làm phẳng phân bố — chỉ báo dương khi tín hiệu mô hình cực kỳ mạnh.',
|
'Làm phẳng phân bố — chỉ báo dương khi tín hiệu mô hình cực kỳ mạnh.',
|
||||||
|
|||||||
@@ -84,12 +84,17 @@ export type ClinicalChatRuntime = 'loading' | 'llm' | 'mock';
|
|||||||
|
|
||||||
export type ModelLoadPhase =
|
export type ModelLoadPhase =
|
||||||
| 'installing-gemma'
|
| 'installing-gemma'
|
||||||
|
| 'resuming-gemma'
|
||||||
// | 'installing-qwen'
|
// | 'installing-qwen'
|
||||||
| 'loading-gemma';
|
| 'loading-gemma';
|
||||||
// | 'loading-qwen';
|
// | 'loading-qwen';
|
||||||
|
|
||||||
|
/** No download progress for this long ⇒ treat the install as stalled (not "loading"). */
|
||||||
|
export const MODEL_INSTALL_STALL_MS = 15_000;
|
||||||
|
const MODEL_INSTALL_STALL_POLL_MS = 3_000;
|
||||||
|
|
||||||
export function isModelInstallPhase(phase: ModelLoadPhase): boolean {
|
export function isModelInstallPhase(phase: ModelLoadPhase): boolean {
|
||||||
return phase === 'installing-gemma';
|
return phase === 'installing-gemma' || phase === 'resuming-gemma';
|
||||||
// || phase === 'installing-qwen';
|
// || phase === 'installing-qwen';
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -137,6 +142,8 @@ export interface UseClinicalChatResult {
|
|||||||
modelLoadProgress: number;
|
modelLoadProgress: number;
|
||||||
/** Byte transfer label during OPFS install (e.g. "842.3 MB / 1.87 GB"). */
|
/** Byte transfer label during OPFS install (e.g. "842.3 MB / 1.87 GB"). */
|
||||||
modelInstallTransferLabel: string | null;
|
modelInstallTransferLabel: string | null;
|
||||||
|
/** True when an install/resume has received no bytes for a while — not actually progressing. */
|
||||||
|
modelLoadStalled: boolean;
|
||||||
modelLoadFading: boolean;
|
modelLoadFading: boolean;
|
||||||
sendMessage: () => void;
|
sendMessage: () => void;
|
||||||
stopGeneration: () => void;
|
stopGeneration: () => void;
|
||||||
@@ -212,6 +219,7 @@ export function useClinicalChat({
|
|||||||
const [modelLoadPhase, setModelLoadPhase] = useState<ModelLoadPhase | null>(null);
|
const [modelLoadPhase, setModelLoadPhase] = useState<ModelLoadPhase | null>(null);
|
||||||
const [modelLoadProgress, setModelLoadProgress] = useState(0);
|
const [modelLoadProgress, setModelLoadProgress] = useState(0);
|
||||||
const [modelInstallTransferLabel, setModelInstallTransferLabel] = useState<string | null>(null);
|
const [modelInstallTransferLabel, setModelInstallTransferLabel] = useState<string | null>(null);
|
||||||
|
const [modelLoadStalled, setModelLoadStalled] = useState(false);
|
||||||
const [modelLoadFading, setModelLoadFading] = useState(false);
|
const [modelLoadFading, setModelLoadFading] = useState(false);
|
||||||
const [modeSuggestion, setModeSuggestion] = useState<ModeSuggestion | null>(null);
|
const [modeSuggestion, setModeSuggestion] = useState<ModeSuggestion | null>(null);
|
||||||
|
|
||||||
@@ -365,11 +373,49 @@ export function useClinicalChat({
|
|||||||
let cancelled = false;
|
let cancelled = false;
|
||||||
let fadeTimer: number | undefined;
|
let fadeTimer: number | undefined;
|
||||||
let stopLoadTicker: (() => void) | undefined;
|
let stopLoadTicker: (() => void) | undefined;
|
||||||
|
let stallWatchdogId: number | undefined;
|
||||||
|
let lastInstallProgressAt = Date.now();
|
||||||
|
// Highest byte offset seen so far. Retry "heartbeats" re-emit the same offset;
|
||||||
|
// only a real increase counts as progress.
|
||||||
|
let lastInstallBytes = -1;
|
||||||
|
|
||||||
|
const clearStallWatchdog = () => {
|
||||||
|
if (stallWatchdogId !== undefined) {
|
||||||
|
window.clearInterval(stallWatchdogId);
|
||||||
|
stallWatchdogId = undefined;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Flip the overlay from "downloading" to "stalled" when no *new bytes* arrive for a
|
||||||
|
// while, so a frozen bar (or a resume stuck retrying the same offset) never
|
||||||
|
// masquerades as active progress.
|
||||||
|
const startStallWatchdog = () => {
|
||||||
|
clearStallWatchdog();
|
||||||
|
lastInstallProgressAt = Date.now();
|
||||||
|
lastInstallBytes = -1;
|
||||||
|
setModelLoadStalled(false);
|
||||||
|
stallWatchdogId = window.setInterval(() => {
|
||||||
|
if (cancelled) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (Date.now() - lastInstallProgressAt > MODEL_INSTALL_STALL_MS) {
|
||||||
|
setModelLoadStalled(true);
|
||||||
|
setStatusLabel('Tải Gemma bị gián đoạn — đang thử kết nối lại…');
|
||||||
|
}
|
||||||
|
}, MODEL_INSTALL_STALL_POLL_MS);
|
||||||
|
};
|
||||||
|
|
||||||
|
const noteInstallProgress = () => {
|
||||||
|
lastInstallProgressAt = Date.now();
|
||||||
|
setModelLoadStalled(false);
|
||||||
|
};
|
||||||
|
|
||||||
async function finishModelLoad(): Promise<void> {
|
async function finishModelLoad(): Promise<void> {
|
||||||
if (cancelled) {
|
if (cancelled) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
clearStallWatchdog();
|
||||||
|
setModelLoadStalled(false);
|
||||||
setModelLoadProgress(100);
|
setModelLoadProgress(100);
|
||||||
setModelLoadFading(true);
|
setModelLoadFading(true);
|
||||||
await new Promise<void>((resolve) => {
|
await new Promise<void>((resolve) => {
|
||||||
@@ -469,9 +515,25 @@ export function useClinicalChat({
|
|||||||
if (!cancelled) {
|
if (!cancelled) {
|
||||||
setModelLoadProgress(8);
|
setModelLoadProgress(8);
|
||||||
if (!initialGemma.loadable) {
|
if (!initialGemma.loadable) {
|
||||||
setModelLoadPhase('installing-gemma');
|
// A partial checkpoint already on disk means we resume, not start fresh.
|
||||||
|
const isPartialResume = initialGemma.bytes > 0;
|
||||||
|
setModelLoadPhase(isPartialResume ? 'resuming-gemma' : 'installing-gemma');
|
||||||
setIsModelLoading(true);
|
setIsModelLoading(true);
|
||||||
setStatusLabel('Đang cài đặt Gemma 4 E2B về máy…');
|
setStatusLabel(
|
||||||
|
isPartialResume
|
||||||
|
? 'Đang tiếp tục tải Gemma 4 E2B…'
|
||||||
|
: 'Đang cài đặt Gemma 4 E2B về máy…',
|
||||||
|
);
|
||||||
|
if (isPartialResume) {
|
||||||
|
setModelInstallTransferLabel(
|
||||||
|
formatInstallTransferLabel({
|
||||||
|
phase: 'resuming',
|
||||||
|
bytesLoaded: initialGemma.bytes,
|
||||||
|
bytesTotal: initialGemma.manifest?.bytes ?? null,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
startStallWatchdog();
|
||||||
}
|
}
|
||||||
// else if (!initialQwen.loadable) {
|
// else if (!initialQwen.loadable) {
|
||||||
// setModelLoadPhase('installing-qwen');
|
// setModelLoadPhase('installing-qwen');
|
||||||
@@ -486,11 +548,26 @@ export function useClinicalChat({
|
|||||||
if (cancelled) {
|
if (cancelled) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
setModelLoadPhase('installing-gemma');
|
|
||||||
setIsModelLoading(true);
|
setIsModelLoading(true);
|
||||||
setStatusLabel('Đang cài đặt Gemma 4 E2B về máy…');
|
|
||||||
setModelInstallTransferLabel(formatInstallTransferLabel(progress));
|
setModelInstallTransferLabel(formatInstallTransferLabel(progress));
|
||||||
setModelLoadProgress(mapDownloadProgress(progress));
|
setModelLoadProgress(mapDownloadProgress(progress));
|
||||||
|
|
||||||
|
// Retry heartbeats re-emit the same offset — those must NOT reset the
|
||||||
|
// watchdog or clear the stalled state, otherwise a stuck resume keeps
|
||||||
|
// looking like active loading.
|
||||||
|
const advanced = progress.bytesLoaded > lastInstallBytes;
|
||||||
|
if (!advanced) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
lastInstallBytes = progress.bytesLoaded;
|
||||||
|
noteInstallProgress();
|
||||||
|
const resuming = progress.phase === 'resuming';
|
||||||
|
setModelLoadPhase(resuming ? 'resuming-gemma' : 'installing-gemma');
|
||||||
|
setStatusLabel(
|
||||||
|
resuming
|
||||||
|
? 'Đang tiếp tục tải Gemma 4 E2B…'
|
||||||
|
: 'Đang cài đặt Gemma 4 E2B về máy…',
|
||||||
|
);
|
||||||
},
|
},
|
||||||
// onQwenDownloadProgress: (progress: QwenDownloadProgress) => {
|
// onQwenDownloadProgress: (progress: QwenDownloadProgress) => {
|
||||||
// if (cancelled) {
|
// if (cancelled) {
|
||||||
@@ -537,17 +614,20 @@ export function useClinicalChat({
|
|||||||
);
|
);
|
||||||
void preloadGemmaIntoMemory();
|
void preloadGemmaIntoMemory();
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
clearStallWatchdog();
|
||||||
if (!cancelled) {
|
if (!cancelled) {
|
||||||
setIsModelLoading(false);
|
setIsModelLoading(false);
|
||||||
setModelLoadFading(false);
|
setModelLoadFading(false);
|
||||||
setModelLoadPhase(null);
|
setModelLoadPhase(null);
|
||||||
setModelInstallTransferLabel(null);
|
setModelInstallTransferLabel(null);
|
||||||
|
setModelLoadStalled(false);
|
||||||
setRuntime('mock');
|
setRuntime('mock');
|
||||||
const message = error instanceof Error ? error.message : 'không tải được mô hình';
|
const message = error instanceof Error ? error.message : 'không tải được mô hình';
|
||||||
const isNetwork =
|
const isNetwork =
|
||||||
error instanceof TypeError ||
|
error instanceof TypeError ||
|
||||||
(error instanceof DOMException && error.name === 'NetworkError') ||
|
(error instanceof DOMException &&
|
||||||
/network|failed to fetch|interrupted|gián đoạn|network_changed|err_network_changed/i.test(
|
(error.name === 'NetworkError' || error.name === 'TimeoutError')) ||
|
||||||
|
/network|failed to fetch|interrupted|timed out|timeout|gián đoạn|network_changed|err_network_changed/i.test(
|
||||||
message,
|
message,
|
||||||
);
|
);
|
||||||
setStatusLabel(
|
setStatusLabel(
|
||||||
@@ -563,6 +643,7 @@ export function useClinicalChat({
|
|||||||
return () => {
|
return () => {
|
||||||
cancelled = true;
|
cancelled = true;
|
||||||
stopLoadTicker?.();
|
stopLoadTicker?.();
|
||||||
|
clearStallWatchdog();
|
||||||
if (fadeTimer !== undefined) {
|
if (fadeTimer !== undefined) {
|
||||||
window.clearTimeout(fadeTimer);
|
window.clearTimeout(fadeTimer);
|
||||||
}
|
}
|
||||||
@@ -765,22 +846,26 @@ export function useClinicalChat({
|
|||||||
let ollamaThoughtAcc = '';
|
let ollamaThoughtAcc = '';
|
||||||
let ollamaAnswerAcc = '';
|
let ollamaAnswerAcc = '';
|
||||||
|
|
||||||
const assistantId = createChatMessageId();
|
// A single generation may spawn continuation segments; each segment renders as
|
||||||
const assistantMessage: ClinicalChatMessage = {
|
// its own bubble with its own reasoning, so continuation thinking never leaks
|
||||||
id: assistantId,
|
// into a prior answer. currentAssistantId tracks the bubble being streamed.
|
||||||
|
let currentAssistantId = createChatMessageId();
|
||||||
|
let segmentCount = 1;
|
||||||
|
const spawnAssistantBubble = (id: string, pondering: boolean): ClinicalChatMessage => ({
|
||||||
|
id,
|
||||||
role: 'assistant',
|
role: 'assistant',
|
||||||
content: '',
|
content: '',
|
||||||
timestamp: new Date(),
|
timestamp: new Date(),
|
||||||
streaming: true,
|
streaming: true,
|
||||||
tracksThought: thoughtActive,
|
tracksThought: thoughtActive,
|
||||||
pondering: useRemote,
|
pondering,
|
||||||
ponderingVariant: mode === 'agent' ? 'agent' : 'chat',
|
ponderingVariant: mode === 'agent' ? 'agent' : 'chat',
|
||||||
};
|
});
|
||||||
setMessages((prev) => [...prev, assistantMessage]);
|
setMessages((prev) => [...prev, spawnAssistantBubble(currentAssistantId, useRemote)]);
|
||||||
setStatusLabel(generationStatusLabel(mode, activeLevel));
|
setStatusLabel(generationStatusLabel(mode, activeLevel));
|
||||||
|
|
||||||
let plainContentAccumulator = '';
|
let plainContentAccumulator = '';
|
||||||
const thoughtParser =
|
let thoughtParser =
|
||||||
thoughtActive && !useOllamaThoughtStream ? createCotStreamParser() : null;
|
thoughtActive && !useOllamaThoughtStream ? createCotStreamParser() : null;
|
||||||
let thoughtCompleteEmitted = false;
|
let thoughtCompleteEmitted = false;
|
||||||
let ponderingCleared = false;
|
let ponderingCleared = false;
|
||||||
@@ -790,7 +875,7 @@ export function useClinicalChat({
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
ponderingCleared = true;
|
ponderingCleared = true;
|
||||||
updateMessage(assistantId, { pondering: false });
|
updateMessage(currentAssistantId, { pondering: false });
|
||||||
};
|
};
|
||||||
|
|
||||||
const useImperativeStreamPaint = useRemote || thoughtActive;
|
const useImperativeStreamPaint = useRemote || thoughtActive;
|
||||||
@@ -801,8 +886,9 @@ export function useClinicalChat({
|
|||||||
thoughtComplete?: boolean;
|
thoughtComplete?: boolean;
|
||||||
tracksThought?: boolean;
|
tracksThought?: boolean;
|
||||||
}>((patch) => {
|
}>((patch) => {
|
||||||
|
const id = currentAssistantId;
|
||||||
setMessages((prev) =>
|
setMessages((prev) =>
|
||||||
prev.map((msg) => (msg.id === assistantId ? { ...msg, ...patch } : msg)),
|
prev.map((msg) => (msg.id === id ? { ...msg, ...patch } : msg)),
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -814,6 +900,7 @@ export function useClinicalChat({
|
|||||||
tracksThought?: boolean;
|
tracksThought?: boolean;
|
||||||
},
|
},
|
||||||
) => {
|
) => {
|
||||||
|
const id = currentAssistantId;
|
||||||
const hasStreamText =
|
const hasStreamText =
|
||||||
patch.thoughtContent !== undefined || patch.content !== undefined;
|
patch.thoughtContent !== undefined || patch.content !== undefined;
|
||||||
|
|
||||||
@@ -822,13 +909,13 @@ export function useClinicalChat({
|
|||||||
if (patch.thoughtContent !== undefined) {
|
if (patch.thoughtContent !== undefined) {
|
||||||
domHandled =
|
domHandled =
|
||||||
setClinicalStreamText(
|
setClinicalStreamText(
|
||||||
streamTargetKey(assistantId, 'thought'),
|
streamTargetKey(id, 'thought'),
|
||||||
patch.thoughtContent,
|
patch.thoughtContent,
|
||||||
) && domHandled;
|
) && domHandled;
|
||||||
}
|
}
|
||||||
if (patch.content !== undefined) {
|
if (patch.content !== undefined) {
|
||||||
domHandled =
|
domHandled =
|
||||||
setClinicalStreamText(streamTargetKey(assistantId, 'answer'), patch.content) &&
|
setClinicalStreamText(streamTargetKey(id, 'answer'), patch.content) &&
|
||||||
domHandled;
|
domHandled;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -840,7 +927,7 @@ export function useClinicalChat({
|
|||||||
if (needsReact) {
|
if (needsReact) {
|
||||||
flushSync(() => {
|
flushSync(() => {
|
||||||
setMessages((prev) =>
|
setMessages((prev) =>
|
||||||
prev.map((msg) => (msg.id === assistantId ? { ...msg, ...patch } : msg)),
|
prev.map((msg) => (msg.id === id ? { ...msg, ...patch } : msg)),
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -861,6 +948,45 @@ export function useClinicalChat({
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// Freeze the bubble being streamed using its OWN parser/accumulator, so its
|
||||||
|
// reasoning + answer slice stay self-contained.
|
||||||
|
const finalizeCurrentBubble = () => {
|
||||||
|
if (thoughtParser) {
|
||||||
|
const snapshot = thoughtParser.finalize();
|
||||||
|
updateMessage(currentAssistantId, {
|
||||||
|
content: snapshot.content,
|
||||||
|
thoughtContent: snapshot.thoughtContent || undefined,
|
||||||
|
tracksThought: true,
|
||||||
|
thoughtComplete: true,
|
||||||
|
streaming: false,
|
||||||
|
pondering: false,
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
updateMessage(currentAssistantId, {
|
||||||
|
content: plainContentAccumulator,
|
||||||
|
streaming: false,
|
||||||
|
pondering: false,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
clearClinicalStreamTargetsForMessage(currentAssistantId);
|
||||||
|
};
|
||||||
|
|
||||||
|
// Continuation → close the current bubble and open a fresh one for the next segment.
|
||||||
|
const startNextSegmentBubble = () => {
|
||||||
|
finalizeCurrentBubble();
|
||||||
|
if (!useImperativeStreamPaint) {
|
||||||
|
edgeStreamRaf.cancel();
|
||||||
|
}
|
||||||
|
const nextId = createChatMessageId();
|
||||||
|
currentAssistantId = nextId;
|
||||||
|
segmentCount += 1;
|
||||||
|
thoughtParser =
|
||||||
|
thoughtActive && !useOllamaThoughtStream ? createCotStreamParser() : null;
|
||||||
|
plainContentAccumulator = '';
|
||||||
|
thoughtCompleteEmitted = false;
|
||||||
|
setMessages((prev) => [...prev, spawnAssistantBubble(nextId, false)]);
|
||||||
|
};
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const runTurn = () =>
|
const runTurn = () =>
|
||||||
runClinicalChatTurn(
|
runClinicalChatTurn(
|
||||||
@@ -884,6 +1010,13 @@ export function useClinicalChat({
|
|||||||
},
|
},
|
||||||
(event: ClinicalChatStreamEvent) => {
|
(event: ClinicalChatStreamEvent) => {
|
||||||
try {
|
try {
|
||||||
|
if (event.type === 'segment_boundary') {
|
||||||
|
// segment 1 is the bubble we already opened; only continuations spawn a new one.
|
||||||
|
if (event.segment >= 2 && !useOllamaThoughtStream) {
|
||||||
|
startNextSegmentBubble();
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
if (event.type === 'thought_token') {
|
if (event.type === 'thought_token') {
|
||||||
if (!event.partial || !useOllamaThoughtStream) {
|
if (!event.partial || !useOllamaThoughtStream) {
|
||||||
return;
|
return;
|
||||||
@@ -955,15 +1088,15 @@ export function useClinicalChat({
|
|||||||
|
|
||||||
if (abortController.signal.aborted) {
|
if (abortController.signal.aborted) {
|
||||||
disposeStreamThrottle();
|
disposeStreamThrottle();
|
||||||
updateMessage(assistantId, { streaming: false });
|
updateMessage(currentAssistantId, { streaming: false });
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
disposeStreamThrottle();
|
disposeStreamThrottle();
|
||||||
clearClinicalStreamTargetsForMessage(assistantId);
|
clearClinicalStreamTargetsForMessage(currentAssistantId);
|
||||||
|
|
||||||
if (useOllamaThoughtStream) {
|
if (useOllamaThoughtStream) {
|
||||||
updateMessage(assistantId, {
|
updateMessage(currentAssistantId, {
|
||||||
content: ollamaAnswerAcc || result.finalAnswer,
|
content: ollamaAnswerAcc || result.finalAnswer,
|
||||||
thoughtContent: ollamaThoughtAcc || undefined,
|
thoughtContent: ollamaThoughtAcc || undefined,
|
||||||
tracksThought: true,
|
tracksThought: true,
|
||||||
@@ -973,8 +1106,11 @@ export function useClinicalChat({
|
|||||||
});
|
});
|
||||||
} else if (thoughtActive && thoughtParser) {
|
} else if (thoughtActive && thoughtParser) {
|
||||||
const snapshot = thoughtParser.finalize();
|
const snapshot = thoughtParser.finalize();
|
||||||
const finalContent = snapshot.content || result.finalAnswer;
|
// Only the sole segment may borrow the merged finalAnswer; continuation
|
||||||
updateMessage(assistantId, {
|
// bubbles must keep just their own parsed slice to avoid duplication.
|
||||||
|
const finalContent =
|
||||||
|
snapshot.content || (segmentCount === 1 ? result.finalAnswer : '');
|
||||||
|
updateMessage(currentAssistantId, {
|
||||||
content: finalContent,
|
content: finalContent,
|
||||||
thoughtContent: snapshot.thoughtContent || undefined,
|
thoughtContent: snapshot.thoughtContent || undefined,
|
||||||
tracksThought: true,
|
tracksThought: true,
|
||||||
@@ -983,8 +1119,9 @@ export function useClinicalChat({
|
|||||||
pondering: false,
|
pondering: false,
|
||||||
});
|
});
|
||||||
} else {
|
} else {
|
||||||
updateMessage(assistantId, {
|
updateMessage(currentAssistantId, {
|
||||||
content: result.finalAnswer,
|
content:
|
||||||
|
plainContentAccumulator || (segmentCount === 1 ? result.finalAnswer : ''),
|
||||||
streaming: false,
|
streaming: false,
|
||||||
pondering: false,
|
pondering: false,
|
||||||
});
|
});
|
||||||
@@ -999,12 +1136,12 @@ export function useClinicalChat({
|
|||||||
);
|
);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
disposeStreamThrottle();
|
disposeStreamThrottle();
|
||||||
clearClinicalStreamTargetsForMessage(assistantId);
|
clearClinicalStreamTargetsForMessage(currentAssistantId);
|
||||||
if (error instanceof DOMException && error.name === 'AbortError') {
|
if (error instanceof DOMException && error.name === 'AbortError') {
|
||||||
updateMessage(assistantId, { streaming: false, pondering: false });
|
updateMessage(currentAssistantId, { streaming: false, pondering: false });
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
updateMessage(assistantId, {
|
updateMessage(currentAssistantId, {
|
||||||
content:
|
content:
|
||||||
error instanceof Error
|
error instanceof Error
|
||||||
? `Không thể trả lời: ${error.message}`
|
? `Không thể trả lời: ${error.message}`
|
||||||
@@ -1132,6 +1269,7 @@ export function useClinicalChat({
|
|||||||
modelLoadPhase,
|
modelLoadPhase,
|
||||||
modelLoadProgress,
|
modelLoadProgress,
|
||||||
modelInstallTransferLabel,
|
modelInstallTransferLabel,
|
||||||
|
modelLoadStalled,
|
||||||
modelLoadFading,
|
modelLoadFading,
|
||||||
sendMessage,
|
sendMessage,
|
||||||
stopGeneration,
|
stopGeneration,
|
||||||
|
|||||||
@@ -12,7 +12,10 @@ import {
|
|||||||
normalizeBackendSeverity,
|
normalizeBackendSeverity,
|
||||||
type SynovitisSeverityResult,
|
type SynovitisSeverityResult,
|
||||||
} from '../data/synovitisSeverity';
|
} from '../data/synovitisSeverity';
|
||||||
import { getCachedCvAnalyzeResult, getCvAnalyzeResultsForProfile } from '../lib/cvAnalyzeApi';
|
import { getCachedCvAnalyzeResult, getCvAnalyzeResultsForProfile, getCvAnalyzeResultsForProfileCelery, clearCvAnalyzeResultCache, type CvFrameAnalyzeResult } from '../lib/cvAnalyzeApi';
|
||||||
|
import { clearSegmentationResultCache } from '../lib/segmentationApi';
|
||||||
|
import { clearAngleClassificationResultCache } from '../lib/angleClassificationApi';
|
||||||
|
import { clearMlInferenceCacheForPatient } from '../lib/mlInferenceCacheStore';
|
||||||
import type { ProfileMlContext } from '../lib/mlInferenceCacheKeys';
|
import type { ProfileMlContext } from '../lib/mlInferenceCacheKeys';
|
||||||
import type { ScanFrame } from '../data/scanFrames';
|
import type { ScanFrame } from '../data/scanFrames';
|
||||||
import { interpretSegmentationForDisplay } from '../lib/interpretSegmentationResult';
|
import { interpretSegmentationForDisplay } from '../lib/interpretSegmentationResult';
|
||||||
@@ -104,6 +107,7 @@ export function useSegmentationOverlay(
|
|||||||
imageSrc: string,
|
imageSrc: string,
|
||||||
profileFrames?: ScanFrame[],
|
profileFrames?: ScanFrame[],
|
||||||
patientMrn?: string,
|
patientMrn?: string,
|
||||||
|
useCelery?: boolean,
|
||||||
): UseSegmentationOverlayResult {
|
): UseSegmentationOverlayResult {
|
||||||
const [overlaySrc, setOverlaySrc] = useState<string | null>(null);
|
const [overlaySrc, setOverlaySrc] = useState<string | null>(null);
|
||||||
const [interpretation, setInterpretation] = useState<SegmentationDisplayInterpretation | null>(null);
|
const [interpretation, setInterpretation] = useState<SegmentationDisplayInterpretation | null>(null);
|
||||||
@@ -117,7 +121,15 @@ export function useSegmentationOverlay(
|
|||||||
const [source, setSource] = useState<'backend' | null>(null);
|
const [source, setSource] = useState<'backend' | null>(null);
|
||||||
const [retryNonce, setRetryNonce] = useState(0);
|
const [retryNonce, setRetryNonce] = useState(0);
|
||||||
|
|
||||||
const retry = () => setRetryNonce((n) => n + 1);
|
const retry = () => {
|
||||||
|
clearCvAnalyzeResultCache();
|
||||||
|
clearSegmentationResultCache();
|
||||||
|
clearAngleClassificationResultCache();
|
||||||
|
if (patientMrn) {
|
||||||
|
clearMlInferenceCacheForPatient(patientMrn);
|
||||||
|
}
|
||||||
|
setRetryNonce((n) => n + 1);
|
||||||
|
};
|
||||||
|
|
||||||
const latestSegmentationRef = useRef<SegmentationApiResult | undefined>(undefined);
|
const latestSegmentationRef = useRef<SegmentationApiResult | undefined>(undefined);
|
||||||
|
|
||||||
@@ -161,7 +173,14 @@ export function useSegmentationOverlay(
|
|||||||
const mlContext: ProfileMlContext | undefined = patientMrn
|
const mlContext: ProfileMlContext | undefined = patientMrn
|
||||||
? { patientMrn }
|
? { patientMrn }
|
||||||
: undefined;
|
: undefined;
|
||||||
const results = await getCvAnalyzeResultsForProfile(frameRefs, mlContext);
|
|
||||||
|
let results: Map<string, CvFrameAnalyzeResult>;
|
||||||
|
if (useCelery) {
|
||||||
|
results = await getCvAnalyzeResultsForProfileCelery(frameRefs, mlContext);
|
||||||
|
} else {
|
||||||
|
results = await getCvAnalyzeResultsForProfile(frameRefs, mlContext);
|
||||||
|
}
|
||||||
|
|
||||||
if (cancelled) return;
|
if (cancelled) return;
|
||||||
|
|
||||||
const cvResult = results.get(frameId);
|
const cvResult = results.get(frameId);
|
||||||
@@ -235,7 +254,7 @@ export function useSegmentationOverlay(
|
|||||||
return () => {
|
return () => {
|
||||||
cancelled = true;
|
cancelled = true;
|
||||||
};
|
};
|
||||||
}, [frameId, imageSrc, profileFrames, patientMrn, retryNonce]);
|
}, [frameId, imageSrc, profileFrames, patientMrn, retryNonce, useCelery]);
|
||||||
|
|
||||||
return {
|
return {
|
||||||
overlaySrc,
|
overlaySrc,
|
||||||
|
|||||||
@@ -31,6 +31,26 @@ export interface BackendCvAnalyzeBatchResponse {
|
|||||||
detail?: string;
|
detail?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface BackendCvCelerySubmitResponse {
|
||||||
|
success: boolean;
|
||||||
|
job_id: string;
|
||||||
|
image_count: number;
|
||||||
|
mode: string;
|
||||||
|
chunk_size: number;
|
||||||
|
detail?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface BackendCvCeleryStatusResponse {
|
||||||
|
status: 'pending' | 'completed' | 'failed' | 'unknown';
|
||||||
|
completed?: number;
|
||||||
|
total?: number;
|
||||||
|
progress?: number;
|
||||||
|
image_count?: number;
|
||||||
|
results?: BackendSegmentationResponse[];
|
||||||
|
errors?: string[];
|
||||||
|
detail?: string;
|
||||||
|
}
|
||||||
|
|
||||||
export interface CvFrameAnalyzeResult {
|
export interface CvFrameAnalyzeResult {
|
||||||
raw: BackendSegmentationResponse;
|
raw: BackendSegmentationResponse;
|
||||||
segmentation: SegmentationApiResult;
|
segmentation: SegmentationApiResult;
|
||||||
@@ -222,3 +242,166 @@ export async function getCvAnalyzeResultsForProfile(
|
|||||||
}
|
}
|
||||||
return resolved;
|
return resolved;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Submit CV batch for async Celery chunk fan-out.
|
||||||
|
* Returns job_id immediately — poll with pollCvAnalyzeBatchCelery().
|
||||||
|
*/
|
||||||
|
export async function submitCvAnalyzeBatchCelery(
|
||||||
|
frames: ProfileFrameRef[],
|
||||||
|
apiBase = getSegmentApiBase(),
|
||||||
|
): Promise<string> {
|
||||||
|
if (frames.length === 0) {
|
||||||
|
throw new Error('frames must not be empty');
|
||||||
|
}
|
||||||
|
|
||||||
|
const formData = new FormData();
|
||||||
|
const files = await Promise.all(
|
||||||
|
frames.map((frame, index) =>
|
||||||
|
imageUrlToFile(frame.src, `${frame.id || `frame-${index}`}.png`),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
files.forEach((file) => formData.append('images', file));
|
||||||
|
formData.append('frame_ids', JSON.stringify(frames.map((frame) => frame.id)));
|
||||||
|
|
||||||
|
const response = await fetch(`${apiBase}/api/test/analyze/batch/celery`, {
|
||||||
|
method: 'POST',
|
||||||
|
body: formData,
|
||||||
|
});
|
||||||
|
|
||||||
|
const payload = await readMlApiJson<BackendCvCelerySubmitResponse>(response);
|
||||||
|
if (!response.ok) {
|
||||||
|
throw new Error(`Celery batch submit error (${response.status})`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!payload.success || !payload.job_id) {
|
||||||
|
throw new Error('Celery batch submit returned success=false or missing job_id');
|
||||||
|
}
|
||||||
|
|
||||||
|
return payload.job_id;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Poll Celery batch job status.
|
||||||
|
* Returns status object — call again if status === 'pending'.
|
||||||
|
*/
|
||||||
|
export async function pollCvAnalyzeBatchCelery(
|
||||||
|
jobId: string,
|
||||||
|
apiBase = getSegmentApiBase(),
|
||||||
|
): Promise<BackendCvCeleryStatusResponse> {
|
||||||
|
const response = await fetch(`${apiBase}/api/test/analyze/batch/celery/${encodeURIComponent(jobId)}`);
|
||||||
|
|
||||||
|
const payload = await readMlApiJson<BackendCvCeleryStatusResponse>(response);
|
||||||
|
if (!response.ok) {
|
||||||
|
throw new Error(payload.detail ?? `Celery batch poll error (${response.status})`);
|
||||||
|
}
|
||||||
|
|
||||||
|
return payload;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Async CV pipeline via Celery chunk fan-out.
|
||||||
|
* Submits job, polls until completed/failed, maps results into cache.
|
||||||
|
* Returns cached results immediately if all frames are already available.
|
||||||
|
*/
|
||||||
|
export async function getCvAnalyzeResultsForProfileCelery(
|
||||||
|
frames: ProfileFrameRef[],
|
||||||
|
mlContext?: ProfileMlContext,
|
||||||
|
apiBase = getSegmentApiBase(),
|
||||||
|
signal?: AbortSignal,
|
||||||
|
): Promise<Map<string, CvFrameAnalyzeResult>> {
|
||||||
|
if (frames.length === 0) {
|
||||||
|
return new Map();
|
||||||
|
}
|
||||||
|
|
||||||
|
const cacheKey = buildBatchCacheKey(frames.map((f) => f.id));
|
||||||
|
|
||||||
|
// Return fully cached results without hitting the backend.
|
||||||
|
const cached = new Map<string, CvFrameAnalyzeResult>();
|
||||||
|
for (const frame of frames) {
|
||||||
|
const hit = cvAnalyzeResultCache.get(frame.id);
|
||||||
|
if (hit) {
|
||||||
|
cached.set(frame.id, hit);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (cached.size === frames.length) {
|
||||||
|
return cached;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Coalesce in-flight requests for the same batch of frames.
|
||||||
|
const existing = inflightBatchByKey.get(cacheKey);
|
||||||
|
if (existing) {
|
||||||
|
return existing;
|
||||||
|
}
|
||||||
|
|
||||||
|
const batchPromise = (async (): Promise<Map<string, CvFrameAnalyzeResult>> => {
|
||||||
|
const jobId = await submitCvAnalyzeBatchCelery(frames, apiBase);
|
||||||
|
const pollIntervalMs = 2000;
|
||||||
|
const submitTime = Date.now();
|
||||||
|
|
||||||
|
while (true) {
|
||||||
|
if (signal?.aborted) {
|
||||||
|
throw new Error('Celery batch poll aborted');
|
||||||
|
}
|
||||||
|
|
||||||
|
const status = await pollCvAnalyzeBatchCelery(jobId, apiBase);
|
||||||
|
|
||||||
|
if (status.status === 'completed') {
|
||||||
|
const byFrameId = new Map<string, CvFrameAnalyzeResult>();
|
||||||
|
if (status.results) {
|
||||||
|
for (const item of status.results) {
|
||||||
|
const frameId = item.frame_id;
|
||||||
|
if (!frameId) continue;
|
||||||
|
const cvResult = mapPayloadToCvResult(item);
|
||||||
|
byFrameId.set(frameId, cvResult);
|
||||||
|
cvAnalyzeResultCache.set(frameId, cvResult);
|
||||||
|
segmentationResultCache.set(frameId, cvResult.segmentation);
|
||||||
|
if (cvResult.angle) {
|
||||||
|
angleResultCache.set(frameId, cvResult.angle);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (mlContext?.patientMrn && status.results) {
|
||||||
|
await persistCvBatch(
|
||||||
|
frames,
|
||||||
|
new Map([...byFrameId.entries()]),
|
||||||
|
mlContext,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const totalMs = Date.now() - submitTime;
|
||||||
|
console.log(
|
||||||
|
`[cvAnalyzeApi] Celery batch completed jobId=${jobId} frames=${frames.length} totalMs=${totalMs}`,
|
||||||
|
);
|
||||||
|
return byFrameId;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (status.status === 'failed') {
|
||||||
|
const totalMs = Date.now() - submitTime;
|
||||||
|
console.error(
|
||||||
|
`[cvAnalyzeApi] Celery batch failed jobId=${jobId} frames=${frames.length} totalMs=${totalMs}`,
|
||||||
|
);
|
||||||
|
throw new Error(status.errors?.join('; ') ?? 'Celery batch job failed');
|
||||||
|
}
|
||||||
|
|
||||||
|
if (status.status === 'unknown') {
|
||||||
|
const totalMs = Date.now() - submitTime;
|
||||||
|
console.error(
|
||||||
|
`[cvAnalyzeApi] Celery batch unknown jobId=${jobId} frames=${frames.length} totalMs=${totalMs}`,
|
||||||
|
);
|
||||||
|
throw new Error(status.detail ?? `Celery job ${jobId} not found`);
|
||||||
|
}
|
||||||
|
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs));
|
||||||
|
}
|
||||||
|
})();
|
||||||
|
|
||||||
|
inflightBatchByKey.set(cacheKey, batchPromise);
|
||||||
|
try {
|
||||||
|
return await batchPromise;
|
||||||
|
} finally {
|
||||||
|
inflightBatchByKey.delete(cacheKey);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -24,12 +24,12 @@ function envFlag(name: string, defaultValue: boolean): boolean {
|
|||||||
|
|
||||||
/** Try local Gemma worker when OPFS model is available. Falls back to mock replies otherwise. */
|
/** Try local Gemma worker when OPFS model is available. Falls back to mock replies otherwise. */
|
||||||
export function useLocalLlmWhenAvailable(): boolean {
|
export function useLocalLlmWhenAvailable(): boolean {
|
||||||
return envFlag('VITE_CLINICAL_CHAT_USE_LLM', true);
|
return import.meta.env.VITE_CLINICAL_CHAT_USE_LLM !== 'false';
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Use fixture tool results instead of BFF (default on in dev). */
|
/** Use fixture tool results instead of BFF (default on in dev). */
|
||||||
export function useMockAgentTools(): boolean {
|
export function useMockAgentTools(): boolean {
|
||||||
return envFlag('VITE_CLINICAL_CHAT_MOCK_TOOLS', true);
|
return import.meta.env.VITE_CLINICAL_CHAT_MOCK_TOOLS !== 'false';
|
||||||
}
|
}
|
||||||
|
|
||||||
export function clinicalChatBffBaseUrl(): string {
|
export function clinicalChatBffBaseUrl(): string {
|
||||||
|
|||||||
@@ -122,6 +122,7 @@ export class LlmWorkerClient {
|
|||||||
promptOptions: PromptOptions,
|
promptOptions: PromptOptions,
|
||||||
decode: DecodeParams,
|
decode: DecodeParams,
|
||||||
onToken?: (partial: string) => void,
|
onToken?: (partial: string) => void,
|
||||||
|
onSegmentStart?: (segment: number) => void,
|
||||||
): Promise<{ rawOutput: string; stats: GenerationStats }> {
|
): Promise<{ rawOutput: string; stats: GenerationStats }> {
|
||||||
const id = requestId();
|
const id = requestId();
|
||||||
this.activeGenerateRequestId = id;
|
this.activeGenerateRequestId = id;
|
||||||
@@ -136,6 +137,7 @@ export class LlmWorkerClient {
|
|||||||
},
|
},
|
||||||
reject,
|
reject,
|
||||||
onToken: (partial) => onToken?.(partial),
|
onToken: (partial) => onToken?.(partial),
|
||||||
|
onSegmentStart: (segment) => onSegmentStart?.(segment),
|
||||||
});
|
});
|
||||||
this.worker.postMessage({
|
this.worker.postMessage({
|
||||||
type: 'generate',
|
type: 'generate',
|
||||||
|
|||||||
@@ -16,19 +16,23 @@ export function formatInstallTransferLabel(progress: DownloadProgress): string {
|
|||||||
export function mapDownloadProgress(progress: DownloadProgress): number {
|
export function mapDownloadProgress(progress: DownloadProgress): number {
|
||||||
switch (progress.phase) {
|
switch (progress.phase) {
|
||||||
case 'downloading':
|
case 'downloading':
|
||||||
case 'resuming':
|
case 'resuming': {
|
||||||
if (!progress.bytesTotal || progress.bytesTotal <= 0) {
|
if (!progress.bytesTotal || progress.bytesTotal <= 0) {
|
||||||
return progress.phase === 'resuming' ? 14 : 12;
|
return progress.phase === 'resuming' ? 6 : 4;
|
||||||
}
|
}
|
||||||
return 10 + Math.round((progress.bytesLoaded / progress.bytesTotal) * 58);
|
// Track the real byte fraction so the bar matches the "X GB / Y GB" label,
|
||||||
|
// reserving the last few percent for verify/write before completion.
|
||||||
|
const fraction = Math.min(1, progress.bytesLoaded / progress.bytesTotal);
|
||||||
|
return 4 + Math.round(fraction * 92);
|
||||||
|
}
|
||||||
case 'hashing':
|
case 'hashing':
|
||||||
return 72;
|
return 97;
|
||||||
case 'writing':
|
case 'writing':
|
||||||
return 76;
|
return 98;
|
||||||
case 'done':
|
case 'done':
|
||||||
return 78;
|
return 99;
|
||||||
default:
|
default:
|
||||||
return 10;
|
return 4;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -66,12 +66,26 @@ interface DownloadProbe {
|
|||||||
supportsRange: boolean;
|
supportsRange: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** HEAD/range probes should return fast; time out so a hung socket surfaces as a retriable error. */
|
||||||
|
const PROBE_TIMEOUT_MS = 12_000;
|
||||||
|
|
||||||
|
function probeTimeoutSignal(): AbortSignal | undefined {
|
||||||
|
return typeof AbortSignal !== 'undefined' && typeof AbortSignal.timeout === 'function'
|
||||||
|
? AbortSignal.timeout(PROBE_TIMEOUT_MS)
|
||||||
|
: undefined;
|
||||||
|
}
|
||||||
|
|
||||||
async function probeModelDownloadUrl(url: string): Promise<DownloadProbe> {
|
async function probeModelDownloadUrl(url: string): Promise<DownloadProbe> {
|
||||||
let response = await fetch(url, { method: 'HEAD', redirect: 'follow' });
|
let response = await fetch(url, {
|
||||||
|
method: 'HEAD',
|
||||||
|
redirect: 'follow',
|
||||||
|
signal: probeTimeoutSignal(),
|
||||||
|
});
|
||||||
if (!response.ok) {
|
if (!response.ok) {
|
||||||
response = await fetch(url, {
|
response = await fetch(url, {
|
||||||
headers: { Range: 'bytes=0-0' },
|
headers: { Range: 'bytes=0-0' },
|
||||||
redirect: 'follow',
|
redirect: 'follow',
|
||||||
|
signal: probeTimeoutSignal(),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -164,7 +178,7 @@ function isRetriableDownloadError(error: unknown): boolean {
|
|||||||
if (error instanceof TypeError) {
|
if (error instanceof TypeError) {
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
if (error instanceof DOMException && error.name === 'NetworkError') {
|
if (error instanceof DOMException && (error.name === 'NetworkError' || error.name === 'TimeoutError')) {
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
const message = (error instanceof Error ? error.message : String(error)).toLowerCase();
|
const message = (error instanceof Error ? error.message : String(error)).toLowerCase();
|
||||||
@@ -175,6 +189,8 @@ function isRetriableDownloadError(error: unknown): boolean {
|
|||||||
message.includes('failed to fetch') ||
|
message.includes('failed to fetch') ||
|
||||||
message.includes('load failed') ||
|
message.includes('load failed') ||
|
||||||
message.includes('interrupted') ||
|
message.includes('interrupted') ||
|
||||||
|
message.includes('timed out') ||
|
||||||
|
message.includes('timeout') ||
|
||||||
message.includes('gián đoạn') ||
|
message.includes('gián đoạn') ||
|
||||||
message.includes('http 502') ||
|
message.includes('http 502') ||
|
||||||
message.includes('http 503') ||
|
message.includes('http 503') ||
|
||||||
@@ -328,6 +344,17 @@ export async function checkOpfsModelLoadable(): Promise<OpfsModelLoadableStatus>
|
|||||||
|
|
||||||
const file = await readOpfsModelFile(MODEL_TASK_FILENAME);
|
const file = await readOpfsModelFile(MODEL_TASK_FILENAME);
|
||||||
if (file && file.size > 0 && (await isReadable(file))) {
|
if (file && file.size > 0 && (await isReadable(file))) {
|
||||||
|
// An interrupted download never wrote a manifest (it is written last). Such a
|
||||||
|
// file can still clear the >500 MB header check while being a truncated,
|
||||||
|
// unloadable checkpoint — treat it as resumable, not loadable, so we resume the
|
||||||
|
// download instead of trying to init MediaPipe on a fragmented model.
|
||||||
|
if (file.size < EXPECTED_MODEL_TASK_BYTES) {
|
||||||
|
return invalidStatus(
|
||||||
|
`Partial download in OPFS (${formatBytes(file.size)} of ${formatBytes(EXPECTED_MODEL_TASK_BYTES)}). Will resume on next install.`,
|
||||||
|
manifest,
|
||||||
|
file,
|
||||||
|
);
|
||||||
|
}
|
||||||
const validationError = await validateTaskCandidate(file);
|
const validationError = await validateTaskCandidate(file);
|
||||||
if (validationError) {
|
if (validationError) {
|
||||||
return invalidStatus(validationError, manifest, file);
|
return invalidStatus(validationError, manifest, file);
|
||||||
|
|||||||
@@ -30,7 +30,9 @@ export interface DirectChatTurnResult {
|
|||||||
|
|
||||||
export type ClinicalChatStreamEvent =
|
export type ClinicalChatStreamEvent =
|
||||||
| AgentEvent
|
| AgentEvent
|
||||||
| { type: 'thought_token'; partial: string };
|
| { type: 'thought_token'; partial: string }
|
||||||
|
/** A new local-model generation segment started (continuation) → spawn a new bubble. */
|
||||||
|
| { type: 'segment_boundary'; segment: number };
|
||||||
|
|
||||||
export interface RunClinicalChatTurnInput {
|
export interface RunClinicalChatTurnInput {
|
||||||
inferenceMode: InferenceMode;
|
inferenceMode: InferenceMode;
|
||||||
@@ -56,10 +58,17 @@ function buildChatHistory(
|
|||||||
if (message.streaming || !message.content.trim()) {
|
if (message.streaming || !message.content.trim()) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (message.role === 'user') {
|
if (message.role !== 'user' && message.role !== 'assistant') {
|
||||||
turns.push({ role: 'user', text: message.content });
|
continue;
|
||||||
} else if (message.role === 'assistant') {
|
}
|
||||||
turns.push({ role: 'assistant', text: message.content });
|
const role: GemmaHistoryTurn['role'] = message.role === 'user' ? 'user' : 'assistant';
|
||||||
|
// Continuation segments render as separate assistant bubbles; coalesce consecutive
|
||||||
|
// same-role turns so Gemma sees one well-formed alternating turn.
|
||||||
|
const previous = turns[turns.length - 1];
|
||||||
|
if (previous && previous.role === role) {
|
||||||
|
previous.text = `${previous.text}${message.content}`;
|
||||||
|
} else {
|
||||||
|
turns.push({ role, text: message.content });
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return turns.slice(-maxHistoryTurns * 2);
|
return turns.slice(-maxHistoryTurns * 2);
|
||||||
@@ -126,6 +135,7 @@ export async function runDirectChatTurn(
|
|||||||
historyMessages: ClinicalChatMessage[];
|
historyMessages: ClinicalChatMessage[];
|
||||||
onToken?: (partial: string) => void;
|
onToken?: (partial: string) => void;
|
||||||
onThoughtToken?: (partial: string) => void;
|
onThoughtToken?: (partial: string) => void;
|
||||||
|
onSegmentStart?: (segment: number) => void;
|
||||||
},
|
},
|
||||||
signal?: AbortSignal,
|
signal?: AbortSignal,
|
||||||
): Promise<DirectChatTurnResult> {
|
): Promise<DirectChatTurnResult> {
|
||||||
@@ -206,6 +216,7 @@ export async function runDirectChatTurn(
|
|||||||
promptOptions,
|
promptOptions,
|
||||||
decode,
|
decode,
|
||||||
input.onToken,
|
input.onToken,
|
||||||
|
input.onSegmentStart,
|
||||||
);
|
);
|
||||||
|
|
||||||
if (signal?.aborted) {
|
if (signal?.aborted) {
|
||||||
@@ -252,6 +263,7 @@ export async function runClinicalChatTurn(
|
|||||||
historyMessages: input.historyMessages,
|
historyMessages: input.historyMessages,
|
||||||
onToken: (partial) => onEvent?.({ type: 'final_token', partial }),
|
onToken: (partial) => onEvent?.({ type: 'final_token', partial }),
|
||||||
onThoughtToken: (partial) => onEvent?.({ type: 'thought_token', partial }),
|
onThoughtToken: (partial) => onEvent?.({ type: 'thought_token', partial }),
|
||||||
|
onSegmentStart: (segment) => onEvent?.({ type: 'segment_boundary', segment }),
|
||||||
},
|
},
|
||||||
signal,
|
signal,
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -206,7 +206,7 @@ function ClinicalWorkspaceContent({
|
|||||||
<WorkspaceShell
|
<WorkspaceShell
|
||||||
zoneA={
|
zoneA={
|
||||||
<div ref={canvasStackRef} className="workspace-canvas-stack">
|
<div ref={canvasStackRef} className="workspace-canvas-stack">
|
||||||
<DiagnosticCanvas
|
<DiagnosticCanvas
|
||||||
showMask={showMask}
|
showMask={showMask}
|
||||||
onToggleMask={() => {
|
onToggleMask={() => {
|
||||||
setShowMask((previous) => {
|
setShowMask((previous) => {
|
||||||
@@ -227,6 +227,7 @@ function ClinicalWorkspaceContent({
|
|||||||
onSegmentationLoadingChange={setIsSegmentationLoading}
|
onSegmentationLoadingChange={setIsSegmentationLoading}
|
||||||
patientMrn={patient.mrn}
|
patientMrn={patient.mrn}
|
||||||
patientId={patient.id}
|
patientId={patient.id}
|
||||||
|
useCelery={import.meta.env.VITE_USE_CV_CELERY !== 'false'}
|
||||||
onRegisterSnapshotCapture={(capture) => {
|
onRegisterSnapshotCapture={(capture) => {
|
||||||
captureSnapshotRef.current = capture;
|
captureSnapshotRef.current = capture;
|
||||||
}}
|
}}
|
||||||
|
|||||||
@@ -32,7 +32,9 @@ interface ImportMetaEnv {
|
|||||||
readonly VITE_OLLAMA_CHAT_URL?: string;
|
readonly VITE_OLLAMA_CHAT_URL?: string;
|
||||||
readonly VITE_OLLAMA_MODEL?: string;
|
readonly VITE_OLLAMA_MODEL?: string;
|
||||||
readonly VITE_USE_BACKEND_SEGMENTATION?: string;
|
readonly VITE_USE_BACKEND_SEGMENTATION?: string;
|
||||||
|
readonly VITE_USE_CV_CELERY?: string;
|
||||||
readonly VITE_SEGMENT_API_BASE?: string;
|
readonly VITE_SEGMENT_API_BASE?: string;
|
||||||
|
readonly VITE_MODAL_OLLAMA_TARGET?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
interface ImportMeta {
|
interface ImportMeta {
|
||||||
|
|||||||
@@ -338,16 +338,16 @@ async function generateResponse(
|
|||||||
}
|
}
|
||||||
|
|
||||||
const baseBeforeSegment = combinedOutput;
|
const baseBeforeSegment = combinedOutput;
|
||||||
let emittedLength = combinedOutput.length;
|
|
||||||
const tokenBatcher = createTokenBatcher(requestId, segmentNumber);
|
const tokenBatcher = createTokenBatcher(requestId, segmentNumber);
|
||||||
|
let emittedSegmentLength = 0;
|
||||||
|
|
||||||
|
// Stream each segment's RAW tokens (its own thought channel + answer)
|
||||||
|
// independently. The client renders one bubble per segment and parses its own
|
||||||
|
// channel, so continuation thinking never leaks into a prior answer. The
|
||||||
|
// cross-segment merge below is only for the model's continuation prompt + stats.
|
||||||
const segmentRaw = await streamSegment(prompt, requestId, (_partial, segmentSoFar) => {
|
const segmentRaw = await streamSegment(prompt, requestId, (_partial, segmentSoFar) => {
|
||||||
const merged =
|
const delta = segmentSoFar.slice(emittedSegmentLength);
|
||||||
segments === 0
|
emittedSegmentLength = segmentSoFar.length;
|
||||||
? segmentSoFar
|
|
||||||
: mergeContinuationOutput(baseBeforeSegment, segmentSoFar, chainOfThought);
|
|
||||||
const delta = merged.slice(emittedLength);
|
|
||||||
emittedLength = merged.length;
|
|
||||||
if (delta.length > 0) {
|
if (delta.length > 0) {
|
||||||
tokenBatcher.push(delta);
|
tokenBatcher.push(delta);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -15,7 +15,8 @@
|
|||||||
"noUnusedLocals": true,
|
"noUnusedLocals": true,
|
||||||
"noUnusedParameters": true,
|
"noUnusedParameters": true,
|
||||||
"noFallthroughCasesInSwitch": true,
|
"noFallthroughCasesInSwitch": true,
|
||||||
"noUncheckedSideEffectImports": true
|
"noUncheckedSideEffectImports": true,
|
||||||
|
"types": ["node"]
|
||||||
},
|
},
|
||||||
"include": ["src"]
|
"include": ["src", "vite.config.ts"]
|
||||||
}
|
}
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
@@ -1,8 +1,27 @@
|
|||||||
import { defineConfig } from 'vite';
|
import { defineConfig } from 'vite';
|
||||||
import react from '@vitejs/plugin-react';
|
import react from '@vitejs/plugin-react';
|
||||||
import path from 'node:path';
|
import path from 'node:path';
|
||||||
|
import fs from 'fs';
|
||||||
|
import { load } from 'js-yaml';
|
||||||
|
|
||||||
|
function loadFrontendConfig(): Record<string, string> {
|
||||||
|
try {
|
||||||
|
const raw = fs.readFileSync('config/frontend.config.yaml', 'utf8');
|
||||||
|
return load(raw) as Record<string, string>;
|
||||||
|
} catch {
|
||||||
|
return {};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const frontendConfig = loadFrontendConfig();
|
||||||
|
|
||||||
|
const defineVars: Record<string, string> = {};
|
||||||
|
for (const [key, value] of Object.entries(frontendConfig)) {
|
||||||
|
defineVars[`import.meta.env.${key}`] = JSON.stringify(String(value));
|
||||||
|
}
|
||||||
|
|
||||||
const MODAL_OLLAMA_TARGET =
|
const MODAL_OLLAMA_TARGET =
|
||||||
|
frontendConfig.VITE_MODAL_OLLAMA_TARGET ??
|
||||||
'https://dtj-tran--ollama-gemma4-e4b-ollamaserver-web.modal.run';
|
'https://dtj-tran--ollama-gemma4-e4b-ollamaserver-web.modal.run';
|
||||||
|
|
||||||
export default defineConfig({
|
export default defineConfig({
|
||||||
@@ -52,4 +71,5 @@ export default defineConfig({
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
|
define: defineVars,
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -1,31 +0,0 @@
|
|||||||
# Install Docker image
|
|
||||||
docker network create jenkins
|
|
||||||
|
|
||||||
# install docker-integratable image
|
|
||||||
docker run --name jenkins-docker --rm --detach \
|
|
||||||
-p 8080:8080 -p 50000:50000 \
|
|
||||||
--restart=on-failure \
|
|
||||||
--privileged --network jenkins --network-alias docker \
|
|
||||||
--env DOCKER_TLS_CERTDIR=/certs \
|
|
||||||
--volume jenkins-docker-certs:/certs/client \
|
|
||||||
--volume jenkins-data:/var/jenkins_home \
|
|
||||||
--publish 2376:2376 \
|
|
||||||
docker:dind --storage-driver overlay2 \
|
|
||||||
-v jenkins_home:/var/jenkins_home jenkins/jenkins:lts
|
|
||||||
|
|
||||||
# install the docker images
|
|
||||||
docker run \
|
|
||||||
--name jenkins-blueocean \
|
|
||||||
--restart=on-failure \
|
|
||||||
--detach \
|
|
||||||
--network jenkins \
|
|
||||||
--env DOCKER_HOST=tcp://docker:2376 \
|
|
||||||
--env DOCKER_CERT_PATH=/certs/client \
|
|
||||||
--env DOCKER_TLS_VERIFY=1 \
|
|
||||||
--publish 8080:8080 \
|
|
||||||
--publish 50000:50000 \
|
|
||||||
--volume jenkins-data:/var/jenkins_home \
|
|
||||||
--volume jenkins-docker-certs:/certs/client:ro \
|
|
||||||
jenkins/jenkins:lts
|
|
||||||
|
|
||||||
|
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
|
||||||
|
### 1. The Core Application Stack (Replacing the streaming engine)
|
||||||
|
|
||||||
|
* **Database:** **PostgreSQL** or **SQLite**
|
||||||
|
* *Why:* Extremely lightweight, universally supported, and standard for medical/relational data (like patient exercises, joint ranges of motion, and logs).
|
||||||
|
|
||||||
|
|
||||||
|
* **Backend:** **Node.js (Express)**, **Python (FastAPI)**, or **Go**
|
||||||
|
* *Why:* Fast to build for a PoC, easily containerized via Docker, and highly efficient. FastAPI is excellent if you plan to incorporate any musculoskeletal data analysis or ML later.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### 2. CI/CD & Infrastructure Components
|
||||||
|
|
||||||
|
* **Codebase:** **Gitea**
|
||||||
|
* *Why:* A lightweight, self-hosted Git platform that runs in a simple local Docker container using very little memory.
|
||||||
|
|
||||||
|
|
||||||
|
* **CI/CD Orchestrator & Build Server:** **Woodpecker CI** or **GitHub Actions** (if you're okay using GitHub until migrating to a private cloud).
|
||||||
|
* *Why:* Woodpecker is an open-source, container-first CI engine that runs locally with almost zero overhead.
|
||||||
|
|
||||||
|
|
||||||
|
* **Artifact Repository:** **Gitea Packages** or **Docker Registry**
|
||||||
|
* *Why:* You can store your built application images right inside Gitea or a tiny local Docker registry container.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### 3. Environments & Deployment
|
||||||
|
|
||||||
|
* **Deployment-Manager:** **Docker Compose** or **Portainer**
|
||||||
|
* *Why:* To deploy your musculoskeletal app, you just need Docker Compose to orchestrate your backend and database containers. Portainer gives you a simple web GUI to manage them locally.
|
||||||
|
|
||||||
|
|
||||||
|
* **Staging & Production Env:** Isolated local containers or separate virtual machines.
|
||||||
|
|
||||||
|
### 4. Feedback & Monitoring Loop
|
||||||
|
|
||||||
|
* **User Feedback Collector:** A custom-built form widget inside your app or an open-source tool like **Feedback Fish** / **Air Table**
|
||||||
|
* **Feedback-Resolve:** **Focalboard** or **Leantime** (Self-hosted, lightweight project boards to track bugs and user feature requests).
|
||||||
|
* **Monitoring & Logging:** **Prometheus + Grafana**
|
||||||
|
* *Why:* Perfect for tracking application uptime, API response times, and server health.
|
||||||
|
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
|
||||||
|
```planUML
|
||||||
|
@startuml C4_Elements
|
||||||
|
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml
|
||||||
|
|
||||||
|
|
||||||
|
title Component Diagram for CI/CD Pipeline Architecture
|
||||||
|
|
||||||
|
Person(developer, "Developer(s)", "Writes code and pushes changes.")
|
||||||
|
Person(enduser, "End-User(s)", "Interacts with the production application.")
|
||||||
|
|
||||||
|
Container_Boundary(pipeline_boundary, "CI/CD Pipeline System") {
|
||||||
|
Component(codebase, "Codebase", "Git Repository", "Stores the source code and tracks version history.")
|
||||||
|
Component(orchestrator, "CI/CD Orchestrator", "Workflow Engine", "Triggers actions based on repository events.")
|
||||||
|
Component(build_server, "Build Server", "Compiler/Packager", "Compiles code and builds release packages.")
|
||||||
|
Component(artifact_repo, "Artifact Repository", "Storage", "Stores compiled binaries or container images.")
|
||||||
|
Component(test_pipeline, "Test-Pipeline", "Automation Suite", "Runs unit, integration, and security tests.")
|
||||||
|
Component(deploy_manager, "Deployment-Manager", "CD Engine", "Orchestrates deployment to various environments.")
|
||||||
|
Component(user_feedback, "User Feedback Collector", "In-App/Portal Widget", "Collects explicit feature requests and bug reports from users.")
|
||||||
|
Component(feedback_resolve, "Feedback-Resolve", "Tracking System", "Manages issues, bugs, and deployment logs.")
|
||||||
|
Component(monitoring, "Monitoring & Logging", "Observability", "Monitors application health and metrics.")
|
||||||
|
|
||||||
|
Container_Boundary(deploy_env, "Deployment Environments") {
|
||||||
|
Component(staging_env, "Staging/QA Env", "Environment", "Pre-production environment for testing.")
|
||||||
|
Component(prod_env, "Production Env", "Environment", "Live environment hosting the customer application.")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
' Directional Flow Links
|
||||||
|
Rel(developer, codebase, "1. Pushes code")
|
||||||
|
Rel(codebase, orchestrator, "2. Triggers event webhook")
|
||||||
|
Rel(orchestrator, build_server, "3. Triggers build job")
|
||||||
|
Rel(build_server, artifact_repo, "4. Stores compiled artifact")
|
||||||
|
Rel(orchestrator, test_pipeline, "5. Runs automated tests")
|
||||||
|
Rel(orchestrator, deploy_manager, "6. Signals ready for deployment")
|
||||||
|
|
||||||
|
Rel(deploy_manager, artifact_repo, "7. Pulls latest artifact")
|
||||||
|
Rel(deploy_manager, staging_env, "8. Deploys to")
|
||||||
|
Rel(deploy_manager, prod_env, "9. Promotes approved changes to")
|
||||||
|
|
||||||
|
Rel(enduser, prod_env, "10. Interacts with application")
|
||||||
|
Rel(enduser, user_feedback, "11. Submits requests & feedback")
|
||||||
|
Rel(user_feedback, feedback_resolve, "12. Forwards user tickets")
|
||||||
|
Rel(monitoring, prod_env, "13. Reads metrics and errors")
|
||||||
|
Rel(monitoring, feedback_resolve, "14. Feeds performance/error logs")
|
||||||
|
Rel(feedback_resolve, developer, "15. Alerts for loop closure")
|
||||||
|
|
||||||
|
@enduml
|
||||||
|
```
|
||||||
@@ -205,6 +205,9 @@ async def forward_list_models():
|
|||||||
min_containers=1, # for keeping warm and prevention,
|
min_containers=1, # for keeping warm and prevention,
|
||||||
buffer_containers=2, # Number of additional idle containers to maintain under active load.
|
buffer_containers=2, # Number of additional idle containers to maintain under active load.
|
||||||
scaledown_window=30, # Max time (in seconds) a container can remain idle while scaling down.
|
scaledown_window=30, # Max time (in seconds) a container can remain idle while scaling down.
|
||||||
|
volumes= {
|
||||||
|
'/mnt/vkist-ml-model' : modal.CloudBucketMount(bucket_name="vkist-ml-model", secret=modal.Secret.from_name("aws-secrets"))
|
||||||
|
},
|
||||||
secrets=[modal.Secret.from_name("aws-secrets")]
|
secrets=[modal.Secret.from_name("aws-secrets")]
|
||||||
)
|
)
|
||||||
@modal.asgi_app()
|
@modal.asgi_app()
|
||||||
@@ -213,7 +216,11 @@ def unified_triton_server():
|
|||||||
|
|
||||||
# Spawns Triton in the background. It will automatically read
|
# Spawns Triton in the background. It will automatically read
|
||||||
# your "aws-secrets" environment keys to mount s3://vkist-ml-model/
|
# your "aws-secrets" environment keys to mount s3://vkist-ml-model/
|
||||||
cmd = ["tritonserver", "--model-repository=s3://vkist-ml-model/"]
|
# cmd = ["tritonserver", "--model-repository=s3://vkist-ml-model/"] # bad pattern causing latency
|
||||||
|
cmd = ["tritonserver", "--model-repository=/mnt/vkist-ml-model/"]
|
||||||
|
|
||||||
|
# triton issue network connection -> AWS -> adding cold-off latency
|
||||||
|
# idea mounting the model to Modal Volume
|
||||||
subprocess.Popen(cmd)
|
subprocess.Popen(cmd)
|
||||||
|
|
||||||
print("📋 Triton background process delegated. Handing routing control over to FastAPI.")
|
print("📋 Triton background process delegated. Handing routing control over to FastAPI.")
|
||||||
|
|||||||
@@ -1,2 +1,2 @@
|
|||||||
cd ../PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/infra/implementation/triton_run
|
cd ../PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/infra/implementation/triton_run
|
||||||
MODAL_PROFILE=dtj-tran modal deploy PILOT_PROJECT/workspace/sprint_1_2/codebase/infra/implementation/triton_run/modal_triton.py
|
MODAL_PROFILE=dtj-tran modal deploy modal_triton.py
|
||||||
@@ -0,0 +1,138 @@
|
|||||||
|
import json
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
import numpy as np
|
||||||
|
from PIL import Image
|
||||||
|
import requests
|
||||||
|
|
||||||
|
# Server configuration
|
||||||
|
TRITON_URL = "https://dtj-tran--triton-s3-service-unified-triton-server.modal.run"
|
||||||
|
TARGET_MODEL = "workspace/sprint_1_2/CODEBASE/infra/tests/models/efficientnet_b0_ultrasound_2_class.pth" # Target a single sub-model directly
|
||||||
|
IMFLAMMATION_CLASSES = ["Không viêm", "Có viêm"]
|
||||||
|
|
||||||
|
BASE_DIR = Path(__file__).resolve().parent
|
||||||
|
|
||||||
|
|
||||||
|
def load_image(path: Path) -> Image.Image:
|
||||||
|
if not path.exists():
|
||||||
|
raise FileNotFoundError(f"Test image not found at: {path}")
|
||||||
|
return Image.open(path).convert("RGB")
|
||||||
|
|
||||||
|
|
||||||
|
def preprocess_224(img: Image.Image) -> np.ndarray:
|
||||||
|
"""Preprocesses image to NCHW FP32 [1, 3, 224, 224] matching ResNet50 input requirements"""
|
||||||
|
img_resized = img.resize((224, 224), Image.Resampling.BILINEAR)
|
||||||
|
arr = np.asarray(img_resized).astype(np.float32) / 255.0
|
||||||
|
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
|
||||||
|
std = np.array([0.229, 0.224, 0.225], dtype=np.float32)
|
||||||
|
arr = (arr - mean) / std
|
||||||
|
arr = arr.transpose(2, 0, 1) # HWC -> CHW
|
||||||
|
arr = np.expand_dims(arr, axis=0) # Add batch dim -> NCHW
|
||||||
|
return arr
|
||||||
|
|
||||||
|
|
||||||
|
def softmax(x: np.ndarray) -> np.ndarray:
|
||||||
|
e = np.exp(x - np.max(x))
|
||||||
|
return e / np.sum(e)
|
||||||
|
|
||||||
|
def build_binary_request(input_tensor: np.ndarray, model_name: str) -> tuple[bytes, dict]:
|
||||||
|
"""Constructs a strict KServe v2 binary payload for a single input tensor"""
|
||||||
|
# NOTE: Check your individual model's config.pbtxt to verify if the
|
||||||
|
# expected input name is 'input_image', 'input_0', etc.
|
||||||
|
input_name = "input_image"
|
||||||
|
|
||||||
|
inputs = [
|
||||||
|
{
|
||||||
|
"name": input_name,
|
||||||
|
"shape": list(input_tensor.shape),
|
||||||
|
"datatype": "FP32",
|
||||||
|
"parameters": {"binary_data_size": input_tensor.nbytes},
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
# We request the standard 'logits' output tensor from this model
|
||||||
|
outputs = [{"name": "logits"}]
|
||||||
|
|
||||||
|
metadata = {
|
||||||
|
"model_name": model_name,
|
||||||
|
"model_version": "",
|
||||||
|
"inputs": inputs,
|
||||||
|
"outputs": outputs
|
||||||
|
}
|
||||||
|
|
||||||
|
metadata_bytes = json.dumps(metadata).encode("utf-8")
|
||||||
|
body = metadata_bytes + input_tensor.tobytes()
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
"Inference-Header-Content-Length": str(len(metadata_bytes)),
|
||||||
|
"Content-Type": "application/octet-stream"
|
||||||
|
}
|
||||||
|
return body, headers
|
||||||
|
|
||||||
|
def parse_binary_response(resp: requests.Response) -> np.ndarray:
|
||||||
|
"""Parses Triton's binary response stream for a single output tensor"""
|
||||||
|
header_length = int(resp.headers.get("Inference-Header-Content-Length", "0"))
|
||||||
|
response_bytes = resp.content
|
||||||
|
|
||||||
|
metadata = json.loads(response_bytes[:header_length].decode("utf-8"))
|
||||||
|
binary_data = response_bytes[header_length:]
|
||||||
|
|
||||||
|
# Extract the first available output tensor
|
||||||
|
output_desc = metadata["outputs"][0]
|
||||||
|
shape = output_desc["shape"]
|
||||||
|
|
||||||
|
# Unpack raw bytes back into numpy array
|
||||||
|
arr = np.frombuffer(binary_data, dtype=np.float32).reshape(shape)
|
||||||
|
return arr
|
||||||
|
|
||||||
|
def main():
|
||||||
|
# Define a path to a real local image to test with
|
||||||
|
image_path = "test_images/sup-up-long_positive/58e7a7ef-de3e-11ee-97e2-0a580a5f5b60_11.png"
|
||||||
|
|
||||||
|
print(f"Targeting Individual Model: {TARGET_MODEL}")
|
||||||
|
print(f"Loading image from: {image_path}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
# 1. Prepare image
|
||||||
|
img = load_image(image_path)
|
||||||
|
input_data = preprocess_224(img)
|
||||||
|
|
||||||
|
# 2. Build the payload
|
||||||
|
body, headers = build_binary_request(input_data, TARGET_MODEL)
|
||||||
|
|
||||||
|
# 3. Fire request to the targeted model endpoint
|
||||||
|
url = f"{TRITON_URL}/v2/models/{TARGET_MODEL}/infer"
|
||||||
|
print("Sending request to Triton server...")
|
||||||
|
|
||||||
|
t0 = time.time()
|
||||||
|
resp = requests.post(url, data=body, headers=headers, timeout=30)
|
||||||
|
resp.raise_for_status()
|
||||||
|
latency = time.time() - t0
|
||||||
|
|
||||||
|
# 4. Parse output logits
|
||||||
|
logits = parse_binary_response(resp)
|
||||||
|
logits = np.squeeze(logits) # Drop batch dimension
|
||||||
|
|
||||||
|
# 5. Decode probabilities
|
||||||
|
probs = softmax(logits)
|
||||||
|
predicted_idx = int(np.argmax(probs))
|
||||||
|
|
||||||
|
print("\n" + "=" * 40)
|
||||||
|
print("🎉 SUCCESSFUL INFERENCE")
|
||||||
|
print(f"Network Roundtrip Time: {latency:.4f}s")
|
||||||
|
print("-" * 40)
|
||||||
|
print("Class Probabilities:")
|
||||||
|
for cls_name, prob in zip(IMFLAMMATION_CLASSES, probs):
|
||||||
|
print(f" {cls_name:<15}: {prob * 100:.2f}%")
|
||||||
|
print("-" * 40)
|
||||||
|
print(f"Prediction: {IMFLAMMATION_CLASSES[predicted_idx]} ({probs[predicted_idx] * 100:.2f}%)")
|
||||||
|
print("=" * 40)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"\n❌ Inference Failed: {e}")
|
||||||
|
if 'resp' in locals() and hasattr(resp, 'text'):
|
||||||
|
print(f"Server Error Message: {resp.text}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
103
workspace/sprint_1_2/CODEBASE/requirements.txt
Normal file
103
workspace/sprint_1_2/CODEBASE/requirements.txt
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
# Automatically generated by https://github.com/damnever/pigar.
|
||||||
|
|
||||||
|
aiosqlite==0.22.1
|
||||||
|
docling==2.70.0
|
||||||
|
fastapi==0.135.1
|
||||||
|
fastembed==0.8.0
|
||||||
|
gliner==0.2.27
|
||||||
|
grpcio==1.81.1
|
||||||
|
httpx==0.28.1
|
||||||
|
ingestion==0.0.42
|
||||||
|
langchain==1.3.7
|
||||||
|
langchain-text-splitters==1.1.2
|
||||||
|
modal==1.5.0
|
||||||
|
numpy==2.1.3
|
||||||
|
opencv-python==4.13.0.92
|
||||||
|
pgvector==0.4.2
|
||||||
|
pillow==12.3.0
|
||||||
|
psycopg2-binary==2.9.12
|
||||||
|
pycocotools==2.0.11
|
||||||
|
pydantic==2.13.4
|
||||||
|
pymupdf==1.28.0
|
||||||
|
pypdf==6.14.2
|
||||||
|
pytest==9.0.3
|
||||||
|
python-dotenv==1.2.2
|
||||||
|
redis==8.0.1
|
||||||
|
removeOaExtension==0.0.6
|
||||||
|
requests==2.34.2
|
||||||
|
starlette==1.2.1
|
||||||
|
supabase==2.31.0
|
||||||
|
timm==1.0.25
|
||||||
|
torch==2.12.1
|
||||||
|
torchinfo==1.8.0
|
||||||
|
torchvision==0.27.1
|
||||||
|
transformers==4.57.6
|
||||||
|
tritonclient==2.69.0
|
||||||
|
uvicorn==0.41.0
|
||||||
|
|
||||||
|
# WARNING(pigar): some manual fixes might be required as pigar has detected duplicate requirements for the same import name (possibly for different submodules).
|
||||||
|
# WARNING(pigar): the following duplicate requirements are for the import name: backend
|
||||||
|
agentbeats==1.2.6
|
||||||
|
AlgoVision-Quant-Research==0.0.2
|
||||||
|
alles-apin==0.0.1
|
||||||
|
asamba==1.0.8
|
||||||
|
backend==0.2.4.1
|
||||||
|
backendcatraca-tektek==0.0.7
|
||||||
|
BackupBackup==0.0.6
|
||||||
|
cinder-ml==1.8.5
|
||||||
|
cmbagent==0.0.1.post63
|
||||||
|
conting-researcher==0.14.0
|
||||||
|
cwyod-base==0.0.2
|
||||||
|
deepread==0.0.3
|
||||||
|
devshare==1.0.0
|
||||||
|
django-nextcloud-storage==2024.5.20
|
||||||
|
doccano==1.8.5
|
||||||
|
doccano-multi-label==1.8.4.2
|
||||||
|
docforge-ai==2.0.3
|
||||||
|
equation-phase-portrait-tool==0.1.0
|
||||||
|
excellxgene==2.9.6
|
||||||
|
flaskreactapp==1.0.21
|
||||||
|
flet-devtools==0.0.0
|
||||||
|
food-fighters==1.0.1
|
||||||
|
Fred-Frechet==1.14.5
|
||||||
|
gpt-researcher==0.15.1
|
||||||
|
hermes-revision-system==0.1.0
|
||||||
|
hotel-spider==1.1.5
|
||||||
|
latch-excellxgene==1.0.0
|
||||||
|
leap-backend==0.1.0
|
||||||
|
lemon-auto-saver==1.0.1
|
||||||
|
LPP0D==0.1.1
|
||||||
|
mcp-feedback-pipe==3.0.15
|
||||||
|
mmsbm==1.0.7
|
||||||
|
nia-cli==0.1.1
|
||||||
|
nmuwd==0.10.3
|
||||||
|
omnbot==2017.5
|
||||||
|
personal-site-msilvasy==1.0.5
|
||||||
|
pipebio==5.1.0
|
||||||
|
prastut-ai==1.0.0
|
||||||
|
productiongraph==0.2.0
|
||||||
|
pyapptest==1.0.0
|
||||||
|
python-backend==0.0.1
|
||||||
|
pyWikiCMS==0.5.0
|
||||||
|
quantum-finance==0.2.0
|
||||||
|
qutritium==1.5.2
|
||||||
|
rasa-storyteller==0.2.1
|
||||||
|
reapply-workflows==0.1.0
|
||||||
|
recall-core==0.1.1
|
||||||
|
Researcher==0.4.3
|
||||||
|
reusing-intent==0.1.2
|
||||||
|
sparseprop==0.1.14
|
||||||
|
terminal-note==0.10.0
|
||||||
|
traiNER-loop==0.1.0
|
||||||
|
typernexrad-cli==0.1.0
|
||||||
|
viasp==2.3.0.post3
|
||||||
|
vibe-reader==0.2.1
|
||||||
|
work-with-database==0.0.1
|
||||||
|
x-lib==0.0.27
|
||||||
|
# WARNING(pigar): the following duplicate requirements are for the import name: langchain_google_vertexai
|
||||||
|
langchain-google-vertexai==3.2.4
|
||||||
|
# WARNING(pigar): the following duplicate requirements are for the import name: optimum
|
||||||
|
optimum==2.1.0
|
||||||
|
optimum-onnx==0.1.0
|
||||||
|
Celery==5.6.3
|
||||||
|
redis==8.0.1
|
||||||
BIN
workspace/sprint_1_2/PRODUCT_VISUALIZATION/home_screen.png
Normal file
BIN
workspace/sprint_1_2/PRODUCT_VISUALIZATION/home_screen.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 722 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.1 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.3 MiB |
Reference in New Issue
Block a user