Lumina-MSK/session_memory/6_Jul_26.md

title, date, status

title	date	status
Session Memory 6 Jul 26	2026-07-06	active

Summary

Sprint_1_2 clinical chat saw major LLM UX, streaming, and inference work (Jul 5–6). The highest-risk open issue is frontend markdown rendering of generated assistant text — it has caused tab freezes and catastrophic crashes during reasoning streams. Reasoning paths now use plain text only (StreamingPlainText) as a mitigation. Beta functionality (Planning mode + 🔥 high reasoning level) must be completed by end of this week (target: Friday 10 Jul 2026).

---

Change log (what was updated)

Clinical chat UI & model lifecycle

Area	Change	Key files
LLM loading bubble	Install vs load phases with distinct copy, progress bar, disabled composer	ClinicalChatPanel.tsx, useClinicalChat.ts, modelLoadProgress.ts
Install vs load semantics	First OPFS download (~1.9 GB) vs cached checkpoint init into worker/GPU	useClinicalChat.ts (ModelLoadPhase: 'installing' | 'loading')
Sidebar card switch	Both diagnosis + review layers stay mounted (CSS hide/show) so Gemma is not torn down on carousel switch	SidebarLayerCarousel.tsx
OPFS persistence	Completed install survives reload; interrupted download not resumable (manifest only written on success)	opfsModelStore.ts
Worker init progress	init_progress events wired through LlmWorkerClient.init(onProgress)	llmWorkerClient.ts, llm.worker.ts

Inference modes, reasoning levels & backends

Area	Change	Key files
Unified chat modes	ask merged into chat; inference modes: Chat, Planning (beta), Agent	clinicalChatModes.ts, analyzePromptComplexity.ts
Reasoning levels	🧘 chill / 🤔 moderate / 🔥 high (beta); bar visibility persisted	chatReasoningLevel.ts, ClinicalChatPanel.tsx
Edge vs server toggle	🤔 moderate: Máy (Gemma 4 E2B edge) vs Server (Gemma 4 E4B Modal Ollama think:true)	reasoningModelBackend.ts, ollamaLlmClient.ts, inferenceBackend.ts
Ollama dev proxy	VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat, model gemma4:e4b	.env.development, vite.config.ts
Agent mode	Uses Modal Ollama E4B when configured; tool loop unchanged	clinicalChatModes.ts, runClinicalChatTurn.ts
OOM mitigation	Bootstrap at 2048 tokens; releaseInference() before reload; reuse engine when configured.maxTokens >= required	clinicalChatConfig.ts, llm.worker.ts, llmModelBootstrap.ts
Qwen3 experiment	Dual-model (Qwen .litertlm for moderate) attempted then disabled — usesQwenReasoningLevel() returns false; moderate uses Gemma CoT again	qwenOpfsModelStore.ts, reasoningLlmClient.ts, chatReasoningLevel.ts

Streaming, CoT split & markdown

Area	Change	Key files
CoT split	splitGemmaThoughtOutput, isThoughtChannelComplete; thought vs answer channels in message state	prompts.ts, clinicalChat.ts, useClinicalChat.ts
Collapsible reasoning	ClinicalChatThought — expand while streaming, auto-collapse when thought completes	ClinicalChatThought.tsx
Stream throttle	RAF-coalesced updates (~60/s) to reduce main-thread pressure	streamUpdateThrottle.ts
Token-by-token Ollama	Imperative DOM via clinicalChatStreamRegistry + StreamingPlainText (bypasses React 18 batching)	clinicalChatStreamRegistry.ts, StreamingPlainText.tsx, ollamaLlmClient.ts
Markdown renderer	Custom ChatMarkdown (bold, italic, code, lists, headings) — deferred until stream ends via requestIdleCallback + startTransition	ChatMarkdown.tsx, ClinicalChatMessageBubble.tsx
Reasoning = plain text	When tracksThought, thought + answer use StreamingPlainText only — no ChatMarkdown	ClinicalChatThought.tsx, ClinicalChatMessageBubble.tsx

Agent tools & Modal testing

Area	Change	Key files
Tool catalog doc	Walkthrough of Edge-LLM agent tools: exa_search, supabase_query, escalate_medgemma	session + agent_tools_contract.md
3-layer smoke harness	Layer 0 Modal Ollama, Layer 1 BFF routes, Layer 2 browser ToolExecutor	ml/tests/agent_tools/
Python reference tests	Modal /api/chat streaming + thinking cases	PILOT_PROJECT/tmp/test_endpoint.py, test_endpoint_img.py
Gemma4 E4B deploy script	Modal Ollama serverless for Gemma 4 E4B	PILOT_PROJECT/tmp/GemmaE4B_ollama_deploy.py
gemma4_e2b lab	Tool smoke panel (no Gemma) for isolated tool calls	ml/tests/gemma4_e2b/src/lib/toolSmoke.ts

---

Critical issue: markdown rendering → catastrophic crash

Symptom

Rendering LLM-generated markdown in the clinical chat UI (especially during or immediately after reasoning / CoT streams) has caused:

• Main-thread freezes (composer unresponsive while tokens still arrive)
• Tab crashes / OOM under combined WebGPU model memory + large text buffers + markdown parse

Root cause (confirmed in debugging)

1. Per-token full re-parse — early implementation ran ChatMarkdown / renderBlocks() on the entire growing thought string every token → hundreds of parses per second.
2. React re-render storm — setMessages on every token re-rendered the full message list.
3. Post-stream markdown — even with “parse after stream ends”, heavy renderBlocks() on long CoT + answer text still spikes CPU/memory.
4. Dual large strings — rawAccumulator, thoughtContent, and content can all hold 2048-token traces simultaneously at 🤔 moderate.

Mitigations applied (6 Jul)

• Stream updates throttled (createStreamUpdateThrottle)
• Plain text while streaming === true
• React.memo on ClinicalChatMessageBubble
• Reasoning paths (tracksThought) → StreamingPlainText only, markdown disabled
• Ollama path → imperative DOM updates per token
• ChatMarkdown uses idle callback + startTransition for post-stream formatting (chill / non-reasoning answers only)

Current policy

Be wary of re-enabling markdown in reasoning mode until a safe renderer is proven.

ClinicalChatThought.tsx comment: "Reasoning panel — plain text only (markdown disabled while isolating stream crashes)."

Follow-up for later sprints

• Reproduce crash with a minimal ChatMarkdown + long fixture (no LLM) — isolate renderBlocks vs React
• Consider react-markdown with strict plugins OR server-side pre-render for final answer only
• Cap thought trace length in UI (truncate + “show more”)
• Virtualize message list for long sessions
• Re-test 🔥 high mode and Planning mode with any new markdown path before removing plain-text guard
• Audit whether chill-mode ChatMarkdown after stream is safe on 8 GB RAM devices

---

Beta functionality — deadline end of week

Target: complete by Friday 10 Jul 2026 (end of sprint week).

Features still marked beta: true (UI tab/button disabled until ready):

Feature	ID	Location	What “done” means
Planning mode	planning	clinicalChatModes.ts	Checklist output stable; remove beta flag; selectable in mode tabs
High reasoning	high 🔥	chatReasoningLevel.ts	2048-token multi-turn works without crash; remove beta flag

Related work not yet beta-flagged but in scope:

• Agent tool integration (Layer 3 after smoke tests pass)
• Live BFF tools (VITE_CLINICAL_CHAT_MOCK_TOOLS=false + credentials)
• Edge/server reasoning toggle polish (default server E4B when Modal up)

---

Architecture snapshot (clinical LLM, 6 Jul EOD)

User message
    │
    ├─ Mode: chat ─┬─ 🧘 chill  → Gemma E2B edge (no CoT)
    │              ├─ 🤔 moderate → Máy: Gemma E2B + CoT (plain text)
    │              │              Server: Gemma E4B Ollama think:true (plain text)
    │              └─ 🔥 high (BETA) → disabled in UI
    │
    ├─ Mode: planning (BETA) → disabled in UI
    │
    └─ Mode: agent → Gemma E4B Ollama + tools (exa, supabase, medgemma)

Display:
    tracksThought → StreamingPlainText (thought panel + answer)
    else          → StreamingPlainText while streaming → ChatMarkdown when done

---

Key env / endpoints (dev)

VITE_CLINICAL_CHAT_USE_LLM=true
VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat
VITE_OLLAMA_MODEL=gemma4:e4b
VITE_CLINICAL_CHAT_MOCK_TOOLS=true   # flip false for live agent tools

Modal Gemma E4B (reference): PILOT_PROJECT/tmp/test_endpoint.py → dtj-tran--ollama-gemma4-e4b-ollamaserver-web.modal.run

---

Session references

• Prior frontend memory: session_memory/27_jun_26/27_jun_26_frontend.md
• Agent tools smoke README: CODEBASE/ml/tests/agent_tools/README.md
• Today's active dev: npm run dev on frontend implementation