160 lines
8.8 KiB
Markdown
160 lines
8.8 KiB
Markdown
---
|
||
title: Session Memory 6 Jul 26
|
||
date: 2026-07-06
|
||
status: active
|
||
---
|
||
|
||
## Summary
|
||
|
||
Sprint_1_2 clinical chat saw major LLM UX, streaming, and inference work (Jul 5–6). The highest-risk open issue is **frontend markdown rendering of generated assistant text** — it has caused tab freezes and catastrophic crashes during reasoning streams. Reasoning paths now use **plain text only** (`StreamingPlainText`) as a mitigation. **Beta functionality** (Planning mode + 🔥 high reasoning level) must be completed **by end of this week** (target: **Friday 10 Jul 2026**).
|
||
|
||
---
|
||
|
||
## Change log (what was updated)
|
||
|
||
### Clinical chat UI & model lifecycle
|
||
|
||
| Area | Change | Key files |
|
||
|------|--------|-----------|
|
||
| LLM loading bubble | Install vs load phases with distinct copy, progress bar, disabled composer | `ClinicalChatPanel.tsx`, `useClinicalChat.ts`, `modelLoadProgress.ts` |
|
||
| Install vs load semantics | First OPFS download (~1.9 GB) vs cached checkpoint init into worker/GPU | `useClinicalChat.ts` (`ModelLoadPhase: 'installing' \| 'loading'`) |
|
||
| Sidebar card switch | Both diagnosis + review layers stay mounted (CSS hide/show) so Gemma is not torn down on carousel switch | `SidebarLayerCarousel.tsx` |
|
||
| OPFS persistence | Completed install survives reload; interrupted download not resumable (manifest only written on success) | `opfsModelStore.ts` |
|
||
| Worker init progress | `init_progress` events wired through `LlmWorkerClient.init(onProgress)` | `llmWorkerClient.ts`, `llm.worker.ts` |
|
||
|
||
### Inference modes, reasoning levels & backends
|
||
|
||
| Area | Change | Key files |
|
||
|------|--------|-----------|
|
||
| Unified chat modes | `ask` merged into `chat`; inference modes: **Chat**, **Planning** (beta), **Agent** | `clinicalChatModes.ts`, `analyzePromptComplexity.ts` |
|
||
| Reasoning levels | 🧘 chill / 🤔 moderate / 🔥 high (beta); bar visibility persisted | `chatReasoningLevel.ts`, `ClinicalChatPanel.tsx` |
|
||
| Edge vs server toggle | 🤔 moderate: **Máy** (Gemma 4 E2B edge) vs **Server** (Gemma 4 E4B Modal Ollama `think:true`) | `reasoningModelBackend.ts`, `ollamaLlmClient.ts`, `inferenceBackend.ts` |
|
||
| Ollama dev proxy | `VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat`, model `gemma4:e4b` | `.env.development`, `vite.config.ts` |
|
||
| Agent mode | Uses Modal Ollama E4B when configured; tool loop unchanged | `clinicalChatModes.ts`, `runClinicalChatTurn.ts` |
|
||
| OOM mitigation | Bootstrap at 2048 tokens; `releaseInference()` before reload; reuse engine when `configured.maxTokens >= required` | `clinicalChatConfig.ts`, `llm.worker.ts`, `llmModelBootstrap.ts` |
|
||
| Qwen3 experiment | Dual-model (Qwen `.litertlm` for moderate) attempted then **disabled** — `usesQwenReasoningLevel()` returns `false`; moderate uses Gemma CoT again | `qwenOpfsModelStore.ts`, `reasoningLlmClient.ts`, `chatReasoningLevel.ts` |
|
||
|
||
### Streaming, CoT split & markdown
|
||
|
||
| Area | Change | Key files |
|
||
|------|--------|-----------|
|
||
| CoT split | `splitGemmaThoughtOutput`, `isThoughtChannelComplete`; thought vs answer channels in message state | `prompts.ts`, `clinicalChat.ts`, `useClinicalChat.ts` |
|
||
| Collapsible reasoning | `ClinicalChatThought` — expand while streaming, auto-collapse when thought completes | `ClinicalChatThought.tsx` |
|
||
| Stream throttle | RAF-coalesced updates (~60/s) to reduce main-thread pressure | `streamUpdateThrottle.ts` |
|
||
| Token-by-token Ollama | Imperative DOM via `clinicalChatStreamRegistry` + `StreamingPlainText` (bypasses React 18 batching) | `clinicalChatStreamRegistry.ts`, `StreamingPlainText.tsx`, `ollamaLlmClient.ts` |
|
||
| Markdown renderer | Custom `ChatMarkdown` (bold, italic, code, lists, headings) — **deferred until stream ends** via `requestIdleCallback` + `startTransition` | `ChatMarkdown.tsx`, `ClinicalChatMessageBubble.tsx` |
|
||
| Reasoning = plain text | When `tracksThought`, thought + answer use `StreamingPlainText` only — **no `ChatMarkdown`** | `ClinicalChatThought.tsx`, `ClinicalChatMessageBubble.tsx` |
|
||
|
||
### Agent tools & Modal testing
|
||
|
||
| Area | Change | Key files |
|
||
|------|--------|-----------|
|
||
| Tool catalog doc | Walkthrough of Edge-LLM agent tools: `exa_search`, `supabase_query`, `escalate_medgemma` | session + `agent_tools_contract.md` |
|
||
| 3-layer smoke harness | Layer 0 Modal Ollama, Layer 1 BFF routes, Layer 2 browser `ToolExecutor` | `ml/tests/agent_tools/` |
|
||
| Python reference tests | Modal `/api/chat` streaming + thinking cases | `PILOT_PROJECT/tmp/test_endpoint.py`, `test_endpoint_img.py` |
|
||
| Gemma4 E4B deploy script | Modal Ollama serverless for Gemma 4 E4B | `PILOT_PROJECT/tmp/GemmaE4B_ollama_deploy.py` |
|
||
| gemma4_e2b lab | Tool smoke panel (no Gemma) for isolated tool calls | `ml/tests/gemma4_e2b/src/lib/toolSmoke.ts` |
|
||
|
||
---
|
||
|
||
## Critical issue: markdown rendering → catastrophic crash
|
||
|
||
### Symptom
|
||
|
||
Rendering **LLM-generated markdown** in the clinical chat UI (especially during or immediately after **reasoning / CoT streams**) has caused:
|
||
|
||
- Main-thread freezes (composer unresponsive while tokens still arrive)
|
||
- Tab crashes / OOM under combined **WebGPU model memory + large text buffers + markdown parse**
|
||
|
||
### Root cause (confirmed in debugging)
|
||
|
||
1. **Per-token full re-parse** — early implementation ran `ChatMarkdown` / `renderBlocks()` on the entire growing thought string every token → hundreds of parses per second.
|
||
2. **React re-render storm** — `setMessages` on every token re-rendered the full message list.
|
||
3. **Post-stream markdown** — even with “parse after stream ends”, heavy `renderBlocks()` on long CoT + answer text still spikes CPU/memory.
|
||
4. **Dual large strings** — `rawAccumulator`, `thoughtContent`, and `content` can all hold 2048-token traces simultaneously at 🤔 moderate.
|
||
|
||
### Mitigations applied (6 Jul)
|
||
|
||
- Stream updates throttled (`createStreamUpdateThrottle`)
|
||
- Plain text while `streaming === true`
|
||
- `React.memo` on `ClinicalChatMessageBubble`
|
||
- Reasoning paths (`tracksThought`) → **`StreamingPlainText` only, markdown disabled**
|
||
- Ollama path → imperative DOM updates per token
|
||
- `ChatMarkdown` uses idle callback + `startTransition` for post-stream formatting (chill / non-reasoning answers only)
|
||
|
||
### Current policy
|
||
|
||
> **Be wary of re-enabling markdown in reasoning mode until a safe renderer is proven.**
|
||
|
||
`ClinicalChatThought.tsx` comment: *"Reasoning panel — plain text only (markdown disabled while isolating stream crashes)."*
|
||
|
||
### Follow-up for later sprints
|
||
|
||
- [ ] Reproduce crash with a minimal `ChatMarkdown` + long fixture (no LLM) — isolate `renderBlocks` vs React
|
||
- [ ] Consider `react-markdown` with strict plugins OR server-side pre-render for final answer only
|
||
- [ ] Cap thought trace length in UI (truncate + “show more”)
|
||
- [ ] Virtualize message list for long sessions
|
||
- [ ] Re-test 🔥 high mode and Planning mode with any new markdown path before removing plain-text guard
|
||
- [ ] Audit whether chill-mode `ChatMarkdown` after stream is safe on 8 GB RAM devices
|
||
|
||
---
|
||
|
||
## Beta functionality — deadline end of week
|
||
|
||
**Target: complete by Friday 10 Jul 2026** (end of sprint week).
|
||
|
||
Features still marked `beta: true` (UI tab/button disabled until ready):
|
||
|
||
| Feature | ID | Location | What “done” means |
|
||
|---------|-----|----------|-------------------|
|
||
| **Planning mode** | `planning` | `clinicalChatModes.ts` | Checklist output stable; remove `beta` flag; selectable in mode tabs |
|
||
| **High reasoning** | `high` 🔥 | `chatReasoningLevel.ts` | 2048-token multi-turn works without crash; remove `beta` flag |
|
||
|
||
Related work not yet beta-flagged but in scope:
|
||
|
||
- Agent tool integration (Layer 3 after smoke tests pass)
|
||
- Live BFF tools (`VITE_CLINICAL_CHAT_MOCK_TOOLS=false` + credentials)
|
||
- Edge/server reasoning toggle polish (default server E4B when Modal up)
|
||
|
||
---
|
||
|
||
## Architecture snapshot (clinical LLM, 6 Jul EOD)
|
||
|
||
```
|
||
User message
|
||
│
|
||
├─ Mode: chat ─┬─ 🧘 chill → Gemma E2B edge (no CoT)
|
||
│ ├─ 🤔 moderate → Máy: Gemma E2B + CoT (plain text)
|
||
│ │ Server: Gemma E4B Ollama think:true (plain text)
|
||
│ └─ 🔥 high (BETA) → disabled in UI
|
||
│
|
||
├─ Mode: planning (BETA) → disabled in UI
|
||
│
|
||
└─ Mode: agent → Gemma E4B Ollama + tools (exa, supabase, medgemma)
|
||
|
||
Display:
|
||
tracksThought → StreamingPlainText (thought panel + answer)
|
||
else → StreamingPlainText while streaming → ChatMarkdown when done
|
||
```
|
||
|
||
---
|
||
|
||
## Key env / endpoints (dev)
|
||
|
||
```env
|
||
VITE_CLINICAL_CHAT_USE_LLM=true
|
||
VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat
|
||
VITE_OLLAMA_MODEL=gemma4:e4b
|
||
VITE_CLINICAL_CHAT_MOCK_TOOLS=true # flip false for live agent tools
|
||
```
|
||
|
||
Modal Gemma E4B (reference): `PILOT_PROJECT/tmp/test_endpoint.py` → `dtj-tran--ollama-gemma4-e4b-ollamaserver-web.modal.run`
|
||
|
||
---
|
||
|
||
## Session references
|
||
|
||
- Prior frontend memory: `session_memory/27_jun_26/27_jun_26_frontend.md`
|
||
- Agent tools smoke README: `CODEBASE/ml/tests/agent_tools/README.md`
|
||
- Today's active dev: `npm run dev` on frontend implementation
|