update session_memory - for refer to the passwork
This commit is contained in:
DatTT127
159
session_memory/6_Jul_26.md
Normal file
159
session_memory/6_Jul_26.md
Normal file
@@ -0,0 +1,159 @@
|
||||
---
|
||||
title: Session Memory 6 Jul 26
|
||||
date: 2026-07-06
|
||||
status: active
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Sprint_1_2 clinical chat saw major LLM UX, streaming, and inference work (Jul 5–6). The highest-risk open issue is **frontend markdown rendering of generated assistant text** — it has caused tab freezes and catastrophic crashes during reasoning streams. Reasoning paths now use **plain text only** (`StreamingPlainText`) as a mitigation. **Beta functionality** (Planning mode + 🔥 high reasoning level) must be completed **by end of this week** (target: **Friday 10 Jul 2026**).
|
||||
|
||||
---
|
||||
|
||||
## Change log (what was updated)
|
||||
|
||||
### Clinical chat UI & model lifecycle
|
||||
|
||||
| Area | Change | Key files |
|
||||
|------|--------|-----------|
|
||||
| LLM loading bubble | Install vs load phases with distinct copy, progress bar, disabled composer | `ClinicalChatPanel.tsx`, `useClinicalChat.ts`, `modelLoadProgress.ts` |
|
||||
| Install vs load semantics | First OPFS download (~1.9 GB) vs cached checkpoint init into worker/GPU | `useClinicalChat.ts` (`ModelLoadPhase: 'installing' \| 'loading'`) |
|
||||
| Sidebar card switch | Both diagnosis + review layers stay mounted (CSS hide/show) so Gemma is not torn down on carousel switch | `SidebarLayerCarousel.tsx` |
|
||||
| OPFS persistence | Completed install survives reload; interrupted download not resumable (manifest only written on success) | `opfsModelStore.ts` |
|
||||
| Worker init progress | `init_progress` events wired through `LlmWorkerClient.init(onProgress)` | `llmWorkerClient.ts`, `llm.worker.ts` |
|
||||
|
||||
### Inference modes, reasoning levels & backends
|
||||
|
||||
| Area | Change | Key files |
|
||||
|------|--------|-----------|
|
||||
| Unified chat modes | `ask` merged into `chat`; inference modes: **Chat**, **Planning** (beta), **Agent** | `clinicalChatModes.ts`, `analyzePromptComplexity.ts` |
|
||||
| Reasoning levels | 🧘 chill / 🤔 moderate / 🔥 high (beta); bar visibility persisted | `chatReasoningLevel.ts`, `ClinicalChatPanel.tsx` |
|
||||
| Edge vs server toggle | 🤔 moderate: **Máy** (Gemma 4 E2B edge) vs **Server** (Gemma 4 E4B Modal Ollama `think:true`) | `reasoningModelBackend.ts`, `ollamaLlmClient.ts`, `inferenceBackend.ts` |
|
||||
| Ollama dev proxy | `VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat`, model `gemma4:e4b` | `.env.development`, `vite.config.ts` |
|
||||
| Agent mode | Uses Modal Ollama E4B when configured; tool loop unchanged | `clinicalChatModes.ts`, `runClinicalChatTurn.ts` |
|
||||
| OOM mitigation | Bootstrap at 2048 tokens; `releaseInference()` before reload; reuse engine when `configured.maxTokens >= required` | `clinicalChatConfig.ts`, `llm.worker.ts`, `llmModelBootstrap.ts` |
|
||||
| Qwen3 experiment | Dual-model (Qwen `.litertlm` for moderate) attempted then **disabled** — `usesQwenReasoningLevel()` returns `false`; moderate uses Gemma CoT again | `qwenOpfsModelStore.ts`, `reasoningLlmClient.ts`, `chatReasoningLevel.ts` |
|
||||
|
||||
### Streaming, CoT split & markdown
|
||||
|
||||
| Area | Change | Key files |
|
||||
|------|--------|-----------|
|
||||
| CoT split | `splitGemmaThoughtOutput`, `isThoughtChannelComplete`; thought vs answer channels in message state | `prompts.ts`, `clinicalChat.ts`, `useClinicalChat.ts` |
|
||||
| Collapsible reasoning | `ClinicalChatThought` — expand while streaming, auto-collapse when thought completes | `ClinicalChatThought.tsx` |
|
||||
| Stream throttle | RAF-coalesced updates (~60/s) to reduce main-thread pressure | `streamUpdateThrottle.ts` |
|
||||
| Token-by-token Ollama | Imperative DOM via `clinicalChatStreamRegistry` + `StreamingPlainText` (bypasses React 18 batching) | `clinicalChatStreamRegistry.ts`, `StreamingPlainText.tsx`, `ollamaLlmClient.ts` |
|
||||
| Markdown renderer | Custom `ChatMarkdown` (bold, italic, code, lists, headings) — **deferred until stream ends** via `requestIdleCallback` + `startTransition` | `ChatMarkdown.tsx`, `ClinicalChatMessageBubble.tsx` |
|
||||
| Reasoning = plain text | When `tracksThought`, thought + answer use `StreamingPlainText` only — **no `ChatMarkdown`** | `ClinicalChatThought.tsx`, `ClinicalChatMessageBubble.tsx` |
|
||||
|
||||
### Agent tools & Modal testing
|
||||
|
||||
| Area | Change | Key files |
|
||||
|------|--------|-----------|
|
||||
| Tool catalog doc | Walkthrough of Edge-LLM agent tools: `exa_search`, `supabase_query`, `escalate_medgemma` | session + `agent_tools_contract.md` |
|
||||
| 3-layer smoke harness | Layer 0 Modal Ollama, Layer 1 BFF routes, Layer 2 browser `ToolExecutor` | `ml/tests/agent_tools/` |
|
||||
| Python reference tests | Modal `/api/chat` streaming + thinking cases | `PILOT_PROJECT/tmp/test_endpoint.py`, `test_endpoint_img.py` |
|
||||
| Gemma4 E4B deploy script | Modal Ollama serverless for Gemma 4 E4B | `PILOT_PROJECT/tmp/GemmaE4B_ollama_deploy.py` |
|
||||
| gemma4_e2b lab | Tool smoke panel (no Gemma) for isolated tool calls | `ml/tests/gemma4_e2b/src/lib/toolSmoke.ts` |
|
||||
|
||||
---
|
||||
|
||||
## Critical issue: markdown rendering → catastrophic crash
|
||||
|
||||
### Symptom
|
||||
|
||||
Rendering **LLM-generated markdown** in the clinical chat UI (especially during or immediately after **reasoning / CoT streams**) has caused:
|
||||
|
||||
- Main-thread freezes (composer unresponsive while tokens still arrive)
|
||||
- Tab crashes / OOM under combined **WebGPU model memory + large text buffers + markdown parse**
|
||||
|
||||
### Root cause (confirmed in debugging)
|
||||
|
||||
1. **Per-token full re-parse** — early implementation ran `ChatMarkdown` / `renderBlocks()` on the entire growing thought string every token → hundreds of parses per second.
|
||||
2. **React re-render storm** — `setMessages` on every token re-rendered the full message list.
|
||||
3. **Post-stream markdown** — even with “parse after stream ends”, heavy `renderBlocks()` on long CoT + answer text still spikes CPU/memory.
|
||||
4. **Dual large strings** — `rawAccumulator`, `thoughtContent`, and `content` can all hold 2048-token traces simultaneously at 🤔 moderate.
|
||||
|
||||
### Mitigations applied (6 Jul)
|
||||
|
||||
- Stream updates throttled (`createStreamUpdateThrottle`)
|
||||
- Plain text while `streaming === true`
|
||||
- `React.memo` on `ClinicalChatMessageBubble`
|
||||
- Reasoning paths (`tracksThought`) → **`StreamingPlainText` only, markdown disabled**
|
||||
- Ollama path → imperative DOM updates per token
|
||||
- `ChatMarkdown` uses idle callback + `startTransition` for post-stream formatting (chill / non-reasoning answers only)
|
||||
|
||||
### Current policy
|
||||
|
||||
> **Be wary of re-enabling markdown in reasoning mode until a safe renderer is proven.**
|
||||
|
||||
`ClinicalChatThought.tsx` comment: *"Reasoning panel — plain text only (markdown disabled while isolating stream crashes)."*
|
||||
|
||||
### Follow-up for later sprints
|
||||
|
||||
- [ ] Reproduce crash with a minimal `ChatMarkdown` + long fixture (no LLM) — isolate `renderBlocks` vs React
|
||||
- [ ] Consider `react-markdown` with strict plugins OR server-side pre-render for final answer only
|
||||
- [ ] Cap thought trace length in UI (truncate + “show more”)
|
||||
- [ ] Virtualize message list for long sessions
|
||||
- [ ] Re-test 🔥 high mode and Planning mode with any new markdown path before removing plain-text guard
|
||||
- [ ] Audit whether chill-mode `ChatMarkdown` after stream is safe on 8 GB RAM devices
|
||||
|
||||
---
|
||||
|
||||
## Beta functionality — deadline end of week
|
||||
|
||||
**Target: complete by Friday 10 Jul 2026** (end of sprint week).
|
||||
|
||||
Features still marked `beta: true` (UI tab/button disabled until ready):
|
||||
|
||||
| Feature | ID | Location | What “done” means |
|
||||
|---------|-----|----------|-------------------|
|
||||
| **Planning mode** | `planning` | `clinicalChatModes.ts` | Checklist output stable; remove `beta` flag; selectable in mode tabs |
|
||||
| **High reasoning** | `high` 🔥 | `chatReasoningLevel.ts` | 2048-token multi-turn works without crash; remove `beta` flag |
|
||||
|
||||
Related work not yet beta-flagged but in scope:
|
||||
|
||||
- Agent tool integration (Layer 3 after smoke tests pass)
|
||||
- Live BFF tools (`VITE_CLINICAL_CHAT_MOCK_TOOLS=false` + credentials)
|
||||
- Edge/server reasoning toggle polish (default server E4B when Modal up)
|
||||
|
||||
---
|
||||
|
||||
## Architecture snapshot (clinical LLM, 6 Jul EOD)
|
||||
|
||||
```
|
||||
User message
|
||||
│
|
||||
├─ Mode: chat ─┬─ 🧘 chill → Gemma E2B edge (no CoT)
|
||||
│ ├─ 🤔 moderate → Máy: Gemma E2B + CoT (plain text)
|
||||
│ │ Server: Gemma E4B Ollama think:true (plain text)
|
||||
│ └─ 🔥 high (BETA) → disabled in UI
|
||||
│
|
||||
├─ Mode: planning (BETA) → disabled in UI
|
||||
│
|
||||
└─ Mode: agent → Gemma E4B Ollama + tools (exa, supabase, medgemma)
|
||||
|
||||
Display:
|
||||
tracksThought → StreamingPlainText (thought panel + answer)
|
||||
else → StreamingPlainText while streaming → ChatMarkdown when done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key env / endpoints (dev)
|
||||
|
||||
```env
|
||||
VITE_CLINICAL_CHAT_USE_LLM=true
|
||||
VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat
|
||||
VITE_OLLAMA_MODEL=gemma4:e4b
|
||||
VITE_CLINICAL_CHAT_MOCK_TOOLS=true # flip false for live agent tools
|
||||
```
|
||||
|
||||
Modal Gemma E4B (reference): `PILOT_PROJECT/tmp/test_endpoint.py` → `dtj-tran--ollama-gemma4-e4b-ollamaserver-web.modal.run`
|
||||
|
||||
---
|
||||
|
||||
## Session references
|
||||
|
||||
- Prior frontend memory: `session_memory/27_jun_26/27_jun_26_frontend.md`
|
||||
- Agent tools smoke README: `CODEBASE/ml/tests/agent_tools/README.md`
|
||||
- Today's active dev: `npm run dev` on frontend implementation
|
||||
Reference in New Issue
Block a user