---
title: Session Memory 6 Jul 26
date: 2026-07-06
status: active
---

## Summary

Sprint_1_2 clinical chat saw major LLM UX, streaming, and inference work (Jul 5–6). The highest-risk open issue is **frontend markdown rendering of generated assistant text** — it has caused tab freezes and catastrophic crashes during reasoning streams. Reasoning paths now use **plain text only** (`StreamingPlainText`) as a mitigation. **Beta functionality** (Planning mode + 🔥 high reasoning level) must be completed **by end of this week** (target: **Friday 10 Jul 2026**).

---

## Change log (what was updated)

### Clinical chat UI & model lifecycle

| Area | Change | Key files |
|------|--------|-----------|
| LLM loading bubble | Install vs load phases with distinct copy, progress bar, disabled composer | `ClinicalChatPanel.tsx`, `useClinicalChat.ts`, `modelLoadProgress.ts` |
| Install vs load semantics | First OPFS download (~1.9 GB) vs cached checkpoint init into worker/GPU | `useClinicalChat.ts` (`ModelLoadPhase: 'installing' \| 'loading'`) |
| Sidebar card switch | Both diagnosis + review layers stay mounted (CSS hide/show) so Gemma is not torn down on carousel switch | `SidebarLayerCarousel.tsx` |
| OPFS persistence | Completed install survives reload; interrupted download not resumable (manifest only written on success) | `opfsModelStore.ts` |
| Worker init progress | `init_progress` events wired through `LlmWorkerClient.init(onProgress)` | `llmWorkerClient.ts`, `llm.worker.ts` |

### Inference modes, reasoning levels & backends

| Area | Change | Key files |
|------|--------|-----------|
| Unified chat modes | `ask` merged into `chat`; inference modes: **Chat**, **Planning** (beta), **Agent** | `clinicalChatModes.ts`, `analyzePromptComplexity.ts` |
| Reasoning levels | 🧘 chill / 🤔 moderate / 🔥 high (beta); bar visibility persisted | `chatReasoningLevel.ts`, `ClinicalChatPanel.tsx` |
| Edge vs server toggle | 🤔 moderate: **Máy** (Gemma 4 E2B edge) vs **Server** (Gemma 4 E4B Modal Ollama `think:true`) | `reasoningModelBackend.ts`, `ollamaLlmClient.ts`, `inferenceBackend.ts` |
| Ollama dev proxy | `VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat`, model `gemma4:e4b` | `.env.development`, `vite.config.ts` |
| Agent mode | Uses Modal Ollama E4B when configured; tool loop unchanged | `clinicalChatModes.ts`, `runClinicalChatTurn.ts` |
| OOM mitigation | Bootstrap at 2048 tokens; `releaseInference()` before reload; reuse engine when `configured.maxTokens >= required` | `clinicalChatConfig.ts`, `llm.worker.ts`, `llmModelBootstrap.ts` |
| Qwen3 experiment | Dual-model (Qwen `.litertlm` for moderate) attempted then **disabled** — `usesQwenReasoningLevel()` returns `false`; moderate uses Gemma CoT again | `qwenOpfsModelStore.ts`, `reasoningLlmClient.ts`, `chatReasoningLevel.ts` |

### Streaming, CoT split & markdown

| Area | Change | Key files |
|------|--------|-----------|
| CoT split | `splitGemmaThoughtOutput`, `isThoughtChannelComplete`; thought vs answer channels in message state | `prompts.ts`, `clinicalChat.ts`, `useClinicalChat.ts` |
| Collapsible reasoning | `ClinicalChatThought` — expand while streaming, auto-collapse when thought completes | `ClinicalChatThought.tsx` |
| Stream throttle | RAF-coalesced updates (~60/s) to reduce main-thread pressure | `streamUpdateThrottle.ts` |
| Token-by-token Ollama | Imperative DOM via `clinicalChatStreamRegistry` + `StreamingPlainText` (bypasses React 18 batching) | `clinicalChatStreamRegistry.ts`, `StreamingPlainText.tsx`, `ollamaLlmClient.ts` |
| Markdown renderer | Custom `ChatMarkdown` (bold, italic, code, lists, headings) — **deferred until stream ends** via `requestIdleCallback` + `startTransition` | `ChatMarkdown.tsx`, `ClinicalChatMessageBubble.tsx` |
| Reasoning = plain text | When `tracksThought`, thought + answer use `StreamingPlainText` only — **no `ChatMarkdown`** | `ClinicalChatThought.tsx`, `ClinicalChatMessageBubble.tsx` |

### Agent tools & Modal testing

| Area | Change | Key files |
|------|--------|-----------|
| Tool catalog doc | Walkthrough of Edge-LLM agent tools: `exa_search`, `supabase_query`, `escalate_medgemma` | session + `agent_tools_contract.md` |
| 3-layer smoke harness | Layer 0 Modal Ollama, Layer 1 BFF routes, Layer 2 browser `ToolExecutor` | `ml/tests/agent_tools/` |
| Python reference tests | Modal `/api/chat` streaming + thinking cases | `PILOT_PROJECT/tmp/test_endpoint.py`, `test_endpoint_img.py` |
| Gemma4 E4B deploy script | Modal Ollama serverless for Gemma 4 E4B | `PILOT_PROJECT/tmp/GemmaE4B_ollama_deploy.py` |
| gemma4_e2b lab | Tool smoke panel (no Gemma) for isolated tool calls | `ml/tests/gemma4_e2b/src/lib/toolSmoke.ts` |

---

## Critical issue: markdown rendering → catastrophic crash

### Symptom

Rendering **LLM-generated markdown** in the clinical chat UI (especially during or immediately after **reasoning / CoT streams**) has caused:

- Main-thread freezes (composer unresponsive while tokens still arrive)
- Tab crashes / OOM under combined **WebGPU model memory + large text buffers + markdown parse**

### Root cause (confirmed in debugging)

1. **Per-token full re-parse** — early implementation ran `ChatMarkdown` / `renderBlocks()` on the entire growing thought string every token → hundreds of parses per second.
2. **React re-render storm** — `setMessages` on every token re-rendered the full message list.
3. **Post-stream markdown** — even with “parse after stream ends”, heavy `renderBlocks()` on long CoT + answer text still spikes CPU/memory.
4. **Dual large strings** — `rawAccumulator`, `thoughtContent`, and `content` can all hold 2048-token traces simultaneously at 🤔 moderate.

### Mitigations applied (6 Jul)

- Stream updates throttled (`createStreamUpdateThrottle`)
- Plain text while `streaming === true`
- `React.memo` on `ClinicalChatMessageBubble`
- Reasoning paths (`tracksThought`) → **`StreamingPlainText` only, markdown disabled**
- Ollama path → imperative DOM updates per token
- `ChatMarkdown` uses idle callback + `startTransition` for post-stream formatting (chill / non-reasoning answers only)

### Current policy

> **Be wary of re-enabling markdown in reasoning mode until a safe renderer is proven.**

`ClinicalChatThought.tsx` comment: *"Reasoning panel — plain text only (markdown disabled while isolating stream crashes)."*

### Follow-up for later sprints

- [ ] Reproduce crash with a minimal `ChatMarkdown` + long fixture (no LLM) — isolate `renderBlocks` vs React
- [ ] Consider `react-markdown` with strict plugins OR server-side pre-render for final answer only
- [ ] Cap thought trace length in UI (truncate + “show more”)
- [ ] Virtualize message list for long sessions
- [ ] Re-test 🔥 high mode and Planning mode with any new markdown path before removing plain-text guard
- [ ] Audit whether chill-mode `ChatMarkdown` after stream is safe on 8 GB RAM devices

---

## Beta functionality — deadline end of week

**Target: complete by Friday 10 Jul 2026** (end of sprint week).

Features still marked `beta: true` (UI tab/button disabled until ready):

| Feature | ID | Location | What “done” means |
|---------|-----|----------|-------------------|
| **Planning mode** | `planning` | `clinicalChatModes.ts` | Checklist output stable; remove `beta` flag; selectable in mode tabs |
| **High reasoning** | `high` 🔥 | `chatReasoningLevel.ts` | 2048-token multi-turn works without crash; remove `beta` flag |

Related work not yet beta-flagged but in scope:

- Agent tool integration (Layer 3 after smoke tests pass)
- Live BFF tools (`VITE_CLINICAL_CHAT_MOCK_TOOLS=false` + credentials)
- Edge/server reasoning toggle polish (default server E4B when Modal up)

---

## Architecture snapshot (clinical LLM, 6 Jul EOD)

```
User message
    │
    ├─ Mode: chat ─┬─ 🧘 chill  → Gemma E2B edge (no CoT)
    │              ├─ 🤔 moderate → Máy: Gemma E2B + CoT (plain text)
    │              │              Server: Gemma E4B Ollama think:true (plain text)
    │              └─ 🔥 high (BETA) → disabled in UI
    │
    ├─ Mode: planning (BETA) → disabled in UI
    │
    └─ Mode: agent → Gemma E4B Ollama + tools (exa, supabase, medgemma)

Display:
    tracksThought → StreamingPlainText (thought panel + answer)
    else          → StreamingPlainText while streaming → ChatMarkdown when done
```

---

## Key env / endpoints (dev)

```env
VITE_CLINICAL_CHAT_USE_LLM=true
VITE_OLLAMA_CHAT_URL=/api/ollama-chat/api/chat
VITE_OLLAMA_MODEL=gemma4:e4b
VITE_CLINICAL_CHAT_MOCK_TOOLS=true   # flip false for live agent tools
```

Modal Gemma E4B (reference): `PILOT_PROJECT/tmp/test_endpoint.py` → `dtj-tran--ollama-gemma4-e4b-ollamaserver-web.modal.run`

---

## Session references

- Prior frontend memory: `session_memory/27_jun_26/27_jun_26_frontend.md`
- Agent tools smoke README: `CODEBASE/ml/tests/agent_tools/README.md`
- Today's active dev: `npm run dev` on frontend implementation