update the codebase poc ver1
This commit is contained in:
DatTT127
@@ -0,0 +1,263 @@
|
||||
# Corpus Profiles — Book-Specific Ingestion Rules
|
||||
|
||||
## Purpose
|
||||
|
||||
Defines per-textbook metadata parsing, logical-unit assembly keys, and boundary sources for the semantic chunking pipeline.
|
||||
|
||||
## Profile Summary
|
||||
|
||||
| Book | `book_id` | Constraint tier | Physical unit | Logical unit | Assembly |
|
||||
|------|-----------|-----------------|---------------|--------------|----------|
|
||||
| ADO | `ado` | Loose | 5 pages, unconstrained | Section (via TOC/headers) | Stitch all batches → detect boundaries |
|
||||
| TNY | `tny` | Loose | 5 pages, unconstrained | Section (via TOC/headers) | Stitch all batches → detect boundaries |
|
||||
| OHO | `oho` | Section | 5 pages within section | Section | Group by `section_id`, stitch batches |
|
||||
| MOR | `mor` | Subsection | Whole subsection (var. pages) | Subsection | One row = one unit (no stitch) |
|
||||
|
||||
---
|
||||
|
||||
## ADO
|
||||
|
||||
### Physical split
|
||||
|
||||
- Pattern: same as TNY — fixed 5-page batches across entire bounded PDF
|
||||
- Output: `corpus/pdf/ado/batches/`
|
||||
- Filename: `batch-{NN}_chunk-{PPP}-{PPP}.pdf`
|
||||
- Example: `batch-02_chunk-006-010.pdf`
|
||||
|
||||
### Filename regex
|
||||
|
||||
```regex
|
||||
^batch-(?P<batch_index>\d+)_chunk-(?P<page_start>\d+)-(?P<page_end>\d+)\.pdf$
|
||||
```
|
||||
|
||||
### Logical unit assembly
|
||||
|
||||
1. Load all checkpoint rows; sort by `page_start`
|
||||
2. Concatenate into `book_stream`
|
||||
3. Subdivide using TOC map (when added) or Docling `#` / `##` headers
|
||||
4. `logical_unit_id` = `{section_number}` or `{header_slug}`
|
||||
|
||||
### Boundary sources
|
||||
|
||||
- TOC map: TBD (`corpus/pdf/ado/index.txt` or equivalent)
|
||||
- Fallback: markdown headers in Docling output
|
||||
|
||||
---
|
||||
|
||||
## TNY
|
||||
|
||||
### Physical split
|
||||
|
||||
- Script: `knowledge/tests/tny_split_page.py`
|
||||
- Master PDF: `corpus/pdf/tny/tny_bounded.pdf`
|
||||
- Output: `corpus/pdf/tny/batches/`
|
||||
- Batch size: 5 pages
|
||||
- Filename: `batch-{NN}_chunk-{PPP}-{PPP}.pdf`
|
||||
- Example: `batch-02_chunk-006-010.pdf`
|
||||
|
||||
### Filename regex
|
||||
|
||||
```regex
|
||||
^batch-(?P<batch_index>\d+)_chunk-(?P<page_start>\d+)-(?P<page_end>\d+)\.pdf$
|
||||
```
|
||||
|
||||
### Parsed metadata example
|
||||
|
||||
```json
|
||||
{
|
||||
"book_id": "tny",
|
||||
"batch_index": 2,
|
||||
"page_start": 6,
|
||||
"page_end": 10,
|
||||
"section_id": null,
|
||||
"subsection_id": null
|
||||
}
|
||||
```
|
||||
|
||||
### Logical unit assembly
|
||||
|
||||
Same as ADO (loose tier):
|
||||
|
||||
1. Stitch all batches in page order
|
||||
2. Detect section boundaries via TOC or headers
|
||||
3. `logical_unit_id` = `sec_{N}` or `h2_{slug}`
|
||||
|
||||
### Boundary sources
|
||||
|
||||
- TOC: TBD
|
||||
- NER glossary (separate concern): `corpus/ner_glossary/TNY_*.txt`
|
||||
|
||||
### Checkpoint DB
|
||||
|
||||
`tny_ingestion_corpus.db` (see `knowledge/tests/ingestion_corpus_db.py`)
|
||||
|
||||
---
|
||||
|
||||
## OHO
|
||||
|
||||
### Physical split
|
||||
|
||||
- Script: `knowledge/tests/oho_split_page.py`
|
||||
- Master PDF: `corpus/pdf/oho/oho_bounded.pdf`
|
||||
- Section index: `corpus/pdf/oho/index.txt`
|
||||
- Output: `corpus/pdf/oho/sections/`
|
||||
- Batch size: 5 pages **within each section** (does not cross section boundaries)
|
||||
- Filename: `sec_{N}_batch-{NN}_chunk-{PPP}-{PPP}.pdf`
|
||||
- Example: `sec_3_batch-01_chunk-104-108.pdf`
|
||||
|
||||
### Section index format (`index.txt`)
|
||||
|
||||
```text
|
||||
section 1: 32-50
|
||||
section 2: 51-103
|
||||
section 3: 104-243
|
||||
...
|
||||
```
|
||||
|
||||
### Filename regex
|
||||
|
||||
```regex
|
||||
^sec_(?P<section_id>\d+)_batch-(?P<batch_index>\d+)_chunk-(?P<page_start>\d+)-(?P<page_end>\d+)\.pdf$
|
||||
```
|
||||
|
||||
### Parsed metadata example
|
||||
|
||||
```json
|
||||
{
|
||||
"book_id": "oho",
|
||||
"section_id": 3,
|
||||
"batch_index": 1,
|
||||
"page_start": 104,
|
||||
"page_end": 108,
|
||||
"subsection_id": null
|
||||
}
|
||||
```
|
||||
|
||||
### Logical unit assembly
|
||||
|
||||
1. Group checkpoint rows by `section_id`
|
||||
2. Sort by `batch_index` (then `page_start`)
|
||||
3. Concatenate within group
|
||||
4. `logical_unit_id` = `sec_{section_id}`
|
||||
|
||||
**Never** merge rows across different `section_id` values.
|
||||
|
||||
### Boundary sources
|
||||
|
||||
- Hard: `corpus/pdf/oho/index.txt` (section page ranges)
|
||||
- Soft: `##` / `###` headers within assembled section text
|
||||
- Index markdown (separate artifact): `corpus/indexs/oho_index_clean.md`
|
||||
|
||||
### Checkpoint DB
|
||||
|
||||
`oho_ingestion_corpus.db`
|
||||
|
||||
---
|
||||
|
||||
## MOR
|
||||
|
||||
### Physical split
|
||||
|
||||
- Script: `knowledge/tests/mor_split_page.py`
|
||||
- Master PDF: `corpus/pdf/mor/mor_bounded.pdf`
|
||||
- TOC: `corpus/pdf/mor/toc_structure.json`
|
||||
- Output: `corpus/pdf/mor/sections/`
|
||||
- Unit: one PDF per **subsection** (page count varies)
|
||||
- Filename: `sub_{subsection_id}_chunk-{PPP}-{PPP}.pdf`
|
||||
- Example: `sub_1.1_chunk-028-032.pdf`
|
||||
|
||||
### Filename regex
|
||||
|
||||
```regex
|
||||
^sub_(?P<subsection_id>\d+\.\d+)_chunk-(?P<page_start>\d+)-(?P<page_end>\d+)\.pdf$
|
||||
```
|
||||
|
||||
### Parsed metadata example
|
||||
|
||||
```json
|
||||
{
|
||||
"book_id": "mor",
|
||||
"subsection_id": "1.1",
|
||||
"section_id": 1,
|
||||
"page_start": 28,
|
||||
"page_end": 32,
|
||||
"batch_index": null
|
||||
}
|
||||
```
|
||||
|
||||
Derive `section_id` as integer part of `subsection_id` (`1.1` → section `1`).
|
||||
|
||||
### Logical unit assembly
|
||||
|
||||
**No cross-row assembly.** Each `processed_chunks` row maps 1:1 to a logical unit.
|
||||
|
||||
- `logical_unit_id` = `sub_{subsection_id}`
|
||||
- `parent_title` from `toc_structure.json` → matching subsection `title`
|
||||
|
||||
### TOC lookup
|
||||
|
||||
```json
|
||||
{
|
||||
"sections": [
|
||||
{
|
||||
"section_number": 1,
|
||||
"section_title": "SECTION I: GENERAL",
|
||||
"subsections": [
|
||||
{
|
||||
"title": "1.1 History Taking for Patients with Rheumatic Complaints",
|
||||
"page_range": "28-32"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Match `subsection_id` prefix to subsection `title` (e.g. `1.1` → title starting with `1.1`).
|
||||
|
||||
### Checkpoint DB
|
||||
|
||||
`mor_ingestion_corpus.db`
|
||||
|
||||
---
|
||||
|
||||
## Metadata parser interface
|
||||
|
||||
Future module: `knowledge/implementation/ingestion/metadata_parser.py`
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class SourceMetadata:
|
||||
book_id: str
|
||||
page_start: int
|
||||
page_end: int
|
||||
section_id: int | None
|
||||
subsection_id: str | None
|
||||
batch_index: int | None
|
||||
source_filename: str
|
||||
|
||||
def parse_source_path(source_path: str) -> SourceMetadata: ...
|
||||
|
||||
def logical_unit_key(meta: SourceMetadata) -> str:
|
||||
"""Stable assembly key for grouping checkpoint rows."""
|
||||
if meta.book_id == "mor":
|
||||
return f"sub_{meta.subsection_id}"
|
||||
if meta.book_id == "oho":
|
||||
return f"sec_{meta.section_id}"
|
||||
# ado, tny — assigned after book_stream section detection
|
||||
raise NotImplementedError("loose tier uses post-assembly logical_unit_id")
|
||||
```
|
||||
|
||||
## Rollout validation matrix
|
||||
|
||||
| Book | Validate assembly | Validate chunk count | Validate citations |
|
||||
|------|-------------------|----------------------|--------------------|
|
||||
| MOR | 1 row → 1 unit | ~1 chunk per short subsection | page_range matches TOC |
|
||||
| OHO | N batches → 1 unit per section | multiple chunks per long section | section_id matches index.txt |
|
||||
| TNY/ADO | all rows → section units | headers align with visual TOC | page ranges contiguous |
|
||||
|
||||
## References
|
||||
|
||||
- `ingestion_pipeline_spec.md`
|
||||
- `semantic_chunking_spec.md`
|
||||
- `schema.md`
|
||||
@@ -0,0 +1,155 @@
|
||||
# Knowledge Ingestion Pipeline Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
Defines the end-to-end pipeline that turns clinical textbook PDFs into retrieval-ready semantic chunks with traceable citations. This spec covers Stages 0–4; Stage 3 (semantic chunking) is detailed in `semantic_chunking_spec.md`.
|
||||
|
||||
## Owner
|
||||
|
||||
Knowledge Engineering Team
|
||||
|
||||
## Scope
|
||||
|
||||
| In scope | Out of scope |
|
||||
|----------|--------------|
|
||||
| PDF physical splitting for Docling | Qdrant collection tuning |
|
||||
| Docling markdown extraction checkpoint (SQLite) | LLM grounding / RAG answer generation |
|
||||
| Book-aware logical-unit assembly | ladybugDB ontology ingestion |
|
||||
| Header + semantic chunking | Production embedding server deployment |
|
||||
| Chunk metadata schema | Frontend citation UI |
|
||||
|
||||
## Pipeline Stages
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
S0[Stage 0: PDF split] --> S1[Stage 1: Docling extract]
|
||||
S1 --> S2[(Stage 2: SQLite checkpoint)]
|
||||
S2 --> S3[Stage 3: Semantic chunk]
|
||||
S3 --> S4[(Stage 4: Embed + Qdrant)]
|
||||
```
|
||||
|
||||
### Stage 0 — Physical PDF split
|
||||
|
||||
Splits master bounded PDFs into Docling-sized slices. **Physical splits are for extraction parallelism only; they are not RAG chunk boundaries.**
|
||||
|
||||
| Book | Constraint tier | Split rule | Output dir | Reference script |
|
||||
|------|-----------------|------------|------------|------------------|
|
||||
| **ADO** | Loose | Fixed 5-page batches, no section boundary | `corpus/pdf/ado/batches/` | (same pattern as TNY) |
|
||||
| **TNY** | Loose | Fixed 5-page batches, no section boundary | `corpus/pdf/tny/batches/` | `knowledge/tests/tny_split_page.py` |
|
||||
| **OHO** | Section | 5-page batches **within** each section | `corpus/pdf/oho/sections/` | `knowledge/tests/oho_split_page.py` |
|
||||
| **MOR** | Subsection | One PDF per TOC subsection (variable page count) | `corpus/pdf/mor/sections/` | `knowledge/tests/mor_split_page.py` |
|
||||
|
||||
Section/subsection page maps:
|
||||
|
||||
- OHO: `corpus/pdf/oho/index.txt`
|
||||
- MOR: `corpus/pdf/mor/toc_structure.json`
|
||||
|
||||
### Stage 1 — Docling extraction
|
||||
|
||||
Convert each physical PDF slice to markdown.
|
||||
|
||||
- **Engine:** Docling `DocumentConverter` (pre-loaded singleton for batch runs)
|
||||
- **Output:** Markdown string per source file
|
||||
- **Reference:** `knowledge/tests/ingestion_corpus_db.py`
|
||||
|
||||
Constraints:
|
||||
|
||||
- One SQLite row per physical PDF file (never merge at this stage)
|
||||
- Preserve full `source_path` for downstream metadata parsing
|
||||
- Idempotent: skip files already present in `processed_chunks` unless `force_reextract=true`
|
||||
|
||||
### Stage 2 — Extraction checkpoint (SQLite)
|
||||
|
||||
Raw markdown checkpoint. **Do not semantic-chunk or embed directly from this table.**
|
||||
|
||||
See `schema.md` → `processed_chunks`.
|
||||
|
||||
Per-book databases (convention):
|
||||
|
||||
| Book | DB file |
|
||||
|------|---------|
|
||||
| ADO | `ado_ingestion_corpus.db` |
|
||||
| TNY | `tny_ingestion_corpus.db` |
|
||||
| OHO | `oho_ingestion_corpus.db` |
|
||||
| MOR | `mor_ingestion_corpus.db` |
|
||||
|
||||
A unified `{book}_ingestion_corpus.db` per textbook keeps checkpoint isolation and allows independent re-chunking.
|
||||
|
||||
### Stage 3 — Semantic chunking
|
||||
|
||||
Transform checkpoint rows into retrieval-sized chunks using book-aware assembly + hybrid splitting.
|
||||
|
||||
**Detailed rules:** `semantic_chunking_spec.md`
|
||||
**Book profiles:** `corpus_profiles.md`
|
||||
|
||||
Input: `processed_chunks` rows for one book
|
||||
Output: `semantic_chunks` rows (same DB or sibling DB)
|
||||
|
||||
Properties:
|
||||
|
||||
- Versioned (`chunker_version`) — re-chunk without re-extracting
|
||||
- Idempotent — skip logical units already chunked at current version
|
||||
- Traceable — every chunk links back to one or more `processed_chunks.id`
|
||||
|
||||
### Stage 4 — Embedding and vector store
|
||||
|
||||
Embed `semantic_chunks.content` and upsert to Qdrant (768-d, EmbeddingGemma per `knowledge_spec.md`).
|
||||
|
||||
Payload fields (minimum):
|
||||
|
||||
```json
|
||||
{
|
||||
"chunk_id": "uuid",
|
||||
"book_id": "mor",
|
||||
"logical_unit_id": "1.1",
|
||||
"parent_title": "1.1 History Taking for Patients with Rheumatic Complaints",
|
||||
"page_start": 28,
|
||||
"page_end": 32,
|
||||
"chunk_index": 0,
|
||||
"chunker_version": "header+semantic_v1",
|
||||
"source_extraction_ids": [12]
|
||||
}
|
||||
```
|
||||
|
||||
Update `semantic_chunks.embedding_status` and `qdrant_point_id` on success.
|
||||
|
||||
## Planned Implementation Layout
|
||||
|
||||
```
|
||||
knowledge/
|
||||
├─ spec/ingestion/ # This spec package
|
||||
├─ implementation/ingestion/ # Stage 1–4 modules (future)
|
||||
│ ├─ extract.py # Docling → processed_chunks
|
||||
│ ├─ assemble.py # Book-aware logical-unit assembly
|
||||
│ ├─ chunk.py # Header + semantic chunking
|
||||
│ ├─ embed.py # EmbeddingGemma → Qdrant
|
||||
│ └─ metadata_parser.py # source_path → structured metadata
|
||||
└─ tests/
|
||||
├─ ingestion_corpus_db.py # Stage 1–2 (existing)
|
||||
├─ *_split_page.py # Stage 0 (existing)
|
||||
└─ markdown_chunking_oho.py # Prototype header split (superseded by chunk.py)
|
||||
```
|
||||
|
||||
## Rollout Order
|
||||
|
||||
1. **MOR** — subsection rows ≈ logical units; validate chunk → embed → retrieve loop
|
||||
2. **OHO** — section-scoped batch assembly from filenames
|
||||
3. **TNY / ADO** — full-book stream assembly + header/TOC boundary detection
|
||||
|
||||
## Breaking-change Policy
|
||||
|
||||
| Change | Version bump |
|
||||
|--------|--------------|
|
||||
| New book profile | MINOR |
|
||||
| `chunker_version` algorithm change | New version string; re-chunk required |
|
||||
| `semantic_chunks` schema additive column | MINOR |
|
||||
| Embedding model dimension change | MAJOR (per `knowledge_spec.md`) |
|
||||
| Removal of citation metadata field | MAJOR |
|
||||
|
||||
## References
|
||||
|
||||
- `semantic_chunking_spec.md` — Stage 3 algorithm
|
||||
- `corpus_profiles.md` — per-book assembly and filename parsing
|
||||
- `schema.md` — SQLite table definitions
|
||||
- `../knowledge_spec.md` — RAG stack (Qdrant, EmbeddingGemma)
|
||||
- `knowledge/tests/ingestion_corpus_db.py` — current Stage 1–2 prototype
|
||||
190
workspace/sprint_1_2/CODEBASE/knowledge/spec/ingestion/schema.md
Normal file
190
workspace/sprint_1_2/CODEBASE/knowledge/spec/ingestion/schema.md
Normal file
@@ -0,0 +1,190 @@
|
||||
# Ingestion SQLite Schema
|
||||
|
||||
## Purpose
|
||||
|
||||
Defines checkpoint and semantic-chunk tables for the knowledge ingestion pipeline. One database file per textbook.
|
||||
|
||||
## Database files
|
||||
|
||||
| Book | Filename |
|
||||
|------|----------|
|
||||
| ADO | `ado_ingestion_corpus.db` |
|
||||
| TNY | `tny_ingestion_corpus.db` |
|
||||
| OHO | `oho_ingestion_corpus.db` |
|
||||
| MOR | `mor_ingestion_corpus.db` |
|
||||
|
||||
Location: `knowledge/corpus/db/` (recommended) or co-located with ingestion scripts during development.
|
||||
|
||||
---
|
||||
|
||||
## Table: `processed_chunks` (Stage 2 — extraction checkpoint)
|
||||
|
||||
Raw Docling markdown per physical PDF slice. **Read-only input for semantic chunking.**
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS processed_chunks (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_path TEXT NOT NULL UNIQUE,
|
||||
markdown_content TEXT NOT NULL,
|
||||
execution_time REAL,
|
||||
processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_processed_chunks_source_path
|
||||
ON processed_chunks (source_path);
|
||||
```
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `id` | INTEGER | Surrogate key; referenced by `semantic_chunks.source_extraction_ids` |
|
||||
| `source_path` | TEXT | Absolute or repo-relative path to source PDF |
|
||||
| `markdown_content` | TEXT | Full Docling markdown for that PDF slice |
|
||||
| `execution_time` | REAL | Docling wall time (seconds) |
|
||||
| `processed_at` | TIMESTAMP | Insert timestamp |
|
||||
|
||||
**Migration from current prototype:** `knowledge/tests/ingestion_corpus_db.py` already creates this table; add `UNIQUE` on `source_path` and index when promoting to implementation.
|
||||
|
||||
---
|
||||
|
||||
## Table: `semantic_chunks` (Stage 3 — retrieval chunks)
|
||||
|
||||
Hybrid header + semantic chunks ready for embedding.
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS semantic_chunks (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
chunk_uuid TEXT NOT NULL UNIQUE,
|
||||
logical_unit_id TEXT NOT NULL,
|
||||
chunk_index INTEGER NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
token_count INTEGER NOT NULL,
|
||||
book_id TEXT NOT NULL,
|
||||
section_id INTEGER,
|
||||
subsection_id TEXT,
|
||||
parent_title TEXT,
|
||||
page_start INTEGER,
|
||||
page_end INTEGER,
|
||||
source_extraction_ids TEXT NOT NULL, -- JSON array of processed_chunks.id
|
||||
chunker_version TEXT NOT NULL,
|
||||
embedding_status TEXT NOT NULL DEFAULT 'pending',
|
||||
qdrant_point_id TEXT,
|
||||
content_hash TEXT NOT NULL,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
|
||||
UNIQUE (logical_unit_id, chunk_index, chunker_version, book_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_semantic_chunks_book_version
|
||||
ON semantic_chunks (book_id, chunker_version);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_semantic_chunks_embedding_status
|
||||
ON semantic_chunks (embedding_status);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_semantic_chunks_logical_unit
|
||||
ON semantic_chunks (logical_unit_id, chunker_version);
|
||||
```
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `chunk_uuid` | TEXT | UUID v4 for Qdrant payload / API citations |
|
||||
| `logical_unit_id` | TEXT | Assembly key, e.g. `sec_3`, `sub_1.1`, `tny_sec_12` |
|
||||
| `chunk_index` | INTEGER | Order within logical unit (0-based) |
|
||||
| `content` | TEXT | Final chunk markdown |
|
||||
| `token_count` | INTEGER | Tokens per embedding model tokenizer |
|
||||
| `book_id` | TEXT | `ado`, `tny`, `oho`, `mor` |
|
||||
| `section_id` | INTEGER | Nullable; from filename or TOC |
|
||||
| `subsection_id` | TEXT | Nullable; e.g. `1.1` (MOR) |
|
||||
| `parent_title` | TEXT | Human-readable section/subsection title |
|
||||
| `page_start`, `page_end` | INTEGER | Inclusive citation page range |
|
||||
| `source_extraction_ids` | TEXT | JSON `[12, 13, 14]` — provenance |
|
||||
| `chunker_version` | TEXT | e.g. `header+semantic_v1` |
|
||||
| `embedding_status` | TEXT | `pending` \| `done` \| `failed` |
|
||||
| `qdrant_point_id` | TEXT | Set after Stage 4 upsert |
|
||||
| `content_hash` | TEXT | SHA-256 of `content` for dedup |
|
||||
|
||||
### `embedding_status` values
|
||||
|
||||
| Value | Meaning |
|
||||
|-------|---------|
|
||||
| `pending` | Chunk written; not yet embedded |
|
||||
| `done` | Embedded and upserted to Qdrant |
|
||||
| `failed` | Embed/upsert error; retry eligible |
|
||||
|
||||
---
|
||||
|
||||
## Table: `chunking_runs` (optional — audit log)
|
||||
|
||||
Track chunk pipeline executions for idempotency and debugging.
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS chunking_runs (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
book_id TEXT NOT NULL,
|
||||
chunker_version TEXT NOT NULL,
|
||||
started_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
finished_at TIMESTAMP,
|
||||
status TEXT NOT NULL DEFAULT 'running',
|
||||
units_processed INTEGER DEFAULT 0,
|
||||
chunks_written INTEGER DEFAULT 0,
|
||||
error_message TEXT,
|
||||
|
||||
UNIQUE (book_id, chunker_version, started_at)
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PostgreSQL target (Supabase)
|
||||
|
||||
Stage 4 upserts into the **`knowledge`** schema on Supabase (pgvector HNSW). See:
|
||||
|
||||
- `../pg_semantic_vector_db/er_diagram.md` — entity model
|
||||
- `../pg_semantic_vector_db/supabase_schema.md` — migration reference
|
||||
- `../../supabase/migrations/` — apply with `supabase db push`
|
||||
|
||||
| SQLite column | Supabase table.column |
|
||||
|---------------|----------------------|
|
||||
| `chunk_uuid` | `knowledge.semantic_chunk.chunk_id` |
|
||||
| `embedding_status` | `knowledge.semantic_chunk_embedding.embedding_status` |
|
||||
| `qdrant_point_id` | **dropped** — use `chunk_id` |
|
||||
|
||||
---
|
||||
|
||||
## Qdrant payload mapping (legacy reference)
|
||||
|
||||
| Qdrant payload field | Source column |
|
||||
|----------------------|---------------|
|
||||
| `chunk_id` | `chunk_uuid` |
|
||||
| `book_id` | `book_id` |
|
||||
| `logical_unit_id` | `logical_unit_id` |
|
||||
| `parent_title` | `parent_title` |
|
||||
| `page_start` | `page_start` |
|
||||
| `page_end` | `page_end` |
|
||||
| `chunk_index` | `chunk_index` |
|
||||
| `chunker_version` | `chunker_version` |
|
||||
| `section_id` | `section_id` |
|
||||
| `subsection_id` | `subsection_id` |
|
||||
|
||||
Vector: 768 dimensions (EmbeddingGemma).
|
||||
|
||||
Collection naming convention: `guidelines_{book_id}_{chunker_version}` (e.g. `guidelines_mor_header+semantic_v1`).
|
||||
|
||||
---
|
||||
|
||||
## Idempotency rules
|
||||
|
||||
| Operation | Skip condition |
|
||||
|-----------|----------------|
|
||||
| Extract (Stage 1–2) | `source_path` already in `processed_chunks` |
|
||||
| Chunk (Stage 3) | `(logical_unit_id, chunk_index, chunker_version, book_id)` exists |
|
||||
| Embed (Stage 4) | `embedding_status = 'done'` |
|
||||
|
||||
Force re-run: pass `--force` flag (implementation) to delete and regenerate rows for scoped book + version.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- `ingestion_pipeline_spec.md`
|
||||
- `semantic_chunking_spec.md`
|
||||
- `corpus_profiles.md`
|
||||
@@ -0,0 +1,198 @@
|
||||
# Semantic Chunking Specification
|
||||
|
||||
## Purpose
|
||||
|
||||
Defines how raw Docling markdown (stored in `processed_chunks`) is transformed into retrieval-ready semantic chunks. Applies a **hybrid strategy**: book-aware assembly → structural pre-split → embedding-based semantic split.
|
||||
|
||||
## Owner
|
||||
|
||||
Knowledge Engineering Team
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Physical PDF splits ≠ RAG chunks.** Extraction batches exist for Docling memory and parallelism.
|
||||
2. **Structure before semantics.** Markdown headers and TOC boundaries are hard guardrails; semantic splitting refines oversized pieces only.
|
||||
3. **Never split inside protected blocks.** Tables, code fences, and figure captions stay intact.
|
||||
4. **Every chunk is citable.** Page range, book, logical unit, and source extraction IDs are mandatory metadata.
|
||||
5. **Re-chunkable.** `chunker_version` allows algorithm updates without re-running Docling.
|
||||
|
||||
## Chunking Pipeline
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Read processed_chunks for book] --> B[Parse source_path metadata]
|
||||
B --> C[Assemble logical units]
|
||||
C --> D[Header pre-split]
|
||||
D --> E{token_count > max_tokens?}
|
||||
E -->|no| F[Emit chunk as-is]
|
||||
E -->|yes| G[Semantic split]
|
||||
G --> H[Enforce min_tokens / merge tiny pieces]
|
||||
F --> I[Write semantic_chunks]
|
||||
H --> I
|
||||
```
|
||||
|
||||
### Step 1 — Parse `source_path`
|
||||
|
||||
Extract structured metadata from each checkpoint row's filename. Regex patterns are defined in `corpus_profiles.md`.
|
||||
|
||||
Minimum parsed fields:
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `book_id` | `ado`, `tny`, `oho`, `mor` |
|
||||
| `page_start`, `page_end` | Inclusive page numbers from filename |
|
||||
| `section_id` | Section number (OHO) or null |
|
||||
| `subsection_id` | Subsection id e.g. `1.1` (MOR) or null |
|
||||
| `batch_index` | Intra-unit batch sequence or null |
|
||||
|
||||
### Step 2 — Assemble logical units
|
||||
|
||||
Group checkpoint rows into **logical units** before any splitting. Assembly rules differ by constraint tier (see `corpus_profiles.md`).
|
||||
|
||||
| Tier | Books | Assembly rule |
|
||||
|------|-------|---------------|
|
||||
| Loose | ADO, TNY | Stitch **all** batches in page order → one stream per book; then subdivide by headers/TOC |
|
||||
| Section | OHO | Group by `section_id`; stitch batches within section in page order |
|
||||
| Subsection | MOR | **No cross-row assembly** — each row is one logical unit |
|
||||
|
||||
Concatenation separator between stitched batches:
|
||||
|
||||
```text
|
||||
\n\n<!-- batch-boundary -->\n\n
|
||||
```
|
||||
|
||||
The HTML comment marker is stripped from final chunk content but logged in assembly metadata for debugging mid-batch topic splits.
|
||||
|
||||
### Step 3 — Header pre-split
|
||||
|
||||
Split assembled text on markdown headers before semantic chunking.
|
||||
|
||||
Default header levels:
|
||||
|
||||
| Level | Metadata key | Use |
|
||||
|-------|--------------|-----|
|
||||
| `#` | `h1` | Part / major division (ADO, TNY when detected) |
|
||||
| `##` | `h2` | Section |
|
||||
| `###` | `h3` | Subsection |
|
||||
| `<!-- image --> ` | `<!-- image -->` | Image|
|
||||
|
||||
Implementation reference: LangChain `MarkdownHeaderTextSplitter` (prototype in `knowledge/tests/markdown_chunking_oho.py`).
|
||||
|
||||
Rules:
|
||||
|
||||
- Propagate header metadata to all descendant chunks (`parent_title` = nearest `##` or `###`)
|
||||
- If a header section is empty after strip, discard it
|
||||
- Do not split inside fenced code blocks (```) or HTML tables
|
||||
|
||||
### Step 4 — Semantic split (oversized pieces only)
|
||||
|
||||
Apply embedding-based breakpoint detection **only** when `token_count(content) > max_tokens`.
|
||||
|
||||
| Parameter | Default | Notes |
|
||||
|-----------|---------|-------|
|
||||
| `max_tokens` | 800 | Target upper bound per chunk |
|
||||
| `min_tokens` | 120 | Merge or carry forward pieces below this |
|
||||
| `overlap_tokens` | 64 | Optional overlap at semantic boundaries |
|
||||
| `breakpoint_threshold_type` | `percentile` | Alternative: `standard_deviation`, `interquartile` |
|
||||
| `breakpoint_threshold_amount` | 95 | Percentile for similarity drop detection |
|
||||
| `embedding_model` | EmbeddingGemma | Must match Stage 4 embed model |
|
||||
|
||||
Recommended library: LangChain Experimental `SemanticChunker` or LlamaIndex `SemanticSplitterNodeParser`.
|
||||
|
||||
Sentence/paragraph granularity:
|
||||
|
||||
- Split input into sentences (or paragraphs for list-heavy clinical text)
|
||||
- Embed consecutive windows; detect largest cosine-similarity drops
|
||||
- Cut at breakpoints; merge segments below `min_tokens` with neighbors
|
||||
|
||||
**MOR shortcut:** If logical unit token count ≤ `max_tokens`, skip semantic split entirely (pass-through after optional header split).
|
||||
|
||||
### Step 5 — Post-processing
|
||||
|
||||
1. **Merge tiny chunks:** If `token_count < min_tokens`, merge with previous chunk in same logical unit (unless previous would exceed `max_tokens * 1.25`)
|
||||
2. **Assign `chunk_index`:** Zero-based, ordered within logical unit
|
||||
3. **Compute page range:** For multi-batch units, use min(`page_start`) and max(`page_end`) across constituent extraction rows; for semantic sub-chunks within a unit, inherit parent page range (fine-grained page mapping is a future enhancement)
|
||||
4. **Deduplicate:** Hash `(book_id, logical_unit_id, chunk_index, chunker_version, content)` — skip exact duplicates on re-run
|
||||
|
||||
## Chunker Versions
|
||||
|
||||
| Version | Description |
|
||||
|---------|-------------|
|
||||
| `header_v1` | Header pre-split only; no semantic split |
|
||||
| `header+semantic_v1` | Header pre-split + semantic split on oversized pieces (default) |
|
||||
|
||||
Bump version string when algorithm parameters or assembly rules change materially.
|
||||
|
||||
## Book-specific behavior summary
|
||||
|
||||
### ADO / TNY (loose tier)
|
||||
|
||||
Highest assembly effort:
|
||||
|
||||
1. Load all `processed_chunks` ordered by `page_start`
|
||||
2. Concatenate into `book_stream`
|
||||
3. Subdivide `book_stream`:
|
||||
- **Preferred:** TOC page→section map (when available)
|
||||
- **Fallback:** Split on `#` / `##` headers from Docling output
|
||||
4. Header pre-split within each section
|
||||
5. Semantic split oversized sections
|
||||
|
||||
Risk: topic spans batch boundary with no header at boundary → semantic chunker must see full assembled section text, not individual batches.
|
||||
|
||||
### OHO (section tier)
|
||||
|
||||
1. Group rows by `section_id` from filename (`sec_N_...`)
|
||||
2. Sort by `batch_index` / `page_start`; concatenate
|
||||
3. Header pre-split within section
|
||||
4. Semantic split if section text > `max_tokens`
|
||||
|
||||
Section boundaries from `corpus/pdf/oho/index.txt` are authoritative — never merge across sections.
|
||||
|
||||
### MOR (subsection tier)
|
||||
|
||||
1. Each row = one logical unit (`subsection_id` from `sub_X.X_...`)
|
||||
2. Optional header pre-split on `##` / `###` within subsection
|
||||
3. Semantic split only if subsection > `max_tokens`
|
||||
4. Many subsections (≤5 pages) may produce a **single chunk**
|
||||
|
||||
Subsection titles from `corpus/pdf/mor/toc_structure.json` populate `parent_title`.
|
||||
|
||||
## Protected content rules
|
||||
|
||||
Do not split inside:
|
||||
|
||||
- Markdown tables (`| ... |` blocks)
|
||||
- Fenced code blocks
|
||||
- Numbered/bulleted lists shorter than `min_tokens` (keep list intact)
|
||||
- Figure/table captions immediately following a figure reference line
|
||||
|
||||
If semantic split would break a table, prefer keeping the entire table in one chunk even if it slightly exceeds `max_tokens` (hard cap: `max_tokens * 1.5`).
|
||||
|
||||
## Output contract
|
||||
|
||||
Each row in `semantic_chunks` must satisfy:
|
||||
|
||||
- `content` is non-empty UTF-8 markdown
|
||||
- `token_count` is computed with the same tokenizer as the embedding model
|
||||
- `source_extraction_ids` is a JSON array of one or more `processed_chunks.id`
|
||||
- `logical_unit_id` is stable across re-runs (see `corpus_profiles.md`)
|
||||
- `chunker_version` matches the active pipeline version
|
||||
|
||||
## Evaluation criteria (acceptance)
|
||||
|
||||
Before promoting a new `chunker_version`:
|
||||
|
||||
| Check | Target |
|
||||
|-------|--------|
|
||||
| Sample retrieval on 20 clinical queries | ≥ baseline vs header-only |
|
||||
| Chunks with `token_count < min_tokens` | < 5% of total |
|
||||
| Chunks with `token_count > max_tokens * 1.5` | < 2% of total |
|
||||
| Chunks missing `parent_title` (MOR/OHO) | 0% |
|
||||
| Citation traceability (manual spot check) | 100% link to source pages |
|
||||
|
||||
## References
|
||||
|
||||
- `ingestion_pipeline_spec.md` — full pipeline context
|
||||
- `corpus_profiles.md` — filename patterns and assembly keys
|
||||
- `schema.md` — `semantic_chunks` table
|
||||
- `knowledge/tests/markdown_chunking_oho.py` — header split prototype
|
||||
@@ -28,7 +28,7 @@ Knowledge Engineering Team
|
||||
- ladybugDB schema details (predicate names, ontology version)
|
||||
- embedding model specifics (model ID, tensor shapes)
|
||||
- LLM prompt templates and decoding parameters
|
||||
- knowledge curation pipeline details (source ingestion, validation)
|
||||
- ingestion SQLite checkpoint paths and chunker version strings (see `spec/ingestion/` for pipeline design; operational paths are internal)
|
||||
|
||||
## Breaking-change Policy
|
||||
- Knowledge schema versioning via semantic versioning (MAJOR.MINOR.PATCH)
|
||||
|
||||
@@ -18,12 +18,33 @@ Qdrant vector database instances, ladybugDB graph database instances, embedding
|
||||
- Retrieval pipeline: hybrid search (vector + BM25) → graph expansion → reranking
|
||||
- Grounding module: verifies LLM outputs against source guidelines with citation extraction
|
||||
- Arbitration engine: resolves conflicting evidence using belief propagation
|
||||
- Continuous integration: automated guideline ingestion from trusted sources (NIH, CDC, radiology societies)
|
||||
- Textbook ingestion pipeline: PDF split → Docling checkpoint (SQLite) → book-aware semantic chunking → pgvector upsert (see `spec/ingestion/`, `spec/pg_semantic_vector_db/`)
|
||||
- Versioned knowledge bases with temporal validity tracking
|
||||
- Monitoring: retrieval relevance, grounding accuracy, latency SLOs
|
||||
|
||||
## Ingestion Specifications
|
||||
See `spec/ingestion/` for the textbook RAG ingestion pipeline:
|
||||
|
||||
| Document | Scope |
|
||||
|----------|-------|
|
||||
| `ingestion_pipeline_spec.md` | Stages 0–4 end-to-end |
|
||||
| `semantic_chunking_spec.md` | Hybrid header + semantic chunking algorithm |
|
||||
| `corpus_profiles.md` | ADO, TNY, OHO, MOR assembly and filename rules |
|
||||
| `schema.md` | SQLite `processed_chunks` and `semantic_chunks` tables |
|
||||
|
||||
## Vector Store Specifications
|
||||
|
||||
See `spec/pg_semantic_vector_db/` for the PostgreSQL + pgvector retrieval schema:
|
||||
|
||||
| Document | Scope |
|
||||
|----------|-------|
|
||||
| `er_diagram.md` | Entities, relationships, and SQLite→PG mapping |
|
||||
| `supabase_schema.md` | Applied Supabase migrations, RPC, security model |
|
||||
|
||||
Migrations: `knowledge/supabase/migrations/`
|
||||
|
||||
## Interface Contract
|
||||
See `bento/knowledge/spec/interface-contract.md`.
|
||||
See `spec/interface_contract.md`.
|
||||
|
||||
## Consumers
|
||||
- frontend:guideline-spec (for displaying grounded explanations in UI)
|
||||
|
||||
@@ -0,0 +1,509 @@
|
||||
# pg-Semantic-Vector-DB — Entity-Relationship Design
|
||||
|
||||
## Purpose
|
||||
|
||||
Define the **PostgreSQL + pgvector** data model for retrieval-ready semantic chunks from clinical textbooks (ADO, TNY, OHO, MOR). This is the **authoritative RAG store** queried at runtime (HNSW similarity search + SQL metadata filters).
|
||||
|
||||
## Scope
|
||||
|
||||
| In scope | Out of scope (separate stores) |
|
||||
|----------|--------------------------------|
|
||||
| Textbook semantic chunks and their EmbeddingGemma vectors | SQLite ingestion checkpoints (`processed_chunks`) |
|
||||
| Citation metadata for NFR-18 grounding | BERT drift/referee embeddings (separate `drift_embeddings` table per architecture) |
|
||||
| Corpus edition and chunker versioning | ladybugDB ontology graph |
|
||||
| Provenance links back to extraction checkpoints | Session/case embeddings (clinical data domain) |
|
||||
| HNSW index lifecycle per corpus + model | Qdrant (Phase 2 hot cache only) |
|
||||
|
||||
## Storage boundary
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph sqlite [SQLite — ingestion staging]
|
||||
PC[processed_chunks]
|
||||
SC_STG[semantic_chunks staging]
|
||||
end
|
||||
subgraph pg [PostgreSQL + pgvector — retrieval]
|
||||
CE[corpus_edition]
|
||||
LU[logical_unit]
|
||||
SC[semantic_chunk]
|
||||
SCE[semantic_chunk_embedding]
|
||||
end
|
||||
PC -->|Stage 3 chunk| SC_STG
|
||||
SC_STG -->|Stage 4 upsert| SC
|
||||
SC --> SCE
|
||||
```
|
||||
|
||||
SQLite remains **per-book, per-pipeline** checkpoint storage. PostgreSQL holds **deduplicated, queryable, versioned** chunks with vectors.
|
||||
|
||||
---
|
||||
|
||||
## ER diagram
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
corpus_source ||--o{ corpus_edition : "has editions"
|
||||
corpus_edition ||--o{ structure_node : "has TOC tree"
|
||||
structure_node ||--o{ structure_node : "parent of"
|
||||
corpus_edition ||--o{ logical_unit : "contains"
|
||||
structure_node |o--o| logical_unit : "may map to"
|
||||
logical_unit ||--|{ semantic_chunk : "split into"
|
||||
chunker_profile ||--o{ semantic_chunk : "produced by"
|
||||
semantic_chunk ||--o{ chunk_provenance : "traced from"
|
||||
semantic_chunk ||--o{ semantic_chunk_embedding : "embedded as"
|
||||
embedding_model ||--o{ semantic_chunk_embedding : "uses"
|
||||
corpus_edition ||--o{ ingestion_run : "processed in"
|
||||
chunker_profile ||--o{ ingestion_run : "uses"
|
||||
corpus_edition ||--o{ vector_index_snapshot : "indexed under"
|
||||
embedding_model ||--o{ vector_index_snapshot : "indexed with"
|
||||
|
||||
corpus_source {
|
||||
uuid corpus_source_id PK
|
||||
text book_id UK "ado | tny | oho | mor"
|
||||
text display_name
|
||||
text constraint_tier "loose | section | subsection"
|
||||
text locale "vi-VN"
|
||||
timestamptz created_at
|
||||
}
|
||||
|
||||
corpus_edition {
|
||||
uuid edition_id PK
|
||||
uuid corpus_source_id FK
|
||||
text edition_label "e.g. 2024-print, v1"
|
||||
text source_pdf_sha256
|
||||
text status "draft | active | superseded | archived"
|
||||
date effective_from
|
||||
date effective_to
|
||||
timestamptz created_at
|
||||
}
|
||||
|
||||
structure_node {
|
||||
uuid structure_node_id PK
|
||||
uuid edition_id FK
|
||||
uuid parent_structure_node_id FK "nullable — root nodes"
|
||||
text node_type "part | section | subsection"
|
||||
text node_key "e.g. sec_3, 1.1"
|
||||
text title
|
||||
int page_start
|
||||
int page_end
|
||||
int sort_order
|
||||
}
|
||||
|
||||
logical_unit {
|
||||
uuid logical_unit_id PK
|
||||
uuid edition_id FK
|
||||
uuid structure_node_id FK "nullable"
|
||||
text unit_key "stable key e.g. sec_3, sub_1.1"
|
||||
text parent_title
|
||||
int page_start
|
||||
int page_end
|
||||
timestamptz assembled_at
|
||||
}
|
||||
|
||||
chunker_profile {
|
||||
text chunker_version PK "e.g. header+semantic_v1"
|
||||
text description
|
||||
int max_tokens
|
||||
int min_tokens
|
||||
int overlap_tokens
|
||||
jsonb params_json
|
||||
timestamptz created_at
|
||||
}
|
||||
|
||||
semantic_chunk {
|
||||
uuid chunk_id PK "API citation id"
|
||||
uuid logical_unit_id FK
|
||||
uuid edition_id FK
|
||||
text chunker_version FK
|
||||
int chunk_index "0-based within logical unit"
|
||||
text content "markdown"
|
||||
int token_count
|
||||
text content_hash "SHA-256"
|
||||
int section_id "nullable"
|
||||
text subsection_id "nullable"
|
||||
text parent_title
|
||||
int page_start
|
||||
int page_end
|
||||
boolean is_active "soft retire on re-chunk"
|
||||
timestamptz created_at
|
||||
}
|
||||
|
||||
chunk_provenance {
|
||||
uuid provenance_id PK
|
||||
uuid chunk_id FK
|
||||
text checkpoint_db "e.g. mor_ingestion_corpus.db"
|
||||
bigint processed_chunk_id "SQLite processed_chunks.id"
|
||||
text source_path "original PDF slice path"
|
||||
}
|
||||
|
||||
embedding_model {
|
||||
uuid model_id PK
|
||||
text model_name "EmbeddingGemma"
|
||||
text model_version
|
||||
int dimensions "768"
|
||||
text purpose "rag"
|
||||
boolean is_active
|
||||
timestamptz registered_at
|
||||
}
|
||||
|
||||
semantic_chunk_embedding {
|
||||
uuid embedding_id PK
|
||||
uuid chunk_id FK
|
||||
uuid model_id FK
|
||||
uuid edition_id FK
|
||||
vector embedding "pgvector, 768-d"
|
||||
text embedding_status "pending | indexed | failed"
|
||||
timestamptz embedded_at
|
||||
}
|
||||
|
||||
ingestion_run {
|
||||
uuid run_id PK
|
||||
uuid edition_id FK
|
||||
text chunker_version FK
|
||||
uuid model_id FK "nullable until Stage 4"
|
||||
text stage "chunk | embed | full"
|
||||
text status "running | succeeded | failed"
|
||||
int units_processed
|
||||
int chunks_written
|
||||
int embeddings_written
|
||||
text error_message
|
||||
timestamptz started_at
|
||||
timestamptz finished_at
|
||||
}
|
||||
|
||||
vector_index_snapshot {
|
||||
uuid index_snapshot_id PK
|
||||
uuid edition_id FK
|
||||
uuid model_id FK
|
||||
text index_name "e.g. hnsw_semantic_chunk_embedding_rag"
|
||||
jsonb hnsw_params "m, ef_construction"
|
||||
boolean is_active
|
||||
timestamptz built_at
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Entity definitions
|
||||
|
||||
### 1. `corpus_source`
|
||||
|
||||
Registry of ingestible textbook corpora. One row per book family.
|
||||
|
||||
| Attribute | Notes |
|
||||
|-----------|-------|
|
||||
| `book_id` | Stable slug: `ado`, `tny`, `oho`, `mor` |
|
||||
| `constraint_tier` | Mirrors `corpus_profiles.md`: drives assembly rules at ingest |
|
||||
| `locale` | Default `vi-VN` for pilot textbooks |
|
||||
| `full_title`, `short_title`, `subtitle` | Bibliographic titles for citations |
|
||||
| `primary_language` | ISO-style language code (`en`, `vi`) |
|
||||
| `subject_area` | RAG classification (e.g. Rheumatology) |
|
||||
| `bibliography_json` | Extensible work metadata (keywords, resource_type) |
|
||||
|
||||
Bibliographic seed data: [`corpus/source_metadata.md`](../../corpus/source_metadata.md).
|
||||
|
||||
**Cardinality:** one `corpus_source` → many `corpus_edition`.
|
||||
|
||||
---
|
||||
|
||||
### 2. `corpus_edition`
|
||||
|
||||
Immutable snapshot of a source document generation. Re-chunking and re-embedding are scoped to an edition.
|
||||
|
||||
| Attribute | Notes |
|
||||
|-----------|-------|
|
||||
| `source_pdf_sha256` | Detects PDF replacement without silent drift |
|
||||
| `status` | Only one `active` edition per `corpus_source` at a time (enforced by partial unique index) |
|
||||
| `effective_from` / `effective_to` | Temporal validity for citations and audit |
|
||||
| `publisher_name`, `publisher_place` | Publication imprint |
|
||||
| `published_year`, `published_date` | Publication date for citations |
|
||||
| `isbn_13`, `isbn_10` | ISBN identifiers |
|
||||
| `edition_number`, `source_uri` | Edition label and canonical source file |
|
||||
| `bibliography_json` | Extensible edition metadata (LCCN, series, etc.) |
|
||||
|
||||
**Why separate from `corpus_source`:** Same book (TNY) may be re-ingested after PDF correction; old chunks stay addressable for audit while new edition becomes `active`.
|
||||
|
||||
---
|
||||
|
||||
### 2b. `corpus_contributor`
|
||||
|
||||
Authors, editors, and other credited contributors for a work or specific edition.
|
||||
|
||||
| Attribute | Notes |
|
||||
|-----------|-------|
|
||||
| `corpus_source_id` | Required FK to work |
|
||||
| `edition_id` | NULL = work-level contributor; set for edition-specific overrides |
|
||||
| `role` | `author`, `editor`, `translator`, `compiler`, `institution` |
|
||||
| `full_name`, `sort_order` | Citation ordering |
|
||||
|
||||
Seed data: [`corpus/source_metadata.md`](../../corpus/source_metadata.md).
|
||||
|
||||
---
|
||||
|
||||
### 3. `structure_node`
|
||||
|
||||
Optional document hierarchy (TOC). Strongly populated for OHO/MOR; ADO/TNY may be partially inferred from headers post-assembly.
|
||||
|
||||
| Attribute | Notes |
|
||||
|-----------|-------|
|
||||
| `node_type` | `part`, `section`, `subsection` |
|
||||
| `node_key` | Matches filename/TOC keys: `sec_3`, `1.1` |
|
||||
| Self-FK | Tree; root nodes have `parent_structure_node_id IS NULL` |
|
||||
|
||||
**Relationship:** `structure_node` **0..1 — 0..1** `logical_unit` (a logical unit may exist without a TOC node when boundaries come from header detection only).
|
||||
|
||||
---
|
||||
|
||||
### 4. `logical_unit`
|
||||
|
||||
The **assembly boundary** before chunking — the unit described in `semantic_chunking_spec.md`.
|
||||
|
||||
| Attribute | Notes |
|
||||
|-----------|-------|
|
||||
| `unit_key` | Stable across runs: `sec_{N}`, `sub_{X.Y}`, `h2_{slug}` |
|
||||
| `edition_id` | Same `unit_key` in two editions = different rows |
|
||||
|
||||
**Cardinality:** one `logical_unit` → one or many `semantic_chunk` (after header/semantic split).
|
||||
|
||||
**Assembly mapping by tier:**
|
||||
|
||||
| Tier | Books | `logical_unit` origin |
|
||||
|------|-------|------------------------|
|
||||
| Loose | ADO, TNY | One row per detected section after full-book stitch |
|
||||
| Section | OHO | One row per `section_id` |
|
||||
| Subsection | MOR | One row per `subsection_id` (= one checkpoint row) |
|
||||
|
||||
---
|
||||
|
||||
### 5. `chunker_profile`
|
||||
|
||||
Version registry for the chunking algorithm. Not a runtime entity for queries, but FK anchor for reproducibility.
|
||||
|
||||
| Attribute | Notes |
|
||||
|-----------|-------|
|
||||
| `chunker_version` | PK string: `header+semantic_v1` |
|
||||
| `params_json` | Thresholds, breakpoint settings, protected-block rules hash |
|
||||
|
||||
**Cardinality:** one profile → many `semantic_chunk` rows (across editions and logical units).
|
||||
|
||||
---
|
||||
|
||||
### 6. `semantic_chunk`
|
||||
|
||||
Core **retrieval text unit**. This is what RAG returns (content + citation metadata).
|
||||
|
||||
| Attribute | Notes |
|
||||
|-----------|-------|
|
||||
| `chunk_id` | UUID v4 — external citation id (replaces SQLite `chunk_uuid` / Qdrant point id) |
|
||||
| `chunk_index` | Order within parent `logical_unit` |
|
||||
| `is_active` | Set `false` when superseded by re-chunk; keeps audit trail |
|
||||
| Citation fields | `parent_title`, `page_start`, `page_end`, `section_id`, `subsection_id` |
|
||||
|
||||
**Uniqueness:**
|
||||
|
||||
```sql
|
||||
UNIQUE (logical_unit_id, chunk_index, chunker_version)
|
||||
-- scoped implicitly by edition via logical_unit.edition_id
|
||||
```
|
||||
|
||||
**Cardinality:** one chunk → zero or one active embedding per model (see below).
|
||||
|
||||
---
|
||||
|
||||
### 7. `chunk_provenance`
|
||||
|
||||
Many-to-one trace from a semantic chunk back to SQLite extraction rows. Supports multi-batch assembly (ADO/TNY/OHO).
|
||||
|
||||
| Attribute | Notes |
|
||||
|-----------|-------|
|
||||
| `checkpoint_db` | Which `{book}_ingestion_corpus.db` |
|
||||
| `processed_chunk_id` | FK into SQLite, not PostgreSQL |
|
||||
| `source_path` | Denormalized for debugging without opening SQLite |
|
||||
|
||||
**Cardinality:** one `semantic_chunk` → **1..N** `chunk_provenance` (N > 1 when batches were stitched).
|
||||
|
||||
---
|
||||
|
||||
### 8. `embedding_model`
|
||||
|
||||
Registry of embedding models allowed to populate the RAG index.
|
||||
|
||||
| Attribute | Notes |
|
||||
|-----------|-------|
|
||||
| `purpose` | `rag` only in this schema; drift models live elsewhere |
|
||||
| `dimensions` | Must match pgvector column width (768 for EmbeddingGemma) |
|
||||
|
||||
PoC: single active row (`EmbeddingGemma`, 768-d).
|
||||
|
||||
---
|
||||
|
||||
### 9. `semantic_chunk_embedding`
|
||||
|
||||
**The vector store.** Separated from `semantic_chunk` so text can be re-embedded under a new model without duplicating content rows.
|
||||
|
||||
| Attribute | Notes |
|
||||
|-----------|-------|
|
||||
| `embedding` | `vector(768)` with HNSW index |
|
||||
| `embedding_status` | Pipeline state: `pending` → `indexed` |
|
||||
| `edition_id` | Denormalized for partition-friendly queries |
|
||||
|
||||
**Uniqueness:**
|
||||
|
||||
```sql
|
||||
UNIQUE (chunk_id, model_id)
|
||||
```
|
||||
|
||||
**Cardinality:** one chunk → at most one embedding per model.
|
||||
|
||||
**Query pattern (runtime RAG):**
|
||||
|
||||
```sql
|
||||
SELECT sc.chunk_id, sc.content, sc.parent_title, sc.page_start, sc.page_end,
|
||||
1 - (sce.embedding <=> :query_vec) AS similarity
|
||||
FROM semantic_chunk_embedding sce
|
||||
JOIN semantic_chunk sc ON sc.chunk_id = sce.chunk_id
|
||||
JOIN corpus_edition ce ON ce.edition_id = sc.edition_id
|
||||
JOIN corpus_source cs ON cs.corpus_source_id = ce.corpus_source_id
|
||||
WHERE ce.status = 'active'
|
||||
AND sc.is_active = TRUE
|
||||
AND sce.embedding_status = 'indexed'
|
||||
AND cs.book_id = ANY(:book_ids)
|
||||
ORDER BY sce.embedding <=> :query_vec
|
||||
LIMIT :k;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 10. `ingestion_run`
|
||||
|
||||
Audit log for chunk and embed pipeline executions (PostgreSQL counterpart to SQLite `chunking_runs` + embed stage).
|
||||
|
||||
| Attribute | Notes |
|
||||
|-----------|-------|
|
||||
| `stage` | `chunk`, `embed`, or `full` |
|
||||
| Counters | `units_processed`, `chunks_written`, `embeddings_written` |
|
||||
|
||||
**Cardinality:** one edition may have many runs (idempotent re-runs, version bumps).
|
||||
|
||||
---
|
||||
|
||||
### 11. `vector_index_snapshot`
|
||||
|
||||
Tracks HNSW index builds for blue/green reindex when `chunker_version` or `embedding_model` changes.
|
||||
|
||||
| Attribute | Notes |
|
||||
|-----------|-------|
|
||||
| `is_active` | Only one active snapshot per `(edition_id, model_id)` |
|
||||
| `hnsw_params` | e.g. `{"m": 16, "ef_construction": 64}` |
|
||||
|
||||
At PoC scale (~15K vectors), a single HNSW index on `semantic_chunk_embedding.embedding` filtered by `edition_id` is sufficient; this entity documents index lifecycle for future corpus growth.
|
||||
|
||||
---
|
||||
|
||||
## Relationship summary
|
||||
|
||||
| From | To | Cardinality | Semantics |
|
||||
|------|-----|-------------|-----------|
|
||||
| `corpus_source` | `corpus_edition` | 1:N | Book has versioned PDF editions |
|
||||
| `corpus_edition` | `structure_node` | 1:N | Edition has TOC tree |
|
||||
| `structure_node` | `structure_node` | 1:N | Parent/child hierarchy |
|
||||
| `corpus_edition` | `logical_unit` | 1:N | Edition contains assembly units |
|
||||
| `structure_node` | `logical_unit` | 0..1:0..1 | Optional TOC alignment |
|
||||
| `logical_unit` | `semantic_chunk` | 1:N | Unit splits into retrieval chunks |
|
||||
| `chunker_profile` | `semantic_chunk` | 1:N | Algorithm version provenance |
|
||||
| `semantic_chunk` | `chunk_provenance` | 1:N | Trace to SQLite extractions |
|
||||
| `semantic_chunk` | `semantic_chunk_embedding` | 1:N | One row per embedding model |
|
||||
| `embedding_model` | `semantic_chunk_embedding` | 1:N | Model produces many vectors |
|
||||
| `corpus_edition` | `ingestion_run` | 1:N | Pipeline execution history |
|
||||
| `corpus_edition` | `vector_index_snapshot` | 1:N | Index rebuild history |
|
||||
|
||||
---
|
||||
|
||||
## Design decisions
|
||||
|
||||
### Why split `semantic_chunk` and `semantic_chunk_embedding`?
|
||||
|
||||
- Re-embed with a new model without cloning text rows
|
||||
- Keep inactive/historical embeddings while marking chunks inactive
|
||||
- HNSW index targets only the embedding table (smaller hot set)
|
||||
|
||||
### Why `corpus_edition` instead of embedding `book_id` only?
|
||||
|
||||
- Supports PDF corrections and temporal citation ("per TNY 2024 edition, p. 42")
|
||||
- Clean supersession: deactivate old edition chunks without deleting audit data
|
||||
- Aligns with architecture "versioned knowledge bases with temporal validity tracking"
|
||||
|
||||
### Why keep provenance as a junction table?
|
||||
|
||||
- ADO/TNY/OHO chunks often span multiple `processed_chunks` rows
|
||||
- SQLite checkpoint DBs stay per-book; provenance stores the cross-reference explicitly
|
||||
|
||||
### Active-edition invariant
|
||||
|
||||
At query time, RAG MUST filter:
|
||||
|
||||
```sql
|
||||
ce.status = 'active' AND sc.is_active = TRUE AND sce.embedding_status = 'indexed'
|
||||
```
|
||||
|
||||
Only one active edition per `corpus_source` prevents mixed-corpus retrieval.
|
||||
|
||||
---
|
||||
|
||||
## Mapping from SQLite staging (`schema.md`)
|
||||
|
||||
| SQLite `semantic_chunks` | PostgreSQL |
|
||||
|--------------------------|------------|
|
||||
| `chunk_uuid` | `semantic_chunk.chunk_id` |
|
||||
| `logical_unit_id` | `logical_unit.unit_key` (+ FK via edition) |
|
||||
| `chunk_index` | `semantic_chunk.chunk_index` |
|
||||
| `content` | `semantic_chunk.content` |
|
||||
| `token_count` | `semantic_chunk.token_count` |
|
||||
| `book_id` | `corpus_source.book_id` (via edition) |
|
||||
| `section_id`, `subsection_id` | same columns on `semantic_chunk` |
|
||||
| `parent_title`, `page_start`, `page_end` | same |
|
||||
| `source_extraction_ids` (JSON) | `chunk_provenance` rows (1 per id) |
|
||||
| `chunker_version` | FK → `chunker_profile` |
|
||||
| `embedding_status` | `semantic_chunk_embedding.embedding_status` |
|
||||
| `qdrant_point_id` | **dropped** — `chunk_id` is the retrieval key |
|
||||
| `content_hash` | `semantic_chunk.content_hash` |
|
||||
|
||||
---
|
||||
|
||||
## Indexes (planned)
|
||||
|
||||
| Index | Table | Purpose |
|
||||
|-------|-------|---------|
|
||||
| HNSW | `semantic_chunk_embedding.embedding` | Cosine similarity search |
|
||||
| `(edition_id, is_active)` | `semantic_chunk` | Filter active chunks per edition |
|
||||
| `(corpus_source_id) WHERE status = 'active'` | `corpus_edition` | Partial unique — one active edition |
|
||||
| `(logical_unit_id, chunk_index, chunker_version)` | `semantic_chunk` | Idempotent upsert |
|
||||
| `(chunk_id, model_id)` | `semantic_chunk_embedding` | Unique embedding per model |
|
||||
| `(book_id)` | `corpus_source` | Filter by textbook at join time |
|
||||
|
||||
---
|
||||
|
||||
## PoC entity subset
|
||||
|
||||
For first implementation (MOR → OHO → TNY/ADO), minimum viable tables:
|
||||
|
||||
1. `corpus_source`
|
||||
2. `corpus_edition`
|
||||
3. `logical_unit`
|
||||
4. `chunker_profile`
|
||||
5. `semantic_chunk`
|
||||
6. `chunk_provenance`
|
||||
7. `embedding_model`
|
||||
8. `semantic_chunk_embedding`
|
||||
|
||||
Defer until needed: `structure_node`, `vector_index_snapshot` (use DDL-only HNSW), full `ingestion_run` UI.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- `../ingestion/schema.md` — SQLite staging schema
|
||||
- `../ingestion/semantic_chunking_spec.md` — logical unit and chunk contract
|
||||
- `../ingestion/corpus_profiles.md` — per-book assembly tiers
|
||||
- `../knowledge_spec.md` — RAG stack (pgvector HNSW, EmbeddingGemma 768-d)
|
||||
- `supabase_schema.md` — Supabase migration reference (`knowledge/supabase/migrations/`)
|
||||
- `../../../Design_Material/SPRINT_1_2_ARCHITECTURE_SPEC.md` — pgvector over Qdrant at PoC scale
|
||||
@@ -0,0 +1,98 @@
|
||||
# Supabase Schema Reference — pg-Semantic-Vector-DB
|
||||
|
||||
Maps the ER design to applied Supabase migrations under `knowledge/supabase/migrations/`.
|
||||
|
||||
## Migration order
|
||||
|
||||
| File | Contents |
|
||||
|------|----------|
|
||||
| `20260705000000_semantic_vector_db_schema.sql` | `knowledge` schema, 11 tables, constraints, HNSW index |
|
||||
| `20260705000001_semantic_vector_db_seed.sql` | ADO/TNY/OHO/MOR sources, `header+semantic_v1`, EmbeddingGemma |
|
||||
| `20260705000002_semantic_vector_db_rls_and_rpc.sql` | RLS, grants, `match_semantic_chunks()` RPC |
|
||||
| `20260705000003_expose_knowledge_api_schema.sql` | PostgREST grants for `knowledge` schema |
|
||||
| `20260705000004_chunker_profile_v2.sql` | `header+semantic_v2` chunker profile |
|
||||
| `20260705000005_corpus_bibliography.sql` | Bibliography columns, `corpus_contributor`, citation view |
|
||||
| `20260705000006_corpus_bibliography_seed.sql` | Bibliographic seed from `corpus/source_metadata.md` |
|
||||
|
||||
## Schema: `knowledge`
|
||||
|
||||
| Table / view | PK | Notable constraints |
|
||||
|-------|-----|---------------------|
|
||||
| `corpus_source` | `corpus_source_id` | `UNIQUE (book_id)`; bibliography columns |
|
||||
| `corpus_edition` | `edition_id` | Partial unique: one `active` per source; publisher/ISBN |
|
||||
| `corpus_contributor` | `contributor_id` | Authors/editors per work or edition |
|
||||
| `v_corpus_citation` | — | View: active edition + contributor string |
|
||||
| `structure_node` | `structure_node_id` | `UNIQUE (edition_id, node_key)` |
|
||||
| `logical_unit` | `logical_unit_id` | `UNIQUE (edition_id, unit_key)` |
|
||||
| `chunker_profile` | `chunker_version` | — |
|
||||
| `semantic_chunk` | `chunk_id` | `UNIQUE (logical_unit_id, chunk_index, chunker_version)` |
|
||||
| `chunk_provenance` | `provenance_id` | `UNIQUE (chunk_id, checkpoint_db, processed_chunk_id)` |
|
||||
| `embedding_model` | `model_id` | `UNIQUE (model_name, model_version)` |
|
||||
| `semantic_chunk_embedding` | `embedding_id` | `UNIQUE (chunk_id, model_id)`; HNSW on `embedding` |
|
||||
| `ingestion_run` | `run_id` | — |
|
||||
| `vector_index_snapshot` | `index_snapshot_id` | Partial unique: one active per edition+model |
|
||||
|
||||
## Vector column
|
||||
|
||||
```sql
|
||||
embedding extensions.vector(768) -- EmbeddingGemma, cosine HNSW
|
||||
```
|
||||
|
||||
Extension: `CREATE EXTENSION vector WITH SCHEMA extensions` (Supabase default pattern).
|
||||
|
||||
## RPC: `knowledge.match_semantic_chunks`
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `query_embedding` | `extensions.vector(768)` | required | Query vector from EmbeddingGemma |
|
||||
| `match_count` | `integer` | `5` | Top-k limit |
|
||||
| `filter_book_ids` | `text[]` | `NULL` | Restrict to `ado`, `tny`, `oho`, `mor` |
|
||||
| `filter_edition_ids` | `uuid[]` | `NULL` | Restrict to specific editions |
|
||||
|
||||
Returns: chunk content, citation fields, `book_id`, `similarity` (1 − cosine distance).
|
||||
|
||||
## Security model
|
||||
|
||||
| Role | Access |
|
||||
|------|--------|
|
||||
| `service_role` | Full CRUD on all `knowledge.*` tables; executes RPC |
|
||||
| `authenticated` | SELECT on active chunks + indexed embeddings; executes RPC |
|
||||
| `anon` | Denied (no policies) |
|
||||
|
||||
Ingestion pipeline and RAG coordinator should use **`service_role`** from the backend only.
|
||||
|
||||
## SQLite staging → Supabase upsert keys
|
||||
|
||||
| SQLite (`semantic_chunks`) | Supabase upsert key |
|
||||
|----------------------------|---------------------|
|
||||
| `book_id` | Resolve → `corpus_edition` via active edition for `corpus_source.book_id` |
|
||||
| `logical_unit_id` | `logical_unit.unit_key` + `edition_id` |
|
||||
| `chunk_index`, `chunker_version` | `semantic_chunk` unique constraint |
|
||||
| `source_extraction_ids[]` | One `chunk_provenance` row per id |
|
||||
| `chunk_uuid` | `semantic_chunk.chunk_id` (preserve on upsert) |
|
||||
|
||||
## Activate a new edition
|
||||
|
||||
```sql
|
||||
BEGIN;
|
||||
UPDATE knowledge.corpus_edition
|
||||
SET status = 'superseded', effective_to = CURRENT_DATE
|
||||
WHERE corpus_source_id = (SELECT corpus_source_id FROM knowledge.corpus_source WHERE book_id = 'mor')
|
||||
AND status = 'active';
|
||||
|
||||
INSERT INTO knowledge.corpus_edition (corpus_source_id, edition_label, source_pdf_sha256, status, effective_from)
|
||||
VALUES (
|
||||
(SELECT corpus_source_id FROM knowledge.corpus_source WHERE book_id = 'mor'),
|
||||
'v1',
|
||||
'<sha256>',
|
||||
'active',
|
||||
CURRENT_DATE
|
||||
);
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [`er_diagram.md`](er_diagram.md) — entity definitions and ER diagram
|
||||
- [`../ingestion/schema.md`](../ingestion/schema.md) — SQLite checkpoint schema
|
||||
- [`../../supabase/README.md`](../../supabase/README.md) — deploy instructions
|
||||
Reference in New Issue
Block a user