update the codebase poc ver1

2026-07-07 15:54:17 +07:00
parent fed5f277f4
commit 1622dc8fc5
452 changed files with 83999 additions and 66328 deletions
--- a/secrets_template/README.md
+++ b/secrets_template/README.md
@@ -0,0 +1,170 @@
+# Secrets Template — Developer Self-Setup Guide
+
+This folder is the **tracked scaffold** for the local `secrets/` directory described in the [repository root README](../README.md#getting-started). It is safe to commit; it contains **no real credentials**.
+
+Each developer creates and maintains their own `secrets/` folder at the repository root. That folder is **gitignored** and never pushed to Git.
+
+---
+
+## Quick Start
+
+From the repository root (`PILOT_PROJECT/`):
+
+```bash
+# 1. Create your local secrets directory
+mkdir -p secrets/aws_secret
+
+# 2. Copy the environment template
+cp secrets_template/.env.example secrets/aws_secret/.env
+
+# 3. Create plaintext secret files (see tables below)
+touch secrets/gcp_access_token.txt secrets/modal_api_key.txt
+```
+
+Then:
+
+1. Open `secrets/aws_secret/.env` and fill in values for the services you need.
+2. Paste single-value secrets into the corresponding `.txt` files (one secret per file, no quotes, no trailing newline required).
+3. Confirm `secrets/` is ignored: `git check-ignore -v secrets/` should report a match.
+
+Optional — add a local pointer file (also gitignored):
+
+```bash
+cp secrets_template/README.md secrets/README.md
+```
+
+---
+
+## Expected Layout
+
+After setup, your local tree should look like this:
+
+```
+PILOT_PROJECT/
+├── secrets/                          # gitignored — your machine only
+│   ├── gcp_access_token.txt          # Vertex AI / Gemini bearer token
+│   ├── modal_api_key.txt             # MedGemma Modal endpoint API key
+│   ├── aws_secret/
+│   │   └── .env                      # consolidated env vars (Supabase, Exa, AWS, Modal SDK, …)
+│   ├── key                           # optional — SSH private key (infra)
+│   ├── key.pub                       # optional — SSH public key
+│   ├── init_ssh.sh                   # optional — SSH setup helper
+│   └── server_vkist.sh               # optional — server bootstrap script
+└── secrets_template/                 # tracked — templates & docs only
+    ├── README.md                     # this file
+    ├── .env.example                  # copy → secrets/aws_secret/.env
+    └── gemini_api_key.txt            # production reference (not a local key file)
+```
+
+Only create the files you need for the components you are working on. Missing secrets for unused services will not block unrelated work.
+
+---
+
+## Secret Inventory
+
+### Plaintext files (`secrets/` root)
+
+| File | Env var | Used by | How to obtain |
+|------|---------|---------|---------------|
+| `gcp_access_token.txt` | `GCP_ACCESS_TOKEN` | Backend Cloud LLM Gateway (Vertex AI / Gemini) | `gcloud auth print-access-token` (short-lived; refresh as needed) |
+| `modal_api_key.txt` | `MEDGEMMA_API_KEY` | Backend + Modal MedGemma endpoint auth | Project lead or Modal dashboard → API keys |
+
+**Override:** point to a file outside `secrets/` with `{NAME}_FILE`, e.g. `GCP_ACCESS_TOKEN_FILE=/path/to/token.txt`. The loader checks `*_FILE` first, then `secrets/<filename>`.
+
+Reference: `workspace/sprint_1_2/CODEBASE/backend/implementation/config.py`
+
+### Environment bundle (`secrets/aws_secret/.env`)
+
+Copy from `secrets_template/.env.example`. This file is auto-loaded by agent-tool smoke tests and the knowledge ingestion pipeline when present.
+
+| Variable | Required for | Notes |
+|----------|--------------|-------|
+| `SUPABASE_URL` | Knowledge RAG, agent `supabase_query` | Supabase project URL |
+| `SUPABASE_SERVICE_ROLE_KEY` | Backend writes / ingestion upload | Dashboard → Project Settings → API |
+| `SUPABASE_PUBLISHABLE_KEY` / `SUPABASE_ANON_KEY` | Client-side or read-only flows | Same dashboard section |
+| `SUPABASE_DB_URL` | Ingestion pipeline (PostgreSQL path) | Dashboard → Database → Connection string |
+| `EXA_API_KEY` | Agent `exa_search` tool | [Exa dashboard](https://docs.exa.ai/) |
+| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION` | S3 uploads, corpus storage | IAM user or role credentials |
+| `MODAL_KEY`, `MODAL_SECRET` | Modal SDK CLI / deploys | Modal account settings |
+| `HF_TOKEN` | Hugging Face model downloads | huggingface.co → Settings → Access Tokens |
+| `FIGMA_ACCESS_TOKEN` | Design automation (optional) | Figma → Personal access tokens |
+
+Non-secret configuration (endpoints, ports, CORS) lives in `workspace/sprint_1_2/CODEBASE/.env.example` — copy that to `CODEBASE/.env` separately; do **not** commit either `.env` file.
+
+Reference paths:
+
+- `workspace/sprint_1_2/CODEBASE/knowledge/implementation/ingestion/config.py` → `secrets/aws_secret/.env`
+- `workspace/sprint_1_2/CODEBASE/backend/tests/smoke_agent_tools_server.py` → same path
+- `workspace/sprint_1_2/CODEBASE/ml/tests/agent_tools/README.md` → smoke test auto-load
+
+### Optional infra files
+
+| File | Purpose |
+|------|---------|
+| `key` / `key.pub` | SSH key pair for remote servers or CI |
+| `init_ssh.sh` | Helper to register SSH keys with `ssh-agent` |
+| `server_vkist.sh` | Server-side bootstrap (project-specific) |
+
+---
+
+## Gemini API Key (Production Reference)
+
+`secrets_template/gemini_api_key.txt` documents where the **production** Gemini key lives (GCP Secret Manager: `gemini-api-key` in project `vkist-project`). It is **not** a file you paste a key into for local dev.
+
+For local backend work, use `gcp_access_token.txt` (Vertex AI bearer token) instead.
+
+---
+
+## Verify Your Setup
+
+### Backend secrets (GCP + Modal)
+
+```bash
+cd workspace/sprint_1_2/CODEBASE
+conda activate vkist_ultra   # or your project env
+PYTHONPATH=. python -c "from backend.implementation import config; print('GCP token loaded:', bool(config.GCP_ACCESS_TOKEN))"
+```
+
+### Agent tools smoke (Exa + Supabase, no GCP)
+
+```bash
+cd workspace/sprint_1_2/CODEBASE
+PYTHONPATH=. python backend/tests/smoke_agent_tools_server.py
+# In another terminal:
+cd workspace/sprint_1_2/CODEBASE/ml/tests/agent_tools && npm run smoke:bff
+```
+
+### Knowledge ingestion
+
+See `workspace/sprint_1_2/CODEBASE/knowledge/implementation/ingestion/README.md` — requires `secrets/aws_secret/.env` with Supabase credentials.
+
+---
+
+## Security Rules
+
+1. **Never commit** `secrets/` or any file containing real keys, tokens, or passwords.
+2. The root `.gitignore` already excludes `secrets/`. Before adding new secret paths, confirm they remain ignored.
+3. Do not paste secrets into source code, logs, issues, or PR comments.
+4. Share credentials only through your team's approved secret manager (1Password, Vault, GCP Secret Manager, etc.).
+5. Rotate tokens if they are ever exposed. GCP access tokens are short-lived; treat `.env` and API keys as long-lived secrets.
+
+For agent and CI guardrails, see `AGENT_SKILL/secrets_and_phi_safety/SKILL.md`.
+
+---
+
+## Getting Credentials
+
+If you do not have access to a required service:
+
+1. Ask the project lead for onboarding to the relevant cloud account (GCP, Modal, Supabase, AWS).
+2. Use the tables above to identify which variables or files you need for your sprint task.
+3. Start with the smoke tests in [Verify Your Setup](#verify-your-setup) to confirm only the secrets you need are wired correctly.
+
+---
+
+## Related Documentation
+
+- [Repository README — Secrets Management](../README.md#secrets-management-important)
+- [CODEBASE `.env.example`](../workspace/sprint_1_2/CODEBASE/.env.example) — non-secret runtime configuration
+- [Agent tools smoke tests](../workspace/sprint_1_2/CODEBASE/ml/tests/agent_tools/README.md)
+- [Knowledge ingestion prerequisites](../workspace/sprint_1_2/CODEBASE/knowledge/implementation/ingestion/README.md)