diff --git a/session_memory/15_Jul_26.md b/session_memory/15_Jul_26.md new file mode 100644 index 0000000..503ffca --- /dev/null +++ b/session_memory/15_Jul_26.md @@ -0,0 +1,71 @@ +# Session Knowledge — 2026-07-15 + +## Goal +Debug why Celery batch API results are not displaying on the client UI and fix the full pipeline: server startup, worker task registration, Triton endpoint routing, polling storm, stale result re-submission. + +## Constraints & Preferences +- Client should not hammer backend with polls; need request coalescing and caching. +- Already-completed batch results should not be re-submitted or re-queried on every page view. +- Worker must target Modal Triton endpoint, not localhost. + +## Progress +### Done +- Fixed `GroupResult.restore()` returning `None` by adding `result.save()` after `apply_async()` in `submit_celery_batch()`. +- Fixed server startup `ModuleNotFoundError: No module named 'backend'` by correcting `CODEBASE_ROOT` path math in `backend/routers/run_cv_inference.sh` (changed `SCRIPT_DIR/..` → `SCRIPT_DIR/../..`). +- Added `celery_app.autodiscover_tasks(["backend.implementation.tasks"])` to `backend/services/celery_app.py` so worker registers `run_cv_chunk`. +- Added `export TRITON_ENDPOINT=...modal.run` to `backend/start_workers.sh` so worker targets Modal instead of `localhost:8000`. +- Added server-side TTL poll cache (`_batch_poll_cache`) in `backend/services/cv_celery_service.py` with default 2000ms TTL to coalesce rapid repeat polls. +- Increased client poll interval from 1000ms → 2000ms and removed unused `pollIntervalMs` parameter from `getCvAnalyzeResultsForProfileCelery()` in `frontend/implementation/src/lib/cvAnalyzeApi.ts`. +- Fixed poll endpoint HTTP status semantics: `GET /api/test/analyze/batch/celery/{job_id}` now returns `202 Accepted` for pending, `404 Not Found` for unknown, `200 OK` for completed/failed. This removes the misleading-200 problem for in-flight jobs. +- Added backend timing logs in `cv_celery_service.py`: submission logs chunk/image counts; completion/failure logs `duration_ms` from submission to terminal state. +- Added frontend timing logs in `getCvAnalyzeResultsForProfileCelery()` for completed/failed/unknown outcomes. +- Migrated frontend config from `.env`/`.env.development` to single YAML source of truth at `config/frontend.config.yaml`, loaded via `js-yaml` in `vite.config.ts` using Vite `define`. Removed old `.env` files. Added `VITE_MODAL_OLLAMA_TARGET` to YAML so proxy target is config-driven. +- Fixed TypeScript errors in `vite.config.ts` by adding `"types": ["node"]` and including `vite.config.ts` in `tsconfig.json`. +- Added client-side in-flight coalescing + completed-result cache in `getCvAnalyzeResultsForProfileCelery()` so repeat page views don’t re-submit identical work. + +### In Progress +- Worker needs force-restart to pick up `autodiscover_tasks` + `TRITON_ENDPOINT` changes. +- One submitted job reached `completed: 1/2` but never finished — need worker restart + fresh batch test. +- Client-side ETA estimation for pending batch jobs is planned but not yet implemented. + +### Blocked +- Worker process restart — old PID/log still active without the new fixes applied. + +## Key Decisions +- Used `result.save()` because Celery does not auto-persist `GroupResult` metadata to Redis. +- Added `autodiscover_tasks` instead of manual imports because worker is launched via `-A backend.services.celery_app` and never imports `cv_tasks.py` otherwise. +- Set worker `TRITON_ENDPOINT` in `start_workers.sh` because worker is separate process that doesn’t run `cv_inference_server.py`’s `os.environ.setdefault()`. +- Poll cache TTL set to 2000ms to match 2s client poll interval. +- Single YAML config replaces `.env` + `.env.development` to eliminate precedence confusion. +- `VITE_MODAL_OLLAMA_TARGET` lives in YAML so proxy target is editable without touching `vite.config.ts` code. + +## Next Steps +- Force-restart Celery worker (`kill -9` stale PID, remove `celery.pid`, run `./start_workers.sh start`). +- Verify worker log shows `backend.implementation.tasks.cv_tasks.run_cv_chunk` under `[tasks]`. +- Submit fresh batch and confirm it transitions `pending` → `completed` with actual results. +- Implement ETA estimation in backend poll response based on observed chunk throughput. +- Address client behavior that submits new jobs while previous jobs are still `pending` (7+ concurrent jobs observed in logs). + +## Critical Context +- Latest server logs show 7+ concurrent jobs all stuck at `pending` with `completed: 0/2` or `1/2`, plus new POST submissions every ~10s — indicating worker is not processing tasks. +- Worker log still shows stale startup at `16:22:18` with empty `[tasks]`; fixes in `celery_app.py` and `start_workers.sh` have not been loaded by running worker. +- One job (`13fd014e`) reached `completed: 1/2, progress: 0.5` and stalled — second chunk likely failed or was never picked up. +- Modal Triton endpoint: `https://dtj-tran--triton-s3-service-unified-triton-server.modal.run`. +- Redis broker/backend: `redis://localhost:6379/0`. +- Chunk size default: 4 frames per chunk (`CELERY_CHUNK_SIZE`). +- `cv_result_cache.py` already exists with in-flight coalescing pattern; new batch poll cache follows same design. +- Frontend not currently in Celery mode (`VITE_USE_CV_CELERY=false` in YAML). +- Direct batch route still shows noisy fallback logs when Triton returns 502 on batched inference — fallback works, but should log at `info` not `warning`. + +## Relevant Files +- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/services/cv_celery_service.py`: Contains `submit_celery_batch()` (fixed with `result.save()`), `get_celery_batch_result()` (now with 2000ms TTL poll cache + timing logs), and poll cache constants. +- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/services/celery_app.py`: Added `autodiscover_tasks(["backend.implementation.tasks"])`; routes `cv_inference.run_chunk` to `cv-inference` queue. +- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/start_workers.sh`: Added `TRITON_ENDPOINT` export so worker targets Modal. +- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/routers/run_cv_inference.sh`: Fixed `CODEBASE_ROOT` path math (`SCRIPT_DIR/../..`) so `PYTHONPATH` resolves `backend` package. +- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/implementation/tasks/cv_tasks.py`: Defines `run_cv_chunk` Celery task. +- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/routers/cv_inference.py`: Contains `/api/test/analyze/batch/celery` endpoints. Poll endpoint now returns 202/404/200. +- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/src/lib/cvAnalyzeApi.ts`: Poll interval 2000ms; client-side cache + in-flight coalescing; timing logs. +- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/src/hooks/useSegmentationOverlay.ts`: Calls `getCvAnalyzeResultsForProfileCelery()`. +- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/config/frontend.config.yaml`: Single source of truth for frontend feature flags and URLs. +- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/vite.config.ts`: Loads YAML config and injects as `import.meta.env.*`. +- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/tsconfig.json`: Added `"types": ["node"]` and included `vite.config.ts`.