72 lines
7.5 KiB
Markdown
72 lines
7.5 KiB
Markdown
# Session Knowledge — 2026-07-15
|
||
|
||
## Goal
|
||
Debug why Celery batch API results are not displaying on the client UI and fix the full pipeline: server startup, worker task registration, Triton endpoint routing, polling storm, stale result re-submission.
|
||
|
||
## Constraints & Preferences
|
||
- Client should not hammer backend with polls; need request coalescing and caching.
|
||
- Already-completed batch results should not be re-submitted or re-queried on every page view.
|
||
- Worker must target Modal Triton endpoint, not localhost.
|
||
|
||
## Progress
|
||
### Done
|
||
- Fixed `GroupResult.restore()` returning `None` by adding `result.save()` after `apply_async()` in `submit_celery_batch()`.
|
||
- Fixed server startup `ModuleNotFoundError: No module named 'backend'` by correcting `CODEBASE_ROOT` path math in `backend/routers/run_cv_inference.sh` (changed `SCRIPT_DIR/..` → `SCRIPT_DIR/../..`).
|
||
- Added `celery_app.autodiscover_tasks(["backend.implementation.tasks"])` to `backend/services/celery_app.py` so worker registers `run_cv_chunk`.
|
||
- Added `export TRITON_ENDPOINT=...modal.run` to `backend/start_workers.sh` so worker targets Modal instead of `localhost:8000`.
|
||
- Added server-side TTL poll cache (`_batch_poll_cache`) in `backend/services/cv_celery_service.py` with default 2000ms TTL to coalesce rapid repeat polls.
|
||
- Increased client poll interval from 1000ms → 2000ms and removed unused `pollIntervalMs` parameter from `getCvAnalyzeResultsForProfileCelery()` in `frontend/implementation/src/lib/cvAnalyzeApi.ts`.
|
||
- Fixed poll endpoint HTTP status semantics: `GET /api/test/analyze/batch/celery/{job_id}` now returns `202 Accepted` for pending, `404 Not Found` for unknown, `200 OK` for completed/failed. This removes the misleading-200 problem for in-flight jobs.
|
||
- Added backend timing logs in `cv_celery_service.py`: submission logs chunk/image counts; completion/failure logs `duration_ms` from submission to terminal state.
|
||
- Added frontend timing logs in `getCvAnalyzeResultsForProfileCelery()` for completed/failed/unknown outcomes.
|
||
- Migrated frontend config from `.env`/`.env.development` to single YAML source of truth at `config/frontend.config.yaml`, loaded via `js-yaml` in `vite.config.ts` using Vite `define`. Removed old `.env` files. Added `VITE_MODAL_OLLAMA_TARGET` to YAML so proxy target is config-driven.
|
||
- Fixed TypeScript errors in `vite.config.ts` by adding `"types": ["node"]` and including `vite.config.ts` in `tsconfig.json`.
|
||
- Added client-side in-flight coalescing + completed-result cache in `getCvAnalyzeResultsForProfileCelery()` so repeat page views don’t re-submit identical work.
|
||
|
||
### In Progress
|
||
- Worker needs force-restart to pick up `autodiscover_tasks` + `TRITON_ENDPOINT` changes.
|
||
- One submitted job reached `completed: 1/2` but never finished — need worker restart + fresh batch test.
|
||
- Client-side ETA estimation for pending batch jobs is planned but not yet implemented.
|
||
|
||
### Blocked
|
||
- Worker process restart — old PID/log still active without the new fixes applied.
|
||
|
||
## Key Decisions
|
||
- Used `result.save()` because Celery does not auto-persist `GroupResult` metadata to Redis.
|
||
- Added `autodiscover_tasks` instead of manual imports because worker is launched via `-A backend.services.celery_app` and never imports `cv_tasks.py` otherwise.
|
||
- Set worker `TRITON_ENDPOINT` in `start_workers.sh` because worker is separate process that doesn’t run `cv_inference_server.py`’s `os.environ.setdefault()`.
|
||
- Poll cache TTL set to 2000ms to match 2s client poll interval.
|
||
- Single YAML config replaces `.env` + `.env.development` to eliminate precedence confusion.
|
||
- `VITE_MODAL_OLLAMA_TARGET` lives in YAML so proxy target is editable without touching `vite.config.ts` code.
|
||
|
||
## Next Steps
|
||
- Force-restart Celery worker (`kill -9` stale PID, remove `celery.pid`, run `./start_workers.sh start`).
|
||
- Verify worker log shows `backend.implementation.tasks.cv_tasks.run_cv_chunk` under `[tasks]`.
|
||
- Submit fresh batch and confirm it transitions `pending` → `completed` with actual results.
|
||
- Implement ETA estimation in backend poll response based on observed chunk throughput.
|
||
- Address client behavior that submits new jobs while previous jobs are still `pending` (7+ concurrent jobs observed in logs).
|
||
|
||
## Critical Context
|
||
- Latest server logs show 7+ concurrent jobs all stuck at `pending` with `completed: 0/2` or `1/2`, plus new POST submissions every ~10s — indicating worker is not processing tasks.
|
||
- Worker log still shows stale startup at `16:22:18` with empty `[tasks]`; fixes in `celery_app.py` and `start_workers.sh` have not been loaded by running worker.
|
||
- One job (`13fd014e`) reached `completed: 1/2, progress: 0.5` and stalled — second chunk likely failed or was never picked up.
|
||
- Modal Triton endpoint: `https://dtj-tran--triton-s3-service-unified-triton-server.modal.run`.
|
||
- Redis broker/backend: `redis://localhost:6379/0`.
|
||
- Chunk size default: 4 frames per chunk (`CELERY_CHUNK_SIZE`).
|
||
- `cv_result_cache.py` already exists with in-flight coalescing pattern; new batch poll cache follows same design.
|
||
- Frontend not currently in Celery mode (`VITE_USE_CV_CELERY=false` in YAML).
|
||
- Direct batch route still shows noisy fallback logs when Triton returns 502 on batched inference — fallback works, but should log at `info` not `warning`.
|
||
|
||
## Relevant Files
|
||
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/services/cv_celery_service.py`: Contains `submit_celery_batch()` (fixed with `result.save()`), `get_celery_batch_result()` (now with 2000ms TTL poll cache + timing logs), and poll cache constants.
|
||
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/services/celery_app.py`: Added `autodiscover_tasks(["backend.implementation.tasks"])`; routes `cv_inference.run_chunk` to `cv-inference` queue.
|
||
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/start_workers.sh`: Added `TRITON_ENDPOINT` export so worker targets Modal.
|
||
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/routers/run_cv_inference.sh`: Fixed `CODEBASE_ROOT` path math (`SCRIPT_DIR/../..`) so `PYTHONPATH` resolves `backend` package.
|
||
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/implementation/tasks/cv_tasks.py`: Defines `run_cv_chunk` Celery task.
|
||
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/routers/cv_inference.py`: Contains `/api/test/analyze/batch/celery` endpoints. Poll endpoint now returns 202/404/200.
|
||
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/src/lib/cvAnalyzeApi.ts`: Poll interval 2000ms; client-side cache + in-flight coalescing; timing logs.
|
||
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/src/hooks/useSegmentationOverlay.ts`: Calls `getCvAnalyzeResultsForProfileCelery()`.
|
||
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/config/frontend.config.yaml`: Single source of truth for frontend feature flags and URLs.
|
||
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/vite.config.ts`: Loads YAML config and injects as `import.meta.env.*`.
|
||
- `/Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/tsconfig.json`: Added `"types": ["node"]` and included `vite.config.ts`.
|