msk_admin/Lumina-MSK
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
test_gitea
Lumina-MSK/session_memory/15_Jul_26.md
DatTT127 1f6b015815 add the session memory
2026-07-15 23:29:26 +07:00

7.5 KiB
Raw Permalink Blame History

Session Knowledge — 2026-07-15

Goal

Debug why Celery batch API results are not displaying on the client UI and fix the full pipeline: server startup, worker task registration, Triton endpoint routing, polling storm, stale result re-submission.

Constraints & Preferences

  • Client should not hammer backend with polls; need request coalescing and caching.
  • Already-completed batch results should not be re-submitted or re-queried on every page view.
  • Worker must target Modal Triton endpoint, not localhost.

Progress

Done

  • Fixed GroupResult.restore() returning None by adding result.save() after apply_async() in submit_celery_batch().
  • Fixed server startup ModuleNotFoundError: No module named 'backend' by correcting CODEBASE_ROOT path math in backend/routers/run_cv_inference.sh (changed SCRIPT_DIR/.. → SCRIPT_DIR/../..).
  • Added celery_app.autodiscover_tasks(["backend.implementation.tasks"]) to backend/services/celery_app.py so worker registers run_cv_chunk.
  • Added export TRITON_ENDPOINT=...modal.run to backend/start_workers.sh so worker targets Modal instead of localhost:8000.
  • Added server-side TTL poll cache (_batch_poll_cache) in backend/services/cv_celery_service.py with default 2000ms TTL to coalesce rapid repeat polls.
  • Increased client poll interval from 1000ms → 2000ms and removed unused pollIntervalMs parameter from getCvAnalyzeResultsForProfileCelery() in frontend/implementation/src/lib/cvAnalyzeApi.ts.
  • Fixed poll endpoint HTTP status semantics: GET /api/test/analyze/batch/celery/{job_id} now returns 202 Accepted for pending, 404 Not Found for unknown, 200 OK for completed/failed. This removes the misleading-200 problem for in-flight jobs.
  • Added backend timing logs in cv_celery_service.py: submission logs chunk/image counts; completion/failure logs duration_ms from submission to terminal state.
  • Added frontend timing logs in getCvAnalyzeResultsForProfileCelery() for completed/failed/unknown outcomes.
  • Migrated frontend config from .env/.env.development to single YAML source of truth at config/frontend.config.yaml, loaded via js-yaml in vite.config.ts using Vite define. Removed old .env files. Added VITE_MODAL_OLLAMA_TARGET to YAML so proxy target is config-driven.
  • Fixed TypeScript errors in vite.config.ts by adding "types": ["node"] and including vite.config.ts in tsconfig.json.
  • Added client-side in-flight coalescing + completed-result cache in getCvAnalyzeResultsForProfileCelery() so repeat page views don’t re-submit identical work.

In Progress

  • Worker needs force-restart to pick up autodiscover_tasks + TRITON_ENDPOINT changes.
  • One submitted job reached completed: 1/2 but never finished — need worker restart + fresh batch test.
  • Client-side ETA estimation for pending batch jobs is planned but not yet implemented.

Blocked

  • Worker process restart — old PID/log still active without the new fixes applied.

Key Decisions

  • Used result.save() because Celery does not auto-persist GroupResult metadata to Redis.
  • Added autodiscover_tasks instead of manual imports because worker is launched via -A backend.services.celery_app and never imports cv_tasks.py otherwise.
  • Set worker TRITON_ENDPOINT in start_workers.sh because worker is separate process that doesn’t run cv_inference_server.py’s os.environ.setdefault().
  • Poll cache TTL set to 2000ms to match 2s client poll interval.
  • Single YAML config replaces .env + .env.development to eliminate precedence confusion.
  • VITE_MODAL_OLLAMA_TARGET lives in YAML so proxy target is editable without touching vite.config.ts code.

Next Steps

  • Force-restart Celery worker (kill -9 stale PID, remove celery.pid, run ./start_workers.sh start).
  • Verify worker log shows backend.implementation.tasks.cv_tasks.run_cv_chunk under [tasks].
  • Submit fresh batch and confirm it transitions pending → completed with actual results.
  • Implement ETA estimation in backend poll response based on observed chunk throughput.
  • Address client behavior that submits new jobs while previous jobs are still pending (7+ concurrent jobs observed in logs).

Critical Context

  • Latest server logs show 7+ concurrent jobs all stuck at pending with completed: 0/2 or 1/2, plus new POST submissions every ~10s — indicating worker is not processing tasks.
  • Worker log still shows stale startup at 16:22:18 with empty [tasks]; fixes in celery_app.py and start_workers.sh have not been loaded by running worker.
  • One job (13fd014e) reached completed: 1/2, progress: 0.5 and stalled — second chunk likely failed or was never picked up.
  • Modal Triton endpoint: https://dtj-tran--triton-s3-service-unified-triton-server.modal.run.
  • Redis broker/backend: redis://localhost:6379/0.
  • Chunk size default: 4 frames per chunk (CELERY_CHUNK_SIZE).
  • cv_result_cache.py already exists with in-flight coalescing pattern; new batch poll cache follows same design.
  • Frontend not currently in Celery mode (VITE_USE_CV_CELERY=false in YAML).
  • Direct batch route still shows noisy fallback logs when Triton returns 502 on batched inference — fallback works, but should log at info not warning.

Relevant Files

  • /Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/services/cv_celery_service.py: Contains submit_celery_batch() (fixed with result.save()), get_celery_batch_result() (now with 2000ms TTL poll cache + timing logs), and poll cache constants.
  • /Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/services/celery_app.py: Added autodiscover_tasks(["backend.implementation.tasks"]); routes cv_inference.run_chunk to cv-inference queue.
  • /Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/start_workers.sh: Added TRITON_ENDPOINT export so worker targets Modal.
  • /Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/routers/run_cv_inference.sh: Fixed CODEBASE_ROOT path math (SCRIPT_DIR/../..) so PYTHONPATH resolves backend package.
  • /Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/implementation/tasks/cv_tasks.py: Defines run_cv_chunk Celery task.
  • /Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/backend/routers/cv_inference.py: Contains /api/test/analyze/batch/celery endpoints. Poll endpoint now returns 202/404/200.
  • /Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/src/lib/cvAnalyzeApi.ts: Poll interval 2000ms; client-side cache + in-flight coalescing; timing logs.
  • /Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/src/hooks/useSegmentationOverlay.ts: Calls getCvAnalyzeResultsForProfileCelery().
  • /Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/config/frontend.config.yaml: Single source of truth for frontend feature flags and URLs.
  • /Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/vite.config.ts: Loads YAML config and injects as import.meta.env.*.
  • /Users/davestran/Downloads/vkist_internship/PILOT_PROJECT/workspace/sprint_1_2/CODEBASE/frontend/implementation/tsconfig.json: Added "types": ["node"] and included vite.config.ts.