diff --git a/.gitea/workflows/test_secret.yaml b/.gitea/workflows/test_secret.yaml new file mode 100644 index 0000000..cbf465c --- /dev/null +++ b/.gitea/workflows/test_secret.yaml @@ -0,0 +1,15 @@ +name: Test Gitea Secrets +on: [push] + +jobs: + print-secret: + runs-on: ubuntu-latest + steps: + - name: Print Secret + env: + # Map the Gitea secret to an environment variable + MY_TEST_SECRET: ${{ secrets.MODAL_KEY }} + run: | + # Use 'echo' to print the variable + # Gitea will automatically mask the actual value if it matches a secret + echo "The secret value is: $MY_TEST_SECRET" \ No newline at end of file diff --git a/secrets_template/modal_api_key.txt b/secrets_template/modal_api_key.txt new file mode 100644 index 0000000..4b1212c --- /dev/null +++ b/secrets_template/modal_api_key.txt @@ -0,0 +1 @@ +# Replace with actual MedGemma Modal endpoint API key (never commit real secrets) \ No newline at end of file diff --git a/secrets_template/modal_builder_secret.sh b/secrets_template/modal_builder_secret.sh new file mode 100644 index 0000000..f611e0f --- /dev/null +++ b/secrets_template/modal_builder_secret.sh @@ -0,0 +1,3 @@ +modal secret create gitea-runner-secrets \ + MODAL_KEY=your-modal-key \ + MODAL_SECRET=your-modal-secret \ No newline at end of file diff --git a/workspace/sprint_1_2/CODEBASE/deps/spec/CICD_architecture.png b/workspace/sprint_1_2/CODEBASE/deps/spec/CICD_architecture.png new file mode 100644 index 0000000..39d37db Binary files /dev/null and b/workspace/sprint_1_2/CODEBASE/deps/spec/CICD_architecture.png differ diff --git a/workspace/sprint_1_2/CODEBASE/deps/spec/build_server_plan.md b/workspace/sprint_1_2/CODEBASE/deps/spec/build_server_plan.md new file mode 100644 index 0000000..76336d3 --- /dev/null +++ b/workspace/sprint_1_2/CODEBASE/deps/spec/build_server_plan.md @@ -0,0 +1,142 @@ + +* The build was for creating the build pipeline in the CI/CD system of the project +that involve: Lightsail instance as broker, Modal-Serverless as the build-worker, and Gitea as the source repository hosted on the same Lightsail VM. + +```mermaid +sequenceDiagram + autonumber + actor Developer + participant Git as Codebase (Local Git) + participant Gitea as Gitea (on Lightsail VM) + participant Webhook as Webhook Listener (Lightsail VM) + participant Modal as Modal Serverless (VM Sandbox) + + Developer->>Git: git push code + Git->>Gitea: Push commits to remote repo (Lightsail) + Note over Gitea: Gitea registers a new pending job
in the Actions queue + Gitea->>Webhook: HTTP POST Webhook (Push event) + + rect rgb(240, 248, 255) + Note over Webhook: Triggered by Webhook (same Lightsail VM) + Webhook->>Modal: Spawn sandbox container via Modal SDK + end + + rect rgb(245, 245, 245) + Note over Modal: Boot Sandbox VM (<2 seconds) + Modal->>Gitea: ./act_runner register (using registration token) + Modal->>Gitea: ./act_runner daemon --once (asks for work) + Gitea->>Modal: Delivers the queued build job + + Note over Modal: Runner clones codebase,
builds Docker container, & runs test/compile + + Modal->>Gitea: Reports build results (Success/Failure) + end + + Note over Modal: act_runner exits automatically (--once)
Modal Sandbox self-destructs + Gitea->>Developer: Displays build status in UI +``` +--- + +### Phase 1: Gitea Preparation (Lightsail VM) + +This phase configures the Gitea instance running on the Lightsail VM to accept serverless runners. + +- [ ] **Task 1.1: Enable Gitea Actions** + - SSH into the Lightsail VM. + - Open `/etc/gitea/app.ini` on the Lightsail instance. + - Add or update the following block to enable the Actions feature:Ini, TOML + + ``` + [actions] + ENABLED = true + ``` + + - Restart Gitea (`sudo systemctl restart gitea`). +- [ ] **Task 1.2: Retrieve Gitea Runner Registration Token** + - Access the Gitea web UI running on the Lightsail VM (e.g., `http://:3000`) as an administrator. + - Navigate to **Site Administration** > **Actions** > **Runners**. + - Click **Registration Token** and copy this token (you will need it for the Modal Sandbox registration). + +> **Note:** All Gitea administration tasks in this plan assume you are managing the Gitea instance hosted on the Lightsail VM, either via SSH or its web UI. + +### Phase 2: Create the Webhook Listener (Lightsail VM) + +The listener runs on the same Lightsail VM as Gitea and catches the git push notification from Gitea to call Modal. + +- [ ] **Task 2.1: Write the Express/Node.js Webhook Server** + - Create a simple script on the Lightsail VM that listens on a port (e.g., `3333`) and exposes a `/deploy` route. + - The route must verify Gitea's Webhook Secret header to ensure requests are authentic. + - When a valid POST request arrives, use the `child_process` module to run a local command: `modal run deploy_runner.py`. +- [ ] **Task 2.2: Configure Webhook in Gitea** + - In your Gitea repository settings (on the Lightsail-hosted Gitea), create a Gitea Webhook pointing to `http://127.0.0.1:3333/deploy` (or your Lightsail public domain). + - Set the trigger event to **Push Events** and assign a strong webhook secret token. + +### Phase 3: Build the Modal Serverless Runner (Modal Cloud) + +This python script defines the sandbox container and coordinates the registration, execution, and termination with the Gitea instance on Lightsail. + +- [ ] **Task 3.1: Define the Modal Image (`deploy_runner.py`)** + - Configure a Modal image containing: + - `git` + - `docker` (or `podman`/`kaniko` if you are building Docker images inside the sandbox) + - The Gitea `act_runner` binary (downloaded directly from Gitea's release page). + - **Build dependencies for the project:** + - Node.js 18+ and npm (for frontend build) + - Python 3.10+ and pip (for any Python-based build steps) + - Build essentials (`build-essential`, `python3-dev`, `pkg-config`) for native module compilation + - `ca-certificates` and `curl` (for downloading dependencies and act_runner) +- [ ] **Task 3.2: Implement the Modal Function Logic** + - Set up a Modal function (e.g., `@app.function()`) that executes the following shell sequence inside the container when invoked: + 1. **Register the runner:**Bash + + ``` + ./act_runner register --no-interactive --instance --token --name modal-sandbox-runner --labels "ubuntu-latest:docker://node:18-bookworm" + ``` + + 2. **Run once and block until job completion:**Bash + + ``` + ./act_runner daemon --once + ``` + + > **Note:** `` must be the publicly reachable address of the Gitea instance on your Lightsail VM (e.g., `http://:3000`), since the Modal sandbox runs outside of your Lightsail network. +- [ ] **Task 3.3: Deploy and Test the Modal App** + - Authenticate your Lightsail server with Modal using `modal setup` (run this on the Lightsail VM). + - Run `modal run deploy_runner.py` manually to verify the sandbox boots, registers in Gitea, checks for a job, and exits gracefully. + +### Phase 4: Configure the Repository CI/CD Workflow + +Define what the runner should actually do when it receives the codebase from the Lightsail-hosted Gitea. + +- [ ] **Task 4.1: Create Gitea Action Workflow File** + - In your repository (hosted on the Lightsail Gitea instance), create a directory structure: `.gitea/workflows/demo.yaml`. + - Define a basic test pipeline:YAML + + # + + ``` + name: Gitea CI/CD + on: [push] + jobs: + test-build: + runs-on: ubuntu-latest + steps: + - name: Check out repository + uses: actions/checkout@v4 + - name: Build Application + run: | + echo "Compiling and testing..." + # Add your test or build commands here + ``` + + +### Phase 5: End-to-End Integration Test + +- [ ] **Task 5.1: Push a Commit** + - Run `git push` from your local machine to the Gitea instance on Lightsail. + - Verify the sequence: + 1. Gitea (on Lightsail) displays a pending job badge. + 2. Webhook triggers on Lightsail (same VM as Gitea). + 3. Modal sandbox boots up. + 4. The sandbox logs into Gitea (via public Lightsail URL), processes the workflow, and turns off. + 5. Gitea UI (on Lightsail) updates to green (Success) or red (Failure). \ No newline at end of file diff --git a/workspace/sprint_1_2/CODEBASE/deps/spec/secrets_management_guideline.md b/workspace/sprint_1_2/CODEBASE/deps/spec/secrets_management_guideline.md new file mode 100644 index 0000000..ff5c612 --- /dev/null +++ b/workspace/sprint_1_2/CODEBASE/deps/spec/secrets_management_guideline.md @@ -0,0 +1,211 @@ +# Secrets Management Guideline + +This document defines how to handle secrets, environment variables, and sensitive configuration across the PILOT project CI/CD pipeline and infrastructure. + +## Principles + +1. **Never commit secrets to version control** — Use `.gitignore` and secret stores +2. **Least-privilege access** — Each component gets only the secrets it needs +3. **Environment isolation** — Separate secrets for local, staging, and production +4. **Auditability** — Track secret creation, rotation, and access + +## Secret Storage Locations + +### 1. Gitea Repository Secrets (CI/CD Workflows) + +Store secrets used by GitHub Actions / Gitea Actions workflows in Gitea's built-in secrets manager. + +**Access path:** Gitea UI → Repository Settings → Secrets → Actions + +```bash +# Repository-level secrets (per project) +AWS_ACCESS_KEY_ID +AWS_SECRET_ACCESS_KEY +MODAL_TOKEN_ID +MODAL_TOKEN_SECRET +POSTGRES_PASSWORD +GITEA_WEBHOOK_SECRET +CADDY_API_TOKEN +``` + +**Access in workflow:** +```yaml +env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }} + MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }} +``` + +### 2. Modal Secrets (Serverless Workers) + +Store Modal-specific configuration in Modal's secrets system. + +**Create secret:** +```bash +modal secret create gitea-runner-secrets \ + MODAL_KEY=your-modal-key \ + MODAL_SECRET=your-modal-secret +``` + +**Use in deploy_runner.py:** +```python +import os +import modal + +app = modal.App("gitea-runner") +secret = modal.Secret.from_name("gitea-runner-secrets") + +@app.function(secret=secret) +def run_runner(): + modal_key = os.environ["MODAL_KEY"] + modal_secret = os.environ["MODAL_SECRET"] +``` + +### 3. AWS Secrets Manager (Production Infrastructure) + +Store AWS credentials and infrastructure secrets in AWS Secrets Manager. + +**Create secret:** +```bash +aws secretsmanager create-secret \ + --name gitea/prod/aws-credentials \ + --secret-string '{"access_key":"...","secret_key":"..."}' +``` + +**Retrieve secret:** +```bash +aws secretsmanager get-secret-value \ + --secret-id gitea/prod/aws-credentials \ + --query SecretString --output text +``` + +**Use in Python (boto3):** +```python +import boto3 +import json + +client = boto3.client('secretsmanager') +secret = client.get_secret_value(SecretId='gitea/prod/aws-credentials') +credentials = json.loads(secret['SecretString']) +``` + +### 4. Local Development (.env) + +Use `.env` files for local development with strict `.gitignore` rules. + +```bash +# .env (NEVER commit this file) +AWS_ACCESS_KEY_ID=local-dev-key +AWS_SECRET_ACCESS_KEY=local-dev-secret +AWS_DEFAULT_REGION=us-east-1 +HF_TOKEN=hf-local-token +MODAL_KEY=local-modal-key +MODAL_SECRET=local-modal-secret +FIGMA_ACCESS_TOKEN=local-figma-token +SUPABASE_URL=http://localhost:54321 +SUPABASE_PUBLISHABLE_KEY=local-supabase-key +SUPABASE_DB_URL=postgresql://postgres:postgres@localhost:54322/postgres +SUPABASE_SERVICE_ROLE_KEY=local-supabase-service-key +EXA_API_KEY=local-exa-key +GITHUB_PAT=local-github-pat +GITHUB_GITEA_CLIENT_SECRET=local-gitea-client-secret +POSTGRES_PASSWORD=local-password +GITEA_URL=http://localhost:3000 +GITEA_WEBHOOK_SECRET=local-webhook-secret +``` + +```bash +# .gitignore +.env +.env.local +.env.*.local +secrets/ +*.pem +*.key +``` + +## Gitea Workflow Secrets Template + +Use this template to reference secrets in `.gitea/workflows/*.yml` files: + +```yaml +env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }} + HF_TOKEN: ${{ secrets.HF_TOKEN }} + MODAL_KEY: ${{ secrets.MODAL_KEY }} + MODAL_SECRET: ${{ secrets.MODAL_SECRET }} + FIGMA_ACCESS_TOKEN: ${{ secrets.FIGMA_ACCESS_TOKEN }} + SUPABASE_URL: ${{ secrets.SUPABASE_URL }} + SUPABASE_PUBLISHABLE_KEY: ${{ secrets.SUPABASE_PUBLISHABLE_KEY }} + SUPABASE_DB_URL: ${{ secrets.SUPABASE_DB_URL }} + SUPABASE_SERVICE_ROLE_KEY: ${{ secrets.SUPABASE_SERVICE_ROLE_KEY }} + EXA_API_KEY: ${{ secrets.EXA_API_KEY }} + GITHUB_PAT: ${{ secrets.GITHUB_PAT }} + GITHUB_GITEA_CLIENT_SECRET: ${{ secrets.GITHUB_GITEA_CLIENT_SECRET }} +``` + +All secrets must be registered in Gitea under **Repository Settings → Secrets → Actions** before the workflow runs. + +## Secret Reference Matrix + +| Secret | Local Dev | Gitea CI/CD | Modal Worker | AWS Infra | +|--------|-----------|-------------|--------------|-----------| +| AWS_ACCESS_KEY_ID | .env | Gitea Secret | — | Secrets Manager | +| AWS_SECRET_ACCESS_KEY | .env | Gitea Secret | — | Secrets Manager | +| AWS_DEFAULT_REGION | .env | Gitea Secret | — | — | +| HF_TOKEN | .env | Gitea Secret | — | — | +| MODAL_KEY | .env | Gitea Secret | Modal Secret | — | +| MODAL_SECRET | .env | Gitea Secret | Modal Secret | — | +| FIGMA_ACCESS_TOKEN | .env | Gitea Secret | — | — | +| SUPABASE_URL | .env | Gitea Secret | — | — | +| SUPABASE_PUBLISHABLE_KEY | .env | Gitea Secret | — | — | +| SUPABASE_DB_URL | .env | Gitea Secret | — | Secrets Manager | +| SUPABASE_SERVICE_ROLE_KEY | .env | Gitea Secret | — | — | +| EXA_API_KEY | .env | Gitea Secret | — | — | +| GITHUB_PAT | .env | Gitea Secret | — | — | +| GITHUB_GITEA_CLIENT_SECRET | .env | Gitea Secret | — | — | +| POSTGRES_PASSWORD | .env | Gitea Secret | — | Secrets Manager | +| GITEA_WEBHOOK_SECRET | .env | Gitea Secret | — | — | +| CADDY_API_TOKEN | .env | Gitea Secret | — | — | + +## Security Best Practices + +1. **Rotate secrets regularly** — Especially AWS keys and tokens (every 90 days) +2. **Use IAM roles** — Prefer IAM roles over hardcoded AWS keys in production +3. **Encrypt at rest** — Use AWS KMS for Secrets Manager encryption +4. **Audit access** — Enable CloudTrail for AWS Secrets Manager, Gitea audit logs +5. **Pre-commit hooks** — Use `git-secrets` or `pre-commit` to block secret commits +6. **Separate environments** — Use different secret namespaces for dev/staging/prod +7. **Minimal scope** — AWS keys should have only required permissions + +## Pre-commit Hook Setup (Optional) + +Install `git-secrets` to prevent committing secrets: + +```bash +# Install git-secrets +git secrets --install + +# Register common patterns +git secrets --register-aws +git secrets --add 'GITEA_TOKEN' +git secrets --add 'MODAL_TOKEN' +``` + +## Emergency: Secret Exposure + +If a secret is accidentally committed: + +1. **Rotate immediately** — Generate a new secret value +2. **Revoke the old secret** — Invalidate the exposed credential +3. **Audit access logs** — Check for unauthorized usage +4. **Update all references** — Deploy the new secret to all environments +5. **Force push cleanup** — Use `git filter-branch` or `BFG` to remove from history + +## Related Files + +- `build_server_plan.md` — CI/CD pipeline architecture +- `CICD_architecture.png` — Architecture diagram