msk_admin/Lumina-MSK
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update the cd-server
Some checks failed
Backend EC - Modal Deployment / notify-deploy (push) Has been cancelled
Details
Backend EC - Modal Deployment / build-and-push (push) Has been cancelled
Details
Deploy CD Server / deploy (push) Failing after 16s
Details

Browse Source
This commit is contained in:
DatTT127
2026-07-19 17:48:27 +07:00
parent 6caa654389
commit 15437a5f18
13 changed files with 1012 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

77
.gitea/workflows/deploy-cd-server.yaml Normal file
View File

@@ -0,0 +1,77 @@
name: Deploy CD Server
on:
push:
# branches:
# - main
paths:
- 'workspace/sprint_1_2/CODEBASE/deps/implementation/CD_server/**'
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Deploy to Lightsail VM
uses: appleboy/ssh-action@v1.0.3
with:
host: ${{ secrets.LIGHTSAIL_HOST }}
username: ${{ secrets.LIGHTSAIL_USERNAME }}
key: ${{ secrets.LIGHTSAIL_SSH_KEY }}
port: ${{ vars.LIGHTSAIL_PORT }}
script: |
set -euo pipefail
DEPLOY_DIR="/opt/cd-server"
SERVICE_NAME="cd-server"
echo "[deploy] Creating deploy user if missing"
if ! id -u deploy &>/dev/null; then
sudo useradd -m -s /bin/bash deploy
fi
echo "[deploy] Adding deploy user to docker group"
sudo usermod -aG docker deploy || true
echo "[deploy] Creating $DEPLOY_DIR"
sudo mkdir -p "$DEPLOY_DIR"
sudo chown deploy:docker "$DEPLOY_DIR"
echo "[deploy] Syncing code to $DEPLOY_DIR"
sudo -u deploy rsync -avz --delete \
--exclude='.git' \
--exclude='__pycache__' \
--exclude='*.pyc' \
--exclude='.env' \
./ "$DEPLOY_DIR/"
echo "[deploy] Building CD server Docker image"
sudo -u deploy docker compose -f "$DEPLOY_DIR/docker-compose.yml" build --no-cache
echo "[deploy] Creating .env file"
sudo -u deploy bash -c "echo 'WEBHOOK_SECRET=${{ secrets.WEBHOOK_SECRET }}' >> $DEPLOY_DIR/.env"
sudo -u deploy bash -c "echo 'AWS_ECR_REGION=${{ secrets.AWS_ECR_REGION }}' >> $DEPLOY_DIR/.env"
sudo chmod 600 "$DEPLOY_DIR/.env"
echo "[deploy] Starting CD server container"
sudo -u deploy docker compose -f "$DEPLOY_DIR/docker-compose.yml" up -d --force-recreate
echo "[deploy] Waiting for service to be healthy"
for i in {1..30}; do
if curl -s http://127.0.0.1:3333/health > /dev/null; then
echo "[deploy] Service is healthy"
break
fi
if [ "$i" -eq 30 ]; then
echo "[deploy] Service failed health check"
sudo docker logs cd-server --no-pager || true
exit 1
fi
sleep 2
done
echo "[deploy] Deployment completed successfully"

2
workspace/sprint_1_2/CODEBASE/deps/implementation/CD_server/.env.example Normal file
View File

@@ -0,0 +1,2 @@
WEBHOOK_SECRET=your-webhook-secret-here
AWS_ECR_REGION=us-east-1

4
workspace/sprint_1_2/CODEBASE/deps/implementation/CD_server/.gitignore vendored Normal file
View File

@@ -0,0 +1,4 @@
.env
__pycache__/
*.pyc
*.pyo

17
workspace/sprint_1_2/CODEBASE/deps/implementation/CD_server/Dockerfile Normal file
View File

@@ -0,0 +1,17 @@
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py config.py docker_ops.py schemas.py config.yaml ./
RUN groupadd -g 998 docker || true
RUN useradd -m -u 998 -g docker appuser
USER appuser
EXPOSE 3333
CMD ["python", "main.py"]

56
workspace/sprint_1_2/CODEBASE/deps/implementation/CD_server/config.py Normal file
View File

@@ -0,0 +1,56 @@
import os
from dataclasses import dataclass, field
import yaml
@dataclass
class Timeouts:
request_read: int = 60
ecr_login: int = 30
docker_pull: int = 600
docker_stop: int = 30
docker_rm: int = 10
docker_run: int = 60
@dataclass
class ServerConfig:
listen_address: str = "127.0.0.1"
listen_port: int = 3333
webhook_secret: str = ""
aws_region: str = ""
service_port_map: dict[str, dict[str, int]] = field(default_factory=dict)
timeouts: Timeouts = field(default_factory=Timeouts)
@classmethod
def from_env(cls) -> "ServerConfig":
secret = os.environ.get("WEBHOOK_SECRET", "")
region = os.environ.get("AWS_ECR_REGION", "")
if not secret:
raise RuntimeError("WEBHOOK_SECRET environment variable must be set")
if not region:
raise RuntimeError("AWS_ECR_REGION environment variable must be set")
return cls(webhook_secret=secret, aws_region=region)
@classmethod
def from_file(cls, path: str) -> "ServerConfig":
with open(path, "r", encoding="utf-8") as f:
data = yaml.safe_load(f) or {}
timeouts_data = data.pop("timeouts", {})
timeouts = Timeouts(**timeouts_data)
# Only non-sensitive fields come from config file
config = cls(
listen_address=data.get("listen_address", "127.0.0.1"),
listen_port=data.get("listen_port", 3333),
service_port_map=data.get("service_port_map", {}),
timeouts=timeouts,
)
# Secrets always from environment — never from config file
config.webhook_secret = os.environ.get("WEBHOOK_SECRET", "")
config.aws_region = os.environ.get("AWS_ECR_REGION", "")
if not config.webhook_secret:
raise RuntimeError("WEBHOOK_SECRET must be set in environment")
if not config.aws_region:
raise RuntimeError("AWS_ECR_REGION must be set in environment")
return config

16
workspace/sprint_1_2/CODEBASE/deps/implementation/CD_server/config.yaml Normal file
View File

@@ -0,0 +1,16 @@
listen_address: "127.0.0.1"
listen_port: 3333
service_port_map:
cv-inference-server:
host: 8001
container: 8001
api-gateway:
host: 8080
container: 8080
timeouts:
request_read: 60
ecr_login: 30
docker_pull: 600
docker_stop: 30
docker_rm: 10
docker_run: 60

13
workspace/sprint_1_2/CODEBASE/deps/implementation/CD_server/docker-compose.yml Normal file
View File

@@ -0,0 +1,13 @@
services:
cd-server:
build: .
container_name: cd-server
restart: unless-stopped
network_mode: host
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./config.yaml:/app/config.yaml:ro
env_file:
- .env
environment:
- CD_SERVER_CONFIG=/app/config.yaml

123
workspace/sprint_1_2/CODEBASE/deps/implementation/CD_server/docker_ops.py Normal file
View File

@@ -0,0 +1,123 @@
import json
import logging
import os
import re
import subprocess
from typing import Any
import os
from config import ServerConfig
logger = logging.getLogger(__name__)
class DockerError(Exception):
def __init__(self, message: str, operation: str):
super().__init__(message)
self.operation = operation
self.message = message
def run_command(cmd: list[str], timeout: int, env: dict[str, str] | None = None) -> str:
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=timeout,
env={**os.environ, **(env or {})},
)
if result.returncode != 0:
stderr = result.stderr.strip()
raise DockerError(stderr, " ".join(cmd))
return result.stdout.strip()
except subprocess.TimeoutExpired:
raise DockerError(f"Command timed out after {timeout}s", " ".join(cmd))
def ecr_login(region: str, registry: str) -> None:
if registry == "public.ecr.aws":
logger.info("Public ECR registry detected, skipping ECR login")
return
account_id = registry.split(".")[0]
ecr_url = f"{account_id}.dkr.ecr.{region}.amazonaws.com"
login_cmd = [
"sh", "-c",
f"aws ecr get-login-password --region {region} | "
f"docker login --username AWS --password-stdin {ecr_url}"
]
run_command(login_cmd, timeout=30)
logger.info("ECR login successful for %s", ecr_url)
def parse_image(image: str) -> tuple[str, str]:
parts = image.rsplit(":", 1)
if len(parts) != 2:
raise DockerError(f"Invalid image format: {image}", "parse_image")
return parts[0], parts[1]
def is_public_ecr(image: str) -> bool:
return image.startswith("public.ecr.aws/")
def pull_image(image: str, timeout: int) -> None:
run_command(["docker", "pull", image], timeout=timeout)
logger.info("Image pulled: %s", image)
def find_running_container(service_name: str) -> str | None:
output = run_command(
["docker", "ps", "--filter", f"name={service_name}", "--filter", "status=running", "--format", "{{.ID}}"],
timeout=10,
)
return output if output else None
def stop_container(service_name: str, timeout: int) -> None:
container_id = find_running_container(service_name)
if not container_id:
logger.info("No running container found for %s, skipping teardown", service_name)
return
logger.info("Stopping container %s (%s)", service_name, container_id)
try:
run_command(["docker", "stop", service_name], timeout=timeout)
except DockerError:
logger.warning("Graceful stop failed for %s, force killing", service_name)
run_command(["docker", "kill", service_name], timeout=10)
logger.info("Removing container %s", service_name)
run_command(["docker", "rm", service_name], timeout=10)
def create_container(
image: str,
service_name: str,
env_vars: dict[str, str],
host_port: int,
container_port: int,
timeout: int,
) -> None:
cmd = ["docker", "run", "-d", "--name", service_name, "--restart", "unless-stopped"]
for key, val in env_vars.items():
cmd.extend(["-e", f"{key}={val}"])
cmd.extend(["-p", f"{host_port}:{container_port}", image])
run_command(cmd, timeout=timeout)
logger.info("Container %s started on port %d", service_name, host_port)
def deploy_service(payload: dict[str, Any], config: ServerConfig) -> None:
service_name = payload["service"]
image = payload["image"]
env_vars = payload.get("env") or {}
timeouts = config.timeouts
if service_name not in config.service_port_map:
raise DockerError(f"Service '{service_name}' not found in port map", "port_map_lookup")
port_map = config.service_port_map[service_name]
host_port = port_map["host"]
container_port = port_map["container"]
registry = image.split("/")[0] if "/" in image else ""
ecr_login(config.aws_region, registry)
pull_image(image, timeouts.docker_pull)
stop_container(service_name, timeouts.docker_stop)
create_container(image, service_name, env_vars, host_port, container_port, timeouts.docker_run)

136
workspace/sprint_1_2/CODEBASE/deps/implementation/CD_server/main.py Normal file
View File

@@ -0,0 +1,136 @@
#!/usr/bin/env python3
"""
CD Server (Deployment Server)
FastAPI-based webhook listener for automated Docker container deployments.
"""
import json
import logging
import os
import sys
import time
from pathlib import Path
from typing import Any
import uvicorn
from fastapi import FastAPI, Request, Response
from fastapi.responses import JSONResponse
from pydantic import ValidationError
import config
import docker_ops
from schemas import DeployPayload
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
logging.basicConfig(
level=logging.INFO,
format="%(message)s",
)
logger = logging.getLogger("cd_server")
logger.setLevel(logging.INFO)
app = FastAPI(title="CD Server", version="1.0.0")
def log_json(level: str, **kwargs: Any) -> None:
entry = {
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"level": level,
**kwargs,
}
log_line = json.dumps(entry, ensure_ascii=False)
if level == "error":
logger.error(log_line)
elif level == "warning":
logger.warning(log_line)
else:
logger.info(log_line)
def load_config() -> config.ServerConfig:
config_path = Path(__file__).parent / "config.yaml"
if config_path.exists():
return config.ServerConfig.from_file(str(config_path))
return config.ServerConfig.from_env()
@app.get("/health")
async def health() -> JSONResponse:
return JSONResponse({"status": "healthy"})
@app.post("/deploy")
async def deploy(request: Request) -> Response:
client_ip = request.client.host if request.client else "unknown"
body_bytes = await request.body()
if len(body_bytes) > 64 * 1024:
log_json("warning", source_ip=client_ip, path="/deploy", method="POST", outcome="body_too_large", status_code=400, message="Request body exceeds 64 KB limit")
return JSONResponse({"error": "bad_request", "message": "Request body too large"}, status_code=400)
cfg = request.app.state.config
signature = request.headers.get("X-Gitea-Signature", "")
if not verify_signature(body_bytes, signature, cfg.webhook_secret):
log_json("warning", source_ip=client_ip, path="/deploy", method="POST", outcome="signature_failed", status_code=401, message="Invalid signature")
return JSONResponse({"error": "unauthorized", "message": "Invalid signature"}, status_code=401)
try:
payload = DeployPayload.model_validate_json(body_bytes)
except ValidationError as e:
log_json("warning", source_ip=client_ip, path="/deploy", method="POST", outcome="validation_failed", status_code=400, message=str(e))
return JSONResponse({"error": "bad_request", "message": "Invalid payload"}, status_code=400)
service = payload.service
image = payload.image
env = payload.env or {}
log_json("info", source_ip=client_ip, path="/deploy", method="POST", outcome="deployment_initiated", status_code=202, message="Deployment initiated", service=service, image=image)
try:
docker_ops.deploy_service(payload.model_dump(), cfg)
except docker_ops.DockerError as e:
log_json("error", source_ip=client_ip, path="/deploy", method="POST", outcome="deployment_failed", status_code=500, message=e.message, service=service, image=image)
return JSONResponse({"error": "deployment_failed", "message": e.message}, status_code=500)
except Exception as e:
log_json("error", source_ip=client_ip, path="/deploy", method="POST", outcome="deployment_failed", status_code=500, message=str(e), service=service, image=image)
return JSONResponse({"error": "deployment_failed", "message": str(e)}, status_code=500)
log_json("info", source_ip=client_ip, path="/deploy", method="POST", outcome="deployment_success", status_code=200, message="Deployment completed", service=service, image=image)
return JSONResponse({"status": "success", "service": service})
def verify_signature(body: bytes, signature: str, secret: str) -> bool:
if not secret:
logger.warning("WEBHOOK_SECRET is empty")
return False
import hmac
import hashlib
expected = "sha256=" + hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, signature)
@app.middleware("http")
async def limit_request_size(request: Request, call_next):
content_length = request.headers.get("content-length")
if content_length and int(content_length) > 64 * 1024:
return JSONResponse({"error": "bad_request", "message": "Request body too large"}, status_code=400)
return await call_next(request)
if __name__ == "__main__":
try:
cfg = load_config()
except RuntimeError as e:
logger.error("Configuration error: %s", e)
sys.exit(1)
app.state.config = cfg
uvicorn.run(
app,
host=cfg.listen_address,
port=cfg.listen_port,
log_config=None,
access_log=False,
)

3
workspace/sprint_1_2/CODEBASE/deps/implementation/CD_server/requirements.txt Normal file
View File

@@ -0,0 +1,3 @@
fastapi==0.135.1
uvicorn==0.41.0
pydantic==2.13.4

41
workspace/sprint_1_2/CODEBASE/deps/implementation/CD_server/schemas.py Normal file
View File

@@ -0,0 +1,41 @@
from pydantic import BaseModel, Field, field_validator, ConfigDict
import re
class DeployPayload(BaseModel):
model_config = ConfigDict(extra="forbid")
service: str = Field(..., min_length=1, max_length=63)
image: str = Field(..., min_length=1)
env: dict[str, str] | None = None
@field_validator("service")
@classmethod
def validate_service(cls, v: str) -> str:
if not re.match(r"^[a-zA-Z0-9_-]{1,63}$", v):
raise ValueError("service must be 1-63 chars: alphanumeric, hyphens, underscores only")
return v
@field_validator("image")
@classmethod
def validate_image(cls, v: str) -> str:
if ":" not in v:
raise ValueError("image must contain a tag separated by ':'")
parts = v.rsplit(":", 1)
if not parts[0] or not parts[1]:
raise ValueError("image must have non-empty registry/repository and tag")
if not re.match(r"^[a-zA-Z0-9._/-]+$", parts[0]):
raise ValueError("image repository contains invalid characters")
if not re.match(r"^[a-zA-Z0-9._-]+$", parts[1]):
raise ValueError("image tag contains invalid characters")
return v
@field_validator("env")
@classmethod
def validate_env(cls, v: dict[str, str] | None) -> dict[str, str] | None:
if v is None:
return v
for key, val in v.items():
if not isinstance(key, str) or not isinstance(val, str):
raise ValueError("env keys and values must be strings")
return v

18
workspace/sprint_1_2/CODEBASE/deps/implementation/CD_server/systemd/cd-server.service Normal file
View File

@@ -0,0 +1,18 @@
[Unit]
Description=CD Server (Deployment Server)
After=docker.service network.target
Requires=docker.service
[Service]
Type=simple
User=deploy
Group=docker
WorkingDirectory=/opt/cd-server
ExecStart=/usr/bin/docker compose up
ExecStop=/usr/bin/docker compose down
Restart=always
RestartSec=5
EnvironmentFile=/opt/cd-server/.env
[Install]
WantedBy=multi-user.target

506
workspace/sprint_1_2/CODEBASE/deps/spec/CD_server/specification.md Normal file
View File

@@ -0,0 +1,506 @@
# CD Server (Deployment Server) — Detailed Technical Specification
## 1. System Overview & Role
The **CD Server (Deployment Server)** is a lightweight, secure background service (daemon) that runs directly on the target host Virtual Machine (VM). Its sole responsibility is to automate the **continuous deployment (CD)** phase of the pipeline.
Instead of opening full SSH access to the repository, this application acts as an isolated agent that listens for secure HTTP POST webhook requests from the CI pipeline (e.g., Gitea Actions), pulls freshly built images from AWS Elastic Container Registry (ECR), and restarts local Docker containers with updated environment configurations.
### 1.1 Success Criteria
- The server must reject all unauthorized requests with a 401 status before parsing any payload.
- The server must not terminate the currently running container if any step in the deployment pipeline (pull, stop, create, run) fails.
- All requests must be logged with timestamp, source IP, path, and outcome status code.
- The server must listen strictly on `127.0.0.1:3333` and must not be exposed externally.
---
## 2. Integration & Architecture
The CD server integrates into the pipeline environment through three distinct boundary interfaces:
### 2.1 Inbound (CI Pipeline → Caddy → CD Server)
1. The CI runner sends a secure HTTPS POST request over the internet to `https://deploy.lumina-msk.io.vn/deploy`.
2. **Caddy Server** acts as the front-edge reverse proxy. It terminates SSL/TLS, filters out traffic not bound for `/deploy`, and forwards the raw traffic internally over the loopback network (`127.0.0.1:3333`) to this CD Server application.
### 2.2 Local Environment (CD Server → Host OS Docker Daemon)
The CD Server executes local system commands or interacts with the Unix socket `/var/run/docker.sock` to control the host machine's Docker engine.
### 2.3 Outbound (Host VM → AWS ECR)
The host machine interacts with AWS ECR to pull private/public container images. The CD Server relies on the host VM having valid AWS credentials (`aws ecr get-login-password`) or public permissions to download images.
---
## 3. Functional Requirements
### FR-1: HTTP Webhook Endpoint
**Requirement:** The application must expose an HTTP server listening strictly on `127.0.0.1` (localhost) on port `3333`.
**Specifications:**
| Item | Requirement |
|------|-------------|
| Bind Address | `127.0.0.1` only |
| Port | `3333` |
| Active Endpoint | `POST /deploy` |
| Disallowed Methods | All methods except `POST` on `/deploy` must return `405 Method Not Allowed` or `404 Not Found` |
| Disallowed Paths | Any path other than `/deploy` must return `404 Not Found` |
| Request Timeout | All incoming requests must have a read timeout of 60 seconds to prevent slow-loris attacks |
**Implementation Notes:**
- The server must not respond to `GET /deploy`, `HEAD /deploy`, or any other HTTP verb on `/deploy`.
- The server must not follow redirects on incoming requests.
- The server must not accept request bodies larger than 64 KB to prevent memory exhaustion.
### FR-2: Cryptographic Signature Verification (HMAC-SHA256)
**Requirement:** The application must protect itself against unauthorized execution by verifying a request signature before parsing any payload.
**Specifications:**
| Item | Requirement |
|------|-------------|
| Header | `X-Gitea-Signature` |
| Algorithm | HMAC-SHA256 |
| Input | Raw, unparsed request body text |
| Secret Key | Pre-configured `WEBHOOK_SECRET` environment variable |
| Timing Attack Prevention | Use a constant-time comparison function |
| Failure Behavior | Drop the request, log a security warning with the client's IP, return HTTP 401 |
**Implementation Notes:**
- Use `hmac.compare_digest` in Python or `crypto.timingSafeEqual` in Node.js.
- Do not parse the JSON body until the signature is verified.
- Log every signature verification attempt (success and failure) with the client IP.
- Return the 401 response immediately upon signature mismatch without consuming the entire request body.
### FR-3: Dynamic JSON Payload Parsing
**Requirement:** Upon successful authentication, the server must parse the inbound JSON body.
**Schema:**
```json
{
"service": "cv-inference-server",
"image": "public.ecr.aws/i9a4e3f6/msk-cv-inference-server:sha-6caa654",
"env": {
"TRITON_ENDPOINT": "https://dtj-tran--triton-s3-service-unified-triton-server.modal.run",
"CV_INFERENCE_HOST": "https://cv.lumina-msk.io.vn",
"CV_INFERENCE_PORT": "8001",
"BASE_URL": "https://api.lumina-msk.io.vn",
"CORS_ORIGINS": "https://cv.lumina-msk.io.vn,https://api.lumina-msk.io.vn,https://www.lumina-msk.io.vn,https://lumina-msk.io.vn",
"ECR_PUBLIC_REGISTRY_ALIAS": "i9a4e3f6"
}
}
```
**Field Specifications:**
| Field | Type | Required | Constraints |
|-------|------|----------|-------------|
| `service` | string | Yes | Alphanumeric, hyphens, and underscores only; max 63 characters |
| `image` | string | Yes | Must contain a registry, repository, and tag separated by `:` |
| `env` | object | No | Key-value pairs of strings; keys must be valid shell variable names |
**Validation Rules:**
- If `service` or `image` is missing or malformed, return HTTP 400.
- If `env` is present, all keys and values must be strings.
- The server must not allow nested objects or arrays in `env` values.
- The server must reject payloads with additional top-level keys beyond `service`, `image`, and `env` with HTTP 400.
### FR-4: Automated Container Lifecycle Management
**Requirement:** Once the payload is verified and parsed, the application must sequentially execute the following execution pipeline:
#### Step 4.1: Authentication Refresh
- Log into AWS ECR if targeting private repositories.
- Command: `aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account>.dkr.ecr.<region>.amazonaws.com`
- If the registry is `public.ecr.aws`, skip this step.
**Error Handling:**
- If the ECR login fails, log the error and abort the deployment with HTTP 500.
#### Step 4.2: Image Fetch
- Execute `docker pull <image>` using the exact image path and tag string provided in the webhook payload.
- Timeout: 600 seconds (10 minutes) for large images.
**Error Handling:**
- If `docker pull` fails (non-zero exit code), the deployment must stop immediately.
- The currently running container must NOT be stopped or deleted.
- Log the failure and return HTTP 500.
#### Step 4.3: Container Teardown
- Identify any currently running Docker container utilizing the specified `service` name.
- Command: `docker ps --filter "name=<service_name>" --filter "status=running" --format "{{.ID}}"` to get the container ID.
- Stop the container: `docker stop <service_name>`
- Remove the container: `docker rm <service_name>`
**Error Handling:**
- If the container is not found, skip teardown (the service may not be running).
- If the container cannot be stopped within 30 seconds, force stop: `docker kill <service_name>`
- Log all teardown operations.
#### Step 4.4: Container Re-creation & Launch
- Spin up a new container instance using the newly pulled image.
- Dynamically pass every environment variable key-value pair provided in the webhook payload into the new container using Docker environment flags (`-e`).
- Preserve persistent port mapping patterns (e.g., binding the container port to the host port) and any network/volume configurations required for that service.
**Command Pattern:**
```bash
docker run -d \
--name <service_name> \
--restart unless-stopped \
-e "TRITON_ENDPOINT=<value>" \
-e "CV_INFERENCE_HOST=<value>" \
... \
-p <host_port>:<container_port> \
<image>
```
**Port Mapping Policy:**
- The server must have a pre-configured port mapping table in a configuration file.
- Each service name maps to a host port and a container port.
- Example configuration: `service_port_map.json` with entries like `{"cv-inference-server": {"host": 8001, "container": 8001}}`.
- If a service is not found in the port mapping table, the deployment fails with HTTP 500.
**Error Handling:**
- If `docker run` fails, the deployment stops and returns HTTP 500.
- Log all container lifecycle events.
---
## 4. Webhook Payload Specification
### 4.1 Full JSON Schema
```json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"required": ["service", "image"],
"properties": {
"service": {
"type": "string",
"pattern": "^[a-zA-Z0-9_-]{1,63}$"
},
"image": {
"type": "string",
"pattern": "^[a-zA-Z0-9._/-]+:[a-zA-Z0-9._-]+$"
},
"env": {
"type": "object",
"additionalProperties": {
"type": "string"
}
}
},
"additionalProperties": false
}
```
### 4.2 Validation Rules
- The `service` field must match the regex `^[a-zA-Z0-9_-]{1,63}$`.
- The `image` field must contain exactly one `:` separating the repository and the tag.
- The `env` object must not contain non-string values.
- Any extra top-level fields must cause the request to be rejected with HTTP 400.
### 4.3 Example Payloads
**Valid:**
```json
{
"service": "cv-inference-server",
"image": "public.ecr.aws/i9a4e3f6/msk-cv-inference-server:sha-6caa654",
"env": {
"TRITON_ENDPOINT": "https://example.com",
"PORT": "8001"
}
}
```
**Invalid (missing service):**
```json
{
"image": "public.ecr.aws/i9a4e3f6/msk-cv-inference-server:sha-6caa654",
"env": {}
}
```
**Invalid (extra fields):**
```json
{
"service": "cv-inference-server",
"image": "public.ecr.aws/i9a4e3f6/msk-cv-inference-server:sha-6caa654",
"env": {},
"extra": "field"
}
```
---
## 5. Non-Functional & Operational Requirements
### 5.1 Logging & Observability
| Requirement | Specification |
|-------------|---------------|
| Log Format | JSON lines (one JSON object per line) |
| Log Fields | `timestamp`, `level`, `source_ip`, `path`, `method`, `outcome`, `status_code`, `message`, `service`, `image` (if applicable) |
| Log Output | Stdout (captured by systemd journal) and optional file output |
| Log Rotation | Managed by `logrotate` or `journald` with a 30-day retention policy |
| Metrics | Optional Prometheus metrics endpoint on `127.0.0.1:9090/metrics` (not required for MVP) |
**Sample Log Entry:**
```json
{
"timestamp": "2026-07-19T16:00:00Z",
"level": "info",
"source_ip": "10.0.0.5",
"path": "/deploy",
"method": "POST",
"outcome": "signature_verified",
"status_code": 200,
"message": "Deployment initiated",
"service": "cv-inference-server",
"image": "public.ecr.aws/i9a4e3f6/msk-cv-inference-server:sha-6caa654"
}
```
### 5.2 Error Handling & Resiliency
| Scenario | Behavior |
|----------|----------|
| Signature mismatch | Drop request, log security warning, return 401 |
| Malformed JSON | Return 400, log error |
| Missing required fields | Return 400, log error |
| ECR login failure | Abort deployment, log error, return 500 |
| Image pull failure | Stop deployment, do not touch running container, log error, return 500 |
| Container stop failure | Force kill container, log warning, continue if possible |
| Container creation failure | Abort deployment, log error, return 500 |
| Subprocess timeout | Kill process, log error, abort deployment, return 500 |
**Timeout Values:**
| Operation | Timeout |
|-----------|---------|
| Incoming request read | 60 seconds |
| ECR login | 30 seconds |
| Docker pull | 600 seconds (10 minutes) |
| Docker stop | 30 seconds |
| Docker rm | 10 seconds |
| Docker run | 60 seconds |
### 5.3 HTTP Response Codes
| Status Code | Condition | Response Body |
|-------------|-----------|---------------|
| 200 OK | Deployment initiated and completed successfully | `{"status": "success", "service": "<name>"}` |
| 202 Accepted | Deployment initiated but still running (async) | `{"status": "accepted", "service": "<name>"}` |
| 400 Bad Request | Malformed payload or missing required fields | `{"error": "bad_request", "message": "<detail>"}` |
| 401 Unauthorized | HMAC signature verification failed | `{"error": "unauthorized", "message": "Invalid signature"}` |
| 404 Not Found | Path not found | `{"error": "not_found"}` |
| 405 Method Not Allowed | Method not allowed on `/deploy` | `{"error": "method_not_allowed"}` |
| 500 Internal Server Error | Deployment failed during Docker operations | `{"error": "deployment_failed", "message": "<detail>"}` |
---
## 6. Implementation Guidelines
### 6.1 Language Selection
**Recommendation:** Go or Python (FastAPI/Flask) wrapped as a systemd service.
| Language | Pros | Cons |
|----------|------|------|
| Go | Low memory, single binary, fast startup, strong standard library | More boilerplate for subprocess handling |
| Python | Rapid development, rich libraries for HMAC and subprocess | Slightly higher memory, requires Python runtime |
| Node.js | Good ecosystem, familiar to frontend teams | Higher memory, callback complexity |
### 6.2 Security Best Practices
- Run the process as a non-root user with access to the Docker socket (add to `docker` group).
- Do not run as root unless absolutely necessary.
- Store `WEBHOOK_SECRET` in environment variables or a systemd credential file, never hardcoded.
- Use `docker context` or Unix socket with restricted permissions rather than TCP socket.
- Implement rate limiting at the Caddy level as a first line of defense.
### 6.3 Systemd Service Configuration
File: `/etc/systemd/system/cd-server.service`
```ini
[Unit]
Description=CD Server (Deployment Server)
After=docker.service network.target
Requires=docker.service
[Service]
Type=simple
User=deploy
Group=docker
WorkingDirectory=/opt/cd-server
ExecStart=/opt/cd-server/cd-server
Restart=always
RestartSec=5
Environment="WEBHOOK_SECRET=<secret>"
Environment="AWS_REGION=<region>"
EnvironmentFile=/opt/cd-server/.env
[Install]
WantedBy=multi-user.target
```
**Commands:**
```bash
sudo systemctl daemon-reload
sudo systemctl enable --now cd-server
```
### 6.4 Configuration File
The server must support a configuration file at `/opt/cd-server/config.json`:
```json
{
"listen_address": "127.0.0.1",
"listen_port": 3333,
"webhook_secret": "<from_env>",
"aws_region": "<region>",
"service_port_map": {
"cv-inference-server": {"host": 8001, "container": 8001},
"api-gateway": {"host": 8080, "container": 8080}
},
"timeouts": {
"request_read": 60,
"ecr_login": 30,
"docker_pull": 600,
"docker_stop": 30,
"docker_run": 60
}
}
```
### 6.5 Health Check
- The server must expose `GET /health` on `127.0.0.1:3333` for health checks.
- Response: `200 OK` with body `{"status": "healthy"}`.
- This endpoint does not require signature verification.
---
## 7. Mermaid Diagrams
### 7.1 Logic-Data Flow Diagram
```mermaid
flowchart TD
A[CI Pipeline POST /deploy] --> B[Caddy Reverse Proxy]
B --> C{Path == /deploy?}
C -->|No| D[Return 404]
C -->|Yes| E{Method == POST?}
E -->|No| F[Return 405]
E -->|Yes| G[CD Server receives request]
G --> H{Verify HMAC-SHA256 Signature}
H -->|Fail| I[Log security warning + IP]
I --> J[Return 401 Unauthorized]
H -->|Pass| K[Parse JSON Payload]
K --> L{Validate Schema}
L -->|Fail| M[Return 400 Bad Request]
L -->|Pass| N{ECR Login Required?}
N -->|Yes| O[Authenticate with AWS ECR]
O --> P{Login Success?}
P -->|No| Q[Abort: Return 500]
N -->|No| R[Pull Docker Image]
P -->|Yes| R
R --> S{Pull Success?}
S -->|No| Q
S -->|Yes| T[Find Running Container]
T --> U{Container Running?}
U -->|Yes| V[Stop Container]
U -->|No| W[Create New Container]
V --> X[Remove Container]
X --> W
W --> Y[Run New Container]
Y --> Z{Run Success?}
Z -->|No| Q
Z -->|Yes| AA[Return 200 OK]
```
### 7.2 Architecture Diagram
```mermaid
flowchart LR
subgraph Internet
A[CI Pipeline\nGitea Actions]
end
subgraph Host VM
B[Caddy Server\nReverse Proxy\nTLS Termination]
C[CD Server\n127.0.0.1:3333\nHMAC Verification\nDocker Orchestrator]
D[Docker Daemon\n/var/run/docker.sock]
E[Service Containers\ncv-inference-server\napi-gateway\n...]
end
subgraph AWS Cloud
F[AWS ECR\nElastic Container Registry]
end
A -->|HTTPS POST| B
B -->|Forward /deploy| C
C -->|docker pull| D
C -->|docker stop/rm| D
C -->|docker run| D
D -->|Run/Pull| E
C -->|Pull Images| F
```
---
## 8. Testing & Verification Strategy
### 8.1 Unit Tests
- HMAC signature verification (valid and invalid signatures)
- JSON payload validation (valid, missing fields, extra fields)
- Port mapping resolution
- Error handling for each Docker operation
### 8.2 Integration Tests
- End-to-end webhook test with a mock Caddy server
- Docker container lifecycle test (pull → stop → remove → run)
- ECR login test with mock AWS credentials
### 8.3 Security Tests
- Reject requests without the `X-Gitea-Signature` header
- Reject requests with an invalid signature
- Reject requests from non-localhost IPs (ensure binding is correct)
- Timing attack resistance verification (constant-time comparison)
---
## 9. Deployment Checklist
- [ ] Build the CD Server binary
- [ ] Create user `deploy` and add to `docker` group
- [ ] Create `/opt/cd-server` directory
- [ ] Place binary at `/opt/cd-server/cd-server`
- [ ] Create `config.json` at `/opt/cd-server/config.json`
- [ ] Set `WEBHOOK_SECRET` and `AWS_REGION` environment variables
- [ ] Create systemd service file at `/etc/systemd/system/cd-server.service`
- [ ] Enable and start the service: `systemctl enable --now cd-server`
- [ ] Verify Caddy forwards `/deploy` to `127.0.0.1:3333`
- [ ] Send a test webhook from CI pipeline
- [ ] Verify container lifecycle in Docker
- [ ] Verify logs in `journalctl -u cd-server`