OC Architecture

# OpenClaw Architecture — Trading Control Plane

High-level view of how OpenClaw orchestrates **cTrader-connected Python strategies**, **Postgres-backed control state**, **Process Hub (Redis + RQ)** as the **intended** control path (with **in-memory fallback** only when Redis is down), and **AI/cron (Coach)** around the same system.

---

## 1) Overview

- **OpenClaw** is the orchestration and governance layer: strategy code, runtime API, DB state, dashboards, cron jobs (Coach Gold, grader, briefings), and safety/demo vs live boundaries.
- **Source of truth for “what should run”** is primarily **PostgreSQL** (`bot_control`, `bot_instances`). Bots poll the API; the dashboard and hub read the same data.
- **Scale-out / core wiring**: **Redis + RQ** decouple “decide start/stop” (API / hub loop) from “execute start/stop” (worker process). Without Redis, the hub **falls back** to in-process execution on a single host (see `automation/process_hub.py`). **Run Redis** locally with **Memurai** or **`tools/redis-local/redis-server.exe`** (`scripts/start_redis_windows.ps1`); Docker compose file exists as an **optional** path — see **`tools/README.md`**.

---

## 2) One strategy codebase, two deployment instances

- **Strategy (code)** — There is **one Python module per `bot_id`** (e.g. `xauusd_bollinger_bounce.py` for S03) plus shared **`bot_engine`**. Signal logic does **not** branch on “demo vs live”; there is no second codebase for the same strategy.
- **Instance (runtime)** — What runs is a **process** bound to a **cTrader account**: `python …/xauusd_….py --account-type demo|live`. That flag only selects **which credentials / account** and **which DB rows** `(bot_id, account_type)` the process polls for `/params` and `/control`. It is a **deployment dimension**, not a different strategy program.
- **Parameters** — `bot_params` is stored **per** `(bot_id, account_type)` so you can size risk differently on live than demo, but the **intent** is: same algorithm and defaults; divergence should be **operational** (lot, safety) or **regime / profile** (e.g. `position_profile`, future regime-specific params), not a forked strategy file.
- **Dashboard Demo / Live swimlanes** — Two **instances** of the **same** strategy set (two accounts), not two different codebases.

More detail: **`docs/STRATEGY_INSTANCES.md`**.

---

## 3) Core Components

| Layer | Role |
|--------|------|
| **`automation/params_api.py`** | FastAPI app: `/params`, `/control`, `/session/*`, `/trade`, `/dashboard/data`, `/hub/*`, floor/coach helpers, embedded dashboard HTML. |
| **Python strategies** (`bots/*.py`) | Shared **`bot_engine`** (cTrader Open API, Twisted): bars, signals, orders, SL/TP, logging to API. |
| **PostgreSQL** | `bot_params`, `bot_control`, `run_sessions`, `trades`, `environments`, `open_positions`, **`bot_instances`** (hub), **`credential_profiles`**, `process_events`, etc. |
| **Process Hub** (`automation/process_hub.py`) | Reconciliation loop: compares `bot_instances.desired_status` vs real PIDs; enqueues **RQ** jobs to start/stop bots (or runs inline if Redis missing). |
| **Redis** | RQ queue **`oc:tasks`**, reconcile lock **`oc:reconcile-lock`**, heartbeat hash **`oc:heartbeats`**, optional event pub/sub. Not the primary trade DB. |
| **RQ worker** (`automation/rq_worker.py`) | Separate process: consumes **`oc:tasks`**, runs `task_start_instance` / `task_stop_instance` / health tasks from `process_hub.py`. |
| **Dashboard** (`automation/dashboard.html`) | Swimlanes (demo / live / backlog), P&L, grades, **merge with `/hub/state`** when local PID visibility differs from hub. |
| **Cron** (`cron/jobs.json`) | Coach Gold, bot grader, morning briefing, etc. Payloads often point at `scripts/coach_files/*`. |
| **GitHub** | Single repo: OpenClaw layout + trading code at repo root (`automation/`, `bots/`). |

**Legacy / diagram note:** Older docs referred to an **MT5 REST bridge** and SQLite-style performance stores. The **current live path** is **cTrader Open API + Postgres (boilerbase/Timescale as deployed)**. MT5 may exist elsewhere in the vision; this file describes the **implemented** stack.

---

## 4) Control plane: `bot_control` vs Process Hub

- **`bot_control`** — Row per `(bot_id, account_type)`: `status` (running/stopped/paused), `lot_size`, `reason`. Bots **poll** `GET /control/{bot_id}`; stopping here causes graceful shutdown in **`bot_engine`**.
- **`bot_instances`** — Row per supervised instance: `desired_status`, `actual_status`, `pid`, script path, `credential_profile_id`, etc. The **hub reconciler** drives lifecycle here.
- **Alignment:** `PUT /control` (and internal `_cmd_set_status`, preset/move paths) also **updates `bot_instances.desired_status`** when the table exists, so the hub does not immediately fight the API (e.g. enqueueing stop while control says running).

---

## 5) Redis and RQ (C#-friendly mental model)

- **Redis** here is **not** a full enterprise service bus; it’s a **fast shared store** used for a **job queue**, a **distributed lock**, and **heartbeats**.
- **RQ** is a **Python job queue** on top of Redis: the API/hub **enqueues** work; **`rq_worker.py`** **dequeues** and runs start/stop (subprocess spawn/kill, PID persisted in Postgres).

**Typical dev (Windows, no Docker):** start **Memurai** or **`redis-server.exe`** from `tools/redis-local/` (or `powershell -File scripts/start_redis_windows.ps1`), then **`params_api`**, then **`python automation/rq_worker.py`**. Env: **`REDIS_URL`** (default `redis://localhost:6379/0`). Optional container: `docker-compose.redis.yml`.

---

## 6) Data model (runtime)

- **Sessions:** `run_sessions` — opened on successful cTrader account auth (`POST /session/start`), closed on bot exit (`POST /session/end`, status e.g. `COMPLETED`).
- **Trades:** `trades` — closed trades via `POST /trade` (and related helpers).
- **Params:** `bot_params` — JSON strategy parameters; bots poll `GET /params`.
- **Presets / snapshots:** `bot_param_presets` and dashboard flows apply to `bot_params` / `bot_control`.
- **Ticks / analytics (optional):** **`tick_streamer`** kan skrive til **PostgreSQL** database **`boilerbase`**, schema **`streamer`**: hypertables **`streamer_ticks`**, **`streamer_bars`**, **`streamer_pulse`** (se `tick_streamer/sql/schema_timescale.sql`). Realtime bots bruger primært **Redis**; DB er til historik/ML. Integrationstest: `python tick_streamer/test_tick_recorder.py`. Se **`docs/MEMURAI_AND_TICK_PIPELINE.md`**.

---

## 7) Data flow (end-to-end)

1. **Operator or Coach** adjusts params/control via API or dashboard → Postgres updated.
2. **Hub** (if enabled) reconciles `bot_instances` → **RQ** or inline → **spawn** `python …/bots/<strategy>.py --account-type demo|live`.
3. **Bot** loads tokens (env / `credential_profiles` when hub-spawned), connects to **cTrader Open API**, starts session, subscribes bars/spots, places orders per strategy.
4. **Closed trades and positions** reported to API → DB; dashboard and ML/grader jobs read same DB.
5. **Cron / Coach** reads files under `scripts/coach_files/` and DB summaries; may suggest actions (subject to `owner_overrides` and safety rules).

---

## 8) Dashboard behaviour (recent design)

- **Swimlanes** show bots with **`display_status === RUNNING`**; payload from `/dashboard/data` can be **merged with `/hub/state`** so hub-managed processes appear even when the API host does not see local PIDs.
- **DB / PY lamps:** DB reflects **`bot_control`** (and **hub-active** when merged); PY reflects **local process** and/or **hub-reported** liveness.

---

## 9) Deployment & safety

- **Demo vs live** — Separate env rows, tokens, and `account_type`; live promotion remains policy-driven (gates, Coach, manual override).
- **Watchdog** (in `params_api`) — Optional restart of stalled/dead **Python** bots (demo-focused by default; live gated by env).
- **Secrets** — `.env` and `credentials/` gitignored; production should use profiles + secure injection.

---

## 10) Future hooks (ideas)

- **Coach → running bot messaging:** e.g. Postgres inbox table, new `GET` poll, or Redis pub/sub; no change required to Redis/RQ for simple “post directive” patterns.
- **Stronger service bus** — Only if multi-host orchestration outgrows RQ (Azure Service Bus, RabbitMQ, NATS, etc.).

---

## 11) Key file map

| Area | Path |
|------|------|
| API + dashboard embed | `automation/params_api.py` |
| Hub + RQ tasks | `automation/process_hub.py` |
| Worker entry | `automation/rq_worker.py` |
| Bot runtime | `bots/bot_engine.py` |
| Strategies | `bots/xauusd_*.py` (one file per strategy; se `bots/README.md`) |
| DB migrations / schema | `Db-setup/files/*.sql` |
| Coach assets | `scripts/coach_files/` |
| Local Redis layout (not in git) | `tools/README.md` |
| Strategy vs demo/live instances | `docs/STRATEGY_INSTANCES.md` |

---

*Last updated to reflect Process Hub, Redis/RQ, `bot_control` / `bot_instances` alignment, cTrader Python runtime, and unified repo root layout.*

**Web:** `http://localhost:8000/architecture` renders this file live from the repo root (`ARCHITECTURE.md`). For proper tables in the browser, install `markdown` in the API venv: `pip install markdown`.