> ## Documentation Index > Fetch the complete documentation index at: https://docs.finwatch.finance/llms.txt > Use this file to discover all available pages before exploring further. # Deploying FinWatch to Production This guide covers everything a DevOps or Platform engineer needs to run FinWatch reliably in a production environment — from system requirements and configuration to Docker/Kubernetes deployment, monitoring, backup, and scaling. *** ## System Requirements ### Minimum Requirements | Resource | Minimum | Recommended | Notes | | ----------- | -------------- | ----------- | ------------------------------------------------------------ | | **RAM** | 2 GB | 4-8 GB | DuckDB is memory-intensive for aggregate queries | | **CPU** | 1 core | 2-4 cores | Single-threaded DuckDB, but Go runtime uses additional cores | | **Disk** | 1 GB | 10-50 GB | Depends on transaction volume and retention | | **Network** | Outbound HTTPS | — | Required for Git sync and WebSocket tunnel | | **OS** | Linux (amd64) | — | Also supports macOS and Windows for development | ### Disk Space Estimation DuckDB stores data in a columnar format which is highly compressed. As a rough guide: * **1 million transactions** ≈ 50-100 MB on disk * **10 million transactions** ≈ 500 MB - 1 GB * **100 million transactions** ≈ 5-10 GB The `instructions.db` (compiled rules) is typically under 1 MB regardless of the number of rules. *** ## Configuration Reference FinWatch is configured entirely through environment variables. Here is the complete reference: ### Core Configuration | Variable | Default | Description | | ----------------------- | --------------- | ------------------------------------------- | | `FINWATCH_PORT` | `8081` | HTTP server port | | `WATCH_SCRIPT_DIR` | `watch_scripts` | Directory where `.ws` rule files are stored | | `FINWATCH_MEMORY_LIMIT` | `2GiB` | Maximum memory DuckDB is allowed to use | ### Git Repository (GitOps) | Variable | Default | Description | | ------------------------- | --------- | ------------------------------------------------------------------ | | `WATCH_SCRIPT_GIT_REPO` | *(empty)* | Git repository URL for rule syncing. Enables GitOps mode when set. | | `WATCH_SCRIPT_GIT_BRANCH` | `main` | Branch to track for rule updates | ### Data Synchronization | Variable | Default | Description | | ---------- | --------- | -------------------------------------------------------------------------------- | | `BLNK_DSN` | *(empty)* | PostgreSQL connection string for Blnk database. Enables watermark sync when set. | ### Database Paths FinWatch creates its databases in the `blnk_agent/` directory relative to the working directory: | Path | Purpose | | ---------------------------- | -------------------------------------------- | | `blnk_agent/blnk.db` | Transaction data (DuckDB) | | `blnk_agent/instructions.db` | Compiled rules (DuckDB) | | `blnk_agent/duckdb_temp/` | Temporary directory for DuckDB spill-to-disk | *** ## Memory Management DuckDB's performance comes from keeping data in memory. As your transaction volume grows, memory management becomes critical. ### How Memory Is Used DuckDB uses memory for: 1. **Buffer pool** — Cached table pages for fast reads. 2. **Query execution** — Intermediate results from aggregate queries. 3. **Write-ahead log** — Buffered writes before checkpointing to disk. ### Configuring the Memory Limit The `FINWATCH_MEMORY_LIMIT` environment variable controls DuckDB's maximum memory usage: ```bash theme={null} # Conservative (low-traffic systems) export FINWATCH_MEMORY_LIMIT="1GiB" # Standard (medium-traffic systems) export FINWATCH_MEMORY_LIMIT="2GiB" # High-performance (high-traffic systems) export FINWATCH_MEMORY_LIMIT="8GiB" ``` Accepted formats: `512MiB`, `1GiB`, `2GiB`, `4GiB`, `8GiB`, `16GiB`. ### DuckDB Pragmas FinWatch initializes DuckDB with the following settings: ```shellscript theme={null} SET access_mode = 'READ_WRITE'; SET threads = 1; SET memory_limit = '2GiB'; SET checkpoint_threshold = '64MiB'; ``` * **`threads = 1`**: Limits DuckDB to a single execution thread. This simplifies the single-writer concurrency model. Go's runtime handles HTTP concurrency separately. * **`memory_limit`**: The upper bound on DuckDB's memory consumption. When exceeded, DuckDB spills intermediate results to the `duckdb_temp/` directory. * **`checkpoint_threshold = '64MiB'`**: Controls how frequently in-memory data is flushed to disk. Lower values mean more frequent writes (safer but slower). ### Memory Sizing Guidelines | Transaction Volume | Recommended Memory | Notes | | ------------------ | ------------------ | ----------------------------------------------- | | \< 10K/day | `1GiB` | Minimal footprint | | 10K - 100K/day | `2GiB` | Default is sufficient | | 100K - 1M/day | `4GiB` | Aggregate queries benefit from more memory | | 1M - 10M/day | `8GiB` | Large time windows need significant buffer pool | | > 10M/day | `16GiB+` | Consider data retention policies | ### Temp Directory When DuckDB exceeds its memory limit, it spills data to `blnk_agent/duckdb_temp/`. Ensure this directory: * Has sufficient disk space (at least 2x the memory limit). * Is on fast storage (SSD recommended). * Is not on a tmpfs or RAM-backed filesystem (defeats the purpose of spilling). *** ## Docker Deployment ### Production Docker Run ```bash theme={null} docker run -d \ --name finwatch \ --restart unless-stopped \ -p 8081:8081 \ -e WATCH_SCRIPT_GIT_REPO="https://github.com/your-org/finwatch-rules.git" \ -e WATCH_SCRIPT_GIT_BRANCH="main" \ -e WATCH_SCRIPT_DIR="/app/watch_scripts" \ -e BLNK_DSN="postgres://user:password@db-host:5432/blnk?sslmode=require" \ -e FINWATCH_MEMORY_LIMIT="4GiB" \ -v finwatch-data:/app/blnk_agent \ --memory 6g \ --cpus 2 \ finwatch/finwatch:latest ``` Key flags: * **`--restart unless-stopped`**: Auto-restart on crash or server reboot. * **`-v finwatch-data:/app/blnk_agent`**: Persistent volume for DuckDB data. Without this, data is lost on container restart. * **`--memory 6g`**: Docker memory limit. Set higher than `FINWATCH_MEMORY_LIMIT` to leave room for Go runtime overhead. * **`--cpus 2`**: Limit CPU usage. ### Production Docker Compose ```yaml theme={null} version: '3.8' services: finwatch: image: finwatch/finwatch:latest container_name: finwatch restart: unless-stopped ports: - "8081:8081" environment: - WATCH_SCRIPT_GIT_REPO=https://github.com/your-org/finwatch-rules.git - WATCH_SCRIPT_GIT_BRANCH=main - WATCH_SCRIPT_DIR=/app/watch_scripts - BLNK_DSN=postgres://user:password@db-host:5432/blnk?sslmode=require - FINWATCH_MEMORY_LIMIT=4GiB volumes: - finwatch-data:/app/blnk_agent deploy: resources: limits: memory: 6G cpus: '2' reservations: memory: 2G cpus: '1' healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8081/instructions"] interval: 30s timeout: 10s retries: 3 start_period: 15s logging: driver: json-file options: max-size: "50m" max-file: "5" volumes: finwatch-data: driver: local ``` ### Health Check The Docker health check uses the `/instructions` endpoint. A `200 OK` response confirms: * The HTTP server is running. * The DuckDB instruction database is accessible. * The API can serve requests. *** ## Kubernetes Deployment ### Deployment Manifest ```yaml theme={null} apiVersion: apps/v1 kind: Deployment metadata: name: finwatch labels: app: finwatch spec: replicas: 1 selector: matchLabels: app: finwatch template: metadata: labels: app: finwatch spec: containers: - name: finwatch image: finwatch/finwatch:latest ports: - containerPort: 8081 name: http env: - name: WATCH_SCRIPT_GIT_REPO value: "https://github.com/your-org/finwatch-rules.git" - name: WATCH_SCRIPT_GIT_BRANCH value: "main" - name: WATCH_SCRIPT_DIR value: "/app/watch_scripts" - name: FINWATCH_MEMORY_LIMIT value: "4GiB" - name: BLNK_DSN valueFrom: secretKeyRef: name: finwatch-secrets key: blnk-dsn resources: requests: memory: "2Gi" cpu: "500m" limits: memory: "6Gi" cpu: "2000m" livenessProbe: httpGet: path: /instructions port: 8081 initialDelaySeconds: 15 periodSeconds: 30 timeoutSeconds: 5 failureThreshold: 3 readinessProbe: httpGet: path: /instructions port: 8081 initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 3 failureThreshold: 2 volumeMounts: - name: data mountPath: /app/blnk_agent volumes: - name: data persistentVolumeClaim: claimName: finwatch-data --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: finwatch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 50Gi storageClassName: ssd --- apiVersion: v1 kind: Service metadata: name: finwatch spec: selector: app: finwatch ports: - port: 8081 targetPort: 8081 name: http type: ClusterIP ``` ### Important Kubernetes Notes * **Replicas: 1.** FinWatch uses an embedded DuckDB database with a single-writer model. Running multiple replicas against the same data volume will cause write contention. If you need horizontal scaling, see the [Scaling Considerations](#scaling-considerations) section. * **PVC with SSD.** DuckDB performance is heavily dependent on disk I/O for spill-to-disk operations. Use SSD-backed persistent volumes. * **Secrets.** Store `BLNK_DSN` (which contains database credentials) in a Kubernetes Secret, not in plain-text environment variables. * **Liveness vs. Readiness.** The liveness probe checks if FinWatch is alive; the readiness probe checks if it's ready to accept traffic. The readiness probe has a shorter interval for faster traffic routing. *** ## Monitoring and Observability ### Log Format FinWatch uses [zerolog](https://github.com/rs/zerolog) for structured JSON logging. In production, logs are formatted as: ```json theme={null} { "level": "info", "time": "2026-04-18T14:30:00Z", "message": "Received inject request", "transaction_id": "txn_001" } ``` ### Key Log Events to Monitor | Log Message | Level | Meaning | | ------------------------------------------ | ------- | -------------------------------- | | `Received inject request` | `info` | Transaction received via API | | `Compiled watch script` | `info` | A rule was successfully compiled | | `Failed to compile watch script` | `error` | A rule has a syntax error | | `Error processing transaction` | `error` | Transaction injection failed | | `WebSocket tunnel not connected` | `error` | Anomaly reporting is offline | | `Failed to clone or update Git repository` | `fatal` | GitOps sync is broken | ### Metrics to Watch | Metric | Source | Warning Threshold | Action | | -------------------------- | ----------------- | ----------------- | ---------------------------------------- | | API response time | HTTP access logs | > 100ms P95 | Investigate DuckDB memory pressure | | Rule compilation errors | Application logs | Any | Fix the malformed `.ws` file | | Transaction ingestion rate | Application logs | Sudden drop | Check API connectivity | | DuckDB file size | Disk monitoring | > 80% of disk | Implement data retention policy | | Memory usage | Container metrics | > 80% of limit | Increase `FINWATCH_MEMORY_LIMIT` | | WebSocket disconnections | Application logs | > 1/hour | Check network connectivity to Blnk Cloud | | Git sync failures | Application logs | Any | Check Git credentials and network | ### Integrating with Log Aggregators FinWatch's JSON logs can be consumed by any standard log aggregator: * **Datadog:** Configure the Docker log driver or use the Datadog agent's log collection. * **ELK Stack:** Forward container logs via Filebeat or Fluentd. * **Grafana Loki:** Use Promtail to ship container logs. * **CloudWatch:** Use the `awslogs` Docker log driver. *** ## Backup and Recovery ### What to Back Up | Data | Location | Backup Strategy | Recovery Priority | | ------------------------- | ------------------------------ | ----------------------------------------- | ------------------------------------------------ | | **Rules (.ws files)** | Git repository | Git itself is the backup | **Critical** — rules are the core logic | | **Transaction data** | `blnk_agent/blnk.db` | File-level backup or sync from PostgreSQL | **Medium** — can be rebuilt from PostgreSQL | | **Compiled instructions** | `blnk_agent/instructions.db` | File-level backup | **Low** — rebuilt automatically from `.ws` files | | **Variable definitions** | Git repository or config store | Same as rules | **High** — needed for rules to function | ### DuckDB File Backup DuckDB database files can be backed up with a simple file copy while FinWatch is running, but for consistency, prefer: ```bash theme={null} # Stop FinWatch, copy the file, restart docker stop finwatch cp blnk_agent/blnk.db blnk_agent/blnk.db.backup docker start finwatch ``` ### Recovery Scenarios **Scenario: DuckDB data is corrupted or lost.** 1. FinWatch restarts and creates a fresh DuckDB database. 2. Compiled rules are rebuilt from the `.ws` files (via Git sync or local directory). 3. Historical transaction data is rebuilt via the watermark sync from PostgreSQL. 4. Recovery is fully automatic — no manual intervention required. **Scenario: Git repository is unavailable.** 1. FinWatch continues to operate with the last-synced rules. 2. Git polling logs warnings but does not crash. 3. When the repository becomes available again, FinWatch catches up automatically. **Scenario: PostgreSQL (BLNK\_DSN) is unavailable.** 1. Watermark sync pauses. Logs warnings. 2. FinWatch continues to evaluate rules against locally-stored data. 3. Aggregate functions use the data available in DuckDB — results may be stale. 4. When PostgreSQL recovers, the watermark sync resumes from where it left off. *** ## Scaling Considerations ### Single-Instance Model FinWatch is designed as a **single-instance** service. This is a deliberate architectural choice driven by DuckDB's single-writer concurrency model. The benefits: * **Simplicity:** No distributed coordination, no consensus protocols, no split-brain scenarios. * **Consistency:** All rules see the same data. No eventual consistency issues. * **Performance:** Local DuckDB queries are faster than any network-based alternative. ### When to Scale A single FinWatch instance can comfortably handle: * **10,000+ transactions per second** for simple rules. * **1,000+ transactions per second** with complex aggregate rules. * **100+ active rules** with no performance impact. If you're hitting these limits, consider: 1. **Vertical scaling:** Increase memory and CPU. DuckDB benefits significantly from more RAM. 2. **Rule optimization:** Ensure cheap conditions are evaluated before expensive aggregates (the gate-and-probe pattern). 3. **Time window reduction:** Smaller aggregate time windows mean less data to scan. 4. **Data retention:** Purge old transaction data that is no longer needed for rule evaluation. ### Multi-Instance Patterns If you truly need horizontal scaling (e.g., processing 100K+ TPS), consider: * **Sharding by source account:** Route transactions to different FinWatch instances based on the source account. This ensures aggregate functions for a given account are always evaluated by the same instance. * **Read replicas:** Run multiple instances in read-only mode for serving API queries, with a single write instance for ingestion. These patterns add significant operational complexity and should only be considered after exhausting vertical scaling options. *** ## Pre-Production Checklist * **Environment variables are set** — Verify all required vars are configured. * **Persistent volume is attached** — DuckDB data survives container restarts. * **Memory limits are appropriate** — Docker/K8s limit > DuckDB `memory_limit` + 1-2 GB headroom. * **Git repository is accessible** — FinWatch can clone and pull the rules repo. * **PostgreSQL is reachable** — If using `BLNK_DSN`, verify connectivity. * **Health checks are configured** — Liveness and readiness probes are active. * **Logging is aggregated** — Logs are shipped to your monitoring platform. * **Backup strategy is in place** — Rules are in Git; data can be rebuilt. * **Alerts are configured** — Monitor for compilation errors, high latency, and sync failures. * **Test transactions have been validated** — Run a full test suite before going live. *** ## Next Steps * [**Integration Guide**](integration-guide.md) — Connect your application to FinWatch. * [**GitOps Rule Management**](gitops-rule-management.md) — Set up Git-based rule deployment for production. * [**Troubleshooting**](troubleshooting.md) — Diagnose and fix common production issues. * [**Understanding the Architecture**](understanding-the-architecture.md) — Deep dive into how FinWatch works.