Deploying FinWatch to Production

​

System Requirements

​

Minimum Requirements

​

Disk Space Estimation

• 1 million transactions ≈ 50-100 MB on disk
• 10 million transactions ≈ 500 MB - 1 GB
• 100 million transactions ≈ 5-10 GB

​

Configuration Reference

​

Core Configuration

​

Git Repository (GitOps)

​

Data Synchronization

​

Database Paths

​

Memory Management

​

How Memory Is Used

1. Buffer pool — Cached table pages for fast reads.
2. Query execution — Intermediate results from aggregate queries.
3. Write-ahead log — Buffered writes before checkpointing to disk.

​

Configuring the Memory Limit

# Conservative (low-traffic systems)
export FINWATCH_MEMORY_LIMIT="1GiB"

# Standard (medium-traffic systems)
export FINWATCH_MEMORY_LIMIT="2GiB"

# High-performance (high-traffic systems)
export FINWATCH_MEMORY_LIMIT="8GiB"

​

DuckDB Pragmas

SET access_mode = 'READ_WRITE';
SET threads = 1;
SET memory_limit = '2GiB';
SET checkpoint_threshold = '64MiB';

• threads = 1: Limits DuckDB to a single execution thread. This simplifies the single-writer concurrency model. Go’s runtime handles HTTP concurrency separately.
• memory_limit: The upper bound on DuckDB’s memory consumption. When exceeded, DuckDB spills intermediate results to the duckdb_temp/ directory.
• checkpoint_threshold = '64MiB': Controls how frequently in-memory data is flushed to disk. Lower values mean more frequent writes (safer but slower).

​

Memory Sizing Guidelines

​

Temp Directory

• Has sufficient disk space (at least 2x the memory limit).
• Is on fast storage (SSD recommended).
• Is not on a tmpfs or RAM-backed filesystem (defeats the purpose of spilling).

​

Docker Deployment

​

Production Docker Run

docker run -d \
  --name finwatch \
  --restart unless-stopped \
  -p 8081:8081 \
  -e WATCH_SCRIPT_GIT_REPO="https://github.com/your-org/finwatch-rules.git" \
  -e WATCH_SCRIPT_GIT_BRANCH="main" \
  -e WATCH_SCRIPT_DIR="/app/watch_scripts" \
  -e BLNK_DSN="postgres://user:password@db-host:5432/blnk?sslmode=require" \
  -e FINWATCH_MEMORY_LIMIT="4GiB" \
  -v finwatch-data:/app/blnk_agent \
  --memory 6g \
  --cpus 2 \
  finwatch/finwatch:latest

• --restart unless-stopped: Auto-restart on crash or server reboot.
• -v finwatch-data:/app/blnk_agent: Persistent volume for DuckDB data. Without this, data is lost on container restart.
• --memory 6g: Docker memory limit. Set higher than FINWATCH_MEMORY_LIMIT to leave room for Go runtime overhead.
• --cpus 2: Limit CPU usage.

​

Production Docker Compose

version: '3.8'

services:
  finwatch:
    image: finwatch/finwatch:latest
    container_name: finwatch
    restart: unless-stopped
    ports:
      - "8081:8081"
    environment:
      - WATCH_SCRIPT_GIT_REPO=https://github.com/your-org/finwatch-rules.git
      - WATCH_SCRIPT_GIT_BRANCH=main
      - WATCH_SCRIPT_DIR=/app/watch_scripts
      - BLNK_DSN=postgres://user:password@db-host:5432/blnk?sslmode=require
      - FINWATCH_MEMORY_LIMIT=4GiB
    volumes:
      - finwatch-data:/app/blnk_agent
    deploy:
      resources:
        limits:
          memory: 6G
          cpus: '2'
        reservations:
          memory: 2G
          cpus: '1'
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8081/instructions"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 15s
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "5"

volumes:
  finwatch-data:
    driver: local

​

Health Check

• The HTTP server is running.
• The DuckDB instruction database is accessible.
• The API can serve requests.

​

Kubernetes Deployment

​

Deployment Manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: finwatch
  labels:
    app: finwatch
spec:
  replicas: 1
  selector:
    matchLabels:
      app: finwatch
  template:
    metadata:
      labels:
        app: finwatch
    spec:
      containers:
        - name: finwatch
          image: finwatch/finwatch:latest
          ports:
            - containerPort: 8081
              name: http
          env:
            - name: WATCH_SCRIPT_GIT_REPO
              value: "https://github.com/your-org/finwatch-rules.git"
            - name: WATCH_SCRIPT_GIT_BRANCH
              value: "main"
            - name: WATCH_SCRIPT_DIR
              value: "/app/watch_scripts"
            - name: FINWATCH_MEMORY_LIMIT
              value: "4GiB"
            - name: BLNK_DSN
              valueFrom:
                secretKeyRef:
                  name: finwatch-secrets
                  key: blnk-dsn
          resources:
            requests:
              memory: "2Gi"
              cpu: "500m"
            limits:
              memory: "6Gi"
              cpu: "2000m"
          livenessProbe:
            httpGet:
              path: /instructions
              port: 8081
            initialDelaySeconds: 15
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /instructions
              port: 8081
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 2
          volumeMounts:
            - name: data
              mountPath: /app/blnk_agent
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: finwatch-data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: finwatch-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: ssd
---
apiVersion: v1
kind: Service
metadata:
  name: finwatch
spec:
  selector:
    app: finwatch
  ports:
    - port: 8081
      targetPort: 8081
      name: http
  type: ClusterIP

​

Important Kubernetes Notes

• Replicas: 1. FinWatch uses an embedded DuckDB database with a single-writer model. Running multiple replicas against the same data volume will cause write contention. If you need horizontal scaling, see the Scaling Considerations section.
• PVC with SSD. DuckDB performance is heavily dependent on disk I/O for spill-to-disk operations. Use SSD-backed persistent volumes.
• Secrets. Store BLNK_DSN (which contains database credentials) in a Kubernetes Secret, not in plain-text environment variables.
• Liveness vs. Readiness. The liveness probe checks if FinWatch is alive; the readiness probe checks if it’s ready to accept traffic. The readiness probe has a shorter interval for faster traffic routing.

​

Monitoring and Observability

​

Log Format

{
  "level": "info",
  "time": "2026-04-18T14:30:00Z",
  "message": "Received inject request",
  "transaction_id": "txn_001"
}

​

Key Log Events to Monitor

​

Metrics to Watch

​

Integrating with Log Aggregators

• Datadog: Configure the Docker log driver or use the Datadog agent’s log collection.
• ELK Stack: Forward container logs via Filebeat or Fluentd.
• Grafana Loki: Use Promtail to ship container logs.
• CloudWatch: Use the awslogs Docker log driver.

​

Backup and Recovery

​

What to Back Up

​

DuckDB File Backup

# Stop FinWatch, copy the file, restart
docker stop finwatch
cp blnk_agent/blnk.db blnk_agent/blnk.db.backup
docker start finwatch

​

Recovery Scenarios

1. FinWatch restarts and creates a fresh DuckDB database.
2. Compiled rules are rebuilt from the .ws files (via Git sync or local directory).
3. Historical transaction data is rebuilt via the watermark sync from PostgreSQL.
4. Recovery is fully automatic — no manual intervention required.

1. FinWatch continues to operate with the last-synced rules.
2. Git polling logs warnings but does not crash.
3. When the repository becomes available again, FinWatch catches up automatically.

1. Watermark sync pauses. Logs warnings.
2. FinWatch continues to evaluate rules against locally-stored data.
3. Aggregate functions use the data available in DuckDB — results may be stale.
4. When PostgreSQL recovers, the watermark sync resumes from where it left off.

​

Scaling Considerations

​

Single-Instance Model

• Simplicity: No distributed coordination, no consensus protocols, no split-brain scenarios.
• Consistency: All rules see the same data. No eventual consistency issues.
• Performance: Local DuckDB queries are faster than any network-based alternative.

​

When to Scale

• 10,000+ transactions per second for simple rules.
• 1,000+ transactions per second with complex aggregate rules.
• 100+ active rules with no performance impact.

1. Vertical scaling: Increase memory and CPU. DuckDB benefits significantly from more RAM.
2. Rule optimization: Ensure cheap conditions are evaluated before expensive aggregates (the gate-and-probe pattern).
3. Time window reduction: Smaller aggregate time windows mean less data to scan.
4. Data retention: Purge old transaction data that is no longer needed for rule evaluation.

​

Multi-Instance Patterns

• Sharding by source account: Route transactions to different FinWatch instances based on the source account. This ensures aggregate functions for a given account are always evaluated by the same instance.
• Read replicas: Run multiple instances in read-only mode for serving API queries, with a single write instance for ingestion.

​

Pre-Production Checklist

• Environment variables are set — Verify all required vars are configured.
• Persistent volume is attached — DuckDB data survives container restarts.
• Memory limits are appropriate — Docker/K8s limit > DuckDB memory_limit + 1-2 GB headroom.
• Git repository is accessible — FinWatch can clone and pull the rules repo.
• PostgreSQL is reachable — If using BLNK_DSN, verify connectivity.
• Health checks are configured — Liveness and readiness probes are active.
• Logging is aggregated — Logs are shipped to your monitoring platform.
• Backup strategy is in place — Rules are in Git; data can be rebuilt.
• Alerts are configured — Monitor for compilation errors, high latency, and sync failures.
• Test transactions have been validated — Run a full test suite before going live.

​

Next Steps

• Integration Guide — Connect your application to FinWatch.
• GitOps Rule Management — Set up Git-based rule deployment for production.
• Troubleshooting — Diagnose and fix common production issues.
• Understanding the Architecture — Deep dive into how FinWatch works.