Troubleshooting - FinWatch

A comprehensive guide to diagnosing and resolving the most common issues you’ll encounter when working with FinWatch. Each section describes the symptom, the likely root cause, and the fix.

Rule Not Triggering

Your rule is deployed and compiled, but transactions that should match it aren’t producing verdicts.

Symptom: Rule compiles successfully but never fires

Cause 1: Field name typo The most common cause. The dig() function silently returns nil for non-existent fields, causing the condition to evaluate to false without any error.

// BUG: "ammount" is misspelled — this condition is ALWAYS false
when ammount > 10000

// FIX:
when amount > 10000

How to diagnose: Log the raw transaction JSON and verify the exact field names. Pay special attention to:

amount vs ammount
metadata vs meta_data (FinWatch uses metadata internally, but the JSON field is meta_data)
Nested field paths: metadata.destination_country requires the transaction to include "meta_data": { "destination_country": "US" }

Cause 2: Wrong operator Using == when you mean !=, or > when you mean >=:

// BUG: This only triggers when amount is EXACTLY 10000
when amount == 10000

// FIX: You probably want "greater than"
when amount > 10000

Cause 3: Type mismatch The field value is a string but you’re comparing it as a number, or vice versa:

// BUG: metadata.kyc_tier might be the STRING "1", not the NUMBER 1
// String "1" == number 1 works due to type coercion, but...
// String "1" > number 0 may not work as expected
when metadata.kyc_tier > 0

// FIX: Use == for string comparisons
when metadata.kyc_tier == "1"

How to diagnose: Check the raw JSON to see whether the value is quoted ("1") or unquoted (1). FinWatch’s toFloat() function can parse string numbers, but the behavior depends on the comparison operator. Cause 4: Aggregate returning 0 Aggregate functions like count() and sum() query historical data in DuckDB. If the database has no historical transactions (e.g., fresh deployment, no watermark sync), aggregates will always return 0.

// This will never trigger if there are no historical transactions
when count(when source == $current.source, "PT24H") > 10

How to diagnose:

Check if historical data exists: curl http://localhost:8081/instructions should show your rules.
Inject several test transactions first to build up history.
If using BLNK_DSN, verify the watermark sync is running and data is flowing.

Cause 5: $current placeholder not resolving If the current transaction is missing the field referenced by $current.*, the placeholder resolves to nil and the filter matches nothing:

// If the transaction has no "source" field, this resolves to:
// count(when source == nil, "PT24H") → always 0
when count(when source == $current.source, "PT24H") > 10

How to diagnose: Ensure the transaction payload includes the field referenced by $current. A transaction with an empty source field will resolve differently than one where the field is missing entirely.

Parse Errors

FinWatch logs parse errors when a .ws file cannot be compiled. The error includes the line number, column number, and a description.

Common Parse Errors and Fixes

Error: expected verdict (allow, block, review)

ERR Failed to compile watch script error="parse error at line 6, column 8: expected verdict (allow, block, review)"

Cause: The first token after then is not a recognized verdict. Valid verdicts: allow, block, review, approve, deny, alert

// BAD: "flag" is not a valid verdict
then flag
     score 0.5

// FIX:
then review
     score 0.5

Error: expected number after 'score'

ERR parse error at line 7, column 15: expected number after 'score'

Cause: The value after score is not a valid number.

// BAD: "high" is not a number
then review
     score high

// FIX:
then review
     score 0.8

Error: expected string after 'reason'

ERR parse error at line 8, column 12: expected string after 'reason'

Cause: The reason is not wrapped in double quotes.

// BAD: Missing quotes
then review
     score 0.5
     reason Transaction is suspicious

// FIX:
then review
     score 0.5
     reason "Transaction is suspicious"

Error: unexpected token in then clause

ERR parse error at line 8, column 6: unexpected token in then clause: verdict

Cause: An unrecognized keyword appears inside the then block. Only score and reason are valid after the verdict.

// BAD: "level" is not a recognized keyword
then review
     score 0.5
     level "high"

// FIX: Remove the invalid keyword
then review
     score 0.5
     reason "High risk transaction"

Missing Braces

// BAD: Missing closing brace
rule MyRule {
    description "Test"
    when amount > 100
    then review
         score 0.5

// FIX: Add the closing brace
rule MyRule {
    description "Test"
    when amount > 100
    then review
         score 0.5
}

Unterminated String

// BAD: Missing closing quote
description "This string never ends

// FIX:
description "This string never ends"

Performance Issues

Symptom: High latency on `POST /inject`

Cause 1: Large aggregate time windows Aggregate functions with large time windows ("P30D", "P7D") scan more data. The larger the window and the more transactions in DuckDB, the slower the query. Fix:

Use the smallest time window that meets your detection needs.
Apply the gate-and-probe pattern: put cheap conditions before aggregates in your and chain.

// SLOW: count() runs on EVERY transaction
when count(when source == $current.source, "P30D") > 100

// FAST: amount check filters out most transactions before count() runs
when amount > 100
 and count(when source == $current.source, "P30D") > 100

Cause 2: Complex regex on long fields Regular expressions on long description fields or large metadata values can be slow. Fix:

Keep regex patterns simple. Use alternation (a|b|c) instead of complex nested groups.
Anchor patterns with ^ or $ when possible.
Gate regex checks with a cheap condition first.

Cause 3: DuckDB memory pressure If DuckDB’s working set exceeds the configured memory_limit, it spills to disk, dramatically slowing queries. Fix:

Increase FINWATCH_MEMORY_LIMIT (e.g., from 2GiB to 4GiB or 8GiB).
Ensure the temp directory (blnk_agent/duckdb_temp/) is on SSD storage.
Implement data retention: delete old transactions that are no longer needed for rule evaluation.

Cause 4: Too many rules with unique aggregates Each unique aggregate (metric, time_window, filter_field, filter_value) tuple generates a separate SQL query. If you have 50 rules each with different aggregate configurations, that’s 50 SQL queries per transaction. Fix:

Consolidate rules that check the same metric. If multiple rules check count(when source == $current.source, "PT24H"), the batch aggregate context only queries once.
Review your rule set and eliminate redundant or overly specific rules.

Data Sync Issues

Symptom: Watermark sync not advancing

Cause 1: PostgreSQL connection failure The BLNK_DSN connection string is incorrect or the database is unreachable. How to diagnose: Check the logs for PostgreSQL connection errors:

ERR Failed to sync data error="failed to connect to PostgreSQL: connection refused"

Fix:

Verify the connection string: postgres://user:password@host:5432/dbname?sslmode=disable
Check network connectivity between FinWatch and the PostgreSQL host.
Verify the database user has SELECT permissions on the required tables.

Cause 2: Schema mismatch The PostgreSQL schema has changed (columns renamed, tables dropped) but the sync query still references the old schema. How to diagnose: Check logs for SQL errors referencing specific columns or tables. Fix:

Ensure the PostgreSQL schema matches what FinWatch expects.
Check the watermark sync configuration for the correct table and column names.

Cause 3: Watermark stuck at old timestamp The watermark advances based on the created_at timestamp of synced records. If new records have timestamps older than the watermark (e.g., backdated transactions), they’ll be skipped. Fix:

Ensure all new records have created_at timestamps that are greater than or equal to the current time.
If backdated records need to be synced, reset the watermark by deleting the sync_watermark entry in DuckDB.

For detailed sync mechanics, see the Watermark Sync Documentation.

Memory Issues

Symptom: DuckDB exceeding memory limit

How to diagnose: Watch for log messages about memory pressure or check container memory usage:

# Docker
docker stats finwatch

# Kubernetes
kubectl top pod finwatch-0

Cause 1: Memory limit too low for transaction volume As your transaction volume grows, DuckDB needs more memory for its buffer pool and query execution. Fix: Increase the memory limit:

export FINWATCH_MEMORY_LIMIT="4GiB"  # or 8GiB, 16GiB

Also increase the Docker/Kubernetes memory limit to accommodate the Go runtime overhead (add 1-2 GB above the DuckDB limit):

# If FINWATCH_MEMORY_LIMIT=4GiB, set container limit to 6Gi
resources:
  limits:
    memory: "6Gi"

Cause 2: Temp directory on insufficient storage When DuckDB spills to disk, it writes to blnk_agent/duckdb_temp/. If this directory runs out of space, queries fail. Fix:

Ensure the temp directory has at least 2x the memory_limit in free disk space.
Mount the temp directory on SSD for better spill performance.

Cause 3: Long-running aggregate queries Aggregate queries over very large time windows ("P30D" on millions of transactions) can consume significant memory during execution. Fix:

Reduce time windows where possible.
Add simple condition gates before aggregate checks.
Consider whether you need the full 30-day window or if 7 days would be sufficient.

Connection Issues

Symptom: Port conflict on startup

FATAL Failed to start server error="listen tcp :8081: bind: address already in use"

Cause: Another process is already using port 8081. Fix:

# Find the process using port 8081
lsof -i :8081

# Option 1: Kill the conflicting process
kill <PID>

# Option 2: Use a different port
export FINWATCH_PORT="8082"

Symptom: WebSocket tunnel disconnections

ERR WebSocket tunnel not connected

Cause: The connection to the Blnk Cloud dashboard has been lost. This can happen due to network interruptions, server restarts, or firewall changes. Impact: Anomaly notifications are not sent to the dashboard. Transaction processing continues normally — verdicts are still computed and logged locally. Fix:

The tunnel automatically reconnects. Check if reconnection is happening by monitoring logs.
Verify outbound HTTPS connectivity from the FinWatch server to the Blnk Cloud endpoint.
Check firewall rules for WebSocket (wss://) connections.

Symptom: API timeouts from your application

Cause: FinWatch is taking too long to respond to POST /inject requests. Fix:

Check if the issue is consistent or intermittent. Intermittent timeouts usually indicate DuckDB memory pressure or spill-to-disk.
Review the Performance Issues section.
Set appropriate timeouts in your application (recommended: 5 seconds).
Implement a fallback: if FinWatch doesn’t respond in time, allow the transaction and log the timeout for later review.

Git Sync Issues

Symptom: Repository clone fails

FATAL Failed to clone or update Git repository error="failed to clone repository: authentication required"

Cause: The Git repository URL requires authentication that isn’t configured. Fix:

For HTTPS repos, include credentials in the URL: https://token:ghp_xxxx@github.com/org/repo.git
For SSH repos, ensure the SSH key is available in the container/server.
For public repos, verify the URL is correct and the repo exists.

Symptom: Rules not updating after merge

Cause 1: The Git polling interval hasn’t elapsed yet. By default, FinWatch checks for updates every 30 seconds. Fix: Trigger an immediate sync:

curl -X POST http://localhost:8081/git/sync

Cause 2: The branch configuration doesn’t match. Fix: Verify WATCH_SCRIPT_GIT_BRANCH matches the branch you merged to:

# Check current configuration
echo $WATCH_SCRIPT_GIT_BRANCH

Symptom: Git not installed

FATAL Git is not installed. Please install Git to use Git repository features.

Fix: Install Git in the container or on the server:

# Debian/Ubuntu
apt-get install -y git

# Alpine (Docker)
apk add git

If using Docker, ensure your Dockerfile includes Git.

Diagnostic Commands

Quick Health Check

# Is FinWatch running?
curl -s http://localhost:8081/instructions | head -c 200

# How many rules are active?
curl -s http://localhost:8081/instructions | python3 -c "import sys,json; print(len(json.load(sys.stdin)))"

# Git sync status
curl -s http://localhost:8081/git/status

Test a Specific Rule

Inject a transaction designed to trigger your rule and watch the logs:

# Inject a test transaction
curl -X POST http://localhost:8081/inject \
  -H "Content-Type: application/json" \
  -d '{
    "transaction_id": "test_debug_001",
    "amount": 99999,
    "currency": "USD",
    "source": "test_source",
    "destination": "test_dest",
    "reference": "debug_ref_001",
    "meta_data": { "destination_country": "IR" }
  }'

# Watch the logs (Docker)
docker logs -f finwatch --tail 50

Check DuckDB Data

If you need to verify what’s in the database, use the instructions API to check compiled rules:

# Get all instructions (compiled rules)
curl -s http://localhost:8081/instructions | python3 -m json.tool

# Get a specific instruction
curl -s http://localhost:8081/instructions/1 | python3 -m json.tool

Check Transaction Storage

Inject a transaction and verify it was stored:

# Inject
curl -X POST http://localhost:8081/inject \
  -H "Content-Type: application/json" \
  -d '{"amount": 100, "currency": "USD", "reference": "test_storage_001"}'

# Retrieve by ID (if you know the transaction_id)
curl -s http://localhost:8081/transactions/test_storage_001

Getting Help

If you’ve worked through this troubleshooting guide and the issue persists:

Collect logs. Capture the full FinWatch log output around the time of the issue.
Reproduce the issue. Create a minimal .ws rule and a specific transaction payload that demonstrates the problem.
Check the rule JSON. Use GET /instructions to see the compiled JSON representation of your rule. This can reveal issues the .ws source doesn’t make obvious.
Review the DSL Reference. The DSL Reference is the authoritative specification for all syntax and behavior.
Open an issue. If you believe you’ve found a bug, open a GitHub issue with:
- The .ws rule file
- The transaction JSON payload
- The expected behavior
- The actual behavior
- FinWatch version and environment details

Next Steps

Getting Started — Revisit the basics.
Conditions Deep Dive — Ensure your conditions are correct.
Production Deployment — Optimize your production configuration.
DSL Reference — The complete language specification.
API Documentation — Full REST API reference.

​Rule Not Triggering

​Symptom: Rule compiles successfully but never fires

​Parse Errors

​Common Parse Errors and Fixes

​Missing Braces

​Unterminated String

​Performance Issues

​Symptom: High latency on POST /inject

​Data Sync Issues

​Symptom: Watermark sync not advancing

​Memory Issues

​Symptom: DuckDB exceeding memory limit

​Connection Issues

​Symptom: Port conflict on startup

​Symptom: WebSocket tunnel disconnections

​Symptom: API timeouts from your application

​Git Sync Issues

​Symptom: Repository clone fails

​Symptom: Rules not updating after merge

​Symptom: Git not installed

​Diagnostic Commands

​Quick Health Check

​Test a Specific Rule

​Check DuckDB Data

​Check Transaction Storage

​Getting Help

​Next Steps