> ## Documentation Index
> Fetch the complete documentation index at: https://docs.finwatch.finance/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

A comprehensive guide to diagnosing and resolving the most common issues you'll encounter when working with FinWatch. Each section describes the symptom, the likely root cause, and the fix.

***

## Rule Not Triggering

Your rule is deployed and compiled, but transactions that should match it aren't producing verdicts.

### Symptom: Rule compiles successfully but never fires

**Cause 1: Field name typo**

The most common cause. The `dig()` function silently returns `nil` for non-existent fields, causing the condition to evaluate to `false` without any error.

```go theme={null}
// BUG: "ammount" is misspelled — this condition is ALWAYS false
when ammount > 10000

// FIX:
when amount > 10000
```

**How to diagnose:** Log the raw transaction JSON and verify the exact field names. Pay special attention to:

* `amount` vs `ammount`
* `metadata` vs `meta_data` (FinWatch uses `metadata` internally, but the JSON field is `meta_data`)
* Nested field paths: `metadata.destination_country` requires the transaction to include `"meta_data": { "destination_country": "US" }`

**Cause 2: Wrong operator**

Using `==` when you mean `!=`, or `>` when you mean `>=`:

```go theme={null}
// BUG: This only triggers when amount is EXACTLY 10000
when amount == 10000

// FIX: You probably want "greater than"
when amount > 10000
```

**Cause 3: Type mismatch**

The field value is a string but you're comparing it as a number, or vice versa:

```go theme={null}
// BUG: metadata.kyc_tier might be the STRING "1", not the NUMBER 1
// String "1" == number 1 works due to type coercion, but...
// String "1" > number 0 may not work as expected
when metadata.kyc_tier > 0

// FIX: Use == for string comparisons
when metadata.kyc_tier == "1"
```

**How to diagnose:** Check the raw JSON to see whether the value is quoted (`"1"`) or unquoted (`1`). FinWatch's `toFloat()` function can parse string numbers, but the behavior depends on the comparison operator.

**Cause 4: Aggregate returning 0**

Aggregate functions like `count()` and `sum()` query historical data in DuckDB. If the database has no historical transactions (e.g., fresh deployment, no watermark sync), aggregates will always return 0.

```go theme={null}
// This will never trigger if there are no historical transactions
when count(when source == $current.source, "PT24H") > 10
```

**How to diagnose:**

1. Check if historical data exists: `curl http://localhost:8081/instructions` should show your rules.
2. Inject several test transactions first to build up history.
3. If using `BLNK_DSN`, verify the watermark sync is running and data is flowing.

**Cause 5: `$current` placeholder not resolving**

If the current transaction is missing the field referenced by `$current.*`, the placeholder resolves to `nil` and the filter matches nothing:

```go theme={null}
// If the transaction has no "source" field, this resolves to:
// count(when source == nil, "PT24H") → always 0
when count(when source == $current.source, "PT24H") > 10
```

**How to diagnose:** Ensure the transaction payload includes the field referenced by `$current`. A transaction with an empty `source` field will resolve differently than one where the field is missing entirely.

***

## Parse Errors

FinWatch logs parse errors when a `.ws` file cannot be compiled. The error includes the line number, column number, and a description.

### Common Parse Errors and Fixes

**Error: `expected verdict (allow, block, review)`**

```text theme={null}
ERR Failed to compile watch script error="parse error at line 6, column 8: expected verdict (allow, block, review)"
```

**Cause:** The first token after `then` is not a recognized verdict.

**Valid verdicts:** `allow`, `block`, `review`, `approve`, `deny`, `alert`

```go theme={null}
// BAD: "flag" is not a valid verdict
then flag
     score 0.5

// FIX:
then review
     score 0.5
```

***

**Error: `expected number after 'score'`**

```text theme={null}
ERR parse error at line 7, column 15: expected number after 'score'
```

**Cause:** The value after `score` is not a valid number.

```go theme={null}
// BAD: "high" is not a number
then review
     score high

// FIX:
then review
     score 0.8
```

***

**Error: `expected string after 'reason'`**

```text theme={null}
ERR parse error at line 8, column 12: expected string after 'reason'
```

**Cause:** The reason is not wrapped in double quotes.

```go theme={null}
// BAD: Missing quotes
then review
     score 0.5
     reason Transaction is suspicious

// FIX:
then review
     score 0.5
     reason "Transaction is suspicious"
```

***

**Error: `unexpected token in then clause`**

```text theme={null}
ERR parse error at line 8, column 6: unexpected token in then clause: verdict
```

**Cause:** An unrecognized keyword appears inside the `then` block. Only `score` and `reason` are valid after the verdict.

```go theme={null}
// BAD: "level" is not a recognized keyword
then review
     score 0.5
     level "high"

// FIX: Remove the invalid keyword
then review
     score 0.5
     reason "High risk transaction"
```

***

### Missing Braces

```go theme={null}
// BAD: Missing closing brace
rule MyRule {
    description "Test"
    when amount > 100
    then review
         score 0.5

// FIX: Add the closing brace
rule MyRule {
    description "Test"
    when amount > 100
    then review
         score 0.5
}
```

### Unterminated String

```go theme={null}
// BAD: Missing closing quote
description "This string never ends

// FIX:
description "This string never ends"
```

***

## Performance Issues

### Symptom: High latency on `POST /inject`

**Cause 1: Large aggregate time windows**

Aggregate functions with large time windows (`"P30D"`, `"P7D"`) scan more data. The larger the window and the more transactions in DuckDB, the slower the query.

**Fix:**

* Use the smallest time window that meets your detection needs.
* Apply the gate-and-probe pattern: put cheap conditions before aggregates in your `and` chain.

```go theme={null}
// SLOW: count() runs on EVERY transaction
when count(when source == $current.source, "P30D") > 100

// FAST: amount check filters out most transactions before count() runs
when amount > 100
 and count(when source == $current.source, "P30D") > 100
```

**Cause 2: Complex regex on long fields**

Regular expressions on long `description` fields or large metadata values can be slow.

**Fix:**

* Keep regex patterns simple. Use alternation (`a|b|c`) instead of complex nested groups.
* Anchor patterns with `^` or `$` when possible.
* Gate regex checks with a cheap condition first.

**Cause 3: DuckDB memory pressure**

If DuckDB's working set exceeds the configured `memory_limit`, it spills to disk, dramatically slowing queries.

**Fix:**

* Increase `FINWATCH_MEMORY_LIMIT` (e.g., from `2GiB` to `4GiB` or `8GiB`).
* Ensure the temp directory (`blnk_agent/duckdb_temp/`) is on SSD storage.
* Implement data retention: delete old transactions that are no longer needed for rule evaluation.

**Cause 4: Too many rules with unique aggregates**

Each unique aggregate `(metric, time_window, filter_field, filter_value)` tuple generates a separate SQL query. If you have 50 rules each with different aggregate configurations, that's 50 SQL queries per transaction.

**Fix:**

* Consolidate rules that check the same metric. If multiple rules check `count(when source == $current.source, "PT24H")`, the batch aggregate context only queries once.
* Review your rule set and eliminate redundant or overly specific rules.

***

## Data Sync Issues

### Symptom: Watermark sync not advancing

**Cause 1: PostgreSQL connection failure**

The `BLNK_DSN` connection string is incorrect or the database is unreachable.

**How to diagnose:** Check the logs for PostgreSQL connection errors:

```go theme={null}
ERR Failed to sync data error="failed to connect to PostgreSQL: connection refused"
```

**Fix:**

* Verify the connection string: `postgres://user:password@host:5432/dbname?sslmode=disable`
* Check network connectivity between FinWatch and the PostgreSQL host.
* Verify the database user has `SELECT` permissions on the required tables.

**Cause 2: Schema mismatch**

The PostgreSQL schema has changed (columns renamed, tables dropped) but the sync query still references the old schema.

**How to diagnose:** Check logs for SQL errors referencing specific columns or tables.

**Fix:**

* Ensure the PostgreSQL schema matches what FinWatch expects.
* Check the watermark sync configuration for the correct table and column names.

**Cause 3: Watermark stuck at old timestamp**

The watermark advances based on the `created_at` timestamp of synced records. If new records have timestamps older than the watermark (e.g., backdated transactions), they'll be skipped.

**Fix:**

* Ensure all new records have `created_at` timestamps that are greater than or equal to the current time.
* If backdated records need to be synced, reset the watermark by deleting the `sync_watermark` entry in DuckDB.

> For detailed sync mechanics, see the [Watermark Sync Documentation](../WATERMARK_SYNC.md).

***

## Memory Issues

### Symptom: DuckDB exceeding memory limit

**How to diagnose:** Watch for log messages about memory pressure or check container memory usage:

```bash theme={null}
# Docker
docker stats finwatch

# Kubernetes
kubectl top pod finwatch-0
```

**Cause 1: Memory limit too low for transaction volume**

As your transaction volume grows, DuckDB needs more memory for its buffer pool and query execution.

**Fix:** Increase the memory limit:

```bash theme={null}
export FINWATCH_MEMORY_LIMIT="4GiB"  # or 8GiB, 16GiB
```

Also increase the Docker/Kubernetes memory limit to accommodate the Go runtime overhead (add 1-2 GB above the DuckDB limit):

```yaml theme={null}
# If FINWATCH_MEMORY_LIMIT=4GiB, set container limit to 6Gi
resources:
  limits:
    memory: "6Gi"
```

**Cause 2: Temp directory on insufficient storage**

When DuckDB spills to disk, it writes to `blnk_agent/duckdb_temp/`. If this directory runs out of space, queries fail.

**Fix:**

* Ensure the temp directory has at least 2x the `memory_limit` in free disk space.
* Mount the temp directory on SSD for better spill performance.

**Cause 3: Long-running aggregate queries**

Aggregate queries over very large time windows (`"P30D"` on millions of transactions) can consume significant memory during execution.

**Fix:**

* Reduce time windows where possible.
* Add simple condition gates before aggregate checks.
* Consider whether you need the full 30-day window or if 7 days would be sufficient.

***

## Connection Issues

### Symptom: Port conflict on startup

```text theme={null}
FATAL Failed to start server error="listen tcp :8081: bind: address already in use"
```

**Cause:** Another process is already using port 8081.

**Fix:**

```bash theme={null}
# Find the process using port 8081
lsof -i :8081

# Option 1: Kill the conflicting process
kill <PID>

# Option 2: Use a different port
export FINWATCH_PORT="8082"
```

### Symptom: WebSocket tunnel disconnections

```text theme={null}
ERR WebSocket tunnel not connected
```

**Cause:** The connection to the Blnk Cloud dashboard has been lost. This can happen due to network interruptions, server restarts, or firewall changes.

**Impact:** Anomaly notifications are not sent to the dashboard. Transaction processing continues normally — verdicts are still computed and logged locally.

**Fix:**

* The tunnel automatically reconnects. Check if reconnection is happening by monitoring logs.
* Verify outbound HTTPS connectivity from the FinWatch server to the Blnk Cloud endpoint.
* Check firewall rules for WebSocket (wss\://) connections.

### Symptom: API timeouts from your application

**Cause:** FinWatch is taking too long to respond to `POST /inject` requests.

**Fix:**

1. Check if the issue is consistent or intermittent. Intermittent timeouts usually indicate DuckDB memory pressure or spill-to-disk.
2. Review the [Performance Issues](#performance-issues) section.
3. Set appropriate timeouts in your application (recommended: 5 seconds).
4. Implement a fallback: if FinWatch doesn't respond in time, allow the transaction and log the timeout for later review.

***

## Git Sync Issues

### Symptom: Repository clone fails

```go theme={null}
FATAL Failed to clone or update Git repository error="failed to clone repository: authentication required"
```

**Cause:** The Git repository URL requires authentication that isn't configured.

**Fix:**

* For HTTPS repos, include credentials in the URL: `https://token:ghp_xxxx@github.com/org/repo.git`
* For SSH repos, ensure the SSH key is available in the container/server.
* For public repos, verify the URL is correct and the repo exists.

### Symptom: Rules not updating after merge

**Cause 1:** The Git polling interval hasn't elapsed yet. By default, FinWatch checks for updates every 30 seconds.

**Fix:** Trigger an immediate sync:

```bash theme={null}
curl -X POST http://localhost:8081/git/sync
```

**Cause 2:** The branch configuration doesn't match.

**Fix:** Verify `WATCH_SCRIPT_GIT_BRANCH` matches the branch you merged to:

```bash theme={null}
# Check current configuration
echo $WATCH_SCRIPT_GIT_BRANCH
```

### Symptom: Git not installed

```go theme={null}
FATAL Git is not installed. Please install Git to use Git repository features.
```

**Fix:** Install Git in the container or on the server:

```bash theme={null}
# Debian/Ubuntu
apt-get install -y git

# Alpine (Docker)
apk add git
```

If using Docker, ensure your Dockerfile includes Git.

***

## Diagnostic Commands

### Quick Health Check

```bash theme={null}
# Is FinWatch running?
curl -s http://localhost:8081/instructions | head -c 200

# How many rules are active?
curl -s http://localhost:8081/instructions | python3 -c "import sys,json; print(len(json.load(sys.stdin)))"

# Git sync status
curl -s http://localhost:8081/git/status
```

### Test a Specific Rule

Inject a transaction designed to trigger your rule and watch the logs:

```bash theme={null}
# Inject a test transaction
curl -X POST http://localhost:8081/inject \
  -H "Content-Type: application/json" \
  -d '{
    "transaction_id": "test_debug_001",
    "amount": 99999,
    "currency": "USD",
    "source": "test_source",
    "destination": "test_dest",
    "reference": "debug_ref_001",
    "meta_data": { "destination_country": "IR" }
  }'

# Watch the logs (Docker)
docker logs -f finwatch --tail 50
```

### Check DuckDB Data

If you need to verify what's in the database, use the instructions API to check compiled rules:

```bash theme={null}
# Get all instructions (compiled rules)
curl -s http://localhost:8081/instructions | python3 -m json.tool

# Get a specific instruction
curl -s http://localhost:8081/instructions/1 | python3 -m json.tool
```

### Check Transaction Storage

Inject a transaction and verify it was stored:

```bash theme={null}
# Inject
curl -X POST http://localhost:8081/inject \
  -H "Content-Type: application/json" \
  -d '{"amount": 100, "currency": "USD", "reference": "test_storage_001"}'

# Retrieve by ID (if you know the transaction_id)
curl -s http://localhost:8081/transactions/test_storage_001
```

***

## Getting Help

If you've worked through this troubleshooting guide and the issue persists:

1. **Collect logs.** Capture the full FinWatch log output around the time of the issue.
2. **Reproduce the issue.** Create a minimal `.ws` rule and a specific transaction payload that demonstrates the problem.
3. **Check the rule JSON.** Use `GET /instructions` to see the compiled JSON representation of your rule. This can reveal issues the `.ws` source doesn't make obvious.
4. **Review the DSL Reference.** The [DSL Reference](../DSL_REFERENCE.md) is the authoritative specification for all syntax and behavior.
5. **Open an issue.** If you believe you've found a bug, open a GitHub issue with:
   * The `.ws` rule file
   * The transaction JSON payload
   * The expected behavior
   * The actual behavior
   * FinWatch version and environment details

***

## Next Steps

* [**Getting Started**](getting-started.md) — Revisit the basics.
* [**Conditions Deep Dive**](conditions-deep-dive.md) — Ensure your conditions are correct.
* [**Production Deployment**](production-deployment.md) — Optimize your production configuration.
* [**DSL Reference**](../DSL_REFERENCE.md) — The complete language specification.
* [**API Documentation**](../../watch/API_DOCUMENTATION.md) — Full REST API reference.