Incremental Sync & Watermarks

What a watermark is

A watermark stores the highest value of one column (such as updated_at or an increasing id) already processed from a source. An incremental connector run reads only rows above it.

Running an incremental read

POST /api/v1/process-connector
{ "source_type": "sql", "credential_id": "cred-uuid", "table": "orders",
  "y_columns": ["churned"], "incremental": true, "watermark_column": "updated_at" }

• The watermark is found by connector type and source key (the credential_id, or a source_key you pass).
• First run: everything is read. Later runs: only rows above the stored value.
• No new rows: 200 with session_id: null.
• The new value is the highest in the rows read, saved only when the run completes, so a failed run re-reads the same rows.
• Needs both incremental and watermark_column, and a table (a custom_query ignores watermarks).

The dashboard has no incremental option, and the Python SDK's process_from_connector(incremental=True) does not currently send a watermark column, so use the REST call. Scheduled runs always read the full table.

Supported connectors

sql, snowflake, bigquery, redshift, oracle, sap_hana, clickhouse, timescaledb, synapse, databricks, fabric, cassandra, dynamodb, elasticsearch. Others read everything.

The Sync tab

• Manual Watermark: pick a connection (fills connector type and source key), then table, column, value and type (Timestamp, Integer or String; default Timestamp). A duplicate for the same source is refused.
• Edit value: the next incremental run resumes after it.
• Reset: clears the value so the next run reads everything, but keeps the watermark.

Watermarks also drive watermark-advance triggers.

Fields

Key	Meaning
connector_type	Required.
source_key	Required; incremental runs use the credential id.
table_name
watermark_column
watermark_value	Null reads everything.
watermark_type	timestamp, integer, number or string.
row_count	Rows in the last sync.
last_sync_at	When it last advanced.

API

Method	Path	Notes
GET	/api/v1/sync/watermarks/list	All watermarks.
GET	/api/sync/watermarks?connector_type=&source_key=	One watermark.
POST	/api/sync/watermarks	Create; 409 if it exists.
PUT	/api/sync/watermarks	Create or update by connector_type and source_key.
DELETE	/api/sync/watermarks?connector_type=&source_key=	Delete.
POST	/api/v1/sync/watermarks/{id}/reset	Clear the value.

Python SDK: list_watermarks(), reset_watermark(watermark_id).