Guide

Set up on-prem scheduler workers for reliable exports

Q: What environment variables does an on-prem scheduler worker need?

A worker needs DP_SCHEDULER_WORKER_ID and DP_SCHEDULER_WORKER_KEY to authenticate, plus one or more DP_CONN_* variables to access your database connections.

Q: How do DP_CONN_* secrets work?

Each saved connection generates a DP_CONN_* variable name. You copy that exact name from the Connections list, store the encrypted value as a secret, and inject it into the worker container.

Q: Where do downloads appear after an on-prem export?

Exports still show up in the workspace notification panel with a download action, and run history keeps the metadata so teams can audit what ran.

Run schedules inside your network, keep database access private, and still deliver CSV / Excel downloads through the DataPilot notification flow. This guide covers the two worker credentials plus the DP_CONN_* secrets your container needs.

Start free No credit card required. Download the agent

Supported databases: PostgreSQL, SQL Server, MySQL, Amazon Redshift. More databases are coming.

3-step quick start

You’ll do three things: create a worker, map connections to DP_CONN_* variables, and start the container. After that, exports show up in notifications like cloud runs—without moving your database outside the firewall.

1) Provision a worker

In Schedules → Workers, create a worker and copy both values: DP_SCHEDULER_WORKER_ID and DP_SCHEDULER_WORKER_KEY. Store the key securely—treat it like a password.

2) Capture connections

Open the Connections list and copy each generated DP_CONN_* name. Pair every name with its encrypted connection string in your secrets manager, then pass them into the same container that runs the worker.

3) Start the worker

Start the container with the env vars in place. The worker will claim pending runs, execute queries near your databases, and publish export results back to the workspace so notifications can deliver downloads.

Worker credentials explained

See local & on-prem execution

A worker authenticates with two exact environment variables: DP_SCHEDULER_WORKER_ID (stable GUID) and DP_SCHEDULER_WORKER_KEY (secret token). If you rotate the key from the worker list, restart the container so it trusts the new secret.

Screenshot placeholder

Add screenshot: worker list/detail showing worker ID, key rotation action, and last seen / heartbeat.

Screenshot needed

Screenshot needed: worker detail showing ID, rotate key controls, and last seen heartbeat.

Sizes: 1600x900 hero + OG 1200x630; Alt: "Scheduler worker detail with rotate key and last seen".

Connections & DP_CONN_* secrets

Catalog & connections

Each connection generates a unique environment variable name like DP_CONN_MYPOSTGRES_ABC123. Copy the exact name from the connection list, store the encrypted value as a secret, and inject it into the worker container. The worker uses these secrets to connect locally while keeping credentials out of the UI.

Screenshot placeholder

Add screenshot: connections list showing DP_CONN_* names with copy action and connection summary.

Screenshot needed

Screenshot needed: connections list showing DP_CONN_* names with copy actions.

Sizes: 1600x900 hero + OG 1200x630; Alt: "Connections list showing DP_CONN env var names".

Operate safely

Keep the worker inside your network and treat the worker key plus DP_CONN_* variables as high-sensitivity secrets. Exports still follow the same delivery loop: run → history → notification → download, so teams can prove what ran and when.

Rotate & monitor

Rotate the worker key from the UI when needed and restart the container. Use worker health and “last seen” plus notification activity to confirm schedules are executing on time.