> AI agents: this is one page from Mammoth Analytics documentation. The index of all pages as Markdown is https://docs.mammoth.io/llms.txt. Append `.md` to any docs URL, or send `Accept: text/markdown`, to get Markdown.

# Connector performance and best practices

> Make Live Connection refreshes faster and more reliable: select only needed columns, pull incrementally, schedule off-peak, and troubleshoot timeouts.

Live Connections let Mammoth query your databases, SaaS tools, and cloud storage directly at refresh time. How you configure those connections, including what data you request and when, affects how fast your pipelines run and how reliably your automations complete. This page is for technical users and administrators who manage Live Connections. Start with [Reduce data at the source](#reduce-data-at-the-source).

## How Live Connection refresh works

When a Dataset Refresh automation runs, Mammoth queries your source system and loads the result into the connected dataset. The entire pipeline attached to that dataset then re-executes on the new data.

This means two factors control performance:

1. **How much data is transferred** from the source system to Mammoth
2. **How the pipeline processes** that data after it arrives

This page focuses on factor 1. For pipeline processing, see [Pipeline optimization](https://docs.mammoth.io/learn/performance/pipeline-optimization).

## Reduce data at the source

The most effective optimization for Live Connections is requesting less data from the source system, not processing it faster after it arrives.

### What "at the source" means in Mammoth

A pipeline's **Conditional Filter** task runs against the copy of your data that a Dataset Refresh already pulled into Mammoth—it does not reach back and re-query your source system. Placing a Conditional Filter early in the pipeline still matters a great deal for processing speed (see [Pipeline optimization](https://docs.mammoth.io/learn/performance/pipeline-optimization#core-principle-1-filter-early)), but it does not, by itself, reduce how much data is transferred from your database on each refresh.

The two levers that actually reduce data transferred from the source are **incremental refresh** (below) and **column selection** (below). Both are configured on the connection/automation itself, not on the pipeline.

### Select only the columns you need

If your Live Connection pulls all columns from a wide table but your pipeline only uses 8 of 45 columns, you're transferring 37 columns of data on every refresh for no benefit.

**For SQL connectors:** Edit the connection query to select only the columns your pipeline uses. Remove any `SELECT *` patterns and explicitly list the needed columns.

**For SaaS connectors (such as Salesforce):** Use field selection in the connector configuration to limit which fields are returned by the API.

Reducing column count directly reduces transfer size and downstream processing overhead.

## Incremental refresh

For datasets that grow over time, such as transaction logs, event streams, and CRM records, refreshing the full table on every run is wasteful. Incremental refresh pulls only new or changed records instead of the full table.

**Native incremental pull (where supported):** for connectors that support it, set the **Update method** to **Add new data** and choose a **Key column**, a column with incremental values such as an ID or date. New data is added based on that column instead of replacing the whole dataset. Check your connector's configuration screen for this option.

**Parameter-driven fallback:** if your connector doesn't expose a native incremental option, you can approximate the same effect at the pipeline level:

**Prerequisite**: You have a Live Connection and can edit its dataset's pipeline.

1. Create a Dataset Refresh automation for your Live Connection
2. In the pipeline attached to the dataset, add a **Conditional Filter** as the first task with a date condition (for example, `created_date > last_30_days`, or a Mammoth parameter tracking the last run timestamp)
3. Schedule the automation at the frequency you need

**Expected result**: Each refresh passes only recent rows through the rest of the pipeline.

Note that this fallback still pulls the full table from the source on every refresh. The filter reduces what flows through the rest of your pipeline, not what's transferred from the source. See [above](#what-at-the-source-means-in-mammoth). Prefer the native option whenever your connector supports it.

Related: [Dataset Refresh](https://docs.mammoth.io/learn/automations/dataset-refresh/) and [Automations](https://docs.mammoth.io/learn/automations/).

## Scheduling for performance

When you schedule Dataset Refresh automations matters, both for your own pipeline performance and to avoid impact on your source systems.

### Off-peak scheduling

| Time window | Recommended use |
| --- | --- |
| Business hours | Light refreshes only; critical updates |
| Evening / overnight | Large dataset refreshes, historical backfills, full-table pulls |
| Weekends | Weekly aggregations, full refresh operations |

**Why it matters:** Large source systems (ERP, databases, Salesforce) often have maintenance windows, rate limits, and peak-usage periods. Scheduling Mammoth refreshes during off-peak hours reduces contention and improves reliability.

### Stagger concurrent automations

If you have multiple Dataset Refresh automations, avoid scheduling them all at the same time. A pile-up of simultaneous refreshes competes for the same source-system connections and processing capacity. Staggering start times keeps each refresh completing within its expected window.

### Run related refreshes in sequence

When one dataset depends on another (for example, a customer master that feeds into a transaction analysis dataset), select both datasets in a single Dataset Refresh automation and choose **In a sequence**. The refreshes then run one after another in the numbered order, which you can change by dragging. **In parallel** runs all refreshes at once.

Related: [Managing automations](https://docs.mammoth.io/learn/automations/managing-automations/).

## Monitoring Live Connection performance

Monitor's Running tab shows currently running automations and pipeline executions. If a Dataset Refresh is taking longer than expected, check Monitor to see its current status and whether it is flagged **Stuck**.

Related: [Monitor](https://docs.mammoth.io/learn/monitor/) and the [Activity Log reference](https://docs.mammoth.io/learn/monitor/activity-log).

## Troubleshooting slow or failing Live Connections

### Connection timing out

**Symptom:** Dataset Refresh fails with a timeout error.

**Likely cause:** The source query is pulling too much data, or the source system itself is under load. Timeout thresholds vary by connector (for example, BigQuery allows up to 5 minutes for a query and up to 1 hour for a long-running job).

**Fix:**

1. Switch to incremental refresh (native, if your connector supports it) so each run pulls less data
2. Reduce the columns requested by the connection to cut transfer size
3. Check whether your source database has indexes on the columns used by your incremental pull or connection query, because unindexed columns cause full table scans
4. If timeouts persist, contact Mammoth support with the connection ID

### Refresh taking longer each week

**Symptom:** A Dataset Refresh that used to complete in 3 minutes now takes 15 minutes.

**Likely cause:** The source table is growing and you're doing a full pull each time.

**Fix:** Switch to incremental refresh (native if your connector supports it, or the parameter-driven fallback otherwise) so each run only pulls new records instead of the full table.

### Automation fails intermittently

**Symptom:** The same automation succeeds most of the time but fails occasionally.

**Likely cause:** Source system rate limits, maintenance windows, or transient network issues.

**Fix:**

1. Reschedule the automation to avoid source system maintenance windows or peak usage periods
2. Check **Monitor** for failed runs, and re-run the automation once the source is available again

## Best practices checklist

**Connection configuration:**

- [ ] Selected only the columns your pipeline uses (no `SELECT *`)
- [ ] Set up native incremental pull, or a parameter-driven fallback, to limit rows transferred on each refresh
- [ ] Scheduled refreshes during off-peak hours

**Monitoring:**

- [ ] Checked Monitor regularly for failed or stuck refreshes
- [ ] Staggered multiple automations to avoid concurrent queuing

**Pipeline:**

- [ ] Conditional Filter is early in your pipeline (reduces processing time, though not the data transferred from the source; see [Pipeline optimization](https://docs.mammoth.io/learn/performance/pipeline-optimization))

---
Source: https://docs.mammoth.io/learn/performance/connector-optimization.md · Updated: 2026-10-13