> AI agents: this is one page from Mammoth Analytics documentation. The index of all pages as Markdown is https://docs.mammoth.io/llms.txt. Append `.md` to any docs URL, or send `Accept: text/markdown`, to get Markdown.

# Pipeline optimization

> Speed up Mammoth pipelines: filter early, order tasks from light to heavy, remove unused columns, avoid join fan-out, and use Draft mode.

Two techniques account for most performance gains: **filter early** and **sequence transforms from light to heavy**. Everything else refines those two principles. This page is for analysts and technical users who build pipelines on large datasets. Start with [Filter early](#core-principle-1-filter-early).

## Understanding pipeline execution

Mammoth executes your pipeline top to bottom, passing the full row set through each transformation in sequence. Every row that exists at task N gets processed at task N+1. Row count at each stage directly controls how much work later tasks must do.

**The implication:** A Conditional Filter at task 2 that removes 70% of rows makes every task after task 2 process only 30% of the data.

## Core principle 1: Filter early

Place **Conditional Filter** tasks as early as possible in your pipeline—ideally as the first or second transformation after Source.

**Slow**:

```plaintext
Source: 2M rows
→ Join with customer table (2M × customer table)
→ Math Function: calculate margin (2M rows)
→ Conditional Filter: keep active accounts only (removes 60%)
→ Group & Aggregate (800K rows)
```

**Optimized**:

```plaintext
Source: 2M rows
→ Conditional Filter: keep active accounts only (→ 800K rows)
→ Join with customer table (800K × customer table — 60% less work)
→ Math Function: calculate margin (800K rows)
→ Group & Aggregate (800K rows)
```

The filter moved from task 4 to task 1. The Join now operates on 800K rows instead of 2M, a 60% reduction in join cost. Every later task benefits.

**Combine multiple conditions into one filter.** A single Conditional Filter with several conditions keeps the pipeline short and easier to read than several separate filters in a row.

## Core principle 2: Sequence transforms from light to heavy

Order transformations so the cheapest operations run first and reduce the dataset before expensive operations process it.

| Category | Transforms | Why |
| --- | --- | --- |
| **Light—run early** | Conditional Filter, Remove Column, Convert Column Type | Reduce rows or columns, or change a type |
| **Medium** | Text Formatting, Label & Insert Values, Fill Missing Values, Bulk Replace | Work through every remaining row |
| **Heavy—run late** | Join, Lookup, Group & Pivot, Window function, Math Function on many columns | Combine tables or compute across rows |

**Recommended sequence for most pipelines:**

1. Conditional Filter (reduce rows)
2. Remove Column / Column selection (reduce width)
3. Data type conversions
4. Text standardization (Bulk Replace, Text Formatting)
5. Enrichment (Join, Lookup)
6. Calculations (Math Function, Add Column)
7. Aggregation (Group & Pivot, Window function)

## Core principle 3: Remove unused columns early

Wide datasets carry more data through every later task. Use **Remove Column** right after Source to keep only the columns your pipeline actually needs.

**Example:** A source with 47 columns but your analysis needs only 10. Removing the 37 unused columns immediately after Source means every downstream task handles less data.

## Join optimization

Joins are the most common performance bottleneck. Three rules apply:

**1. Filter both datasets before joining.**\
The Join operation processes every row in dataset A against every row in dataset B that shares the join key. Reducing either input directly reduces join work.

**2. Use Lookup instead of Join for single-column key matching.**\
When you're matching on a single column and expect one-to-one matches (for example, adding a product name from a reference table), Lookup avoids accidental row multiplication.

**3. Check for duplicate join keys.**\
If your join key has duplicates on either side, the result can multiply rows (a "fan-out"). A dataset with 100K rows joined to a reference table where each key appears 5 times produces 500K rows—every later task processes 5× the expected volume. Use **Remove Duplicate Rows** on the reference dataset before joining if needed.

**Mammoth rejects a join that bloats the row count.** Join estimates the resulting row count before running. When the projected output exceeds both 100,000 rows and 5× the larger of your two input row counts, the join fails with a row-count error instead of running. If you see that error, check for duplicate keys on both sides and remove duplicates or filter first. Lookup doesn't need this safeguard—it's designed to always return at most one match per input row, so it can never fan out row counts, which is the other reason to prefer it over Join for simple one-to-one enrichment.

## Draft mode

Draft mode keeps new pipeline tasks pending and applies them all at once when you're ready, instead of running the full pipeline after every individual addition.

**When Draft mode helps:**

- Building several new tasks on a large dataset
- Redesigning a section of an existing pipeline
- Testing a new transformation sequence before committing

**How it works:**

1. Open the pipeline view and turn on **Draft mode** in the **More Pipeline options** (`...`) menu in the pipeline toolbar
2. Add your new transformations. They stay pending and don't take effect yet.
3. When ready, open **Pending Updates** and click **Apply now**. The pending tasks are applied together. **Review** shows the pending changes, and **Discard** drops them.

**Expected result**: The pipeline runs once with all the new tasks applied.

## Structuring long pipelines

Pipelines with many tasks benefit from splitting into multiple views, where each view handles a distinct processing stage—both for performance and for maintainability. Use intermediate **Send to Dataset** tasks if a complex sub-pipeline is shared by multiple views.

**Architecture pattern:**

```plaintext
Source dataset
└── View 1: Data quality (filter, clean, standardize)
    └── View 2: Enrichment (joins, lookups)
        └── View 3: Calculations and output
```

Each view can be independently monitored and run. This also makes troubleshooting easier—if a stage fails, the error is isolated.

## Monitoring pipeline performance

Open **Monitor** from the left sidebar. The **Running** tab shows pipelines, tasks, and automations that are executing or queued, and flags a task as **Stuck** if it runs unusually long. The **Needs Attention** tab lists failed runs.

### When a pipeline is consistently slow

1. Check **Monitor** for a **Stuck** status or failed runs
2. Look for the slowest task: Join operations are the most common culprit
3. Check for duplicate join keys that might be causing row fan-out
4. Move any Conditional Filter tasks earlier in the sequence
5. Contact Mammoth support with the pipeline ID if the issue persists. The support team can inspect server-side execution logs

See [Monitor](https://docs.mammoth.io/learn/monitor/) for details.

---
Source: https://docs.mammoth.io/learn/performance/pipeline-optimization.md · Updated: 2026-10-13