> AI agents: this is one page from Mammoth Analytics documentation. The index of all pages as Markdown is https://docs.mammoth.io/llms.txt. Append `.md` to any docs URL, or send `Accept: text/markdown`, to get Markdown.

# Data Consolidation

> Create a Data Consolidation automation that merges files uploaded to a source folder into a destination dataset using Combine or Replace.

Data Consolidation automatically processes files uploaded to a source folder and merges them into a destination dataset. Every time a new file lands in the folder, Mammoth triggers the consolidation—no schedule needed.

## How it works

```plaintext
New file uploaded to source folder
        │
        ▼
Mammoth detects the upload
        │
        ▼
Schema validation (columns must be compatible)
        │
        ▼
Consolidate into destination dataset (Combine or Replace)
```

The consolidation fires on each upload event.

## Before you start

- **Source folder**: A folder in your Data Library that your team uploads files to. Create one if it doesn't exist.
- **Destination dataset**: The dataset that will receive the consolidated data. It must already exist and have a schema compatible with the files you plan to upload.
- **Schema compatibility**: All files uploaded to the source folder must have columns that match the destination dataset's schema. How Mammoth handles mismatches depends on the destination dataset's **If column type doesn't match** and column mapping settings. See [Add or replace data](https://docs.mammoth.io/learn/data-library/add-or-replace-data/).

## Creating a Data Consolidation automation

### Step 1: Open the creation steps

From the **Automations** tab, click **New automation**, then select **Data Consolidation**.

The Data Consolidation steps open: select the destination, select the source folder, then review and save.

![Data Consolidation form with Select folder for the source folder, Select dataset(s), Name and Description fields](https://docs.mammoth.io/api/v1/images/20260418_184240_data-consolidation-form1.jpg)

### Step 2: Select the destination dataset

In the **Select destination** step, choose the dataset that receives the data.

> *"New files picked up by this automation will be added to this dataset."*

The destination dataset must have a schema that's compatible with the files in your source folder.

---

### Step 3: Select the source folder

In the **Select source folder** step, choose the folder that contains (or will contain) the incoming files.

> *"Files added to this folder are picked up automatically."*

Once selected, the folder name appears. Any file uploaded to this folder after you create the automation will trigger consolidation.

You can also limit which files are processed. Turn on **Only process files matching a name pattern** to pick up only files whose names match a pattern. Matching is not case sensitive. If you leave this off, all files in the folder are processed.

### Step 4: Review and save

Name the automation, check the settings, and click **Save automation**.

## Consolidation modes

Mammoth offers two modes for how incoming data is written to the destination dataset:

| Mode | Behavior | Best for |
| --- | --- | --- |
| **Combine** | Appends rows from the new file to the existing dataset. Data accumulates over time. | Incremental data (daily transactions, event logs) |
| **Replace** | Overwrites the destination dataset with the contents of the new file. Previous data is removed. | Full snapshots (weekly inventory exports) |

> Configure the update method on the destination dataset before creating the automation.

## Schema requirements

Data Consolidation validates column schemas before merging. The columns in uploaded files must be compatible with the destination dataset's schema.

**What Mammoth checks:**

- Column names must match (by display name)
- Data types must be compatible

**If a schema mismatch occurs:**

- Depending on the dataset's settings, Mammoth converts the column to text, converts incompatible values to null, or does not update the dataset
- If the consolidation fails, the automation status shows **Error**

Keep your source files consistent. If different team members upload files with slightly different column names, the columns may not map to the destination dataset. Define a standard template for uploads.

## Monitoring consolidations

Click the automation row to open the detail panel. The **Automation Summary** shows:

- The source folder name
- The destination dataset(s)
- The last run result (Success or failure reason)

The automation status in the list reflects the outcome of the most recent consolidation:

| Status | Meaning |
| --- | --- |
| **Active** | Watching the folder; last consolidation succeeded |
| **Error** | The automation has an error (for example, a deleted source or destination) |
| **Suspended** | Paused—uploads to the folder will not trigger consolidation |

## Troubleshooting

### Status shows Error after a file upload

**Cause:** The uploaded file's columns or data types do not match the destination dataset schema.

**Fix:** Open the detail panel and check the **Last run result**. Compare the uploaded file's columns with the destination dataset schema, then correct the file and upload it again.

### Source folder or destination dataset was deleted

**Cause:** The automation references a folder or dataset that no longer exists. It shows an error indicator.

**Fix:** Edit the automation to point to new resources, or delete it.

---
Source: https://docs.mammoth.io/learn/automations/data-consolidation.md · Updated: 2026-10-03