> AI agents: this is one page from Mammoth Analytics documentation. The index of all pages as Markdown is https://docs.mammoth.io/llms.txt. Append `.md` to any docs URL, or send `Accept: text/markdown`, to get Markdown.

# Dataset lifecycle workflows

> Manage the dataset lifecycle in Mammoth with dev, staging and production separation, View versioning, change control, rollback plans and deprecation.

This guide is for data architects and administrators who run Mammoth in production. As your deployment matures, you need to manage datasets over time: test changes before they affect production dashboards, version Views that other pipelines depend on, and deprecate old datasets without breaking downstream consumers. It covers environment separation, versioning, branching, change management, and deprecation.

![Promotion path: Dev (sample data), Staging (full-volume validation), Production (versioned View with change control).](https://docs.mammoth.io/api/v1/images/20260805_052024_dataset-workflows-environment-promotion1.jpg)

## Environment separation

### The three-environment model

For production-grade Mammoth deployments, separate work into three logical environments:

**Development (Dev):**

- Where data engineers build and test new pipelines
- Uses a sample of production data (not full volume)
- Outputs go to Dev-prefixed datasets: `dev_sales_summary`
- No external stakeholders access Dev datasets

**Staging:**

- Where new pipelines are validated before production
- Uses recent full-volume data
- Outputs go to Staging-prefixed datasets: `stg_sales_summary`
- Internal stakeholders (analytics team) validate accuracy

**Production:**

- Where dashboards and downstream systems consume data
- Runs on the full dataset with the full schedule
- Outputs are versioned Views: `sales_summary_v3`
- Change control applies: no direct edits to production pipelines

### Implementing environments in Mammoth

Mammoth has no built-in environments, but you can implement them with folder structure and naming conventions:

```plaintext
Data Library/
├── _dev/
│   ├── [engineer_name]/
│   │   └── pipeline-in-progress/
├── _staging/
│   └── validated-before-promotion/
└── Shared/          # Production layer
    └── sales_summary_v3/
```

Mammoth's role-based access is project-level, not per-folder, so a folder naming convention alone does not stop someone with edit access to the project from writing to `Shared/`. Project roles are Project Admin and Project Editor, and both can change data in the project. To actually enforce the separation, put Dev/Staging in a project scoped to the data engineering team, and add only the people who should be able to promote to the Shared project as its members; give other teams access to the results through dashboards shared as Dashboard Viewer, or through an export.

## Dataset versioning

### When to create a new version

Create a new version of a Shared View when:

- The schema changes (columns added, renamed, or removed)
- The business logic changes in a way that affects values (not just performance)
- You want to test the new logic against the old in parallel before switching

**Do not create a new version for:**

- Performance improvements with identical output
- Bug fixes that correct wrong values (these should update the existing version, with documentation)
- Adding columns that don't affect existing columns

### Version naming

Use semantic versioning in View names:

- `customer_master_v1` → `customer_master_v2` for breaking schema changes
- For minor additions that don't break consumers: document the change in the description, don't bump the version

### Version migration process

1. Build the new version in `_dev/`, test thoroughly
2. Promote to `_staging/`, and have existing consumers validate the new version against their expected outputs
3. Announce to consumers: "v2 is available in staging, v1 will be deprecated on [date]"
4. Promote v2 to `Shared/`
5. Give consumers 30 days to migrate their pipelines from v1 to v2
6. After migration deadline: mark v1 as deprecated in its description, stop refreshing it
7. Confirm no pipeline still references v1 as a source, then archive it. Mammoth's Activity Log records edit, create, and delete events, not passive "last viewed" access, so checking pipeline references is the reliable way to confirm nothing depends on it

## Pipeline branching for experimentation

When you want to try a new approach without touching the production pipeline:

### Branch-and-compare pattern

1. Duplicate the production pipeline: `sales_kpi_v3` → `sales_kpi_v3_experiment`
2. Make changes to the experiment version
3. Run both in parallel on the same input data
4. Add a **comparison View** that joins both outputs on the primary key and highlights differences
5. If the experiment is better: promote it, deprecate the old version

### A/B logic testing

For testing two different business logic implementations:

1. Run both logic pipelines on the same dataset
2. The comparison View shows: rows where outputs agree, rows where they differ, magnitude of difference
3. Present both options to business stakeholders—let them choose which is correct
4. Document the decision in the winner's pipeline description

## Change management for production pipelines

### Promotion checklist

Before promoting a pipeline change to production:

- [ ] Tested on sample data in Dev
- [ ] Tested on full recent data in Staging
- [ ] Output compared to previous version (no unexpected changes in values)
- [ ] Row count within expected range
- [ ] Validation/quality checks pass
- [ ] All downstream consumers notified (if schema changed)
- [ ] Pipeline description updated with what changed and why
- [ ] Rollback plan documented (can you revert to previous version if issues appear?)

### Rollback plan

For every production pipeline, document:

- What the output looks like if the current run fails (does the View use stale data or produce nothing?)
- How to revert to the previous version (do you have a copy of the previous pipeline configuration?)

For critical pipelines, keep the previous version's output in a `_backup/` dataset until the new version has run successfully 3 times.

## Dataset deprecation

### Steps to retire a dataset

1. **Identify consumers:** Mammoth's Activity Log records mutation events (created, edited, renamed), not passive access, so it cannot tell you who is merely reading a dataset. Instead, check which pipelines and dashboards still reference it as a source; a dataset with no upstream references is a real deprecation candidate.
2. **Announce deprecation:** Add to the dataset description: "DEPRECATED: will be removed on [date]. Use [replacement] instead."
3. **Notify direct consumers:** If you know who uses the dataset, message them directly.
4. **Wait period:** 30 days minimum for non-critical datasets; 60 days for anything with known consumers.
5. **Archive:** Stop the refreshing automation but keep the dataset in a `_archive/` folder for 90 days in case questions arise.
6. **Delete:** After 90 days in archive with no pipeline referencing it, delete the dataset.

### Tracking deprecated datasets

Maintain a deprecation log in the master data catalog document:

| Dataset | Deprecated | Reason | Replacement | Removal date |
| --- | --- | --- | --- | --- |
| `raw_sf_leads_v1` | 2026-03-01 | Schema change | `raw_sf_leads_v2` | 2026-06-01 |
| `marketing_weekly_old` | 2026-02-15 | No longer used | None | 2026-05-15 |

## Handling breaking changes gracefully

When you must make a breaking change (unavoidable schema change that affects downstream consumers):

1. **3-week notice rule:** Announce breaking changes at least 3 weeks in advance
2. **Parallel period:** Run both old and new versions for 2 weeks
3. **Migration support:** Offer to help consumers update their pipelines (most are small changes to a source reference)
4. **Verified migration:** Before removing the old version, verify no pipeline still references it as a source. The Activity Log does not show read access, only edit, create, and delete events

The goal is to make breaking changes manageable, not to avoid them entirely.

---
Source: https://docs.mammoth.io/guides/advanced/dataset-workflows.md · Updated: 2026-10-13