> AI agents: this is one page from Mammoth Analytics documentation. The index of all pages as Markdown is https://docs.mammoth.io/llms.txt. Append `.md` to any docs URL, or send `Accept: text/markdown`, to get Markdown.

# Pull data from a REST API with the CLI

> Connect a REST or Elasticsearch source with the mammoth CLI, sample it, create a dataset, and refresh it on a schedule. Needs mammoth-cli 2.2.73 or later.

`mammoth-cli` 2.2.73 and later can connect a REST or Elasticsearch source, sample it, create a dataset from it, and refresh that dataset on a schedule. This guide covers the connection, the sample, the dataset and the refresh. The connector key is `generic_rest_api`, and every connector command runs in a project (`--project PROJECT_ID`).

Connections, dataset creation and automations are writes. Confirm each with the person who owns the data before you run it.

```bash
mammoth connector get generic_rest_api
mammoth connector connection list generic_rest_api
```

The first command shows the connection fields and the query keys the connector accepts. The second lists connections you can reuse.

## 1. Create the connection

Put the settings under `config`, with flat auth fields: `auth_type` (`none`, `api_key`, `bearer` or `basic`), then `username` and `password`, or `token`, or `key_name`, `key_value` and `inject_into` (`header` or `query_param`). Don't send a nested `auth` object. A backend with the fix rejects it with error 4CONN003, and an older backend saves the connection as `auth_type: none`, so the API answers 401 later.

The base URL must be a public host. Localhost and private addresses are refused.

```bash
# request.json (keep it private, mode 0600)
# {"config": {"name": "My API", "base_url": "https://api.example.com/v1", "auth_type": "basic", "username": "u", "password": "p"}}
mammoth connector connection create generic_rest_api --input request.json --yes
```

The reply holds `identity_key`. Use it as `CONNECTION_KEY` below. Error 4SUBS007 means the workspace plan doesn't include this connector.

## 2. Sample the data

`query` is a JSON string, not an object. Its keys are `endpoint_path`, `http_method` (`GET` or `POST`), `request_body`, `query_params`, `data_root_path` (a JSONPath to the records), `pagination`, `flatten_nested_json` and `max_pages`. This step creates no dataset.

```bash
# sample.json
# {"query": "{\"endpoint_path\": \"/posts\", \"data_root_path\": \"$\", \"max_pages\": 1}", "validate": false, "data_sample": true}
mammoth connector ds-config create generic_rest_api CONNECTION_KEY --input sample.json --yes
```

The reply returns real sample rows. Read the column names from `data.data_sample.dataschema`.

## 3. Create the dataset with a schedule

`schedule_properties` and `recurrence_info` are required. Without them the job fails with "Invalid parameters". `on_refresh_action` is `replace` (swap the rows on each refresh) or `combine` (append). With `combine`, every refresh adds every row the API returns again, so rows duplicate.

```bash
# dataset.json
# {"ds_creation_type": "cloud", "dataset_spec": {"connector_key": "generic_rest_api",
#   "connection_key": "CONNECTION_KEY",
#   "query_properties": {"ds_name": "My data", "table_name": "posts", "query": "<same JSON string as step 2>"},
#   "schedule_properties": {"schedule_type": "moment", "first_pull_at": "now", "on_refresh_action": "replace"},
#   "recurrence_info": {"interval": 1, "start_at": "2027-01-01T02:00:00"}}}
mammoth dataset create --input dataset.json
```

## 4. Refresh automatically

```bash
# auto.json
# {"description": "Refresh My data", "tasks": [{"task_type": "run_data_retrieval", "details": {"ds_details": [{"ds_id": DATASET_ID}]}}],
#   "conditions": [{"condition_type": "at_specific_time", "details": {"frequency": "daily", "interval": 1, "start_at": "2027-01-01T02:00:00Z"}}]}
mammoth automation create 'Refresh My data' --input auto.json --yes
```

`ds_details` must be a list. A `start_at` less than 2 minutes ahead is rejected as too soon, and a past one starts now. To run the refresh once now, use `mammoth automation update AUTOMATION_ID --yes --input '{"patch": [{"op": "command", "path": "run", "value": {}}]}'`.

## Elasticsearch

Use the same connection with the cluster's address as `base_url`, and `POST` on `/INDEX/_search`. Name the fields you need in `_source`. A field such as `asctime` (`2026-10-09 15:52:46,290`, with a comma before the milliseconds) breaks the load.

- **Documents:** `"request_body": {"size": 500, "_source": ["@timestamp", "task"]}` with `"data_root_path": "$.hits.hits[*]._source"`.
- **Counts per bucket:** `"request_body": {"size": 0, "aggs": {"c": {"composite": {"size": 10000, "sources": [...]}}}}` with `"data_root_path": "$.aggregations.c.buckets[*]"`. The columns are `doc_count` and `key.<source>`.

## Next up

- [API-driven workflows](https://docs.mammoth.io/guides/advanced/api-workflows): trigger automations through the REST API.
- [CLI reference](https://docs.mammoth.io/build/cli/): every command, with its blast radius and endpoint.

---
Source: https://docs.mammoth.io/guides/advanced/rest-api-import.md · Updated: 2026-10-12