---
slug: examples/batch-and-pagination
title: Batch upsert and pagination
kind: howto
surface: vectors
summary: Complete Python and JavaScript examples of batched upserts into Aetherfy plus both pagination styles, the scroll_iter/scrollIter generators and the manual scroll cursor, with the batch-size bounds and the type-export caveat.
sources:
  - aetherfy-vectors-python-sdk:aetherfy_vectors/client.py
  - aetherfy-vectors-python-sdk:aetherfy_vectors/models.py
  - aetherfy-vectors-js-sdk:src/client.ts
  - aetherfy-vectors-js-sdk:src/models.ts
  - aetherfy-vectors-js-sdk:src/index.ts
---

# Batch upsert and pagination

Writing many points and reading them all back are the two operations where the naive version falls over. Aetherfy gives you a batched `upsert` and two ways to page: a generator that hides the cursor, and a manual cursor you drive yourself.

```bash
export AETHERFY_API_KEY="afy_live_your_key_here"
```

## Batched upsert into Aetherfy with Python

`upsert` takes a list of points, so batching is a matter of chunking your input rather than calling a different method. Install with `pip install aetherfy-vectors`.

```python
from aetherfy_vectors import AetherfyVectorsClient
from aetherfy_vectors.models import VectorConfig, DistanceMetric, Point

client = AetherfyVectorsClient()

client.create_collection(
    "documents",
    VectorConfig(size=4, distance=DistanceMetric.COSINE),
)

# 1,000 points to write. Vectors are literal floats here so the example runs
# without an embedding model; see /examples/embeddings for the real thing.
all_points = [
    Point(
        id=i,
        vector=[i / 1000, 0.20, 0.30, 0.40],
        payload={"doc_id": i, "shard": "a" if i % 2 == 0 else "b"},
    )
    for i in range(1000)
]

BATCH = 100
for start in range(0, len(all_points), BATCH):
    chunk = all_points[start:start + BATCH]
    ok = client.upsert("documents", chunk)
    print(start, len(chunk), ok)

# 0 100 True
# 100 100 True
# ... one line per batch ...
# 900 100 True

print(client.count("documents"))  # 1000

client.close()
```

`upsert` returns `True` for the whole batch; it does not report per-point status. Re-sending a point with an existing id replaces it, so a retried batch is safe.

## Batched upsert into Aetherfy with JavaScript

Install with `npm install aetherfy-vectors` (Node >= 20). Save as `main.mjs` or set `"type": "module"`.

```typescript
import { AetherfyVectorsClient, DistanceMetric } from 'aetherfy-vectors';

const client = new AetherfyVectorsClient();

async function main() {
  await client.createCollection('documents', { size: 4, distance: DistanceMetric.COSINE });

  const allPoints = Array.from({ length: 1000 }, (_, i) => ({
    id: i,
    vector: [i / 1000, 0.20, 0.30, 0.40],
    payload: { doc_id: i, shard: i % 2 === 0 ? 'a' : 'b' },
  }));

  const BATCH = 100;
  for (let start = 0; start < allPoints.length; start += BATCH) {
    const chunk = allPoints.slice(start, start + BATCH);
    const ok = await client.upsert('documents', chunk);
    console.log(start, chunk.length, ok);
  }
  // 0 100 true
  // 100 100 true
  // ... one line per batch ...
  // 900 100 true

  console.log(await client.count('documents', { exact: true })); // 1000

  client.dispose();
}

main();
```

Batches are sequential above on purpose. Firing every batch concurrently with `Promise.all` will hit rate limits on a large import; see [Limits](/platform/limits).

## Paging an Aetherfy collection with the Python scroll_iter generator

`scroll_iter` yields points one at a time and fetches the next page for you. It is the right default for a full scan. **Its parameters are keyword-only** — passing `batch_size` positionally raises `TypeError`.

```python
from aetherfy_vectors import AetherfyVectorsClient

client = AetherfyVectorsClient()

seen = 0
for point in client.scroll_iter("documents", batch_size=256, with_payload=True):
    seen += 1
    if seen <= 2:
        print(point)

print("total", seen)
# {'id': 0, 'payload': {'doc_id': 0, 'shard': 'a'}}
# {'id': 1, 'payload': {'doc_id': 1, 'shard': 'b'}}
# total 1000

client.close()
```

Filtering while scrolling uses `scroll_filter`, the same object shape as a search filter:

```python
count = 0
for point in client.scroll_iter(
    "documents",
    batch_size=256,
    scroll_filter={"must": [{"key": "shard", "match": {"value": "a"}}]},
):
    count += 1

print(count)  # 500

client.close()
```

## Paging an Aetherfy collection with the JavaScript scrollIter generator

`scrollIter` is an async generator, so drive it with `for await`.

```typescript
import { AetherfyVectorsClient } from 'aetherfy-vectors';

const client = new AetherfyVectorsClient();

async function main() {
  let seen = 0;
  for await (const point of client.scrollIter('documents', {
    batchSize: 256,
    withPayload: true,
  })) {
    seen += 1;
    if (seen <= 2) console.log(point);
  }
  console.log('total', seen);
  // { id: 0, payload: { doc_id: 0, shard: 'a' } }
  // { id: 1, payload: { doc_id: 1, shard: 'b' } }
  // total 1000

  // Filtered scan.
  let filtered = 0;
  for await (const _point of client.scrollIter('documents', {
    batchSize: 256,
    scrollFilter: { must: [{ key: 'shard', match: { value: 'a' } }] },
  })) {
    filtered += 1;
  }
  console.log(filtered); // 500

  client.dispose();
}

main();
```

Breaking out of the `for await` loop early stops the iteration and no further pages are fetched.

## The Aetherfy scroll iterator options

| Python keyword | JavaScript option | Default | Notes |
| --- | --- | --- | --- |
| `batch_size` | `batchSize` | `256` | Must be between 1 and 1000 inclusive |
| `scroll_filter` | `scrollFilter` | `None` / undefined | Same filter shape as search |
| `with_payload` | `withPayload` | `True` / `true` | Include stored payloads |
| `with_vectors` | `withVectors` | `False` / `false` | Include stored vectors |

The batch size is validated client-side and the two SDKs raise different error types for the same mistake:

| Bad value | Python | JavaScript |
| --- | --- | --- |
| `0`, `1001`, negative | `ValueError` | `RangeError` |

`batch_size` only controls how many points each network round trip carries. It does not change how many points you receive in total — the generator runs until the collection (or the filtered subset) is exhausted.

## Manual pagination in Aetherfy with Python scroll

Use `scroll` when you need to hand a cursor to something else — a web request, a resumable job, a queue message. You page by feeding the previous response's `next_page_offset` back in as `offset`.

```python
from aetherfy_vectors import AetherfyVectorsClient

client = AetherfyVectorsClient()

offset = None
pages = 0
total = 0

while True:
    result = client.scroll(
        "documents",
        limit=250,
        offset=offset,
        with_payload=True,
    )
    pages += 1
    total += len(result["points"])
    offset = result["next_page_offset"]
    print(pages, len(result["points"]), offset)
    if offset is None:
        break

print("pages", pages, "total", total)
# 1 250 250
# 2 250 500
# 3 250 750
# 4 250 None
# pages 4 total 1000

client.close()
```

`scroll` returns a plain `dict` in Python with exactly two keys:

| Key | Type | Meaning |
| --- | --- | --- |
| `points` | list of dicts | The page of points |
| `next_page_offset` | cursor value, or `None` | Feed back as `offset`; `None` means the scan is finished |

Stop when `next_page_offset` is `None`. Do not stop on a short page — a page can be shorter than `limit` and still have a successor.

## Manual pagination in Aetherfy with JavaScript scroll

The JS shape is the same contract in camelCase.

```typescript
import { AetherfyVectorsClient } from 'aetherfy-vectors';

const client = new AetherfyVectorsClient();

async function main() {
  let offset: unknown = undefined;
  let pages = 0;
  let total = 0;

  for (;;) {
    const result = await client.scroll('documents', {
      limit: 250,
      offset,
      withPayload: true,
    });
    pages += 1;
    total += result.points.length;
    offset = result.nextPageOffset;
    console.log(pages, result.points.length, offset);
    if (offset === null || offset === undefined) break;
  }

  console.log('pages', pages, 'total', total);
  // 1 250 250
  // 2 250 500
  // 3 250 750
  // 4 250 null
  // pages 4 total 1000

  client.dispose();
}

main();
```

| Key | Type | Meaning |
| --- | --- | --- |
| `points` | array | The page of points |
| `nextPageOffset` | cursor value, or null | Feed back as `offset`; nullish means the scan is finished |

## A TypeScript caveat in the Aetherfy JS package

The npm package's `exports` map contains only `"."`, so every import comes from `'aetherfy-vectors'`. Within that root export, **not every type name is re-exported.**

| Type name | Exported from the Aetherfy package root |
| --- | --- |
| `ScrollIterOptions` | yes |
| `ScrollPoint` | yes |
| `ScrollOptions` | **no** — do not import it |
| `ScrollResult` | **no** — do not import it |

So this compiles:

```typescript
import { AetherfyVectorsClient, ScrollIterOptions, ScrollPoint } from 'aetherfy-vectors';

const opts: ScrollIterOptions = { batchSize: 256, withPayload: true };

async function collect(client: AetherfyVectorsClient): Promise<ScrollPoint[]> {
  const out: ScrollPoint[] = [];
  for await (const p of client.scrollIter('documents', opts)) out.push(p);
  return out;
}
```

and this does not:

```typescript
// Does NOT compile — these names are not exported from the Aetherfy package root.
import { ScrollOptions, ScrollResult } from 'aetherfy-vectors';
```

For the manual `scroll` path, let TypeScript infer the return type from `await client.scroll(...)` instead of naming it.

## Choosing between the two Aetherfy pagination styles

| Use | When |
| --- | --- |
| `scroll_iter` / `scrollIter` | You are scanning inside one process and just want every point. Fewer moving parts, no cursor to lose. |
| `scroll` / `scroll` | You need the cursor as a value — resumable imports, an HTTP endpoint that returns a page plus a "next" token, work spread over several processes. |

Related: [Retrieve, delete, count](/examples/retrieve-delete-count) for counting what you imported, and [Filtered search](/examples/filtered-search) for the filter object accepted by `scroll_filter` / `scrollFilter`.