Batch upsert and pagination

Writing many points and reading them all back are the two operations where the naive version falls over. Aetherfy gives you a batched upsert and two ways to page: a generator that hides the cursor, and a manual cursor you drive yourself.


export AETHERFY_API_KEY="afy_live_your_key_here"

Batched upsert into Aetherfy with Python

upsert takes a list of points, so batching is a matter of chunking your input rather than calling a different method. Install with pip install aetherfy-vectors.


from aetherfy_vectors import AetherfyVectorsClient
from aetherfy_vectors.models import VectorConfig, DistanceMetric, Point
 
client = AetherfyVectorsClient()
 
client.create_collection(
    "documents",
    VectorConfig(size=4, distance=DistanceMetric.COSINE),
)
 
# 1,000 points to write. Vectors are literal floats here so the example runs
# without an embedding model; see /examples/embeddings for the real thing.
all_points = [
    Point(
        id=i,
        vector=[i / 1000, 0.20, 0.30, 0.40],
        payload={"doc_id": i, "shard": "a" if i % 2 == 0 else "b"},
    )
    for i in range(1000)
]
 
BATCH = 100
for start in range(0, len(all_points), BATCH):
    chunk = all_points[start:start + BATCH]
    ok = client.upsert("documents", chunk)
    print(start, len(chunk), ok)
 
# 0 100 True
# 100 100 True
# ... one line per batch ...
# 900 100 True
 
print(client.count("documents"))  # 1000
 
client.close()

upsert returns True for the whole batch; it does not report per-point status. Re-sending a point with an existing id replaces it, so a retried batch is safe.

Batched upsert into Aetherfy with JavaScript

Install with npm install aetherfy-vectors (Node >= 20). Save as main.mjs or set "type": "module".


import { AetherfyVectorsClient, DistanceMetric } from 'aetherfy-vectors';
 
const client = new AetherfyVectorsClient();
 
async function main() {
  await client.createCollection('documents', { size: 4, distance: DistanceMetric.COSINE });
 
  const allPoints = Array.from({ length: 1000 }, (_, i) => ({
    id: i,
    vector: [i / 1000, 0.20, 0.30, 0.40],
    payload: { doc_id: i, shard: i % 2 === 0 ? 'a' : 'b' },
  }));
 
  const BATCH = 100;
  for (let start = 0; start < allPoints.length; start += BATCH) {
    const chunk = allPoints.slice(start, start + BATCH);
    const ok = await client.upsert('documents', chunk);
    console.log(start, chunk.length, ok);
  }
  // 0 100 true
  // 100 100 true
  // ... one line per batch ...
  // 900 100 true
 
  console.log(await client.count('documents', { exact: true })); // 1000
 
  client.dispose();
}
 
main();

Batches are sequential above on purpose. Firing every batch concurrently with Promise.all will hit rate limits on a large import; see Limits.

Paging an Aetherfy collection with the Python scroll_iter generator

scroll_iter yields points one at a time and fetches the next page for you. It is the right default for a full scan. Its parameters are keyword-only — passing batch_size positionally raises TypeError.


from aetherfy_vectors import AetherfyVectorsClient
 
client = AetherfyVectorsClient()
 
seen = 0
for point in client.scroll_iter("documents", batch_size=256, with_payload=True):
    seen += 1
    if seen <= 2:
        print(point)
 
print("total", seen)
# {'id': 0, 'payload': {'doc_id': 0, 'shard': 'a'}}
# {'id': 1, 'payload': {'doc_id': 1, 'shard': 'b'}}
# total 1000
 
client.close()

Filtering while scrolling uses scroll_filter, the same object shape as a search filter:


count = 0
for point in client.scroll_iter(
    "documents",
    batch_size=256,
    scroll_filter={"must": [{"key": "shard", "match": {"value": "a"}}]},
):
    count += 1
 
print(count)  # 500
 
client.close()

Paging an Aetherfy collection with the JavaScript scrollIter generator

scrollIter is an async generator, so drive it with for await.


import { AetherfyVectorsClient } from 'aetherfy-vectors';
 
const client = new AetherfyVectorsClient();
 
async function main() {
  let seen = 0;
  for await (const point of client.scrollIter('documents', {
    batchSize: 256,
    withPayload: true,
  })) {
    seen += 1;
    if (seen <= 2) console.log(point);
  }
  console.log('total', seen);
  // { id: 0, payload: { doc_id: 0, shard: 'a' } }
  // { id: 1, payload: { doc_id: 1, shard: 'b' } }
  // total 1000
 
  // Filtered scan.
  let filtered = 0;
  for await (const _point of client.scrollIter('documents', {
    batchSize: 256,
    scrollFilter: { must: [{ key: 'shard', match: { value: 'a' } }] },
  })) {
    filtered += 1;
  }
  console.log(filtered); // 500
 
  client.dispose();
}
 
main();

Breaking out of the for await loop early stops the iteration and no further pages are fetched.

The Aetherfy scroll iterator options

Python keyword	JavaScript option	Default	Notes
`batch_size`	`batchSize`	`256`	Must be between 1 and 1000 inclusive
`scroll_filter`	`scrollFilter`	`None` / undefined	Same filter shape as search
`with_payload`	`withPayload`	`True` / `true`	Include stored payloads
`with_vectors`	`withVectors`	`False` / `false`	Include stored vectors

The batch size is validated client-side and the two SDKs raise different error types for the same mistake:

Bad value	Python	JavaScript
`0`, `1001`, negative	`ValueError`	`RangeError`

batch_size only controls how many points each network round trip carries. It does not change how many points you receive in total — the generator runs until the collection (or the filtered subset) is exhausted.

Manual pagination in Aetherfy with Python scroll

Use scroll when you need to hand a cursor to something else — a web request, a resumable job, a queue message. You page by feeding the previous response’s next_page_offset back in as offset.


from aetherfy_vectors import AetherfyVectorsClient
 
client = AetherfyVectorsClient()
 
offset = None
pages = 0
total = 0
 
while True:
    result = client.scroll(
        "documents",
        limit=250,
        offset=offset,
        with_payload=True,
    )
    pages += 1
    total += len(result["points"])
    offset = result["next_page_offset"]
    print(pages, len(result["points"]), offset)
    if offset is None:
        break
 
print("pages", pages, "total", total)
# 1 250 250
# 2 250 500
# 3 250 750
# 4 250 None
# pages 4 total 1000
 
client.close()

scroll returns a plain dict in Python with exactly two keys:

Key	Type	Meaning
`points`	list of dicts	The page of points
`next_page_offset`	cursor value, or `None`	Feed back as `offset`; `None` means the scan is finished

Stop when next_page_offset is None. Do not stop on a short page — a page can be shorter than limit and still have a successor.

Manual pagination in Aetherfy with JavaScript scroll

The JS shape is the same contract in camelCase.


import { AetherfyVectorsClient } from 'aetherfy-vectors';
 
const client = new AetherfyVectorsClient();
 
async function main() {
  let offset: unknown = undefined;
  let pages = 0;
  let total = 0;
 
  for (;;) {
    const result = await client.scroll('documents', {
      limit: 250,
      offset,
      withPayload: true,
    });
    pages += 1;
    total += result.points.length;
    offset = result.nextPageOffset;
    console.log(pages, result.points.length, offset);
    if (offset === null || offset === undefined) break;
  }
 
  console.log('pages', pages, 'total', total);
  // 1 250 250
  // 2 250 500
  // 3 250 750
  // 4 250 null
  // pages 4 total 1000
 
  client.dispose();
}
 
main();

Key	Type	Meaning
`points`	array	The page of points
`nextPageOffset`	cursor value, or null	Feed back as `offset`; nullish means the scan is finished

A TypeScript caveat in the Aetherfy JS package

The npm package’s exports map contains only ".", so every import comes from 'aetherfy-vectors'. Within that root export, not every type name is re-exported.

Type name	Exported from the Aetherfy package root
`ScrollIterOptions`	yes
`ScrollPoint`	yes
`ScrollOptions`	no — do not import it
`ScrollResult`	no — do not import it

So this compiles:


import { AetherfyVectorsClient, ScrollIterOptions, ScrollPoint } from 'aetherfy-vectors';
 
const opts: ScrollIterOptions = { batchSize: 256, withPayload: true };
 
async function collect(client: AetherfyVectorsClient): Promise<ScrollPoint[]> {
  const out: ScrollPoint[] = [];
  for await (const p of client.scrollIter('documents', opts)) out.push(p);
  return out;
}

and this does not:


// Does NOT compile — these names are not exported from the Aetherfy package root.
import { ScrollOptions, ScrollResult } from 'aetherfy-vectors';

For the manual scroll path, let TypeScript infer the return type from await client.scroll(...) instead of naming it.

Choosing between the two Aetherfy pagination styles

Use	When
`scroll_iter` / `scrollIter`	You are scanning inside one process and just want every point. Fewer moving parts, no cursor to lose.
`scroll` / `scroll`	You need the cursor as a value — resumable imports, an HTTP endpoint that returns a page plus a “next” token, work spread over several processes.

Related: Retrieve, delete, count for counting what you imported, and Filtered search for the filter object accepted by scroll_filter / scrollFilter.