afy collections, index, and points

The Aetherfy CLI’s vector commands

Three command groups of the Aetherfy CLI read and manage the Aetherfy vector database, with the same API key as every other afy command. They cover inspecting collections, creating and deleting them, managing payload indexes, browsing and reading points, writing a few, and deleting them. Bulk loading is not among them: the Python and JavaScript SDKs do that, with the chunking and retries a bulk load needs (see /vectors/sdk).

The collections group also answers to collection.

Flags every Aetherfy vector command takes

Each of the thirteen commands takes these four flags, in addition to the global flags on /cli.

How the Aetherfy CLI chooses the vectors endpoint

The Aetherfy CLI picks the endpoint in the same order as the Aetherfy SDKs:

1. --vectors-url, when given.
2. AETHERFY_VECTORS_URL, when set. It also wins over an API region, with a warning on stderr: Aetherfy sets this variable on every agent machine.
3. --api-region, else AETHERFY_VECTORS_API_REGION. The region is turned into a URL by asking GET /api/v1/regions on https://vectors.aetherfy.com.
4. https://vectors.aetherfy.com.

An API region outside the three above is refused before any request, with exit code 2. The API region only chooses which Aetherfy endpoint answers; it is not where a collection’s data lives, which afy collections create --regions sets.

How the Aetherfy CLI chooses the workspace

An Aetherfy collection belongs to one workspace, or to none, and is reachable only through it: articles in workspace research is not found outside it. Names are unique per account, so there is only ever one articles. The Aetherfy CLI uses --workspace when it is given, else AETHERFY_WORKSPACE, else no workspace, which is the SDKs’ workspace="auto" default. Aetherfy sets AETHERFY_WORKSPACE only on an agent that belongs to a workspace. --workspace "" forces no workspace even where the variable is set. Windows PowerShell 5.1 drops an empty argument before the CLI sees it, so there write --workspace=; the same holds for every "" on this page. A --workspace or --to value starting with - is refused with exit 2 before anything is sent: it is what a dropped "" leaves, the next argument read as the value.

Every --json object says which endpoint and workspace the command used:

{
  "endpoint": "https://vectors.aetherfy.com",
  "workspace": "research",
  "collections": []
}

workspace is null when no workspace was used.

Listing and inspecting Aetherfy collections

afy collections list lists the collections in the chosen workspace, or the collections outside any workspace when none is chosen. The two are separate: neither list includes the other.

afy collections list
afy collections list --workspace research --json

Each entry of collections in the --json output carries name, description (or null), size, distance, status, points_count and created_at. An empty result is "collections": [].

afy collections get <name> shows one collection. Its --json output holds one collection object with the same fields plus regions and updated_at.

afy collections get articles --json

Creating and deleting Aetherfy collections

afy collections create <name> creates a collection for vectors of one size, compared with one distance.

afy collections create articles --size 1536 --distance cosine
afy collections create articles --size 768 --distance dot --regions us-east-1,eu-central-1 --json

The --json output’s collection holds name, size, distance and the regions Aetherfy placed the collection in. Creating a collection that already exists with the same settings succeeds and changes nothing.

afy collections delete <name> deletes a collection and every point in it. It asks you to type the collection’s name first.

When stdin is not a terminal, nobody can answer the question, so without --yes the Aetherfy CLI deletes nothing and exits 2. A script passes --yes:

afy collections delete articles --yes --json

Changing an Aetherfy collection’s regions: preview, then confirm

afy collections regions <name> changes the regions a collection’s data is placed in. A region it adds gets a copy of the collection’s data; a region it removes has its copy deleted. So the change takes two steps. First the Aetherfy CLI shows the control plane’s preview, which is built by the same planner the change runs, so the change does what its preview says. Then it asks Proceed? [y/N] on stderr.

afy collections regions articles --regions us-east-1,eu-central-1

Collection 'articles' in no workspace: us-east-1 -> us-east-1, eu-central-1
  Adds eu-central-1: copies 3 GB of data into it.
  Region change 4 of 10 that copy data in the last 30 days.
  50 GB of 400 GB copied between regions in the last 30 days; 347 GB left after this change.
Proceed? [y/N]

A change that only removes regions copies no data, and the preview says it does not count toward your plan’s region-change limits. Answering anything but y or yes changes nothing and exits 2. When stdin is not a terminal, or with --json, nobody can answer the question, so without --yes the Aetherfy CLI sends nothing and exits 2. A script passes --yes:

afy collections regions articles --regions us-east-1 --workspace research --yes --wait --json

A change the control plane would refuse is refused at the preview, and nothing is sent: the preview’s lines are followed by the refusal’s message and code on stderr, and the exit code is 1. The refusals, among them REGION_CHANGE_COUNT_LIMIT_EXCEEDED and REGION_CHANGE_SIZE_LIMIT_EXCEEDED with the time the next change is allowed, are listed on /agents/api-workspaces. A change to the regions the collection is already in changes nothing and exits 0.

The --json object holds collection, preview (the control plane’s preview: from_regions, to_regions, regions_to_add, regions_to_remove, no_op, copy_bytes, data_deleted_from and limits), operation_id (null for no change) and operation, the finished operation under --wait, else null.

Moving an Aetherfy collection to another workspace

afy collections move <name> moves a collection to the workspace --to names, or out of any workspace with --to "" (--to= on Windows PowerShell 5.1). The collection is found in the workspace --workspace chooses, like every collections command, so --to names only where it goes. A move changes only which workspace reaches the collection: its regions and its data stay where they are, so it copies nothing and asks nothing. The target workspace must allow each of the regions the collection is placed in. The collection’s aliases are deleted; recreate the ones you need.

afy collections move articles --to research
afy collections move articles --workspace research --to "" --wait --json

The --json object holds collection, to (null for no workspace), no_op, regions, operation_id and operation.

Both commands return once the control plane accepts the change, which then runs in the background. --wait follows it, as afy start --wait follows a start: exit 0 when it succeeds, 1 when it fails, with the failure’s message, and 4 when it is still running after 30 minutes. It keeps running then, and afy collections get <name> shows the result once it finishes.

Aetherfy payload indexes: afy index create waits until the index is built

afy index create <collection> <field> creates a payload index on one field and exits 0 only once the index is built, so a filter or an ordered scroll on the field works as soon as it returns.

The default wait is ten minutes; these two commands wait the same way:

afy index create articles thread_id --type keyword
afy index create articles thread_id --type keyword --timeout 600

afy index create articles ts --type integer --timeout 300 --json
afy index create articles body --type '{"type":"text","tokenizer":"word","lowercase":true}'

This is the Aetherfy SDKs’ create_field_index behaviour, step for step:

A --timeout that is not a number of seconds above 0 is refused before any request, naming the value, with exit code 2.

afy index delete <collection> <field> drops the index on one field in a single request, allowed as long as a create, because Aetherfy may hold a delete the same way. Dropping an index the field does not have succeeds; a collection that does not exist is an error.

afy index delete articles thread_id

Reading Aetherfy points: count, get, and search

--filter takes a filter as a JSON object, in the shape the SDKs send (see /vectors/filtering), on every points command that has it.

afy points count <collection> counts the points exactly. Without --json it prints the bare number, so $(afy points count articles) is the count.

afy points count articles
afy points count articles --filter '{"must":[{"key":"lang","match":{"value":"en"}}]}' --json

afy points get <collection> <id>... reads points by id, with their payloads and without their vectors. An id made only of digits is sent as a number, anything else as a string, such as a UUID. An id that does not exist is left out of the answer rather than reported.

afy points get articles 17 42 --json

afy points search <collection> returns the points nearest to a vector.

afy points search articles --vector '[0.12, -0.03, 0.88]' --limit 5
afy points search articles --vector @query.json --json

In --json output each entry of points carries id and payload, and a search hit carries score as well.

Browsing Aetherfy points: afy points list pages through a collection

afy points list <collection> pages through a collection’s points in id order, with their payloads. It is the SDKs’ scroll.

1000 is the most the Aetherfy vectors API returns in one read; a --limit outside 1 to 1000 is refused before any request, with exit code 2. When there are more points, the page ends with the id the next one starts at. Pass it as --offset to read the next page, with the same --filter:

afy points list articles
afy points list articles --offset 11

Without --json each row is a point’s id and the start of its payload, your own keys first, and the page ends with More points: pass --offset 11 for the next page. when there is one. With --json the object holds points (each with id, payload, and vector under --with-vectors) and next_page_offset, which is null on the last page. A script passes it back until it is null:

afy points list articles --limit 100 --json > page.json
afy points list articles --limit 100 --offset "$(jq -r .next_page_offset page.json)" --json

Writing Aetherfy points: afy points upsert adds or replaces by id

afy points upsert <collection> writes the points --points gives: one point object or a JSON array of them, inline or @path to read them from a file. A point is the vectors API’s own shape: an id (a whole number or a UUID), a vector, and an optional payload object. A point whose id already exists is replaced, vector and payload, without a question, as the SDKs’ upsert does.

afy points upsert articles --points '{"id": 1, "vector": [0.12, -0.03, 0.88], "payload": {"lang": "en"}}'
afy points upsert articles --points @points.json --json

Upserted 1 point(s) into collection 'articles'.

With --json the object holds collection and upserted, the number of points written. A missing --points, input that is not JSON, an empty array, or a point without an id or a vector is refused before any request, with exit code 2. Every other rule (the vector’s size, reserved __aetherfy_ payload keys, the size caps) is the Aetherfy vectors API’s, and its error is printed as it came, with exit code 1.

The points are sent in order, in requests of at most 10 000 points or 24 MiB, the way the SDKs split an upsert (see /vectors/limits). Each request waits up to 100 seconds, longer than the vectors API lets any request run, so the API always answers or ends a request before the CLI gives up. If a request fails, the error says how many points were written before it (written in the --json error object). When the failed request’s own outcome cannot be known, the error also says how many points it held (unconfirmed): they may or may not have been written. That is the case when the connection was lost before an answer, when the server failed, and when the server refused a request larger than one 12 MB chunk, because the vectors API writes an upsert in chunks as it reads it and a refusal leaves the chunks already written in place. A refused request no larger than one 12 MB chunk wrote nothing. Re-running the same upsert is safe in every case: points are replaced by id.

Points written with your API key read back with __aetherfy_agent_id null: a person’s write, not an agent’s.

Deleting Aetherfy points: afy points delete counts first

afy points delete <collection> <id>... deletes points by id, or with --filter instead of ids, every point a filter matches. Give one or the other: both, or neither, is refused before any request, with exit code 2.

Before deleting anything, the Aetherfy CLI counts what will go and prints the count on stderr: by id, the ids that exist (only those are deleted); by filter, the points the filter matches now, the same count afy points count <collection> --filter gives. Then it asks you to type the collection’s name; typing anything else deletes nothing and exits 2. When nothing matches, nothing is deleted and nothing is asked.

afy points delete articles 17 42

2 point(s) in collection 'articles' will be deleted.
Type the collection name to confirm: articles
Deleted 2 point(s) from collection 'articles'.

When stdin is not a terminal, nobody can answer the question, so without --yes the Aetherfy CLI deletes nothing, sends nothing and exits 2, saying why. A script passes --yes; with --json the object holds collection and deleted, the count shown before the delete. A filter delete removes what matches when it runs, so a point written in between that matches is deleted too and is not in that count:

afy points delete articles --filter '{"must":[{"key":"lang","match":{"value":"de"}}]}' --yes --json

An empty filter is refused, with exit code 2: {}, or a filter whose clauses are all empty, matches every point, and deleting every point is afy collections delete <name>.

afy points delete articles --filter '{}' --yes

Error: not deleting: an empty --filter matches every point in 'articles'. To delete the whole collection, run afy collections delete articles.

Aetherfy vector command errors and exit codes

Nothing goes to stdout on a failure, except what was printed before it without --json: the preview of a regions change the control plane refuses, or the line --wait prints before following a change that then fails. On stderr, the Aetherfy vectors API’s or the control plane’s own status, code and message are copied through unchanged. With --json stderr holds one object, for every failure except a command line that does not parse, which is a plain Error: line because parsing stopped before --json was necessarily read:

{"error": {"status": 404, "code": "NOT_FOUND", "message": "Collection 'articles' not found"}}

Without --json it is one line, Error: Collection 'articles' not found (NOT_FOUND). A failure that got no answer from the API has no status or code. The codes themselves are listed on /vectors/errors.