Runs and logs
Listing the runs of an Aetherfy agent
A run on Aetherfy is one ephemeral execution of a type: job agent. List
recent runs newest-first:
afy runs nightly-report
afy runs nightly-report --limit 50The default limit is 20 and the maximum is 100.
The listing shows six columns:
| Column | Contains |
|---|---|
| When | When the run was created, in UTC and labelled UTC — the instant --json gives as created_at |
| Trigger | Why it ran — cron or manual, or spawn (by <agent>) for a run another agent spawned on this one |
| State | How it ended, or where it is |
| Release | The release the run executed, as afy deployments numbers it (v3), or a dash when none was recorded |
| Duration | How long the machine ran |
| Run ID | The identifier to pass to afy logs --run |
Aetherfy lists every run the agent executed: a scheduled one, a manual one, and one another agent spawned on it. A run is the history of the agent that executed it, so a spawned run is in the child’s listing, naming the agent that spawned it.
What the Aetherfy runs API returns
The equivalent Aetherfy REST call is:
GET /agents/{id}/runs| Query parameter | Accepted values | On anything else |
|---|---|---|
trigger_source | cron, manual or spawn only | 422 |
limit | 1–100 | 422 |
before | The cursor of the last run of the previous page | 422 |
Each row Aetherfy returns carries these fields:
| Field | Contains |
|---|---|
id | The run’s identifier |
trigger_source | cron, manual, or spawn |
state | See the state table below |
created_at | When the run was created |
error_message | The failure reason, when the run failed |
release_version | The release the run executed — see below |
machine_started_at | When the machine began executing |
machine_stopped_at | When the machine stopped |
duration_seconds | Execution duration |
cursor | Where this row sits in the list, to pass as before |
The two machine fields are null on a run of a type: service agent, which is a
request to a machine the service already has rather than a machine of its own. Its
duration_seconds runs from the moment that machine accepted the run to the moment
the run’s outcome was reported.
release_version is the version of the release a run executed — the number
afy deployments and afy rollback use, not the run’s own place in the
agent’s deployment history. A type: job run executes the release whose image it
was started from, and that stays true when the image was gone and the run
rebuilt it from stored code. After an exact rollback, the rebuilt code comes from
the version you rolled back to, but the run still names the release the rollback
created. A type: service run executes the
release the machine that took it was serving. It is null for a service run no
machine accepted, and for runs recorded before Aetherfy kept it.
The agent record carries its most recent entry of this list as last_run, which
is what afy status prints as Last run.
before is a keyset cursor rather than an offset, so paging backwards through a
long history stays correct even as new runs arrive.
Run and deployment states on Aetherfy
Aetherfy uses one state vocabulary for both deployments and runs. Four of the states are terminal — once a record reaches them it will not change again.
| State | Meaning | Terminal |
|---|---|---|
queued | Waiting in the build queue | No |
building | The image is building | No |
deploying | Machines are launching and being health-checked, or, for a run, the run is being handed to a machine | No |
active | Serving, or executing in the case of a run | No |
completed | A run’s machine ended on its own with exit code 0 | Yes |
failed | The build, the deploy, or the run failed | Yes |
superseded | Replaced by a newer successful deployment | Yes |
rolled_back | Replaced by a rollback | Yes |
completed applies to runs, which end by design. A long-lived service
deployment that is working sits at active and stays there until a newer
deployment supersedes it.
A run is active only once a machine has accepted it. Until then it is
deploying, and a run that no machine accepts within 2 minutes of Aetherfy
starting to hand it over is failed with “The run was never accepted by a
machine, so your code was not executed.” Your code did not start, so there is
nothing of yours to debug.
A run that stalls before that point, still queued, building or deploying 7
minutes after it was created, is also recorded as failed, with a reason naming the
step that never finished. The 7 minutes run from when the run was created, so time
the run spends queued behind other deployments counts toward them (see
/agents/task-contract). Aetherfy checks for stalled runs
every 5 minutes, so such a run can show its pending state for up to 5 minutes past
those 7 minutes before it is failed.
What decides completed versus failed for a run is the process exit code —
see /agents/task-contract.
Agent status values on Aetherfy
An agent has its own status, separate from the state of any individual deployment or run.
| Status | Meaning |
|---|---|
pending | Created, not yet built |
building | An image is building |
deploying | A version is launching and being health-checked; a version already deployed keeps serving meanwhile |
running | Deployed and operating |
failed | The most recent deployment failed and no earlier version is serving. A redeploy that fails leaves the agent running on the version it was replacing |
paused | Paused with afy stop — every machine is stopped, and incoming traffic does not wake it. Resume with afy start |
stopped | Stopped by Aetherfy because the account was suspended for billing. You do not set this and cannot clear it directly — settling the account returns the agent to running. See Billing and spend caps |
usage_paused | Paused by Aetherfy because usage was used up — at a spend limit, or, on Free, once the agent is awake beyond 75% of the month. Shown as paused for usage |
archived | Archived, freeing the plan quota slot |
deleting | Deletion in progress |
deleted | Deleted |
usage_paused is the one to recognise on sight: the agent is not broken and
nothing needs repairing in your code. On a paid plan, raise the limit or settle the payment at
https://app.aetherfy.com/dashboard/settings/billing
and the agent resumes. A Free account has no spend limit: its agent is paused because it
was awake beyond 75% of the month, and it resumes by itself within a day of the new month starting
(UTC), or as soon as you upgrade.
Retrieving logs from Aetherfy
Everything an agent writes to stdout and stderr becomes its logs on Aetherfy.
afy logs nightly-reportScope the output to a single run, using a Run ID from afy runs:
afy logs nightly-report --run 9f3a1c72-5b8e-4d61-a0f4-7c2e9b5d3a18| Flag | Short | Default | Effect |
|---|---|---|---|
--tail | -n | 50 | How many lines to return. The server maximum is 1000. |
--follow | -f | off | Keep streaming new lines as they arrive |
--since | — | — | Only lines newer than a duration, e.g. 1h, 30m |
--level | — | — | Comma-separated severities, e.g. ERROR,WARN |
--stream | — | — | Comma-separated streams: stdout, stderr, system |
--run | — | — | Restrict to a single run id. On a type: service agent this returns Aetherfy’s lines about that run — sent, answered, how long — not your service’s own output |
Combined:
afy logs nightly-report --tail 200 --since 1h --level ERROR,WARN --stream stderrOne caveat worth knowing before you debug against it: on Aetherfy --follow
polls, and it ignores both --tail and --since. It also does not honour
JSON output. Use --follow to watch what happens next; use --tail and
--since to look at what already happened.
Log record fields on Aetherfy
Every log record Aetherfy stores carries:
| Field | Contains |
|---|---|
| agent | Which agent emitted it |
| deployment | Which deployment was running |
| timestamp | When the line was emitted |
| stream | stdout, stderr, or system |
| level | The severity |
| message | The line itself |
The system stream holds lines Aetherfy itself emits about the machine, as
distinct from your program’s own output on stdout and stderr. Their message
carries no tag of its own: afy logs prints each line as [level] message, so
a platform line reads [SYSTEM] …, and one Aetherfy writes at ERROR reads
[SYSTEM ERROR] … — never like your program’s own [ERROR] lines. In the
dashboard the stream is its own column, and afy logs --stream system selects
only Aetherfy’s lines.
The platform also writes into this stream when an agent’s own key is refused or removes data from the vector database. Each is one line, in plain words:
| What happened | The line |
|---|---|
| The agent’s key was refused (its access, a protected collection or the plan’s collection limit) | Refused DELETE /api/v1/workspaces/research/collections/kb: … followed by the refusal’s sentence, such as This agent's key holds read and write access in workspace "research", and this request needs full access. … |
| The agent deleted a collection | Deleted collection "kb" in workspace "research". |
| The agent deleted points | Deleted 12 points from collection "kb" in workspace "research". The count is of points that existed; a delete by filter counts the points the filter selected. |
The same refusal of the same request repeated within a minute is one line ending
(3 times), and deletes from one collection within a minute add up to one line
((4 requests)). The lines are written in batches, so one appears within about a
minute of the event, from any region. Reading data, and anything an account key
does, writes no line. See agent access.
How Aetherfy reads a line’s level
Your output carries no level field, so Aetherfy reads one from the start of each
line, and only from there. A level is recognised when the line begins — after an
optional timestamp such as 2026-09-17 12:40:36,123 or [2026-09-17T12:40:36Z] —
with one of DEBUG, INFO, WARN, WARNING, ERROR, CRITICAL, FATAL or
TRACE, written in one of three ways:
| Written as | Example |
|---|---|
| In square brackets, any case | [error] upstream closed |
| Followed by a colon, any case | error: config file missing |
| In capitals, followed by a space | ERROR disk full |
WARNING is stored as WARN, CRITICAL and FATAL as ERROR, and TRACE as
DEBUG. A line with no level at its start takes its stream’s: INFO on
stdout, ERROR on stderr. The word anywhere else is part of the message — No Error found on stdout is INFO — and so is a capitalised word opening a
sentence, such as Error handling is on. To set a line’s level, print it in one of
the three forms above.
What Aetherfy does not record for you
Aetherfy logging is plain stdout/stderr capture, and nothing more. It does
not auto-instrument calls your agent makes to model providers — there is no
automatic capture of prompts, completions, token counts, latency or tool calls
from an OpenAI, Anthropic or other SDK running inside your agent.
If you want that, log it yourself. A print() or logger.info() around the call
lands in exactly the same place as the rest of your output and is searchable with
afy logs --level:
import json, time
started = time.monotonic()
response = call_your_model(prompt)
print(json.dumps({
"event": "llm_call",
"model": "your-model-id",
"elapsed_ms": round((time.monotonic() - started) * 1000),
"prompt_chars": len(prompt),
}))Keep the per-line and per-deployment limits below in mind if you log full prompts — they are easy to exceed.
Log limits and retention on Aetherfy
Aetherfy caps log volume in several dimensions. Logs are an operational aid, not a durable data store — write anything you need to keep to a real destination.
| Limit | Value |
|---|---|
| Retention | Your plan’s: 3 days on Free, 14 on Starter, 30 on Performance, negotiated on Enterprise |
| Stored per account | Your plan’s, across all your agents: 50 MB on Free, 250 MB on Starter, 1 GB on Performance, negotiated on Enterprise. A line counts its message bytes, at least 64. The oldest lines are removed first, and an account may run over its size for up to an hour |
| Per deployment | 5 MB, every line counting at least 64 bytes |
| Per line | 4 KB |
| Chunks per minute, per agent | 60 |
| Per upload body | 256 KB |
Deleting an agent deletes its logs once its teardown finishes; the retention applies to agents that still exist.
Your plan keeps logs for its retention and at most its size per account, and whichever limit is reached first applies: 3 days and 50 MB on Free, 14 days and 250 MB on Starter, 30 days and 1 GB on Performance, and negotiated on Enterprise. The size counts each line’s message bytes, at least 64 for a short line, across all your agents. When an account is over its size, the oldest lines are removed first, so the newest lines are never the ones lost. The cleanup runs hourly, so an account may run over its size for up to an hour before the oldest lines go.
afy logs never shows a log older than your plan’s retention. A --since that reaches
further back than the retention still shows what is kept, and afy logs says on
stderr that the window was cut to your plan’s retention. When the size limit has
removed lines the retention would still have kept, afy logs and the dashboard’s Logs
tab say so, with the time before which nothing is left. With --json the output
is an object with logs, retention_days (null where the retention is negotiated)
and since_clamped; see logs.
A downgrade shortens the window and lowers the size at once for what you can read (the window) and at the next hourly cleanup for what is stored. An upgrade does not bring back logs the cleanup has already deleted.
When an agent exceeds the per-line or volume caps, Aetherfy drops the excess lines
and emits a marker in the stream so the gap is visible rather than silent. As afy logs prints it:
[SYSTEM] N log line(s) droppedIf you see that marker, the agent is logging faster than Aetherfy will accept. Reduce per-line size or log volume — a very large payload dumped per iteration is the usual cause. Short lines are not free: each one counts at least 64 bytes against the per-deployment cap, so a stream of empty or one-word lines reaches it after about 80,000 lines.
A run’s output is read to its last line before the run’s outcome is reported.
If your code starts a background process that keeps the run’s output open after
your main process exits, Aetherfy stops reading 2 seconds later, ends the run,
and says so in the run’s log. As afy logs prints it:
[SYSTEM] output from a process run <run id> left running was cut off 2s after the run endedAnything that background process prints after that point is not captured.
When Aetherfy cannot put an idle agent to sleep
A service with auto-sleep is put to sleep by Aetherfy once it has had no real
request for its idle_timeout_minutes, from inside the machine. If that does
not work, the agent’s system stream says so, and Aetherfy puts the machine to
sleep another way (below). As afy logs prints it:
[SYSTEM ERROR] IDLE_SUSPEND_FAILED machine=<machine id> reason=<reason> attempts=<n> failing_for_s=<seconds> every_s=<seconds>: this idle machine could not be put to sleep and stays awake; it is retried every <interval>
[SYSTEM] IDLE_SUSPEND_RECOVERED machine=<machine id> after=<n>: the machine was put to sleep after <n> failed attempt(s)IDLE_SUSPEND_FAILEDis written once for each reason, the first time it happens while the machine is idle, and again every ten minutes while it keeps happening.attemptscounts the tries so far,failing_for_show long it has been failing, andevery_show long until the next such line.machineis the machine’s id, asGET /agents/{id}/statuslists it. A line may also carrydropped=<n>: that many earlier lines of this kind were lost before they could be delivered.IDLE_SUSPEND_RECOVEREDis written when the machine is finally put to sleep after failing, with the number of failed tries.
The reason says what stood in the way, in Aetherfy’s terms:
| Reason | What it means |
|---|---|
permission_denied | The process the machine runs as was not allowed to ask for sleep |
socket_missing | The machine had no way to ask for sleep |
connection_refused | The request to sleep was refused before it was read |
timed_out | The request to sleep got no answer in time |
unreachable | The request to sleep could not be made, for any other reason |
not_configured | The machine did not know its own identity, so it could not ask |
http_<status> | Aetherfy’s platform answered the request with that status instead of accepting it |
Aetherfy then puts the machine to sleep itself. When an
IDLE_SUSPEND_FAILED line arrives, Aetherfy suspends that machine from outside
it, within seconds. If the platform will not suspend it, Aetherfy stops it
instead: the next request then starts the machine fresh, a cold start rather
than a resume, and your agent’s in-memory state is gone. Either way the machine
is billed as awake until it is asleep, like any other awake time, and these
lines count against the log limits above like any other line.
There is nothing to change in your agent. Every reason above is on Aetherfy’s
side. If an agent keeps writing IDLE_SUSPEND_FAILED, contact support with the agent
and the line.