Skip to Content
Agent computeRuns & logs
Raw

Runs and logs

Listing the runs of an Aetherfy agent

A run on Aetherfy is one ephemeral execution of a type: job agent. List recent runs newest-first:

afy runs nightly-report afy runs nightly-report --limit 50

The default limit is 20 and the maximum is 100.

The listing shows six columns:

ColumnContains
WhenWhen the run was created, in UTC and labelled UTC — the instant --json gives as created_at
TriggerWhy it ran — cron or manual, or spawn (by <agent>) for a run another agent spawned on this one
StateHow it ended, or where it is
ReleaseThe release the run executed, as afy deployments numbers it (v3), or a dash when none was recorded
DurationHow long the machine ran
Run IDThe identifier to pass to afy logs --run

Aetherfy lists every run the agent executed: a scheduled one, a manual one, and one another agent spawned on it. A run is the history of the agent that executed it, so a spawned run is in the child’s listing, naming the agent that spawned it.

What the Aetherfy runs API returns

The equivalent Aetherfy REST call is:

GET /agents/{id}/runs
Query parameterAccepted valuesOn anything else
trigger_sourcecron, manual or spawn only422
limit1–100422
beforeThe cursor of the last run of the previous page422

Each row Aetherfy returns carries these fields:

FieldContains
idThe run’s identifier
trigger_sourcecron, manual, or spawn
stateSee the state table below
created_atWhen the run was created
error_messageThe failure reason, when the run failed
release_versionThe release the run executed — see below
machine_started_atWhen the machine began executing
machine_stopped_atWhen the machine stopped
duration_secondsExecution duration
cursorWhere this row sits in the list, to pass as before

The two machine fields are null on a run of a type: service agent, which is a request to a machine the service already has rather than a machine of its own. Its duration_seconds runs from the moment that machine accepted the run to the moment the run’s outcome was reported.

release_version is the version of the release a run executed — the number afy deployments and afy rollback use, not the run’s own place in the agent’s deployment history. A type: job run executes the release whose image it was started from, and that stays true when the image was gone and the run rebuilt it from stored code. After an exact rollback, the rebuilt code comes from the version you rolled back to, but the run still names the release the rollback created. A type: service run executes the release the machine that took it was serving. It is null for a service run no machine accepted, and for runs recorded before Aetherfy kept it.

The agent record carries its most recent entry of this list as last_run, which is what afy status prints as Last run.

before is a keyset cursor rather than an offset, so paging backwards through a long history stays correct even as new runs arrive.

Run and deployment states on Aetherfy

Aetherfy uses one state vocabulary for both deployments and runs. Four of the states are terminal — once a record reaches them it will not change again.

StateMeaningTerminal
queuedWaiting in the build queueNo
buildingThe image is buildingNo
deployingMachines are launching and being health-checked, or, for a run, the run is being handed to a machineNo
activeServing, or executing in the case of a runNo
completedA run’s machine ended on its own with exit code 0Yes
failedThe build, the deploy, or the run failedYes
supersededReplaced by a newer successful deploymentYes
rolled_backReplaced by a rollbackYes

completed applies to runs, which end by design. A long-lived service deployment that is working sits at active and stays there until a newer deployment supersedes it.

A run is active only once a machine has accepted it. Until then it is deploying, and a run that no machine accepts within 2 minutes of Aetherfy starting to hand it over is failed with “The run was never accepted by a machine, so your code was not executed.” Your code did not start, so there is nothing of yours to debug.

A run that stalls before that point, still queued, building or deploying 7 minutes after it was created, is also recorded as failed, with a reason naming the step that never finished. The 7 minutes run from when the run was created, so time the run spends queued behind other deployments counts toward them (see /agents/task-contract). Aetherfy checks for stalled runs every 5 minutes, so such a run can show its pending state for up to 5 minutes past those 7 minutes before it is failed.

What decides completed versus failed for a run is the process exit code — see /agents/task-contract.

Agent status values on Aetherfy

An agent has its own status, separate from the state of any individual deployment or run.

StatusMeaning
pendingCreated, not yet built
buildingAn image is building
deployingA version is launching and being health-checked; a version already deployed keeps serving meanwhile
runningDeployed and operating
failedThe most recent deployment failed and no earlier version is serving. A redeploy that fails leaves the agent running on the version it was replacing
pausedPaused with afy stop — every machine is stopped, and incoming traffic does not wake it. Resume with afy start
stoppedStopped by Aetherfy because the account was suspended for billing. You do not set this and cannot clear it directly — settling the account returns the agent to running. See Billing and spend caps
usage_pausedPaused by Aetherfy because usage was used up — at a spend limit, or, on Free, once the agent is awake beyond 75% of the month. Shown as paused for usage
archivedArchived, freeing the plan quota slot
deletingDeletion in progress
deletedDeleted

usage_paused is the one to recognise on sight: the agent is not broken and nothing needs repairing in your code. On a paid plan, raise the limit or settle the payment at https://app.aetherfy.com/dashboard/settings/billing  and the agent resumes. A Free account has no spend limit: its agent is paused because it was awake beyond 75% of the month, and it resumes by itself within a day of the new month starting (UTC), or as soon as you upgrade.

Retrieving logs from Aetherfy

Everything an agent writes to stdout and stderr becomes its logs on Aetherfy.

afy logs nightly-report

Scope the output to a single run, using a Run ID from afy runs:

afy logs nightly-report --run 9f3a1c72-5b8e-4d61-a0f4-7c2e9b5d3a18
FlagShortDefaultEffect
--tail-n50How many lines to return. The server maximum is 1000.
--follow-foffKeep streaming new lines as they arrive
--since——Only lines newer than a duration, e.g. 1h, 30m
--level——Comma-separated severities, e.g. ERROR,WARN
--stream——Comma-separated streams: stdout, stderr, system
--run——Restrict to a single run id. On a type: service agent this returns Aetherfy’s lines about that run — sent, answered, how long — not your service’s own output

Combined:

afy logs nightly-report --tail 200 --since 1h --level ERROR,WARN --stream stderr

One caveat worth knowing before you debug against it: on Aetherfy --follow polls, and it ignores both --tail and --since. It also does not honour JSON output. Use --follow to watch what happens next; use --tail and --since to look at what already happened.

Log record fields on Aetherfy

Every log record Aetherfy stores carries:

FieldContains
agentWhich agent emitted it
deploymentWhich deployment was running
timestampWhen the line was emitted
streamstdout, stderr, or system
levelThe severity
messageThe line itself

The system stream holds lines Aetherfy itself emits about the machine, as distinct from your program’s own output on stdout and stderr. Their message carries no tag of its own: afy logs prints each line as [level] message, so a platform line reads [SYSTEM] …, and one Aetherfy writes at ERROR reads [SYSTEM ERROR] … — never like your program’s own [ERROR] lines. In the dashboard the stream is its own column, and afy logs --stream system selects only Aetherfy’s lines.

The platform also writes into this stream when an agent’s own key is refused or removes data from the vector database. Each is one line, in plain words:

What happenedThe line
The agent’s key was refused (its access, a protected collection or the plan’s collection limit)Refused DELETE /api/v1/workspaces/research/collections/kb: … followed by the refusal’s sentence, such as This agent's key holds read and write access in workspace "research", and this request needs full access. …
The agent deleted a collectionDeleted collection "kb" in workspace "research".
The agent deleted pointsDeleted 12 points from collection "kb" in workspace "research". The count is of points that existed; a delete by filter counts the points the filter selected.

The same refusal of the same request repeated within a minute is one line ending (3 times), and deletes from one collection within a minute add up to one line ((4 requests)). The lines are written in batches, so one appears within about a minute of the event, from any region. Reading data, and anything an account key does, writes no line. See agent access.

How Aetherfy reads a line’s level

Your output carries no level field, so Aetherfy reads one from the start of each line, and only from there. A level is recognised when the line begins — after an optional timestamp such as 2026-09-17 12:40:36,123 or [2026-09-17T12:40:36Z] — with one of DEBUG, INFO, WARN, WARNING, ERROR, CRITICAL, FATAL or TRACE, written in one of three ways:

Written asExample
In square brackets, any case[error] upstream closed
Followed by a colon, any caseerror: config file missing
In capitals, followed by a spaceERROR disk full

WARNING is stored as WARN, CRITICAL and FATAL as ERROR, and TRACE as DEBUG. A line with no level at its start takes its stream’s: INFO on stdout, ERROR on stderr. The word anywhere else is part of the message — No Error found on stdout is INFO — and so is a capitalised word opening a sentence, such as Error handling is on. To set a line’s level, print it in one of the three forms above.

What Aetherfy does not record for you

Aetherfy logging is plain stdout/stderr capture, and nothing more. It does not auto-instrument calls your agent makes to model providers — there is no automatic capture of prompts, completions, token counts, latency or tool calls from an OpenAI, Anthropic or other SDK running inside your agent.

If you want that, log it yourself. A print() or logger.info() around the call lands in exactly the same place as the rest of your output and is searchable with afy logs --level:

import json, time started = time.monotonic() response = call_your_model(prompt) print(json.dumps({ "event": "llm_call", "model": "your-model-id", "elapsed_ms": round((time.monotonic() - started) * 1000), "prompt_chars": len(prompt), }))

Keep the per-line and per-deployment limits below in mind if you log full prompts — they are easy to exceed.

Log limits and retention on Aetherfy

Aetherfy caps log volume in several dimensions. Logs are an operational aid, not a durable data store — write anything you need to keep to a real destination.

LimitValue
RetentionYour plan’s: 3 days on Free, 14 on Starter, 30 on Performance, negotiated on Enterprise
Stored per accountYour plan’s, across all your agents: 50 MB on Free, 250 MB on Starter, 1 GB on Performance, negotiated on Enterprise. A line counts its message bytes, at least 64. The oldest lines are removed first, and an account may run over its size for up to an hour
Per deployment5 MB, every line counting at least 64 bytes
Per line4 KB
Chunks per minute, per agent60
Per upload body256 KB

Deleting an agent deletes its logs once its teardown finishes; the retention applies to agents that still exist.

Your plan keeps logs for its retention and at most its size per account, and whichever limit is reached first applies: 3 days and 50 MB on Free, 14 days and 250 MB on Starter, 30 days and 1 GB on Performance, and negotiated on Enterprise. The size counts each line’s message bytes, at least 64 for a short line, across all your agents. When an account is over its size, the oldest lines are removed first, so the newest lines are never the ones lost. The cleanup runs hourly, so an account may run over its size for up to an hour before the oldest lines go.

afy logs never shows a log older than your plan’s retention. A --since that reaches further back than the retention still shows what is kept, and afy logs says on stderr that the window was cut to your plan’s retention. When the size limit has removed lines the retention would still have kept, afy logs and the dashboard’s Logs tab say so, with the time before which nothing is left. With --json the output is an object with logs, retention_days (null where the retention is negotiated) and since_clamped; see logs.

A downgrade shortens the window and lowers the size at once for what you can read (the window) and at the next hourly cleanup for what is stored. An upgrade does not bring back logs the cleanup has already deleted.

When an agent exceeds the per-line or volume caps, Aetherfy drops the excess lines and emits a marker in the stream so the gap is visible rather than silent. As afy logs prints it:

[SYSTEM] N log line(s) dropped

If you see that marker, the agent is logging faster than Aetherfy will accept. Reduce per-line size or log volume — a very large payload dumped per iteration is the usual cause. Short lines are not free: each one counts at least 64 bytes against the per-deployment cap, so a stream of empty or one-word lines reaches it after about 80,000 lines.

A run’s output is read to its last line before the run’s outcome is reported. If your code starts a background process that keeps the run’s output open after your main process exits, Aetherfy stops reading 2 seconds later, ends the run, and says so in the run’s log. As afy logs prints it:

[SYSTEM] output from a process run <run id> left running was cut off 2s after the run ended

Anything that background process prints after that point is not captured.

When Aetherfy cannot put an idle agent to sleep

A service with auto-sleep is put to sleep by Aetherfy once it has had no real request for its idle_timeout_minutes, from inside the machine. If that does not work, the agent’s system stream says so, and Aetherfy puts the machine to sleep another way (below). As afy logs prints it:

[SYSTEM ERROR] IDLE_SUSPEND_FAILED machine=<machine id> reason=<reason> attempts=<n> failing_for_s=<seconds> every_s=<seconds>: this idle machine could not be put to sleep and stays awake; it is retried every <interval> [SYSTEM] IDLE_SUSPEND_RECOVERED machine=<machine id> after=<n>: the machine was put to sleep after <n> failed attempt(s)
  • IDLE_SUSPEND_FAILED is written once for each reason, the first time it happens while the machine is idle, and again every ten minutes while it keeps happening. attempts counts the tries so far, failing_for_s how long it has been failing, and every_s how long until the next such line. machine is the machine’s id, as GET /agents/{id}/status lists it. A line may also carry dropped=<n>: that many earlier lines of this kind were lost before they could be delivered.
  • IDLE_SUSPEND_RECOVERED is written when the machine is finally put to sleep after failing, with the number of failed tries.

The reason says what stood in the way, in Aetherfy’s terms:

ReasonWhat it means
permission_deniedThe process the machine runs as was not allowed to ask for sleep
socket_missingThe machine had no way to ask for sleep
connection_refusedThe request to sleep was refused before it was read
timed_outThe request to sleep got no answer in time
unreachableThe request to sleep could not be made, for any other reason
not_configuredThe machine did not know its own identity, so it could not ask
http_<status>Aetherfy’s platform answered the request with that status instead of accepting it

Aetherfy then puts the machine to sleep itself. When an IDLE_SUSPEND_FAILED line arrives, Aetherfy suspends that machine from outside it, within seconds. If the platform will not suspend it, Aetherfy stops it instead: the next request then starts the machine fresh, a cold start rather than a resume, and your agent’s in-memory state is gone. Either way the machine is billed as awake until it is asleep, like any other awake time, and these lines count against the log limits above like any other line.

There is nothing to change in your agent. Every reason above is on Aetherfy’s side. If an agent keeps writing IDLE_SUSPEND_FAILED, contact support with the agent and the line.

Last updated on