Diagnostics
A diagnostic bundle is a read-only snapshot of a host's state, collected by
pulse-agent and stored by Pulse. It answers "why did this break" without
opening an SSH session — useful when the host is remote or hard to reach.
Today bundles are collected on demand with pulse-agent --diagnose. Attaching
them to incidents automatically is not yet implemented.
What a bundle contains
| Section | Source | Contents |
|---|---|---|
kernel |
dmesg |
Out-of-memory kills, with the process, PID, and resident size |
disk |
df -Pk |
Usage per mount, and which mounts are at capacity |
processes |
ps |
The 15 busiest processes by CPU share |
docker |
docker ps -a, docker logs |
All containers with state and exit status, which are not running, and recent output from those |
systemd |
systemctl list-units --failed, journalctl |
Units in the failed state, and the recent journal for each |
proxmox |
qm list |
Guest VMs and their status, and which are not running |
Every collector is read-only. The agent runs a fixed set of commands and never executes anything supplied by the server.
df is invoked with -k deliberately: POSIX -P alone reports 512-byte blocks
on BSD and under POSIXLY_CORRECT, which would make available_kb double the
true value. The unit is part of the contract, not an implementation detail.
Logs are captured only for what already failed
An OOM kill tells you a service died; its log tells you why. The agent pulls logs only for units already in the failed state and containers already stopped — it never takes a log target from the server.
Capture is bounded so a bundle stays sendable: the last 200 lines, for at most
5 failing units and 5 stopped containers, truncated to 32 KiB each. Truncated
logs keep their tail, since that is where the failure is, and are marked
[truncated: log exceeded capture limit].
Sections degrade independently
A host that denies dmesg, has no Docker, or is not a Proxmox node still
produces a useful bundle. Collectors that cannot run record their error in
their own section and leave the rest intact:
{
"sections": {
"disk": { "data": { "mounts": [], "full": ["/"] } },
"proxmox": { "error": "qm list: exec: \"qm\": executable file not found in $PATH" }
}
}
Each command is individually timeout-bounded, so a wedged command cannot stall collection — which matters most on exactly the hosts worth diagnosing.
A failing command's own output is folded into its section error, so the reason survives rather than a bare exit status:
"error": "docker ps: exit status 1: failed to connect to the docker API ..."
"error": "dmesg: exit status 1: usage: sudo dmesg"
Reading the kernel ring buffer requires root on most systems. Run the agent as
root — as the systemd unit does — or the kernel section degrades.
Mounts on filesystems with no backing store are listed but never reported as
full, because they read 100% permanently: devfs, devtmpfs, udev, proc,
sysfs, and efivarfs.
Everything else is flagged when it reaches capacity, including tmpfs,
overlay, and loop-backed mounts. A full tmpfs is real memory-backed
exhaustion and a full overlay is a container's writable layer filling up, so
both are actionable.
Loop devices are a deliberate trade-off. A read-only image mount such as a snap
package sits at 100% permanently and will be reported as full, but df -Pk
names the device rather than the filesystem type, so it cannot be distinguished
from a writable loop-mounted ext4 or XFS volume. Reporting a harmless snap mount is the
lesser failure — hiding a genuinely full filesystem is a missed incident.
Installing the agent
pulse-agent is published with each release for Linux and macOS on amd64 and
arm64. Install it on the host you want to diagnose:
VERSION=$(curl -fsSL https://api.github.com/repos/memetics19/pulse/releases/latest | grep -o '"tag_name": *"[^"]*"' | cut -d'"' -f4)
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/')
BASE="https://github.com/memetics19/pulse/releases/download/${VERSION}"
curl -fsSLo pulse-agent "${BASE}/pulse-agent_${OS}_${ARCH}"
curl -fsSLo SHA256SUMS "${BASE}/SHA256SUMS"
grep " pulse-agent_${OS}_${ARCH}\$" SHA256SUMS | sed "s|pulse-agent_${OS}_${ARCH}|pulse-agent|" | shasum -a 256 -c -
chmod +x pulse-agent && sudo mv pulse-agent /usr/local/bin/
Run it as root so the kernel section can read the ring buffer; without that
the section degrades to a permission error. On a Proxmox node, install it on the
host rather than inside a guest: a hung VM cannot report on itself, so the
host-side view is the only reliable one.
Pulse does not ship a systemd unit for the agent. Invoke it directly, or write your own unit if you want it scheduled.
Collecting a bundle
Print a bundle locally, without contacting Pulse:
pulse-agent --diagnose
This works even when Pulse itself is unreachable, which is when you are most likely to need it.
The agent targets Linux. On macOS the processes section degrades, because
ps --sort is procps-specific, and systemd and proxmox are absent.
Collect and upload in one step by supplying the agent's credentials:
PULSE_AGENT_TOKEN=<agent-token> pulse-agent --diagnose --server https://status.example.com
On a host with the pulse-agent package, read the token from its file instead:
sudo pulse-agent --diagnose --server https://status.example.com --token-file /etc/pulse-agent/token
The token is the one shown once when the agent was created. The agent never
takes the token as a command line argument, because any local user can read
arguments through ps and /proc. See Architecture for how
agents are registered.
API
POST /api/ingest/diagnostics
Authorization: Bearer <agent-token>
{
"bundle": { "collected_at": "...", "sections": {} }
}
A bundle belongs to the agent that produced it, identified by the bearer token,
and carries no other association. Uploads are limited to one per agent every
five seconds; a faster caller receives 429 with Retry-After. The agent
refuses to send a bundle larger than 900 KiB, which would be rejected by the
1 MiB request cap anyway — the local copy is still printed. The server stores it verbatim, so collectors
can change without a server-side migration. The bundle must be a JSON object;
its section contents are not validated. Responds 204 No Content.
Bundles fall under the same retention_days window as check results and are
removed by the retention worker. Lowering that setting therefore discards older
diagnostic evidence as well as older check history — see
Data retention.
Reading bundles back
GET /api/agents/{agentID}/diagnostics?limit=5
Authorization: Bearer <api-key>
Returns an agent's recent bundles, newest first, as JSON. Requires the
diagnostics:read scope — bundles carry journal entries, container logs,
process names, and filesystem paths, so they need an explicit grant rather than
riding on agents:read. Reading a host's evidence is an admin action, not
something the agent's own ingest token can do.
limit defaults to 3 and is capped at 5. Bundles are large, so the ceiling is
deliberately low.