Homelab & Self-hosting8 min read

Docker Container Monitoring: Restart Loops, Healthchecks and Image Updates

Up 4 seconds isn't up. How to catch restart loops, failing healthchecks and stale images on your Docker hosts.

The Ensju team
On this page

docker ps is an optimist. A container that crashes every thirty seconds and gets restarted shows as "Up 4 seconds". A container whose app has deadlocked shows as "Up 3 weeks". And a container running a tag that's three security releases behind looks exactly like one that's current. Watching a Docker host means watching for the states that look fine.

The short version

  • A restart policy hides crashes: alert on a rising restart count, not on the container being up.
  • Give important containers a HEALTHCHECK so Docker can tell you when the app inside is broken.
  • Only alert on exited containers you expect to run forever; one-shot jobs exit on purpose.
  • Tags move: compare the digest you run with the one the registry serves to know when to update.
  • Reading the Docker socket is root-equivalent, so make it an explicit opt-in.

The failures that matter

FailureWhat docker ps showsWhat to watch
Crash loopUp 4 secondsRestart count rising
App hung or broken insideUp 3 weeksHealthcheck status
Container goneExited (1)State, for containers that should run
Running an old imageUp 3 weeksRunning digest vs. the registry's
One container starving the hostUp 3 weeksPer-container CPU and memory

Restart loops

With a restart policy like unless-stopped or always, Docker brings a crashed container straight back, which is great for uptime and terrible for noticing that something is wrong. The tell is the restart count:

How many times has it restarted?
docker inspect --format '{{.RestartCount}}' my-container

One restart after a host reboot is normal. Three in ten minutes is a crash loop. That's the default Ensju alerts on, and you can tune both numbers per server.

Healthchecks: let Docker say "broken"

A process can be running and still be useless: deadlocked, out of database connections, returning 500s. A healthcheck gives Docker a way to notice. Add one to any container you care about:

docker-compose.yml
services:
  app:
    image: ghcr.io/example/app:1.4
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8080/health || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3

Docker then reports the container as healthy or unhealthy. Alert on unhealthy, but only after it holds for two check-ins in a row: a single failed probe is noise, and a container that fails once and recovers shouldn't page anyone.

Tip

A healthcheck tests the app from inside the container. It's worth pairing important services with an outside uptime check on their real URL too, which catches the proxy, DNS and certificate problems a healthcheck can't see.

Containers that should never stop

Plenty of containers are meant to exit: migrations, one-shot jobs, anything you stopped yourself. Alerting on every exited container would page you about normal housekeeping on day one. Instead, list the handful that should never stop (the database, the reverse proxy, the app itself) and alert only when one of those exits.

Is my image out of date?

A tag like nginx:1.27 isn't a fixed image. The registry moves it forward as patch releases and security fixes ship, but your container keeps running whatever it pulled the day you started it. The way to know you're behind is to compare digests: the one your container runs, and the one the registry serves for that tag today.

Ensju does that check centrally rather than on your hosts. Your agent reports which image and digest each container runs and never contacts a registry. Ensju then makes one anonymous HTTPS HEAD request per unique tag, about every six hours, and shows an arrow next to any container whose digest differs. HEAD requests don't count toward Docker Hub's pull limits, and one request covers every host running that tag.

When you see the arrow, update on your own schedule:

Update a Compose project
docker compose pull && docker compose up -d

When there's no arrow

No arrow doesn't always mean up to date. The check can't see:

  • Private registries, or ones on your LAN.
  • Images you built locally (docker build), which have no registry copy.
  • Digest-pinned images (image@sha256:...), where the digest is the version.

Per-container resources

When a host slows down, the first question is which container did it. Per-container charts of CPU, memory, network and disk I/O answer it. Two details make the numbers easier to read:

  • CPU as a share of the whole host. A container maxing one core on a 16-core box shows about 6%, not 100%, so the stacked chart adds up to the machine.
  • Memory as the working set, without the page cache. A container that reads a lot of files doesn't look like it's leaking, and the number is the one the OOM killer cares about.

History follows your plan: 7 days on Free, 30 on Homelab.

Setting it up with Ensju

Docker reporting is opt-in, because reading the Docker socket is root-equivalent. Tick Report Docker containers in the install dialog, or add ENSJU_DOCKER=true to the install command:

Linux install with Docker reporting
curl -sSL https://ensju.com/agent/install.sh | sudo ENSJU_TOKEN=ensrv_xxxxxxxx \
  ENSJU_DOCKER=true sh
  • Container stats need agent 0.5.0 or newer; restart-loop, unhealthy and exited alerts need 0.6.0 or newer. Docker reporting runs on Linux.
  • The agent only reads the socket: it never starts, stops, restarts or updates a container.
  • Rootless Docker or Podman? Point the agent at your socket with ENSJU_DOCKER_SOCKET.

The alerts follow the defaults above with nothing to configure, and you can tune them per server. Containers created by Compose are named project/container, so you can tell app/web from staging/web. You get one alert per container until it recovers, and at most ten from a single check-in, so a host-wide problem gives you the signal without a hundred pages. It's on every plan, including Free with 1 server agent; Homelab ($10/mo) covers 5.

Running ZFS or a UPS on the same box? The same agent can watch those too: see monitoring ZFS, SMART and UPS.

Frequently asked questions

How do I know if a Docker container is stuck in a restart loop?
Its restart count keeps climbing: check it with docker inspect --format '{{.RestartCount}}' <container>. A restart policy like unless-stopped hides the crash from docker ps, which shows the container as up for a few seconds at a time. Alert when the count rises several times within a few minutes.
How can I tell when a newer Docker image is available?
Compare the digest your container runs with the digest the registry currently serves for the same tag. When the registry has moved the tag forward, the digests differ and a pull would get you the newer image. Ensju does that comparison centrally and shows an arrow next to the image.
Is it safe to give a monitoring agent access to the Docker socket?
Access to the Docker socket is effectively root on the host, so it should always be an explicit choice. Ensju's agent only reads the socket when you install it with Docker reporting on, and it only reads: it never starts, stops or changes a container.
Does Ensju update or restart containers for me?
No. It tells you a container is crash-looping, unhealthy or behind its tag, and leaves the fix to you. Monitoring that changes production on its own is a different kind of tool.
Does it work with Docker Compose?
Yes. Containers created by Compose are grouped and named project/container in alerts, so you can tell app/web from staging/web at a glance.

Get the next one in your inbox

Practical, vendor-neutral monitoring tips — what to watch and how to alert on it — plus the occasional Ensju update. No spam, unsubscribe anytime.

Monitoring tips + Ensju updates. No spam, unsubscribe anytime.