Monitoring APIs: Past the Health Endpoint

The trivial health check problem

The most common health endpoint in production returns a hardcoded 200 with the body "ok". It confirms that the web server accepted a connection and that routing works. It cannot detect a failed database connection, an exhausted connection pool, a broken cache, or a downstream dependency that is timing out.

In other words, it stays green through most of the incidents you actually care about.

Dependency-aware health

A better health endpoint verifies the dependencies the service genuinely needs, cheaply. A single lightweight query against the database. A read from the cache. A check that a required downstream is reachable.

Keep it fast and cheap, because it runs constantly. And be deliberate about which dependencies are fatal — a service that returns unhealthy because an optional analytics sink is down will cause more outages than it prevents.

Assert on the contract

The failures that hurt most return 200. A search endpoint returning an empty array because the index is gone. A pricing service silently serving stale cache. An API returning an error object inside a successful response envelope.

• Keyword assertionsRequire a string that only appears in a genuinely correct response. Cheap and catches most soft failures.
• Structural checksVerify the response actually contains the fields clients depend on, catching schema drift after a deploy.
• Non-empty resultsFor endpoints that should always return data, an empty set is a failure even with a 200 status.
• Negative assertionsFail if the body contains an error marker, even when the status line says success.

Latency budgets are failure conditions

For an API, slow is a flavour of broken. If your client library times out at three seconds, a response that takes eight has failed regardless of what it eventually returned.

Set an explicit maximum per endpoint, based on measured p95 with headroom, and treat a breach as a failure rather than a note on a graph.

Monitoring authenticated endpoints

Public endpoints are easy to monitor and rarely representative of what your customers use. The routes that matter need credentials.

• Use a dedicated synthetic account, never a real customer credential
• Give it the narrowest permissions that let the check work
• Set a renewal reminder — expired probe tokens are a top source of false alarms
• Store secrets encrypted, never in a monitor name or URL query string

Write endpoints and idempotency

Monitoring a POST route that creates real records will fill your production database with synthetic junk. Either point the check at an idempotent verification route, or scope it to a synthetic tenant whose data you can purge on a schedule.

The alternative — only monitoring reads — leaves the write path, which is usually where the money is, completely unwatched.