Working with a link API: pagination, rate limits and retries

Pagination: cursors beat offsets

Listing links with page numbers and offsets works until the list changes while you are reading it. A link created during your loop shifts everything, so you either see a record twice or miss one entirely. Cursor-based pagination, where each response hands you an opaque token for the next page, is stable under concurrent writes and is what you should use if the API offers it.

Whichever you use, never assume the page size you asked for is the page size you got. Read until the API says there is no more, rather than counting pages yourself.

Rate limits are a contract, not an obstacle

Read the response headers. Most APIs return the limit, the remaining budget and the reset time on every response, which means you can pace yourself precisely instead of guessing. A client that watches the remaining budget and slows down before hitting zero never sees a 429 in normal operation.

Back off exponentially, and add jitter

When you do get a 429 or a 5xx, wait and retry with an exponentially growing delay. Add a random component: without jitter, every worker that failed at the same moment retries at the same moment, and you rebuild the exact spike that caused the problem. Honour a Retry-After header when one is present, because it is more accurate than your formula.

Retry only what is safe to retry

A failed GET can always be retried. A failed POST might have succeeded on the server before the connection dropped, so retrying it can create a duplicate link. Either use an idempotency key so the server can recognise the repeat, or make creation naturally idempotent by specifying the slug yourself and treating a conflict response as success.

Handle partial failure in bulk operations

Bulk endpoints frequently return a per-item result rather than a single status. A 200 on the request does not mean every item succeeded. Parse the per-item outcomes, log the failures with enough context to fix them, and never assume that a successful HTTP status means the work is done.

Cache what does not change

Domains, tags and workspace membership change rarely. Fetching them on every operation wastes your rate budget on data you already have. Cache them for minutes at least, and use conditional requests with an ETag where the API supports it, so an unchanged resource costs a small response instead of a full one.

Log the request id

Most APIs return an identifier on each response. Store it alongside your own logs. When something goes wrong and you need support, a request id turns a vague description into a specific lookup, and it is the difference between a same-day answer and a week of correspondence.