Robots policy and what search engines see

Short links should generally not be indexed. They are pointers, they duplicate content that already has a canonical home, and an indexed short link competes with the page it points at.

Search engine policy

Domain settings, Robots policy, offers three values:

• Allow indexing — crawlers may index short links. Rarely what you want.
• Don't index short links — a noindex directive. Crawlers may follow the redirect but should not list the short URL.
• Don't follow short links — crawlers are asked not to follow. This also stops well-behaved preview bots from consuming clicks against a click limit, which matters if you use limited links.

robots.txt behaviour

The domain serves its own robots.txt, and you choose how:

• Redirect to the origin robots.txt — a 301 to the robots.txt of your main page redirect's origin, so one file governs everything.
• Redirect to origin (302, immediate) — the same, without the permanent cache.
• Serve a custom robots.txt — the text you paste is served verbatim. Use this when you want rules specific to the short domain.
• Disallow everything — serves User-agent: * and Disallow: /.

With no explicit mode, a file is generated from the search engine policy above. There is a link in the settings panel to open the live file, which is worth checking after any change.

Preview bots are a separate concern

Messaging apps and social networks fetch a link before showing a card. Those requests are recorded as clicks with the bot flagged, and they count against a click limit unless you set the policy to "Don't follow". If a link has social metadata set, a preview crawler is served an HTML page carrying those tags rather than a redirect — otherwise the crawler follows the redirect and reads the destination's tags, and everything you configured is ignored.

Statistics and bots

Bot traffic is never dropped silently. Every request that reaches the redirector is logged, with a human flag and the detected bot name, and separated afterwards in reporting. That is why the statistics dashboard shows total clicks and human clicks as two different numbers, and why a link that looks busy the moment you post it usually is not.

Find this in your dashboard

1. Sign in and select the workspace and domain you want to manage.
2. Open Domains. Review the article’s steps and your plan’s available controls.
3. Save your changes, reopen the record and verify the saved result. For routing changes, check the short link with a test visit.

API workflow guides · Open dashboard