This host publishes the public signing key for a research crawler. If you are
reading this because Signature-Agent: "https://agent.hiem.al"
appeared in your logs, that request came from us.
It requests the home page of sampled sites, and occasionally a few pages linked from it, to measure how origins treat different declared crawler identities. It reads only what is publicly served. It does not submit forms, authenticate, or attempt to reach anything not linked from the page it was given.
Every request carries a From header with a working contact
address, and robots.txt is fetched and obeyed
per identity: a path disallowed for the token a request declares is
not fetched. Requests to one host are serialised with a delay between them.
A User-Agent string is a claim anyone can make. Web Bot Auth (RFC 9421 HTTP Message Signatures, with the conventions in draft-meunier-web-bot-auth-architecture) lets a client prove which key it holds instead. This crawler signs so that the claim it makes is one you can check rather than one you have to take on trust.
curl -s https://agent.hiem.al/.well-known/http-message-signatures-directory
The directory is a JWKS. Take keyid from the request's
Signature-Input, find the matching kid, and verify
the Ed25519 signature over the covered components
(@authority and signature-agent).
Disallow the token aicloak-research in your
robots.txt and we will stop, or write to the address below and
we will exclude your domain by hand. Neither requires a reason.
User-agent: aicloak-research Disallow: /