Skip to main content
This page covers traffic that arrives with the TextQL-Agent User-Agent. These requests come from TextQL agent code running in a sandbox, described below.

​
What TextQL-Agent is

TextQL is an analytics platform. Agent code running in a TextQL sandbox may fetch a URL when a user asks a question that requires data from it, or when customer-configured activity runs, such as a scheduled playbook, a scheduled dashboard or app refresh, or a trigger like a webhook. Requests are driven by customer activity. TextQL does not fetch URLs on its own initiative. TextQL does not crawl. It does not spider sites, it does not follow links to discover new pages, and it does not build an index of web content.

​
How to identify it

Requests that agent code in a TextQL sandbox sends through TextQL’s egress proxy carry a TextQL-Agent token as the first product in the User-Agent header. The token is always present and always first, and its value is exactly:
Everything after the token is the originating client’s own User-Agent, passed through unchanged. The proxy composes the header rather than replacing it, so the header is variable length and its tail differs from request to request. A request from a client that sent python-requests/2.31 arrives with:
A client that sent no User-Agent of its own produces the token alone. TextQL passes the client’s User-Agent through so that site owners and API providers can still see which client library or SDK made the request.
Match the TextQL-Agent token as a substring or a prefix, never by comparing the whole header for equality. The token is stable, the rest of the header is not, and an equality rule silently stops matching as soon as the originating client’s User-Agent changes. That leaves you believing the traffic is blocked when it is not.

​
Rate limits

Each TextQL sandbox is limited to 5,000 requests per minute to a given destination host, enforced at the proxy. This is a per-sandbox limit, not a site-wide ceiling: several sandboxes can each reach it against the same host at the same time. It is a protective limit rather than a guaranteed ceiling, and it fails open if TextQL’s rate limit store is unavailable. If you need a hard ceiling on the traffic you accept, enforce it on your own side.

​
How to opt out

To stop these requests, block the TextQL-Agent User-Agent at your WAF or CDN, matching the token as a substring of the header rather than comparing the whole header for equality. That block takes effect immediately and applies to every TextQL-Agent request that reaches your edge. This page covers only requests that TextQL sandboxes send through TextQL’s proxy. Some TextQL features retrieve web pages through third party services that do not carry this User-Agent, and blocking it does not affect those requests. If you have questions about that traffic, email support@textql.com.
Blocking the User-Agent at your WAF or CDN is what actually stops traffic. TextQL does not read robots.txt in any fetch path, so a robots.txt entry on its own does not block requests.
If you prefer to publish a declarative record of preference, you can add this to your robots.txt:

​
Contact

For abuse reports or questions about this traffic, email support@textql.com.