# Rat Radar

> Rat Radar tells a person or an AI agent whether a website will let them collect its data, and the best way to do it. It probes 653 public sites each week and publishes a machine-readable verdict for each: bot protection, robots.txt and AI-crawler policy, sitemap, llms.txt, JSON-LD, and a recommended approach. Free, read-only, by RatLabs.

Data last probed: 2026-10-11. It is a heuristic from one residential machine. Sites can behave differently by country, IP type, or volume. Always respect robots.txt and each site's terms.

## Look up one site
- [Site record](https://ratlabs.tech/radar/api/v1/site/reddit.com.json): replace "reddit.com" with the host, no "www." Fields: verdict (open, guarded, hostile, blocked), hostility_score 0-100, recommended_approach, steps, risks, protections, robots, machine_readable, can_scrape, and playbook (golden_data, best_stack, workflows with a trigger, collect, shape, store, act pipeline, and step-by-step recipes for n8n, Make, Zapier and Apify). A robots.txt ban or a login wall makes a site at least hostile, and recommended_approach is then respect_robots_skip or official_api_or_skip: follow it.
- Not in the dataset? Live check from a datacenter IP: https://rat-radar.github-a18.workers.dev/v1/check?domain=example.com
- Find sites by word (name, category, country, verdict): https://rat-radar.github-a18.workers.dev/v1/search?q=jobs
- May I fetch this URL? Live robots.txt check: https://rat-radar.github-a18.workers.dev/v1/robots?url=https://example.com/products&agent=GPTBot
- How do I extract THIS page? Ranked plan with selectors and proof rows: https://rat-radar.github-a18.workers.dev/v1/xray?url=https://example.com/products&code=1

## Bulk data
- [All sites, one line each](https://ratlabs.tech/radar/api/v1/index.json)
- [All sites, full records](https://ratlabs.tech/radar/api/v1/sites.json)
- [CSV](https://ratlabs.tech/radar/api/v1/sites.csv)
- [Playbook library](https://ratlabs.tech/radar/api/v1/library.json): cross-site workflow templates (job market tracker, price index, news wire, and more), each with the sites that can feed it, a merged pipeline, and n8n, Make, Zapier and Apify recipes.
- [Headline numbers](https://ratlabs.tech/radar/api/v1/summary.json): verdict mix, how many sites block AI crawlers, protection vendors, EU AI-training signals, by category and country
- EU AI training: every site record has `eu_ai_training` (level none, some or strong, and the signals seen: robots.txt AI blocks, ai.txt, CAPTCHAs, login walls), the signals the EDPB's draft Guidelines 03/2026 on web scraping for generative AI count against collecting personal data. Signals, not legal advice.
- [What changed this week](https://ratlabs.tech/radar/api/v1/changes.json) and [history per site](https://ratlabs.tech/radar/api/v1/history.json): weekly snapshots, so an agent can see when a site started blocking.
- Badge for a README: https://ratlabs.tech/radar/api/v1/badge/reddit.com.svg
- [OpenAPI](https://ratlabs.tech/radar/api/v1/openapi.json)

## MCP server (tools for agents)
- Streamable HTTP endpoint: https://rat-radar.github-a18.workers.dev/mcp
- Tools: check_site(domain), get_playbook(domain, goal, platform), search_sites(query), list_sites(category, verdict, limit), can_fetch(url, user_agent), xray_page(url, include_code), xray_diagram(url, include_scene), summary()
- Typical flow: check_site (is it allowed and how) then get_playbook (what to build, for a goal like "price alerts") then can_fetch (this URL) then xray_page (the plan for this page). REST twin: /v1/playbook?domain=&goal=&platform=

## Connect your AI app
Server: https://rat-radar.github-a18.workers.dev/mcp (Streamable HTTP, no key, read-only tools).

### Claude Code (In a terminal)
- Run this once. --scope user makes it available in every project.

```
claude mcp add --transport http --scope user rat-radar https://rat-radar.github-a18.workers.dev/mcp
```

### Claude app (claude.ai and Claude Desktop)
- Open Settings, then Connectors.
- Click Add custom connector.
- Name it Rat Radar and paste the URL below. Leave the OAuth fields empty, then click Add.
- In a chat, turn Rat Radar on from the tools menu. The Free plan allows one custom connector.

```
https://rat-radar.github-a18.workers.dev/mcp
```

### ChatGPT (chatgpt.com, a paid plan)
- Open Settings, then Apps & Connectors, then Advanced settings, and turn on Developer mode.
- Back in Apps & Connectors, click Create.
- Name it Rat Radar, paste the URL below, choose No authentication, confirm you trust it, then Create.
- In a chat, pick Developer mode from the + menu and switch Rat Radar on.

```
https://rat-radar.github-a18.workers.dev/mcp
```

### Cursor (~/.cursor/mcp.json (all projects) or .cursor/mcp.json (one project))
- Add this to the file, then reload Cursor. Rat Radar shows up under Settings, MCP.

```
{
  "mcpServers": {
    "rat-radar": {
      "url": "https://rat-radar.github-a18.workers.dev/mcp"
    }
  }
}
```

### VS Code (.vscode/mcp.json in your project)
- Add this to the file. VS Code shows a Start button above the server; Copilot agent mode can then use it.

```
{
  "servers": {
    "rat-radar": {
      "type": "http",
      "url": "https://rat-radar.github-a18.workers.dev/mcp"
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)
- Add this to the file, then refresh the MCP list in Cascade. Windsurf uses serverUrl, not url.

```
{
  "mcpServers": {
    "rat-radar": {
      "serverUrl": "https://rat-radar.github-a18.workers.dev/mcp"
    }
  }
}
```

### OpenCode (opencode.json (project) or ~/.config/opencode/opencode.json)
- Add this to the file and restart OpenCode.

```
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "rat-radar": {
      "type": "remote",
      "url": "https://rat-radar.github-a18.workers.dev/mcp",
      "enabled": true
    }
  }
}
```

### Codex CLI (In a terminal, or ~/.codex/config.toml)
- Run this once, or add the two lines below it to ~/.codex/config.toml.

```
codex mcp add rat-radar --url https://rat-radar.github-a18.workers.dev/mcp

[mcp_servers.rat-radar]
url = "https://rat-radar.github-a18.workers.dev/mcp"
```

### Gemini CLI (~/.gemini/settings.json)
- Add this to the file and restart Gemini CLI. Check it with /mcp.

```
{
  "mcpServers": {
    "rat-radar": {
      "httpUrl": "https://rat-radar.github-a18.workers.dev/mcp"
    }
  }
}
```

### Anything else (Clients that only run local (stdio) servers)
- Use the mcp-remote bridge (needs Node.js). It runs locally and talks to the server for you.

```
{
  "mcpServers": {
    "rat-radar": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://rat-radar.github-a18.workers.dev/mcp"
      ]
    }
  }
}
```

Then ask: "Can I scrape reddit.com? Use Rat Radar." or "Give me a Rat Radar playbook for price alerts on target.com."

## Privacy and limits
- What is collected and what is not: https://ratlabs.tech/radar/privacy.html
- The edge API is rate limited (about 40 requests a minute per IP, about 10 a minute to any one target site). Please be gentle and cache.

## How to read a verdict
- open: a plain HTTP request gets in.
- guarded: works, but go slow.
- hostile: needs a real browser and proxies.
- blocked: the front door refuses bots. Look for an official API.
- recommended_approach values: plain_http, plain_http_slow, real_browser, browser_plus_residential_proxy, official_api_or_skip, respect_robots_skip.

## Human interface
- [Rat Radar](https://ratlabs.tech/radar/): map, ranked list, and X-Ray for any page.