DOCS // BOT TRACKING

Monitor AI crawler traffic in your Next.js app

Track every time a known AI bot crawls your site. Lightweight middleware, zero dependencies beyond Next.js, 5-minute setup. Free for all CLICON clients.

What is bot tracking?

AI crawlers from Anthropic, OpenAI, Perplexity, Google, and others now regularly scan the web. Knowing when and what they crawl on your site is critical infrastructure for the agentic era.

The CLICON bot tracking middleware detects 9 known AI bots by User-Agent and reports each hit to your Lotus dashboard in real time. No performance impact, no client-side JavaScript, no extra dependencies.

9 bots detected

The middleware matches User-Agent headers against these known AI crawlers using case-insensitive regex patterns:

PerplexityBot
Perplexity AI crawler
GPTBot
OpenAI GPTBot crawler
Claude-Bot
Anthropic ClaudeBot (matches Claude-Bot and ClaudeBot)
Google-Extended
Google Extended (Gemini training opt-out crawler)
Bingbot
Microsoft Bing crawler
Applebot-Extended
Applebot Extended (Apple Intelligence crawler)
Meta-ExternalAgent
Meta External Agent (Llama training crawler)
Bytespider
ByteDance Bytespider crawler
CCBot
Common Crawl bot

Installation

Three steps. Five minutes. No new dependencies.

STEP 1

Set LOTUS_TRACKING_KEY in Vercel

Add the environment variable LOTUS_TRACKING_KEY to your Vercel project. You can find your key in the Lotus admin panel under Settings → API Keys. The middleware uses this key to authenticate tracking requests to the Lotus API.

STEP 2

Add the middleware to your Next.js project

Copy this file to middleware.ts at the root of your Next.js project. This single file handles both bot detection and i18n routing. If you already have a middleware, merge the bot detection block into yours — it runs before your existing logic.

middleware.tstypescript
import createMiddleware from "next-intl/middleware";
import { NextRequest } from "next/server";
import { routing } from "./i18n/routing";

const intlMiddleware = createMiddleware(routing);

const KNOWN_BOTS: Array<{ pattern: RegExp; canonical: string }> = [
  { pattern: /PerplexityBot/i,      canonical: "PerplexityBot" },
  { pattern: /GPTBot/i,             canonical: "GPTBot" },
  { pattern: /Claude-?Bot/i,        canonical: "Claude-Bot" },
  { pattern: /Google-Extended/i,    canonical: "Google-Extended" },
  { pattern: /bingbot/i,            canonical: "Bingbot" },
  { pattern: /Applebot-Extended/i,  canonical: "Applebot-Extended" },
  { pattern: /meta-externalagent/i, canonical: "Meta-ExternalAgent" },
  { pattern: /Bytespider/i,         canonical: "Bytespider" },
  { pattern: /CCBot/i,              canonical: "CCBot" },
];

function detectBot(userAgent: string): string | null {
  for (const { pattern, canonical } of KNOWN_BOTS) {
    if (pattern.test(userAgent)) return canonical;
  }
  return null;
}

export default async function middleware(req: NextRequest) {
  const userAgent = req.headers.get("user-agent") || "";
  const bot = detectBot(userAgent);

  if (bot) {
    const trackingKey = process.env.LOTUS_TRACKING_KEY;
    if (trackingKey) {
      const url = new URL(req.url);
      const clientDomain = url.hostname.replace(/^www\./, "");
      try {
        await fetch("https://lotus.clicon.app/api/v1/track/bot-hit", {
          method: "POST",
          headers: {
            "Content-Type": "application/json",
            "X-API-Key": trackingKey,
          },
          body: JSON.stringify({
            client_domain: clientDomain,
            bot_name: bot,
            endpoint_accessed: url.pathname,
          }),
        });
      } catch (err) {
        console.error("[bot-tracking] POST failed:", err);
      }
    }
  }

  return intlMiddleware(req);
}

export const config = {
  matcher: [
    "/((?!api|_next|_vercel|outstatic|.*\\..*).*)",
  ],
};
STEP 3

Add LOTUS_TRACKING_KEY to env.d.ts (TypeScript only)

If you use TypeScript, declare the env variable so Next.js picks it up. Add it to your env.d.ts or next-env.d.ts declarations. This step is optional but recommended for type safety.

Verify it works

Deploy your app and run this curl command from your terminal. It simulates a PerplexityBot request to your production domain:

curlbash
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \\
  -o /dev/null -w "%{http_code}\\n" https://tu-dominio.com/

You should see a 200 response. Check your Lotus dashboard — the hit will appear under Bot Traffic within seconds. If you get a different status code, check the troubleshooting section below.

Common issues

Middleware isn't firing

Check that middleware.ts is at the root of your Next.js project (same level as next.config). Also verify your matcher config isn't excluding the route you're testing.

LOTUS_TRACKING_KEY is undefined

Confirm the env var is set in Vercel (Settings → Environment Variables) and redeployed. For local development, add it to .env.local — never commit this file.

Bot hits not showing in dashboard

The tracking POST runs async and its errors are caught silently to avoid breaking your app. Check Vercel function logs for [bot-tracking] entries. Also verify your API key in the Lotus admin panel.

Getting 401 from tracking endpoint

Your LOTUS_TRACKING_KEY is invalid or expired. Generate a new key from the Lotus admin panel and update the env var in Vercel.

Need help?

If you run into issues or need a tracking key, reach out at martin@clicon.app.

← Back to home