xseek Logo
Back to integrations
Integration docs·xseek.io

Custom API.

Integrate with our API directly from your server or proxy layer. The most reliable method for tracking AI bot visits with full control and accuracy.

01

Overview

Our API integration is the most reliable way to track AI bot interactions with your website. Unlike client-side JavaScript solutions, server-side detection ensures accurate tracking because AI bots typically don't execute JavaScript - they only fetch HTML content. By implementing detection on your server or through a proxy layer (nginx, Apache, Cloudflare Workers, etc.), you can capture 100% of AI bot traffic. This approach works with any platform and gives you complete control over the tracking implementation.

02

What you get

Most accurate tracking method (bots don't execute JavaScript)
100% coverage of AI bot visits
Full control over implementation
Works with any platform or technology stack
Supports server-side, middleware, and proxy implementations (nginx, Apache, Cloudflare, etc.)
Real-time API with CORS support for flexibility
Secure API key authentication
Comprehensive bot detection patterns included
03

Requirements

  • API key from xseek
  • Website ID from your xseek dashboard
  • Server-side code access, middleware capability, or proxy layer (nginx, Apache, Cloudflare Workers, etc.)
04

Setup process

  1. 01

    Choose your implementation approach: server-side code, middleware, or proxy layer (nginx, Cloudflare Workers, etc.)

  2. 02

    Get your API key from the xseek dashboard

  3. 03

    Implement bot detection in your server code or proxy

  4. 04

    Call the /api/track-ai-bot endpoint when a bot is detected

  5. 05

    Deploy your changes to start tracking AI visits

05

Integration setup

Environment variables requiredrequired
Add these environment variables to your project:
XSEEK_API_KEY=your_api_keyXSEEK_WEBSITE_ID=your_website_id

Your API key can be found in your account settings. Make sure it has the ai_visits:push privilege.

Why Server-Side Detection?

AI bots typically don't execute JavaScript - they only fetch HTML content. Server-side or proxy-based detection ensures you capture 100% of AI bot traffic, not just the small percentage that might execute JavaScript.

✓Recommended: Server-Side Detection

Implement detection in your server code (Node.js, PHP, Python, etc.):

javascript
// Node.js/Express - patterns auto-generated from xSeek's bot database
import express from 'express';

const AI_BOTS = [
  { name: 'anthropic-ai', pattern: /anthropic-ai/i },
  { name: 'claudebot', pattern: /ClaudeBot/i },
  { name: 'claude-web', pattern: /claude-web/i },
  { name: 'claude-user', pattern: /Claude-User/i },
  { name: 'claude-searchbot', pattern: /Claude-SearchBot/i },
  { name: 'claude-code', pattern: /claude-code\//i },
  { name: 'perplexitybot', pattern: /PerplexityBot/i },
  { name: 'perplexity-user', pattern: /Perplexity-User/i },
  { name: 'grokbot', pattern: /GrokBot(?!.*DeepSearch)/i },
  { name: 'grok-search', pattern: /xAI-Grok/i },
  { name: 'grok-deepsearch', pattern: /Grok-DeepSearch/i },
  { name: 'GPTBot', pattern: /GPTBot/i },
  { name: 'chatgpt-user', pattern: /ChatGPT-User/i },
  { name: 'oai-searchbot', pattern: /OAI-SearchBot/i },
  { name: 'google-extended', pattern: /Google-Extended/i },
  { name: 'Google-Agent', pattern: /Google-Agent/i },
  { name: 'applebot', pattern: /Applebot(?!-Extended)/i },
  { name: 'applebot-extended', pattern: /Applebot-Extended/i },
  { name: 'meta-external', pattern: /meta-externalagent/i },
  { name: 'meta-externalfetcher', pattern: /meta-externalfetcher/i },
  { name: 'bingbot', pattern: /Bingbot(?!.*AI)/i },
  { name: 'bingpreview', pattern: /bingbot.*Chrome/i },
  { name: 'microsoftpreview', pattern: /MicrosoftPreview/i },
  { name: 'cohere-ai', pattern: /cohere-ai/i },
  { name: 'cohere-training-data-crawler', pattern: /cohere-training-data-crawler/i },
  { name: 'youbot', pattern: /YouBot/i },
  { name: 'duckassistbot', pattern: /DuckAssistBot/i },
  { name: 'semanticscholarbot', pattern: /SemanticScholarBot/i },
  { name: 'ccbot', pattern: /CCBot/i },
  { name: 'ai2bot', pattern: /AI2Bot/i },
  { name: 'ai2bot-dolma', pattern: /AI2Bot-Dolma/i },
  { name: 'aihitbot', pattern: /aiHitBot/i },
  { name: 'amazonbot', pattern: /Amazonbot/i },
  { name: 'novaact', pattern: /NovaAct/i },
  { name: 'brightbot', pattern: /Brightbot/i },
  { name: 'bytespider', pattern: /Bytespider/i },
  { name: 'tiktokspider', pattern: /TikTokSpider/i },
  { name: 'cotoyogi', pattern: /Cotoyogi/i },
  { name: 'crawlspace', pattern: /Crawlspace/i },
  { name: 'pangubot', pattern: /PanguBot/i },
  { name: 'petalbot', pattern: /PetalBot/i },
  { name: 'sidetrade-indexer', pattern: /Sidetrade indexer bot/i },
  { name: 'timpibot', pattern: /Timpibot/i },
  { name: 'omgili', pattern: /omgili/i },
  { name: 'omgilibot', pattern: /omgilibot/i },
  { name: 'webzio-extended', pattern: /Webzio-Extended/i },
  { name: 'baiduspider', pattern: /Baiduspider/i },
  { name: 'mistralai-user', pattern: /MistralAI-User/i }
];

app.use(async (req, res, next) => {
  const userAgent = req.headers['user-agent'] || '';
  
  let detectedBot = null;
  for (const bot of AI_BOTS) {
    if (bot.pattern.test(userAgent)) {
      detectedBot = bot.name;
      break;
    }
  }
  
  if (detectedBot) {
    const ip = req.ip || req.headers['x-forwarded-for']?.split(',')[0] || '';
    
    try {
      const response = await fetch('https://www.xseek.io/api/track-ai-bot', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'x-api-key': process.env.XSEEK_API_KEY,
        },
        body: JSON.stringify({
          botName: detectedBot,
          userAgent,
          url: req.protocol + '://' + req.get('host') + req.originalUrl,
          ip: ip || undefined,
          referer: req.headers['referer'] || undefined,
          websiteId: process.env.XSEEK_WEBSITE_ID,
        }),
      });

      if (!response.ok) {
        const text = await response.text();
        console.error('[xSeek] Tracking failed', {
          status: response.status,
          statusText: response.statusText,
          body: text,
        });
      }
    } catch (err) {
      console.error('[xSeek] Tracking request error', {
        error: err?.message || err,
        botName: detectedBot,
        url: req.protocol + '://' + req.get('host') + req.originalUrl,
      });
    }
  }

  next();
});

Best for: Self-hosted applications, Next.js, custom backends, WordPress with server access

↻Alternative: Proxy/Edge Detection

Use any reverse proxy (nginx, Apache, Cloudflare Workers, etc.) for platforms without server access:

javascript
// Cloudflare Worker proxy - patterns from xSeek's bot database
// Add secrets: wrangler secret put XSEEK_API_KEY && wrangler secret put XSEEK_WEBSITE_ID

const AI_BOTS = [
  { name: 'anthropic-ai', pattern: /anthropic-ai/i },
  { name: 'claudebot', pattern: /ClaudeBot/i },
  { name: 'claude-web', pattern: /claude-web/i },
  { name: 'claude-user', pattern: /Claude-User/i },
  { name: 'claude-searchbot', pattern: /Claude-SearchBot/i },
  { name: 'claude-code', pattern: /claude-code\//i },
  { name: 'perplexitybot', pattern: /PerplexityBot/i },
  { name: 'perplexity-user', pattern: /Perplexity-User/i },
  { name: 'grokbot', pattern: /GrokBot(?!.*DeepSearch)/i },
  { name: 'grok-search', pattern: /xAI-Grok/i },
  { name: 'grok-deepsearch', pattern: /Grok-DeepSearch/i },
  { name: 'GPTBot', pattern: /GPTBot/i },
  { name: 'chatgpt-user', pattern: /ChatGPT-User/i },
  { name: 'oai-searchbot', pattern: /OAI-SearchBot/i },
  { name: 'google-extended', pattern: /Google-Extended/i },
  { name: 'Google-Agent', pattern: /Google-Agent/i },
  { name: 'applebot', pattern: /Applebot(?!-Extended)/i },
  { name: 'applebot-extended', pattern: /Applebot-Extended/i },
  { name: 'meta-external', pattern: /meta-externalagent/i },
  { name: 'meta-externalfetcher', pattern: /meta-externalfetcher/i },
  { name: 'bingbot', pattern: /Bingbot(?!.*AI)/i },
  { name: 'bingpreview', pattern: /bingbot.*Chrome/i },
  { name: 'microsoftpreview', pattern: /MicrosoftPreview/i },
  { name: 'cohere-ai', pattern: /cohere-ai/i },
  { name: 'cohere-training-data-crawler', pattern: /cohere-training-data-crawler/i },
  { name: 'youbot', pattern: /YouBot/i },
  { name: 'duckassistbot', pattern: /DuckAssistBot/i },
  { name: 'semanticscholarbot', pattern: /SemanticScholarBot/i },
  { name: 'ccbot', pattern: /CCBot/i },
  { name: 'ai2bot', pattern: /AI2Bot/i },
  { name: 'ai2bot-dolma', pattern: /AI2Bot-Dolma/i },
  { name: 'aihitbot', pattern: /aiHitBot/i },
  { name: 'amazonbot', pattern: /Amazonbot/i },
  { name: 'novaact', pattern: /NovaAct/i },
  { name: 'brightbot', pattern: /Brightbot/i },
  { name: 'bytespider', pattern: /Bytespider/i },
  { name: 'tiktokspider', pattern: /TikTokSpider/i },
  { name: 'cotoyogi', pattern: /Cotoyogi/i },
  { name: 'crawlspace', pattern: /Crawlspace/i },
  { name: 'pangubot', pattern: /PanguBot/i },
  { name: 'petalbot', pattern: /PetalBot/i },
  { name: 'sidetrade-indexer', pattern: /Sidetrade indexer bot/i },
  { name: 'timpibot', pattern: /Timpibot/i },
  { name: 'omgili', pattern: /omgili/i },
  { name: 'omgilibot', pattern: /omgilibot/i },
  { name: 'webzio-extended', pattern: /Webzio-Extended/i },
  { name: 'baiduspider', pattern: /Baiduspider/i },
  { name: 'mistralai-user', pattern: /MistralAI-User/i }
];

export default {
  async fetch(request, env) {
    const userAgent = request.headers.get('user-agent') || '';
    const ip = request.headers.get('cf-connecting-ip') || '';
    
    let detectedBot = null;
    for (const bot of AI_BOTS) {
      if (bot.pattern.test(userAgent)) {
        detectedBot = bot.name;
        break;
      }
    }

    if (detectedBot && env.XSEEK_API_KEY && env.XSEEK_WEBSITE_ID) {
      try {
        const res = await fetch('https://www.xseek.io/api/track-ai-bot', {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
            'x-api-key': env.XSEEK_API_KEY,
          },
          body: JSON.stringify({
            botName: detectedBot,
            userAgent,
            url: request.url,
            ip: ip || undefined,
            referer: request.headers.get('referer') || undefined,
            websiteId: env.XSEEK_WEBSITE_ID,
          }),
        });

        if (!res.ok) {
          const text = await res.text();
          console.error('[xSeek] Tracking failed', {
            status: res.status,
            statusText: res.statusText,
            body: text,
          });
        }
      } catch (err) {
        console.error('[xSeek] Tracking request error', {
          error: err?.message || err,
          botName: detectedBot,
          url: request.url,
        });
      }
    }

    return fetch(request);
  },
};

Best for: WordPress.com, Wix, Framer, static sites, or when you can't modify server code. Works with nginx, Apache mod_proxy, Cloudflare Workers, Vercel Edge, or any reverse proxy.

✗NOT Recommended: Client-Side JavaScript

This approach misses 90%+ of AI bot traffic. Shown for educational purposes only:

javascript
<!-- ⚠️ CLIENT-SIDE DETECTION IS NOT RECOMMENDED ⚠️ -->
<!--
    This approach does NOT work for most AI bots because:
      1. AI bots typically don't execute JavaScript
  2. They only fetch the HTML content
  3. You'll miss 90%+ of actual AI bot traffic
  
Use server-side or proxy detection instead (see examples above)
  -->

    <script>
  // This will NOT detect most AI bots
      function checkForAIBot() {
    const userAgent = navigator.userAgent;
    // ... detection code ...
  }
      window.addEventListener('load', checkForAIBot);
    </script>

Why it doesn't work: AI bots fetch your HTML but don't execute JavaScript, so client-side detection will never see them. Always use server-side or proxy detection for accurate tracking.

Complete API Endpoint Documentation

Endpoint URL

POST https://www.xseek.io/api/track-ai-bot

Request Headers

  • Content-Type: application/json - Required
  • x-api-key: YOUR_API_KEY - Required for authentication

Request Body (JSON)

{ "botName": "string", // Required: Name of the AI bot detected "userAgent": "string", // Required: Full user agent string "url": "string", // Required: URL being accessed "websiteId": "string", // Required: Your website ID from dashboard "ip": "string", // Optional: IP address of the bot "referer": "string" // Optional: Referer header value }

Response Formats

Success (200):

{ "success": true }

Error (400/401/500):

{ "error": "Error message", "details": "Additional error details" }

Status Codes

  • 200 - Successfully tracked bot visit
  • 400 - Invalid request body or missing required fields
  • 401 - Invalid or missing API key
  • 500 - Server error

Rate Limits

The API supports high-volume tracking with generous rate limits. Contact support if you need higher limits for enterprise deployments.

Need more help?

We can walk you through the setup or review your configuration.

Contact support