Skill: Web Search Plus

Owner: robbyczgw-cla

Summary: Unified search skill with Intelligent Auto-Routing. Uses multi-signal analysis to automatically select between Serper (Google), Tavily (Research), Exa (Neura...

Tags: latest:2.8.5

Version history:

v2.8.5 | 2026-02-20T15:58:22.076Z | auto

web-search-plus 2.8.5

• Updated package version to 2.8.5.
• Minor updates in CHANGELOG.md and scripts/search.py.
• No changes to SKILL.md; user-facing documentation is unchanged.

v2.8.4 | 2026-02-20T13:56:48.196Z | user

Security: SSRF protection added to setup.py SearXNG connection test

v2.8.3 | 2026-02-20T13:52:13.265Z | user

Fix: Perplexity provider returned 0 results — answer now mapped as primary result

v2.8.2 | 2026-02-20T13:28:17.364Z | user

Fix SKILL.md metadata inconsistencies: add KILOCODE_API_KEY to env declarations, update all provider references to include Perplexity

v2.8.1 | 2026-02-20T13:25:21.665Z | user

Detailed changelog + docs for Perplexity provider and routing rebalance

v2.8.0 | 2026-02-20T13:23:07.723Z | user

Add Perplexity provider via Kilo Gateway + rebalance routing (Tavily for research, Exa for discovery, Perplexity for direct answers)

v2.7.2 | 2026-02-15T07:32:28.504Z | user

Security: Implement actual SSRF protection - resolve hostnames and block private/internal IPs (loopback, RFC1918, link-local, cloud metadata). Add SEARXNG_ALLOW_PRIVATE=1 escape hatch for intentional self-hosted setups.

v2.7.1 | 2026-02-15T07:24:08.041Z | user

Security fix: Add SSRF protection for SearXNG instance URL validation. Enforce http/https scheme, prevent prompt injection from redirecting to internal endpoints.

v2.7.0 | 2026-02-14T14:52:12.877Z | auto

Version 2.7.0 (web-search-plus):

• Documentation fully updated across README, FAQ, SKILL, and TROUBLESHOOTING for clarity and completeness.
• Improved instructions and API key setup guides.
• Enhanced configuration examples and usage explanations.
• No changes to provider support or algorithm logic.
• Version updated in metadata and docs.

v2.6.5 | 2026-02-11T09:02:15.723Z | user

Fix: SKILL.md metadata env vars now correctly marked optional (was array format interpreted as required by registry)

v2.6.4 | 2026-02-11T08:46:01.334Z | user

cache fix

v2.6.3 | 2026-02-11T08:36:14.538Z | user

Added file-based result caching (1h TTL), --no-cache, --clear-cache, --cache-stats flags.

v2.6.2 | 2026-02-05T09:30:14.248Z | user

Improve SKILL.md user-friendliness for ClawHub (add FAQ section, clearer intro)

v2.6.1 | 2026-02-04T23:03:31.408Z | auto

• Bumped version to 2.6.1.
• Updated documentation files for accuracy and clarification.
• No major code or feature changes in this release.

v2.6.0 | 2026-02-04T12:19:43.253Z | auto

• Added FAQ.md and TROUBLESHOOTING.md for improved user support and documentation.
• SKILL.md fully rewritten for clarity, concise setup, and easier navigation.
• Expanded and reorganized usage instructions, configuration details, and provider comparison tables.
• Now highlights additional documentation and troubleshooting resources for new and existing users.

v2.5.2 | 2026-02-03T17:17:49.757Z | user

Add ClawHub runtime requirements metadata (SKILL.md frontmatter)

v2.5.1 | 2026-02-03T16:58:10.723Z | user

Add runtime requirements metadata for ClawHub discovery

v2.5.0 | 2026-02-03T15:32:22.706Z | user

NEW: SearXNG provider - Privacy-first meta-search! 🔒 70+ upstream engines, $0 API cost, self-hosted. Perfect for privacy-conscious users and multi-source aggregation. Now with 5 intelligent providers: Serper (Google), Tavily (Research), Exa (Neural), You.com (RAG), SearXNG (Privacy). Comprehensive FAQ added with 15 questions. Auto-routing for privacy/multi-source queries. Tested all provider combinations (0-5). Designed for self-hosters - bring your own instance!

v2.4.4 | 2026-02-03T15:11:26.320Z | user

Documentation fix: Corrected provider count from 'all 3' to 'all 4' in setup wizard description (we have 4 providers: Serper, Tavily, Exa, You.com).

v2.4.3 | 2026-02-03T14:44:05.657Z | user

Documentation update: Added 'NEW in v2.4.2' badge for You.com in SKILL.md to properly highlight the new provider on ClawHub.

v2.4.2 | 2026-02-03T14:40:35.558Z | user

Critical fix: Corrected You.com API hostname (ydc-index.io) and header name (X-API-KEY uppercase). You.com provider now works correctly - fully tested and verified with perfect results!

v2.4.1 | 2026-02-03T14:36:58.532Z | user

Bugfix: Fixed URL encoding for You.com queries - spaces and special characters now work correctly. Queries like 'OpenClaw AI framework' no longer cause 403 errors.

v2.4.0 | 2026-02-03T14:32:29.725Z | user

Add You.com as 4th search provider. Features: Auto-routing for RAG/real-time queries, LLM-ready snippets, unified web/news results, live crawl option for full page content. Perfect for AI-assisted research and real-time information needs.

v2.3.0 | 2026-01-31T23:02:52.885Z | auto

Highlights: Adds interactive setup wizard for first-time users.

• Introduced an interactive setup script (scripts/setup.py) for guided provider and API key configuration.
• Updated documentation to highlight the setup wizard and improved onboarding experience.
• Existing manual configuration via .env or config.json still supported.
• Minor metadata and documentation improvements.

v2.2.6 | 2026-01-31T22:55:13.239Z | auto

• Updated example city in usage and intent sections from "Vienna" to "Berlin" for consistency.
• No code changes or feature updates; documentation adjustments only.

v2.2.5 | 2026-01-31T08:55:23.621Z | user

Auto error fallback between providers

v2.2.4 | 2026-01-31T08:54:05.168Z | user

Automatic error fallback - when one provider fails (rate limit, timeout), automatically tries next provider in priority order (serper → tavily → exa)

v2.1.9 | 2026-01-29T18:40:00.157Z | user

Migrate from Clawdbot to Moltbot

v2.2.2 | 2026-01-28T20:51:59.487Z | user

Gitignore config.json to prevent API key leaks. Added config.example.json as safe template.

v2.2.1 | 2026-01-28T20:50:55.570Z | user

Support API keys in config.json - Priority: config.json > .env > environment

v2.2.0 | 2026-01-28T20:46:15.447Z | user

Auto-load .env file - no more 'source .env' needed! Keys load automatically from skill directory.

v2.1.8 | 2026-01-28T13:47:37.649Z | user

fix: correct free tier limits (Serper 2,500, Tavily 1,000), update version badges

v2.1.7 | 2026-01-28T13:46:11.702Z | user

fix: remove incorrect perplexity reference

v2.1.6 | 2026-01-28T12:12:55.279Z | user

fix: add missing confidence_level, User-Agent header, English defaults

v2.1.5 | 2026-01-27T07:58:42.652Z | user

docs: Add warning about NOT using Tavily/Serper/Exa in core clawdbot.json

v2.1.3 | 2026-01-25T11:17:51.767Z | user

v2.1.3: Expanded FAQ (API keys, troubleshooting, Clawdbot), more tags

v2.1.2 | 2026-01-25T06:24:49.025Z | user

SECURITY: Removed hardcoded API keys from test script

v2.1.1 | 2026-01-25T06:22:16.903Z | user

Fix: .env requires export prefix for Python to read environment variables

v2.0.0 | 2026-01-23T20:54:07.068Z | auto

Web Search Plus 2.0.0 – Smart Auto-Routing (major upgrade!)

• Added automatic query analysis to select the best search provider (Serper, Tavily, or Exa) with no user intervention required.
• Included explainable routing: use --explain-routing to see why a provider was chosen.
• Updated usage guides and documentation to reflect auto-routing as the new default and recommended workflow.
• Introduced tests for auto-routing logic (see test-auto-routing.sh).
• Minor config and script updates for improved clarity and maintainability.

v1.0.8 | 2026-01-20T22:35:48.740Z | user

Update author email to robbyczgw@gmail.com

v1.0.7 | 2026-01-20T22:35:05.769Z | user

Update author email

v1.0.6 | 2026-01-20T22:25:57.841Z | user

Add SKILL.md frontmatter for ClawdHub summary display

v1.0.5 | 2026-01-20T22:22:45.872Z | user

Add decision guide to README.md, update badges to v1.0.4 and add GitHub badge

v1.0.4 | 2026-01-20T22:19:24.367Z | user

Add clear decision guide: when to use built-in Brave search vs web-search-plus providers (Serper/Tavily/Exa)

v1.0.3 | 2026-01-20T22:14:12.343Z | user

Fix config.json example defaults to en/us for consistency

v1.0.2 | 2026-01-20T22:11:51.811Z | user

Add package.json with full description for ClawdHub display

v1.0.1 | 2026-01-20T22:11:12.153Z | user

Fix defaults to English (en/us) for international use, add package.json for proper ClawdHub metadata

v1.1.0 | 2026-01-20T22:10:17.360Z | auto

Web Search Plus v1.1.0

• Major documentation overhaul: SKILL.md rewritten to include agent-focused decision trees, detailed feature matrix, usage patterns, cost optimization, and advanced workflows.
• Added config.json for new configuration capabilities.
• Updated README.md and scripts/search.py for improved clarity and usability.
• Enhanced provider comparison and multi-provider workflow instructions.

v1.0.0 | 2026-01-20T22:05:37.164Z | auto

• Initial release of Web Search Plus: a multi-provider web search utility supporting Serper (Google Search API), Tavily (Research Search), and Exa (Neural/Semantic Search).
• Added guidance on choosing the best provider for various query types.
• Unified JSON output format across all providers for streamlined results handling.
• Default settings for provider options (such as language, result count, and type) included.
• Environment variable setup documented for easy API integration.

Archive index:

Archive v2.8.5: 11 files, 58666 bytes

Files: CHANGELOG.md (18449b), config.example.json (4972b), FAQ.md (9360b), package.json (1801b), README.md (23073b), scripts/search.py (89216b), scripts/setup.py (18486b), SKILL.md (9693b), test-auto-routing.sh (555b), TROUBLESHOOTING.md (6664b), _meta.json (134b)

File v2.8.5:SKILL.md

---

name: web-search-plus version: 2.8.1 description: Unified search skill with Intelligent Auto-Routing. Uses multi-signal analysis to automatically select between Serper (Google), Tavily (Research), Exa (Neural), Perplexity (AI Answers), You.com (RAG/Real-time), and SearXNG (Privacy/Self-hosted) with confidence scoring. tags: [search, web-search, serper, tavily, exa, perplexity, you, searxng, google, research, semantic-search, auto-routing, multi-provider, shopping, rag, free-tier, privacy, self-hosted, kilo] metadata: {"openclaw":{"requires":{"bins":["python3","bash"],"env":{"SERPER_API_KEY":"optional","TAVILY_API_KEY":"optional","EXA_API_KEY":"optional","YOU_API_KEY":"optional","SEARXNG_INSTANCE_URL":"optional","KILOCODE_API_KEY":"optional — required for Perplexity provider (via Kilo Gateway)"},"note":"Only ONE provider key needed. All are optional."}}}

Web Search Plus

Stop choosing search providers. Let the skill do it for you.

This skill connects you to 6 search providers (Serper, Tavily, Exa, Perplexity, You.com, SearXNG) and automatically picks the best one for each query. Shopping question? → Google results. Research question? → Deep research engine. Need a direct answer? → AI-synthesized with citations. Want privacy? → Self-hosted option.

---

✨ What Makes This Different?

• Just search — No need to think about which provider to use
• Smart routing — Analyzes your query and picks the best provider automatically
• 6 providers, 1 interface — Google results, research engines, neural search, AI answers with citations, RAG-optimized, and privacy-first all in one
• Works with just 1 key — Start with any single provider, add more later
• Free options available — SearXNG is completely free (self-hosted)

---

🚀 Quick Start

# Interactive setup (recommended for first run)
python3 scripts/setup.py

# Or manual: copy config and add your keys
cp config.example.json config.json

The wizard explains each provider, collects API keys, and configures defaults.

---

🔑 API Keys

You only need ONE key to get started. Add more providers later for better coverage.

| Provider | Free Tier | Best For | Sign Up | |----------|-----------|----------|---------| | Serper | 2,500/mo | Shopping, prices, local, news | serper.dev | | Tavily | 1,000/mo | Research, explanations, academic | tavily.com | | Exa | 1,000/mo | "Similar to X", startups, papers | exa.ai | | Perplexity | Via Kilo | Direct answers with citations | kilo.ai | | You.com | Limited | Real-time info, AI/RAG context | api.you.com | | SearXNG | FREE ✅ | Privacy, multi-source, $0 cost | Self-hosted |

Setting your keys:

# Option A: .env file (recommended)
export SERPER_API_KEY="your-key"
export TAVILY_API_KEY="your-key"

# Option B: config.json
{ "serper": { "api_key": "your-key" } }

---

🎯 When to Use Which Provider

| I want to... | Provider | Example Query | |--------------|----------|---------------| | Find product prices | Serper | "iPhone 16 Pro Max price" | | Find restaurants/stores nearby | Serper | "best pizza near me" | | Understand how something works | Tavily | "how does HTTPS encryption work" | | Do deep research | Tavily | "climate change research 2024" | | Find companies like X | Exa | "startups similar to Notion" | | Find research papers | Exa | "transformer architecture papers" | | Get a direct answer with sources | Perplexity | "events in Berlin this weekend" | | Know the current status of something | Perplexity | "what is the status of Ethereum upgrades" | | Get real-time info | You.com | "latest AI regulation news" | | Search without being tracked | SearXNG | anything, privately |

Pro tip: Just search normally! Auto-routing handles most queries correctly. Override with -p provider when needed.

---

🧠 How Auto-Routing Works

The skill looks at your query and picks the best provider:

"iPhone 16 price"              → Serper (shopping keywords)
"how does quantum computing work" → Tavily (research question)
"companies like stripe.com"    → Exa (URL detected, similarity)
"events in Graz this weekend"  → Perplexity (local + direct answer)
"latest news on AI"            → You.com (real-time intent)
"search privately"             → SearXNG (privacy keywords)

What if it picks wrong? Override it: python3 scripts/search.py -p tavily -q "your query"

Debug routing: python3 scripts/search.py --explain-routing -q "your query"

---

📖 Usage Examples

Let Auto-Routing Choose (Recommended)

python3 scripts/search.py -q "Tesla Model 3 price"
python3 scripts/search.py -q "explain machine learning"
python3 scripts/search.py -q "startups like Figma"

Force a Specific Provider

python3 scripts/search.py -p serper -q "weather Berlin"
python3 scripts/search.py -p tavily -q "quantum computing" --depth advanced
python3 scripts/search.py -p exa --similar-url "https://stripe.com" --category company
python3 scripts/search.py -p you -q "breaking tech news" --include-news
python3 scripts/search.py -p searxng -q "linux distros" --engines "google,bing"

---

⚙ Configuration

{
  "auto_routing": {
    "enabled": true,
    "fallback_provider": "serper",
    "confidence_threshold": 0.3,
    "disabled_providers": []
  },
  "serper": {"country": "us", "language": "en"},
  "tavily": {"depth": "advanced"},
  "exa": {"type": "neural"},
  "you": {"country": "US", "include_news": true},
  "searxng": {"instance_url": "https://your-instance.example.com"}
}

---

📊 Provider Comparison

| Feature | Serper | Tavily | Exa | Perplexity | You.com | SearXNG | |---------|:------:|:------:|:---:|:----------:|:-------:|:-------:| | Speed | ⚡⚡⚡ | ⚡⚡ | ⚡⚡ | ⚡⚡ | ⚡⚡⚡ | ⚡⚡ | | Direct Answers | ✗ | ✗ | ✗ | ✓✓ | ✗ | ✗ | | Citations | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | | Factual Accuracy | ⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ | | Semantic Understanding | ⭐ | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ | ⭐ | | Full Page Content | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | | Shopping/Local | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | | Find Similar Pages | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | | RAG-Optimized | ✗ | ✓ | ✗ | ✗ | ✓✓ | ✗ | | Privacy-First | ✗ | ✗ | ✗ | ✗ | ✗ | ✓✓ | | API Cost | $$ | $$ | $$ | Via Kilo | $ | FREE |

---

❓ Common Questions

Do I need API keys for all providers?

No. You only need keys for providers you want to use. Start with one (Serper recommended), add more later.

Which provider should I start with?

Serper — fastest, cheapest, largest free tier (2,500 queries/month), and handles most queries well.

What if I run out of free queries?

The skill automatically falls back to your other configured providers. Or switch to SearXNG (unlimited, self-hosted).

How much does this cost?

• Free tiers: 2,500 (Serper) + 1,000 (Tavily) + 1,000 (Exa) = 4,500+ free searches/month
• SearXNG: Completely free (just ~$5/mo if you self-host on a VPS)
• Paid plans: Start around $10-50/month depending on provider

Is SearXNG really private?

Yes, if self-hosted. You control the server, no tracking, no profiling. Public instances depend on the operator's policy.

How do I set up SearXNG?

# Docker (5 minutes)
docker run -d -p 8080:8080 searxng/searxng

Then enable JSON API in settings.yml. See docs.searxng.org.

Why did it route my query to the "wrong" provider?

Sometimes queries are ambiguous. Use --explain-routing to see why, then override with -p provider if needed.

---

🔄 Automatic Fallback

If one provider fails (rate limit, timeout, error), the skill automatically tries the next provider. You'll see routing.fallback_used: true in the response when this happens.

---

📤 Output Format

{
  "provider": "serper",
  "query": "iPhone 16 price",
  "results": [{"title": "...", "url": "...", "snippet": "...", "score": 0.95}],
  "routing": {
    "auto_routed": true,
    "provider": "serper",
    "confidence": 0.78,
    "confidence_level": "high"
  }
}

---

⚠ Important Note

Tavily, Serper, and Exa are NOT core OpenClaw providers.

❌ Don't modify ~/.openclaw/openclaw.json for these
✅ Use this skill's scripts — keys auto-load from .env

---

🔒 Security

SearXNG SSRF Protection: The SearXNG instance URL is validated with defense-in-depth:

• Enforces http/https schemes only
• Blocks cloud metadata endpoints (169.254.169.254, metadata.google.internal)
• Resolves hostnames and blocks private/internal IPs (loopback, RFC1918, link-local, reserved)
• Operators who intentionally self-host on private networks can set SEARXNG_ALLOW_PRIVATE=1

📚 More Documentation

• FAQ.md — Detailed answers to more questions
• TROUBLESHOOTING.md — Fix common errors
• README.md — Full technical reference

---

🔗 Quick Links

• Serper — Google Search API
• Tavily — AI Research Search
• Exa — Neural Search
• Perplexity — AI-Synthesized Answers (via Kilo Gateway)
• You.com — RAG/Real-time Search
• SearXNG — Privacy-First Meta-Search

File v2.8.5:README.md

Web Search Plus

Unified multi-provider web search with Intelligent Auto-Routing — uses multi-signal analysis to automatically select between Serper, Tavily, Exa, You.com, and SearXNG with confidence scoring.

---

🧠 Features (v2.7.0)

Intelligent Multi-Signal Routing — The skill uses sophisticated query analysis:

• Intent Classification: Shopping vs Research vs Discovery vs RAG/Real-time vs Privacy
• Linguistic Patterns: "how much" (price) vs "how does" (research) vs "privately" (privacy)
• Entity Detection: Product+brand combos, URLs, domains
• Complexity Analysis: Long queries favor research providers
• Confidence Scoring: Know how reliable the routing decision is

python3 scripts/search.py -q "how much does iPhone 16 cost"     # → Serper (68% confidence)
python3 scripts/search.py -q "how does quantum entanglement work"  # → Tavily (86% HIGH)
python3 scripts/search.py -q "startups similar to Notion"       # → Exa (76% HIGH)
python3 scripts/search.py -q "companies like stripe.com"        # → Exa (100% HIGH - URL detected)
python3 scripts/search.py -q "summarize key points on AI"       # → You.com (68% MEDIUM - RAG intent)
python3 scripts/search.py -q "search privately without tracking" # → SearXNG (74% HIGH - privacy intent)

---

🔍 When to Use Which Provider

Built-in Brave Search (OpenClaw default)

• ✅ General web searches
• ✅ Privacy-focused
• ✅ Quick lookups
• ✅ Default fallback

Serper (Google Results)

• 🛍 Product specs, prices, shopping
• 📍 Local businesses, places
• 🎯 "Google it" - explicit Google results
• 📰 Shopping/images needed
• 🏆 Knowledge Graph data

Tavily (AI-Optimized Research)

• 📚 Research questions, deep dives
• 🔬 Complex multi-part queries
• 📄 Need full page content (not just snippets)
• 🎓 Academic/technical research
• 🔒 Domain filtering (trusted sources)

Exa (Neural Semantic Search)

• 🔗 Find similar pages
• 🏢 Company/startup discovery
• 📝 Research papers
• 💻 GitHub projects
• 📅 Date-specific content

You.com (RAG/Real-time)

• 🤖 RAG applications (LLM-ready snippets)
• 📰 Combined web + news (single API call)
• ⚡ Real-time information (current events)
• 📋 Summarization context ("What's the latest...")
• 🔄 Live crawling (full page content on demand)

SearXNG (Privacy-First/Self-Hosted)

• 🔒 Privacy-preserving search (no tracking)
• 🌐 Multi-source aggregation (70+ engines)
• 💰 $0 API cost (self-hosted)
• 🎯 Diverse perspectives (results from multiple engines)
• 🏠 Self-hosted environments (full control)

---

Table of Contents

• Quick Start
• Smart Auto-Routing
• Configuration Guide
• Provider Deep Dives
• Usage Examples
• Workflow Examples
• Optimization Tips
• FAQ & Troubleshooting
• API Reference

---

Quick Start

Option A: Interactive Setup (Recommended)

# Run the setup wizard - it guides you through everything
python3 scripts/setup.py

The wizard explains each provider, collects your API keys, and creates config.json automatically.

Option B: Manual Setup

# 1. Set up at least one API key (or SearXNG instance)
export SERPER_API_KEY="your-key"   # https://serper.dev
export TAVILY_API_KEY="your-key"   # https://tavily.com
export EXA_API_KEY="your-key"      # https://exa.ai
export YOU_API_KEY="your-key"      # https://api.you.com
export SEARXNG_INSTANCE_URL="https://your-instance.example.com"  # Self-hosted

# 2. Run a search (auto-routed!)
python3 scripts/search.py -q "best laptop 2024"

Run a Search

# Auto-routed to best provider
python3 scripts/search.py -q "best laptop 2024"

# Or specify a provider explicitly
python3 scripts/search.py -p serper -q "iPhone 16 specs"
python3 scripts/search.py -p tavily -q "quantum computing explained" --depth advanced
python3 scripts/search.py -p exa -q "AI startups 2024" --category company

---

Smart Auto-Routing

How It Works

When you don't specify a provider, the skill analyzes your query and routes it to the best provider:

| Query Contains | Routes To | Example | |---------------|-----------|---------| | "price", "buy", "shop", "cost" | Serper | "iPhone 16 price" | | "near me", "restaurant", "hotel" | Serper | "pizza near me" | | "weather", "news", "latest" | Serper | "weather Berlin" | | "how does", "explain", "what is" | Tavily | "how does TCP work" | | "research", "study", "analyze" | Tavily | "climate research" | | "tutorial", "guide", "learn" | Tavily | "python tutorial" | | "similar to", "companies like" | Exa | "companies like Stripe" | | "startup", "Series A" | Exa | "AI startups Series A" | | "github", "research paper" | Exa | "LLM papers arxiv" | | "private", "anonymous", "no tracking" | SearXNG | "search privately" | | "multiple sources", "aggregate" | SearXNG | "results from all engines" |

Examples

# These are all auto-routed to the optimal provider:
python3 scripts/search.py -q "MacBook Pro M3 price"           # → Serper
python3 scripts/search.py -q "how does HTTPS work"            # → Tavily
python3 scripts/search.py -q "startups like Notion"           # → Exa
python3 scripts/search.py -q "best sushi restaurant near me"  # → Serper
python3 scripts/search.py -q "explain attention mechanism"    # → Tavily
python3 scripts/search.py -q "alternatives to Figma"          # → Exa
python3 scripts/search.py -q "search privately without tracking" # → SearXNG

Result Caching (NEW in v2.7.0!)

Search results are automatically cached for 1 hour to save API costs:

# First request: fetches from API ($)
python3 scripts/search.py -q "AI startups 2024"

# Second request: uses cache (FREE!)
python3 scripts/search.py -q "AI startups 2024"
# Output includes: "cached": true

# Bypass cache (force fresh results)
python3 scripts/search.py -q "AI startups 2024" --no-cache

# View cache stats
python3 scripts/search.py --cache-stats

# Clear all cached results
python3 scripts/search.py --clear-cache

# Custom TTL (in seconds, default: 3600 = 1 hour)
python3 scripts/search.py -q "query" --cache-ttl 7200

Cache location: .cache/ in skill directory (override with WSP_CACHE_DIR environment variable)

Debug Auto-Routing

See exactly why a provider was selected:

python3 scripts/search.py --explain-routing -q "best laptop to buy"

Output:

{
  "query": "best laptop to buy",
  "selected_provider": "serper",
  "reason": "matched_keywords (score=2)",
  "matched_keywords": ["buy", "best"],
  "available_providers": ["serper", "tavily", "exa"]
}

Routing Info in Results

Every search result includes routing information:

{
  "provider": "serper",
  "query": "iPhone 16 price",
  "results": [...],
  "routing": {
    "auto_routed": true,
    "selected_provider": "serper",
    "reason": "matched_keywords (score=1)",
    "matched_keywords": ["price"]
  }
}

---

Configuration Guide

Environment Variables

Create a .env file or set these in your shell:

# Required: Set at least one
export SERPER_API_KEY="your-serper-key"
export TAVILY_API_KEY="your-tavily-key"
export EXA_API_KEY="your-exa-key"

Config File (config.json)

The config.json file lets you customize auto-routing and provider defaults:

{
  "defaults": {
    "provider": "serper",
    "max_results": 5
  },
  
  "auto_routing": {
    "enabled": true,
    "fallback_provider": "serper",
    "provider_priority": ["serper", "tavily", "exa"],
    "disabled_providers": [],
    "keyword_mappings": {
      "serper": ["price", "buy", "shop", "cost", "deal", "near me", "weather"],
      "tavily": ["how does", "explain", "research", "what is", "tutorial"],
      "exa": ["similar to", "companies like", "alternatives", "startup", "github"]
    }
  },
  
  "serper": {
    "country": "us",
    "language": "en"
  },
  
  "tavily": {
    "depth": "basic",
    "topic": "general"
  },
  
  "exa": {
    "type": "neural"
  }
}

Configuration Examples

Example 1: Disable Exa (Only Use Serper + Tavily)

{
  "auto_routing": {
    "disabled_providers": ["exa"]
  }
}

Example 2: Make Tavily the Default

{
  "auto_routing": {
    "fallback_provider": "tavily"
  }
}

Example 3: Add Custom Keywords

{
  "auto_routing": {
    "keyword_mappings": {
      "serper": [
        "price", "buy", "shop", "amazon", "ebay", "walmart",
        "deal", "discount", "coupon", "sale", "cheap"
      ],
      "tavily": [
        "how does", "explain", "research", "what is",
        "coursera", "udemy", "learn", "course", "certification"
      ],
      "exa": [
        "similar to", "companies like", "competitors",
        "YC company", "funded startup", "Series A", "Series B"
      ]
    }
  }
}

Example 4: German Locale for Serper

{
  "serper": {
    "country": "de",
    "language": "de"
  }
}

Example 5: Disable Auto-Routing

{
  "auto_routing": {
    "enabled": false
  },
  "defaults": {
    "provider": "serper"
  }
}

Example 6: Research-Heavy Config

{
  "auto_routing": {
    "fallback_provider": "tavily",
    "provider_priority": ["tavily", "serper", "exa"]
  },
  "tavily": {
    "depth": "advanced",
    "include_raw_content": true
  }
}

---

Provider Deep Dives

Serper (Google Search API)

What it is: Direct access to Google Search results via API — the same results you'd see on google.com.

Strengths

| Strength | Description | |----------|-------------| | 🎯 Accuracy | Google's search quality, knowledge graph, featured snippets | | 🛒 Shopping | Product prices, reviews, shopping results | | 📍 Local | Business listings, maps, places | | 📰 News | Real-time news with Google News integration | | 🖼 Images | Google Images search | | ⚡ Speed | Fastest response times (~200-400ms) |

Best Use Cases

• ✅ Product specifications and comparisons
• ✅ Shopping and price lookups
• ✅ Local business searches ("restaurants near me")
• ✅ Quick factual queries (weather, conversions, definitions)
• ✅ News headlines and current events
• ✅ Image searches
• ✅ When you need "what Google shows"

Getting Your API Key

1. Go to serper.dev
2. Sign up with email or Google
3. Copy your API key from the dashboard
4. Set SERPER_API_KEY environment variable

---

Tavily (Research Search)

What it is: AI-optimized search engine built for research and RAG applications — returns synthesized answers plus full content.

Strengths

| Strength | Description | |----------|-------------| | 📚 Research Quality | Optimized for comprehensive, accurate research | | 💬 AI Answers | Returns synthesized answers, not just links | | 📄 Full Content | Can return complete page content (raw_content) | | 🎯 Domain Filtering | Include/exclude specific domains | | 🔬 Deep Mode | Advanced search for thorough research | | 📰 Topic Modes | Specialized for general vs news content |

Best Use Cases

• ✅ Research questions requiring synthesized answers
• ✅ Academic or technical deep dives
• ✅ When you need actual page content (not just snippets)
• ✅ Multi-source information comparison
• ✅ Domain-specific research (filter to authoritative sources)
• ✅ News research with context
• ✅ RAG/LLM applications

Getting Your API Key

1. Go to tavily.com
2. Sign up and verify email
3. Navigate to API Keys section
4. Generate and copy your key
5. Set TAVILY_API_KEY environment variable

---

Exa (Neural Search)

What it is: Neural/semantic search engine that understands meaning, not just keywords — finds conceptually similar content.

Strengths

| Strength | Description | |----------|-------------| | 🧠 Semantic Understanding | Finds results by meaning, not keywords | | 🔗 Similar Pages | Find pages similar to a reference URL | | 🏢 Company Discovery | Excellent for finding startups, companies | | 📑 Category Filters | Filter by type (company, paper, tweet, etc.) | | 📅 Date Filtering | Precise date range searches | | 🎓 Academic | Great for research papers and technical content |

Best Use Cases

• ✅ Conceptual queries ("companies building X")
• ✅ Finding similar companies or pages
• ✅ Startup and company discovery
• ✅ Research paper discovery
• ✅ Finding GitHub projects
• ✅ Date-filtered searches for recent content
• ✅ When keyword matching fails

Getting Your API Key

1. Go to exa.ai
2. Sign up with email or Google
3. Navigate to API section in dashboard
4. Copy your API key
5. Set EXA_API_KEY environment variable

---

SearXNG (Privacy-First Meta-Search)

What it is: Open-source, self-hosted meta-search engine that aggregates results from 70+ search engines without tracking.

Strengths

| Strength | Description | |----------|-------------| | 🔒 Privacy-First | No tracking, no profiling, no data collection | | 🌐 Multi-Engine | Aggregates Google, Bing, DuckDuckGo, and 70+ more | | 💰 Free | $0 API cost (self-hosted, unlimited queries) | | 🎯 Diverse Results | Get perspectives from multiple search engines | | ⚙ Customizable | Choose which engines to use, SafeSearch, language | | 🏠 Self-Hosted | Full control over your search infrastructure |

Best Use Cases

• ✅ Privacy-sensitive searches (no tracking)
• ✅ When you want diverse results from multiple engines
• ✅ Budget-conscious (no API fees)
• ✅ Self-hosted/air-gapped environments
• ✅ Fallback when paid APIs are rate-limited
• ✅ When "aggregate everything" is the goal

Setting Up Your Instance

# Docker (recommended, 5 minutes)
docker run -d -p 8080:8080 searxng/searxng

# Enable JSON API in settings.yml:
# search:
#   formats: [html, json]

1. See docs.searxng.org
2. Deploy via Docker, pip, or your preferred method
3. Enable JSON format in settings.yml
4. Set SEARXNG_INSTANCE_URL environment variable

---

Usage Examples

Auto-Routed Searches (Recommended)

# Just search — the skill picks the best provider
python3 scripts/search.py -q "Tesla Model 3 price"
python3 scripts/search.py -q "how do neural networks learn"
python3 scripts/search.py -q "YC startups like Stripe"
python3 scripts/search.py -q "search privately without tracking"

Serper Options

# Different search types
python3 scripts/search.py -p serper -q "gaming monitor" --type shopping
python3 scripts/search.py -p serper -q "coffee shop" --type places
python3 scripts/search.py -p serper -q "AI news" --type news

# With time filter
python3 scripts/search.py -p serper -q "OpenAI news" --time-range day

# Include images
python3 scripts/search.py -p serper -q "iPhone 16 Pro" --images

# Different locale
python3 scripts/search.py -p serper -q "Wetter Wien" --country at --language de

Tavily Options

# Deep research mode
python3 scripts/search.py -p tavily -q "quantum computing applications" --depth advanced

# With full page content
python3 scripts/search.py -p tavily -q "transformer architecture" --raw-content

# Domain filtering
python3 scripts/search.py -p tavily -q "AI research" --include-domains arxiv.org nature.com

Exa Options

# Category filtering
python3 scripts/search.py -p exa -q "AI startups Series A" --category company
python3 scripts/search.py -p exa -q "attention mechanism" --category "research paper"

# Date filtering
python3 scripts/search.py -p exa -q "YC companies" --start-date 2024-01-01

# Find similar pages
python3 scripts/search.py -p exa --similar-url "https://stripe.com" --category company

SearXNG Options

# Basic search
python3 scripts/search.py -p searxng -q "linux distros"

# Specific engines only
python3 scripts/search.py -p searxng -q "AI news" --engines "google,bing,duckduckgo"

# SafeSearch (0=off, 1=moderate, 2=strict)
python3 scripts/search.py -p searxng -q "privacy tools" --searxng-safesearch 2

# With time filter
python3 scripts/search.py -p searxng -q "open source projects" --time-range week

# Custom instance URL
python3 scripts/search.py -p searxng -q "test" --searxng-url "http://localhost:8080"

---

Workflow Examples

🛒 Product Research Workflow

# Step 1: Get product specs (auto-routed to Serper)
python3 scripts/search.py -q "MacBook Pro M3 Max specs"

# Step 2: Check prices (auto-routed to Serper)
python3 scripts/search.py -q "MacBook Pro M3 Max price comparison"

# Step 3: In-depth reviews (auto-routed to Tavily)
python3 scripts/search.py -q "detailed MacBook Pro M3 Max review"

📚 Academic Research Workflow

# Step 1: Understand the topic (auto-routed to Tavily)
python3 scripts/search.py -q "explain transformer architecture in deep learning"

# Step 2: Find recent papers (Exa)
python3 scripts/search.py -p exa -q "transformer improvements" --category "research paper" --start-date 2024-01-01

# Step 3: Find implementations (Exa)
python3 scripts/search.py -p exa -q "transformer implementation" --category github

🏢 Competitive Analysis Workflow

# Step 1: Find competitors (auto-routed to Exa)
python3 scripts/search.py -q "companies like Notion"

# Step 2: Find similar products (Exa)
python3 scripts/search.py -p exa --similar-url "https://notion.so" --category company

# Step 3: Deep dive comparison (Tavily)
python3 scripts/search.py -p tavily -q "Notion vs Coda comparison" --depth advanced

---

Optimization Tips

Cost Optimization

| Tip | Savings | |-----|---------| | Use SearXNG for routine queries | $0 API cost | | Use auto-routing (defaults to Serper, cheapest paid) | Best value | | Use Tavily basic before advanced | ~50% cost reduction | | Set appropriate max_results | Linear cost savings | | Use Exa only for semantic queries | Avoid waste |

Performance Optimization

| Tip | Impact | |-----|--------| | Serper is fastest (~200ms) | Use for time-sensitive queries | | Tavily basic faster than advanced | ~2x faster | | Lower max_results = faster response | Linear improvement |

---

FAQ & Troubleshooting

General Questions

Q: Do I need API keys for all three providers?

No. You only need keys for providers you want to use. Auto-routing skips providers without keys.

Q: Which provider should I start with?

Serper — it's the fastest, cheapest, and has the largest free tier (2,500 queries).

Q: Can I use multiple providers in one workflow?

Yes! That's the recommended approach. See Workflow Examples.

Q: How do I reduce API costs?

Use auto-routing (defaults to cheapest), start with lower max_results, use Tavily basic before advanced.

Auto-Routing Questions

Q: Why did my query go to the wrong provider?

Use --explain-routing to debug. Add custom keywords to config.json if needed.

Q: Can I add my own keywords?

Yes! Edit config.json → auto_routing.keyword_mappings.

Q: How does keyword scoring work?

Multi-word phrases get higher weights. "companies like" (2 words) scores higher than "like" (1 word).

Q: What if no keywords match?

Uses the fallback provider (default: Serper).

Q: Can I force a specific provider?

Yes, use -p serper, -p tavily, or -p exa.

Troubleshooting

Error: "Missing API key"

# Check if key is set
echo $SERPER_API_KEY

# Set it
export SERPER_API_KEY="your-key"

Error: "API Error (401)"

Your API key is invalid or expired. Generate a new one.

Error: "API Error (429)"

Rate limited. Wait and retry, or upgrade your plan.

Empty results?

Try a different provider, broaden your query, or remove restrictive filters.

Slow responses?

Reduce max_results, use Tavily basic, or use Serper (fastest).

---

API Reference

Output Format

All providers return unified JSON:

{
  "provider": "serper|tavily|exa",
  "query": "original search query",
  "results": [
    {
      "title": "Page Title",
      "url": "https://example.com/page",
      "snippet": "Content excerpt...",
      "score": 0.95,
      "date": "2024-01-15",
      "raw_content": "Full page content (Tavily only)"
    }
  ],
  "images": ["url1", "url2"],
  "answer": "Synthesized answer",
  "knowledge_graph": { },
  "routing": {
    "auto_routed": true,
    "selected_provider": "serper",
    "reason": "matched_keywords (score=1)",
    "matched_keywords": ["price"]
  }
}

CLI Options Reference

| Option | Providers | Description | |--------|-----------|-------------| | -q, --query | All | Search query | | -p, --provider | All | Provider: auto, serper, tavily, exa, you, searxng | | -n, --max-results | All | Max results (default: 5) | | --auto | All | Force auto-routing | | --explain-routing | All | Debug auto-routing | | --images | Serper, Tavily | Include images | | --country | Serper, You | Country code (default: us) | | --language | Serper, SearXNG | Language code (default: en) | | --type | Serper | search/news/images/videos/places/shopping | | --time-range | Serper, SearXNG | hour/day/week/month/year | | --depth | Tavily | basic/advanced | | --topic | Tavily | general/news | | --raw-content | Tavily | Include full page content | | --exa-type | Exa | neural/keyword | | --category | Exa | company/research paper/news/pdf/github/tweet | | --start-date | Exa | Start date (YYYY-MM-DD) | | --end-date | Exa | End date (YYYY-MM-DD) | | --similar-url | Exa | Find similar pages | | --searxng-url | SearXNG | Instance URL | | --searxng-safesearch | SearXNG | 0=off, 1=moderate, 2=strict | | --engines | SearXNG | Specific engines (google,bing,duckduckgo) | | --categories | SearXNG | Search categories (general,images,news) | | --include-domains | Tavily, Exa | Only these domains | | --exclude-domains | Tavily, Exa | Exclude these domains | | --compact | All | Compact JSON output |

---

License

MIT

---

Links

• Serper — Google Search API
• Tavily — AI Research Search
• Exa — Neural Search
• ClawHub — OpenClaw Skills

File v2.8.5:_meta.json

{ "ownerId": "kn73gpe8xz2630jrknkb3ya96h7zb84h", "slug": "web-search-plus", "version": "2.8.5", "publishedAt": 1771603102076 }

File v2.8.5:CHANGELOG.md

Changelog - Web Search Plus

[2.8.5] - 2026-02-20

✨ Feature: Perplexity freshness filter

• Added freshness parameter to Perplexity provider (day, week, month, year)
• Maps to Perplexity's native search_recency_filter parameter
• Example: python3 scripts/search.py -p perplexity -q "latest AI news" --freshness day
• Consistent with freshness support in Serper and Brave providers

[2.8.4] - 2026-02-20

🔒 Security Fix: SSRF protection in setup wizard

• Fixed: setup.py SearXNG connection test had no SSRF protection (unlike search.py)
• Before: Operator could be tricked into probing internal networks during setup
• After: Same IP validation as search.py — blocks private IPs, cloud metadata, loopback
• Credit: ClawHub security scanner

[2.8.3] - 2026-02-20

🐛 Critical Fix: Perplexity results empty

• Fixed: Perplexity provider returned 0 results because the AI-synthesized answer wasn't mapped into the results array
• Before: Only extracted URLs from the answer text were returned as results (often 0)
• After: The full answer is now the primary result (title, snippet with cleaned text), extracted source URLs follow as additional results
• Impact: Perplexity queries now always return at least 1 result with the synthesized answer

[2.8.0] - 2026-02-20

🆕 New Provider: Perplexity (AI-Synthesized Answers)

Added Perplexity as the 6th search provider via Kilo Gateway — the first provider that returns direct answers with citations instead of just links:

Features

• AI-Synthesized Answers: Get a complete answer, not a list of links
• Inline Citations: Every claim backed by [1][2][3] source references
• Real-Time Web Search: Perplexity searches the web live, reads pages, and summarizes
• Zero Extra Config: Works through Kilo Gateway with your existing KILOCODE_API_KEY
• Model: perplexity/sonar-pro (best quality, supports complex queries)

Auto-Routing Signals

New direct-answer intent detection routes to Perplexity for:

• Status queries: "status of", "current state of", "what is the status"
• Local info: "events in [city]", "things to do in", "what's happening in"
• Direct questions: "what is", "who is", "when did", "how many"
• Current affairs: "this week", "this weekend", "right now", "today"

Usage Examples

# Auto-routed
python3 scripts/search.py -q "events in Graz Austria this weekend"  # → Perplexity
python3 scripts/search.py -q "what is the current status of Ethereum"  # → Perplexity

# Explicit
python3 scripts/search.py -p perplexity -q "latest AI regulation news"

Configuration

Requires KILOCODE_API_KEY environment variable (Kilo Gateway account). No additional API key needed — Perplexity is accessed through Kilo's unified API.

export KILOCODE_API_KEY="your-kilo-key"

🔧 Routing Rebalance

Major overhaul of the auto-routing confidence scoring to fix Serper dominance:

Problem

Serper (Google) was winning ~90% of queries due to:

• High recency multiplier boosting Serper on any query with dates/years
• Default provider priority placing Serper first in ties
• Research and discovery signals not strong enough to override

Changes

• Lowered Serper recency multiplier — date mentions no longer auto-route to Google
• Strengthened research signals for Tavily:
  • Added: "status of", "what happened with", "how does X compare"
  • Boosted weights for comparison patterns (4.0 → 5.0)
• Strengthened discovery signals for Exa:
  • Added: "events in", "things to do in", "startups similar to"
  • Boosted weights for local discovery patterns
• Updated provider priority order: tavily → exa → perplexity → serper → you → searxng
  • Serper moved from 1st to 4th in tie-breaking
  • Research/discovery providers now win on ambiguous queries

Routing Test Results

| Query | Before | After | ✓ | |-------|--------|-------|---| | "latest OpenClaw version Feb 2026" | Serper | Serper | ✅ | | "Ethereum Pectra upgrade status" | Serper | Tavily | ✅ | | "events in Graz this weekend" | Serper | Perplexity | ✅ | | "compare SearXNG vs Brave for AI agents" | Serper | Tavily | ✅ | | "Sam Altman OpenAI news this week" | Serper | Serper | ✅ | | "find startups similar to Kilo Code" | Serper | Exa | ✅ |

📊 Updated Provider Comparison

| Feature | Serper | Tavily | Exa | Perplexity | You.com | SearXNG | |---------|:------:|:------:|:---:|:----------:|:-------:|:-------:| | Speed | ⚡⚡⚡ | ⚡⚡ | ⚡⚡ | ⚡⚡ | ⚡⚡⚡ | ⚡ | | Direct Answers | ✗ | ✗ | ✗ | ✓✓ | ✗ | ✗ | | Citations | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | | Local Events | ✓ | ✗ | ✓ | ✓✓ | ✗ | ✓ | | Research | ✗ | ✓✓ | ✓ | ✓ | ✓ | ✗ | | Discovery | ✗ | ✗ | ✓✓ | ✗ | ✗ | ✗ | | Self-Hosted | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |

[2.7.0] - 2026-02-14

✨ Added

• Provider cooldown tracking in .cache/provider_health.json
• Exponential cooldown on provider failures: 1m → 5m → 25m → 1h (cap)
• Retry strategy for transient failures (timeout, 429, 503): up to 2 retries with backoff 1s → 3s → 9s
• Smarter cache keys hashed from full request context (query/provider/max_results + locale, freshness, time_range, topic, search_engines, include_news, and related params)
• Cross-provider result deduplication by normalized URL during fallback merge

🔧 Changed

• Cooldown providers are skipped in routing while their cooldown is active
• Provider health is reset automatically after successful requests
• Fallback output now includes dedup metadata:
  • deduplicated: true|false
  • metadata.dedup_count

[2.6.5] - 2026-02-11

🆕 File-Based Result Caching

Added local caching to save API costs on repeated searches:

Features

• Automatic Caching: Search results cached locally by default
• 1-Hour TTL: Results expire after 3600 seconds (configurable)
• Cache Indicators: Response includes cached: true/false and cache_age_seconds
• Zero-Cost Repeats: Cached requests don't hit APIs

New CLI Options

• --cache-ttl SECONDS — Custom cache TTL (default: 3600)
• --no-cache — Bypass cache, always fetch fresh
• --clear-cache — Delete all cached results
• --cache-stats — Show cache statistics (entries, size, age)

Configuration

• Cache directory: .cache/ in skill directory
• Environment variable: WSP_CACHE_DIR to override location
• Cache key: Based on query + provider + max_results (SHA256)

Usage Examples

# First request costs API credits
python3 scripts/search.py -q "AI startups"

# Second request is FREE (uses cache)
python3 scripts/search.py -q "AI startups"

# Force fresh results
python3 scripts/search.py -q "AI startups" --no-cache

# View stats
python3 scripts/search.py --cache-stats

# Clear everything
python3 scripts/search.py --clear-cache

Technical Details

• Cache files: JSON with metadata (_cache_timestamp, _cache_key, etc.)
• Automatic cleanup of expired entries on access
• Graceful handling of corrupted cache files

[2.6.1] - 2026-02-04

• Privacy cleanup: removed hardcoded paths and personal info from docs

[2.5.0] - 2026-02-03

🆕 New Provider: SearXNG (Privacy-First Meta-Search)

Added SearXNG as the 5th search provider, focused on privacy and self-hosted search:

Features

• Privacy-Preserving: No tracking, no profiling — your searches stay private
• Multi-Source Aggregation: Queries 70+ upstream engines (Google, Bing, DuckDuckGo, etc.)
• $0 API Cost: Self-hosted = unlimited queries with no API fees
• Diverse Results: Get perspectives from multiple search engines in one query
• Customizable: Choose which engines to use, set SafeSearch levels, language preferences

Auto-Routing Signals

New privacy/multi-source intent detection routes to SearXNG for:

• Privacy queries: "private", "anonymous", "without tracking", "no tracking"
• Multi-source: "aggregate results", "multiple sources", "diverse perspectives"
• Budget/free: "free search", "no api cost", "self-hosted search"
• German: "privat", "anonym", "ohne tracking", "verschiedene quellen"

Usage Examples

# Auto-routed
python3 scripts/search.py -q "search privately without tracking"  # → SearXNG

# Explicit
python3 scripts/search.py -p searxng -q "linux distros"
python3 scripts/search.py -p searxng -q "AI news" --engines "google,bing,duckduckgo"
python3 scripts/search.py -p searxng -q "privacy tools" --searxng-safesearch 2

Configuration

{
  "searxng": {
    "instance_url": "https://your-instance.example.com",
    "safesearch": 0,
    "engines": null,
    "language": "en"
  }
}

Setup

SearXNG requires a self-hosted instance with JSON format enabled:

# Docker setup (5 minutes)
docker run -d -p 8080:8080 searxng/searxng

# Enable JSON in settings.yml:
# search:
#   formats: [html, json]

# Set instance URL
export SEARXNG_INSTANCE_URL="http://localhost:8080"

See: https://docs.searxng.org/admin/installation.html

📊 Updated Provider Comparison

| Feature | Serper | Tavily | Exa | You.com | SearXNG | |---------|:------:|:------:|:---:|:-------:|:-------:| | Privacy-First | ✗ | ✗ | ✗ | ✗ | ✓✓ | | Self-Hosted | ✗ | ✗ | ✗ | ✗ | ✓ | | API Cost | $$ | $$ | $$ | $ | FREE | | Multi-Engine | ✗ | ✗ | ✗ | ✗ | ✓ (70+) |

🔧 Technical Changes

• Added search_searxng() function with full error handling
• Added PRIVACY_SIGNALS to QueryAnalyzer for auto-routing
• Updated setup wizard with SearXNG option (instance URL validation)
• Updated config.example.json with searxng section
• New CLI args: --searxng-url, --searxng-safesearch, --engines, --categories

---

[2.4.4] - 2026-02-03

📝 Documentation: Provider Count Fix

• Fixed: "You can use 1, 2, or all 3" → "1, 2, 3, or all 4" (we have 4 providers now!)
• Impact: Accurate documentation for setup wizard

[2.4.3] - 2026-02-03

📝 Documentation: Updated README

• Added: "NEW in v2.4.2" badge for You.com in SKILL.md
• Impact: ClawHub README now properly highlights You.com as new feature

[2.4.2] - 2026-02-03

🐛 Critical Fix: You.com API Configuration

• Fixed: Incorrect hostname (api.ydc-index.io → ydc-index.io)
• Fixed: Incorrect header name (X-API-Key → X-API-KEY uppercase)
• Impact: You.com now works correctly - was giving 403 Forbidden before
• Status: ✅ Fully tested and working

[2.4.1] - 2026-02-03

🐛 Bugfix: You.com URL Encoding

• Fixed: URL encoding for You.com queries - spaces and special characters now properly encoded
• Impact: Queries with spaces (e.g., "OpenClaw AI framework") work correctly now
• Technical: Added urllib.parse.quote for parameter encoding

[2.4.0] - 2026-02-03

🆕 New Provider: You.com

Added You.com as the 4th search provider, optimized for RAG applications and real-time information:

Features

• LLM-Ready Snippets: Pre-extracted, query-aware text excerpts perfect for feeding into AI models
• Unified Web + News: Get both web pages and news articles in a single API call
• Live Crawling: Fetch full page content on-demand in Markdown format (--livecrawl)
• Automatic News Classification: Intelligently includes news results based on query intent
• Freshness Controls: Filter by recency (day, week, month, year, or date range)
• SafeSearch Support: Content filtering (off, moderate, strict)

Auto-Routing Signals

New RAG/Real-time intent detection routes to You.com for:

• RAG context queries: "summarize", "key points", "tldr", "context for"
• Real-time info: "latest news", "current status", "right now", "what's happening"
• Information synthesis: "updates on", "situation", "main takeaways"

Usage Examples

# Auto-routed
python3 scripts/search.py -q "summarize key points about AI regulation"  # → You.com

# Explicit
python3 scripts/search.py -p you -q "climate change" --livecrawl all
python3 scripts/search.py -p you -q "tech news" --freshness week

Configuration

{
  "you": {
    "country": "US",
    "language": "en",
    "safesearch": "moderate",
    "include_news": true
  }
}

API Key Setup

export YOU_API_KEY="your-key"  # Get from https://api.you.com

📊 Updated Provider Comparison

| Feature | Serper | Tavily | Exa | You.com | |---------|:------:|:------:|:---:|:-------:| | Speed | ⚡⚡⚡ | ⚡⚡ | ⚡⚡ | ⚡⚡⚡ | | News Integration | ✓ | ✗ | ✗ | ✓ | | RAG-Optimized | ✗ | ✓ | ✗ | ✓✓ | | Full Page Content | ✗ | ✓ | ✓ | ✓ |

---

[2.1.5] - 2026-01-27

📝 Documentation

• Added warning about NOT using Tavily/Serper/Exa in core OpenClaw config
• Core OpenClaw only supports brave as the built-in provider
• This skill's providers must be used via environment variables and scripts, not openclaw.json

[2.1.0] - 2026-01-23

🧠 Intelligent Multi-Signal Routing

Completely overhauled auto-routing with sophisticated query analysis:

Intent Classification

• Shopping Intent: Detects price patterns ("how much", "cost of"), purchase signals ("buy", "order"), deal keywords, and product+brand combinations
• Research Intent: Identifies explanation patterns ("how does", "why does"), analysis signals ("pros and cons", "compare"), learning keywords, and complex multi-clause queries
• Discovery Intent: Recognizes similarity patterns ("similar to", "alternatives"), company discovery signals, URL/domain detection, and academic patterns

Linguistic Pattern Detection

• "How much" / "price of" → Shopping (Serper)
• "How does" / "Why does" / "Explain" → Research (Tavily)
• "Companies like" / "Similar to" / "Alternatives" → Discovery (Exa)
• Product + Brand name combos → Shopping (Serper)
• URLs and domains in query → Similar search (Exa)

Query Analysis Features

• Complexity scoring: Long, multi-clause queries get routed to research providers
• URL detection: Automatic detection of URLs/domains triggers Exa similar search
• Brand recognition: Tech brands (Apple, Samsung, Sony, etc.) with product terms → shopping
• Recency signals: "latest", "2026", "breaking" boost news mode

Confidence Scoring

• HIGH (70-100%): Strong signal match, very reliable routing
• MEDIUM (40-69%): Good match, should work well
• LOW (0-39%): Ambiguous query, using fallback provider
• Confidence based on absolute signal strength + relative margin over alternatives

Enhanced Debug Mode

python3 scripts/search.py --explain-routing -q "your query"

Now shows:

• Routing decision with confidence level
• All provider scores
• Top matched signals with weights
• Query analysis (complexity, URL detection, recency focus)
• All matched patterns per provider

🔧 Technical Changes

QueryAnalyzer Class

New QueryAnalyzer class with:

• SHOPPING_SIGNALS: 25+ weighted patterns for shopping intent
• RESEARCH_SIGNALS: 30+ weighted patterns for research intent
• DISCOVERY_SIGNALS: 20+ weighted patterns for discovery intent
• LOCAL_NEWS_SIGNALS: 25+ patterns for local/news queries
• BRAND_PATTERNS: Tech brand detection regex

Signal Weighting

• Multi-word phrases get higher weights (e.g., "how much" = 4.0 vs "price" = 3.0)
• Strong signals: price patterns (4.0), similarity patterns (5.0), URLs (5.0)
• Medium signals: product terms (2.5), learning keywords (2.5)
• Bonus scoring: Product+brand combo (+3.0), complex query (+2.5)

Improved Output Format

{
  "routing": {
    "auto_routed": true,
    "provider": "serper",
    "confidence": 0.78,
    "confidence_level": "high",
    "reason": "high_confidence_match",
    "top_signals": [{"matched": "price", "weight": 3.0}],
    "scores": {"serper": 7.0, "tavily": 0.0, "exa": 0.0}
  }
}

📚 Documentation Updates

• SKILL.md: Complete rewrite with signal tables and confidence scoring guide
• README.md: Updated with intelligent routing examples and confidence levels
• FAQ: Updated to explain multi-signal analysis

🧪 Test Results

| Query | Provider | Confidence | Signals | |-------|----------|------------|---------| | "how much does iPhone 16 cost" | Serper | 68% | "how much", brand+product | | "how does quantum entanglement work" | Tavily | 86% HIGH | "how does", "what are", "implications" | | "startups similar to Notion" | Exa | 76% HIGH | "similar to", "Series A" | | "companies like stripe.com" | Exa | 100% HIGH | URL detected, "companies like" | | "MacBook Pro M3 specs review" | Serper | 70% HIGH | brand+product, "specs", "review" | | "Tesla" | Serper | 0% LOW | No signals (fallback) | | "arxiv papers on transformers" | Exa | 58% | "arxiv" | | "latest AI news 2026" | Serper | 77% HIGH | "latest", "news", "2026" |

---

[2.0.0] - 2026-01-23

🎉 Major Features

Smart Auto-Routing

• Automatic provider selection based on query analysis
• No need to manually choose provider - just search!
• Intelligent keyword matching for routing decisions
• Pattern detection for query types (shopping, research, discovery)
• Scoring system for provider selection

User Configuration

• config.json: Full control over auto-routing behavior
• Configurable keyword mappings: Add your own routing keywords
• Provider priority: Set tie-breaker order
• Disable providers: Turn off providers you don't have API keys for
• Enable/disable auto-routing: Opt-in or opt-out as needed

Debugging Tools

• --explain-routing flag: See exactly why a provider was selected
• Detailed routing metadata in JSON responses
• Shows matched keywords and routing scores

📚 Documentation

• README.md: Complete auto-routing guide with examples
• SKILL.md: Detailed routing logic and configuration reference
• FAQ section: Common questions about auto-routing
• Configuration examples: Pre-built configs for common use cases

---

[1.0.x] - Initial Release

• Multi-provider search: Serper, Tavily, Exa
• Manual provider selection with -p flag
• Unified JSON output format
• Provider-specific options (--depth, --category, --similar-url, etc.)
• Domain filtering for Tavily/Exa
• Date filtering for Exa

File v2.8.5:FAQ.md

Frequently Asked Questions

Caching (NEW in v2.7.0!)

How does caching work?

Search results are automatically cached locally for 1 hour (3600 seconds). When you make the same query again, you get instant results at $0 API cost. The cache key is based on: query text + provider + max_results.

Where are cached results stored?

In .cache/ directory inside the skill folder by default. Override with WSP_CACHE_DIR environment variable:

export WSP_CACHE_DIR="/path/to/custom/cache"

How do I see cache stats?

python3 scripts/search.py --cache-stats

This shows total entries, size, oldest/newest entries, and breakdown by provider.

How do I clear the cache?

python3 scripts/search.py --clear-cache

Can I change the cache TTL?

Yes! Default is 3600 seconds (1 hour). Set a custom TTL per request:

python3 scripts/search.py -q "query" --cache-ttl 7200  # 2 hours

How do I skip the cache?

Use --no-cache to always fetch fresh results:

python3 scripts/search.py -q "query" --no-cache

How do I know if a result was cached?

The response includes:

• "cached": true/false — whether result came from cache
• "cache_age_seconds": 1234 — how old the cached result is (when cached)

---

General

How does auto-routing decide which provider to use?

Multi-signal analysis scores each provider based on: price patterns, explanation phrases, similarity keywords, URLs, product+brand combos, and query complexity. Highest score wins. Use --explain-routing to see the decision breakdown.

What if it picks the wrong provider?

Override with -p serper/tavily/exa. Check --explain-routing to understand why it chose differently.

What does "low confidence" mean?

Query is ambiguous (e.g., "Tesla" could be cars, stock, or company). Falls back to Serper. Results may vary.

Can I disable a provider?

Yes! In config.json: "disabled_providers": ["exa"]

---

API Keys

Which API keys do I need?

At minimum ONE key (or SearXNG instance). You can use just Serper, just Tavily, just Exa, just You.com, or just SearXNG. Missing keys = that provider is skipped.

Where do I get API keys?

• Serper: https://serper.dev (2,500 free queries, no credit card)
• Tavily: https://tavily.com (1,000 free searches/month)
• Exa: https://exa.ai (1,000 free searches/month)
• You.com: https://api.you.com (Limited free tier for testing)
• SearXNG: Self-hosted, no key needed! https://docs.searxng.org/admin/installation.html

How do I set API keys?

Two options (both auto-load):

Option A: .env file

export SERPER_API_KEY="your-key"

Option B: config.json (v2.2.1+)

{ "serper": { "api_key": "your-key" } }

---

Routing Details

How do I know which provider handled my search?

Check routing.provider in JSON output, or [🔍 Searched with: Provider] in chat responses.

Why does it sometimes choose Serper for research questions?

If the query has brand/product signals (e.g., "how does Tesla FSD work"), shopping intent may outweigh research intent. Override with -p tavily.

What's the confidence threshold?

Default: 0.3 (30%). Below this = low confidence, uses fallback. Adjustable in config.json.

---

You.com Specific

When should I use You.com over other providers?

You.com excels at:

• RAG applications: Pre-extracted snippets ready for LLM consumption
• Real-time information: Current events, breaking news, status updates
• Combined sources: Web + news results in a single API call
• Summarization tasks: "What's the latest on...", "Key points about..."

What's the livecrawl feature?

You.com can fetch full page content on-demand. Use --livecrawl web for web results, --livecrawl news for news articles, or --livecrawl all for both. Content is returned in Markdown format.

Does You.com include news automatically?

Yes! You.com's intelligent classification automatically includes relevant news results when your query has news intent. You can also use --include-news to explicitly enable it.

---

SearXNG Specific

Do I need my own SearXNG instance?

Yes! SearXNG is self-hosted. Most public instances disable the JSON API to prevent bot abuse. You need to run your own instance with JSON format enabled. See: https://docs.searxng.org/admin/installation.html

How do I set up SearXNG?

Docker is the easiest way:

docker run -d -p 8080:8080 searxng/searxng

Then enable JSON in settings.yml:

search:
  formats:
    - html
    - json

Why am I getting "403 Forbidden"?

The JSON API is disabled on your instance. Enable it in settings.yml under search.formats.

What's the API cost for SearXNG?

$0! SearXNG is free and open-source. You only pay for hosting (~$5/month VPS). Unlimited queries.

When should I use SearXNG?

• Privacy-sensitive queries: No tracking, no profiling
• Budget-conscious: $0 API cost
• Diverse results: Aggregates 70+ search engines
• Self-hosted requirements: Full control over your search infrastructure
• Fallback provider: When paid APIs are rate-limited

Can I limit which search engines SearXNG uses?

Yes! Use --engines google,bing,duckduckgo to specify engines, or configure defaults in config.json.

---

Provider Selection

Which provider should I use?

| Query Type | Best Provider | Why | |------------|---------------|-----| | Shopping ("buy laptop", "cheap shoes") | Serper | Google Shopping, price comparisons, local stores | | Research ("how does X work?", "explain Y") | Tavily | Deep research, academic quality, full-page content | | Startups/Papers ("companies like X", "arxiv papers") | Exa | Semantic/neural search, startup discovery | | RAG/Real-time ("summarize latest", "current events") | You.com | LLM-ready snippets, combined web+news | | Privacy ("search without tracking") | SearXNG | No tracking, multi-source, self-hosted |

Tip: Enable auto-routing and let the skill choose automatically! 🎯

Do I need all 5 providers?

No! All providers are optional. You can use:

• 1 provider (e.g., just Serper for everything)
• 2-3 providers (e.g., Serper + You.com for most needs)
• All 5 (maximum flexibility + fallback options)

How much do the APIs cost?

| Provider | Free Tier | Paid Plan | |----------|-----------|-----------| | Serper | 2,500 queries/mo | $50/mo (5,000 queries) | | Tavily | 1,000 queries/mo | $150/mo (10,000 queries) | | Exa | 1,000 queries/mo | $1,000/mo (100,000 queries) | | You.com | Limited free | ~$10/mo (varies by usage) | | SearXNG | FREE ✅ | Only VPS cost (~$5/mo if self-hosting) |

Budget tip: Use SearXNG as primary + others as fallback for specialized queries!

How private is SearXNG really?

| Setup | Privacy Level | |-------|---------------| | Self-hosted (your VPS) | ⭐⭐⭐⭐⭐ You control everything | | Self-hosted (Docker local) | ⭐⭐⭐⭐⭐ Fully private | | Public instance | ⭐⭐⭐ Depends on operator's logging policy |

Best practice: Self-host if privacy is critical.

Which provider has the best results?

| Metric | Winner | |--------|--------| | Most accurate for facts | Serper (Google) | | Best for research depth | Tavily | | Best for semantic queries | Exa | | Best for RAG/AI context | You.com | | Most diverse sources | SearXNG (70+ engines) | | Most private | SearXNG (self-hosted) |

Recommendation: Enable multiple providers + auto-routing for best overall experience.

How does auto-routing work?

The skill analyzes your query for keywords and patterns:

"buy cheap laptop"     → Serper (shopping signals)
"how does AI work?"    → Tavily (research/explanation)
"companies like X"     → Exa (semantic/similar)
"summarize latest news" → You.com (RAG/real-time)
"search privately"     → SearXNG (privacy signals)

Confidence threshold: Only routes if confidence > 30%. Otherwise uses default provider.

Override: Use -p provider to force a specific provider.

---

Production Use

Can I use this in production?

Yes! Web-search-plus is production-ready:

• ✅ Error handling with automatic fallback
• ✅ Rate limit protection
• ✅ Timeout handling (30s per provider)
• ✅ API key security (.env + config.json gitignored)
• ✅ 5 providers for redundancy

Tip: Monitor API usage to avoid exceeding free tiers!

What if I run out of API credits?

1. Fallback chain: Other enabled providers automatically take over
2. Use SearXNG: Switch to self-hosted (unlimited queries)
3. Upgrade plan: Paid tiers have higher limits
4. Rate limit: Use disabled_providers to skip exhausted APIs temporarily

---

Updates

How do I update to the latest version?

Via ClawHub (recommended):

clawhub update web-search-plus --registry "https://www.clawhub.ai" --no-input

Manually:

cd /path/to/workspace/skills/web-search-plus/
git pull origin main
python3 scripts/setup.py  # Re-run to configure new features

Where can I report bugs or request features?

• GitHub Issues: https://github.com/robbyczgw-cla/web-search-plus/issues
• ClawHub: https://www.clawhub.ai/skills/web-search-plus

File v2.8.5:TROUBLESHOOTING.md

Troubleshooting Guide

Caching Issues (v2.7.0+)

Cache not working / always fetching fresh

Symptoms:

• Every request hits the API
• "cached": false even for repeated queries

Solutions:

1. Check cache directory exists and is writable:

   ls -la .cache/  # Should exist in skill directory
   

2. Verify --no-cache isn't being passed
3. Check disk space isn't full
4. Ensure query is EXACTLY the same (including provider and max_results)

Stale results from cache

Symptoms:

• Getting outdated information
• Cache TTL seems too long

Solutions:

1. Use --no-cache to force fresh results
2. Reduce TTL: --cache-ttl 1800 (30 minutes)
3. Clear cache: python3 scripts/search.py --clear-cache

Cache growing too large

Symptoms:

• Disk space filling up
• Many .json files in .cache/

Solutions:

1. Clear cache periodically:

   python3 scripts/search.py --clear-cache
   

2. Set up a cron job to clear weekly
3. Use a smaller TTL so entries expire faster

"Permission denied" when caching

Symptoms:

• Cache write errors in stderr
• Searches work but don't cache

Solutions:

1. Check directory permissions: chmod 755 .cache/
2. Use custom cache dir: export WSP_CACHE_DIR="/tmp/wsp-cache"

---

Common Issues

"No API key found" error

Symptoms:

Error: No API key found for serper

Solutions:

1. Check .env exists in skill folder with export VAR=value format
2. Keys auto-load from skill's .env since v2.2.0
3. Or set in system environment: export SERPER_API_KEY="..."
4. Verify key format in config.json:

   { "serper": { "api_key": "your-key" } }
   

Priority order: config.json > .env > environment variable

---

Getting empty results

Symptoms:

• Search returns no results
• "results": [] in JSON output

Solutions:

1. Check API key is valid (try the provider's web dashboard)
2. Try a different provider with -p
3. Some queries have no results (very niche topics)
4. Check if provider is rate-limited
5. Verify internet connectivity

Debug:

python3 scripts/search.py -q "test query" --verbose

---

Rate limited

Symptoms:

Error: 429 Too Many Requests
Error: Rate limit exceeded

Good news: Since v2.2.5, automatic fallback kicks in! If one provider hits rate limits, the script automatically tries the next provider.

Solutions:

1. Wait for rate limit to reset (usually 1 hour or end of day)
2. Use a different provider: -p tavily instead of -p serper
3. Check free tier limits:
   • Serper: 2,500 free total
   • Tavily: 1,000/month free
   • Exa: 1,000/month free
4. Upgrade to paid tier for higher limits
5. Use SearXNG (self-hosted, unlimited)

Fallback info: Response will include routing.fallback_used: true when fallback was used.

---

SearXNG: "403 Forbidden"

Symptoms:

Error: 403 Forbidden
Error: JSON format not allowed

Cause: Most public SearXNG instances disable JSON API to prevent bot abuse.

Solution: Self-host your own instance:

docker run -d -p 8080:8080 searxng/searxng

Then enable JSON in settings.yml:

search:
  formats:
    - html
    - json  # Add this!

Restart the container and update your config:

{
  "searxng": {
    "instance_url": "http://localhost:8080"
  }
}

---

SearXNG: Slow responses

Symptoms:

• SearXNG takes 2-5 seconds
• Other providers are faster

Explanation: This is expected behavior. SearXNG queries 70+ upstream engines in parallel, which takes longer than direct API calls.

Trade-off: Slower but privacy-preserving + multi-source + $0 cost.

Solutions:

1. Accept the trade-off for privacy benefits
2. Limit engines for faster results:

   python3 scripts/search.py -p searxng -q "query" --engines "google,bing"
   

3. Use SearXNG as fallback (put last in priority list)

---

Auto-routing picks wrong provider

Symptoms:

• Query about research goes to Serper
• Query about shopping goes to Tavily

Debug:

python3 scripts/search.py --explain-routing -q "your query"

This shows the full analysis:

{
  "query": "how much does iPhone 16 Pro cost",
  "routing_decision": {
    "provider": "serper",
    "confidence": 0.68,
    "reason": "moderate_confidence_match"
  },
  "scores": {"serper": 7.0, "tavily": 0.0, "exa": 0.0},
  "top_signals": [
    {"matched": "how much", "weight": 4.0},
    {"matched": "brand + product detected", "weight": 3.0}
  ]
}

Solutions:

1. Override with explicit provider: -p tavily
2. Rephrase query to be more explicit about intent
3. Adjust confidence_threshold in config.json (default: 0.3)

---

Config not loading

Symptoms:

• Changes to config.json not applied
• Using default values instead

Solutions:

1. Check JSON syntax (use a validator)
2. Ensure file is in skill directory: /path/to/skills/web-search-plus/config.json
3. Check file permissions
4. Run setup wizard to regenerate:

   python3 scripts/setup.py --reset
   

Validate JSON:

python3 -m json.tool config.json

---

Python dependencies missing

Symptoms:

ModuleNotFoundError: No module named 'requests'

Solution:

pip3 install requests

Or install all dependencies:

pip3 install -r requirements.txt

---

Timeout errors

Symptoms:

Error: Request timeout after 30s

Causes:

• Slow network connection
• Provider API issues
• SearXNG instance overloaded

Solutions:

1. Try again (temporary issue)
2. Switch provider: -p serper
3. Check your internet connection
4. If using SearXNG, check instance health

---

Duplicate results

Symptoms:

• Same result appears multiple times
• Results overlap between providers

Solution: This is expected when using auto-fallback or multiple providers. The skill doesn't deduplicate across providers.

For single-provider results:

python3 scripts/search.py -p serper -q "query"

---

Debug Mode

For detailed debugging:

# Verbose output
python3 scripts/search.py -q "query" --verbose

# Show routing decision
python3 scripts/search.py -q "query" --explain-routing

# Dry run (no actual search)
python3 scripts/search.py -q "query" --dry-run

# Test specific provider
python3 scripts/search.py -p tavily -q "query" --verbose

---

Getting Help

Still stuck?

1. Check the full documentation in README.md
2. Run the setup wizard: python3 scripts/setup.py
3. Review FAQ.md for common questions
4. Open an issue: https://github.com/robbyczgw-cla/web-search-plus/issues

File v2.8.5:config.example.json

{ "$schema": "https://json-schema.org/draft/2020-12/schema", "$comment": "Web Search Plus configuration — intelligent routing and provider settings", "defaults": { "provider": "serper", "max_results": 5 }, "auto_routing": { "enabled": true, "fallback_provider": "serper", "provider_priority": [ "serper", "tavily", "exa", "you", "searxng" ], "disabled_providers": [], "confidence_threshold": 0.3, "keyword_mappings": { "serper": [ "price", "buy", "shop", "shopping", "cost", "deal", "sale", "purchase", "cheap", "expensive", "store", "product", "review", "specs", "specification", "where to buy", "near me", "local", "restaurant", "hotel", "weather", "news", "latest", "breaking", "map", "directions", "phone number", "preis", "kaufen", "bestellen", "günstig", "billig", "teuer", "kosten", "angebot", "rabatt", "shop", "händler", "geschäft", "laden", "test", "bewertung", "technische daten", "spezifikationen", "wo kaufen", "in der nähe", "wetter", "nachrichten", "aktuell", "neu" ], "tavily": [ "how does", "how to", "explain", "research", "what is", "why does", "analyze", "compare", "study", "academic", "detailed", "comprehensive", "in-depth", "understand", "learn", "tutorial", "guide", "overview", "history of", "background", "context", "implications", "pros and cons", "wie funktioniert", "erklärung", "erklären", "was ist", "warum", "analyse", "vergleich", "vergleichen", "studie", "verstehen", "lernen", "anleitung", "tutorial", "überblick", "hintergrund", "vor- und nachteile" ], "exa": [ "similar to", "companies like", "find sites like", "alternatives to", "competitors", "startup", "github", "paper", "research paper", "arxiv", "pdf", "academic paper", "similar pages", "related sites", "who else", "other companies", "comparable to", "ähnlich wie", "firmen wie", "alternativen zu", "konkurrenten", "vergleichbar mit", "andere unternehmen" ], "you": [ "rag", "context for", "summarize", "brief", "quick overview", "tldr", "key points", "key facts", "main points", "main takeaways", "latest news", "latest updates", "current events", "current situation", "current status", "right now", "as of today", "up to date", "real time", "what's happening", "what's the latest", "updates on", "status of", "zusammenfassung", "aktuelle nachrichten", "neueste updates" ], "searxng": [ "private", "privately", "anonymous", "anonymously", "without tracking", "no tracking", "privacy", "privacy-focused", "privacy-first", "duckduckgo alternative", "private search", "aggregate results", "multiple sources", "diverse results", "diverse perspectives", "meta search", "all engines", "free search", "no api cost", "self-hosted search", "zero cost", "privat", "anonym", "ohne tracking", "datenschutz", "verschiedene quellen", "aus mehreren quellen", "alle suchmaschinen", "kostenlose suche", "keine api kosten" ] } }, "serper": { "country": "us", "language": "en", "type": "search", "autocorrect": true, "include_images": false }, "tavily": { "depth": "advanced", "topic": "general", "max_results": 8 }, "exa": { "type": "neural", "category": null, "include_domains": [], "exclude_domains": [] }, "you": { "country": "US", "language": "en", "safesearch": "moderate", "include_news": true }, "searxng": { "$comment": "SearXNG requires a self-hosted instance. No API key needed, just your instance URL.", "instance_url": null, "safesearch": 0, "engines": null, "language": "en" } }

File v2.8.5:package.json

{ "name": "@openclaw/web-search-plus", "version": "2.8.5", "description": "Unified search skill with Intelligent Auto-Routing. Uses multi-signal analysis (intent classification, linguistic patterns, URL/brand detection) to automatically select between Serper (Google), Tavily (Research), Exa (Neural), and You.com (RAG/Real-time) with confidence scoring.", "keywords": [ "openclaw", "skill", "search", "web-search", "serper", "tavily", "exa", "you", "you.com", "google-search", "research", "semantic-search", "ai-agent", "auto-routing", "smart-routing", "multi-provider", "shopping", "product-search", "similar-sites", "company-discovery", "rag", "real-time", "free-tier", "api-aggregator" ], "author": "robbyczgw-cla", "license": "MIT", "repository": { "type": "git", "url": "https://github.com/robbyczgw-cla/web-search-plus.git" }, "homepage": "https://clawhub.ai/robbyczgw-cla/web-search-plus", "bugs": { "url": "https://github.com/robbyczgw-cla/web-search-plus/issues" }, "openclaw": { "skill": true, "triggers": [ "search", "find", "look up", "research" ], "capabilities": [ "web-search", "image-search", "semantic-search", "multi-provider" ], "providers": [ "serper", "tavily", "exa", "you" ], "requirements": { "bins": ["python3", "bash"], "env": { "SERPER_API_KEY": "optional", "TAVILY_API_KEY": "optional", "EXA_API_KEY": "optional", "YOU_API_KEY": "optional", "SEARXNG_INSTANCE_URL": "optional" } } }, "files": [ "SKILL.md", "README.md", "scripts/", ".env.example" ] }

Archive v2.8.4: 11 files, 58352 bytes

Files: CHANGELOG.md (18072b), config.example.json (4972b), FAQ.md (9360b), package.json (1801b), README.md (23073b), scripts/search.py (88415b), scripts/setup.py (18486b), SKILL.md (9693b), test-auto-routing.sh (555b), TROUBLESHOOTING.md (6664b), _meta.json (134b)

File v2.8.4:SKILL.md

---

name: web-search-plus version: 2.8.1 description: Unified search skill with Intelligent Auto-Routing. Uses multi-signal analysis to automatically select between Serper (Google), Tavily (Research), Exa (Neural), Perplexity (AI Answers), You.com (RAG/Real-time), and SearXNG (Privacy/Self-hosted) with confidence scoring. tags: [search, web-search, serper, tavily, exa, perplexity, you, searxng, google, research, semantic-search, auto-routing, multi-provider, shopping, rag, free-tier, privacy, self-hosted, kilo] metadata: {"openclaw":{"requires":{"bins":["python3","bash"],"env":{"SERPER_API_KEY":"optional","TAVILY_API_KEY":"optional","EXA_API_KEY":"optional","YOU_API_KEY":"optional","SEARXNG_INSTANCE_URL":"optional","KILOCODE_API_KEY":"optional — required for Perplexity provider (via Kilo Gateway)"},"note":"Only ONE provider key needed. All are optional."}}}

Web Search Plus

Stop choosing search providers. Let the skill do it for you.

This skill connects you to 6 search providers (Serper, Tavily, Exa, Perplexity, You.com, SearXNG) and automatically picks the best one for each query. Shopping question? → Google results. Research question? → Deep research engine. Need a direct answer? → AI-synthesized with citations. Want privacy? → Self-hosted option.

---

✨ What Makes This Different?

• Just search — No need to think about which provider to use
• Smart routing — Analyzes your query and picks the best provider automatically
• 6 providers, 1 interface — Google results, research engines, neural search, AI answers with citations, RAG-optimized, and privacy-first all in one
• Works with just 1 key — Start with any single provider, add more later
• Free options available — SearXNG is completely free (self-hosted)

---

🚀 Quick Start

# Interactive setup (recommended for first run)
python3 scripts/setup.py

# Or manual: copy config and add your keys
cp config.example.json config.json

The wizard explains each provider, collects API keys, and configures defaults.

---

🔑 API Keys

You only need ONE key to get started. Add more providers later for better coverage.

| Provider | Free Tier | Best For | Sign Up | |----------|-----------|----------|---------| | Serper | 2,500/mo | Shopping, prices, local, news | serper.dev | | Tavily | 1,000/mo | Research, explanations, academic | tavily.com | | Exa | 1,000/mo | "Similar to X", startups, papers | exa.ai | | Perplexity | Via Kilo | Direct answers with citations | kilo.ai | | You.com | Limited | Real-time info, AI/RAG context | api.you.com | | SearXNG | FREE ✅ | Privacy, multi-source, $0 cost | Self-hosted |

Setting your keys:

# Option A: .env file (recommended)
export SERPER_API_KEY="your-key"
export TAVILY_API_KEY="your-key"

# Option B: config.json
{ "serper": { "api_key": "your-key" } }

---

🎯 When to Use Which Provider

| I want to... | Provider | Example Query | |--------------|----------|---------------| | Find product prices | Serper | "iPhone 16 Pro Max price" | | Find restaurants/stores nearby | Serper | "best pizza near me" | | Understand how something works | Tavily | "how does HTTPS encryption work" | | Do deep research | Tavily | "climate change research 2024" | | Find companies like X | Exa | "startups similar to Notion" | | Find research papers | Exa | "transformer architecture papers" | | Get a direct answer with sources | Perplexity | "events in Berlin this weekend" | | Know the current status of something | Perplexity | "what is the status of Ethereum upgrades" | | Get real-time info | You.com | "latest AI regulation news" | | Search without being tracked | SearXNG | anything, privately |

Pro tip: Just search normally! Auto-routing handles most queries correctly. Override with -p provider when needed.

---

🧠 How Auto-Routing Works

The skill looks at your query and picks the best provider:

"iPhone 16 price"              → Serper (shopping keywords)
"how does quantum computing work" → Tavily (research question)
"companies like stripe.com"    → Exa (URL detected, similarity)
"events in Graz this weekend"  → Perplexity (local + direct answer)
"latest news on AI"            → You.com (real-time intent)
"search privately"             → SearXNG (privacy keywords)

What if it picks wrong? Override it: python3 scripts/search.py -p tavily -q "your query"

Debug routing: python3 scripts/search.py --explain-routing -q "your query"

---

📖 Usage Examples

Let Auto-Routing Choose (Recommended)

python3 scripts/search.py -q "Tesla Model 3 price"
python3 scripts/search.py -q "explain machine learning"
python3 scripts/search.py -q "startups like Figma"

Force a Specific Provider

python3 scripts/search.py -p serper -q "weather Berlin"
python3 scripts/search.py -p tavily -q "quantum computing" --depth advanced
python3 scripts/search.py -p exa --similar-url "https://stripe.com" --category company
python3 scripts/search.py -p you -q "breaking tech news" --include-news
python3 scripts/search.py -p searxng -q "linux distros" --engines "google,bing"

---

⚙ Configuration

{
  "auto_routing": {
    "enabled": true,
    "fallback_provider": "serper",
    "confidence_threshold": 0.3,
    "disabled_providers": []
  },
  "serper": {"country": "us", "language": "en"},
  "tavily": {"depth": "advanced"},
  "exa": {"type": "neural"},
  "you": {"country": "US", "include_news": true},
  "searxng": {"instance_url": "https://your-instance.example.com"}
}

---

📊 Provider Comparison

| Feature | Serper | Tavily | Exa | Perplexity | You.com | SearXNG | |---------|:------:|:------:|:---:|:----------:|:-------:|:-------:| | Speed | ⚡⚡⚡ | ⚡⚡ | ⚡⚡ | ⚡⚡ | ⚡⚡⚡ | ⚡⚡ | | Direct Answers | ✗ | ✗ | ✗ | ✓✓ | ✗ | ✗ | | Citations | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | | Factual Accuracy | ⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ | | Semantic Understanding | ⭐ | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ | ⭐ | | Full Page Content | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | | Shopping/Local | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | | Find Similar Pages | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | | RAG-Optimized | ✗ | ✓ | ✗ | ✗ | ✓✓ | ✗ | | Privacy-First | ✗ | ✗ | ✗ | ✗ | ✗ | ✓✓ | | API Cost | $$ | $$ | $$ | Via Kilo | $ | FREE |

---

❓ Common Questions

Do I need API keys for all providers?

No. You only need keys for providers you want to use. Start with one (Serper recommended), add more later.

Which provider should I start with?

Serper — fastest, cheapest, largest free tier (2,500 queries/month), and handles most queries well.

What if I run out of free queries?

The skill automatically falls back to your other configured providers. Or switch to SearXNG (unlimited, self-hosted).

How much does this cost?

• Free tiers: 2,500 (Serper) + 1,000 (Tavily) + 1,000 (Exa) = 4,500+ free searches/month
• SearXNG: Completely free (just ~$5/mo if you self-host on a VPS)
• Paid plans: Start around $10-50/month depending on provider

Is SearXNG really private?

Yes, if self-hosted. You control the server, no tracking, no profiling. Public instances depend on the operator's policy.

How do I set up SearXNG?

# Docker (5 minutes)
docker run -d -p 8080:8080 searxng/searxng

Then enable JSON API in settings.yml. See docs.searxng.org.

Why did it route my query to the "wrong" provider?

Sometimes queries are ambiguous. Use --explain-routing to see why, then override with -p provider if needed.

---

🔄 Automatic Fallback

If one provider fails (rate limit, timeout, error), the skill automatically tries the next provider. You'll see routing.fallback_used: true in the response when this happens.

---

📤 Output Format

{
  "provider": "serper",
  "query": "iPhone 16 price",
  "results": [{"title": "...", "url": "...", "snippet": "...", "score": 0.95}],
  "routing": {
    "auto_routed": true,
    "provider": "serper",
    "confidence": 0.78,
    "confidence_level": "high"
  }
}

---

⚠ Important Note

Tavily, Serper, and Exa are NOT core OpenClaw providers.

❌ Don't modify ~/.openclaw/openclaw.json for these
✅ Use this skill's scripts — keys auto-load from .env

---

🔒 Security

SearXNG SSRF Protection: The SearXNG instance URL is validated with defense-in-depth:

• Enforces http/https schemes only
• Blocks cloud metadata endpoints (169.254.169.254, metadata.google.internal)
• Resolves hostnames and blocks private/internal IPs (loopback, RFC1918, link-local, reserved)
• Operators who intentionally self-host on private networks can set SEARXNG_ALLOW_PRIVATE=1

📚 More Documentation

• FAQ.md — Detailed answers to more questions
• TROUBLESHOOTING.md — Fix common errors
• README.md — Full technical reference

---

🔗 Quick Links

• Serper — Google Search API
• Tavily — AI Research Search
• Exa — Neural Search
• Perplexity — AI-Synthesized Answers (via Kilo Gateway)
• You.com — RAG/Real-time Search
• SearXNG — Privacy-First Meta-Search

File v2.8.4:README.md

Web Search Plus

Unified multi-provider web search with Intelligent Auto-Routing — uses multi-signal analysis to automatically select between Serper, Tavily, Exa, You.com, and SearXNG with confidence scoring.

---

🧠 Features (v2.7.0)

Intelligent Multi-Signal Routing — The skill uses sophisticated query analysis:

• Intent Classification: Shopping vs Research vs Discovery vs RAG/Real-time vs Privacy
• Linguistic Patterns: "how much" (price) vs "how does" (research) vs "privately" (privacy)
• Entity Detection: Product+brand combos, URLs, domains
• Complexity Analysis: Long queries favor research providers
• Confidence Scoring: Know how reliable the routing decision is

python3 scripts/search.py -q "how much does iPhone 16 cost"     # → Serper (68% confidence)
python3 scripts/search.py -q "how does quantum entanglement work"  # → Tavily (86% HIGH)
python3 scripts/search.py -q "startups similar to Notion"       # → Exa (76% HIGH)
python3 scripts/search.py -q "companies like stripe.com"        # → Exa (100% HIGH - URL detected)
python3 scripts/search.py -q "summarize key points on AI"       # → You.com (68% MEDIUM - RAG intent)
python3 scripts/search.py -q "search privately without tracking" # → SearXNG (74% HIGH - privacy intent)

---

🔍 When to Use Which Provider

Built-in Brave Search (OpenClaw default)

• ✅ General web searches
• ✅ Privacy-focused
• ✅ Quick lookups
• ✅ Default fallback

Serper (Google Results)

• 🛍 Product specs, prices, shopping
• 📍 Local businesses, places
• 🎯 "Google it" - explicit Google results
• 📰 Shopping/images needed
• 🏆 Knowledge Graph data

Tavily (AI-Optimized Research)

• 📚 Research questions, deep dives
• 🔬 Complex multi-part queries
• 📄 Need full page content (not just snippets)
• 🎓 Academic/technical research
• 🔒 Domain filtering (trusted sources)

Exa (Neural Semantic Search)

• 🔗 Find similar pages
• 🏢 Company/startup discovery
• 📝 Research papers
• 💻 GitHub projects
• 📅 Date-specific content

You.com (RAG/Real-time)

• 🤖 RAG applications (LLM-ready snippets)
• 📰 Combined web + news (single API call)
• ⚡ Real-time information (current events)
• 📋 Summarization context ("What's the latest...")
• 🔄 Live crawling (full page content on demand)

SearXNG (Privacy-First/Self-Hosted)

• 🔒 Privacy-preserving search (no tracking)
• 🌐 Multi-source aggregation (70+ engines)
• 💰 $0 API cost (self-hosted)
• 🎯 Diverse perspectives (results from multiple engines)
• 🏠 Self-hosted environments (full control)

---

Table of Contents

• Quick Start
• Smart Auto-Routing
• Configuration Guide
• Provider Deep Dives
• Usage Examples
• Workflow Examples
• Optimization Tips
• FAQ & Troubleshooting
• API Reference

---

Quick Start

Option A: Interactive Setup (Recommended)

# Run the setup wizard - it guides you through everything
python3 scripts/setup.py

The wizard explains each provider, collects your API keys, and creates config.json automatically.

Option B: Manual Setup

# 1. Set up at least one API key (or SearXNG instance)
export SERPER_API_KEY="your-key"   # https://serper.dev
export TAVILY_API_KEY="your-key"   # https://tavily.com
export EXA_API_KEY="your-key"      # https://exa.ai
export YOU_API_KEY="your-key"      # https://api.you.com
export SEARXNG_INSTANCE_URL="https://your-instance.example.com"  # Self-hosted

# 2. Run a search (auto-routed!)
python3 scripts/search.py -q "best laptop 2024"

Run a Search

# Auto-routed to best provider
python3 scripts/search.py -q "best laptop 2024"

# Or specify a provider explicitly
python3 scripts/search.py -p serper -q "iPhone 16 specs"
python3 scripts/search.py -p tavily -q "quantum computing explained" --depth advanced
python3 scripts/search.py -p exa -q "AI startups 2024" --category company

---

Smart Auto-Routing

How It Works

When you don't specify a provider, the skill analyzes your query and routes it to the best provider:

| Query Contains | Routes To | Example | |---------------|-----------|---------| | "price", "buy", "shop", "cost" | Serper | "iPhone 16 price" | | "near me", "restaurant", "hotel" | Serper | "pizza near me" | | "weather", "news", "latest" | Serper | "weather Berlin" | | "how does", "explain", "what is" | Tavily | "how does TCP work" | | "research", "study", "analyze" | Tavily | "climate research" | | "tutorial", "guide", "learn" | Tavily | "python tutorial" | | "similar to", "companies like" | Exa | "companies like Stripe" | | "startup", "Series A" | Exa | "AI startups Series A" | | "github", "research paper" | Exa | "LLM papers arxiv" | | "private", "anonymous", "no tracking" | SearXNG | "search privately" | | "multiple sources", "aggregate" | SearXNG | "results from all engines" |

Examples

# These are all auto-routed to the optimal provider:
python3 scripts/search.py -q "MacBook Pro M3 price"           # → Serper
python3 scripts/search.py -q "how does HTTPS work"            # → Tavily
python3 scripts/search.py -q "startups like Notion"           # → Exa
python3 scripts/search.py -q "best sushi restaurant near me"  # → Serper
python3 scripts/search.py -q "explain attention mechanism"    # → Tavily
python3 scripts/search.py -q "alternatives to Figma"          # → Exa
python3 scripts/search.py -q "search privately without tracking" # → SearXNG

Result Caching (NEW in v2.7.0!)

Search results are automatically cached for 1 hour to save API costs:

# First request: fetches from API ($)
python3 scripts/search.py -q "AI startups 2024"

# Second request: uses cache (FREE!)
python3 scripts/search.py -q "AI startups 2024"
# Output includes: "cached": true

# Bypass cache (force fresh results)
python3 scripts/search.py -q "AI startups 2024" --no-cache

# View cache stats
python3 scripts/search.py --cache-stats

# Clear all cached results
python3 scripts/search.py --clear-cache

# Custom TTL (in seconds, default: 3600 = 1 hour)
python3 scripts/search.py -q "query" --cache-ttl 7200

Cache location: .cache/ in skill directory (override with WSP_CACHE_DIR environment variable)

Debug Auto-Routing

See exactly why a provider was selected:

python3 scripts/search.py --explain-routing -q "best laptop to buy"

Output:

{
  "query": "best laptop to buy",
  "selected_provider": "serper",
  "reason": "matched_keywords (score=2)",
  "matched_keywords": ["buy", "best"],
  "available_providers": ["serper", "tavily", "exa"]
}

Routing Info in Results

Every search result includes routing information:

{
  "provider": "serper",
  "query": "iPhone 16 price",
  "results": [...],
  "routing": {
    "auto_routed": true,
    "selected_provider": "serper",
    "reason": "matched_keywords (score=1)",
    "matched_keywords": ["price"]
  }
}

---

Configuration Guide

Environment Variables

Create a .env file or set these in your shell:

# Required: Set at least one
export SERPER_API_KEY="your-serper-key"
export TAVILY_API_KEY="your-tavily-key"
export EXA_API_KEY="your-exa-key"

Config File (config.json)

The config.json file lets you customize auto-routing and provider defaults:

{
  "defaults": {
    "provider": "serper",
    "max_results": 5
  },
  
  "auto_routing": {
    "enabled": true,
    "fallback_provider": "serper",
    "provider_priority": ["serper", "tavily", "exa"],
    "disabled_providers": [],
    "keyword_mappings": {
      "serper": ["price", "buy", "shop", "cost", "deal", "near me", "weather"],
      "tavily": ["how does", "explain", "research", "what is", "tutorial"],
      "exa": ["similar to", "companies like", "alternatives", "startup", "github"]
    }
  },
  
  "serper": {
    "country": "us",
    "language": "en"
  },
  
  "tavily": {
    "depth": "basic",
    "topic": "general"
  },
  
  "exa": {
    "type": "neural"
  }
}

Configuration Examples

Example 1: Disable Exa (Only Use Serper + Tavily)

{
  "auto_routing": {
    "disabled_providers": ["exa"]
  }
}

Example 2: Make Tavily the Default

{
  "auto_routing": {
    "fallback_provider": "tavily"
  }
}

Example 3: Add Custom Keywords

{
  "auto_routing": {
    "keyword_mappings": {
      "serper": [
        "price", "buy", "shop", "amazon", "ebay", "walmart",
        "deal", "discount", "coupon", "sale", "cheap"
      ],
      "tavily": [
        "how does", "explain", "research", "what is",
        "coursera", "udemy", "learn", "course", "certification"
      ],
      "exa": [
        "similar to", "companies like", "competitors",
        "YC company", "funded startup", "Series A", "Series B"
      ]
    }
  }
}

Example 4: German Locale for Serper

{
  "serper": {
    "country": "de",
    "language": "de"
  }
}

Example 5: Disable Auto-Routing

{
  "auto_routing": {
    "enabled": false
  },
  "defaults": {
    "provider": "serper"
  }
}

Example 6: Research-Heavy Config

{
  "auto_routing": {
    "fallback_provider": "tavily",
    "provider_priority": ["tavily", "serper", "exa"]
  },
  "tavily": {
    "depth": "advanced",
    "include_raw_content": true
  }
}

---

Provider Deep Dives

Serper (Google Search API)

What it is: Direct access to Google Search results via API — the same results you'd see on google.com.

Strengths

| Strength | Description | |----------|-------------| | 🎯 Accuracy | Google's search quality, knowledge graph, featured snippets | | 🛒 Shopping | Product prices, reviews, shopping results | | 📍 Local | Business listings, maps, places | | 📰 News | Real-time news with Google News integration | | 🖼 Images | Google Images search | | ⚡ Speed | Fastest response times (~200-400ms) |

Best Use Cases

• ✅ Product specifications and comparisons
• ✅ Shopping and price lookups
• ✅ Local business searches ("restaurants near me")
• ✅ Quick factual queries (weather, conversions, definitions)
• ✅ News headlines and current events
• ✅ Image searches
• ✅ When you need "what Google shows"

Getting Your API Key

1. Go to serper.dev
2. Sign up with email or Google
3. Copy your API key from the dashboard
4. Set SERPER_API_KEY environment variable

---

Tavily (Research Search)

What it is: AI-optimized search engine built for research and RAG applications — returns synthesized answers plus full content.

Strengths

| Strength | Description | |----------|-------------| | 📚 Research Quality | Optimized for comprehensive, accurate research | | 💬 AI Answers | Returns synthesized answers, not just links | | 📄 Full Content | Can return complete page content (raw_content) | | 🎯 Domain Filtering | Include/exclude specific domains | | 🔬 Deep Mode | Advanced search for thorough research | | 📰 Topic Modes | Specialized for general vs news content |

Best Use Cases

• ✅ Research questions requiring synthesized answers
• ✅ Academic or technical deep dives
• ✅ When you need actual page content (not just snippets)
• ✅ Multi-source information comparison
• ✅ Domain-specific research (filter to authoritative sources)
• ✅ News research with context
• ✅ RAG/LLM applications

Getting Your API Key

1. Go to tavily.com
2. Sign up and verify email
3. Navigate to API Keys section
4. Generate and copy your key
5. Set TAVILY_API_KEY environment variable

---

Exa (Neural Search)

What it is: Neural/semantic search engine that understands meaning, not just keywords — finds conceptually similar content.

Strengths

| Strength | Description | |----------|-------------| | 🧠 Semantic Understanding | Finds results by meaning, not keywords | | 🔗 Similar Pages | Find pages similar to a reference URL | | 🏢 Company Discovery | Excellent for finding startups, companies | | 📑 Category Filters | Filter by type (company, paper, tweet, etc.) | | 📅 Date Filtering | Precise date range searches | | 🎓 Academic | Great for research papers and technical content |

Best Use Cases

• ✅ Conceptual queries ("companies building X")
• ✅ Finding similar companies or pages
• ✅ Startup and company discovery
• ✅ Research paper discovery
• ✅ Finding GitHub projects
• ✅ Date-filtered searches for recent content
• ✅ When keyword matching fails

Getting Your API Key

1. Go to exa.ai
2. Sign up with email or Google
3. Navigate to API section in dashboard
4. Copy your API key
5. Set EXA_API_KEY environment variable

---

SearXNG (Privacy-First Meta-Search)

What it is: Open-source, self-hosted meta-search engine that aggregates results from 70+ search engines without tracking.

Strengths

| Strength | Description | |----------|-------------| | 🔒 Privacy-First | No tracking, no profiling, no data collection | | 🌐 Multi-Engine | Aggregates Google, Bing, DuckDuckGo, and 70+ more | | 💰 Free | $0 API cost (self-hosted, unlimited queries) | | 🎯 Diverse Results | Get perspectives from multiple search engines | | ⚙ Customizable | Choose which engines to use, SafeSearch, language | | 🏠 Self-Hosted | Full control over your search infrastructure |

Best Use Cases

• ✅ Privacy-sensitive searches (no tracking)
• ✅ When you want diverse results from multiple engines
• ✅ Budget-conscious (no API fees)
• ✅ Self-hosted/air-gapped environments
• ✅ Fallback when paid APIs are rate-limited
• ✅ When "aggregate everything" is the goal

Setting Up Your Instance

# Docker (recommended, 5 minutes)
docker run -d -p 8080:8080 searxng/searxng

# Enable JSON API in settings.yml:
# search:
#   formats: [html, json]

1. See docs.searxng.org
2. Deploy via Docker, pip, or your preferred method
3. Enable JSON format in settings.yml
4. Set SEARXNG_INSTANCE_URL environment variable

---

Usage Examples

Auto-Routed Searches (Recommended)

# Just search — the skill picks the best provider
python3 scripts/search.py -q "Tesla Model 3 price"
python3 scripts/search.py -q "how do neural networks learn"
python3 scripts/search.py -q "YC startups like Stripe"
python3 scripts/search.py -q "search privately without tracking"

Serper Options

# Different search types
python3 scripts/search.py -p serper -q "gaming monitor" --type shopping
python3 scripts/search.py -p serper -q "coffee shop" --type places
python3 scripts/search.py -p serper -q "AI news" --type news

# With time filter
python3 scripts/search.py -p serper -q "OpenAI news" --time-range day

# Include images
python3 scripts/search.py -p serper -q "iPhone 16 Pro" --images

# Different locale
python3 scripts/search.py -p serper -q "Wetter Wien" --country at --language de

Tavily Options

# Deep research mode
python3 scripts/search.py -p tavily -q "quantum computing applications" --depth advanced

# With full page content
python3 scripts/search.py -p tavily -q "transformer architecture" --raw-content

# Domain filtering
python3 scripts/search.py -p tavily -q "AI research" --include-domains arxiv.org nature.com

Exa Options

# Category filtering
python3 scripts/search.py -p exa -q "AI startups Series A" --category company
python3 scripts/search.py -p exa -q "attention mechanism" --category "research paper"

# Date filtering
python3 scripts/search.py -p exa -q "YC companies" --start-date 2024-01-01

# Find similar pages
python3 scripts/search.py -p exa --similar-url "https://stripe.com" --category company

SearXNG Options

# Basic search
python3 scripts/search.py -p searxng -q "linux distros"

# Specific engines only
python3 scripts/search.py -p searxng -q "AI news" --engines "google,bing,duckduckgo"

# SafeSearch (0=off, 1=moderate, 2=strict)
python3 scripts/search.py -p searxng -q "privacy tools" --searxng-safesearch 2

# With time filter
python3 scripts/search.py -p searxng -q "open source projects" --time-range week

# Custom instance URL
python3 scripts/search.py -p searxng -q "test" --searxng-url "http://localhost:8080"

---

Workflow Examples

🛒 Product Research Workflow

# Step 1: Get product specs (auto-routed to Serper)
python3 scripts/search.py -q "MacBook Pro M3 Max specs"

# Step 2: Check prices (auto-routed to Serper)
python3 scripts/search.py -q "MacBook Pro M3 Max price comparison"

# Step 3: In-depth reviews (auto-routed to Tavily)
python3 scripts/search.py -q "detailed MacBook Pro M3 Max review"

📚 Academic Research Workflow

# Step 1: Understand the topic (auto-routed to Tavily)
python3 scripts/search.py -q "explain transformer architecture in deep learning"

# Step 2: Find recent papers (Exa)
python3 scripts/search.py -p exa -q "transformer improvements" --category "research paper" --start-date 2024-01-01

# Step 3: Find implementations (Exa)
python3 scripts/search.py -p exa -q "transformer implementation" --category github

🏢 Competitive Analysis Workflow

# Step 1: Find competitors (auto-routed to Exa)
python3 scripts/search.py -q "companies like Notion"

# Step 2: Find similar products (Exa)
python3 scripts/search.py -p exa --similar-url "https://notion.so" --category company

# Step 3: Deep dive comparison (Tavily)
python3 scripts/search.py -p tavily -q "Notion vs Coda comparison" --depth advanced

---

Optimization Tips

Cost Optimization

| Tip | Savings | |-----|---------| | Use SearXNG for routine queries | $0 API cost | | Use auto-routing (defaults to Serper, cheapest paid) | Best value | | Use Tavily basic before advanced | ~50% cost reduction | | Set appropriate max_results | Linear cost savings | | Use Exa only for semantic queries | Avoid waste |

Performance Optimization

| Tip | Impact | |-----|--------| | Serper is fastest (~200ms) | Use for time-sensitive queries | | Tavily basic faster than advanced | ~2x faster | | Lower max_results = faster response | Linear improvement |

---

FAQ & Troubleshooting

General Questions

Q: Do I need API keys for all three providers?

No. You only need keys for providers you want to use. Auto-routing skips providers without keys.

Q: Which provider should I start with?

Serper — it's the fastest, cheapest, and has the largest free tier (2,500 queries).

Q: Can I use multiple providers in one workflow?

Yes! That's the recommended approach. See Workflow Examples.

Q: How do I reduce API costs?

Use auto-routing (defaults to cheapest), start with lower max_results, use Tavily basic before advanced.

Auto-Routing Questions

Q: Why did my query go to the wrong provider?

Use --explain-routing to debug. Add custom keywords to config.json if needed.

Q: Can I add my own keywords?

Yes! Edit config.json → auto_routing.keyword_mappings.

Q: How does keyword scoring work?

Multi-word phrases get higher weights. "companies like" (2 words) scores higher than "like" (1 word).

Q: What if no keywords match?

Uses the fallback provider (default: Serper).

Q: Can I force a specific provider?

Yes, use -p serper, -p tavily, or -p exa.

Troubleshooting

Error: "Missing API key"

# Check if key is set
echo $SERPER_API_KEY

# Set it
export SERPER_API_KEY="your-key"

Error: "API Error (401)"

Your API key is invalid or expired. Generate a new one.

Error: "API Error (429)"

Rate limited. Wait and retry, or upgrade your plan.

Empty results?

Try a different provider, broaden your query, or remove restrictive filters.

Slow responses?

Reduce max_results, use Tavily basic, or use Serper (fastest).

---

API Reference

Output Format

All providers return unified JSON:

{
  "provider": "serper|tavily|exa",
  "query": "original search query",
  "results": [
    {
      "title": "Page Title",
      "url": "https://example.com/page",
      "snippet": "Content excerpt...",
      "score": 0.95,
      "date": "2024-01-15",
      "raw_content": "Full page content (Tavily only)"
    }
  ],
  "images": ["url1", "url2"],
  "answer": "Synthesized answer",
  "knowledge_graph": { },
  "routing": {
    "auto_routed": true,
    "selected_provider": "serper",
    "reason": "matched_keywords (score=1)",
    "matched_keywords": ["price"]
  }
}

CLI Options Reference

| Option | Providers | Description | |--------|-----------|-------------| | -q, --query | All | Search query | | -p, --provider | All | Provider: auto, serper, tavily, exa, you, searxng | | -n, --max-results | All | Max results (default: 5) | | --auto | All | Force auto-routing | | --explain-routing | All | Debug auto-routing | | --images | Serper, Tavily | Include images | | --country | Serper, You | Country code (default: us) | | --language | Serper, SearXNG | Language code (default: en) | | --type | Serper | search/news/images/videos/places/shopping | | --time-range | Serper, SearXNG | hour/day/week/month/year | | --depth | Tavily | basic/advanced | | --topic | Tavily | general/news | | --raw-content | Tavily | Include full page content | | --exa-type | Exa | neural/keyword | | --category | Exa | company/research paper/news/pdf/github/tweet | | --start-date | Exa | Start date (YYYY-MM-DD) | | --end-date | Exa | End date (YYYY-MM-DD) | | --similar-url | Exa | Find similar pages | | --searxng-url | SearXNG | Instance URL | | --searxng-safesearch | SearXNG | 0=off, 1=moderate, 2=strict | | --engines | SearXNG | Specific engines (google,bing,duckduckgo) | | --categories | SearXNG | Search categories (general,images,news) | | --include-domains | Tavily, Exa | Only these domains | | --exclude-domains | Tavily, Exa | Exclude these domains | | --compact | All | Compact JSON output |

---

License

MIT

---

Links

• Serper — Google Search API
• Tavily — AI Research Search
• Exa — Neural Search
• ClawHub — OpenClaw Skills

File v2.8.4:_meta.json

{ "ownerId": "kn73gpe8xz2630jrknkb3ya96h7zb84h", "slug": "web-search-plus", "version": "2.8.4", "publishedAt": 1771595808196 }

File v2.8.4:CHANGELOG.md

Changelog - Web Search Plus

[2.8.4] - 2026-02-20

🔒 Security Fix: SSRF protection in setup wizard

• Fixed: setup.py SearXNG connection test had no SSRF protection (unlike search.py)
• Before: Operator could be tricked into probing internal networks during setup
• After: Same IP validation as search.py — blocks private IPs, cloud metadata, loopback
• Credit: ClawHub security scanner

[2.8.3] - 2026-02-20

🐛 Critical Fix: Perplexity results empty

• Fixed: Perplexity provider returned 0 results because the AI-synthesized answer wasn't mapped into the results array
• Before: Only extracted URLs from the answer text were returned as results (often 0)
• After: The full answer is now the primary result (title, snippet with cleaned text), extracted source URLs follow as additional results
• Impact: Perplexity queries now always return at least 1 result with the synthesized answer

[2.8.0] - 2026-02-20

🆕 New Provider: Perplexity (AI-Synthesized Answers)

Added Perplexity as the 6th search provider via Kilo Gateway — the first provider that returns direct answers with citations instead of just links:

Features

• AI-Synthesized Answers: Get a complete answer, not a list of links
• Inline Citations: Every claim backed by [1][2][3] source references
• Real-Time Web Search: Perplexity searches the web live, reads pages, and summarizes
• Zero Extra Config: Works through Kilo Gateway with your existing KILOCODE_API_KEY
• Model: perplexity/sonar-pro (best quality, supports complex queries)

Auto-Routing Signals

New direct-answer intent detection routes to Perplexity for:

• Status queries: "status of", "current state of", "what is the status"
• Local info: "events in [city]", "things to do in", "what's happening in"
• Direct questions: "what is", "who is", "when did", "how many"
• Current affairs: "this week", "this weekend", "right now", "today"

Usage Examples

# Auto-routed
python3 scripts/search.py -q "events in Graz Austria this weekend"  # → Perplexity
python3 scripts/search.py -q "what is the current status of Ethereum"  # → Perplexity

# Explicit
python3 scripts/search.py -p perplexity -q "latest AI regulation news"

Configuration

Requires KILOCODE_API_KEY environment variable (Kilo Gateway account). No additional API key needed — Perplexity is accessed through Kilo's unified API.

export KILOCODE_API_KEY="your-kilo-key"

🔧 Routing Rebalance

Major overhaul of the auto-routing confidence scoring to fix Serper dominance:

Problem

Serper (Google) was winning ~90% of queries due to:

• High recency multiplier boosting Serper on any query with dates/years
• Default provider priority placing Serper first in ties
• Research and discovery signals not strong enough to override

Changes

• Lowered Serper recency multiplier — date mentions no longer auto-route to Google
• Strengthened research signals for Tavily:
  • Added: "status of", "what happened with", "how does X compare"
  • Boosted weights for comparison patterns (4.0 → 5.0)
• Strengthened discovery signals for Exa:
  • Added: "events in", "things to do in", "startups similar to"
  • Boosted weights for local discovery patterns
• Updated provider priority order: tavily → exa → perplexity → serper → you → searxng
  • Serper moved from 1st to 4th in tie-breaking
  • Research/discovery providers now win on ambiguous queries

Routing Test Results

| Query | Before | After | ✓ | |-------|--------|-------|---| | "latest OpenClaw version Feb 2026" | Serper | Serper | ✅ | | "Ethereum Pectra upgrade status" | Serper | Tavily | ✅ | | "events in Graz this weekend" | Serper | Perplexity | ✅ | | "compare SearXNG vs Brave for AI agents" | Serper | Tavily | ✅ | | "Sam Altman OpenAI news this week" | Serper | Serper | ✅ | | "find startups similar to Kilo Code" | Serper | Exa | ✅ |

📊 Updated Provider Comparison

| Feature | Serper | Tavily | Exa | Perplexity | You.com | SearXNG | |---------|:------:|:------:|:---:|:----------:|:-------:|:-------:| | Speed | ⚡⚡⚡ | ⚡⚡ | ⚡⚡ | ⚡⚡ | ⚡⚡⚡ | ⚡ | | Direct Answers | ✗ | ✗ | ✗ | ✓✓ | ✗ | ✗ | | Citations | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | | Local Events | ✓ | ✗ | ✓ | ✓✓ | ✗ | ✓ | | Research | ✗ | ✓✓ | ✓ | ✓ | ✓ | ✗ | | Discovery | ✗ | ✗ | ✓✓ | ✗ | ✗ | ✗ | | Self-Hosted | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |

[2.7.0] - 2026-02-14

✨ Added

• Provider cooldown tracking in .cache/provider_health.json
• Exponential cooldown on provider failures: 1m → 5m → 25m → 1h (cap)
• Retry strategy for transient failures (timeout, 429, 503): up to 2 retries with backoff 1s → 3s → 9s
• Smarter cache keys hashed from full request context (query/provider/max_results + locale, freshness, time_range, topic, search_engines, include_news, and related params)
• Cross-provider result deduplication by normalized URL during fallback merge

🔧 Changed

• Cooldown providers are skipped in routing while their cooldown is active
• Provider health is reset automatically after successful requests
• Fallback output now includes dedup metadata:
  • deduplicated: true|false
  • metadata.dedup_count

[2.6.5] - 2026-02-11

🆕 File-Based Result Caching

Added local caching to save API costs on repeated searches:

Features

• Automatic Caching: Search results cached locally by default
• 1-Hour TTL: Results expire after 3600 seconds (configurable)
• Cache Indicators: Response includes cached: true/false and cache_age_seconds
• Zero-Cost Repeats: Cached requests don't hit APIs

New CLI Options

• --cache-ttl SECONDS — Custom cache TTL (default: 3600)
• --no-cache — Bypass cache, always fetch fresh
• --clear-cache — Delete all cached results
• --cache-stats — Show cache statistics (entries, size, age)

Configuration

• Cache directory: .cache/ in skill directory
• Environment variable: WSP_CACHE_DIR to override location
• Cache key: Based on query + provider + max_results (SHA256)

Usage Examples

# First request costs API credits
python3 scripts/search.py -q "AI startups"

# Second request is FREE (uses cache)
python3 scripts/search.py -q "AI startups"

# Force fresh results
python3 scripts/search.py -q "AI startups" --no-cache

# View stats
python3 scripts/search.py --cache-stats

# Clear everything
python3 scripts/search.py --clear-cache

Technical Details

• Cache files: JSON with metadata (_cache_timestamp, _cache_key, etc.)
• Automatic cleanup of expired entries on access
• Graceful handling of corrupted cache files

[2.6.1] - 2026-02-04

• Privacy cleanup: removed hardcoded paths and personal info from docs

[2.5.0] - 2026-02-03

🆕 New Provider: SearXNG (Privacy-First Meta-Search)

Added SearXNG as the 5th search provider, focused on privacy and self-hosted search:

Features

• Privacy-Preserving: No tracking, no profiling — your searches stay private
• Multi-Source Aggregation: Queries 70+ upstream engines (Google, Bing, DuckDuckGo, etc.)
• $0 API Cost: Self-hosted = unlimited queries with no API fees
• Diverse Results: Get perspectives from multiple search engines in one query
• Customizable: Choose which engines to use, set SafeSearch levels, language preferences

Auto-Routing Signals

New privacy/multi-source intent detection routes to SearXNG for:

• Privacy queries: "private", "anonymous", "without tracking", "no tracking"
• Multi-source: "aggregate results", "multiple sources", "diverse perspectives"
• Budget/free: "free search", "no api cost", "self-hosted search"
• German: "privat", "anonym", "ohne tracking", "verschiedene quellen"

Usage Examples

# Auto-routed
python3 scripts/search.py -q "search privately without tracking"  # → SearXNG

# Explicit
python3 scripts/search.py -p searxng -q "linux distros"
python3 scripts/search.py -p searxng -q "AI news" --engines "google,bing,duckduckgo"
python3 scripts/search.py -p searxng -q "privacy tools" --searxng-safesearch 2

Configuration

{
  "searxng": {
    "instance_url": "https://your-instance.example.com",
    "safesearch": 0,
    "engines": null,
    "language": "en"
  }
}

Setup

SearXNG requires a self-hosted instance with JSON format enabled:

# Docker setup (5 minutes)
docker run -d -p 8080:8080 searxng/searxng

# Enable JSON in settings.yml:
# search:
#   formats: [html, json]

# Set instance URL
export SEARXNG_INSTANCE_URL="http://localhost:8080"

See: https://docs.searxng.org/admin/installation.html

📊 Updated Provider Comparison

| Feature | Serper | Tavily | Exa | You.com | SearXNG | |---------|:------:|:------:|:---:|:-------:|:-------:| | Privacy-First | ✗ | ✗ | ✗ | ✗ | ✓✓ | | Self-Hosted | ✗ | ✗ | ✗ | ✗ | ✓ | | API Cost | $$ | $$ | $$ | $ | FREE | | Multi-Engine | ✗ | ✗ | ✗ | ✗ | ✓ (70+) |

🔧 Technical Changes

• Added search_searxng() function with full error handling
• Added PRIVACY_SIGNALS to QueryAnalyzer for auto-routing
• Updated setup wizard with SearXNG option (instance URL validation)
• Updated config.example.json with searxng section
• New CLI args: --searxng-url, --searxng-safesearch, --engines, --categories

---

[2.4.4] - 2026-02-03

📝 Documentation: Provider Count Fix

• Fixed: "You can use 1, 2, or all 3" → "1, 2, 3, or all 4" (we have 4 providers now!)
• Impact: Accurate documentation for setup wizard

[2.4.3] - 2026-02-03

📝 Documentation: Updated README

• Added: "NEW in v2.4.2" badge for You.com in SKILL.md
• Impact: ClawHub README now properly highlights You.com as new feature

[2.4.2] - 2026-02-03

🐛 Critical Fix: You.com API Configuration

• Fixed: Incorrect hostname (api.ydc-index.io → ydc-index.io)
• Fixed: Incorrect header name (X-API-Key → X-API-KEY uppercase)
• Impact: You.com now works correctly - was giving 403 Forbidden before
• Status: ✅ Fully tested and working

[2.4.1] - 2026-02-03

🐛 Bugfix: You.com URL Encoding

• Fixed: URL encoding for You.com queries - spaces and special characters now properly encoded
• Impact: Queries with spaces (e.g., "OpenClaw AI framework") work correctly now
• Technical: Added urllib.parse.quote for parameter encoding

[2.4.0] - 2026-02-03

🆕 New Provider: You.com

Added You.com as the 4th search provider, optimized for RAG applications and real-time information:

Features

• LLM-Ready Snippets: Pre-extracted, query-aware text excerpts perfect for feeding into AI models
• Unified Web + News: Get both web pages and news articles in a single API call
• Live Crawling: Fetch full page content on-demand in Markdown format (--livecrawl)
• Automatic News Classification: Intelligently includes news results based on query intent
• Freshness Controls: Filter by recency (day, week, month, year, or date range)
• SafeSearch Support: Content filtering (off, moderate, strict)

Auto-Routing Signals

New RAG/Real-time intent detection routes to You.com for:

• RAG context queries: "summarize", "key points", "tldr", "context for"
• Real-time info: "latest news", "current status", "right now", "what's happening"
• Information synthesis: "updates on", "situation", "main takeaways"

Usage Examples

# Auto-routed
python3 scripts/search.py -q "summarize key points about AI regulation"  # → You.com

# Explicit
python3 scripts/search.py -p you -q "climate change" --livecrawl all
python3 scripts/search.py -p you -q "tech news" --freshness week

Configuration

{
  "you": {
    "country": "US",
    "language": "en",
    "safesearch": "moderate",
    "include_news": true
  }
}

API Key Setup

export YOU_API_KEY="your-key"  # Get from https://api.you.com

📊 Updated Provider Comparison

| Feature | Serper | Tavily | Exa | You.com | |---------|:------:|:------:|:---:|:-------:| | Speed | ⚡⚡⚡ | ⚡⚡ | ⚡⚡ | ⚡⚡⚡ | | News Integration | ✓ | ✗ | ✗ | ✓ | | RAG-Optimized | ✗ | ✓ | ✗ | ✓✓ | | Full Page Content | ✗ | ✓ | ✓ | ✓ |

---

[2.1.5] - 2026-01-27

📝 Documentation

• Added warning about NOT using Tavily/Serper/Exa in core OpenClaw config
• Core OpenClaw only supports brave as the built-in provider
• This skill's providers must be used via environment variables and scripts, not openclaw.json

[2.1.0] - 2026-01-23

🧠 Intelligent Multi-Signal Routing

Completely overhauled auto-routing with sophisticated query analysis:

Intent Classification

• Shopping Intent: Detects price patterns ("how much", "cost of"), purchase signals ("buy", "order"), deal keywords, and product+brand combinations
• Research Intent: Identifies explanation patterns ("how does", "why does"), analysis signals ("pros and cons", "compare"), learning keywords, and complex multi-clause queries
• Discovery Intent: Recognizes similarity patterns ("similar to", "alternatives"), company discovery signals, URL/domain detection, and academic patterns

Linguistic Pattern Detection

• "How much" / "price of" → Shopping (Serper)
• "How does" / "Why does" / "Explain" → Research (Tavily)
• "Companies like" / "Similar to" / "Alternatives" → Discovery (Exa)
• Product + Brand name combos → Shopping (Serper)
• URLs and domains in query → Similar search (Exa)

Query Analysis Features

• Complexity scoring: Long, multi-clause queries get routed to research providers
• URL detection: Automatic detection of URLs/domains triggers Exa similar search
• Brand recognition: Tech brands (Apple, Samsung, Sony, etc.) with product terms → shopping
• Recency signals: "latest", "2026", "breaking" boost news mode

Confidence Scoring

• HIGH (70-100%): Strong signal match, very reliable routing
• MEDIUM (40-69%): Good match, should work well
• LOW (0-39%): Ambiguous query, using fallback provider
• Confidence based on absolute signal strength + relative margin over alternatives

Enhanced Debug Mode

python3 scripts/search.py --explain-routing -q "your query"

Now shows:

• Routing decision with confidence level
• All provider scores
• Top matched signals with weights
• Query analysis (complexity, URL detection, recency focus)
• All matched patterns per provider

🔧 Technical Changes

QueryAnalyzer Class

New QueryAnalyzer class with:

• SHOPPING_SIGNALS: 25+ weighted patterns for shopping intent
• RESEARCH_SIGNALS: 30+ weighted patterns for research intent
• DISCOVERY_SIGNALS: 20+ weighted patterns for discovery intent
• LOCAL_NEWS_SIGNALS: 25+ patterns for local/news queries
• BRAND_PATTERNS: Tech brand detection regex

Signal Weighting

• Multi-word phrases get higher weights (e.g., "how much" = 4.0 vs "price" = 3.0)
• Strong signals: price patterns (4.0), similarity patterns (5.0), URLs (5.0)
• Medium signals: product terms (2.5), learning keywords (2.5)
• Bonus scoring: Product+brand combo (+3.0), complex query (+2.5)

Improved Output Format

{
  "routing": {
    "auto_routed": true,
    "provider": "serper",
    "confidence": 0.78,
    "confidence_level": "high",
    "reason": "high_confidence_match",
    "top_signals": [{"matched": "price", "weight": 3.0}],
    "scores": {"serper": 7.0, "tavily": 0.0, "exa": 0.0}
  }
}

📚 Documentation Updates

• SKILL.md: Complete rewrite with signal tables and confidence scoring guide
• README.md: Updated with intelligent routing examples and confidence levels
• FAQ: Updated to explain multi-signal analysis

🧪 Test Results

| Query | Provider | Confidence | Signals | |-------|----------|------------|---------| | "how much does iPhone 16 cost" | Serper | 68% | "how much", brand+product | | "how does quantum entanglement work" | Tavily | 86% HIGH | "how does", "what are", "implications" | | "startups similar to Notion" | Exa | 76% HIGH | "similar to", "Series A" | | "companies like stripe.com" | Exa | 100% HIGH | URL detected, "companies like" | | "MacBook Pro M3 specs review" | Serper | 70% HIGH | brand+product, "specs", "review" | | "Tesla" | Serper | 0% LOW | No signals (fallback) | | "arxiv papers on transformers" | Exa | 58% | "arxiv" | | "latest AI news 2026" | Serper | 77% HIGH | "latest", "news", "2026" |

---

[2.0.0] - 2026-01-23

🎉 Major Features

Smart Auto-Routing

• Automatic provider selection based on query analysis
• No need to manually choose provider - just search!
• Intelligent keyword matching for routing decisions
• Pattern detection for query types (shopping, research, discovery)
• Scoring system for provider selection

User Configuration

• config.json: Full control over auto-routing behavior
• Configurable keyword mappings: Add your own routing keywords
• Provider priority: Set tie-breaker order
• Disable providers: Turn off providers you don't have API keys for
• Enable/disable auto-routing: Opt-in or opt-out as needed

Debugging Tools

• --explain-routing flag: See exactly why a provider was selected
• Detailed routing metadata in JSON responses
• Shows matched keywords and routing scores

📚 Documentation

• README.md: Complete auto-routing guide with examples
• SKILL.md: Detailed routing logic and configuration reference
• FAQ section: Common questions about auto-routing
• Configuration examples: Pre-built configs for common use cases

---

[1.0.x] - Initial Release

• Multi-provider search: Serper, Tavily, Exa
• Manual provider selection with -p flag
• Unified JSON output format
• Provider-specific options (--depth, --category, --similar-url, etc.)
• Domain filtering for Tavily/Exa
• Date filtering for Exa

File v2.8.4:FAQ.md

Frequently Asked Questions

Caching (NEW in v2.7.0!)

How does caching work?

Search results are automatically cached locally for 1 hour (3600 seconds). When you make the same query again, you get instant results at $0 API cost. The cache key is based on: query text + provider + max_results.

Where are cached results stored?

In .cache/ directory inside the skill folder by default. Override with WSP_CACHE_DIR environment variable:

export WSP_CACHE_DIR="/path/to/custom/cache"

How do I see cache stats?

python3 scripts/search.py --cache-stats

This shows total entries, size, oldest/newest entries, and breakdown by provider.

How do I clear the cache?

python3 scripts/search.py --clear-cache

Can I change the cache TTL?

Yes! Default is 3600 seconds (1 hour). Set a custom TTL per request:

python3 scripts/search.py -q "query" --cache-ttl 7200  # 2 hours

How do I skip the cache?

Use --no-cache to always fetch fresh results:

python3 scripts/search.py -q "query" --no-cache

How do I know if a result was cached?

The response includes:

• "cached": true/false — whether result came from cache
• "cache_age_seconds": 1234 — how old the cached result is (when cached)

---

General

How does auto-routing decide which provider to use?

Multi-signal analysis scores each provider based on: price patterns, explanation phrases, similarity keywords, URLs, product+brand combos, and query complexity. Highest score wins. Use --explain-routing to see the decision breakdown.

What if it picks the wrong provider?

Override with -p serper/tavily/exa. Check --explain-routing to understand why it chose differently.

What does "low confidence" mean?

Query is ambiguous (e.g., "Tesla" could be cars, stock, or company). Falls back to Serper. Results may vary.

Can I disable a provider?

Yes! In config.json: "disabled_providers": ["exa"]

---

API Keys

Which API keys do I need?

At minimum ONE key (or SearXNG instance). You can use just Serper, just Tavily, just Exa, just You.com, or just SearXNG. Missing keys = that provider is skipped.

Where do I get API keys?

• Serper: https://serper.dev (2,500 free queries, no credit card)
• Tavily: https://tavily.com (1,000 free searches/month)
• Exa: https://exa.ai (1,000 free searches/month)
• You.com: https://api.you.com (Limited free tier for testing)
• SearXNG: Self-hosted, no key needed! https://docs.searxng.org/admin/installation.html

How do I set API keys?

Two options (both auto-load):

Option A: .env file

export SERPER_API_KEY="your-key"

Option B: config.json (v2.2.1+)

{ "serper": { "api_key": "your-key" } }

---

Routing Details

How do I know which provider handled my search?

Check routing.provider in JSON output, or [🔍 Searched with: Provider] in chat responses.

Why does it sometimes choose Serper for research questions?

If the query has brand/product signals (e.g., "how does Tesla FSD work"), shopping intent may outweigh research intent. Override with -p tavily.

What's the confidence threshold?

Default: 0.3 (30%). Below this = low confidence, uses fallback. Adjustable in config.json.

---

You.com Specific

When should I use You.com over other providers?

You.com excels at:

• RAG applications: Pre-extracted snippets ready for LLM consumption
• Real-time information: Current events, breaking news, status updates
• Combined sources: Web + news results in a single API call
• Summarization tasks: "What's the latest on...", "Key points about..."

What's the livecrawl feature?

You.com can fetch full page content on-demand. Use --livecrawl web for web results, --livecrawl news for news articles, or --livecrawl all for both. Content is returned in Markdown format.

Does You.com include news automatically?

Yes! You.com's intelligent classification automatically includes relevant news results when your query has news intent. You can also use --include-news to explicitly enable it.

---

SearXNG Specific

Do I need my own SearXNG instance?

Yes! SearXNG is self-hosted. Most public instances disable the JSON API to prevent bot abuse. You need to run your own instance with JSON format enabled. See: https://docs.searxng.org/admin/installation.html

How do I set up SearXNG?

Docker is the easiest way:

docker run -d -p 8080:8080 searxng/searxng

Then enable JSON in settings.yml:

search:
  formats:
    - html
    - json

Why am I getting "403 Forbidden"?

The JSON API is disabled on your instance. Enable it in settings.yml under search.formats.

What's the API cost for SearXNG?

$0! SearXNG is free and open-source. You only pay for hosting (~$5/month VPS). Unlimited queries.

When should I use SearXNG?

• Privacy-sensitive queries: No tracking, no profiling
• Budget-conscious: $0 API cost
• Diverse results: Aggregates 70+ search engines
• Self-hosted requirements: Full control over your search infrastructure
• Fallback provider: When paid APIs are rate-limited

Can I limit which search engines SearXNG uses?

Yes! Use --engines google,bing,duckduckgo to specify engines, or configure defaults in config.json.

---

Provider Selection

Which provider should I use?

| Query Type | Best Provider | Why | |------------|---------------|-----| | Shopping ("buy laptop", "cheap shoes") | Serper | Google Shopping, price comparisons, local stores | | Research ("how does X work?", "explain Y") | Tavily | Deep research, academic quality, full-page content | | Startups/Papers ("companies like X", "arxiv papers") | Exa | Semantic/neural search, startup discovery | | RAG/Real-time ("summarize latest", "current events") | You.com | LLM-ready snippets, combined web+news | | Privacy ("search without tracking") | SearXNG | No tracking, multi-source, self-hosted |

Tip: Enable auto-routing and let the skill choose automatically! 🎯

Do I need all 5 providers?

No! All providers are optional. You can use:

• 1 provider (e.g., just Serper for everything)
• 2-3 providers (e.g., Serper + You.com for most needs)
• All 5 (maximum flexibility + fallback options)

How much do the APIs cost?

| Provider | Free Tier | Paid Plan | |----------|-----------|-----------| | Serper | 2,500 queries/mo | $50/mo (5,000 queries) | | Tavily | 1,000 queries/mo | $150/mo (10,000 queries) | | Exa | 1,000 queries/mo | $1,000/mo (100,000 queries) | | You.com | Limited free | ~$10/mo (varies by usage) | | SearXNG | FREE ✅ | Only VPS cost (~$5/mo if self-hosting) |

Budget tip: Use SearXNG as primary + others as fallback for specialized queries!

How private is SearXNG really?

| Setup | Privacy Level | |-------|---------------| | Self-hosted (your VPS) | ⭐⭐⭐⭐⭐ You control everything | | Self-hosted (Docker local) | ⭐⭐⭐⭐⭐ Fully private | | Public instance | ⭐⭐⭐ Depends on operator's logging policy |

Best practice: Self-host if privacy is critical.

Which provider has the best results?

| Metric | Winner | |--------|--------| | Most accurate for facts | Serper (Google) | | Best for research depth | Tavily | | Best for semantic queries | Exa | | Best for RAG/AI context | You.com | | Most diverse sources | SearXNG (70+ engines) | | Most private | SearXNG (self-hosted) |

Recommendation: Enable multiple providers + auto-routing for best overall experience.

How does auto-routing work?

The skill analyzes your query for keywords and patterns:

"buy cheap laptop"     → Serper (shopping signals)
"how does AI work?"    → Tavily (research/explanation)
"companies like X"     → Exa (semantic/similar)
"summarize latest news" → You.com (RAG/real-time)
"search privately"     → SearXNG (privacy signals)

Confidence threshold: Only routes if confidence > 30%. Otherwise uses default provider.

Override: Use -p provider to force a specific provider.

---

Production Use

Can I use this in production?

Yes! Web-search-plus is production-ready:

• ✅ Error handling with automatic fallback
• ✅ Rate limit protection
• ✅ Timeout handling (30s per provider)
• ✅ API key security (.env + config.json gitignored)
• ✅ 5 providers for redundancy

Tip: Monitor API usage to avoid exceeding free tiers!

What if I run out of API credits?

1. Fallback chain: Other enabled providers automatically take over
2. Use SearXNG: Switch to self-hosted (unlimited queries)
3. Upgrade plan: Paid tiers have higher limits
4. Rate limit: Use disabled_providers to skip exhausted APIs temporarily

---

Updates

How do I update to the latest version?

Via ClawHub (recommended):

clawhub update web-search-plus --registry "https://www.clawhub.ai" --no-input

Manually:

cd /path/to/workspace/skills/web-search-plus/
git pull origin main
python3 scripts/setup.py  # Re-run to configure new features

Where can I report bugs or request features?

• GitHub Issues: https://github.com/robbyczgw-cla/web-search-plus/issues
• ClawHub: https://www.clawhub.ai/skills/web-search-plus

File v2.8.4:TROUBLESHOOTING.md

Troubleshooting Guide

Caching Issues (v2.7.0+)

Cache not working / always fetching fresh

Symptoms:

• Every request hits the API
• "cached": false even for repeated queries

Solutions:

1. Check cache directory exists and is writable:

   ls -la .cache/  # Should exist in skill directory
   

2. Verify --no-cache isn't being passed
3. Check disk space isn't full
4. Ensure query is EXACTLY the same (including provider and max_results)

Stale results from cache

Symptoms:

• Getting outdated information
• Cache TTL seems too long

Solutions:

1. Use --no-cache to force fresh results
2. Reduce TTL: --cache-ttl 1800 (30 minutes)
3. Clear cache: python3 scripts/search.py --clear-cache

Cache growing too large

Symptoms:

• Disk space filling up
• Many .json files in .cache/

Solutions:

1. Clear cache periodically:

   python3 scripts/search.py --clear-cache
   

2. Set up a cron job to clear weekly
3. Use a smaller TTL so entries expire faster

"Permission denied" when caching

Symptoms:

• Cache write errors in stderr
• Searches work but don't cache

Solutions:

1. Check directory permissions: chmod 755 .cache/
2. Use custom cache dir: export WSP_CACHE_DIR="/tmp/wsp-cache"

---

Common Issues

"No API key found" error

Symptoms:

Error: No API key found for serper

Solutions:

1. Check .env exists in skill folder with export VAR=value format
2. Keys auto-load from skill's .env since v2.2.0
3. Or set in system environment: export SERPER_API_KEY="..."
4. Verify key format in config.json:

   { "serper": { "api_key": "your-key" } }
   

Priority order: config.json > .env > environment variable

---

Getting empty results

Symptoms:

• Search returns no results
• "results": [] in JSON output

Solutions:

1. Check API key is valid (try the provider's web dashboard)
2. Try a different provider with -p
3. Some queries have no results (very niche topics)
4. Check if provider is rate-limited
5. Verify internet connectivity

Debug:

python3 scripts/search.py -q "test query" --verbose

---

Rate limited

Symptoms:

Error: 429 Too Many Requests
Error: Rate limit exceeded

Good news: Since v2.2.5, automatic fallback kicks in! If one provider hits rate limits, the script automatically tries the next provider.

Solutions:

1. Wait for rate limit to reset (usually 1 hour or end of day)
2. Use a different provider: -p tavily instead of -p serper
3. Check free tier limits:
   • Serper: 2,500 free total
   • Tavily: 1,000/month free
   • Exa: 1,000/month free
4. Upgrade to paid tier for higher limits
5. Use SearXNG (self-hosted, unlimited)

Fallback info: Response will include routing.fallback_used: true when fallback was used.

---

SearXNG: "403 Forbidden"

Symptoms:

Error: 403 Forbidden
Error: JSON format not allowed

Cause: Most public SearXNG instances disable JSON API to prevent bot abuse.

Solution: Self-host your own instance:

docker run -d -p 8080:8080 searxng/searxng

Then enable JSON in settings.yml:

search:
  formats:
    - html
    - json  # Add this!

Restart the container and update your config:

{
  "searxng": {
    "instance_url": "http://localhost:8080"
  }
}

---

SearXNG: Slow responses

Symptoms:

• SearXNG takes 2-5 seconds
• Other providers are faster

Explanation: This is expected behavior. SearXNG queries 70+ upstream engines in parallel, which takes longer than direct API calls.

Trade-off: Slower but privacy-preserving + multi-source + $0 cost.

Solutions:

1. Accept the trade-off for privacy benefits
2. Limit engines for faster results:

   python3 scripts/search.py -p searxng -q "query" --engines "google,bing"
   

3. Use SearXNG as fallback (put last in priority list)

---

Auto-routing picks wrong provider

Symptoms:

• Query about research goes to Serper
• Query about shopping goes to Tavily

Debug:

python3 scripts/search.py --explain-routing -q "your query"

This shows the full analysis:

{
  "query": "how much does iPhone 16 Pro cost",
  "routing_decision": {
    "provider": "serper",
    "confidence": 0.68,
    "reason": "moderate_confidence_match"
  },
  "scores": {"serper": 7.0, "tavily": 0.0, "exa": 0.0},
  "top_signals": [
    {"matched": "how much", "weight": 4.0},
    {"matched": "brand + product detected", "weight": 3.0}
  ]
}

Solutions:

1. Override with explicit provider: -p tavily
2. Rephrase query to be more explicit about intent
3. Adjust confidence_threshold in config.json (default: 0.3)

---

Config not loading

Symptoms:

• Changes to config.json not applied
• Using default values instead

Solutions:

1. Check JSON syntax (use a validator)
2. Ensure file is in skill directory: /path/to/skills/web-search-plus/config.json
3. Check file permissions
4. Run setup wizard to regenerate:

   python3 scripts/setup.py --reset
   

Validate JSON:

python3 -m json.tool config.json

---

Python dependencies missing

Symptoms:

ModuleNotFoundError: No module named 'requests'

Solution:

pip3 install requests

Or install all dependencies:

pip3 install -r requirements.txt

---

Timeout errors

Symptoms:

Error: Request timeout after 30s

Causes:

• Slow network connection
• Provider API issues
• SearXNG instance overloaded

Solutions:

1. Try again (temporary issue)
2. Switch provider: -p serper
3. Check your internet connection
4. If using SearXNG, check instance health

---

Duplicate results

Symptoms:

• Same result appears multiple times
• Results overlap between providers

Solution: This is expected when using auto-fallback or multiple providers. The skill doesn't deduplicate across providers.

For single-provider results:

python3 scripts/search.py -p serper -q "query"

---

Debug Mode

For detailed debugging:

# Verbose output
python3 scripts/search.py -q "query" --verbose

# Show routing decision
python3 scripts/search.py -q "query" --explain-routing

# Dry run (no actual search)
python3 scripts/search.py -q "query" --dry-run

# Test specific provider
python3 scripts/search.py -p tavily -q "query" --verbose

---

Getting Help

Still stuck?

1. Check the full documentation in README.md
2. Run the setup wizard: python3 scripts/setup.py
3. Review FAQ.md for common questions
4. Open an issue: https://github.com/robbyczgw-cla/web-search-plus/issues

File v2.8.4:config.example.json

{ "$schema": "https://json-schema.org/draft/2020-12/schema", "$comment": "Web Search Plus configuration — intelligent routing and provider settings", "defaults": { "provider": "serper", "max_results": 5 }, "auto_routing": { "enabled": true, "fallback_provider": "serper", "provider_priority": [ "serper", "tavily", "exa", "you", "searxng" ], "disabled_providers": [], "confidence_threshold": 0.3, "keyword_mappings": { "serper": [ "price", "buy", "shop", "shopping", "cost