SKILL: skill-web-scraper

Extract anything. Understand everything.
Powered by Jnana (Sanskrit: ज्ञान — knowledge/wisdom)
By Darshj.me

Overview

skill-web-scraper is a production-grade intelligent web scraping skill for OpenClaw. It exposes the Jnana extraction engine — a pipeline that combines CSS selectors, Zod schema validation, and optional LLM post-processing to extract typed, structured data from any web page.

Tagline: Extract anything. Understand everything.

Capabilities

| Capability | Description | |---|---| | extract(url, schema) | Extract structured data from a single URL using CSS selectors + Zod validation | | crawl(url, schema) | Follow pagination/next-page links, depth-limited BFS crawl | | batch(urls, schema) | Concurrent extraction from multiple URLs with configurable parallelism | | Article extractor | Title, author, date, body, tags, image — works on most blogs/news sites | | Product extractor | Name, price, currency, SKU, availability, rating — broad e-commerce coverage | | Table extractor | Parse HTML tables into typed {headers, rows} structures | | Link extractor | All <a> links, resolved, filtered, with rel/title attributes | | Contact extractor | Emails (regex), phones, addresses, social profile links | | Anti-detection | Rotating user agents, stealth headers, per-domain rate limiting, robots.txt compliance | | JS rendering | Optional Playwright integration for SPAs and JS-heavy sites | | Output formats | JSON, CSV, Markdown, plain text | | Caching | ETag + Last-Modified aware — skip unchanged pages | | LLM fallback | When CSS selectors miss, pass page text to an LLM for extraction |

Installation

npm install skill-web-scraper
# Optional: for JS rendering
npm install playwright
npx playwright install chromium

Quick Start

import { WebScraper, articleExtractor } from 'skill-web-scraper';

const scraper = new WebScraper();

const result = await scraper.extract({
  url: 'https://example.com/article',
  schema: articleExtractor(),
});

console.log(result.data.title);    // "Article Title"
console.log(result.data.body);     // "Full article text..."
console.log(result.fromCache);     // false (first fetch)

OpenClaw Integration

// In your OpenClaw skill handler
import { WebScraper, articleExtractor, productExtractor } from 'skill-web-scraper';

export async function handleScrapeRequest(input: { url: string; type: string }) {
  const scraper = new WebScraper({
    antiDetection: { minDelayMs: 1000, maxDelayMs: 3000 },
    llmComplete: async (prompt) => {
      // Hook into your LLM provider
      return await callYourLLM(prompt);
    },
  });

  const schema = input.type === 'product' ? productExtractor() : articleExtractor();
  const result = await scraper.extract({ url: input.url, schema });
  return result.data;
}

API Reference

WebScraper

Main class. Instantiate once and reuse across requests.

const scraper = new WebScraper(config?: WebScraperConfig);

Config options:

| Option | Type | Default | Description | |---|---|---|---| | antiDetection | AntiDetectionConfig | {} | Rate limiting, user agents, robots.txt | | cache | CacheStore | MemoryCacheStore | Custom cache backend | | cacheEnabled | boolean | true | Enable/disable caching | | defaultRenderMode | 'static' \| 'playwright' | 'static' | Default render mode | | llmComplete | (prompt: string) => Promise<string> | undefined | LLM fallback for extraction |

scraper.extract(options)

const result: ExtractResult<T> = await scraper.extract({
  url: string,
  schema: ExtractionSchema<T>,
  renderMode?: 'static' | 'playwright',
  requestOptions?: RequestOptions,
  outputFormat?: 'json' | 'csv' | 'markdown' | 'text',
});

scraper.crawl(options)

const result: CrawlResult<T> = await scraper.crawl({
  startUrls: string | string[],
  schema: ExtractionSchema<T>,
  nextPageSelector?: string,   // CSS selector for next-page link
  maxPages?: number,           // default: 50
  maxDepth?: number,           // default: 3
  sameDomainOnly?: boolean,    // default: true
  renderMode?: 'static' | 'playwright',
  onPage?: (result, pageIndex) => void,
});

scraper.batch(options)

const result: BatchResult<T> = await scraper.batch({
  urls: string[],
  schema: ExtractionSchema<T>,
  concurrency?: number,        // default: 3
  renderMode?: 'static' | 'playwright',
});

Built-in Extractors

articleExtractor()

Extracts: title, author, publishedAt, body, summary, tags[], imageUrl, canonicalUrl

productExtractor()

Extracts: name, price, currency, description, sku, availability, imageUrl, rating, reviewCount

extractLinks(html, baseUrl?)

Returns: LinkData[] — { href, text, title?, rel? }

extractContacts(html)

Returns: ContactData — { emails[], phones[], addresses[], socialLinks{} }

extractTables(html)

Returns: TableData[] — { headers[], rows[][] }

Custom Schema

import { WebScraper } from 'skill-web-scraper';
import { z } from 'zod';

const scraper = new WebScraper();

const result = await scraper.extract({
  url: 'https://site.com/page',
  schema: {
    selectors: {
      title: 'h1',
      price: { selector: '.price', transform: (v) => parseFloat(v.replace('$', '')) },
      images: { selector: 'img', attr: 'src', multiple: true },
    },
    schema: z.object({
      title: z.string(),
      price: z.number(),
      images: z.array(z.string()),
    }),
    llmPrompt: 'Extract title, price (as number), and all image URLs from this page.',
  },
});

Anti-Detection Guide

The Jnana engine uses a layered anti-detection approach:

1. User-Agent rotation — 15-agent pool covering Chrome, Firefox, Safari, Edge, mobile browsers
2. Stealth headers — Sec-Fetch-*, Accept, Accept-Language, Accept-Encoding exactly mirroring browser behaviour
3. Randomized delays — Per-domain jitter between configurable min/max bounds
4. robots.txt compliance — Fetched and cached per-domain; configurable (respectRobotsTxt: false to disable)
5. Conditional requests — ETag/Last-Modified; sends 304-aware headers to avoid re-fetching unchanged pages
6. Playwright stealth — When using JS rendering, headers are passed directly into the browser context

Configuration

const scraper = new WebScraper({
  antiDetection: {
    rotateUserAgents: true,    // Enable UA rotation (default: true)
    minDelayMs: 1000,          // Min delay per domain (default: 1000ms)
    maxDelayMs: 3000,          // Max delay per domain (default: 3000ms)
    respectRobotsTxt: true,    // Respect robots.txt (default: true)
    extraHeaders: {            // Additional custom headers
      'X-Forwarded-For': '203.0.113.0',
    },
  },
});

Environment

• Node.js: ≥ 18.0.0 (uses native Fetch API)
• TypeScript: 5.x (strict mode)
• ESM only
• Optional: Playwright 1.x for JS rendering

Ethical Use Notice

This skill is built for legitimate data extraction use cases: research, monitoring, archiving, and integration. Always:

• Respect robots.txt (enabled by default)
• Keep rate limits reasonable (1–3 second delays by default)
• Check a site's Terms of Service before scraping
• Never use for unauthorized access, PII harvesting, or harmful purposes

skill-web-scraper — By Darshj.me | MIT License
Jnana engine: Extract anything. Understand everything.