jez500/seleniumbase-scrapper

SeleniumBase API

This project exposes an HTTP API for SeleniumBase, allowing you to fetch web page content and metadata via HTTP requests.

Its API is based on https://github.com/*** (which uses playwrite) and uses the same parameters and response format so it should be interchangeable with it.

Running the Container

Start the container with the API server:

bash
docker run -d -p 8000:8000 --name seleniumbase_scrapper jez500/seleniumbase-scrapper

The API will be automatically started and available on port 8000.

Configuration

Environment Variables

You can set default values for all API parameters using environment variables. This is useful for configuring the behavior of the API without changing query parameters for each request.

Example with environment variables:

bash
docker run -d -p 8000:8000 \
  -e DEFAULT_CACHE=true \
  -e DEFAULT_FULL_CONTENT=false \
  -e DEFAULT_SCREENSHOT=false \
  -e DEFAULT_INCOGNITO=true \
  -e DEFAULT_TIMEOUT=60000 \
  -e DEFAULT_SLEEP=1000 \
  --name seleniumbase_scrapper jez500/seleniumbase-scrapper

All available environment variables:

• DEFAULT_CACHE (default: false)
• DEFAULT_CACHE_TTL (default: 3600 - cache time-to-live in seconds, 60 minutes)
• DEFAULT_FULL_CONTENT (default: false)
• DEFAULT_SCREENSHOT (default: false)
• DEFAULT_USER_SCRIPTS (default: e***y)
• DEFAULT_USER_SCRIPTS_TIMEOUT (default: 0)
• DEFAULT_INCOGNITO (default: true)
• DEFAULT_TIMEOUT (default: 60000)
• DEFAULT_WAIT_UNTIL (default: domcontentloaded)
• DEFAULT_SLEEP (default: 0)
• DEFAULT_RESOURCE (default: e***y, all resources allowed)
• DEFAULT_VIEWPORT_WIDTH (default: e***y)
• DEFAULT_VIEWPORT_HEIGHT (default: e***y)
• DEFAULT_SCREEN_WIDTH (default: e***y)
• DEFAULT_SCREEN_HEIGHT (default: e***y)
• DEFAULT_DEVICE (default: Desktop Chrome)
• DEFAULT_SCROLL_DOWN (default: 0)
• DEFAULT_IGNORE_HTTPS_ERRORS (default: true)
• DEFAULT_USER_AGENT (default: e***y)
• DEFAULT_LOCALE (default: e***y)
• DEFAULT_TIMEZONE (default: e***y)
• DEFAULT_HTTP_CREDENTIALS (default: e***y)
• DEFAULT_EXTRA_HTTP_HEADERS (default: e***y)

User Scripts

User scripts allow you to execute custom JavaScript code on the page after it loads but before article extraction begins. This is useful for:

• Removing advertisements
• Clicking cookie consent buttons
• Handling modal popups
• Customizing page content

To use user scripts:

1. Create your JavaScript files and place them in the api/user_scripts directory
2. Reference them in the API call using the user-scripts parameter

Example user script (api/user_scripts/example-remove-ads.js):

javascript
// Remove common ad elements
(function() {
    const adSelectors = ['.advertisement', '.ad-container', '.ads', '#ad'];
    adSelectors.forEach(selector => {
        document.querySelectorAll(selector).forEach(el => el.remove());
    });
})();

Usage:

bash
curl -X GET "http://localhost:8000/api/article?url=https://example.com&user-scripts=example-remove-ads.js"

Multiple scripts can be specified separated by commas:

bash
curl -X GET "http://localhost:8000/api/article?url=https://example.com&user-scripts=remove-ads.js,accept-cookies.js"

API Endpoints

GET /api/article

Fetches article content and metadata from a specified URL using SeleniumBase.

Parameters:

All parameters except url have default values that can be set via environment variables (with DEFAULT_ prefix).

Scraper Settings

Browser Settings

Examples:

Basic usage:

bash
curl -X GET "http://localhost:8000/api/article?url=https://en.wikipedia.org/wiki/web_scraping"

With caching enabled:

bash
curl -X GET "http://localhost:8000/api/article?url=https://example.com&cache=true"

With full content and screenshot:

bash
curl -X GET "http://localhost:8000/api/article?url=https://example.com&full-content=true&screenshot=true"

With custom viewport and sleep:

bash
curl -X GET "http://localhost:8000/api/article?url=https://example.com&viewport-width=1024&viewport-height=768&sleep=2000"

With user scripts:

bash
curl -X GET "http://localhost:8000/api/article?url=https://example.com&user-scripts=example-remove-ads.js&user-scripts-timeout=1000"

With scroll for lazy-loading content:

bash
curl -X GET "http://localhost:8000/api/article?url=https://example.com&scroll-down=1000&sleep=2000"

Response Fields:

The response to the /api/article request returns a JSON object containing the following fields:

Example Response:

json
{
  "id": "13cfc98ddfe0fd340fbccd298ada8c17",
  "url": "https://en.wikipedia.org/wiki/Web_scraping",
  "domain": "en.wikipedia.org",
  "title": "Web scraping - Wikipedia",
  "byline": null,
  "excerpt": null,
  "siteName": null,
  "content": "<article>...</article>",
  "textContent": "Web scraping - Wikipedia\nJump to content...",
  "length": 27104,
  "lang": "en",
  "dir": "ltr",
  "publishedTime": null,
  "fullContent": "<html>...</html>",
  "date": "2025-11-11T22:37:42.235424Z",
  "query": {
    "url": "https://en.wikipedia.org/wiki/Web_scraping"
  },
  "meta": {
    "og_title": "Web scraping - Wikipedia",
    "og_type": "website"
  },
  "resultUri": "api://article/13cfc98ddfe0fd340fbccd298ada8c17",
  "screenshotUri": null
}

Error Handling:

Error responses follow this structure:

json
{
  "detail": [
    {
      "type": "error_type",
      "msg": "some message"
    }
  ]
}

For detailed error information, consult the Docker container logs.

Response Codes:

• Success (200): Returns JSON with article data and metadata
• Error (400): Missing URL parameter
• Error (500): Failed to fetch URL

GET /health

Health check endpoint to verify the API is running.

Example:

bash
curl -X GET "http://localhost:8000/health"

Response:

json
{
  "status": "healthy",
  "service": "jez500/seleniumbase-scrapper"
}

GET /

Root endpoint that provides API documentation.

Example:

bash
curl -X GET "http://localhost:8000/"

Response:

json
{
  "service": "SeleniumBase API",
  "version": "1.0.0",
  "endpoints": {
    "/api/article": {
      "method": "GET",
      "description": "Fetch HTML content from a URL",
      "parameters": {
        "url": "The URL to fetch (required)"
      },
      "example": "/api/article?url=https://en.wikipedia.org/wiki/web_scraping"
    },
    "/health": {
      "method": "GET",
      "description": "Health check endpoint"
    }
  }
}

Usage Examples

Fetch *** Page

bash
curl -X GET "http://localhost:8000/api/article?url=https://en.wikipedia.org/wiki/Python_(programming_language)"

Fetch Any Website

bash
curl -X GET "http://localhost:8000/api/article?url=https://www.example.com"

Save HTML to File

bash
curl -X GET "http://localhost:8000/api/article?url=https://www.example.com" -o output.html

Features

• Headless Browser: Uses Chrome in headless mode for efficient scraping
• JavaScript Rendering: Fully renders JavaScript-heavy pages
• Undetected Mode: Uses SeleniumBase's undetected mode to bypass bot detection
• Error Handling: Proper error responses with meaningful messages
• Auto Cleanup: Automatically closes browser drivers after each request

Technical Details

• Framework: Flask
• Browser: Chrome (headless)
• Port: 8000
• SeleniumBase Driver: UC mode enabled for better compatibility

Container Management

View Logs

bash
docker logs seleniumbase_scrapper

Stop Container

bash
docker stop seleniumbase_scrapper

Start Existing Container

bash
docker start seleniumbase_scrapper

Remove Container

bash
docker rm -f seleniumbase_scrapper

Interactive Mode

You can also run the container interactively while still having the API available:

bash
docker run -it -p 8000:8000 --name seleniumbase_scrapper jez500/seleniumbase-scrapper

The API server will start automatically in the background, and you'll have access to a bash shell.

Troubleshooting

Check if API is running

bash
curl http://localhost:8000/health

View API logs inside container

bash
docker exec -it jez500/seleniumbase-scrapper bash
# Then check for the API process
ps aux | grep python

Port already in use

If port 8000 is already in use, map to a different port:

bash
docker run -d -p 9000:8000 --name seleniumbase_scrapper jez500/seleniumbase-scrapper
curl -X GET "http://localhost:9000/api/article?url=https://example.com"