vimagick/scrapyd

scrapyd

https://github.com/easypi/docker-scrapyd/actions/workflows/build.yaml/badge.svg](https://github.com/EasyPi/docker-scrapyd)

https://github.com/scrapy/scrapy 是一个开源协作框架，用于以快速、简单且可扩展的方式从网站提取所需数据。

https://github.com/scrapy/scrapyd 是一个运行Scrapy爬虫的服务，允许通过HTTP JSON API部署Scrapy项目并控制爬虫。

https://github.com/scrapy/scrapyd-client 是scrapyd的客户端，提供scrapyd-deploy工具，用于将项目部署到Scrapyd服务器。

https://github.com/scrapinghub/scrapy-splash 通过Splash实现Scrapy与JavaScript的集成。

https://github.com/scrapinghub/scrapyrt 允许为现有Scrapy项目轻松添加HTTP API。

https://github.com/scrapinghub/spidermon 是构建Scrapy爬虫监控器的框架。

https://github.com/scrapinghub/scrapy-poet 是Web-Poet页面对象模式在Scrapy中的实现。

此镜像基于 debian:bullseye，安装了七个最新稳定版Python包：

• scrapy==2.8.0
• scrapyd==1.4.1
• scrapyd-client==v1.2.3
• scrapy-splash==latest
• scrapyrt==v0.13
• spidermon==1.17.1
• scrapy-poet==0.10.1

请将此镜像用作您自己项目的基础镜像。

:warning: Scrapy（自 2.0.0 起）已放弃对Python 2.7的支持，Python 2.7已于2020-01-01停止维护。

docker-compose.yml

yaml
version: "3.8"

services:

  scrapyd:
    image: vimagick/scrapyd
    ports:
      - "6800:6800"
    volumes:
      - ./data:/var/lib/scrapyd
      - /usr/local/lib/python3.9/dist-packages
    restart: unless-stopped

  scrapy:
    image: vimagick/scrapyd
    command: bash
    volumes:
      - .:/code
    working_dir: /code
    restart: unless-stopped

  scrapyrt:
    image: vimagick/scrapyd
    command: scrapyrt -i 0.0.0.0 -p 9080
    ports:
      - "9080:9080"
    volumes:
      - .:/code
    working_dir: /code
    restart: unless-stopped

以后台守护进程方式运行scrapyd

bash
$ docker-compose up -d scrapyd
$ docker-compose logs -f scrapyd
$ docker cp scrapyd_scrapyd_1:/var/lib/scrapyd/items .
$ tree items
└── myproject
    └── myspider
        └── ad6153ee5b0711e68bc70242ac110005.jl

bash
$ mkvirtualenv -p python3 webbot
$ pip install scrapy scrapyd-client

$ scrapy startproject myproject
$ cd myproject
$ setvirtualenvproject

$ scrapy genspider myspider mydomain.com
$ scrapy edit myspider
$ scrapy list

$ vi scrapy.cfg
$ scrapyd-client deploy
$ curl http://localhost:6800/schedule.json -d project=myproject -d spider=myspider
$ firefox http://localhost:6800

文件：scrapy.cfg

ini
[settings]
default = myproject.settings

[deploy]
url = http://localhost:6800/
project = myproject

以交互式shell方式运行scrapy

bash
$ cat > stackoverflow_spider.py << _EOF_
import scrapy

class StackOverflowSpider(scrapy.Spider):
    name = 'stackoverflow'
    start_urls = ['http://stackoverflow.com/questions?sort=votes']

    def parse(self, response):
        for href in response.css('.question-summary h3 a::attr(href)'):
            full_url = response.urljoin(href.extract())
            yield scrapy.Request(full_url, callback=self.parse_question)

    def parse_question(self, response):
        yield {
            'title': response.css('h1 a::text').extract()[0],
            'votes': response.css('.question div[itemprop="upvoteCount"]::text').extract()[0],
            'body': response.css('.question .postcell').extract()[0],
            'tags': response.css('.question .post-tag::text').extract(),
            'link': response.url,
        }
_EOF_

$ docker-compose run --rm scrapy
>>> scrapy runspider stackoverflow_spider.py -o top-stackoverflow-questions.jl
>>> cat top-stackoverflow-questions.jl
>>> exit

以实时爬虫方式运行scrapyrt

bash
$ git clone https://github.com/scrapy/quotesbot.git .
$ docker-compose up -d scrapyrt
$ curl -s 'http://localhost:9080/crawl.json?spider_name=toscrape-css&callback=parse&url=http://quotes.toscrape.com/&max_requests=5' | jq -c '.items[]'