Nutch Dockerfile

!Docker Pulls !Docker Image Size (latest by date) !Docker Image Version (latest semver) !Docker Stars !Docker Automated build

快速使用Docker搭建Nutch服务。

什么是Nutch？

!Nutch logo

Apache Nutch是一个高度可扩展、可伸缩的开源网络爬虫软件项目。

Nutch可以在单机上运行，但在Hadoop集群中运行时能充分发挥其优势。

Docker镜像

当前镜像配置包含以下组件：

• Nutch 1.x（"master"分支）

基础镜像

• alpine:3.19

提示

如果遇到类似以下错误，可能需要将docker命令别名设置为"docker --tls"：

2015/04/07 09:19:56 Post [***] malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"

最简单的方法：

1. alias docker="docker --tls"

安装

1. 安装Docker。

2. 从当前目录的文件构建：

有三种构建模式，可通过--build-arg BUILD_MODE=0标志激活。以下为默认值：

• 0 == 安装Nutch master分支源码，crawl和nutch脚本位于$PATH中
• 1 == 与模式0相同，额外添加Nutch REST服务器；额外构建参数--build-arg SERVER_PORT=8081和--build-arg SERVER_HOST=0.0.0.0
• 2 == 与模式1相同，额外添加Nutch Web应用；额外构建参数--build-arg WEBAPP_PORT=8080

例如，如果要安装Nutch master分支并同时运行Nutch REST服务器和Web应用，运行以下命令：

$(boot2docker shellinit | grep export) #可能不需要
docker build -t apache/nutch . --build-arg BUILD_MODE=2 --build-arg SERVER_PORT=8081 --build-arg SERVER_HOST=0.0.0.0 --build-arg WEBAPP_PORT=8080

使用方法

如果尚未运行Docker，启动Docker：

boot2docker up
$(boot2docker shellinit | grep export)

运行容器：

docker run -t -i -d -p 8080:8080 -p 8081:8081 --name nutchcontainer apache/nutch
c5401810e50a606f43256b4b24602443508bd9badcf2b7493bd97839834571fc

查看容器日志：

docker logs c5401810e50a606f43256b4b24602443508bd9badcf2b7493bd97839834571fc
2021-06-29 19:14:32,922 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2021-06-29 19:14:32,925 INFO supervisord started with pid 1
2021-06-29 19:14:33,929 INFO spawned: 'nutchserver' with pid 8
2021-06-29 19:14:33,932 INFO spawned: 'nutchwebapp' with pid 9
2021-06-29 19:14:36,012 INFO success: nutchserver entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2021-06-29 19:14:36,012 INFO success: nutchwebapp entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)

现在可以通过http://localhost:8080访问Web应用，并与REST API交互，例如：

curl http://localhost:8080/admin
{"startDate":1625118207995,"configuration":["default"],"jobs":[],"runningJobs":[]}

进入容器：

docker exec -it c5401810e50a606f43256b4b24602443508bd9badcf2b7493bd97839834571fc /bin/bash

查看supervisord日志：

cat /tmp/supervisord.log
2021-06-29 19:14:32,922 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2021-06-29 19:14:32,925 INFO supervisord started with pid 1
2021-06-29 19:14:33,929 INFO spawned: 'nutchserver' with pid 8
2021-06-29 19:14:33,932 INFO spawned: 'nutchwebapp' with pid 9
2021-06-29 19:14:36,012 INFO success: nutchserver entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2021-06-29 19:14:36,012 INFO success: nutchwebapp entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)

查看supervisord子进程日志：

ls /var/log/supervisord/
nutchserver_stderr.log  nutchserver_stdout.log  nutchwebapp_stderr.log  nutchwebapp_stdout.log

Nutch位于$NUTCH_HOME，几乎可以运行。 需要设置种子URL，并在$NUTCH_HOME/conf/nutch-site.xml中更新http.agent.name配置属性，设置爬虫的代理名称。 有关更多“入门”信息，请查看Nutch教程。