wires160/archivebox-proxy

ArchiveBox Proxy

A proxy that saves navigated URLs to https://github.com/ArchiveBox/ArchiveBox, implemented with https://github.com/mitmproxy/mitmproxy.

✨ Originally developed by https://github.com/brunocek/archivebox-proxy

This container is designed to be used with docker compose to build an ArchiveBox application that can monitor traffic through a mitmproxy proxy and automatically add visited pages to the ArchiveBox archive.

To allow archivebox-proxy to access ArchiveBox running in a separate container, we expose the docker socket. This may have security implications but seems to be the best way to communicate between containers at this time. ArchiveBox v0.9 may include an HTTP API that would make this method obsolete.

This project is intended to meet ArchiveBox's ticket 557: https://github.com/ArchiveBox/ArchiveBox/issues/577. The main challenge is to serve ios, as ios does not allow firefox plugins to be installed.

Installation

Follow the https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#docker-compose for installing a docker compose application. First create the folder where you plan to save your ArchiveBox archive. This will be a folder on your host computer (e.g., /archivebox) and you will also create a data folder (e.g., /archivebox/data). Download docker-compose.yml into the main ArchiveBox folder, not the data directory.

You will need two edits of the base docker-compose.yml file. First, add a container name to the archivebox service. This allows archivebox-proxy to identify the container running the main archivebox service.

    archivebox:
        image: archivebox/archivebox:latest
        container_name: archivebox_main
        ports:
            - 8000:8000

Next, you will add a service description for the archivebox-proxy service:

    ### This runs the archivebox-proxy built on a mitmproxy image
    # https://github.com/ArchiveBox/archivebox-proxy

    archivebox-proxy:
        image: wires160/archivebox-proxy:latest
        ports:
            - 8080:8080
        volumes:
            - ./data/.mitmproxy:/home/mitmproxy/.mitmproxy
            - ./data/.mitmproxy:/root/.mitmproxy
            - /var/run/docker.sock:/var/run/docker.sock
            - ./data:/data

Note that you can assign a different external port number to the service to avoid conflicts with other services on your server.

Run all docker compose commands from within the archivebox directory containing your docker-compose.yml file. First start the application and initialize the archive collection:

sudo docker compose run archivebox init

Next, start the full application:

sudo docker compose up

On the first run, archivebox-proxy copies its configuration file to the data directory where it can be edited as described below.

To test the proxy, open an HTTP only website like http://mitm.it which is a built-in magic site. See below instructions for configuring HTTPS proxying.

Configuration

ArchiveBox

See https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration regarding configuration changes. These can be made using sudo docker compose run archivebox config or by directly editing ArchiveBox.conf in the ./data directory. *** changing which types of archives are saved (e.g., singlefile, PDF screenshot, Chrome screenshot, wget archive, etc.) and the USER_AGENT that is sent since some sites block automated or headless requests.

Configuring an HTTPS client

HTTPS will not work out of the box since mitmproxy works by signing connections with its own certificate that is not trusted by your client until you install the certificate. Open http://mitm.it and follow the instructions for your platform to install and trust the certificate. This should only need to be done once since the container stores the mitmproxy keys in the ./data directory that is persistent.

Archivebox-proxy Modes

The config file ( config-archivebox-proxy.yaml ) holds a parameter for mode that can be: record, or archive.

The reason for two modes is explained in the section below Identifying User HTTP Requests - not trivial.

On record mode, archivebox-proxy will record all the navigation on record.yaml file, and the user will need to latter on manually run archivebox add record.yaml. The user may edit the file with vim and remove some of the lines ( dd ) with URLs not for archiving.

On archive mode, archivebox-proxy will run archivebox add to each of the identified URLs. Please read section Identifying User HTTP requests - not trivial bellow, before using this mode.

Troubleshooting

While moving my application from a test server to my production server, I encountered a permissions error with the archivebox-proxy container. When this container starts, docker-entrypoint.sh atte***s to change the UID of the mitmproxy user account using usermod so that it matches the UID that owns the mitmproxy configuration files. Since I have bound /root/.mitmproxy to our data directory, the container needs a user with UID that matches the UID that owns the files on your server. *** creating an archivebox user:group on the server to fill this role. The ArchiveBox container defaults archivebox:archivebox to 911:911.

Comments

Identifying User HTTP Requests - not trivial

When developing this proxy, I came across research papers trying to solve the open problem of identifying User Actions in a HTTP Flow. It is not a trivial problem to solve as you can attest in the article bellow.

At that time (2016), some of the evidence on HTTP flows:

"..a single request for the Huffington Post website results in the download of 408 objects from 113 unique domains. A similar analysis by Butkiewicz et al. [4], of 1,700 popular websites showed that the median landing page consists of at least 40 objects, requested from 10 or more servers, most of which are operated by third-party services."

"..Here, the pool of starting pages is randomly selected from the top-1,000 most popular webpages according to alexa.com, excluding HTTPS pages and Chinese websites (using non-Roman script). HTTPS pages were omitted to allow fair head-to-head comparison. On average, each trace of 500 page requests resulted in 29,506 HTTP requests, distributed over 14.168 connections."

As of today, 2024, traffic is HTTPS, but this problem still exists.

I implemented filters based on the authors insights and these can be tweaked by changing the float constants __time_window_next and __reset_timer in the script. (I may externalise them to the config file if users demand to constantly tinker with it).

Some more filters may be in place:

• mitmdump filter expressions, specially '!~a', that filters out webpage assets. For more on these: ([***] )
• commercial s such as i and *** filter adverts
• projects such as https://github.com/arkenfox/user.js
• firefox hardening such as https://github.com/arkenfox/user.js

With all these filters running, I still get a lot of URLs that are not user action. More research must be invested on this. I count with your help on the issues forum.

Georgios Rizothanasis; Niklas Carlsson; Aniket Mahanti Identifying User Actions from HTTP(S) Traffic IEEE, 2016 ( [***] )

historic

2024-01 Bruno Schroeder kick-starts and asks for contribution with the architectural decisions, and delivers a script for mitmproxy. 2025-08 Wires160 changes: - Fixed subprocess not imported error - Added functionality to allow configuration file changes to be persistent by moving the base file to ./data - Changed archive process call to use docker socket to communicate with the ArchiveBox container

ios alternative solution

For each tab:

1. Hit share, and share it to iMarkdown or Obsidian
2. Obsidian asks which file to append to - one may have one file per tag/subject
3. ios appends the url there (but sometimes it appends the page title and work must be re-done)
4. Tab must be closed