widepath/phpfpm Docker 镜像 - 轩辕镜像

airwp

App Containers & Deployments

• app: The WidePath web application (signup, create/delete sites, etc)
• caddy-proxy: Proxy direct to app and to php nodes
• nodesftpd: Multi-node sftp server
• worker: Backend processes via worker queue (site create/delete, backup, password reset, etc.)
• vector: Sends event data to logtail from kube

Phpnode Containers

• caddy-phpnode: phpnode local http server, passes requests to specific phpfpm instance
• phpfpm: Container for phpfpm daemon
• purger: Php node daemon that forwards purge requests to CDN (deprecated since StackPath retirement)
• smtp-relay: An smtp relay daemon that manages a quota of outbound mail per site
• vector: Sends event data to logtail

Other

• app-cli: Control the web app from the command line
• deploy: Tools, scripts, schemes for running all components in dev or production environments
• purger-plugin: WP plugin that posts requests to purger to perform cache invalidation on site edits

Build a Ubuntu dev node on Mac using Canonical Multipass

• brew install multipass
• multipass launch docker --name=docker
• multipass exec docker -- sudo bash
• Install the public key for ansible_user into ./.ssh/authorized_keys
• systemctl edit docker
  • add this at the top (need to explicitly clear ExecStart):

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 --containerd=/run/containerd/containerd.sock

• Get inside as root: multipass exec docker -- sudo bash
• systemctl restart docker
• docker login
• docker pull widepath/phpfpm:release-0.0.98
• docker network create widepath_localbackend
• apt install s3cmd zip
• docker run -d --name=mysql -e MYSQL_ROOT_PASSWORD=root --network widepath_localbackend -p 3306:3306 mariadb:10.4
• Then run worker like this: ansible_user=root ansible_pass= MYSQL_ROOT_PASSWORD=root ansible_port=22 WP_UID_FILES=1001 WP_GID_FILES=1001 nodemon
• Also add the node's IP to an entry in mongo nodes collection

Dev Mode

• On mac, to get Docker Desktop to expose the Docker API, run this container: docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p 127.0.0.1:2375:2375 bobrik/socat TCP-LISTEN:2375,fork UNIX-CONNECT:/var/run/docker.sock
• Run rastasheep/ubuntu-sshd to emulate the filesystem of a phpnode and sshd on a custom port
• Have to install sshpass to allow ansible to connect the rastasheep/ubuntu-sshd container over ssh using passwords`

How to Run Scripts Against Production

• Port forward mongo on local
• Run portal-meteor app
• In another shell run meteor shell
• Now at the prompt, run .load <physical path to script>
• Output will be in the first windows (the meteor-portal one)

Build for DigitalOcean

• docker build --platform=linux/amd64 -t <image-name>:<version> .
• Important to add the --platform=linux/amd64

Keys

• worker-ansible-phpnode.key

  • Generate this RSA private key
  • Use ansible to install the key on all phpnodes
  • Add the key as a secret in kube
  • Mount the key secret in worker

• Private key generated on app nodes then authorized on php nodes - this is so the sftpd container can mount the key and ssh to the php nodes

• Private keys generated on computation nodes then authorized on php nodes - this is so the worker container can ansible to php nodes in order to do backups, etc

• Install ssh key (just copy paste to ~/.ssh/id_rsa) to bastion host

• ./deploy/kube/secret-nodesftpd-phpnodes.yml and ./deploy/kube/secret-worker-ansible-phpnodes.yml are the keys that grant nodesftpd and worker to communicate with phpnodes.

• To copy the regcred secret to the kube-logging namespace: kubectl get secret regcred --namespace=default -o yaml | grep -v '^\s*namespace:\s' | kubectl apply --namespace=kube-logging -f -

• Mailgun SMTP credentials for *** are used for

  • MAIL_URL in production kube widepath-secret-prod
  • Phpnode smtp-relay daemon SMTP_OUTBOUND_PASSWORD in deploy/ansible/phpnode-service/smtp-relay.yml
  • monitor

Mongo & mgob

• Dump: mongodump --db=dbname --gzip --archive=wp.gz
• Restore: mongorestore --gzip --archive=wp.gz

DigitalOcean

• Create bastion manually and make sure to install ssh private key in ~/.ssh/id_rsa
• DO_API_TOKEN is required as an environment variable to make the dynamic inventory work
• Ensure to create tags before launching else errors will occur: doctl compute tag create phpnode
• launch-inventory is static and represents the base instances. This means, run the launch.yml playbook like this: ansible-playbook -i launch-inventory launch.yml
• To get ssh key ids: curl -X GET -H 'Content-Type: application/json' -H 'Authorization: Bearer $DO_API_TOKEN' "[***]" or doctl compute ssh-key list

Provision Additional PHP nodes

• Set the API key: export DO_AUTH_TOKEN=abc123
• cd ./deploy/ansible
• Add the host to the ./deploy/ansible/phpnode-provision/launch-inventory-php file
• ansible-playbook -i phpnode-provision/launch-inventory-php phpnode-provision/launch-php.yml
• The new node should have been created, now provision required packages and config on the new node:
• ansible-playbook phpnode-provision/droplet-basics.yml --limit "phpXX"
• ansible-playbook phpnode-provision/docker.yml --limit "phpXX"
• ansible-playbook phpnode-provision/php.yml --limit "phpXX"
• ansible-playbook phpnode-provision/php-packages-outside-docker.yml --limit "phpXX"
• ansible-playbook phpnode-provision/python.yml --limit "phpXX"
• ansible-playbook phpnode-provision/reinstall-docker-sdk.yml --limit "phpXX"
• ansible-playbook phpnode-provision/block-storage.yml --limit "phpXX"

Install key to new node

• If you need to generate a public key from the private key: ssh-keygen -y -f worker-ansible-phpnode.key > worker-ansible-phpnode.pub
• ansible-playbook phpnode-provision/keys.yml --limit "phpXX"

Create phpnode services

• ansible-playbook phpnode-service/caddy-phpnode.yml --limit "phpXX"
• ansible-playbook phpnode-service/vector/vector-phpnode.yml --limit "phpXX"
• ansible-playbook phpnode-service/mysql.yml --limit "phpXX"
• ansible-playbook phpnode-service/purger.yml --limit "phpXX"
• ansible-playbook phpnode-service/smtp-relay.yml --limit "phpXX"

Install node in app

• Now add the node ip and name to app: node app-cli/app-cli node add phpXX <private_ip> "websitesca" "docker-standalone"

Add DNS Entry in Cloudflare for new node

• A phpXX.tor1.airwp.com >> public_ip

Migrate a Node

• src=phpX1
• dst=phpX2
• candidates=$(echo "show databases" | mysql -u root -p -h $src | grep -Ev "^(Database|mysql|performance_schema|information_schema)$")
• mysqldump --databases $candidates -u root -p -h $src > all.sql
• MYSQL_CONN="-uroot -p -h $src"
• mysql ${MYSQL_CONN} --skip-column-names -A -e"SELECT CONCAT('SHOW GRANTS FOR ''',user,'''@''',host,''';') FROM mysql.user WHERE user<>''" | mysql ${MYSQL_CONN} --skip-column-names -A | sed 's/$/;/g' > grants.sql
• Edit grants and remove weird stuff
• mysql -u root -p -h $dst < all.sql
• mysql -u root -p -h $dst < grants.sql
• rsync -chavzP --stats $src:/mnt/airwp/sites/ ./sites
• rmdir sites/_trash
• rsync -chavzP --stats ./sites/* $dst:/mnt/airwp/sites
• app-cli site list --nodename=$src | jq "[.[].siteStub]" > ~/Desktop/nodes/$src.json
• cat ~/Desktop/nodes/$src.json | python deploy/script/node-change-all.js $dst

Ansible Setup

• There are 2 files to setup:
• ~/.ansible/airwp.cfg (get from ./deploy/ansible.cfg.sample)
• ~/.ansible/airwp-ssh.cfg (get from ./deploy/ssh.cfg.sample)
• Then export ANSIBLE_CONFIG=~/.ansible/airwp.cfg
• Should be able to run ansible-playbook in any folder and it will find the inventory
• Note that the inventory requires the DO_API_TOKEN environment variable

NODESFTPD

Sftpd server for WidePath.

Keys and Auth

Keys need to be generated in a specific format for use with the ssh2 module.

• ssh-keygen -m PEM -t rsa -f /path/to/wp_key

This will generate /path/to/wp_key (the private key) and /path/to/wp_key.pub (the public key)

Can use ssh-copy-id to authorize the public key on the destination server or use the ansible modules authorized_key.

The format seems to be RSA PKCS#8 ASN1 with a containing public key RSA X.509 ASN1

Production

The app nodes need to be able to ssh to the php nodes. This means we can generate keys on the app nodes then authorize those keys on the php nodes.

Dev

In dev, we just authorize a key from main to php1.

Migrate by Node

From jump node:

• export source=php6
• export dest=php10
• candidates=$(echo "show databases" | mysql -u root -p -h $source | grep -Ev "^(Database|mysql|performance_schema|information_schema)$")
• mysqldump --databases $candidates -u root -p -h $source > all.sql
• export MYSQL_CONN="-uroot -p -h $source
• mysql ${MYSQL_CONN} --skip-column-names -A -e"SELECT CONCAT('SHOW GRANTS FOR ''',user,'''@''',host,''';') FROM mysql.user WHERE user<>''" | mysql ${MYSQL_CONN} --skip-column-names -A | sed 's/$/;/g' > grants.sql

Edit grants and remove weird stuff

• mysql -u root -p -h $dest < all.sql
• mysql -u root -p -h $dest < grants.sql
• rsync -chavzP --stats $source:/mnt/airwp/sites/ ./sites
• rmdir sites/_trash
• rsync -chavzP --stats ./sites/* $dest:/mnt/airwp/sites
• app-cli site list --nodename=$source | jq "[.[].siteStub]" > ~/Desktop/nodes/$source.json
• cat ~/Desktop/nodes/$source.json | python deploy/script/node-change-all.js $dest

Clean up

• chown -R wp:wp *
• find . -type f -exec chmod 644 {} \;
• find . -type d -exec chmod 755 {} \;
• find . -regex '.*.archive.zip' -exec rm {} \;

Misc

wpcli tricks and shortcuts

• To add the Formidable Pro API key: wp option add frmpro-credentials 'a:1:{s:7:\"license\";s:23:\"BDN68-YZF5W-GZHKY-YOJJL\";}'
• To add the Akismet API key: wp option add wordpress_api_key 92f0f8e7bee3

Other stuff:

wp role reset --all
wp user set-role websitesca administrator
wp user update websitesca --nickname=websitesca
sudo -u \#$WP_UID_FILES wp plugin uninstall hello
sudo -u \#$WP_UID_FILES wp theme uninstall twentyfifteen twentysixteen

See memory usage of a container: kubectl exec -it app-d86fcdd6c-lwp9d cat /sys/fs/cgroup/memory/memory.usage_in_bytes

Test fcgi on the command line

• See: [***]
• Install this: apt-get install libfcgi0ldbl
• Then run like this:

Clean up backups that get stuck pending

• List them all: db.sites.find({backups:{$elemMatch:{'status.pending':true}}}).map(x=>x.siteStub)
• Delete them all: db.sites.update({}, {$pull:{backups:{'status.pending':true}}}, {multi:true})

DigitalOcean Spaces Lifecycle Policies

• Must set BucketLifecycle rules via api (can't even use doctl).
• Set this using scripts/update_do_spaces_widepath_lifecycle.py

To curl or httpie direct to an origin (ie bypass stackpath):

• httpie: http --headers [***] Host:<sitestub>.widepath.app X-OriginalHost:<hostname> X-SiteStub:<sitestub> --verify=no
• curl: curl -I [***] -H "X-OriginalHost: <host>" -H "X-SiteStub: <sitestub>" -H "Host: <sitestub>.widepath.app" -k