capyfile/capycmd

capycmd - 文件处理命令行应用Docker镜像

镜像概述

capycmd是一款用于文件处理的命令行工具Docker镜像，可实现文件读取、时间验证、S3兼容存储上传及旧文件清理等自动化任务，适用于日志归档、旧文件管理等场景。

核心功能

• 从文件系统读取指定路径的文件
• 验证文件修改时间是否满足预设条件
• 将符合要求的文件上传至S3兼容存储服务
• 删除处理完成且无错误的本地文件

使用场景

• 日志文件定期归档：自动上传超过指定天数的日志文件至存储服务并清理本地副本
• 旧数据迁移：批量将本地旧文件迁移至云存储以释放空间
• 自动化文件管理：按自定义规则处理本地文件，提升运维效率

配置说明

需准备JSON格式的服务定义文件，描述文件处理流程。示例定义如下：

json
{
  "version": "1.0",
  "name": "logs",
  "processors": [
    {
      "name": "archive",
      "operations": [
        {
          "name": "filesystem_input_read",
          "params": {
            "target": {
              "sourceType": "value",
              "source": "/var/log/rotated-logs/*"
            }
          }
        },
        {
          "name": "file_time_validate",
          "params": {
            "maxMtime": {
              "sourceType": "env_var",
              "source": "MAX_LOG_FILE_AGE_RFC3339"
            }
          }
        },
        {
          "name": "s3_upload",
          "targetFiles": "without_errors",
          "params": {
            "accessKeyId": {
              "sourceType": "secret",
              "source": "aws_access_key_id"
            },
            "secretAccessKey": {
              "sourceType": "secret",
              "source": "aws_secret_access_key"
            },
            "endpoint": {
              "sourceType": "value",
              "source": "s3.amazonaws.com"
            },
            "region": {
              "sourceType": "value",
              "source": "us-east-1"
            },
            "bucket": {
              "sourceType": "env_var",
              "source": "AWS_LOGS_BUCKET"
            }
          }
        },
        {
          "name": "filesystem_input_remove",
          "targetFiles": "without_errors"
        }
      ]
    }
  ]
}

Docker部署示例

拉取镜像

docker pull docker.xuanyuan.run/capyfile/capycmd:<TAG>

可用标签：

• latest：最新稳定版本
• <version>：特定版本（如1.0.0）
• <major>-latest：最新主版本（如1-latest）

运行镜像

准备服务定义文件后，执行以下命令启动处理流程：

docker run \
    --name capyfile_server \
    --mount type=bind,source=./service-definition.json,target=/etc/capyfile/service-definition.json \
    --mount type=bind,source=/var/log/rotated-logs,target=/var/log/rotated-logs \
    --env CAPYFILE_SERVICE_DEFINITION_FILE=/etc/capyfile/service-definition.json \
    --env MAX_LOG_FILE_AGE_RFC3339=$(date -d "30 days ago" -u +"%Y-%m-%dT%H:%M:%SZ") \
    --env AWS_LOGS_BUCKET=logs \
    --secret aws_access_key_id \
    --secret aws_secret_access_key \
    capyfile/capycmd:latest logs:archive

输出示例

运行后将输出处理过程日志，示例如下：

Running logs:archive service processor...

[/var/log/rotated-logs/access-2023-08-27.log] filesystem_input_read FINISHED file read finished
[/var/log/rotated-logs/access-2023-08-28.log] filesystem_input_read FINISHED file read finished
[/var/log/rotated-logs/access-2023-09-27.log] filesystem_input_read FINISHED file read finished
...
[/var/log/rotated-logs/access-2023-08-27.log] s3_upload FINISHED S3 file upload has finished
[/var/log/rotated-logs/access-2023-08-27.log] filesystem_input_remove FINISHED file remove finished
...

附加信息

capycmd的源码及贡献指南可在GitHub查看：https://github.com/capyfile/capyfile