harisekhon/kafka Docker 镜像 - 轩辕镜像 | Docker 镜像高效稳定拉取服务
harisekhon/kafka自动构建
harisekhon
Apache Kafka (tags 0.8 - 0.10 with scala 2.10 - 2.11)
5 次收藏下载次数: 0状态：自动构建维护者：harisekhon仓库类型：镜像最近更新：5 年前
Big Data, NoSQL & DevOps Docker Images
- NoSQL datastores (HBase, Cassandra, Riak, SolrCloud)
- OS & development images (Alpine, CentOS, Debian, Fedora, Ubuntu)
- DevOps tech, anything cool and open source (RabbitMQ Cluster, Mesos, Consul)
- My GitHub repos containing hundreds of tools related to these technologies with all dependencies pre-built
These images are all available pre-built on My DockerHub - [***]
Quality and Testing - this repo has entire test suites run against it from various GitHub repositories to validate the docker images' functionality, branches vs tagged versions align, latest contains correct version from master branch, syntax checks covering all common build and file formats (Make/JSON/CSV/INI/XML/YAML configurations) etc.
These are reusable tests that can anybody can implement and can be found in my DevOps Python Tools and DevOps Bash Tools repos as well as the Advanced Nagios Plugins Collection which contains hundreds of technology specific API-level test programs to ensure the docker images are functioning as intended.
Continuous Integration in run on this and adjacent repos that form a bi-directional validation between these docker images and several other repositories full of hundreds of programs. All of this is intended to keep the quality of this repo as high as possible.
Hari Sekhon
Cloud & Big Data Contractor, United Kingdom
(ex-Cloudera, former Hortonworks Consultant)
[***]
(you're welcome to connect with me on LinkedIn)
Ready to run Docker images
docker search harisekhon docker run harisekhon/nagios-plugins
To see more than the 25 DockerHub repos limited by docker search (docker issue 23055) I wrote dockerhub_search.py using the DockerHub API, available in my DevOps Python Tools github repo and as a pre-built docker image:
docker run harisekhon/pytools dockerhub_search.py harisekhon
There are lots of tagged versions of official software in my repos to allow development testing across multiple versions, usually more versions than available from the official repos (and new version updates available on request, just raise a GitHub issue).
DockerHub tags are not shown by docker search (docker issue ***) so I wrote dockerhub_show_tags.py available in my DevOps Python Tools github repo and as a pre-built docker image - eg. to see an organized list of all CentOS tags:
docker run harisekhon/pytools dockerhub_show_tags.py centos
For service technologies like Hadoop, HBase, ZooKeeper etc for which you'll also want port mappings, each directory in the GitHub project contains both a standard docker-compose configuration as well as a make run shortcut (which doesn't require docker-compose to be installed) - either way you don't have to remember all the command line switches and port number specifics:
cd zookeeper docker-compose up
or for technologies with interactive shells like Spark, ZooKeeper, HBase, Drill, Cassandra where you want to be dropped in to an interactive shell, use the make run shortcut instead:
cd zookeeper make run
which is much easier to type and remember than the equivalent bigger commands like:
docker run -ti -p 2181:2181 harisekhon/zookeeper
and avoid this for more complex services like Hadoop / HBase:
docker run -ti -p 2181:2181 -p 8080:8080 -p 8085:8085 -p 9090:9090 -p 9095:9095 -p ***:*** -p ***:*** -p ***:*** -p ***:*** harisekhon/hbase
docker run -ti -p 8020:8020 -p 8032:8032 -p 8088:8088 -p 9000:9000 -p ***:*** -p ***:*** -p 50010:50010 -p 50020:50020 -p 50070:50070 -p 50075:50075 -p 50090:50090 harisekhon/hadoop
Full Inventory:
Official Standard Open Source Technologies:
More specific information can be found in the readme page under each respective directory in the Dockerfiles git repo.
- Alluxio - distributed in-memory filesystem for cluster computing frameworks by UC Berkely's AMPLab - readme
- Apache Drill - distributed SQL engine by MapR (opens Drill SQL shell) - readme
- Awless - a Mighty CLI for AWS - readme
- Cassandra - distributed NoSQL datastore by *** and DataStax (opens CQL shell, bundled with nagios-plugins)
- Consul - distributed service discovery by HashiCorp
- FakeS3 - Amazon S3 API simulator for testing without incurring AWS S3 costs - readme
- H2O - distributed machine learning framework by 0xdata
- Hadoop (HDFS + Yarn) - distributed storage and compute cluster by ***, Cloudera and Hortonworks - readme
- HBase - distributed NoSQL datastore by *** (opens HBase shell) - readme
- Jython - Python on Java JVM (useful for Hadoop python utilities using Hadoop's Java API. eg. DevOps Python Tools)
- Kafka - pub-sub data broker by LinkedIn and Confluent. Deprecated, see new Confluent docker images instead
- Mesos - datacenter resource manager by Mesosphere (mostly obsoleted by more free Hortonworks / Hadoop Yarn resource manager)
- Nifi - IOT data flow engine by NSA and Hortonworks
- OpenTSDB TCollector - metrics collector - sends metrics to OpenTSDB - readme
- Presto - distributed SQL engine by *** (opens Presto SQL shell) - readme
- Presto (Teradata distribution) - Teradata's Presto distribution including ODBC and JDBC drivers (opens Presto SQL shell) - readme
- RabbitMQ Cluster - pub-sub message queue broker by Pivotal (extension of RabbitMQ official image with added plugins)
- Riak KV - distributed NoSQL datastore by Basho
- Riak KV (bundled with nagios-plugins)
- Serf - decentralized cluster coordination engine by HashiCorp
- Solr - mature indexing engine built on Lucene search library
- SolrCloud - clustered distributed indexing engine version of Solr
- Spark - fast distributed cluster compute engine usually used on Hadoop, by UC Berkely's AMPLab and Databricks (opens Spark shell)
- Superset - data visualization by Airbnb
- Tachyon (Alluxio < 1.0) - distributed in-memory filesystem for cluster computing frameworks by UC Berkely's AMPLab
- ZooKeeper (opens ZK shell) - distributed coordination and sychronization service by ***
Repos suffixed with -dev are the official technologies + development & debugging tools + my github repos with all dependencies pre-built.
My GitHub Repos (with all libs + deps pre-built):
-
Advanced Nagios Plugins Collection - 450+ nagios plugins for every Hadoop distribution and every major NoSQL technology - Hadoop, Redis, Elasticsearch, Solr, HBase, Cassandra & DataStax OpsCenter, MongoDB, MySQL, Kafka, Riak, Memcached, Couchbase, CouchDB, Mesos, Spark, Neo4j, Datameer, H2O, WanDisco, Yarn, HDFS, Impala, Apache Drill, Presto, ZooKeeper, Cloudera, Hortonworks, MapR, IBM BigInsights, Infrastructure - Linux, DNS, Whois, SSL Certs etc
harisekhon/nagios-plugins:centos/harisekhon/nagios-plugins:latestharisekhon/nagios-plugins:alpineharisekhon/nagios-plugins:fedoraharisekhon/nagios-plugins:debianharisekhon/nagios-plugins:ubuntuharisekhon/nagios-plugins:perlharisekhon/nagios-plugins:python
-
harisekhon/tools- DevOps Tools superset of the below images, containing hundreds of programs:harisekhon/pytools- DevOps Python Tools - 80+ DevOps CLI tools tools for AWS, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, Ambari, Blueprints, CloudFormation, Elasticsearch, Solr, Pig etc.harisekhon/bash-tools- DevOps Bash Tools - 100+ DevOps CLI tools for AWS, GCP, Kubernetes, Hadoop, Hive, Impala, Kafka, Docker, LDAP, Git, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodiesharisekhon/perl-tools- DevOps Perl Tools - 25+ DevOps CLI Tools - Log Anonymizer, Hadoop HDFS & Hive tools, Solr/SolrCloud CLI, SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, ***, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Linux, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Ambari FreeIPA Kerberos, Datameer etc.- all of the above repos come with tags for
alpine,centos,debian,fedoraandubuntubuilds
-
Spotify Tools - Backup & Play Automation: Spotify Lookup - converts Spotify URIs to 'Artist - Track' form by querying the Spotify Metadata API. Spotify Cmd - command line control of Spotify on Mac via AppleScript for automation, auto timed track flick through etc.
-
all Github repos pre-built on major Linux distros with CLI programs located at
/github/<project>harisekhon/github:centos/harisekhon/github:latestharisekhon/github:alpineharisekhon/github:debianharisekhon/github:fedoraharisekhon/github:ubuntu
Base Images:
Dev images:
harisekhon/dev:latest=harisekhon/dev:centosharisekhon/dev:centos/ - CentOS latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect, EPEL etc.harisekhon/dev:alpine- Alpine latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect etc.harisekhon/dev:debian- Debian latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect etc.harisekhon/dev:fedora- Fedora latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect etc.harisekhon/dev:ubuntu- Ubuntu latest with Java JDK, Perl, Python, Jython, Ruby, Scala, Groovy, GCC, Maven, SBT, Gradle, Make, Expect etc.
Base Images of Java / Scala:
All builds use OpenJDK with jre and jdk numbered tags. See this article below for why it might be illegal to bundle Oracle Java (and why no Linux distributions do this either):
[***]
harisekhon/alpine-java- Alpine latest with Java 8harisekhon/centos-java- CentOS latest combinations of Java 7 / 8 and Scala 2.10 / 2.11harisekhon/debian-java- Debian latest with Java 7, 8harisekhon/fedora-java- Fedora latest combinations of Java 7/8 and Scala 2.10/2.11harisekhon/ubuntu-java- Ubuntu 14.04 with Java 7
- Ubuntu latest with Java 8, 9
Build from Source
All images come pre-built on DockerHub but if you want to compile from source for any reason such as developing improvements, I've made this easy to do:
git clone [***] cd Dockerfiles
To build all Docker images, just run the make command at the top level:
make
To build a specific Docker image, enter its directory and run make:
cd nagios-plugins make
You can also build a specific version by checking out the git branch for the version and running the build:
cd consul git checkout consul-0.9 make
or build all versions of a given software project like so:
cd hadoop make build-versions
See the top level Makefile as well as the Makefile.in which is sourced per project with any project specific overrides in the <project_directory>/Makefile.
Support
Please raise tickets for issues and improvements at [***]
See Also:
-
DevOps Python Tools - 80+ DevOps CLI tools for AWS, Hadoop, HBase, Spark, Log Anonymizer, Ambari Blueprints, AWS CloudFormation, Linux, Docker, Spark Data Converters & Validators (Avro / Parquet / JSON / CSV / INI / XML / YAML), Elasticsearch, Solr, Travis CI, Pig, IPython
-
The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)
-
DevOps Bash Tools - 100+ DevOps Bash scripts, advanced
.bashrc,.vimrc,.screenrc,.tmux.conf,.toprc, Utility Code Library used by CI and all my GitHub repos - includes code for AWS, Kubernetes, Kafka, Docker, Git, Code & build linting, package management for Linux / Mac / Perl / Python / Ruby / Golang, and lots more random goodies -
DevOps Perl Tools - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, ***, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...
-
HAProxy-configs - 80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, Cloudera, Hortonworks, MapR, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, ZooKeeper, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, SSH, RabbitMQ, Redis, Riak, Rancher etc.
Stargazers over time
bitnami/kafka
bitnami
比特纳米Kafka安全镜像是一款针对分布式流处理平台Kafka的预配置、安全加固容器镜像，集成行业最佳安全实践，涵盖漏洞扫描、最小权限原则、加密通信支持及合规性检查，旨在简化Kafka部署流程，确保数据传输与存储安全性，适用于企业级流数据处理场景，帮助用户快速搭建安全可靠的Kafka集群。
956 次收藏1亿+ 次下载
6 个月前更新
bitnamicharts/kafka
bitnamicharts
Bitnami为Apache Kafka提供的Helm Chart是一款预配置的Kubernetes包管理工具，旨在简化分布式流处理平台Apache Kafka在Kubernetes集群中的部署、配置与全生命周期运维管理，集成了高可用性集群设置、安全认证机制、Prometheus监控指标及自动伸缩策略等核心功能，帮助用户无需手动处理复杂的集群参数配置，即可快速搭建稳定、可扩展且符合生产级标准的Kafka服务，适用于从开发测试到大规模生产环境的各类场景。
4 次收藏1000万+ 次下载
6 个月前更新
ubuntu/kafka
Ubuntu 官方镜像
Apache Kafka 是一个分布式事件流平台，它支持高吞吐量、低延迟的实时数据流处理与传输，可广泛应用于消息传递、日志聚合、实时分析、数据集成等场景，其长期维护轨道由 Canonical 负责，以确保平台在稳定性、安全性及功能迭代方面获得持续支持，为企业级用户提供可靠的事件流处理解决方案。
60 次收藏100万+ 次下载
1 个月前更新
apache/kafka
Apache 软件基金会镜像
Apache Kafka是一个开源的分布式流处理平台，旨在提供高吞吐量、低延迟的实时数据流传递服务，支持发布/订阅消息模式，能够持久化存储海量数据流并确保数据可靠性，具备水平扩展能力和容错机制，广泛应用于日志收集、事件驱动架构、实时数据集成及流处理系统等场景，为企业级应用提供高效、稳定的数据流传输与处理解决方案。
198 次收藏1000万+ 次下载
29 天前更新
manageiq/kafka
manageiq
Kafka container for ManageIQ
1万+ 次下载
7 年前更新
adobe/kafka
adobe
暂无描述
5.5千+ 次下载
5 个月前更新
镜像拉取常见问题
使用与功能问题
错误码与失败问题
manifest unknown 错误：镜像不存在或标签错误
manifest unknown 错误
TLS/SSL 证书验证失败：Docker pull 时 HTTPS 证书错误
TLS 证书验证失败
DNS 解析超时：无法解析镜像仓库地址或连接超时
DNS 解析超时
410 Gone 错误：Docker 版本过低导致协议不兼容
410 错误：版本过低
402 Payment Required 错误：流量耗尽错误提示
402 错误：流量耗尽
401 UNAUTHORIZED 错误：身份认证失败或登录信息错误
身份认证失败错误
429 Too Many Requests 错误：请求频率超出专业版限制
429 限流错误
Docker login 凭证保存错误：Cannot autolaunch D-Bus（不影响登录）
凭证保存错误
账号 / 计费 / 权限
用户好评
来自真实用户的反馈，见证轩辕镜像的优质服务
oldzhang
运维工程师
Linux服务器
5
"Docker访问体验非常流畅，大镜像也能快速完成下载。"