mirantis/magma-ci

magma-ci-infra

Magma CI/CD基础设施构建说明。

概述

magma-ci-infra是一套用于在Azure云平台上构建Magma项目CI/CD基础设施的工具集。通过Terraform实现基础设施即代码(IaC)，自动化部署和管理AKS（Azure Kubernetes Service）集群，并集成ArgoCD、Prow、Prometheus、Grafana等CI/CD关键组件，为Magma项目提供完整的持续集成和持续部署环境。

前提条件

构建AKS集群需安装以下工具：

• Azure账户
• az（Azure CLI）
• kubectl（Kubernetes命令行工具）
• terraform（基础设施即代码工具）
• https://stedolan.github.io/jq/

架构图

!Magma CI/CD

创建AKS集群

按照以下步骤使用Terraform安装AKS集群。

环境变量配置

导出访问Azure所需的环境变量。若不了解如何配置这些变量，请参考Terraform中配置服务主体或相关文档，也可参考https://ruzickap.github.io/k8s-flagger-istio-flux/part-01/#prepare-the-azure-environment%E3%80%82

shell
export ARM_CLIENT_ID="axxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx8"
export ARM_CLIENT_SECRET="dxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx0"
export ARM_SUBSCRIPTION_ID="exxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxb"
export ARM_TENANT_ID="5xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx8"
export TF_VAR_client_id="${ARM_CLIENT_ID}"
export TF_VAR_client_secret="${ARM_CLIENT_SECRET}"
export TF_VAR_subscription_id="${ARM_SUBSCRIPTION_ID}"
export TF_VAR_tenant_id="${ARM_TENANT_ID}"

上述变量为必填项，必须在执行后续步骤前正确设置。

以下为可选配置变量，可根据需要修改（默认值如下，可直接复制到终端）：

shell
export TF_VAR_admin_email_address="pruzicka@mirantis.com"
export TF_VAR_dns_zone_name="magma.mirantis.cloud"
export TF_VAR_kubernetes_cluster_name="k8s"
export TF_VAR_kubernetes_version="1.15.11"
export TF_VAR_location="francecentral"
export TF_VAR_prefix="magmaci"
export TF_VAR_resource_group_name="${TF_VAR_prefix}"
export TF_VAR_resource_group_name_dns="magma-dns"
export TF_VAR_letsencrypt_environment="staging"  # staging / production（测试/生产环境）
export TF_VAR_argocd_application_git_repository="Mirantis/magma-argocd-apps"
# bcrypt哈希密码，参考：https://github.com/argoproj/argo-helm/blob/master/charts/argo-cd/values.yaml#L747
export TF_VAR_argocd_server_admin_password="htpasswd_generated_password"
export TF_VAR_harbor_admin_password="harbor_admin_password"
export TF_VAR_prow_git_repository="Mirantis/magma"
# 为机器人用户生成的GitHub个人访问令牌
export TF_VAR_prow_oauth_token='my_prow_oauth_token'
# Slack应用访问令牌：https://github.com/kubernetes/test-infra/tree/master/prow/cmd/crier
export TF_VAR_prow_slack_access_token='my_prow_slack_access_token'
export TF_VAR_oidc_client_id="1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxr.apps.googleusercontent.com"
export TF_VAR_oidc_client_secret="_xxxxxxxxxxxxxxxxxxxxxx4"
export TF_VAR_alertmanager_slack_api_url='https://hooks.slack.com/services/TxxxxxxxT/BxxxxxxxM/Mxxxxxxxxxxxxxxxxxxxxxxr'
export TF_VAR_ssh_public_key="../../secrets/k8s-nodes-ssh-key/id_rsa.pub"
export TF_VAR_default_node_pool="{
  node_count                     = 3,
  vm_size                        = \"Standard_D2as_v4\",
  vm_disk_size                   = 40,
  zones                          = [\"1\"],
  cluster_auto_scaling           = false,
  cluster_auto_scaling_min_count = null,
  cluster_auto_scaling_max_count = null
}"
export TF_VAR_additional_node_pools="{
  p1 = {
    node_count                     = 2,
    vm_size                        = \"Standard_D2as_v4\",
    vm_disk_size                   = 40,
    zones                          = [\"1\"],
    cluster_auto_scaling           = false,
    cluster_auto_scaling_min_count = null,
    cluster_auto_scaling_max_count = null,
  },
}"
export STORAGE_ACCOUNT_NAME="${TF_VAR_prefix}tf"
export CONTAINER_NAME="${TF_VAR_prefix}-tfstate"

变量详细说明请参考terraform/azure/variables.tf。

注意：若需创建第二个集群，TF_VAR_prefix变量需设置为唯一值。

注意：若有权限访问secrets/environment_variables中的解密内容，可直接使用以下命令加载所有必要变量：

bash
source secrets/environment_variables

最佳实践：将所有变量存储在文件中，执行Terraform前通过source命令加载（如source your_env_file）。

创建Azure上的Terraform状态后端

执行Terraform前必须完成以下步骤，因为Terraform需要使用新创建的"存储容器"保存"状态文件"。

登录Azure：

bash
az login --service-principal --username "${ARM_CLIENT_ID}" --password "${ARM_CLIENT_SECRET}" --tenant "${ARM_TENANT_ID}" | jq

输出示例：

json
[
  {
    "cloudName": "AzureCloud",
    "id": "exxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxb",
    "isDefault": true,
    "name": "Pay-As-You-Go",
    "state": "Enabled",
    "tenantId": "5xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx8",
    "user": {
      "name": "axxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx8",
      "type": "servicePrincipal"
    }
  }
]

创建资源组：

bash
az group create --name "${TF_VAR_resource_group_name}" --location "${TF_VAR_location}" --tags "Owner=${TF_VAR_admin_email_address}" "Environment=Testing" | jq

输出示例：

json
{
  "id": "/subscriptions/exxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxb/resourceGroups/magmaci",
  "location": "eastus",
  "managedBy": null,
  "name": "magmaci",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": {
    "Environment": "Testing",
    "Owner": "pruzicka@mirantis.com"
  },
  "type": "Microsoft.Resources/resourceGroups"
}

创建存储账户：

bash
az storage account create --resource-group "${TF_VAR_resource_group_name}" --name "${STORAGE_ACCOUNT_NAME}" --access-tier Cool --kind StorageV2 --sku Standard_LRS --tags "Owner=${TF_VAR_admin_email_address}" "Environment=Testing" | jq

输出示例：

json
{
  "accessTier": "Cool",
  "azureFilesIdentityBasedAuthentication": null,
  "creationTime": "2019-12-11T14:22:31.572168+00:00",
  "customDomain": null,
  "enableHttpsTrafficOnly": true,
  "encryption": {
    "keySource": "Microsoft.Storage",
    "keyVaultProperties": null,
    "services": {
      "blob": {
        "enabled": true,
        "lastEnabledTime": "2019-12-11T14:22:31.634694+00:00"
      },
      "file": {
        "enabled": true,
        "lastEnabledTime": "2019-12-11T14:22:31.634694+00:00"
      },
      "queue": null,
      "table": null
    }
  },
  "failoverInProgress": null,
  "geoReplicationStats": null,
  "id": "/subscriptions/exxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxb/resourceGroups/magmaci/providers/Microsoft.Storage/storageAccounts/magmacitf",
  "identity": null,
  "isHnsEnabled": null,
  "kind": "StorageV2",
  "largeFileSharesState": null,
  "lastGeoFailoverTime": null,
  "location": "eastus",
  "name": "magmacitf",
  "networkRuleSet": {
    "bypass": "AzureServices",
    "defaultAction": "Allow",
    "ipRules": [],
    "virtualNetworkRules": []
  },
  "primaryEndpoints": {
    "blob": "https://magmacitf.blob.core.windows.net/",
    "dfs": "https://magmacitf.dfs.core.windows.net/",
    "file": "https://magmacitf.file.core.windows.net/",
    "queue": "https://magmacitf.queue.core.windows.net/",
    "table": "https://magmacitf.table.core.windows.net/",
    "web": "https://magmacitf.z28.web.core.windows.net/"
  },
  "primaryLocation": "eastus",
  "provisioningState": "Succeeded",
  "resourceGroup": "magmaci",
  "secondaryEndpoints": null,
  "secondaryLocation": null,
  "sku": {
    "capabilities": null,
    "kind": null,
    "locations": null,
    "name": "Standard_LRS",
    "resourceType": null,
    "restrictions": null,
    "tier": "Standard"
  },
  "statusOfPrimary": "available",
  "statusOfSecondary": null,
  "tags": {
    "Environment": "Testing",
    "Owner": "pruzicka@mirantis.com"
  },
  "type": "Microsoft.Storage/storageAccounts"
}

创建容器：

bash
az storage container create --name "${CONTAINER_NAME}" --account-name "${STORAGE_ACCOUNT_NAME}" | jq

输出示例：

json
{
  "created": true
}

使用Terraform创建集群

克隆git仓库magma-ci-infra：

bash
test -d ".git" || ( git clone git@github.com:Mirantis/magma-ci-infra.git && cd magma-ci-infra )

初始化Terraform：

bash
cd terraform/azure
terraform init \
  -backend-config="container_name=${CONTAINER_NAME}" \
  -backend-config="storage_account_name=${STORAGE_ACCOUNT_NAME}" \
  -backend-config="resource_group_name=${TF_VAR_resource_group_name}"

输出示例：

text
Initializing the backend...

Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "azurerm" (hashicorp/azurerm) 1.37.0...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.local: version = "~> 1.3"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

执行Terraform创建AKS集群：

bash
terraform apply -auto-approve

输出示例：

text
...
...
...
Apply complete! Resources: 44 added, 0 changed, 0 destroyed.
Releasing state lock. This may take a few moments...

Outputs:

acr_admin_password = Vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx1
acr_admin_username = magmacik8smagmamirantiscloud
acr_login_server = magmacik8smagmamirantiscloud.azurecr.io
azure_kubernetes_cluster_login = az aks get-credentials --name magmaci-k8s-magma-mirantis-cloud --resource-group magmaci
kubeconfig_export_command = export KUBECONFIG=./kubeconfig_magmaci-k8s-magma-mirantis-cloud.conf
kubeconfig_file = ./kubeconfig_magmaci-k8s-magma-mirantis-cloud.conf
location = eastus
url_alertmanager = https://alertmanager.magmaci.magma.mirantis.cloud
url_argocd = https://argocd.magmaci.magma.mirantis.cloud
url_grafana = https://grafana.magmaci.magma.mirantis.cloud
url_prometheus = https://prometheus.magmaci.magma.mirantis.cloud
url_prow = https://prow.magmaci.magma.mirantis.cloud
url_tekton-dashboard = https://tekton-dashboard.magmaci.magma.mirantis.cloud

集群创建完成后，验证节点状态：

bash
export KUBECONFIG=$PWD/kubeconfig_magmaci-k8s-magma-mirantis-cloud.conf
kubectl get nodes

输出示例：

text
NAME                                  STATUS   ROLES   AGE   VERSION
aks-magmacik8sd-34239724-vmss000000   Ready    agent   49m   v1.15.11
aks-magmacik8sd-34239724-vmss000001   Ready    agent   43m   v1.15.11

删除AKS集群

删除集群前，需确保环境变量与创建集群时一致（见"环境变量配置"部分）。

登录Azure：

bash
az login --service-principal --username "${ARM_CLIENT_ID}" --password "${ARM_CLIENT_SECRET}" --tenant "${ARM_TENANT_ID}" | jq

输出示例：

json
[
  {
    "cloudName": "AzureCloud",
    "id": "exxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxb",
    "isDefault": true,
    "name": "Pay-As-You-Go",
    "state": "Enabled",
    "tenantId": "5xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx8",
    "user": {
      "name": "axxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx8",
      "type": "servicePrincipal"
    }
  }
]

进入terraform/azure目录：

bash
cd terraform/azure

删除AKS集群及依赖资源：

bash
terraform destroy -auto-approve

输出示例：

text
data.azurerm_resource_group.resource_group: Refreshing state...
azurerm_kubernetes_cluster.aks: Refreshing state... [id=/subscriptions/exxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxb/resourcegroups/magmaci/providers/Microsoft.ContainerService/managedClusters/magmaci-k8s-magma-mirantis-cloud]
local_file.file: Refreshing state... [id=e90e46639b01fedc353cbb70664748b643f823a0]
local_file.file: Destroying... [id=e90e46639b01fedc353cbb70664748b643f823a0]
local_file.file: Destruction complete after 0s
...
...
...
Destroy complete!

删除包含Terraform容器/存储的资源组：

bash
az group delete --yes --name "${TF_VAR_resource_group_name}"

清理本地Terraform和Azure相关文件：

bash
rm -rf .terraform
az logout
rm -rf ~/.azure

从零构建Magma CI/CD环境