intel/apphsm

KMRA - Key Management Reference Application with SGX - docker images

KMRA (Key Management Reference Application) is a proof-of-concept software created to demonstrate the integration of SGX asymmetric / symmetric key capability with a third party HSM. For source code and documentation, go to: [***]

Preconditions

Install SGX kernel driver on the host. Out-of-tree SGX kernel driver can be installed using KMRA ansible scripts in kmaas/ansible/sgx-infra-setup directory.

Tags

KMRA v1.4 - v1

KMRA v2.0.1 - v2

KMRA v2.1 - v2.1

KMRA v2.2 - v2.2

KMRA v2.2.1 - v2.2.1

KMRA v2.2.2 - v2.2.2

KMRA v2.3 - v2.3

KMRA Docker Image deployment

Download KMRA Source Code: [***]

Generate mTLS certificates and keys to share with containers:
$ cd /kmra/apphsm/ca
$ bash -c "APPHSM_HOSTNAME=apphsm ./gen_all.sh"

NOTE: APPHSM_HOSTNAME must match the name of AppHSM container

Create custom bridge network:
$ docker network create kmra-net

PCCS

Prepare certificates and configuration file for PCCS container (on host):
$ cd /kmaas/containers/pccs
$ mkdir -p certs && rm -rf certs/*
$ cd certs
$ bash ../scripts/pccs_generate_certificates.sh

The primary key from the subscription will be used in the next steps. This key is accessible at any time on this website: [***]

Set PCCS_* environment variables before generating config file using export (on host):
$ export PCCS_ADMIN_PASS="example-admin-pass"
$ export PCCS_API_KEY="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

Run pccs_generate_config.sh script (on host):
$ bash ../scripts/pccs_generate_config.sh

PCCS_API_KEY environment variable is required, others are optional, default values will be used for optional variables. See list of environment variable and default values below.

See "Environment variables" below for a list of available variables. Generate configuration for PCCS (on host):
$ cd kmra/containers/pccs/config
$ ../scripts/pccs_generate_config.sh

Run PCCS container with mounted generated certificates and config file:
$ cd /kmra/containers/pccs
$ docker run --user "65333:65333" --rm --network kmra-net --name pccs -it --cpu-shares 512 --pids-limit 100 --memory=512m --security-opt=no-new-privileges -v `pwd`/certs/private.pem:/opt/intel/pccs/ssl_key/private.pem -v `pwd`/certs/file.crt:/opt/intel/pccs/ssl_key/file.crt -v `pwd`/config/pccs.config:/opt/intel/pccs/config/default.json --read-only --tmpfs /opt/intel/pccs/logs/ --tmpfs /tmp --cap-drop=all pccs

Build arguments:

• USER - a name of the user inside the container (kmra by default)

• UID - UID for the above user (1000 by default)

Environment variables:

• PCCS_API_KEY - value of 'primary key' from the subscription from site: <[***]>

• PCCS_ADMIN_PASS - password for PCCS admin (not used in this demo but value can be provided) - PCCS_USER_PASS - password for PCCS user (not used in this demo but value can be provided)

• PCCS_USER_PASS - password for PCCS user (not used in this demo but value can be provided)

• http_proxy, https_proxy - have to be set if using PCCS behind corporate proxy: '-e https_proxy="[***]"' (to be set at image build time)

• PCCS_PORT - port on which PCCS service will listen (8081 by default)

• PCCS_LOCAL_ONLY - flag (Y/N) indicating whether PCCS service should listen on local network interface only (N by default)

AppHSM

Run AppHSM container from main directory:
$ cd /kmra

Sample command to run AppHSM docker container:
docker run -it --rm --cpu-shares 512 --pids-limit 100 --memory=2048m --security-opt=no-new-privileges --read-only --tmpfs /var/lib/softhsm/tokens --tmpfs /tmp -v `pwd`/containers/apphsm/sgx_default_qcnl.conf:/etc/sgx_default_qcnl.conf:ro -v `pwd`/apphsm/ca/:/opt/intel/ca:ro --name apphsm --env no_proxy="pccs" --cap-drop=all --network kmra-net apphsm:latest

[Optional] Run AppHSM with custom configuration: Create custom key and certificate. They can be generated by script:
$ cd containers/apphsm/custom_config
$ bash ./gen_key_cert.sh Edit the configuration file "apphsm.conf" and add the required key(s). The "token_name" name field must be different for every key. The "key_name" and "certificate_file" fields must correspond to file names with key and certificate placed in "custom_config" directory. The provided default configuration match the default key and certificate file names generated by "gen_key_cert.sh" script. Mount the directory with custom config file, keys and certificates:
$ cd kmra/
$ docker run -it --rm --cpu-shares 512 --pids-limit 100 --memory=2048m --security-opt=no-new-privileges --read-only --tmpfs /var/lib/softhsm/tokens --tmpfs /tmp -v `pwd`/containers/apphsm/sgx_default_qcnl.conf:/etc/sgx_default_qcnl.conf:ro -v `pwd`/apphsm/ca/:/opt/intel/ca:ro -v `pwd`/containers/apphsm/custom_config:/opt/apphsm_config:ro --name apphsm --env no_proxy="pccs" --cap-drop=all --network kmra-net apphsm:latest

Build arguments:

• DCAP_VERSION - 1.15 by default

• DCAP_LIB_VERSION - 1.15.100.3 by default

• SGX_LIB_VERSION - 2.18.101.1 by default

• USER - a name of the user inside the container (kmra by default)

• UID - UID for the above user (1000 by default)

Set environment variables:

• APPHSM_PORT - port on which AppHSM service will listen (5000 by default)

• no_proxy - a list of host names, IP addresses, IP subnets for that the proxy will not be used

• APPHSM_KEY_IN_TOKEN_NAME - label of the key in the softhsm for apphsm (key_1 by default)

• APPHSM_TOKEN_NAME -label of the token in the softhsm for apphsm (token_1 by default)

• TEST_CTK_LOADKEY_CERT_USER_ID - name of the client that will be visible in the 'OU' field of generated certificate for the client side. This name need to be defined in apphsm.conf in the 'clients' section. (ctk_loadkey_user_id_01234 by default)

• TEST_UNIQUE_UID - unique id of the key definition located on AppHSM side in apphsm.conf. Client can obtain this key by requesting this unique id using ctk_loadkey

• DEFAULT_USER_PIN - softhsm user pin. Valid length is 4-16 chars. (1234 is default)

• DEFAULT_SO_PIN - softhsm security officer pin. Valid length is 4-16 chars. Please consult your security officer. (*** is default)

• APPHSM_CUSTOM_CONFIG_DIR - location of custom configuration inside container (/opt/apphsm_config by default)

PCCS configuration file containers/apphsm/sgx_default_qcnl.conf:

• PCCS_URL - valid url address of pccs service (<[***]> by default)

• USE_SECURE_CERT - to accept insecure HTTPS cert for PCCS URL, set this option to FALSE (FALSE by default)

gen_key_cert.sh script:

• DEFAULT_KEY_NAME - file name to store the generated key

• DEFAULT_CERTIFICATE_NAME - file name to store the generated certificate

Ctk_loadkey

Run Ctk_loadkey container from main directory:
$ cd /kmra

Sample command to run Ctk_loadkey container:
docker run -it --rm --device /dev/sgx_*** --cpu-shares 512 --pids-limit 100 --memory=2048m --security-opt=no-new-privileges --device /dev/sgx_provision --cap-drop=all --env PCCS_HOSTNAME=pccs --env APPHSM_HOSTNAME=apphsm --env no_proxy="apphsm,pccs" --name ctk_loadkey --network kmra-net --read-only --tmpfs /opt/intel/***apitoolkit/tokens --tmpfs /tmp -v `pwd`/containers/ctk/sgx_default_qcnl.conf:/etc/sgx_default_qcnl.conf:ro -v `pwd`/containers/nginx/p11_proxy_tls.psk:/etc/p11_proxy_tls.psk:ro -v `pwd`/apphsm/ca/:/opt/intel/ca:ro --user kmra:$(getent group sgx_prv | cut -d: -f3) ctk_loadkey:latest

NOTE:

On some host systems (e.g Ubuntu 22) additional argument may be required to mount /dev/sgx_*** inside container:
--group-add $(getent group sgx | cut -d: -f3) You can check if sgx group is needed by executing command below on the host:
$ ls -lg /dev/sgx_enclave
crw-rw---- 1 sgx 10, 125 Dec 13 10:54 /dev/sgx_enclave

Build arguments:

• DCAP_VERSION - 1.15 by default

• DCAP_LIB_VERSION - 1.15.100.3 by default

• SGX_VERSION - 2.18.1 by default

• SGX_LIB_VERSION - 2.18.101.1 by default

• USER - a name of the user inside the container (kmra by default)

• UID - UID for the above user (1000 by default)

Environment variables:

• APPHSM_HOSTNAME - host on which AppHSM service is listening (localhost by default) Usually it should be set to a name of the AppHSM container - "apphsm" (the last argument in docker run ... command starting AppHSM container)

• APPHSM_PORT - port on which AppHSM service is listening (5000 by default)

• NGINX_HOSTNAME - the host name on which nginx will listen (0.0.0.0 by default)

• NGINX_PORT - port on which nginx will listen (8082 by default)

• no_proxy - a list of host names, IP addresses, IP subnets for that the proxy will not be used

• CLIENT_TOKEN - private key for the certificate for nginx will be stored by ctk_loadkey in this token label (client_token by default)

• CLIENT_KEY_LABEL - label of the key which nginx will use to finds its key (client_key_priv by default)

• TEST_UNIQUE_UID - unique id of the key definition located on AppHSM side in apphsm.conf. Client can obtain this key by requesting this unique id using ctk_loadkey

• TEST_UNIQUE_UID - unique id of the key definition located on AppHSM side in apphsm.conf. Client can obtain this key by requesting this unique id using ctk_loadkey

• DEFAULT_USER_PIN - ***-Api-Toolkit user pin. Valid length is 4-16 chars. (1234 is default)

• DEFAULT_SO_PIN - -Api-Toolkit security officer pin. Valid length is 4-16 chars. Please consult your security officer. ( is default)

• DEFAULT_CLIENT_TOKEN_ID - ID of the key pair. The ID is in hexadecimal with a variable length, used by pkcs11-tool as argument to write certificate into token

• KEEP_TOKENS - Do not cleanup *** api toolkit tokens on startup

PCCS configuration file containers/ctk/sgx_default_qcnl.conf:

• PCCS_URL - valid url address of pccs service (<[***]> by default)

• USE_SECURE_CERT - to accept insecure HTTPS cert for PCCS URL, set this option to FALSE (FALSE by default)

NGINX

Run NGINX container::
$ cd /kmra

Sample command to run NGINX container:
docker run -it --rm --cpu-shares 512 --pids-limit 100 --memory=2048m --read-only --tmpfs /tmp --env PKCS11_PROXY_SOCKET=tls://<ctk_loakey container IP address>:<port_number> --env no_proxy="ctk_loadkey" -v `pwd`/containers/nginx/p11_proxy_tls.psk:/etc/p11_proxy_tls.psk:ro --name nginx --network kmra-net -p 8082:8082 nginx:latest

Environment variables:

• PKCS11_PROXY_SOCKET - ctk_loakey container IP address and exposed port number of pkcs11-proxy

• NGINX_HOSTNAME - the host name on which nginx will listen (0.0.0.0 by default)

• NGINX_PORT - port on which nginx will listen (8082 by default)

• CLIENT_TOKEN - token name with private key for for nginx (client_token by default)

• CLIENT_KEY_LABEL - label of the key which nginx will use to finds its key (client_key_priv by default)

• DEFAULT_USER_PIN - ***-Api-Toolkit user pin, valid length is 4-16 chars (1234 is default)

REMARKS:

• TLS pre-shared key file format: key_name:16 bytes of key in hexadecimal format. The key is common for both ctk and nginx conatainer. See sample p11_proxy_tls.psk file in source code.

Common Problems

1. Failure in task '[create_empty_token_in_hsm: Create token ...]'

-- Error log: TASK [create_empty_token_in_hsm : Create token with name 'client_token'] ******* fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["softhsm2-util", "--module", "/usr/local/lib/libp11sgx.so.0.0.0", "--init-token", "--free", "--label", "client_token", "--pin", "1234", "--so-pin", "*** 678"], ... Could not initialize the PKCS#11 library/module: /usr/local/lib/libp11sgx.so.0.0.0\nERROR: Please check log files for additional information.", "stderr_lines": ["[get_driver_type /home/sgx/jenkins/ubuntuServer2004-release-build-tr unk-213.3/build_target/PROD/label/Builder-UbuntuSrv20/label_exp/ubuntu64/linux-trunk-opensource/psw/urts/linux/edmm_utility.cpp:111] Failed to open Intel SGX device.", "ERROR: Could not initialize the PKCS#11 li brary/module: /usr/local/lib/libp11sgx.so.0.0.0", "ERROR: Please check log files for additional information."], "stdout": "", "stdout_lines": []} -- Root cause: There is a problem with sharing SGX services (e.g. lack of --device /dev/sgx/*** passed to the container during runtime)

2. Failure in task [install_ctk_loadkey: Copy ca cert and ctk_loadkey keys ...]

-- Error log: TASK [install_ctk_loadkey : Copy ca cert and ctk_loadkey keys to /opt/intel/ctk_loadkey] *** changed: [localhost] => (item=ctk_loadkey.crt) fatal: [localhost]: FAILED! => {"msg": "an error occurred while trying to read the file '/opt/intel/ca/ctk_loadkey.key': [Errno 13] Permission denied: b'/opt/intel/ca/ctk_loadkey.key'. [Errno 13] Permission denied: b'/opt/intel/ca/ctk_loadkey.key'"} -- Root cause: There is mismatch between user id in the container and the user that owns shared certificate/key files. Make sure that certificate/key files for ctk_loadkey are accessible for 'kmra' user inside container.

3. ***** not authorized to run in task [provision_ctk_with_key_from_apphsm ...]**

-- Error log: TASK [provision_ctk_with_key_from_apphsm : Provision token client_token with key client_key_priv from AppHSM] *** fatal: [localhost]: FAILED! => {"changed": true, "cmd": "cd /opt/intel/ctk_loadkey; https_proxy="" ./ctk_loadkey -t client_token -p 1234 -u unique_id_1234 -P 5000 -H silpixa00400537", "delta": "0:00:00.478512", "end": "2021-07-19 13:33:45.", "msg": "non-zero return code", "rc": 5, "start": "2021-07-19 13:33:44.687554", "stderr": "[error_driver2api sgx_common.cpp:247] *** not authorized to run, .e.g. provisioning *** hosted in app without access rights to /dev/sgx_provision. You need add the user id to group sgx_prv or run the app as root.\n[load_pce ../pce_wrapper.cpp:175] Error, call sgx_create*** for PCE fail [load_pce], SGXError:4004.", "stderr_lines": ["[error_driver2api sgx_common.cpp:247] *** not authorized to run, .e.g. provisioning *** hosted in app without access rights to /dev/sgx_provision. You need add the user id to group sgx_prv or run the app as root.", "[load_pce ../pce_wrapper.cpp:175] Error, call sgx_create for PCE fail [load_pce], SGXError:4004."], "stdout": "Error during C_WrapKey-size: CKR_GENERAL_ERROR\nError during creating ecdsa_quote: CKR_GENERAL_ERROR\nError during ctk_quote generation", "stdout_lines": ["Error during C_WrapKey-size: CKR_GENERAL_ERROR", "Error during creating ecdsa_quote: CKR_GENERAL_ERROR", "Error during ctk_quote generation"]} -- Root cause: There is a mismatch between group id for 'sgx_prv' group in the container and the same group on the host. Make sure that both group ids are equal.

4. SSL peer certificate error in task [provision_ctk_with_key_from_apphsm..]

-- Error log: fatal: [localhost]: FAILED! => {"changed": true, "cmd": "cd /opt/intel/ctk_loadkey; https_proxy="" ./ctk_loadkey -t client_token -p 1234 -u unique_id_1234 -P 5000 -H apphsm", "delta": "0:00:01.702477", "end": "2021-07-20 12:15:58.***", "msg": "non-zero return code", "rc": 5, "start": "2021-07-20 12:15:56.408581", "stderr": "", "stderr_lines": [], "stdout": "rest_api_check_version: Supports AppHSM v0.1 (or newer) API v0.1.\nrest_api_perform_request: REST API request failed 'SSL peer certificate or SSH remote key was not OK'!\nrest_api_check_version: Failed to get AppHSM version!\nFAILED REST API initialization for host 'apphsm' on port 5000: -93\nFailed to send export key request: CKR_GENERAL_ERROR", "stdout_lines": ["rest_api_check_version: Supports AppHSM v0.1 (or newer) API v0.1.", "rest_api_perform_request: REST API request failed 'SSL peer certificate or SSH remote key was not OK'!", "rest_api_check_version: Failed to get AppHSM version!", "FAILED REST API initialization for host 'apphsm' on port 5000: -93", "Failed to send export key request: CKR_GENERAL_ERROR"]} -- Root cause: AppHSM common name in the AppHSM certificate does not match domain name used for connecting to AppHSM from ctk_loadkey container (e.g. in the error above, certificate was generated for 'localhost' but it should be generated for 'apphsm' instead). Generate certificates again with proper APPHSM_HOSTNAME variable set and restart apphsm and ctk_loadkey containers.