Bug 1965086

Description of problem:
OSP 16.1.5 environment with two Atos HSM devices in HA configuration. Getting/Storing secrets using Barbican occasionally fails. When Barbican transactions succeed, access to one of the two Atos HSM devices can be seen. When a failure happens, there is no connection to either of the two Atos HSM devices.

This error is seen in the atos-barbican-api.log:

>>> TW_CK [Tue May 25 19:51:42:142] [pid:94 thid:140396351485696]
    C_GenerateKey(0x11000003,{CKM_AES_KEY_GEN, NULL_PTR, 0x00000000},((CLASS 04:00:00:00:00:00:00:00 #x00000008)
 (KEY_TYPE 1f:00:00:00:00:00:00:00 #x00000008)
 (VALUE_LEN 20:00:00:00:00:00:00:00 #x00000008)
 (TOKEN 00 #x00000001)
 (PRIVATE 01 #x00000001)
 (SENSITIVE 00 #x00000001)
 (ENCRYPT 01 #x00000001)
 (DECRYPT 01 #x00000001)
 (SIGN 00 #x00000001)
 (VERIFY 00 #x00000001)
 (WRAP 00 #x00000001)
 (UNWRAP 00 #x00000001)
 (EXTRACTABLE 01 #x00000001)
 ),13) ...starting
>>> TW_CK [Tue May 25 19:51:42:142] [pid:94 thid:140396351485696]
    cl_call network error (ERROR 0X81000071);
>>> TW_CK [Tue May 25 19:51:42:142] [pid:94 thid:140396351485696]
    processFinalize cnx = 0x843c2ea0 modeMonoCnx=0 (OK 0X0);
>>> TW_CK [Tue May 25 19:51:42:142] [pid:94 thid:140396351485696]
    C_GenerateKey ... failed (ERROR 0X81000071);
>>> TW_CK [Tue May 25 19:51:42:143] [pid:94 thid:140396351485696]
    C_Finalize((nil)) ...starting
>>> TW_CK [Tue May 25 19:51:42:143] [pid:94 thid:140396351485696]
    cl_call network error (ERROR 0X81000071);
>>> TW_CK [Tue May 25 19:51:42:143] [pid:94 thid:140396351485696]
    processFinalize cnx = 0x85037f00 modeMonoCnx=0 (OK 0X0);
>>> TW_CK [Tue May 25 19:51:42:143] [pid:94 thid:140396351485696]
    C_Finalize ...  0  HSM failed(OK 0X0);
>>> TW_CK [Tue May 25 19:51:42:143] [pid:94 thid:140396351485696]
    C_Finalize ... failed (ERROR 0X81000071);

And this error is in the barbican-api.log:

2021-05-25 19:51:42.143 94 ERROR barbican.api.controllers barbican.common.exception.P11CryptoPluginException: HSM returned response code: 0x81000071 CKR_????


Version-Release number of selected component (if applicable):

python3-barbicanclient-4.9.0-1.20201113110223.9c0e02d.el8ost.noarch
python3-barbican-9.0.1-1.20210310093023.el8ost.noarch
openstack-barbican-api-9.0.1-1.20210310093023.el8ost.noarch
openstack-barbican-common-9.0.1-1.20210310093023.el8ost.noarch


How reproducible:
Run repeated Barbican secret commands. Some will fail and some will not:

# date; openstack secret store --name mysecret --payload 'shhhhhh'
Tue May 25 19:51:39 UTC 2021
5xx Server error: Internal Server Error: Secret creation failure seen - please contact site administrator.
Internal Server Error: Secret creation failure seen - please contact site administrator.

# date; openstack secret store --name myothersecret --payload 'shhhhhh'
Tue May 25 20:25:40 UTC 2021
+---------------+-----------------------------------------------------------------------------------------------------+
| Field         | Value                                                                                               |
+---------------+-----------------------------------------------------------------------------------------------------+
| Secret href   | https://redacted.local:9311/v1/secrets/ac897b44-bfd0-4c9c-b4dd-bd86c77764b3 |
| Name          | myothersecret                                                                                         |
| Created       | None                                                                                                |
| Status        | None                                                                                                |
| Content types | None                                                                                                |
| Algorithm     | aes                                                                                                 |
| Bit length    | 256                                                                                                 |
| Secret type   | opaque                                                                                              |
| Mode          | cbc                                                                                                 |
| Expiration    | None                                                                                                |
+---------------+-----------------------------------------------------------------------------------------------------+

# date; openstack secret get https://redacted.local:9311/v1/secrets/ac897b44-bfd0-4c9c-b4dd-bd86c77764b3 --payload
Tue May 25 20:28:59 UTC 2021
5xx Server error: Internal Server Error: Secret payload retrieval failure seen - please contact site administrator.
Internal Server Error: Secret payload retrieval failure seen - please contact site administrator.

# date; openstack secret get https://redacted.local:9311/v1/secrets/ac897b44-bfd0-4c9c-b4dd-bd86c77764b3 --payload
Tue May 25 20:29:32 UTC 2021
+---------+---------+
| Field   | Value   |
+---------+---------+
| Payload | shhhhhh |
+---------+---------+


Actual results:
Secrets will get stored/retrieved if multiple attempts are made.

Expected results:
Secrets should get stored/retrieved on the first attempt.

Additional info:
CU has applied a fix available from BZ 1918480