Bug 1932705

I believe the problem is not related to the project, but rather the user. Barbican has rules around who owns a secret, and if you delete the user who created the volume then the issue may be who, including an admin, can delete the user's barbican secret.

Can you please check if the problem can be reproduced by simply deleting the user, and leave the project intact?

We will likely need to coordinate with the Security DFG on how to proceed. The reason we block deleting the cinder volume unless the barbican key can be deleted is we do not want to "leak" the key. HSM's have a limited amount of storage, and we cannot abandon keys by simply deleting the associated cinder volume.

Hi Alan,

Thanks for your reply!

(In reply to Alan Bishop from comment #1)
> I believe the problem is not related to the project, but rather the user.

I added "and user" to the title to make it accurate.

> Barbican has rules around who owns a secret, and if you delete the user who
> created the volume then the issue may be who, including an admin, can delete
> the user's barbican secret.

Yeah, according to the code, I thought the admin needs to be in the project, IIUC.

https://github.com/openstack/barbican/blob/stable/train/barbican/common/policies/secrets.py#L57-L69
~~~
...
    policy.DocumentedRuleDefault(
        name='secret:delete',
        check_str='rule:secret_project_admin or ' +
                  'rule:secret_project_creator',
        scope_types=[],
        description='Delete a secret by uuid.',
        operations=[
            {
                'path': '/v1/secrets/{secret-id}',
                'method': 'DELETE'
            }
        ]
    ),
...
~~~

> Can you please check if the problem can be reproduced by simply deleting the
> user, and leave the project intact?

Sure, I'll check it.

> We will likely need to coordinate with the Security DFG on how to proceed.
> The reason we block deleting the cinder volume unless the barbican key can
> be deleted is we do not want to "leak" the key. HSM's have a limited amount
> of storage, and we cannot abandon keys by simply deleting the associated
> cinder volume.

Thanks, it makes sense.

Best Regards,
Masayuki

Hi Alan,

(In reply to Alan Bishop from comment #1)
> I believe the problem is not related to the project, but rather the user.
> Barbican has rules around who owns a secret, and if you delete the user who
> created the volume then the issue may be who, including an admin, can delete
> the user's barbican secret.
> 
> Can you please check if the problem can be reproduced by simply deleting the
> user, and leave the project intact?

Yes, that is reproduced by simply deleting the user. An admin user can't delete the volume after deleting the user.

Best Regards,
-- Masayuki

I'm going to run some experiments to see if I can either identify a workaround, or if it's time to pull in a Barbican expert.

Here are the results of running some tests and discussing the situation with people in the Security DFG.

It turns out there are two different scenarios, each of which has a different solution.

Scenario #1: The user is deleted, but the project is intact (not deleted).
Scenario #2: The project is deleted (it doesn't matter whether or not the user is also deleted).

For #1, the barbican secret associated with the cinder volume can be deleted by any user who has admin privilge ***in that project***. The last part is critical. So, the global "admin" user will be able to delete the secret, but only after you add the "admin" user with admin privileges, like this:

  # openstack role add --project <project> --user admin admin

Now the "admin" should be able to delete the cinder volume, which in turn will delete the barbican secret. Note you should undo this change (remove the role) after deleting all encrypted volumes created by the user who was deleted.

Scenario #2 is more complex, and the reason is deleting the project makes it impossible to satisfy barbican's default policy for deleting secrets, which you show in the comment #2:

    policy.DocumentedRuleDefault(
        name='secret:delete',
        check_str='rule:secret_project_admin or ' +
                  'rule:secret_project_creator',
        scope_types=[],
        description='Delete a secret by uuid.',
        operations=[
            {
                'path': '/v1/secrets/{secret-id}',
                'method': 'DELETE'
            }
        ]
    ),

The "secret_project_admin" and "secret_project_creator" rules can never be satisfied when the project no longer exists. In this case, the workaround is to loosen the delete policy long enough to delete all encrypted volumes associated with the deleted project.

1. Create a policy.yaml file and restart the barbican_api container

  # echo '"secret:delete": "rule:admin or rule:secret_project_creator"' | sudo tee /var/lib/config-data/puppet-generated/barbican/etc/barbican/policy.yaml
  # sudo podman restart barbican_api

2. Use the global "admin" user to delete all volumes associated with the deleted project

3. Remove the policy.yaml file, and ensure it's deleted from the barbican_api container

  # sudo rm -f /var/lib/config-data/puppet-generated/barbican/etc/barbican/policy.yaml
  # sudo podman exec barbican_api rm -f /etc/barbican/policy.yaml
  # sudo podman restart barbican_api

You will need to execute steps 1 and 3 on all three controllers in order to update all instances of the barbican-api service.

NOTE: It's important to remove the modified policy (step 3) to ensure admins cannot delete secrets in someone else's projects !!!

Hi Alan,

Thank you very much for the workaround!
It makes sense to me. I'll check that.

And I have one more question.
Is there any plan to provide a feature for "Scenario #2"?
Or do we need this workaround in this scenario for the future?

Because I think "to loosen the delete policy" sounds a bit risky, and the steps a bit complicated.

Best Regards,
Masayuki

I'm going to redirect the question to Ade, as this is a barbican issue rather than something for cinder to handle.

Thanks to input from Ade, I have additional info to share, but I'm also closing this BZ.

Barbican does have a provision that will cause all secrets to be automatically deleted whenever a project is deleted. In this situation, the global admin (as opposed to a project specific admin, where the project itself has been deleted) *will* be able to delete encrypted volumes associated with the deleted project. In other words, as long as barbican deletes the project's secrets when the project is deleted, the global admin will be able to delete cinder volumes that reference the now-deleted secrets.

The next issue, though, is that barbican relies on keystone notifications to purge secrets when a project is deleted, and by default the keystone notification driver is "noop," meaning there's no keystone driver to deliver the notifications. The notification driver is enabled whenever telemetry is enabled, but it's also possible to enable it manually by including this in the deployment:

parameter_defaults:
  NotificationDriver: messagingv2

With this in place, I was able to successfully do this:
- Create a project and a user for that project
- Create an encrypted volume by that user
- Delete the project (and observe the project's secret is automatically deleted)
- Delete the volume using the global admin account

However, it's not reasonable to force keystone notifications to be enabled simply to facilitate deleting encrypted cinder volumes in situations where the associated project has been deleted. I'm providing the information here so readers have the full context.

With this in mind, I'm closing this BZ as NOTABUG. It's true that, unless keystone notifications are enabled, barbican's default policy will prevent deleting secrets when a project no longer exists, and that prevents any use (even the global admin) from deleting an encrypted cinder volume associated with the deleted project. From cinder's point of view, the best we can offer is the workaround described in comment #6.

My suggestion is you file another BZ against barbican that focuses on the difficulty of deleting a secret after its associated project was deleted.

I chatted with Ade on irc, and he reminded my that OSP-16.1 (train) is still using policy.json (not .yaml). Here is an updated set of instructions for creating a JSON policy file:

1. Create a policy.json file and restart the barbican_api container

  # echo '{"secret:delete": "rule:admin or rule:secret_project_creator"}' | sudo tee /var/lib/config-data/puppet-generated/barbican/etc/barbican/policy.json
  # sudo podman restart barbican_api

2. Use the global "admin" user to delete all volumes associated with the deleted project

3. Remove the policy.json file, and ensure it's deleted from the barbican_api container

  # sudo rm -f /var/lib/config-data/puppet-generated/barbican/etc/barbican/policy.json
  # sudo podman exec barbican_api rm -f /etc/barbican/policy.json
  # sudo podman restart barbican_api

You will need to execute steps 1 and 3 on all three controllers in order to update all instances of the barbican-api service.

NOTE: It's important to remove the modified policy (step 3) to ensure admins cannot delete secrets in someone else's projects !!!

The change to policy.json does allow an admin delete request to get past RBAC, but there seems to be a bug in Barbican's model layer that is throwing an unexpected error.  I am reopening this bug to get this fixed in Barbican.

Hello,

Why not just change :

{"secret_project_creator": "rule:creator and rule:secret_project_match and rule:secret_creator_user}
by
{"secret_project_creator": "rule:creator and rule:secret_project_match"}

Thanks for your feedback.
CL

This bug was fixed upstream in stable/wallaby by this patch: https://opendev.org/openstack/barbican/commit/b0ec7edf1dd69eaf5ec30a1a5f6e8db7ecfa3024

The fix was included in 17.0 and will also be includedin 17.1.  With this fix the custom policy to allow deletion of a secret after the project that owns it has been deleted from keystone works as intended.

Hi Douglas,

Is the fix merged into stable/train and included in OSP16. As far as I saw the code in OSP16.1, the code seems to be merged.
Can we consider that this issue is fixed in OSP16?

@knoha this fix has not been backported to any branch before stable/wallaby.  The fix is currently only available in 17.x.

Hi Douglas,

Thank you for your comment. It looks that I overlooked the code in OSP16.2. I'm sorry for that.
My customer would like to know how to remove the orphaned resource in barbican and cinder when they hit an issue.

Is there a way to recover from the situation? Or manual DB operation is the only option to get the customer out of situation?

Best regards,
Keigo Noha