Interestingly, in our lab the quota set command was working as expected:

~~~
(ess13latest) [stack@undercloud-0 ~]$ cinder --os-volume-api-version 3.40 absolute-limits dbfe4a15e64b4b8291c1b779caa426df |grep -i giga
| maxTotalBackupGigabytes  | 1000  |
| maxTotalVolumeGigabytes  | 50    |
| totalBackupGigabytesUsed | 0     |
| totalGigabytesUsed       | 0     |
(ess13latest) [stack@undercloud-0 ~]$ openstack quota set --gigabytes 2000 dbfe4a15e64b4b8291c1b779caa426df
(ess13latest) [stack@undercloud-0 ~]$ cinder --os-volume-api-version 3.40 absolute-limits dbfe4a15e64b4b8291c1b779caa426df |grep -i giga
| maxTotalBackupGigabytes  | 1000  |
| maxTotalVolumeGigabytes  | 2000  |
| totalBackupGigabytesUsed | 0     |
| totalGigabytesUsed       | 0     |
(ess13latest) [stack@undercloud-0 ~]$ cinder  absolute-limits dbfe4a15e64b4b8291c1b779caa426df |grep -i giga
| maxTotalBackupGigabytes  | 1000  |
| maxTotalVolumeGigabytes  | 1000  |
| totalBackupGigabytesUsed | 0     |
| totalGigabytesUsed       | 34    |
(ess13latest) [stack@undercloud-0 ~]$ openstack project list
+----------------------------------+------------------------------------------------------------------+
| ID                               | Name                                                             |
+----------------------------------+------------------------------------------------------------------+
| 809fe9b277a641f289fb8a8d7cd6219f | service                                                          |
| 9ebe06a47c334a17b6269f77010b883a | admin                                                            |
| d886086adc49453ba30cfbdf9528db9a | 9ebe06a47c334a17b6269f77010b883a-e6c0c370-1e0a-43f3-9c30-3bd6f82 |
| dbfe4a15e64b4b8291c1b779caa426df | ess                                                              |
+----------------------------------+------------------------------------------------------------------+
~~~

There's a lot of data to parse, and I need it distilled to something actionable.

(In reply to David Vallee Delisle from comment #0)
> Apparently, the cinder api doesn't support overriding the --project --user
> based on this story [a]. This was deprecated with this commit [b] because
> they were, apparently, silently ignored. This article [c] seems to describe
> this situation but I don't believe this is totally accurate.

Bear in mind the referenced story [a] and commit [b] (bug #1559425) pertain to the cinder API for creating a volume. This is not meant to imply the entire cinder API ignores these paramaters.

> These tests here executed on a Queens deployment by jansari++ shows the
> issue with the limits. When we limits show as admin with --project, we
> always get the admin's limits.
> 
> We went in the code and changed this line [d] to this:
> ~~~
> volume_limits = volume_client.limits.get(tenant_id=project_id)
> ~~~
> 
> And it solved the issue.

At this point I don't quite understand the issue. The referenced code [d] is an entirely different OSC command that uses different API calls.

> We were about to propose a change upstream, but before, we wanted to
> validate if it was the same thing on 16.1. Changing the same line in 16.1
> didn't yield the expected results of having the right limits.
> 
> And this brings us to our issue reported by our customer:
> - They try to spin an instance for a tenant but the quota is preventing it:
> ~~~
> (overcloud) [stack@undercloud ~]$ openstack volume create --image
> 5661d940-ad84-4a7b-8c49-4cf49829be99 --size 100 zabbix_boot_vda --project
> d1adca49347148f8bf10d3cd25f1dc7e

Right, this is bug #1559425. The OSC volume create command is trying to supply a --project value to cinder, and it's ignored in the cinder API. The OSC code in Stein was updated to reject the command, and advise the user to use the --os-project-id and --os-username options instead. Unfortunately, the OSP-13 code (Queens) silently passes the --project and --user values to cinder, where they're ignored.

We could certainly do a downstream backport of the Stein patch for OSP-13. Would that help?

> VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds
> allowed gigabytes quota. Requested 100G, quota is 1000G and 990G has been
> consumed. (HTTP 413) (Request-ID: req-3b02cc4a-d5fd-47a6-b729-c8ee736c2a03)
> ~~~
> 
> - They adjust the quota for said tenant but they have an error about their
> usage that is exceeding the desired quota. This usage is presumably the
> admin project:
> ~~~
> (overcloud) [stack@undercloud ~]$ openstack quota set --gigabytes 2000
> d1adca49347148f8bf10d3cd25f1dc7e
> Quota gigabytes limit must be equal or greater than existing resources.
> Current usage is 17170 and the requested limit is 2000. (HTTP 400)
> (Request-ID: req-eda5e7a0-ba98-41ab-b276-a51ef8956f09)
> ~~~
> 
> - They end up setting it to "20000" instead:
> ~~~
> (overcloud) [stack@undercloud ~]$ openstack quota set --gigabytes 20000
> d1adca49347148f8bf10d3cd25f1dc7e
> ~~~
> 
> - And they have the same error, with the same quota.
> ~~~
> (overcloud) [stack@undercloud ~]$ openstack volume create --image
> 5661d940-ad84-4a7b-8c49-4cf49829be99 --size 100 zabbix_boot_vda --project
> d1adca49347148f8bf10d3cd25f1dc7e

Yes, same error because the --project is ignored. But --os-project-id should work.

> VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds
> allowed gigabytes quota. Requested 100G, quota is 1000G and 990G has been
> consumed. (HTTP 413) (Request-ID: req-6fea1bcd-6e2a-4291-91bf-e86abba8560d)
> ~~~
> 
> So apparently the quota set command is also not passing the project to
> cinder, or cinder just ignores it.

That's entirely possible, but would need further investigation to confirm. The cinder code for each API call would need to be analyzed to see if it's mishandling (e.g. ignoring) any parameters. And I would be curious to know if OSC commands behave correctly when using --os-project-id and --os-username instead of --project and --user.

Also bear in mind the OSC volume commands don't support API microversions (this is a long standing point of contention). The 'cinder' command is necessary for a lot of things.

With all of that said, I'm not sure of the goal of this BZ. Can you identify something specific?

These are public responses to questions raised in comment #4 (currently private).

1. "openstack limits show --absolute --project foo" behavior

You can see cinder's own CLI working correctly in the BZ description's footnote [2]. The reason the OSC command does not appear to function correctly is the cinder API for displaying another project's absolute limits requires a microversion, and OSC does not support volume microversions.  Your patch [d] would also be necessary, but I'm not sure how you observed it working in queens without OSC microversion support. The patch alone will not be sufficient.

2. Support for creating volumes in another project

I went (way) back in time to 2012, and found the origin of the python-cinderclient code for the --project_id parameter [x]. It was added erroneously, but even so a comment in the code states it's the "Project id derived from context." Cinder itself only supports creating resources based on the request's context. There never was an intent to support creating resources in a project other than the one associated with the request.

[x] https://opendev.org/openstack/python-cinderclient/commit/778d2d90227771976f9b98ab686dbe1d88154806

As noted, it's is possible to use --os-project-id to override the default context, and this does not require and Administrator create a user inside each tenant's project, only that the admin be added as a user in the tenant's project (see [y]). I verified this sequence works, in which an admin creates a volume in the "demo" project:

(overcloud) [stack@rhos-undercloud ~]$ openstack --os-project-name demo volume create --size 1 demo-vol
The request you have made requires authentication. (HTTP 401) (Request-ID: req-1ce5d187-c44b-4deb-a38d-167bb123a96f)

(overcloud) [stack@rhos-undercloud ~]$ openstack role add --project demo --user admin admin

(overcloud) [stack@rhos-undercloud ~]$ openstack --os-project-name demo volume create --size 1 demo-vol
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2020-10-12T20:06:33.000000           |
| description         | None                                 |
| encrypted           | False                                |
| id                  | 9c61817b-31a8-4712-836a-f189c5b4aeaf |
| migration_status    | None                                 |
| multiattach         | False                                |
| name                | demo-vol                             |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 1                                    |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | tripleo                              |
| updated_at          | None                                 |
| user_id             | 3503e7b8f57a41f0b71b36f06bc83f83     |
+---------------------+--------------------------------------+

Another alternative would be to use trusts to impersonate one of the tenant's users. I was able to follow the steps outlined in [z] for an admin creating a volume in a project by impersonating (via a trust) a user in that project.

[y] https://stackoverflow.com/questions/36571228/openstack-can-admin-impersonate-as-another-user-to-create-a-vm
[z] https://blog.zhaw.ch/icclab/trust-delegation-in-openstack-using-keystone-trusts/

I don't know how other projects such as nova support creating resources in another project. For cinder, perhaps the two I listed above are viable but not as user friendly. Adding a third method would definitely require an API enhancement, and I'm not sure how the cinder community would embrace the idea.

Alan,

1) You're right about this, I just confirmed and the patch doesn't work in my lab. It was confirmed to work by one of my colleague in their lab, but maybe there was an error in the testing. So we will stick with the current solution of using cinderclient, or passing --os-project-id, I'll validate.

2) I'll test this and document in a KCS

3) I totally understand, this request wasn't coming from the customer so we can certainly put this on hold. I just personally think it would be great to have if this was standard across openstack projects because it would be "intuitively userfriendly".

Thanks for the analysis,

DVD

I just tried the trust impersonification methot and it doesn't work for me on OSP13 and OSP16.1.

First, I had a policy issue [1] that I fixed by adding this policy:
echo '"identity:create_trust": "role:admin or user_id:%(trust.trustor_user_id)s"' > /var/lib/config-data/puppet-generated/keystone/etc/keystone/policy.json && docker restart keystone

But now I get this error [2]. Apparently, we have a double validation for the trustor (the policy and this code bit [a]). I'll raise a keystone BZ and put the KCS [b] on hold for now. I'll write a new one for the --os-project-id one.


[a] https://opendev.org/openstack/keystone/src/branch/master/keystone/api/trusts.py#L286-L288
[b] https://access.redhat.com/solutions/5486601
[1]
~~~
(overcloud) [stack@undercloud-0 ~]$ openstack trust create --impersonate --project 412e7b23341c4a7dbce5d1aad3b23fd4 --role 146ce6560e2b4182950a7b88533e3c4f 17dab54e244f45fab1c0bc979bb10a1b 82db35a909714fe0b036adc18f49159a
You are not authorized to perform the requested action: identity:create_trust. (HTTP 403) (Request-ID: req-e128a508-1346-43b4-b973-bada2e1b6ef0)
~~~

[2]
~~~
(overcloud) [stack@undercloud-0 ~]$ openstack trust create --impersonate --project 412e7b23341c4a7dbce5d1aad3b23fd4 --role 146ce6560e2b4182950a7b88533e3c4f 17dab54e244f45fab1c0bc979bb10a1b 82db35a909714fe0b036adc18f49159a
You are not authorized to perform the requested action: The authenticated user should match the trustor. (HTTP 403) (Request-ID: req-af186bea-bade-4093-bb54-173e8bcf4c46)
~~~

The test I did that worked for me was on OSP-13. The key (no pun intended) is the trustor must be the one to create the trust. So if the goal is for an admin to create a volume in a tenant's project, the tenant will need to be the one who creates the trust. Yeah, this is not ideal from a usability perspective (I suspect the goal is for the admin to "take care of things" without needing to ask the tenant to create the trust).

Oh, and I did not need to tweak a policy to allow my test tenant to create the trust.

That's what I thought.

I don't think you needed the tweak the policies because you did the trust as the tenant, so that check in the code is a bit redundant with the policy. I don't know if we can remove this redundant check in the code or if there's a security concern here, I'll let the keystone team decide.

I'll prepare a KCS with --os-project-id until this is addressed.

Alan,

I think I summarized everything here in this article [a]. I can work on the downstream backport of this change [b].

Thanks for your help,

DVD

[a] https://access.redhat.com/solutions/5486601
[b] https://review.opendev.org/#/c/575804/

Excellent KCS!

What can we do now with this BZ? It has good background material, but maybe it can be closed NOTABUG or WORKSFORME?

Thanks Alan,

Well, can we use it to backport this [a]?

[a] https://review.opendev.org/#/c/575804/

Well, this BZ title makes it seem unrelated to the OSC's volume create command. For backporting the OSC patch to OSP-13 I recommend cloning bug #1559425.

You're totally right Alan, I'll close this one.

Thanks again,

DVD