Bug 1816918

Hi, please post your /var/log/audit/audit.log (grep for AVC) from the overcloud-novacompute-0 (or full sos report); so we can help and identify what rule is missing.

Some additional context since we don't have everything here.

This issue was also discussed on #tripleo this week. owalsh stepped in with some proposal and all.

The thing is:
- that trilio thing mounts an NFS share in a subdirectory of /var/lib/nova - this prevents the relabelling, because NFS
- this solution was apparently approved at some point by Red Hat eng

Solution
- one of the proposal made was to actually mount that NFS elsewhere in the containers (there are, iirc, 3 containers, including triliovault)
- in order to do so, a new param would be needed for the libvirt container, something like NovaLibvirtOptVolumes 
- then NovaComputeOptVolumes could be used in addition

Doing so would allow triliovault to work as expected, but would require some work in order to properly get the NFS share name (there's apparently some kind of hashing at some point for the name)

The current solution (mounting the share directly in /var/lib/nova) was working with Docker because the selinux separation was deactivated back then. The move to podman enforces selinux separation, therefore we had to add some flags to the shares, such as that "z" one, which requires a recursive relabelling of the volume.

NFS doesn't really support SELinux. We can pass a "context" (mount -t nfs -o context="...") but this won't make the relabelling work (recursion).... Moving the share elsewhere is probably the best move at this point.

@Shyam: would the proposed solution be OK for you? Adding a new param in order to get extras volumes in libvirt container? I know this means you'll need to rehash some data in order to provide the right path during deploy time, but it shouldn't be that complicated, right? Maybe you can even pre-hash things once and re-use that generated during the whole deploy (i.e. as a param).

Thank you for your feedback.

Cheers,

C. (aka Tengu on #tripleo)

Hi Team,


Moving NFS share to somewhere else is something very difficult. In that case we need to achive many things a the same time.
1. We need to make NFS share available to nova_compute container
2. We need to make NFS share available to nova_libvirt container
3. Mount point of NFS share is static, our datamover service calculates the hash for given NFS share and uses it as mount point.
4. This approach makes it less dynamic.

owalsh is proposing something else. Here is PR owalsh raised for this: https://review.opendev.org/#/c/715015/
You will get more details on the approach owalsh is using here.

Thank you.

Let me know if you need additional information.

Hello Shyam,

Thank you for your feedback. I've indeed seen Oliver proposal and it seems to solve your issue (among others).

I let Oliver manage this BZ as well (I've put him as assignee yesterday).

Cheers,

C.

Thank you Cedric.

*** Bug 1813941 has been marked as a duplicate of this bug. ***

According to our records, this should be resolved by openstack-tripleo-heat-templates-11.3.2-0.20200405044625.ec9970c.el8ost.  This build is available now.

Hi,


With this fix, our trilio_datamover container is not getting started. It's remained in 'Created' state.
I tried to start this container using 'podman start' command, it's failing following error.

[root@overcloud-novacompute-0 heat-admin]# podman ps --all | grep trilio
70464925affe  devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.91-rhosp16                    kolla_start           18 hours ago  Created                         trilio_datamover


[root@overcloud-novacompute-0 heat-admin]# podman start trilio_datamover
Error: unable to start container "trilio_datamover": relabel failed "/var/lib/nova": operation not supported



When this is happening:
We deployed 4.0.90 containers of Triliovault, it worked fine. trilio_datamover container started well. NFS share mounted under '/var/lib/nova/...'.
But when I tried to upgrade the cloud with 4.0.91 containers of triliovault, 4.0.91 trilio_datamover container is not getting started.
Overcloud deployment intermittently failing. 

Let us know your thoughts.

Hi,


Following is the error found in ansible logs. Deployment failed at step5 while starting 'trilio_datamover' container.

------------------------------------------------------------------------------------
2020-05-26 13:41:53,780 p=25153 u=mistral |  FAILED - RETRYING: Wait for containers to start for step 5 using paunch (1189 retries left).
2020-05-26 13:41:57,105 p=25153 u=mistral |  FAILED - RETRYING: Wait for containers to start for step 5 using paunch (1188 retries left).
2020-05-26 13:42:00,377 p=25153 u=mistral |  ok: [overcloud-controller-0] => {"action": ["Applying config_id tripleo_step5"], "ansible_job_id": "439545201027.925672", "attempts": 14, "changed": false, "finished": 1, "rc": 0, "stderr": "Did not find container with "['podman', 'ps', '-a', '--filter', 'label=container_name=cinder_volume_init_bundle', '--filter', 'label=config_id=tripleo_step5', '--format', '{{.Names}}']" - retrying without config_id\nDid not find container with "['podman', 'ps', '-a', '--filter', 'label=container_name=cinder_volume_init_bundle', '--format', '{{.Names}}']"\nRemoved /etc/systemd/system/multi-user.target.wants/tripleo_trilio_dmapi.service.\nDid not find container with "['podman', 'ps', '-a', '--filter', 'label=container_name=trilio_dmapi', '--filter', 'label=config_id=tripleo_step5', '--format', '{{.Names}}']" - retrying without config_id\nDid not find container with "['podman', 'ps', '-a', '--filter', 'label=container_name=trilio_dmapi', '--format', '{{.Names}}']"\nCreated symlink /etc/systemd/system/multi-user.target.wants/tripleo_trilio_dmapi.service → /etc/systemd/system/tripleo_trilio_dmapi.service.\n", "stderr_lines": ["Did not find container with "['podman', 'ps', '-a', '--filter', 'label=container_name=cinder_volume_init_bundle', '--filter', 'label=config_id=tripleo_step5', '--format', '{{.Names}}']" - retrying without config_id", "Did not find container with "['podman', 'ps', '-a', '--filter', 'label=container_name=cinder_volume_init_bundle', '--format', '{{.Names}}']"", "Removed /etc/systemd/system/multi-user.target.wants/tripleo_trilio_dmapi.service.", "Did not find container with "['podman', 'ps', '-a', '--filter', 'label=container_name=trilio_dmapi', '--filter', 'label=config_id=tripleo_step5', '--format', '{{.Names}}']" - retrying without config_id", "Did not find container with "['podman', 'ps', '-a', '--filter', 'label=container_name=trilio_dmapi', '--format', '{{.Names}}']"", "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_trilio_dmapi.service → /etc/systemd/system/tripleo_trilio_dmapi.service."], "stdout": "Info: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: Loading facts\nInfo: 
------------------------------------------------------------------------------------

(In reply to Shyam from comment #13)
> Hi,
> 
> 
> With this fix, our trilio_datamover container is not getting started. It's
> remained in 'Created' state.
> I tried to start this container using 'podman start' command, it's failing
> following error.
> 
> [root@overcloud-novacompute-0 heat-admin]# podman ps --all | grep trilio
> 70464925affe 
> devundercloud.ctlplane.localdomain:8787/trilio/trilio-datamover:4.0.91-
> rhosp16                    kolla_start           18 hours ago  Created      
> trilio_datamover
> 
> 
> [root@overcloud-novacompute-0 heat-admin]# podman start trilio_datamover
> Error: unable to start container "trilio_datamover": relabel failed
> "/var/lib/nova": operation not supported

This suggests /var/lib/nova is bind mounted with selinux relabelling enabled e.g /var/lib/nova:/var/lib/nova:shared,z
Change this to /var/lib/nova:/var/lib/nova:shared instead.

> 
> 
> 
> When this is happening:
> We deployed 4.0.90 containers of Triliovault, it worked fine.
> trilio_datamover container started well. NFS share mounted under
> '/var/lib/nova/...'.
> But when I tried to upgrade the cloud with 4.0.91 containers of triliovault,
> 4.0.91 trilio_datamover container is not getting started.

I doubt the version matters, just that the NFS mounts exist when the container is restarted.

> Overcloud deployment intermittently failing. 
> 
> Let us know your thoughts.

Hi,

Yes, we use 'shared,z' flags while mounting the '/var/lib/nova'.
Here is the code.
https://github.com/trilioData/triliovault-cfg-scripts/blob/stable/4.0/redhat-director-scripts/docker/services/trilio-datamover-osp16.yaml#L158

Let me try by removing 'z' flag.

Thank you.

(In reply to Shyam from comment #16)
> Hi,
> 
> Yes, we use 'shared,z' flags while mounting the '/var/lib/nova'.
> Here is the code.
> https://github.com/trilioData/triliovault-cfg-scripts/blob/stable/4.0/redhat-
> director-scripts/docker/services/trilio-datamover-osp16.yaml#L158
> 
> Let me try by removing 'z' flag.
> 
> Thank you.

Hi,

I can see this was committed in https://github.com/trilioData/triliovault-cfg-scripts/commit/55db096d5ab7496797dfed5c9f23670f119a74ba so I assume it works and closing the BZ.

Thanks,
Ollie