Bug 1752148

env : seal15.qa.lab.tlv.redhat.com

sosreports and stack home are at : http://rhos-release.virt.bos.redhat.com/log/pkomarov_sosreports/BZ_1752148/

After looking at the env with Herve, we see that the pkg_resources module is badly imported on that env:

$ python3 -s
Python 3.6.8 (default, Jun 12 2019, 01:12:31) 
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>> pkg_resources
<module 'pkg_resources' (namespace)>
>>> pkg_resources.__path__
_NamespacePath(['/usr/lib/python3.6/site-packages/pkg_resources'])
>>> dir(pkg_resources)
['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']

This is due to the fact that on this directory, the package has been removed, there's no python __init__.py to load anymore 

$ ls /usr/lib/python3.6/site-packages/pkg_resources
extern  __pycache__  _vendor

Oddly, the package which ships the missing file is still installed

$ rpm -qf /usr/lib/python3.6/site-packages/pkg_resources
platform-python-setuptools-39.2.0-4.el8.noarch

$ rpm -qa platform-python-setuptools
platform-python-setuptools-39.2.0-4.el8.noarch


But something has deleted files from /usr/lib/python3.6/site-packages

rpm -Va platform-python-setuptools | grep missing | wc -l
181


We also noticed that some python packages has been installed under /usr/local/lib/python3.6/site-packages, which indicates that a pip command has most probably been run manually. 
$ python3
Python 3.6.8 (default, Jun 12 2019, 01:12:31) 
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>> pkg_resources
<module 'pkg_resources' from '/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py'>


At this point it's still unclear whether pip could have deleted the files from the original package, but I doubt this is due to the minor update steps themselves, because nothing has removed package platform-python-setuptools.

Just to complete the previous Damien' message, notice that we use the option `-s` in the first case (the broken use case) because the shebang of the `tripleo-ansible-inventory` use the `-s` option too:

```
!#/usr/bin/python3 -s
```

The `-s` option IS NOT AN ISSUE and it work perfectly on other env, but in our case the `-s` option introduce some behaviours within
the python paths loaded that we need to notice to understand things more clearly.


The `-s` option don’t add the user site-packages directory to sys.path [1]:
```
(undercloud) [stack@undercloud-0 site-packages]$ python3 -s -c 'import sys;print(sys.path)'
['', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/usr/lib64/py
thon3.6/site-packages', '/usr/lib/python3.6/site-packages']
(undercloud) [stack@undercloud-0 site-packages]$ python3 -c 'import sys;print(sys.path)'
['', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/usr/local/lib/python3.6/site-packages', '/usr/lib64/python3.6/site-packages', '/usr/lib/python3.6/site-packages']
```

So, when we use the `-s` we don't add the `/usr/local/lib/python3.6/site-packages` to your path and so we retrieve `pkg_resources` from `/usr/lib/python3.6/site-packages/pkg_resources` in this case. Notice that without the `-s` `/usr/local/lib/python3.6/site-packages` is before 
`/usr/lib/python3.6/site-packages` so when we import things, then python retrieve things first from `/usr/local/lib/python3.6/site-packages` there.

The sys.path is a python list so it is ordered and if something loopover it, then it will retrieve first from `/usr/local/lib/python3.6/site-packages`, this is why it work in the previous Damien's example. Also it work because something have installed some package under this path: 
`/usr/local/lib/python3.6/site-packages` (in your case `pkg_resources`).

It can be useful to also check if we use some specific pip config there:
```
pip config list
```

[1] https://docs.python.org/3.6/using/cmdline.html#cmdoption-s

Does not reproduce 100% , during undercloud update , the tripleo-ansible-inventory works without breaking...
[stack@undercloud-0 ~]$ cat core_puddle_version 
RHOS_TRUNK-15.0-RHEL-8-20190913.n.3[stack@undercloud-0 ~]$ 
[stack@undercloud-0 ~]$ 
[stack@undercloud-0 ~]$  /usr/bin/tripleo-ansible-inventory --help
usage: tripleo-ansible-inventory [-h]
                                 [--ansible_python_interpreter ANSIBLE_PYTHON_INTERPRETER]
                                 [--ansible_ssh_user ANSIBLE_SSH_USER]
                                 [--auth-token AUTH_TOKEN]
                                 [--auth-url AUTH_URL] [--cacert CACERT]
[...]

optional arguments:
  -h, --help            show this help message and exit
  --ansible_python_interpreter ANSIBLE_PYTHON_INTERPRETER
  --ansible_ssh_user ANSIBLE_SSH_USER
[...]

does not reproduce , composable roles tests are passing ...
https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-pidone-updates-15_director-rhel-virthost-3cont_3db_3msg_2net_2comp-ipv4-geneve-ansible-sts-composable_roles/26/artifact/.sh/ansible_sts-ha-tests.log

Seeing this error on osp16 : 


[stack@undercloud-0 ~]$ head -n 1 `which tripleo-ansible-inventory`
#!/usr/bin/python3 -s

[stack@undercloud-0 ~]$ tripleo-ansible-inventory --help
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pbr/version.py", line 441, in _get_version_from_pkg_resources
    requirement = pkg_resources.Requirement.parse(self.package)
AttributeError: module 'pkg_resources' has no attribute 'Requirement'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/tripleo-ansible-inventory", line 29, in <module>
    from oslo_config import cfg
  File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 53, in <module>
    from oslo_config import types
  File "/usr/lib/python3.6/site-packages/oslo_config/types.py", line 28, in <module>
    from debtcollector import removals
  File "/usr/lib/python3.6/site-packages/debtcollector/__init__.py", line 20, in <module>
    'debtcollector').version_string()
  File "/usr/lib/python3.6/site-packages/pbr/version.py", line 467, in version_string
    return self.semantic_version().brief_string()
  File "/usr/lib/python3.6/site-packages/pbr/version.py", line 462, in semantic_version
    self._semantic = self._get_version_from_pkg_resources()
  File "/usr/lib/python3.6/site-packages/pbr/version.py", line 444, in _get_version_from_pkg_resources
    except pkg_resources.DistributionNotFound:
AttributeError: module 'pkg_resources' has no attribute 'DistributionNotFound'

#but when we remove the '-s' from the python shabang: 

[stack@undercloud-0 ~]$ sudo vi `which tripleo-ansible-inventory`

#it works : 
[stack@undercloud-0 ~]$ head -n 1 `which tripleo-ansible-inventory`
#!/usr/bin/python3
[stack@undercloud-0 ~]$ tripleo-ansible-inventory --help
usage: tripleo-ansible-inventory [-h]
                                 [--ansible_python_interpreter ANSIBLE_PYTHON_INTERPRETER]
                                 [--ansible_ssh_user ANSIBLE_SSH_USER]
                                 [--auth-token AUTH_TOKEN]

Is there any env where I can look ?
I'm trying to reproduce it, but with no luck atm.

This problem looks to be the exact same as what I wrote in comment #3. pip got installed and a log of packages got installed in /usr/local rather that consuming rpms directly.

Now directory pkg_resources from platform-python-setuptools-39.2.0-5.el8.noarch has been partially removed and makes  
tripleo-ansible-inventory load a partial module, which breaks the command.

The only thing I see right now is that the directory seems to have been cleaned up on 2019-12-10 08:53:55:

[stack@undercloud-0 ~]$ stat /usr/lib/python3.6/site-packages/pkg_resources
  File: /usr/lib/python3.6/site-packages/pkg_resources
  Size: 54        	Blocks: 0          IO Block: 4096   directory
Device: fc01h/64513d	Inode: 336445      Links: 5
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:lib_t:s0
Access: 2019-12-23 10:46:37.349816563 +0000
Modify: 2019-12-10 08:53:55.690317993 +0000
Change: 2019-12-10 08:53:55.690317993 +0000

But I don't know for sure what command ran in the QE job at that time. Pini can you correlate that deployment run with your QE job and check what triggered the installation of pip?

I believe the failure is not indicative of a problem in RHOSP16 per se, but rather a job setup issue.

Thanks Damen , I think you're right , I retested it and it seems not a bug on a fresh deployment .

https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Phase3/view/OSP%2016/view/PidOne/job/DFG-pidone-sanity-16_director-rhel-virthost-3cont_2comp-ipv4-geneve-ansible-sts-sanity/9/

closing..