Description of problem: ___________________________ It looks like PEP8 job for OSP15-OVN is failing because of module deprecation: TASK [run_tests : Running tests] *********************************************** task path: /home/rhos-ci/jenkins/workspace/DFG-network-networking-ovn-15-pep8-rhos/infrared/plugins/octario/playbooks/roles/run_tests/tasks/main.yml:7 Sunday 19 May 2019 10:35:48 +0200 (0:00:13.034) 0:01:48.452 ************ fatal: [networking-ovn-35-1558254559-pep8-tester-0]: FAILED! => { "changed": false, "cmd": "export PATH=$PATH:/usr/sbin\n set -euo pipefail\n set -o pipefail; rpm -qa | sort > ../logs/installed-rpms.log;\ntruncate --size 0 requirements.txt; truncate --size 0 test-requirements.txt;\nsudo pip3 install tox;\ngit checkout -- test-requirements.txt requirements.txt; tox -v -e pep8 2>&1 | tee ../logs/testrun.log;\n", "delta": "0:02:34.959641", "end": "2019-05-19 04:38:25.345856", "rc": 1, "start": "2019-05-19 04:35:50.386215" } STDOUT: Collecting tox Downloading https://files.pythonhosted.org/packages/8b/38/71c2fe0c3915fc0e93bdd1bf8cd697be48cdacedbdcd438e0f0629c69024/tox-3.11.1-py2.py3-none-any.whl (76kB) Requirement already satisfied: virtualenv>=14.0.0 in /usr/lib/python3.6/site-packages (from tox) Requirement already satisfied: pluggy<1,>=0.3.0 in /usr/lib/python3.6/site-packages (from tox) Requirement already satisfied: setuptools>=30.0.0 in /usr/lib/python3.6/site-packages (from tox) Requirement already satisfied: six<2,>=1.0.0 in /usr/lib/python3.6/site-packages (from tox) Collecting toml>=0.9.4 (from tox) Downloading https://files.pythonhosted.org/packages/a2/12/ced7105d2de62fa7c8fb5fce92cc4ce66b57c95fb875e9318dba7f8c5db0/toml-0.10.0-py2.py3-none-any.whl Collecting filelock<4,>=3.0.0 (from tox) Downloading https://files.pythonhosted.org/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl Requirement already satisfied: py<2,>=1.4.17 in /usr/lib/python3.6/site-packages (from tox) Installing collected packages: toml, filelock, tox Successfully installed filelock-3.0.12 toml-0.10.0 tox-3.11.1 using tox.ini: /home/cloud-user/networking-ovn/tox.ini (pid 23234) using tox-3.11.1 from /usr/local/lib/python3.6/site-packages/tox/__init__.py (pid 23234) pep8 cannot reuse: no previous config /home/cloud-user/networking-ovn/.tox/shared/.tox-config1 pep8 create: /home/cloud-user/networking-ovn/.tox/shared [23257] /home/cloud-user/networking-ovn/.tox$ /usr/bin/python3.6 -m virtualenv --no-download --python /usr/bin/python3.6 shared >shared/log/pep8-0.log pep8 installdeps: -chttps://releases.openstack.org/constraints/upper/stein, -r/home/cloud-user/networking-ovn/requirements.txt, -r/home/cloud-user/networking-ovn/test-requirements.txt [23264] /home/cloud-user/networking-ovn$ /home/cloud-user/networking-ovn/.tox/shared/bin/pip install -chttps://releases.openstack.org/constraints/upper/stein -r/home/cloud-user/networking-ovn/requirements.txt -r/home/cloud-user/networking-ovn/test-requirements.txt >.tox/shared/log/pep8-1.log pep8 develop-inst: /home/cloud-user/networking-ovn write config to /home/cloud-user/networking-ovn/.tox/shared/.tox-config1 as 'cb8411eddb57a0beb4829d05ab9c45e1 /usr/bin/python3.6\n3.11.1 0 1 0\n00000000000000000000000000000000 -chttps://releases.openstack.org/constraints/upper/stein\n00000000000000000000000000000000 -r/home/cloud-user/networking-ovn/requirements.txt\n00000000000000000000000000000000 -r/home/cloud-user/networking-ovn/test-requirements.txt' [23546] /home/cloud-user/networking-ovn$ /home/cloud-user/networking-ovn/.tox/shared/bin/pip install --exists-action w -e . >.tox/shared/log/pep8-2.log [23581] /home/cloud-user/networking-ovn$ /home/cloud-user/networking-ovn/.tox/shared/bin/python -m pip freeze >.tox/shared/log/pep8-3.log pep8 installed: You are using pip version 9.0.1, however version 19.1.1 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,alembic==1.0.8,amqp==2.4.2,appdirs==1.4.3,asn1crypto==0.24.0,astroid==2.1.0,Babel==2.6.0,bandit==1.6.0,bcrypt==3.1.6,beautifulsoup4==4.7.1,cachetools==3.1.0,certifi==2018.11.29,cffi==1.12.2,chardet==3.0.4,cliff==2.14.1,cmd2==0.8.9,coverage==4.5.2,cryptography==2.6.1,debtcollector==1.21.0,decorator==4.3.2,dnspython==1.15.0,dogpile.cache==0.7.1,eventlet==0.24.1,extras==1.0.0,fasteners==0.14.1,fixtures==3.0.0,flake8==2.6.2,flake8-import-order==0.12,future==0.17.1,futurist==1.8.1,gitdb2==2.0.5,GitPython==2.1.11,greenlet==0.4.15,hacking==1.1.0,httplib2==0.12.1,idna==2.8,iso8601==0.1.12,isort==4.3.20,Jinja2==2.10,jmespath==0.9.4,jsonpatch==1.23,jsonpointer==2.0,jsonschema==2.6.0,keystoneauth1==3.13.1,keystonemiddleware==6.0.0,kombu==4.4.0,lazy-object-proxy==1.4.1,linecache2==1.0.0,logutils==0.3.5,Mako==1.0.7,MarkupSafe==1.1.1,mccabe==0.6.1,mock==2.0.0,monotonic==1.5,mox3==0.27.0,msgpack==0.6.1,munch==2.3.2,netaddr==0.7.19,netifaces==0.10.9,-e git+https://code.engineering.redhat.com/gerrit/networking-ovn.git@e5d6d099e60dd5ef252cc2e04bd44307b44bca3a#egg=networking_ovn,neutron==14.0.1,neutron-lib==1.25.0,octavia-lib==1.1.1,openstacksdk==0.27.0,os-client-config==1.32.0,os-ken==0.3.1,os-service-types==1.6.0,os-testr==1.0.0,os-traits==0.11.0,os-xenapi==0.3.4,osc-lib==1.12.1,oslo.cache==1.33.3,oslo.concurrency==3.29.1,oslo.config==6.8.1,oslo.context==2.22.1,oslo.db==4.45.0,oslo.i18n==3.23.1,oslo.log==3.42.3,oslo.messaging==9.5.0,oslo.middleware==3.37.1,oslo.policy==2.1.1,oslo.privsep==1.32.1,oslo.reports==1.29.2,oslo.rootwrap==5.15.2,oslo.serialization==2.28.2,oslo.service==1.38.0,oslo.upgradecheck==0.2.1,oslo.utils==3.40.3,oslo.versionedobjects==1.35.1,oslotest==3.7.1,osprofiler==2.6.0,ovs==2.10.0,ovsdbapp==0.15.0,paramiko==2.4.2,Paste==3.0.7,PasteDeploy==2.0.1,pbr==5.1.3,pecan==1.3.2,prettytable==0.7.2,psutil==5.6.0,pyasn1==0.4.5,pycadf==2.9.0,pycodestyle==2.5.0,pycparser==2.19,pyflakes==1.2.3,pyinotify==0.9.6,pylint==2.3.0,PyNaCl==1.3.0,pyOpenSSL==19.0.0,pyparsing==2.3.1,pyperclip==1.7.0,pyroute2==0.5.4,python-dateutil==2.8.0,python-designateclient==2.11.0,python-editor==1.0.4,python-keystoneclient==3.19.0,python-mimeparse==1.6.0,python-neutronclient==6.12.0,python-novaclient==13.0.0,python-subunit==1.3.0,pytz==2018.9,PyYAML==3.13,repoze.lru==0.7,requests==2.21.0,requestsexceptions==1.4.0,rfc3986==1.2.0,Routes==2.4.1,setproctitle==1.1.10,simplejson==3.16.0,six==1.12.0,smmap2==2.0.5,sortedcontainers==2.1.0,soupsieve==1.8,SQLAlchemy==1.2.18,sqlalchemy-migrate==0.12.0,sqlparse==0.2.4,statsd==3.3.0,stestr==2.2.0,stevedore==1.30.1,Tempita==0.5.2,tenacity==5.0.3,testresources==2.0.1,testscenarios==0.5.0,testtools==2.3.0,tinyrpc==0.9.4,traceback2==1.4.0,typed-ast==1.3.1,unittest2==1.1.0,urllib3==1.24.1,vine==1.2.0,voluptuous==0.11.5,waitress==1.2.1,wcwidth==0.1.7,WebOb==1.8.5,WebTest==2.0.33,wrapt==1.11.1,yappi==1.0 pep8 run-test-pre: PYTHONHASHSEED='1239935353' pep8 run-test: commands[0] | flake8 [23589] /home/cloud-user/networking-ovn$ /home/cloud-user/networking-ovn/.tox/shared/bin/flake8 pep8 run-test: commands[1] | /home/cloud-user/networking-ovn/tools/coding-checks.sh --pylint '' [23603] /home/cloud-user/networking-ovn$ /home/cloud-user/networking-ovn/tools/coding-checks.sh --pylint '' Running pylint... You can speed this up by running it on 'HEAD~[0-9]' (e.g. HEAD~1, this change only)... /home/cloud-user/networking-ovn/.tox/shared/lib/python3.6/site-packages/astroid/interpreter/_import/spec.py:12: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp <unknown>:76: DeprecationWarning: invalid escape sequence \. ------------------------------------ Your code has been rated at 10.00/10 pep8 run-test: commands[2] | neutron-db-manage --subproject=networking-ovn check_migration [23605] /home/cloud-user/networking-ovn$ /home/cloud-user/networking-ovn/.tox/shared/bin/neutron-db-manage --subproject=networking-ovn check_migration Running branches for networking-ovn ... initial_branchpoint (branchpoint) -> ac094507b7f4 (expand) -> 1d271ead4eb6 (contract) (head) OK pep8 run-test: commands[3] | mkdir -p etc/neutron/plugins/ml2 WARNING: test command found but not installed in testenv cmd: /usr/bin/mkdir env: /home/cloud-user/networking-ovn/.tox/shared Maybe you forgot to specify a dependency? See also the whitelist_externals envconfig setting. DEPRECATION WARNING: this will be an error in tox 4 and above! [23607] /home/cloud-user/networking-ovn$ /usr/bin/mkdir -p etc/neutron/plugins/ml2 pep8 run-test: commands[4] | oslo-config-generator --config-file etc/oslo-config-generator/ml2_conf.ini [23608] /home/cloud-user/networking-ovn$ /home/cloud-user/networking-ovn/.tox/shared/bin/oslo-config-generator --config-file etc/oslo-config-generator/ml2_conf.ini /home/cloud-user/networking-ovn/.tox/shared/lib/python3.6/site-packages/eventlet/patcher.py:1: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp pep8 run-test: commands[5] | oslo-config-generator --config-file etc/oslo-config-generator/networking_ovn_metadata_agent.ini [23614] /home/cloud-user/networking-ovn$ /home/cloud-user/networking-ovn/.tox/shared/bin/oslo-config-generator --config-file etc/oslo-config-generator/networking_ovn_metadata_agent.ini /home/cloud-user/networking-ovn/.tox/shared/lib/python3.6/site-packages/eventlet/patcher.py:1: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp pep8 run-test: commands[6] | bandit -r networking_ovn -x tests -n5 -s B104 [23619] /home/cloud-user/networking-ovn$ /home/cloud-user/networking-ovn/.tox/shared/bin/bandit -r networking_ovn -x tests -n5 -s B104 [main] INFO profile include tests: None [main] INFO profile exclude tests: None [main] INFO cli include tests: None [main] INFO cli exclude tests: B104 [main] INFO running on Python 3.6.8 114 [0.. 50.. 100.. ] Run started:2019-05-19 08:38:25.166533 Test results: >> Issue: [B108:hardcoded_tmp_directory] Probable insecure usage of temp file/directory. Severity: Medium Confidence: Medium Location: networking_ovn/tests/functional/base.py:47 More Info: https://bandit.readthedocs.io/en/latest/plugins/b108_hardcoded_tmp_directory.html 45 46 # This is the directory from which infra fetches log files for functional tests 47 DEFAULT_LOG_DIR = os.path.join(os.environ.get('OS_LOG_PATH', '/tmp'), 48 'dsvm-functional-logs') 49 SQL_FIXTURE_LOCK = 'sql_fixture_lock' -------------------------------------------------- >> Issue: [B110:try_except_pass] Try, Except, Pass detected. Severity: Low Confidence: High Location: networking_ovn/tests/functional/resources/process.py:64 More Info: https://bandit.readthedocs.io/en/latest/plugins/b110_try_except_pass.html 62 stop_cmd = ['ovs-appctl', '-t', self.unixctl_path, 'exit'] 63 utils.execute(stop_cmd) 64 except Exception: 65 pass 66 -------------------------------------------------- >> Issue: [B110:try_except_pass] Try, Except, Pass detected. Severity: Low Confidence: High Location: networking_ovn/tests/functional/resources/process.py:189 More Info: https://bandit.readthedocs.io/en/latest/plugins/b110_try_except_pass.html 187 'exit'] 188 utils.execute(stop_cmd) 189 except Exception: 190 pass 191 -------------------------------------------------- >> Issue: [B106:hardcoded_password_funcarg] Possible hardcoded password: 'secret' Severity: Low Confidence: Medium Location: networking_ovn/tests/unit/agent/metadata/test_server.py:37 More Info: https://bandit.readthedocs.io/en/latest/plugins/b106_hardcoded_password_funcarg.html 35 meta_conf.register_meta_conf_opts( 36 meta_conf.METADATA_PROXY_HANDLER_OPTS, self.conf) 37 self.config(auth_ca_cert=None, 38 nova_metadata_host='9.9.9.9', 39 nova_metadata_port=8775, 40 metadata_proxy_shared_secret='secret', 41 nova_metadata_protocol='http', 42 nova_metadata_insecure=True, 43 nova_client_cert='nova_cert', 44 nova_client_priv_key='nova_priv_key') 45 46 -------------------------------------------------- >> Issue: [B311:blacklist] Standard pseudo-random generators are not suitable for security/cryptographic purposes. Severity: Low Confidence: High Location: networking_ovn/tests/unit/l3/test_l3_ovn_scheduler.py:93 More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b311-random 91 def test_no_chassis_available_for_existing_gateway(self): 92 mapping = self.fake_chassis_gateway_mappings['None'] 93 gateway_name = random.choice(list(mapping['Gateways'].keys())) 94 chassis = self.select(mapping, gateway_name) 95 self.assertEqual([ovn_const.OVN_GATEWAY_INVALID_CHASSIS], chassis) -------------------------------------------------- >> Issue: [B311:blacklist] Standard pseudo-random generators are not suitable for security/cryptographic purposes. Severity: Low Confidence: High Location: networking_ovn/tests/unit/l3/test_l3_ovn_scheduler.py:111 More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b311-random 109 def test_existing_chassis_available_for_existing_gateway(self): 110 mapping = self.fake_chassis_gateway_mappings['Multiple1'] 111 gateway_name = random.choice(list(mapping['Gateways'].keys())) 112 chassis = self.select(mapping, gateway_name) 113 self.assertEqual(mapping['Gateways'][gateway_name], chassis) -------------------------------------------------- >> Issue: [B311:blacklist] Standard pseudo-random generators are not suitable for security/cryptographic purposes. Severity: Low Confidence: High Location: networking_ovn/tests/unit/l3/test_l3_ovn_scheduler.py:124 More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b311-random 122 def test_no_chassis_available_for_existing_gateway(self): 123 mapping = self.fake_chassis_gateway_mappings['None'] 124 gateway_name = random.choice(list(mapping['Gateways'].keys())) 125 chassis = self.select(mapping, gateway_name) 126 self.assertEqual([ovn_const.OVN_GATEWAY_INVALID_CHASSIS], chassis) -------------------------------------------------- >> Issue: [B311:blacklist] Standard pseudo-random generators are not suitable for security/cryptographic purposes. Severity: Low Confidence: High Location: networking_ovn/tests/unit/l3/test_l3_ovn_scheduler.py:159 More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b311-random 157 def test_existing_chassis_available_for_existing_gateway(self): 158 mapping = self.fake_chassis_gateway_mappings['Multiple1'] 159 gateway_name = random.choice(list(mapping['Gateways'].keys())) 160 chassis = self.select(mapping, gateway_name) 161 self.assertEqual(mapping['Gateways'][gateway_name], chassis) -------------------------------------------------- Code scanned: Total lines of code: 24042 Total lines skipped (#nosec): 0 Run metrics: Total issues (by severity): Undefined: 0.0 Low: 7.0 Medium: 1.0 High: 0.0 Total issues (by confidence): Undefined: 0.0 Low: 0.0 Medium: 2.0 High: 6.0 Files skipped (0): ERROR: InvocationError for command /home/cloud-user/networking-ovn/.tox/shared/bin/bandit -r networking_ovn -x tests -n5 -s B104 (exited with code 1) ___________________________________ summary ____________________________________ ERROR: pep8: commands failed STDERR: WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead. MSG: non-zero return code Version-Release number of selected component (if applicable): ___________________________ OSP15 > git fetch --tags --progress https://code.engineering.redhat.com/gerrit/networking-ovn.git +refs/heads/rhos-15.0-patches:refs/remotes/origin/rhos-15.0-patches > git rev-parse FETCH_HEAD^{commit} # timeout=10 Checking out Revision e5d6d099e60dd5ef252cc2e04bd44307b44bca3a (rhos-15.0-patches) > git config core.sparsecheckout # timeout=10 > git checkout -f e5d6d099e60dd5ef252cc2e04bd44307b44bca3a Commit message: "RHOS: use internal gerrit - DROP-IN-RPM" > git rev-parse FETCH_HEAD^{commit} # timeout=10 > git rev-list --no-walk 2769a9c09be3da66d808a7fa8924704878cdc795 # timeout=10 Job logs: ___________________________ https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-15-pep8-rhos/35/consoleFull How reproducible: ___________________________ 100% Steps to Reproduce: ___________________________ 1. run job : https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-15-pep8-rhos/
This error happens because of a regression in the bandit library version 1.6.0. It's already been fixed in the 1.6.1 version [0] Upstream in networking-ovn we are skipping the 1.6.0: https://review.opendev.org/#/c/658234/ [0] https://github.com/PyCQA/bandit/issues/488
So there is a fix for this upstream, and in the downstream rhos-15.0-trunk-patches networking-ovn repo, 9e8a199871b2a4c9cb56bc91a8d3e6e88e3f5fee That branch is closely tracking trunk and being used for the latest beta builds. The rhos-15.0-patches branch doesn't have this fix yet, but should eventually be synced to the 'trunk' branch, which is something reldel is discussing. Is this causing issues or just something you noticed, as I'm inclined to just wait for the sync instead of cherry-picking into rhos-15.0-patches. I did also send an upstream patch to cap bandit at <1.6.0 in stable/stein since that will prevent us from getting broken in the future, instead of blacklisting 1.6.0.
Cherry-picked change from trunk.
The downstream rhos-15.0-patches branch was rebased to pick-up this change, so this issue should be fixed.
Ignore my last comment, doesn't seem to be fixed in this branch yet.
fix verified https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-15-pep8-rhos/48/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811