Bug 1547065
Summary: | Failed to create OVN Logical Switch names with unicode characters 2.9.0 | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Marcin Mirecki <mmirecki> | ||||
Component: | openvswitch | Assignee: | Timothy Redaelli <tredaelli> | ||||
Status: | CLOSED ERRATA | QA Contact: | haidong li <haili> | ||||
Severity: | medium | Docs Contact: | |||||
Priority: | medium | ||||||
Version: | 7.7 | CC: | atragler, ctrautma, danken, jhsiao, jsitnick, mmirecki, pvauter, tredaelli | ||||
Target Milestone: | rc | Keywords: | Reopened | ||||
Target Release: | --- | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | openvswitch-2.9.0-25.el7fdn | Doc Type: | If docs needed, set a value | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2018-06-21 13:36:35 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1527828 | ||||||
Attachments: |
|
Description
Marcin Mirecki
2018-02-20 13:02:15 UTC
Selinux policies should be ok for both 7.5 and 7.4.z so why can't you update to 2.8 or 2.9? I verified the problem with 2.9. It worked fine with my own build of 2.9, but I still see the problem with a version from brew. It failed with the same issue in our system test: https://rhv-devops-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/rhv-system-tests/job/ds-ovirt-system-tests_manual/291/ which used the following 2.0 build: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=655288 The error can be seen here: https://rhv-devops-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/rhv-system-tests/job/ds-ovirt-system-tests_manual/291/artifact/exported-artifacts/test_logs/rhv-suite-4.2/post-098_ovirt_provider_ovn.py/lago-rhv-suite-4-2-engine/_var_log/ovirt-provider-ovn.log The error seen is: BadRequestError: 'ascii' codec can't decode byte 0xe3 in position 87: ordinal not in range(128) The openvswitch logs are here: https://rhv-devops-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/rhv-system-tests/job/ds-ovirt-system-tests_manual/291/artifact/exported-artifacts/test_logs/rhv-suite-4.2/post-098_ovirt_provider_ovn.py/lago-rhv-suite-4-2-engine/_var_log/openvswitch/ The name used to fail the test: u'ネットワーク_1' Could you please look at this? Verified with 2.9.0-8.el7 and it works fine. The automatic test failure was a result of a problem with package name provides in python-openvswitch (2.7 had python2-openvswitch and was not updated with 2.9 which has python-openvswitch). Reopening due to Marcin reports that RHEV QA is still seeing the failures: http://pastebin.test.redhat.com/569135 Reproduced outside of ovirt-provider-ovn. Interestingly the error that points to the root cause is missing from ovirt-provider-ovn logs. May have something to do with logging handlers. # ./repro.py ERROR:ovsdbapp.backend.ovs_idl.transaction:Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 100, in run txn.results.put(txn.do_commit()) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 92, in do_commit status = txn.commit_block() File "/usr/lib/python2.7/site-packages/ovs/db/idl.py", line 1402, in commit_block status = self.commit() File "/usr/lib/python2.7/site-packages/ovs/db/idl.py", line 1385, in commit if not self.idl._session.send(msg): File "/usr/lib/python2.7/site-packages/ovs/jsonrpc.py", line 539, in send return self.rpc.send(msg) File "/usr/lib/python2.7/site-packages/ovs/jsonrpc.py", line 244, in send self.run() File "/usr/lib/python2.7/site-packages/ovs/jsonrpc.py", line 203, in run retval = self.stream.send(self.output) File "/usr/lib/python2.7/site-packages/ovs/stream.py", line 808, in send return super(SSLStream, self).send(buf) File "/usr/lib/python2.7/site-packages/ovs/stream.py", line 391, in send buf = buf.encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 85: ordinal not in range(128) Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File "/usr/lib64/python2.7/threading.py", line 765, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 96, in run self.idl.run() File "/usr/lib/python2.7/site-packages/ovs/db/idl.py", line 179, in run assert not self.txn AssertionError ERROR:ovsdbapp.backend.ovs_idl.command:Error executing command Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 35, in execute txn.add(self) File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 94, in transaction self._nested_txn = None File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 54, in __exit__ self.result = self.commit() File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit raise result.ex UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 85: ordinal not in range(128) Traceback (most recent call last): File "./repro.py", line 40, in <module> main() File "./repro.py", line 36, in main add_switch(idl) File "./repro.py", line 30, in add_switch cmd.execute(check_error=True) File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 35, in execute txn.add(self) File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 94, in transaction self._nested_txn = None File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 54, in __exit__ self.result = self.commit() File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit raise result.ex UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 85: ordinal not in range(128) Enviroment: # cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.5 (Maipo) # rpm -qf /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py python-ovsdbapp-0.6.0-1.el7ev.noarch # rpm -qf /usr/lib/python2.7/site-packages/ovs/stream.py python-openvswitch-2.9.0-1.el7fdp.noarch Created attachment 1417334 [details]
Reproducer for the bug.
Fix needed upstream. Changes from: commit 2254074e3067fd306c37ac80347ef75075b40a89 Author: Alin Balutoiu <abalutoiu> Date: Wed Aug 16 15:01:39 2017 +0000 python: fix python3 encode/decode on Windows are conflicting with an earlier patch: commit d90ed7d65ba8cfbdada2286f5adb167ca26048c9 Author: Numan Siddique <nusiddiq> Date: Wed Oct 5 17:50:24 2016 +0530 python: Add SSL support to the python ovs client library and we end up attempting to encode a Unicode string twice. Fix (with regression test) posted upstream: https://mail.openvswitch.org/pipermail/ovs-dev/2018-April/346248.html Fix is now backported to 2.9 branch upstream: https://github.com/openvswitch/ovs/commit/cef8e1fa7a8f030ca221f8cac5893c686b9d0f00 Can you supply a scratch build ovs-2.9+this patch for us to verify? (In reply to Dan Kenigsberg from comment #13) > Can you supply a scratch build ovs-2.9+this patch for us to verify? Yes, will do. This is what I should have done. Jakub, did you intend to add a URL to the scratch build? Yes, I'm on it. I apologize for the delay. would you clear the needinfo? only when the needed info is provided? it helps me notice if action is needed on me. (In reply to Dan Kenigsberg from comment #17) > would you clear the needinfo? only when the needed info is provided? it > helps me notice if action is needed on me. Understood. Sorry for the confusion. Here's a scratch build for RHEL-7.5: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=15904507 I'm not sure what RHEL version you would like to test with. BZ target is set for 7.7 but we don't have that release in Brew yet. Please let me know if you need another scratch build targeted for a different release. Once I get a green light from RHV team, and manage to collect all needed flags (pm_ack, fast‑datapath‑rhel‑7 ack, qa_ack) I will backport it to FDP. Correction, I believe the backport first has to go to Fast Datapath Next, not FDP. (In reply to Jakub Sitnicki from comment #18) > Here's a scratch build for RHEL-7.5: > > https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=15904507 Link to yum repo: http://brew-task-repos.usersys.redhat.com/repos/scratch/jsitnick/openvswitch/2.9.0/19.el7_5.bz1547065.1/openvswitch-2.9.0-19.el7_5.bz1547065.1-scratch.repo I just fired https://rhv-devops-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/ds-ovirt-system-tests_manual/310/ to test this (I must admit that I am not sure if I filled all the right data for this to succeed). (I have not ran it properly, as openvswitch-2.9.0-19.el7_5.bz1547065.1 was not pulled in by the job.) Since this bug has resurfaced after being solved before, I am asking to add qe coverage for it. I can confirm that your scratch build supports creating of unicode-named logical switches. (ovirt, however, seem to have a bug elsewhere in enumerating them.) (In reply to Dan Kenigsberg from comment #24) > I can confirm that your scratch build supports creating of unicode-named > logical switches. (ovirt, however, seem to have a bug elsewhere in > enumerating them.) Thank you. I will backport the fix to Fast Datapath Next branch, from there it will be automatically pulled into Fast Datapath Production branch for next release - FDP 18.06 (in planning for 19 June 2018). In case this doesn't work for RHV, please let us know. Regarding QE coverage, I can chip in that regression tests have been added upstream but they have not been backported to 2.9 because they were not trivial additions (required some test suite refactoring). https://mail.openvswitch.org/pipermail/ovs-dev/2018-April/346248.html Ovirt system tests pass using this change: https://rhv-devops-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/rhv-system-tests/job/ds-ovirt-system-tests_manual/315/ The tests needed a small adjustment to handle unicode properly too: https://code.engineering.redhat.com/gerrit/#/c/137482/ I used the build provided by Jakub: http://brew-task-repos.usersys.redhat.com/repos/scratch/jsitnick/openvswitch/2.9.0/19.el7_5.bz1547065.1/openvswitch-2.9.0-19.el7_5.bz1547065.1-scratch.repo One remaining thing is to get the exceptions from Python IDL logged in ovirt logs so that it's easier to troubleshoot similar issues in the future. Marcin, do you happen to have an env where we could experiment with the logger config? I will set it up, and ping you once it's up. The logger conf we are using: https://github.com/oVirt/ovirt-provider-ovn/blob/master/provider/logger.conf (In reply to Marcin Mirecki from comment #29) > I will set it up, and ping you once it's up. > > The logger conf we are using: > https://github.com/oVirt/ovirt-provider-ovn/blob/master/provider/logger.conf Thanks, Marcin. this issue is verified on the latest verison:
[root@dell-per730-20 ~]# ovn-nbctl set-connection pssl:6641
[root@dell-per730-20 ~]# ovn-nbctl set-ssl /etc/openvswitch/northdb-privkey.pem \
> /etc/openvswitch/northdb-cert.pem \
> /var/lib/openvswitch/pki/controllerca/cacert.pem
[root@dell-per730-20 ~]# python 1.py
[root@dell-per730-20 ~]# ovn-nbctl show
switch a311658f-4eb4-4a76-bee6-bc998564d1c7 (רשת חיצונית)
[root@dell-per730-20 ~]# cat 1.py
#!/usr/bin/env python2
import logging
import ovs.stream
from ovsdbapp.schema.ovn_northbound.impl_idl import OvnNbApiIdlImpl
from ovsdbapp.backend.ovs_idl.connection import Connection
from ovsdbapp.backend.ovs_idl.connection import OvsdbIdl
# ovn-nbctl get-ssl
SSL_KEY_FILE = "/etc/openvswitch/northdb-privkey.pem"
SSL_CERT_FILE = "/etc/openvswitch/northdb-cert.pem"
SSL_CACERT_FILE = "/var/lib/openvswitch/pki/controllerca/cacert.pem"
def connect():
ovs.stream.Stream.ssl_set_private_key_file(SSL_KEY_FILE)
ovs.stream.Stream.ssl_set_certificate_file(SSL_CERT_FILE)
ovs.stream.Stream.ssl_set_ca_cert_file(SSL_CACERT_FILE)
ovsidl = OvsdbIdl.from_server("ssl:127.0.0.1:6641", "OVN_Northbound")
ovsdb_connection = Connection(idl=ovsidl, timeout=100)
idl = OvnNbApiIdlImpl(ovsdb_connection)
return idl
def add_switch(idl):
name = u'\u05e8\u05e9\u05ea \u05d7\u05d9\u05e6\u05d5\u05e0\u05d9\u05ea'
cmd = idl.ls_add(switch=name, may_exist=False)
cmd.execute(check_error=True)
def main():
logging.basicConfig()
idl = connect()
add_switch(idl)
if __name__ == "__main__":
main()
[root@dell-per730-20 ~]# rpm -qa | grep openvswitch
openvswitch-selinux-extra-policy-1.0-3.el7fdp.noarch
openvswitch-ovn-host-2.9.0-36.el7fdp.x86_64
openvswitch-2.9.0-36.el7fdp.x86_64
openvswitch-ovn-central-2.9.0-36.el7fdp.x86_64
python-openvswitch-2.9.0-36.el7fdp.noarch
openvswitch-ovn-vtep-2.9.0-36.el7fdp.x86_64
openvswitch-ovn-common-2.9.0-36.el7fdp.x86_64
openvswitch-test-2.9.0-36.el7fdp.noarch
[root@dell-per730-20 ~]#
[root@dell-per730-20 ~]# uname -a
Linux dell-per730-20.rhts.eng.pek2.redhat.com 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-per730-20 ~]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1962 |