Bug 1504527
| Summary: | beaker-watchdog crashes when saving console logs from OpenStack: UnicodeEncodeError: 'ascii' codec can't encode... | ||
|---|---|---|---|
| Product: | [Retired] Beaker | Reporter: | Dan Callaghan <dcallagh> |
| Component: | general | Assignee: | Dan Callaghan <dcallagh> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Anwesha Chatterjee <achatter> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 24 | CC: | achatter, dcallagh, mjia, rjoost |
| Target Milestone: | 25.0 | Keywords: | Patch |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-03-19 04:18:12 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Yoink. So the first thing to note is that the servers.get_console_output call in novaclient is actually returning unicode not str. Which is odd because the serial console is really a stream of bytes. Any non-ASCII bytes that appear are being converted to U+FFFD REPLACEMENT CHARACTER inside OpenStack, even if it was a valid UTF-8 sequence. So that's annoying. Probably related to the ANSI escape stripping which OpenStack is also doing (bug 1395029). Regardless how right or wrong that behaviour is, I guess we just need to treat the return value of get_console_output as pre-decoded unicode. Even if OpenStack fixes the handling of non-ASCII console output in a future version, they will need to keep the return type as unicode to avoid breaking compatibility. In some ways it makes this problem easier, because it means we don't have to worry about decoding UTF-8 or handling UTF-8 invalid byte sequences through XMLRPC or anything. We just need to re-encode to UTF-8 at the last layer before we write bytes to disk. https://gerrit.beaker-project.org/5903 Verified that beaker-watchdog no longer crashes when rendering non-ascii characters in the console log. For example in : https://beaker-devel.app.eng.bos.redhat.com/beaker-logs/2018/01/136/13643/20543/console.log This line appears: Firmware for Intel® Wireless 5150 A/G/N network adaptors Apologies, please disregard (https://bugzilla.redhat.com/show_bug.cgi?id=1504527#c4) Comment 4 This bug is verified with this log file: https://beaker-devel.app.eng.bos.redhat.com/beaker-logs/2018/01/136/13626/20526/console.log containing this line: Firmware for Intel�� PRO/Wireless 5000 A/G/N network adaptors which shows that the console is rendering non-ascii characters as a unicode replacement character, and beaker-watchdog is not crashing. This bug was fixed on the release-24 branch, to appear in Beaker 24.6, but we are not planning any more maintenance releases for the 24.x series. The next release will be Beaker 25.0. Updating the milestone accordingly. Beaker 25.0 has been released. Release notes are available upstream: https://beaker-project.org/docs/whats-new/release-25.html |
Version-Release number of selected component (if applicable): 24.5.git.15.5807bc1a3 but the bug likely existed since OpenStack integration was added How reproducible: unsure Steps to Reproduce: 1. Run beaker-watchdog -f, otherwise the traceback goes to stderr and is lost 2. See if it dies? Actual results: The crash is: Traceback (most recent call last): File "/usr/bin/beaker-watchdog", line 9, in <module> load_entry_point('beaker-lab-controller==24.5.git.15.5807bc1a3', 'console_scripts', 'beaker-watchdog')() File "/usr/lib/python2.6/site-packages/bkr/labcontroller/watchdog.py", line 118, in main main_loop(watchdog, conf) File "/usr/lib/python2.6/site-packages/bkr/labcontroller/watchdog.py", line 66, in main_loop if not watchdog.run(): File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 552, in run updated |= monitor.run() File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 616, in run return self.console_watch.update() File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 321, in update self.process_log(block) File "/usr/lib/python2.6/site-packages/bkr/labcontroller/proxy.py", line 254, in process_log log_file.update_chunk(block, self.where) File "/usr/lib/python2.6/site-packages/bkr/labcontroller/log_storage.py", line 72, in update_chunk self.f.write(data) UnicodeEncodeError: 'ascii' codec can't encode characters in position 37337-37338: ordinal not in range(128) Observed this in the beaker-devel environment, recipe in question is R:19642 which ran on OpenStack. Expected results: Not crash. Should deal only in raw bytes when storing the console log. Additional info: Most likely, somewhere we are fetching the OpenStack console is trying to convert to/from unicode instead of treating everything as raw bytes.