Bug 1136141

Handling a misconfigured serial console on RHEL 6 and earlier requires enhancements to the custom handling of /dev/console.

That moves this into new feature territory, rather than bug fix territory (on systemd based systems, we can just drop beah's custom console handling entirely and let systems deal with it).


+++ This bug was initially created as a clone of Bug #967502 +++

Description of problem:
If a system has misconfigured serial console then beah can't run any tests and all of them fail with:

Redirecting to /bin/systemctl status  rhts-compat.service
+++ basename /var/lib/beah/tortilla/wrappers.d/runtest
++ tortilla get-next-wrapper runtest
+ WRAP_NEXT=
+ /usr/bin/rhts-test-runner.sh
logger: /usr/bin/rhts-test-runner.sh rhts-extend lab-02 12662707 5400
logger: /usr/bin/rhts-test-runner.sh rhts-test-checkin 127.0.0.1:7091 intel-mccreary-01 422430 /kernel/networking/kdump 5400 12662707
Traceback (most recent call last):
  File "/usr/bin/rhts-test-checkin", line 41, in <module>
    res = os.write(fd, "%s JobID:%s Test:%s Response:%s\n" % (timetext, jobid, test, resp))
OSError: [Errno 5] Input/output error
+ rc=1
+ '[' -n '' ']'
+ exit 1


Version-Release number of selected component (if applicable):
0.12

How reproducible:
100%

Steps to Reproduce:
1. run any test on system with misconfigured serial console, such that any write to /dev/console fails with "Input/output error"

Actual results:
beah fails to run every test

Expected results:
beah should report failure to write to stdout (/dev/console), but this shouldn't stop tests from running

Additional info:
Bug 966942 - /dev/ttyS0: not a character device, starting with 3.10.0-0.rc1.56.el7.x86_64

--- Additional comment from Dan Callaghan on 2014-09-02 12:28:48 EST ---

/dev/console is not a real file and we really shouldn't be pretending it is. beah also needs to handle the case where open("/dev/console") fails with ENODEV or EIO [1].

On systemd distros we should just stop all this logging business and write everything to stdout/stderr, and let the systemd journal capture it and forward it to the console. I think restraint does this currently. Then the quirks of the console are systemd's problem.

For RHEL6 and earlier we may need to introduce a new beah config option for logging to the console, which uses a different code path (not the stdlib logging file handler) which is resilient to the various failure cases with /dev/console. In this case we will also need to be careful about redirecting stdout/stderr for child processes -- see the traceback in comment 0 for an example of why.

[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/554172/comments/245

--- Additional comment from Dan Callaghan on 2014-09-02 12:34:09 EST ---

I forgot the context for comment 1... We had a report of a similar problem, with the harness services failing to start on RHEL7 due to IOError opening /dev/console. It is essentially the same issue (/dev/console has some weird behaviour, we can't treat it like a regular file).

systemd[1]: Starting The Beaker backend server....
systemd[1]: Started The Beaker backend server..
beah-beaker-backend[1319]: --- WARNING: Value for DEFAULT.HOSTNAME (None) is not an string.
beah-beaker-backend[1319]: Traceback (most recent call last):
beah-beaker-backend[1319]: File "/usr/bin/beah-beaker-backend", line 9, in <module>
beah-beaker-backend[1319]: load_entry_point('beah==0.7.3.dev201403262029', 'console_scripts', 'beah-beaker-backend')()
beah-beaker-backend[1319]: File "/usr/lib/python2.7/site-packages/beah/backends/beakerlc.py", line 2070, in main
beah-beaker-backend[1319]: log_handler()
beah-beaker-backend[1319]: File "/usr/lib/python2.7/site-packages/beah/wires/internals/twbackend.py", line 90, in log_handler
beah-beaker-backend[1319]: console=parse_bool(conf.get('DEFAULT', 'CONSOLE_LOG')))
beah-beaker-backend[1319]: File "/usr/lib/python2.7/site-packages/beah/misc/__init__.py", line 214, in make_log_handler
beah-beaker-backend[1319]: lhandler = logging.FileHandler('/dev/console')
beah-beaker-backend[1319]: File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
beah-beaker-backend[1319]: StreamHandler.__init__(self, self._open())
beah-beaker-backend[1319]: File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
beah-beaker-backend[1319]: stream = open(self.baseFilename, self.mode)
beah-beaker-backend[1319]: IOError: [Errno 5] Input/output error: '/dev/console'
systemd[1]: beah-beaker-backend.service: main process exited, code=exited, status=1/FAILURE
systemd[1]: Unit beah-beaker-backend.service entered failed state.

--- Additional comment from Nick Coghlan on 2014-09-02 12:45:51 EST ---

I agree relying on systemd is the right thing to do for systems that offer it, but I'm less convinced it's worth the hassle of coming up with our own solution for older RHEL versions.

I'll split this into two issues (this one for systemd based systems, a new one for RHEL6 and earlier), as I'd like to separate the two decisions.