Bug 1600587

Summary: Beaker Exports Failing - Excludes and Key/Value pairs
Product: [Community] Beaker Reporter: Anthony Gialluca <agialluc>
Component: generalAssignee: Martin Styk <mastyk>
Status: CLOSED NEXTRELEASE QA Contact: tools-bugs <tools-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 25CC: alemay, gcase, mastyk, mboswell, yliberma
Target Milestone: 28.0Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-24 09:10:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Anthony Gialluca 2018-07-12 14:38:47 UTC
Description of problem:

With the latest beaker we are unable to export the 'Excludes' and the 'Key/Value' pairs.

This is going to severely impact the ability to administer the Excluded Distros for the Admins.

Version-Release number of selected component (if applicable):

beaker-server-25.4-1.el6bkr.hotfix.1.noarch
python-beaker-1.3.1-7.el6.noarch
beaker-common-25.4-1.el6bkr.hotfix.1.noarch
beaker-server-redhat-3-1.el6eng.noarch


How reproducible:

Try to export via:
https://beaker.engineering.redhat.com/csv/action_export?csv_type=exclude
-or-
https://beaker.engineering.redhat.com/csv/action_export?csv_type=keyvalue

Steps to Reproduce:
1. Connect to beaker server beaker.engineering.redhat.com
2. Try to export either 'System Key/Values' or 'System Excluded Families'


Actual results:
Receive a page with "500 Internal error"

Expected results:
Should be presented with a CSV file to save/view.

Additional info:
The following was observed in /var/log/beaker/server-errors.log
Jul 12 14:29:05 beaker-server beaker-server[7143]: bkr.server ERROR Exception on /csv/action_export [GET]
Jul 12 14:29:05 beaker-server beaker-server[7143]:  Traceback (most recent call last):
Jul 12 14:29:05 beaker-server beaker-server[7143]:    File "/usr/lib/python2.6/site-packages/flask/app.py", line 1817, in wsgi_app
Jul 12 14:29:05 beaker-server beaker-server[7143]:      response = self.full_dispatch_request()
Jul 12 14:29:05 beaker-server beaker-server[7143]:    File "/usr/lib/python2.6/site-packages/flask/app.py", line 1479, in full_dispatch_request
Jul 12 14:29:05 beaker-server beaker-server[7143]:      response = self.process_response(response)
Jul 12 14:29:05 beaker-server beaker-server[7143]:    File "/usr/lib/python2.6/site-packages/flask/app.py", line 1691, in process_response
Jul 12 14:29:05 beaker-server beaker-server[7143]:      response = handler(response)
Jul 12 14:29:05 beaker-server beaker-server[7143]:    File "/usr/lib/python2.6/site-packages/bkr/server/wsgi.py", line 150, in commit_or_rollback_session
Jul 12 14:29:05 beaker-server beaker-server[7143]:      session.rollback()
Jul 12 14:29:05 beaker-server beaker-server[7143]:    File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/scoping.py", line 150, in do
Jul 12 14:29:05 beaker-server beaker-server[7143]:      return getattr(self.registry(), name)(*args, **kwargs)
Jul 12 14:29:05 beaker-server beaker-server[7143]:    File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 740, in rollback
Jul 12 14:29:05 beaker-server beaker-server[7143]:      self.transaction.rollback()
Jul 12 14:29:05 beaker-server beaker-server[7143]:    File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 403, in rollback
Jul 12 14:29:05 beaker-server beaker-server[7143]:      transaction._rollback_impl()
Jul 12 14:29:05 beaker-server beaker-server[7143]:    File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 434, in _rollback_impl
Jul 12 14:29:05 beaker-server beaker-server[7143]:      self._restore_snapshot(dirty_only=self.nested)
Jul 12 14:29:05 beaker-server beaker-server[7143]:    File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 286, in _restore_snapshot
Jul 12 14:29:05 beaker-server beaker-server[7143]:      for s in self.session.identity_map.all_states():
Jul 12 14:29:05 beaker-server beaker-server[7143]:    File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/identity.py", line 177, in all_states
Jul 12 14:29:05 beaker-server beaker-server[7143]:      return dict.values(self)
Jul 12 14:29:05 beaker-server beaker-server[7143]:  MemoryError: <bound method CSV.action_export of <bkr.server.CSV_import_export.CSV object at 0x7f40651edf90>>

Comment 1 Roman Joost 2018-07-16 04:12:32 UTC
This looks like it's the same problem as it was initially reported by Bug 1568751. I would bump the priority of 1568751 since it seems it would benefit from the same solution...

Comment 2 drohwer 2018-07-16 17:00:55 UTC
Bumping the priority on this as well as https://bugzilla.redhat.com/show_bug.cgi?id=1568751 since they both seem related.  Though they get different errors and I believe the beaker logs get different errors:

The key/value gets:

500 Internal error
The server encountered an unexpected condition which prevented it from fulfilling the request.

System Excluded Families gets:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Comment 5 Dan Callaghan 2018-10-29 18:32:44 UTC
As a workaround, you can extract a similar CSV to what Beaker would produce for the "System Excluded Families" CSV export by using a Teiid query like this:

SELECT 'exclude' AS csv_type, fqdn, arch.arch, osmajor.osmajor as family, NULL as "update", 'True' as excluded
FROM Beaker.system
INNER JOIN Beaker.exclude_osmajor ON system_id = system.id
INNER JOIN Beaker.osmajor ON exclude_osmajor.osmajor_id = osmajor.id
INNER JOIN Beaker.arch ON exclude_osmajor.arch_id = arch.id
UNION
SELECT 'exclude' AS csv_type, fqdn, arch.arch, osmajor.osmajor as family, osversion.osminor as "update", 'True' as excluded
FROM Beaker.system
INNER JOIN exclude_osversion ON system_id = system.id
INNER JOIN osversion ON exclude_osversion.osversion_id = osversion.id
INNER JOIN osmajor ON osversion.osmajor_id = osmajor.id
INNER JOIN arch ON exclude_osversion.arch_id = arch.id;

Unfortunately there is no easy way to make the psql client produce a CSV formatted result from that query.

We can kind of fake it, by using the following psql options:

psql postgresql://... -o exclude.csv -c '\pset border 1' -c '\pset format unaligned' -c '\pset fieldsep ,' -c '\pset footer off' -c "SELECT ..."

Be careful to escape " inside the query from the shell as \".

Also note that this trick for faking the CSV output will only work if the values do not contain commas or quotes, because it is not doing any proper CSV escaping. For this particular query (exclude families) it should be fine since system FQDNs and OS major names will not normally contain a comma or quote character. But it won't necessarily work for other Teiid queries.

Comment 6 MikeBoswell 2019-04-25 14:48:01 UTC
This is still an issue in prod.  
https://beaker.engineering.redhat.com/csv/action_export?csv_type=exclude

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.
Apache/2.2.15 (Red Hat) Server at beaker.engineering.redhat.com Port 443

Comment 7 Yaniv Liberman 2020-01-09 14:12:35 UTC
Still an issue in Beaker version 26 as well.

Should be fixed in version 27, which prod will soon be updated to. Hopefully.

Comment 8 Martin Styk 2020-01-14 08:20:21 UTC
(In reply to Yaniv Liberman from comment #7)
> Still an issue in Beaker version 26 as well.
> 
> Should be fixed in version 27, which prod will soon be updated to. Hopefully.

This will be not fixed in 27. This will land in Beaker in 28.

Comment 9 Yaniv Liberman 2020-01-14 10:45:29 UTC
Ack. I was told by rebarbos on #beaker that he tested it in v27 development and it worked...