Bug 1568751 - Beaker CSV Exports Failing
Summary: Beaker CSV Exports Failing
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Beaker
Classification: Retired
Component: general
Version: 25
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
Assignee: beaker-dev-list
QA Contact: tools-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-18 08:48 UTC by Theodoros Apazoglou
Modified: 2020-10-21 14:12 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-21 14:12:38 UTC


Attachments (Terms of Use)
memory error on the server (6.25 KB, text/plain)
2018-04-19 02:03 UTC, Roman Joost
no flags Details

Description Theodoros Apazoglou 2018-04-18 08:48:31 UTC
Description of problem:
Hello,

while i am on the main beaker website, im trying to export the system key/values   and i am getting the below HTTP 500 error message.

"500 Internal error

The server encountered an unexpected condition which prevented it from fulfilling the request.

Powered by CherryPy 2.3.0 "



Version-Release number of selected component (if applicable): Beaker 25.1


How reproducible:
Easy to reproduce.

Steps to Reproduce:
1. Go to https://beaker.engineering.redhat.com/
2. log in
3. Click at the "Admin" tab
4. Export
5. Select "System Key/Values"
6. click the button "Export CSV"

Actual results:
500 Internal error

Expected results:
Downloadable CSV file expected.

Additional info:
Feel free to ask me for more information if needed. irc:theo email:tapazogl@redhat.com

thank you,
Theo

Comment 1 Roman Joost 2018-04-19 02:03:33 UTC
Created attachment 1423857 [details]
memory error on the server

Comment 2 Roman Joost 2018-04-19 02:05:03 UTC
Found the problem on the server. Seems like while compiling the CSV it ran out of memory.

Comment 3 Roman Joost 2018-04-19 02:06:33 UTC
Dear Theodoros,

do you need the key/values of *all* systems, or just a specific subset?

Comment 4 Theodoros Apazoglou 2018-04-19 08:26:49 UTC
Hello Roman,

because of https://redhat.service-now.com/surl.do?n=PNT0187703, i need key/values of *all* systems, so i can find the ones that have "mptsas/mptspi disks and/or e1000 network interfaces" in order to exclude those systems from installing RHEL-8 since these drivers are not supported.

Few days ago i did export of *all* systems "excluded families" and worked flawlessly. 



thanks,
Theo
PnT & CEE Labs team

Comment 5 Dan Callaghan 2018-04-19 22:54:09 UTC
There is no easy solution to this problem (too much memory needed to export all systems) and thus it won't be fixed any time soon.

Can you instead construct a suitable XML host filter and use `bkr list-systems --xml-filter=` to get a list of relevant FQDNs, and then do the exclusions just for those?

Comment 6 Bill Therrien 2018-05-04 13:58:59 UTC
Hi,
I think the problem here is really the functionality of the CSV exporting.
When you export any of the CSV types it dumps literally everything from the database.
Considering that this includes systems that were "Removed" years ago, there are extremely few cases where someone actually needs to export info for ALL the systems.

Instead of considering this a bug against a MemoryError maybe we can treat it as an RFE to improve the usability of CSV exports.

Is it possible to implement CSV exporting based on system search results?
For example:
- I search from the Beaker home page (All Systems) for 'Devices/Description' containing the string "be3"
- 87 systems are returned
- Perhaps, under the "+Add" button there is an "Export CSV" button which will pass the results to the main CSV Export page where you can then select the CSV type that you want to export for just those 87 systems.

Comment 7 Roman Joost 2018-05-08 06:38:27 UTC
Dear Bill,

I like the idea. Not sure how much of a change it is with the current code, but the use case sounds reasonable.

Comment 8 Theodoros Apazoglou 2018-06-14 11:00:04 UTC
Hi Dan/Roman,

while running the export of key/values i am checking the cpu/ram resources and i don't see running out of memory. 
Yes, beaker RAM resources is not in a good shape but, that's why there is the swap memory and i don't see the system to allocate any. 

I don't know in deep how beaker works at the background that's why i am trying to understand. Exporting is a useful features for my team and the key/values is the only one it doesn't work.

I don't know how to construct a suitable XML host filter.

Is there an update on this issue or maybe at Bill's proposal ?

thanks,
Theo

Comment 9 Roman Joost 2018-06-15 03:29:30 UTC
Dear Theodoros,

I believe you that this would make your life a lot easier. Thank you as well for your help in checking the memory usage. We haven't had time so far to look into this issue and therefore do not know how much work is involved with it. With that I can't unfortunately give you an estimate when we will ship this.

Having said that, I bumped the priority in our board so that we'll be looking at this item next, once all the fire fighting is done (RHEL8).

A suitable host filter is whatever you would usually insert in a hostRequires element. You can find examples here:

https://beaker-project.org/docs/user-guide/job-xml.html#job-workflow-details

Note, that you'll have to scroll down a fair bit.

For example, if I would like to find all systems which have a network card with a description of: NetXtreme II BCM5708 Gigabit Ethernet I would use:

bkr list-systems --xml-filter='<and><device op="=" type="network" description="NetXtreme II BCM5708 Gigabit Ethernet" /></and>'

Mind the single quotes around the filter to avoid my shell interpolating any values away.

I do know it's not ideal, but hopefully will get you going in the mean time.

Comment 11 drohwer 2018-07-16 17:00:59 UTC
Bumping the priority on this as well as https://bugzilla.redhat.com/show_bug.cgi?id=1600587 since they both seem related.  Though they get different errors and I believe the beaker logs get different errors:

The key/value gets:

500 Internal error
The server encountered an unexpected condition which prevented it from fulfilling the request.

System Excluded Families gets:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Comment 12 MikeBoswell 2019-04-25 14:47:51 UTC
This is still an issue in prod.  
https://beaker.engineering.redhat.com/csv/action_export?csv_type=exclude

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.
Apache/2.2.15 (Red Hat) Server at beaker.engineering.redhat.com Port 443

Comment 13 Theodoros Apazoglou 2019-06-06 10:53:24 UTC
After one year,

this is still an issue.

500 Internal error

The server encountered an unexpected condition which prevented it from fulfilling the request.

Powered by CherryPy 2.3.0


Note You need to log in before you can comment on or make changes to this bug.