Description of problem: Hammer silently truncates output to 22 lines when redirecting standard output. Version-Release number of selected component (if applicable): rubygem-hammer_cli-0.1.1-5.el6sat.noarch but also affects GA version. How reproducible: every time Steps to Reproduce: 1. hammer --csv template list > /tmp/templates.csv 2. wc -l /tmp/templatex.csv 3. Actual results: 21 Expected results: 42 (the actual number of templates on my system) Additional info: This is due to a poor implementation of the pager In any case, the pager should always be switched off when stdout is redirected, rather than simply silently truncating the output and exiting. Workaround: specifying: hammer --csv template list --per-page 9999 > /tmp/templates.csv will list all templates (up to a maximum of 9999) - however this is not a satisfactory solution, particularly in the case of hosts where we may have more than 9999 hosts.
Since this issue was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release.
Connecting redmine issue http://projects.theforeman.org/issues/3898 from this bug
Moving to POST since upstream bug http://projects.theforeman.org/issues/3898 has been closed ------------- Anonymous Applied in changeset commit:hammer-cli-foreman|ae520bbb7bccbc38ba55938fb5f05c3cb5a71298.
Moving back to new since the related PR actually fixed something different. Please see the upstream issue for more details.
Tomas, Should this also address the following upstream request? http://projects.theforeman.org/issues/3652 If a user is attempting to report on 30,000+ systems, would they just need to use the example posted by Tom: current_results = get first page all results += current_results while current_results.size < per_page current_results = get next page all_results += current_results end And set the per_page option to how many systems they want to receive in each batch? Is the above functionality possible by this commit: hammer-cli-foreman|ae520bbb7bccbc38ba55938fb5f05c3cb5a71298? Thanks, Brandon
Thanks, Brandon. (This is Marty; I'm using bugzi credentials that predate the account.) Tomas, we've got code that does this pagination. It's relatively obvious how to do that. My question is, what are the semantics in the API for that? What http headers/cookies need to be set so that the API knows how to deal with the result set? If, for example, we sent repeated requests via curl (I wouldn't), what I expect is that each paginated request would generate the complete result set, and only give the paginated piece we're asking for for that request. Similarly, I don't expect that such requests would be transactional, in that if boxes were added to or removed from the result set, that the pagination approach would be racy for them: request 1 generates 100 results; request 2 (2 seconds later) generates 150. With the pagination approach (say 50 per request), which set am I going to get, or am I going to get the first 50, then the next 50, then the next 50 (if there are 50 left). Granted the "window" for such problems is fairly limited, but we are evaluating just how much operational tooling we can build around the foreman API and this is an important piece of data for us. I hope these questions make sense. I fear I haven't asked very well.
Brandon, your proposed algorithm was possible even before the mentioned commit. The commit actually only switches off the interactive "show more?" prompt in csv mode. The problem with your algorithm is what Marty outlined - the api queries are not transactional. The api takes parameters: page - page to start from per_page - number of items to show per page See the docs for listing hosts in [1]. So if an item is added (or removed) between query for page 1 and page 2 there can be overlap (or items missing). The solution is to add direct support for listing items as [2] proposes. I'm going to clone [2] into BZ and reference it with this bug. Marty, did it answer your question? [1] http://www.theforeman.org/api/apidoc/v2/hosts/index.html [2] http://projects.theforeman.org/issues/3652
Tomas, yes, that answers my question. Thank you! I've added myself to the other bug so I can follow it. Foreman is a truly excellent piece of work, by the way. I really appreciate all the work you and the team have done on it.
Upstream bug assigned to tcaspy
this is not a bug. the data can't return without pagination (imagine a query returning 100K results, which is definitely possible) the idea is that the machine reading the data will go through the pagination programatically. i.e. this pseudo code: current_results = get first page all results += current_results while current_results.size < per_page current_results = get next page all_results += current_results end
I can imagine it just fine. Do you really expect all of your users to do this? Shouldn't foreman be expected to be able to report on its entire node set? Most data processing tools are easily able to handle 100k+ rows.
*** Bug 1179361 has been marked as a duplicate of this bug. ***
It's a bug - this is a command line tool and command line tools in UNIX environments should operate with standard command pipelines and redirection. And yes, I fully expect the command to return 100,000 lines if that is how many systems there are registered.
*** Bug 1179096 has been marked as a duplicate of this bug. ***
Per all teh comments... I agree.. we need to support this. I am fine with a command line parameter having to be used ti disable pagination. That it is a valid requirement. However, the option needs to be there.
okay, I'll think of a way to support this, will need to find a way to make this memory efficient...
*** This bug is verified in upstream. This fix should eventually land in future downstream builds *** Version Tested: # rpm -qa | grep foreman nec-em17.rhts.eng.bos.redhat.com-foreman-client-1.0-1.noarch foreman-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch tfm-rubygem-hammer_cli_foreman_docker-0.0.3-4.el7.noarch nec-em17.rhts.eng.bos.redhat.com-foreman-proxy-client-1.0-1.noarch tfm-rubygem-hammer_cli_foreman-0.4.0-1.201510071112git33fd59b.el7.noarch foreman-debug-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch foreman-release-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch foreman-postgresql-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch foreman-vmware-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch tfm-rubygem-foreman_hooks-0.3.9-1.el7.noarch tfm-rubygem-foreman-tasks-0.7.6-1.fm1_10.el7.noarch tfm-rubygem-hammer_cli_foreman_tasks-0.0.8-1.el7.noarch tfm-rubygem-foreman_bootdisk-6.0.0-2.fm1_10.el7.noarch foreman-release-scl-1-1.el7.x86_64 foreman-libvirt-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch foreman-selinux-1.11.0-0.develop.201510071426git6234447.el7.noarch foreman-ovirt-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch tfm-rubygem-hammer_cli_foreman_bootdisk-0.1.3-3.el7.noarch tfm-rubygem-foreman_gutterball-0.0.1-3.el7.noarch nec-em17.rhts.eng.bos.redhat.com-foreman-proxy-1.0-2.noarch tfm-rubygem-foreman_discovery-4.1.0-1.fm1_10.el7.noarch tfm-rubygem-foreman_docker-1.4.1-2.fm1_10.el7.noarch foreman-proxy-1.11.0-0.develop.201510120849git5f36f2e.el7.noarch foreman-compute-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch foreman-gce-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch steps: # hammer -u admin -p changeme --csv template list > /tmp/templates.csv # wc -l /tmp/templates.csv 55 /tmp/templates.csv shows the actual number of templates on my system . so nothing is truncated
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1501