Bug 1142258

Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Connecting redmine issue http://projects.theforeman.org/issues/3898 from this bug

Moving to POST since upstream bug http://projects.theforeman.org/issues/3898 has been closed
-------------
Anonymous
Applied in changeset commit:hammer-cli-foreman|ae520bbb7bccbc38ba55938fb5f05c3cb5a71298.

Moving back to new since the related PR actually fixed something different. Please see the upstream issue for more details.

Tomas,

Should this also address the following upstream request?

http://projects.theforeman.org/issues/3652

If a user is attempting to report on 30,000+ systems, would they just need to use the example posted by Tom:

current_results = get first page
all results += current_results
while current_results.size < per_page
current_results = get next page
all_results += current_results
end

And set the per_page option to how many systems they want to receive in each batch? Is the above functionality possible by this commit: hammer-cli-foreman|ae520bbb7bccbc38ba55938fb5f05c3cb5a71298?

Thanks,
Brandon

Thanks, Brandon.  (This is Marty; I'm using bugzi credentials that predate the account.)

Tomas, we've got code that does this pagination.  It's relatively obvious how to do that.

My question is, what are the semantics in the API for that?  What http headers/cookies need to be set so that the API knows how to deal with the result set?

If, for example, we sent repeated requests via curl (I wouldn't), what I expect is that each paginated request would generate the complete result set, and only give the paginated piece we're asking for for that request.  Similarly, I don't expect that such requests would be transactional, in that if boxes were added to or removed from the result set, that the pagination approach would be racy for them:  request 1 generates 100 results; request 2 (2 seconds later) generates 150.  With the pagination approach (say 50 per request), which set am I going to get, or am I going to get the first 50, then the next 50, then the next 50 (if there are 50 left).  Granted the "window" for such problems is fairly limited, but we are evaluating just how much operational tooling we can build around the foreman API and this is an important piece of data for us.

I hope these questions make sense.  I fear I haven't asked very well.

Brandon, your proposed algorithm was possible even before the mentioned commit. The commit actually only switches off the interactive "show more?" prompt in csv mode.

The problem with your algorithm is what Marty outlined - the api queries are not transactional. The api takes parameters:
 page - page to start from
 per_page - number of items to show per page
See the docs for listing hosts in [1].

So if an item is added (or removed) between query for page 1 and page 2 there can be overlap (or items missing). The solution is to add direct support for listing items as [2] proposes. I'm going to clone [2] into BZ and reference it with this bug.

Marty, did it answer your question?

[1] http://www.theforeman.org/api/apidoc/v2/hosts/index.html
[2] http://projects.theforeman.org/issues/3652

Tomas, yes, that answers my question.  Thank you!   I've added myself to the other bug so I can follow it.

Foreman is a truly excellent piece of work, by the way.  I really appreciate all the work you and the team have done on it.

Upstream bug assigned to tcaspy

this is not a bug.
the data can't return without pagination (imagine a query returning 100K results, which is definitely possible)
the idea is that the machine reading the data will go through the pagination programatically.
i.e. this pseudo code:

current_results = get first page
all results += current_results
while current_results.size < per_page
current_results = get next page
all_results += current_results
end

I can imagine it just fine.

Do you really expect all of your users to do this?  Shouldn't foreman be expected to be able to report on its entire node set?

Most data processing tools are easily able to handle 100k+ rows.

*** Bug 1179361 has been marked as a duplicate of this bug. ***

It's a bug - this is a command line tool and command line tools in UNIX environments should operate with standard command pipelines and redirection.

And yes, I fully expect the command to return 100,000 lines if that is how many systems there are registered.

*** Bug 1179096 has been marked as a duplicate of this bug. ***

Per all teh comments... I agree.. we need to support this. I am fine with a command line parameter having to be used ti disable pagination. That it is a valid requirement. However, the option needs to be there.

okay, I'll think of a way to support this, will need to find a way to make this memory efficient...

*** This bug is verified in upstream.  This fix should eventually land in future downstream builds ***
Version Tested:
# rpm -qa  | grep foreman
nec-em17.rhts.eng.bos.redhat.com-foreman-client-1.0-1.noarch
foreman-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.3-4.el7.noarch
nec-em17.rhts.eng.bos.redhat.com-foreman-proxy-client-1.0-1.noarch
tfm-rubygem-hammer_cli_foreman-0.4.0-1.201510071112git33fd59b.el7.noarch
foreman-debug-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch
foreman-release-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch
foreman-postgresql-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch
foreman-vmware-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch
tfm-rubygem-foreman_hooks-0.3.9-1.el7.noarch
tfm-rubygem-foreman-tasks-0.7.6-1.fm1_10.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.8-1.el7.noarch
tfm-rubygem-foreman_bootdisk-6.0.0-2.fm1_10.el7.noarch
foreman-release-scl-1-1.el7.x86_64
foreman-libvirt-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch
foreman-selinux-1.11.0-0.develop.201510071426git6234447.el7.noarch
foreman-ovirt-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.1.3-3.el7.noarch
tfm-rubygem-foreman_gutterball-0.0.1-3.el7.noarch
nec-em17.rhts.eng.bos.redhat.com-foreman-proxy-1.0-2.noarch
tfm-rubygem-foreman_discovery-4.1.0-1.fm1_10.el7.noarch
tfm-rubygem-foreman_docker-1.4.1-2.fm1_10.el7.noarch
foreman-proxy-1.11.0-0.develop.201510120849git5f36f2e.el7.noarch
foreman-compute-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch
foreman-gce-1.11.0-0.develop.201510121538gitb6b977a.el7.noarch

steps:
#  hammer -u admin -p changeme --csv template list > /tmp/templates.csv
# wc -l /tmp/templates.csv
55 /tmp/templates.csv

shows the actual number of templates on my system . so nothing is truncated

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501