Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1325880 - hammer import config-file on RHEL7 fails to generate puppet modules
Summary: hammer import config-file on RHEL7 fails to generate puppet modules
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Uncategorized
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Tom McKay
QA Contact: Lukas Pramuk
URL: http://projects.theforeman.org/issues...
Whiteboard:
: 1315260 1350192 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-11 12:17 UTC by Lukas Pramuk
Modified: 2020-07-16 08:44 UTC (History)
5 users (show)

Fixed In Version: rubygem-hammer_cli_import-0.11.1-1
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-27 11:02:03 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 14616 0 None None None 2016-04-22 16:13:51 UTC

Description Lukas Pramuk 2016-04-11 12:17:01 UTC 
Description of problem:
hammer import config-file on RHEL7 fails to generate puppet modules.
It uses "puppet module generate <name>" and on RHEL7 it fails to parse command interview questions at all, while on RHEL6 command output with interview questions is parsed successfully.

Version-Release number of selected component (if applicable):
@Sat6.2.0-Beta-Snap
tfm-rubygem-hammer_cli_import-0.10.23-1.el7sat.noarch

How reproducible:
always on RHEL7

Steps to Reproduce:
1. # cat users.csv 
organization_id,organization,user_id,username,last_name,first_name,position,email,role,creation_time,last_login_time,active
1,SatelliteQE,1,admin,Admin,Admin,,root@localhost,Satellite Administrator,2014-09-10 19:39:59,2016-04-07 01:00:14,enabled

2. # hammer import organization --verbose --csv-file users.csv 
Importing from users.csv
Creating new organization: SatelliteQE
Summary
  Created 1 organization.

3. # cat config-files-latest.csv 
org_id,channel_id,channel,channel_type,path,file_type,file_id,revision,is_binary,contents,delim_start,delim_end,username,groupname,filemode,symbolic_link,selinux_ctx
1,3,config-1,normal,/etc/config1,file,3,2,N,config1 file ver.2,{|,|},apache,apache,0,,

4. # hammer import config-file --debug --csv-file config-files-latest.csv --generate-only
Importing from config-files-latest.csv
Caught EOFError:end of file reached while processing CSV line:
{"org_id"=>"1", "channel_id"=>"3", "channel"=>"config-1", "channel_type"=>"normal", "path"=>"/etc/config1", "file_type"=>"file", "file_id"=>"3", "revision"=>"2", "is_binary"=>"N", "contents"=>"config1 file ver.2", "delim_start"=>"{|", "delim_end"=>"|}", "username"=>"apache", "groupname"=>"apache", "filemode"=>"0", "symbolic_link"=>nil, "selinux_ctx"=>nil}
Writing converted files
Found module satelliteqe-config_1
Caught Errno::ENOENT:No such file or directory @ dir_s_mkdir - /root/puppet_work_dir/satelliteqe-config_1/files while post_import
Summary
  No action taken.

5. # less import.log:
...
I, [2016-04-11T08:03:46.664346 #17137]  INFO -- : Importing from config-files-latest.csv
D, [2016-04-11T08:03:46.665697 #17137] DEBUG -- : Removing /root/puppet_work_dir/satelliteqe-config_1
E, [2016-04-11T08:03:47.255283 #17137] ERROR -- : Caught EOFError:end of file reached while processing CSV line:
{"org_id"=>"1", "channel_id"=>"3", "channel"=>"config-1", "channel_type"=>"normal", "path"=>"/etc/config1", "file_type"=>"file", "file_id"=>"3", "revision"=>"2", "is_binary"=>"N", "contents"=>"config1 file ver.2", "delim_start"=>"{|", "delim_end"=>"|}", "username"=>"apache", "groupname"=>"apache", "filemode"=>"0", "symbolic_link"=>nil, "selinux_ctx"=>nil}
E, [2016-04-11T08:03:47.255556 #17137] ERROR -- : /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:146:in `readline'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:146:in `block (2 levels) in generate_module_template_for'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:143:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:143:in `block in generate_module_template_for'
/opt/rh/rh-ruby22/root/usr/share/ruby/open3.rb:199:in `popen_run'
/opt/rh/rh-ruby22/root/usr/share/ruby/open3.rb:93:in `popen3'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:141:in `generate_module_template_for'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:193:in `generate_module'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:251:in `import_single_row'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/base.rb:527:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/base.rb:527:in `block (2 levels) in cvs_iterate'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/importtools.rb:324:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/importtools.rb:324:in `handle_missing_and_supress'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/base.rb:526:in `block in cvs_iterate'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/csvhelper.rb:55:in `block in csv_each'
/opt/rh/rh-ruby22/root/usr/share/ruby/csv.rb:1739:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/csvhelper.rb:44:in `csv_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/base.rb:525:in `cvs_iterate'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/base.rb:533:in `import'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/base.rb:569:in `execute'


Actual results:
fails to generate a puppet module (afterwards it is uploaded)

Expected results:
generates a puppet module successfully (to be then uploaded onto server)
Comment 1 Lukas Pramuk 2016-04-11 12:22:50 UTC 
Additional info:
lib/hammer_cli_import/configfile.rb:146 in `readline' - it fails to parse command output with interview questions at all on RHEL7.

puppet module generate command which is being run by hammer import works itself:
# cd /root/puppet_work_dir/
# puppet module generate satelliteqe-config_1
We need to create a metadata.json file for this module.  Please answer the
following questions; if the question is not applicable to this module, feel free
to leave it blank.

Puppet uses Semantic Versioning (semver.org) to version modules.
What version is this module?  [0.1.0]
--> 

Who wrote this module?  [satelliteqe]
--> 

What license does this module code fall under?  [Apache 2.0]
--> 

How would you describe this module in a single sentence?
--> 

Where is this module's source code repository?
--> 

Where can others go to learn more about this module?
--> 

Where can others go to file issues about this module?
--> 

----------------------------------------
{
  "name": "satelliteqe-config_1",
  "version": "0.1.0",
  "author": "satelliteqe",
  "summary": null,
  "license": "Apache 2.0",
  "source": "",
  "project_page": null,
  "issues_url": null,
  "dependencies": [
    {"name":"puppetlabs-stdlib","version_requirement":">= 1.0.0"}
  ]
}
----------------------------------------

About to generate this metadata; continue? [n/Y]
--> Y

Notice: Generating module at /root/puppet_work_dir/satelliteqe-config_1...
Notice: Populating templates...
Finished; module generated in satelliteqe-config_1.
satelliteqe-config_1/Gemfile
satelliteqe-config_1/Rakefile
satelliteqe-config_1/manifests
satelliteqe-config_1/manifests/init.pp
satelliteqe-config_1/spec
satelliteqe-config_1/spec/classes
satelliteqe-config_1/spec/classes/init_spec.rb
satelliteqe-config_1/spec/spec_helper.rb
satelliteqe-config_1/tests
satelliteqe-config_1/tests/init.pp
satelliteqe-config_1/README.md
satelliteqe-config_1/metadata.json
Comment 2 Lukas Pramuk 2016-04-11 12:25:52 UTC 
lib/hammer_cli_import/configfile.rb:

            rd = ''
            until rd.include? '?'
              rd = stdout.readline  <<< 146: failing to parse output, RHEL6 is ok 
              #debug "Read #{rd}"
            end
Comment 4 Tom McKay 2016-04-13 14:49:23 UTC 
Created redmine issue http://projects.theforeman.org/issues/14616 from this bug
Comment 5 Bryan Kearney 2016-04-13 16:00:45 UTC 
Upstream bug component is Uncategorized
Comment 6 Bryan Kearney 2016-04-13 20:00:45 UTC 
Upstream bug component is Uncategorized
Comment 7 Lukas Pramuk 2016-06-27 08:27:51 UTC 
*** Bug 1350192 has been marked as a duplicate of this bug. ***
Comment 8 Bryan Kearney 2016-07-05 12:45:52 UTC 
*** Bug 1315260 has been marked as a duplicate of this bug. ***
Comment 9 Bryan Kearney 2016-07-14 18:14:29 UTC 
Moving to POST since upstream bug http://projects.theforeman.org/issues/14616 has been closed
Comment 10 Lukas Pramuk 2016-07-18 11:38:10 UTC 
FailedQA.

@satellite-6.2.0-20.1.el7sat.noarch
tfm-rubygem-hammer_cli_import-0.11.0-1.el7sat.noarch

1) # cat > users.csv << EOF
organization_id,organization,user_id,username,last_name,first_name,position,email,role,creation_time,last_login_time,active
1,SatelliteQE,1,admin,Admin,Admin,,root@localhost,Satellite Administrator,2014-09-10 19:39:59,2016-04-07 01:00:14,enabled
EOF

2) # hammer import organization --verbose --csv-file users.csv
Importing from users.csv
Creating new organization: SatelliteQE
Summary
  Created 1 organization.

3) # cat > config-files-latest.csv << EOF
org_id,channel_id,channel,channel_type,path,file_type,file_id,revision,is_binary,contents,delim_start,delim_end,username,groupname,filemode,symbolic_link,selinux_ctx
1,3,config-1,normal,/etc/config1,file,3,2,N,config1 file ver.2,{|,|},apache,apache,0,,
EOF

4) # hammer import config-file --debug --csv-file config-files-latest.csv --generate-only
Importing from config-files-latest.csv
Failed puppet module satelliteqe-config_1: metadata.json not created
Writing converted files
Summary
  Failed 1 puppet module.


>>> failed to create puppet module (on rhel7)
Comment 11 Lukas Pramuk 2016-07-18 11:42:17 UTC 
...however, puppet module generate is successful:

# puppet module generate satelliteqe-config_1
We need to create a metadata.json file for this module.  Please answer the
following questions; if the question is not applicable to this module, feel free
to leave it blank.

Puppet uses Semantic Versioning (semver.org) to version modules.
What version is this module?  [0.1.0]
--> 

Who wrote this module?  [satelliteqe]
--> 

What license does this module code fall under?  [Apache 2.0]
--> 

How would you describe this module in a single sentence?
--> 

Where is this module's source code repository?
--> 

Where can others go to learn more about this module?
--> 

Where can others go to file issues about this module?
--> 

----------------------------------------
{
  "name": "satelliteqe-config_1",
  "version": "0.1.0",
  "author": "satelliteqe",
  "summary": null,
  "license": "Apache 2.0",
  "source": "",
  "project_page": null,
  "issues_url": null,
  "dependencies": [
    {"name":"puppetlabs-stdlib","version_requirement":">= 1.0.0"}
  ]
}
----------------------------------------

About to generate this metadata; continue? [n/Y]
--> 

Notice: Generating module at /root/puppet_work_dir/satelliteqe-config_1...
Notice: Populating templates...
Finished; module generated in satelliteqe-config_1.
satelliteqe-config_1/Gemfile
satelliteqe-config_1/Rakefile
satelliteqe-config_1/manifests
satelliteqe-config_1/manifests/init.pp
satelliteqe-config_1/spec
satelliteqe-config_1/spec/classes
satelliteqe-config_1/spec/classes/init_spec.rb
satelliteqe-config_1/spec/spec_helper.rb
satelliteqe-config_1/tests
satelliteqe-config_1/tests/init.pp
satelliteqe-config_1/README.md
satelliteqe-config_1/metadata.json
Comment 12 Lukas Pramuk 2016-07-18 11:45:29 UTC 
In previous comment I missed to paste one cmd:

# cd /root/puppet_work_dir/
# puppet module generate satelliteqe-config_1
...
Comment 13 Lukas Pramuk 2016-07-18 11:54:01 UTC 
forgot to paste relevant part of import.log (with debug msg)

I, [2016-07-18T07:52:36.146520 #30537]  INFO -- : Importing from config-files-latest.csv
D, [2016-07-18T07:52:36.148544 #30537] DEBUG -- : Removing /root/puppet_work_dir/satelliteqe-config_1
D, [2016-07-18T07:52:36.852570 #30537] DEBUG -- : Done reading
I, [2016-07-18T07:52:36.852805 #30537]  INFO -- : Failed puppet module satelliteqe-config_1: metadata.json not created
D, [2016-07-18T07:52:36.853055 #30537] DEBUG -- : name etcconfig1, path /etc/config1, type file
I, [2016-07-18T07:52:36.853158 #30537]  INFO -- : Writing converted files
I, [2016-07-18T07:52:36.853321 #30537]  INFO -- : Summary
I, [2016-07-18T07:52:36.853490 #30537]  INFO -- :   Failed 1 puppet module.
Comment 14 Lukas Pramuk 2016-07-18 12:09:25 UTC 
On debug level I cannot find output of debug "Read #{rd}"

This means that whole parsing cycle L:143-153 is being skipped as was before and thrown EOFException is just masked.

Please see comment #2: 
rd = stdout.readline  <<< 146: failing to parse output, RHEL6 is ok
Comment 15 Tom McKay 2016-07-18 18:24:11 UTC 
Running 'gem install json_pure' will fix the import of configfiles. Still investigating solution.
Comment 16 Lukas Pramuk 2016-07-20 14:25:20 UTC 
VERIFIED.

@satellite-6.2.0-20.3.el7sat.noarch
tfm-rubygem-hammer_cli_import-0.11.1-1.el7sat.noarch

used manual reproducer describe in comment #0

...
4) # hammer import config-file --debug --csv-file config-files-latest.csv --generate-only
Importing from config-files-latest.csv
Writing converted files
Found module defaultorganization-config_1
Summary
  Wrote 1 puppet module.
  Wrote 1 puppet file.
Comment 17 Bryan Kearney 2016-07-27 11:02:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501
Note You need to log in before you can comment on or make changes to this bug.