Bug 1325880 - hammer import config-file on RHEL7 fails to generate puppet modules
Summary: hammer import config-file on RHEL7 fails to generate puppet modules
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Uncategorized
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high vote
Target Milestone: Unspecified
Assignee: Tom McKay
QA Contact: Lukas Pramuk
URL: http://projects.theforeman.org/issues...
Whiteboard:
: 1315260 1350192 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-11 12:17 UTC by Lukas Pramuk
Modified: 2020-07-16 08:44 UTC (History)
5 users (show)

Fixed In Version: rubygem-hammer_cli_import-0.11.1-1
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-27 11:02:03 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 14616 0 None None None 2016-04-22 16:13:51 UTC

Description Lukas Pramuk 2016-04-11 12:17:01 UTC
Description of problem:
hammer import config-file on RHEL7 fails to generate puppet modules.
It uses "puppet module generate <name>" and on RHEL7 it fails to parse command interview questions at all, while on RHEL6 command output with interview questions is parsed successfully.

Version-Release number of selected component (if applicable):
@Sat6.2.0-Beta-Snap
tfm-rubygem-hammer_cli_import-0.10.23-1.el7sat.noarch

How reproducible:
always on RHEL7

Steps to Reproduce:
1. # cat users.csv 
organization_id,organization,user_id,username,last_name,first_name,position,email,role,creation_time,last_login_time,active
1,SatelliteQE,1,admin,Admin,Admin,,root@localhost,Satellite Administrator,2014-09-10 19:39:59,2016-04-07 01:00:14,enabled

2. # hammer import organization --verbose --csv-file users.csv 
Importing from users.csv
Creating new organization: SatelliteQE
Summary
  Created 1 organization.

3. # cat config-files-latest.csv 
org_id,channel_id,channel,channel_type,path,file_type,file_id,revision,is_binary,contents,delim_start,delim_end,username,groupname,filemode,symbolic_link,selinux_ctx
1,3,config-1,normal,/etc/config1,file,3,2,N,config1 file ver.2,{|,|},apache,apache,0,,

4. # hammer import config-file --debug --csv-file config-files-latest.csv --generate-only
Importing from config-files-latest.csv
Caught EOFError:end of file reached while processing CSV line:
{"org_id"=>"1", "channel_id"=>"3", "channel"=>"config-1", "channel_type"=>"normal", "path"=>"/etc/config1", "file_type"=>"file", "file_id"=>"3", "revision"=>"2", "is_binary"=>"N", "contents"=>"config1 file ver.2", "delim_start"=>"{|", "delim_end"=>"|}", "username"=>"apache", "groupname"=>"apache", "filemode"=>"0", "symbolic_link"=>nil, "selinux_ctx"=>nil}
Writing converted files
Found module satelliteqe-config_1
Caught Errno::ENOENT:No such file or directory @ dir_s_mkdir - /root/puppet_work_dir/satelliteqe-config_1/files while post_import
Summary
  No action taken.

5. # less import.log:
...
I, [2016-04-11T08:03:46.664346 #17137]  INFO -- : Importing from config-files-latest.csv
D, [2016-04-11T08:03:46.665697 #17137] DEBUG -- : Removing /root/puppet_work_dir/satelliteqe-config_1
E, [2016-04-11T08:03:47.255283 #17137] ERROR -- : Caught EOFError:end of file reached while processing CSV line:
{"org_id"=>"1", "channel_id"=>"3", "channel"=>"config-1", "channel_type"=>"normal", "path"=>"/etc/config1", "file_type"=>"file", "file_id"=>"3", "revision"=>"2", "is_binary"=>"N", "contents"=>"config1 file ver.2", "delim_start"=>"{|", "delim_end"=>"|}", "username"=>"apache", "groupname"=>"apache", "filemode"=>"0", "symbolic_link"=>nil, "selinux_ctx"=>nil}
E, [2016-04-11T08:03:47.255556 #17137] ERROR -- : /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:146:in `readline'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:146:in `block (2 levels) in generate_module_template_for'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:143:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:143:in `block in generate_module_template_for'
/opt/rh/rh-ruby22/root/usr/share/ruby/open3.rb:199:in `popen_run'
/opt/rh/rh-ruby22/root/usr/share/ruby/open3.rb:93:in `popen3'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:141:in `generate_module_template_for'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:193:in `generate_module'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/configfile.rb:251:in `import_single_row'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/base.rb:527:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/base.rb:527:in `block (2 levels) in cvs_iterate'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/importtools.rb:324:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/importtools.rb:324:in `handle_missing_and_supress'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/base.rb:526:in `block in cvs_iterate'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/csvhelper.rb:55:in `block in csv_each'
/opt/rh/rh-ruby22/root/usr/share/ruby/csv.rb:1739:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/csvhelper.rb:44:in `csv_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/base.rb:525:in `cvs_iterate'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/base.rb:533:in `import'
/opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_import-0.10.23/lib/hammer_cli_import/base.rb:569:in `execute'


Actual results:
fails to generate a puppet module (afterwards it is uploaded)

Expected results:
generates a puppet module successfully (to be then uploaded onto server)

Comment 1 Lukas Pramuk 2016-04-11 12:22:50 UTC
Additional info:
lib/hammer_cli_import/configfile.rb:146 in `readline' - it fails to parse command output with interview questions at all on RHEL7.

puppet module generate command which is being run by hammer import works itself:
# cd /root/puppet_work_dir/
# puppet module generate satelliteqe-config_1
We need to create a metadata.json file for this module.  Please answer the
following questions; if the question is not applicable to this module, feel free
to leave it blank.

Puppet uses Semantic Versioning (semver.org) to version modules.
What version is this module?  [0.1.0]
--> 

Who wrote this module?  [satelliteqe]
--> 

What license does this module code fall under?  [Apache 2.0]
--> 

How would you describe this module in a single sentence?
--> 

Where is this module's source code repository?
--> 

Where can others go to learn more about this module?
--> 

Where can others go to file issues about this module?
--> 

----------------------------------------
{
  "name": "satelliteqe-config_1",
  "version": "0.1.0",
  "author": "satelliteqe",
  "summary": null,
  "license": "Apache 2.0",
  "source": "",
  "project_page": null,
  "issues_url": null,
  "dependencies": [
    {"name":"puppetlabs-stdlib","version_requirement":">= 1.0.0"}
  ]
}
----------------------------------------

About to generate this metadata; continue? [n/Y]
--> Y

Notice: Generating module at /root/puppet_work_dir/satelliteqe-config_1...
Notice: Populating templates...
Finished; module generated in satelliteqe-config_1.
satelliteqe-config_1/Gemfile
satelliteqe-config_1/Rakefile
satelliteqe-config_1/manifests
satelliteqe-config_1/manifests/init.pp
satelliteqe-config_1/spec
satelliteqe-config_1/spec/classes
satelliteqe-config_1/spec/classes/init_spec.rb
satelliteqe-config_1/spec/spec_helper.rb
satelliteqe-config_1/tests
satelliteqe-config_1/tests/init.pp
satelliteqe-config_1/README.md
satelliteqe-config_1/metadata.json

Comment 2 Lukas Pramuk 2016-04-11 12:25:52 UTC
lib/hammer_cli_import/configfile.rb:

            rd = ''
            until rd.include? '?'
              rd = stdout.readline  <<< 146: failing to parse output, RHEL6 is ok 
              #debug "Read #{rd}"
            end

Comment 4 Tom McKay 2016-04-13 14:49:23 UTC
Created redmine issue http://projects.theforeman.org/issues/14616 from this bug

Comment 5 Bryan Kearney 2016-04-13 16:00:45 UTC
Upstream bug component is Uncategorized

Comment 6 Bryan Kearney 2016-04-13 20:00:45 UTC
Upstream bug component is Uncategorized

Comment 7 Lukas Pramuk 2016-06-27 08:27:51 UTC
*** Bug 1350192 has been marked as a duplicate of this bug. ***

Comment 8 Bryan Kearney 2016-07-05 12:45:52 UTC
*** Bug 1315260 has been marked as a duplicate of this bug. ***

Comment 9 Bryan Kearney 2016-07-14 18:14:29 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/14616 has been closed

Comment 10 Lukas Pramuk 2016-07-18 11:38:10 UTC
FailedQA.

@satellite-6.2.0-20.1.el7sat.noarch
tfm-rubygem-hammer_cli_import-0.11.0-1.el7sat.noarch

1) # cat > users.csv << EOF
organization_id,organization,user_id,username,last_name,first_name,position,email,role,creation_time,last_login_time,active
1,SatelliteQE,1,admin,Admin,Admin,,root@localhost,Satellite Administrator,2014-09-10 19:39:59,2016-04-07 01:00:14,enabled
EOF

2) # hammer import organization --verbose --csv-file users.csv
Importing from users.csv
Creating new organization: SatelliteQE
Summary
  Created 1 organization.

3) # cat > config-files-latest.csv << EOF
org_id,channel_id,channel,channel_type,path,file_type,file_id,revision,is_binary,contents,delim_start,delim_end,username,groupname,filemode,symbolic_link,selinux_ctx
1,3,config-1,normal,/etc/config1,file,3,2,N,config1 file ver.2,{|,|},apache,apache,0,,
EOF

4) # hammer import config-file --debug --csv-file config-files-latest.csv --generate-only
Importing from config-files-latest.csv
Failed puppet module satelliteqe-config_1: metadata.json not created
Writing converted files
Summary
  Failed 1 puppet module.


>>> failed to create puppet module (on rhel7)

Comment 11 Lukas Pramuk 2016-07-18 11:42:17 UTC
...however, puppet module generate is successful:

# puppet module generate satelliteqe-config_1
We need to create a metadata.json file for this module.  Please answer the
following questions; if the question is not applicable to this module, feel free
to leave it blank.

Puppet uses Semantic Versioning (semver.org) to version modules.
What version is this module?  [0.1.0]
--> 

Who wrote this module?  [satelliteqe]
--> 

What license does this module code fall under?  [Apache 2.0]
--> 

How would you describe this module in a single sentence?
--> 

Where is this module's source code repository?
--> 

Where can others go to learn more about this module?
--> 

Where can others go to file issues about this module?
--> 

----------------------------------------
{
  "name": "satelliteqe-config_1",
  "version": "0.1.0",
  "author": "satelliteqe",
  "summary": null,
  "license": "Apache 2.0",
  "source": "",
  "project_page": null,
  "issues_url": null,
  "dependencies": [
    {"name":"puppetlabs-stdlib","version_requirement":">= 1.0.0"}
  ]
}
----------------------------------------

About to generate this metadata; continue? [n/Y]
--> 

Notice: Generating module at /root/puppet_work_dir/satelliteqe-config_1...
Notice: Populating templates...
Finished; module generated in satelliteqe-config_1.
satelliteqe-config_1/Gemfile
satelliteqe-config_1/Rakefile
satelliteqe-config_1/manifests
satelliteqe-config_1/manifests/init.pp
satelliteqe-config_1/spec
satelliteqe-config_1/spec/classes
satelliteqe-config_1/spec/classes/init_spec.rb
satelliteqe-config_1/spec/spec_helper.rb
satelliteqe-config_1/tests
satelliteqe-config_1/tests/init.pp
satelliteqe-config_1/README.md
satelliteqe-config_1/metadata.json

Comment 12 Lukas Pramuk 2016-07-18 11:45:29 UTC
In previous comment I missed to paste one cmd:

# cd /root/puppet_work_dir/
# puppet module generate satelliteqe-config_1
...

Comment 13 Lukas Pramuk 2016-07-18 11:54:01 UTC
forgot to paste relevant part of import.log (with debug msg)

I, [2016-07-18T07:52:36.146520 #30537]  INFO -- : Importing from config-files-latest.csv
D, [2016-07-18T07:52:36.148544 #30537] DEBUG -- : Removing /root/puppet_work_dir/satelliteqe-config_1
D, [2016-07-18T07:52:36.852570 #30537] DEBUG -- : Done reading
I, [2016-07-18T07:52:36.852805 #30537]  INFO -- : Failed puppet module satelliteqe-config_1: metadata.json not created
D, [2016-07-18T07:52:36.853055 #30537] DEBUG -- : name etcconfig1, path /etc/config1, type file
I, [2016-07-18T07:52:36.853158 #30537]  INFO -- : Writing converted files
I, [2016-07-18T07:52:36.853321 #30537]  INFO -- : Summary
I, [2016-07-18T07:52:36.853490 #30537]  INFO -- :   Failed 1 puppet module.

Comment 14 Lukas Pramuk 2016-07-18 12:09:25 UTC
On debug level I cannot find output of debug "Read #{rd}"

This means that whole parsing cycle L:143-153 is being skipped as was before and thrown EOFException is just masked.

Please see comment #2: 
rd = stdout.readline  <<< 146: failing to parse output, RHEL6 is ok

Comment 15 Tom McKay 2016-07-18 18:24:11 UTC
Running 'gem install json_pure' will fix the import of configfiles. Still investigating solution.

Comment 16 Lukas Pramuk 2016-07-20 14:25:20 UTC
VERIFIED.

@satellite-6.2.0-20.3.el7sat.noarch
tfm-rubygem-hammer_cli_import-0.11.1-1.el7sat.noarch

used manual reproducer describe in comment #0

...
4) # hammer import config-file --debug --csv-file config-files-latest.csv --generate-only
Importing from config-files-latest.csv
Writing converted files
Found module defaultorganization-config_1
Summary
  Wrote 1 puppet module.
  Wrote 1 puppet file.

Comment 17 Bryan Kearney 2016-07-27 11:02:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501


Note You need to log in before you can comment on or make changes to this bug.