Bug 1871043

Summary: [6.8beta] Installation failing with message "Exit with status code: 6 (signal was 6)"
Product: Red Hat Satellite Reporter: Steffen Froemer <sfroemer>
Component: InstallerAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED NOTABUG QA Contact: Devendra Singh <desingh>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.8.0CC: ehelms, zhunting
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-30 15:39:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Steffen Froemer 2020-08-21 08:24:24 UTC
Description of problem:
Installing Satellite-6.8 beta on a fresh RHEL-7.8 system failing with error message Exit with status code: 6 (signal was 6)

Version-Release number of selected component (if applicable):
6.8.0

How reproducible:
only once tries

Steps to Reproduce:
1. RHEL-7.8 DVD installation with yum update
2. satellite-installer --scenario satellite --foreman-initial-organization "ACME.CORP" --foreman-initial-location "LOCAL" --foreman-initial-admin-username admin --foreman-initial-admin-password secret


Actual results:
    [DEBUG 2020-08-20T14:31:58 main] Executing: foreman-maintain packages lock --assumeyes
    [DEBUG 2020-08-20T14:32:01 main] Running preparation steps required to run the next scenarios
    [DEBUG 2020-08-20T14:32:01 main] ================================================================================
    [DEBUG 2020-08-20T14:32:01 main] Check if tooling for package locking is installed:                    ESC[32mESC[1m[OK]ESC[0m
    [DEBUG 2020-08-20T14:32:01 main] --------------------------------------------------------------------------------
    [DEBUG 2020-08-20T14:32:01 main]
    [DEBUG 2020-08-20T14:32:01 main]
    [DEBUG 2020-08-20T14:32:01 main] Running locking of package versions
    [DEBUG 2020-08-20T14:32:01 main] ================================================================================
    [DEBUG 2020-08-20T14:32:01 main] Lock packages:                                                        ESC[32mESC[1m[OK]ESC[0m
    [DEBUG 2020-08-20T14:32:01 main] --------------------------------------------------------------------------------
    [DEBUG 2020-08-20T14:32:01 main]
    [DEBUG 2020-08-20T14:32:01 main] foreman-maintain packages lock --assumeyes finished successfully!
    [DEBUG 2020-08-20T14:32:01 main] Hook /usr/share/foreman-installer/katello/hooks/post/99-version_locking.rb returned true
    [ INFO 2020-08-20T14:32:01 main] All hooks in group post finished
    [DEBUG 2020-08-20T14:32:01 main] Exit with status code: 6 (signal was 6)
    [ERROR 2020-08-20T14:32:01 main] Errors encountered during run:
    [ERROR 2020-08-20T14:32:01 main]  'pulp-manage-db' returned 70 instead of one of [0]
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:183:in `sync'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:241:in `sync'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:136:in `sync_if_needed'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:90:in `block in perform_changes'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `each'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `perform_changes'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:267:in `apply'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:287:in `eval_resource'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:521:in `block in thinmark'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:520:in `thinmark'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:521:in `block in thinmark'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:520:in `thinmark'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:521:in `block in thinmark'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:520:in `thinmark'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:233:in `block in benchmark'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `benchmark'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:387:in `run_internal'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:290:in `override'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:343:in `apply_catalog'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:260:in `block (2 levels) in main'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:290:in `override'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:243:in `block in main'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:290:in `override'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:710:in `exit_on_fail'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
    [ERROR 2020-08-20T14:32:01 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
    [ERROR 2020-08-20T14:32:01 main]  /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns: change from 'notrun' to ['0'] failed: 'pulp-manage-db' returned 70 instead of one of [0]
    [DEBUG 2020-08-20T14:32:01 main] Cleaning /tmp/kafo_installation20200820-21596-1q9cy1q
    [DEBUG 2020-08-20T14:32:01 main] Cleaning /tmp/kafo_installation20200820-21596-4ikhve
    [DEBUG 2020-08-20T14:32:01 main] Cleaning /tmp/default_values.yaml




Expected results:
Installation should work

Additional info:

Comment 1 Steffen Froemer 2020-08-21 08:28:11 UTC
I analyzed further and found it seems some relation to docker

[root@satellite ~]# runuser -u apache pulp-manage-db
...
Applying migration pulp_docker.plugins.migrations.0001_v2_support failed.                                        

Halting migrations due to a migration failure.                                                                   
[Errno 13] Permission denied: '/var/lib/pulp/published/docker'                                                   
Traceback (most recent call last):                                                                               
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 280, in main                                                                                                                                             
    return _auto_manage_db(options)                                                                              
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 347, in _auto_manage_db                                                                                                                                  
    migrate_database(options)                                                                                    
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 175, in migrate_database                                                                                                                                 
    update_current_version=not options.test)                                                                     
  File "/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line 206, in apply_migration                                                                                                                          
    migration.migrate()                                                                                          
  File "/usr/lib/python2.7/site-packages/pulp_docker/plugins/migrations/0001_v2_support.py", line 20, in migrate                                                                                                                  
    misc.mkdir(NEW_DOCKER_V1_PUBLISH_PATH)                                                                       
  File "/usr/lib/python2.7/site-packages/pulp/plugins/util/misc.py", line 49, in mkdir                                                                                                                                            
    os.makedirs(*args, **kwargs)                                                                                 
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs                                                       
    makedirs(head, mode)                                                                                         
  File "/usr/lib64/python2.7/os.py", line 157, in makedirs                                                       
    mkdir(name, mode)                                                                                            
OSError: [Errno 13] Permission denied: '/var/lib/pulp/published/docker' 


[root@satellite ~]# mkdir /var/lib/pulp/published/docker
[root@satellite ~]# chown -R apache:apache /var/lib/pulp/published

==> test again, still failing

[root@satellite ~]# runuser -u apache pulp-manage-db 
...
Applying migration pulp_puppet.plugins.migrations.0005_puppet_module_name_change failed.                                                                                                                                          
                                                                                                                                                                                                                                  
Halting migrations due to a migration failure.                                                                                                                                                                                    
[Errno 13] Permission denied: '/var/lib/pulp/0005_puppet_module_name_change.txt'                                                                                                                                                  
Traceback (most recent call last):                                                                                                                                                                                                
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 280, in main                                                                                                                                             
    return _auto_manage_db(options)                                                                                                                                                                                               
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 347, in _auto_manage_db                                                                                                                                  
    migrate_database(options)                                                                                                                                                                                                     
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 175, in migrate_database                                                                                                                                 
    update_current_version=not options.test)                                                                                                                                                                                      
  File "/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line 206, in apply_migration                                                                                                                          
    migration.migrate()                                                                                                                                                                                                           
  File "/usr/lib/python2.7/site-packages/pulp_puppet/plugins/migrations/0005_puppet_module_name_change.py", line 56, in migrate                                                                                                   
    f = open(path, 'w')                                                                                                                                                                                                           
IOError: [Errno 13] Permission denied: '/var/lib/pulp/0005_puppet_module_name_change.txt'                                                                                                                                         
                                                                                                                                                                                                                                  
[root@satellite ~]# ll /var/lib/pulp/0005_puppet_module_name_change.txt                                                                                                                                                         
ls: cannot access /var/lib/pulp/0005_puppet_module_name_change.txt: No such file or directory  

[root@satellite ~]# touch /var/lib/pulp/0005_puppet_module_name_change.txt                                                                                                                                                      
[root@satellite ~]# chown -R apache:apache /var/lib/pulp/0005_puppet_module_name_change.txt                                                                                                                                     
                                                                                                                                                                                                                                  
[root@satellite ~]# runuser -u apache pulp-manage-db  
...
Applying pulp_puppet.plugins.migrations version 5                                                                
***Note. You may want to re-publish the list of repos found in /var/lib/pulp/0005_puppet_module_name_change.txt.                                                                                                                  
   This migration fixed an issue with modules installation related to wrong puppet_module name.                                                                                                                                   
Migration to pulp_puppet.plugins.migrations version 5 complete in 0.005 seconds.                                                                                                                                                  
Database migrations complete.     


Now satellite-installer finished successfully

Comment 2 Eric Helms 2020-08-26 19:07:11 UTC
Was there anything unique about this system setup? Permissions on it? Mounted directories anywhere related to /var/lib/pulp ?

Comment 3 Steffen Froemer 2020-09-02 07:07:59 UTC
(In reply to Eric Helms from comment #2)
> Was there anything unique about this system setup? Permissions on it?
> Mounted directories anywhere related to /var/lib/pulp ?

not that I'm aware of.
I did it again and than it worked flawless. The only difference I've made on the second try, was not re-use the LVs for 
  - pulp
  - pulp-cache
  - mongodb

In my first try, where I run into that error, I cloned the VM (including disks) from another, older Satellite 6 with separate logical volumes for 
  - pulp
  - pulp-cache
  - mongodb

Those disks were formatted (mkfs.xfs ... --force) before installing satellite packages.

On my second try, I removed these virtual disks completely and created new ones. Everything else was same.

Hope that helps.

Comment 5 Eric Helms 2020-11-11 16:06:36 UTC
Given this issue stems from the re-use of logical volumes attached from an older Satellite 6, I am inclined to close not a bug this issue.

Comment 6 Steffen Froemer 2020-11-13 08:10:48 UTC
I'm fine with closing this issue. Although it's hard to understand, why a re-use of a logical volume will prevent files from creation.