Justin,

    When the root cause is pinned down, can you provide some verification steps?

Root cause was that a repository was missing in pulp but not katello.  This was likely due to a skipped or deleted task prior to the upgrade.

This is the command that resolved the issue:



foreman-rake console

User.current = User.first

::Katello::Repository.all.each do |repo|
  begin
    puts "Checking #{repo.name} - #{repo.id}"
    Katello.pulp_server.extensions.repository.retrieve_with_details(repo.pulp_id)
  rescue RestClient::ResourceNotFound
    puts "Deleting #{repo.name} - #{repo.id}"
    repo.destroy!
  end
end



I will go ahead and close this as not a bug as it was more of a data integrity issue.

** WORKAROUND INSTRUCTIONS **

See comment #6

To see which repositories are problematic:


foreman-rake console

User.current = User.first

::Katello::Repository.all.each do |repo|
  begin
    Katello.pulp_server.extensions.repository.retrieve_with_details(repo.pulp_id)
  rescue RestClient::ResourceNotFound
    puts "Problem with Repo:  #{repo.name} - #{repo.id}"
  end
end

This was a data error, if you see this again please feel free to re-open and provide additional inforamtion.

this is still happening and we need to auto-correct this on upgrade

Build: Satellite 6.2.10 snap3 

Upgrade is performed from 6.1.z to 6.2.10 snap3

Verification steps:
1. To reproduce the issue , deleted a repo from pulp

[root@qe-sat6-upgrade-rhel6 ~]# pulp-admin -u admin -p pass rpm repo delete --repo-id=Default_Organization-Dev-rhel6_cv-custom_prd_zoo-custom_repo_zoo
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Repository [Default_Organization-Dev-rhel6_cv-custom_prd_zoo-custom_repo_zoo] successfully deleted.

2. Used the snippet in https://bugzilla.redhat.com/show_bug.cgi?id=1365223#c8 , to check for problem

irb(main):001:0> User.current = User.first
        2017-06-06 07:10:17 [D]   User Load (0.7ms)  SELECT "users".* FROM "users" ORDER BY firstname LIMIT 1
        => #<User id: 3, login: "admin", firstname: "Admin", lastname: "User", mail: "root@satellite", admin: true, last_login_on: "2017-02-08 06:04:17", auth_source_id: 1, create
        d_at: "2016-11-16 15:42:18", updated_at: "2017-02-08 06:04:17", password_hash: "hash", password_salt: "saltedhash", locale: n
        il, helptips_enabled: true, page_size: 25, disabled: false, preferences: #<struct ActiveRecord::AttributeMethods::Serialization::Attribute coder=#<ActiveRecord::Coders::YAMLColumn:0x000000081
        074f0 @object_class=Hash>, value="--- {}\n", state=:serialized>, remote_id: "admin-18911e27", avatar_hash: nil, default_organization_id: nil, default_location_id: nil, lower_login: "admin", m
        ail_enabled: true>
        irb(main):002:0>
        irb(main):003:0* ::Katello::Repository.all.each do |repo|
        irb(main):004:1*   begin
        irb(main):005:2*     Katello.pulp_server.extensions.repository.retrieve_with_details(repo.pulp_id)
        irb(main):006:2>   rescue RestClient::ResourceNotFound
        irb(main):007:2>     puts "Problem with Repo:  #{repo.name} - #{repo.id}"
        irb(main):008:2>   end
        irb(main):009:1> end
        2017-06-06 07:10:20 [D]   Katello::Repository Load (6.4ms)  SELECT "katello_repositories".* FROM "katello_repositories" ORDER BY katello_repositories.name ASC
        Problem with Repo:  custom_repo_zoo - 28
        => [#<Katello::Repository id: 138, name: "busybox", pulp_id: "default_organization-library-docker_view-docker-bus...",



3. Ran the upgrade, satellite-installer --upgrade

Upgrade Step: correct_repositories (this may take a while) ...
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.43/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.135/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
Processing Repository 1/111: Red Hat Enterprise Linux 6 Server Kickstart x86_64 6.8 (1)
.....
.....
...
Processing Repository 23/111: custom_repo_zoo (28)
Repository 28 Missing
Deleting 28
Processing Repository 24/110: Red Hat Enterprise Linux 6 Server Kickstart x86_64 6.8 (31)
......
Processing Repository 111/110: Red Hat Satellite Tools 6.1 for RHEL 7 Server RPMs x86_64 (183)


4. Upgrade was successful
Upgrade Step: set_virt_who_on_pools (this may take a while) ...
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.43/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.135/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
Importing Subscriptions

Upgrade completed!

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1553