Bug 2055336

Summary: null value in column "issued_date" violates not-null constraint while syncing EPEL7 repository
Product: Red Hat Satellite Reporter: Yogendra <yyadav>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED NOTABUG QA Contact: Vladimír Sedmík <vsedmik>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.9.9CC: ahumbe, dalley, ggainey, hyu, jsherril, mkushwah
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-10-31 14:57:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yogendra 2022-02-16 17:59:55 UTC
Description of problem:

After upgrading to Red Hat Satellite 6.10 , EPEL 7 repository sync is failing with "null value in column "issued_date" violates not-null constraint" error. 
 

Version-Release number of selected component (if applicable):
6.10.2

How reproducible:
100%

Steps to Reproduce:
1.Create a local repository with empty issued date.
2. Sync the repository against pulp 2 it synced.
3. Sync the repository against pulp 3 it fails.

Actual results:

Below error is observed in /var/log/foreman/production.log

2022-02-10T03:47:36 [E|bac|23373571] null value in column "issued_date" violates not-null constraint
 23373571 | DETAIL:  Failing row contains (4701f8a2-90c8-47a3-bd1f-3e0040634446, FEDORA-EPEL-2018-d57e8992f6, 2021-05-12 05:02:13, Initial package release, null, updates, stable, ykushcmd-1.1.0
-1.el7, ykushcmd-1.1.0-1.el7 enhancement update, 2.0, enhancement, None, , Fedora EPEL 7, Copyright (C) 2021 Red Hat, Inc. and others., , 8eef5f8957f9e9837b8eceff5f9ce0c885dcca2e2606cb351a6ece6362b7f122, f).
 23373571 |  (Katello::Errors::Pulp3Error)
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/pulp3/abstract_async_task.rb:139:in `poll_external_task'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/polling.rb:22:in `run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/cancellable.rb:14:in `run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:572:in `block (3 levels) in execute_run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:32:in `run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.42/app/lib/actions/middleware/remote_action.rb:16:in `run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/progress.rb:17:in `run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/world.rb:31:in `execute'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:571:in `block (2 levels) in execute_run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:570:in `catch'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:570:in `block in execute_run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `block in with_error_handling'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `catch'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `with_error_handling'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:565:in `execute_run'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:286:in `execute'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director.rb:94:in `execute'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors.rb:18:in `run_user_code'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 23373571 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 23373571 | [ sidekiq ]
 23373571 | [ concurrent-ruby ]
2022-02-10T03:47:38 [I|bac|23373571] Task {label: Actions::Katello::Repository::Sync, id: 4c1bdccb-6b6d-4187-8c8e-8b2a683d5ee5, execution_plan_id: b65da9bd-fd6b-4fe2-94e5-9627449a22ca} state changed: stopped  result: warning
2022-02-10T03:47:38 [I|bac|23373571] Task {label: Actions::Katello::Repository::Sync, id: 4c1bdccb-6b6d-4187-8c8e-8b2a683d5ee5, execution_plan_id: b65da9bd-fd6b-4fe2-94e5-9627449a22ca} state changed: stopped  result: warning


Expected results:
Syncing for  repository should complete successfully.

Additional info:

Customer is using Satellite 6.9 as an Upstream URL for EPEL 7 repository configured on Satellite 6.10.2

Comment 14 Yogendra 2022-08-15 11:40:22 UTC
Hello Grant,

First i verified on Customer affected 6.10 Satellite Server if there is any  number of empty-string-issue-date-advisories with below command.
---
# DJANGO_SETTINGS_MODULE=pulpcore.app.settings PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager shell
Python 3.6.8 (default, Aug 13 2020, 07:46:32) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from pulp_rpm.app.models.advisory import UpdateRecord
>>> UpdateRecord.objects.filter(issued_date="").count()
0
---

As the count was zero so requested him to update the issued date on Red Hat Satellite 6.9 from where content is fetched with below command and then try the sync of affected repository on Satellite 6.10.
---
# mongo pulp_database --eval 'db.units_erratum.find({$or: [{issued: null}, {issued: ""}]}, {_id: 1, errata_id: 1, issued: 1, updated: 1}).forEach(function(erratum) { print("Updating issued date for " + erratum.errata_id + " to " + erratum.updated); db.units_erratum.update({_id: erratum._id}, {"$set": {issued: erratum.updated}}) })'

MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017/pulp_database
MongoDB server version: 3.4.9
Updating issued date for FEDORA-EPEL-2018-d57e8992f6 to 2021-05-12 05:02:13
Updating issued date for FEDORA-EPEL-2018-701ce7a3d5 to 2021-05-12 05:06:32
---

But still repository sync is failing with same error  on Red Hat Satellite 6.10
-------------------------------------------
null value in column "issued_date" violates not-null constraint
DETAIL:  Failing row contains (72ef1381-e881-45d1-96ff-54b8f498629a, FEDORA-EPEL-2018-d57e8992f6, 2021-05-12 05:02:13, Initial package release, null, updates, stable, ykushcmd-1.1.0-1.el7, ykushcmd-1.1.0-1.el7 enhancement update, 2.0, enhancement, None, , Fedora EPEL 7, Copyright (C) 2021 Red Hat, Inc. and others., , 8eef5f8957f9e9837b8eceff5f9ce0c885dcca2e2606cb351a6ece6362b7f122, f).
-------------------------------------------

Comment 21 Daniel Alley 2022-10-31 14:57:32 UTC
Closing as NOTABUG because Pulp 3 is correct to reject the metadata - however, this is triggered by a Pulp 2 bug which can be resolved using the command in comment #14

Note that while comment #14 states that it didn't work, it did work later, presumably after the repository got republished.  Making sure the repo is republished (forcibly if need be) after the data is fixed in the database is a necessary step to being able to sync the repo afterwards.

Comment 22 Red Hat Bugzilla 2023-09-18 04:32:11 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days