Description of problem: When running spacewalk-clone-by-date on both RHEL 6/7 x86_64 bit channels a 500 is thrown: [root@demosat ~]# spacewalk-clone-by-date -u rhnadmin -d 2016-05-28 -l rhel-x86_64-server-7 sus_20160528-rhel-x86_64-server-7-pjw Password: Reading repository information. . . . Cloning Errata into sus_20160528-rhel-x86_64-server-7-pjw (1198): ________________________________________ Traceback (most recent call last): File "/usr/bin/spacewalk-clone-by-date", line 253, in <module> sys.exit(abs(main() or 0)) File "/usr/bin/spacewalk-clone-by-date", line 243, in main return cloneByDate.main(args) File "/usr/share/rhn/utils/cloneByDate.py", line 197, in main cloner.clone(options.skip_depsolve) File "/usr/share/rhn/utils/cloneByDate.py", line 358, in clone cloner.process() File "/usr/share/rhn/utils/cloneByDate.py", line 502, in process self.clone() File "/usr/share/rhn/utils/cloneByDate.py", line 594, in clone self.remote_api.clone_errata(self.to_label, errata_set) File "/usr/share/rhn/utils/cloneByDate.py", line 723, in clone_errata self.client.errata.cloneAsOriginal(self.auth_token, to_label, errata_list) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/lib64/python2.6/xmlrpclib.py", line 1243, in request headers xmlrpclib.ProtocolError: <ProtocolError for localhost/rpc/api: 500 Internal Server Error> Looking deeper into the catalina.out logs we find that this is a unique constraint violation in the database: Hibernate: select nextval ('RHN_ERRATAFILE_ID_SEQ') Hibernate: insert into rhnErrataFile (checksum_id, fileName, type, errata_id, id) values (?, ?, ?, ?, ?) 2016-07-12 11:45:49,838 [TP-Processor5] WARN org.hibernate.util.JDBCExceptionReporter - SQL Error: 0, SQLState: 23505 2016-07-12 11:45:49,838 [TP-Processor5] ERROR org.hibernate.util.JDBCExceptionReporter - ERROR: duplicate key value violates unique constraint "rhn_erratafile_eid_file_uq" . . . Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "rhn_erratafile_eid_file_uq" at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2094) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1827) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:508) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:384) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:330) at net.rkbloom.logdriver.LogPreparedStatement.executeUpdate(LogPreparedStatement.java:92) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:79) at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:46) There is a workaround that seems to clear up the database to allow the spacewalk-clone-by-date to run. The following procedure must be completed for each channel a clone would be cloned from: 1) in webui clone a new channel using the same original channel that was used in the failed spacewalk-clone-by-date with no errata, original channel state 2) go to the new channel in the webui (manage channels), click on the channel, then errata and clone errata 3) now go back to channel an sync the errata 4) after syncing is complete, rerun the original spacewalk-clone-by-date and it works I don't know the RCA for this issue but given the WEBUI procedure seems to clean up the DB so clone by date might give us some insight. The packages/files that were hitting UQ were not part of the recent bad errata that was released as I cross-referenced these files. Also, I did not see any git commits that were committed in the past few months that might have caused an issue.
Not sure if this is helpful or will just get me laughed at, but I am seeing the same thing in Spacewalk 2.2 on Oracle Linux 6 [the version of spacewalk supported by Oracle from their spacewalk repo] and running against an Oracle database. # spacewalk-clone-by-date -u USER -c itmisc-dev.cbd ... Cloning Errata into dev-itmisc-ol6u7_x86_64_parent (2573): ________________________________________ Traceback (most recent call last): File "/usr/bin/spacewalk-clone-by-date", line 375, in <module> sys.exit(abs(main() or 0)) File "/usr/bin/spacewalk-clone-by-date", line 365, in main return cloneByDate.main(args) File "/usr/share/rhn/utils/cloneByDate.py", line 214, in main cloner.clone(options.skip_depsolve) File "/usr/share/rhn/utils/cloneByDate.py", line 385, in clone cloner.process() File "/usr/share/rhn/utils/cloneByDate.py", line 525, in process self.clone() File "/usr/share/rhn/utils/cloneByDate.py", line 591, in clone self.remote_api.clone_errata(self.to_label, errata_set) File "/usr/share/rhn/utils/cloneByDate.py", line 735, in clone_errata errata_list) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/lib64/python2.6/xmlrpclib.py", line 1243, in request headers xmlrpclib.ProtocolError: <ProtocolError for localhost/rpc/api: 500 Internal Server Error> And then in catalina.out: Oct 27, 2016 9:02:26 PM redstone.xmlrpc.XmlRpcDispatcher writeError WARNING: redstone.xmlrpc.XmlRpcFault: unhandled internal exception: could not insert: [com.redhat.rhn.domain.errata.impl.PublishedErrataFile] 2016-10-27 21:02:26,550 [TP-Processor15] WARN org.hibernate.util.JDBCExceptionReporter - SQL Error: 1, SQLState: 23000 2016-10-27 21:02:26,550 [TP-Processor15] ERROR org.hibernate.util.JDBCExceptionReporter - ORA-00001: unique constraint (SPACEWALK.RHN_ERRATAFILE_EID _FILE_UQ) violated ... 2016-10-27 21:02:26,550 [TP-Processor15] ERROR org.hibernate.event.def.AbstractFlushingEventListener - Could not synchronize database state with ses sion org.hibernate.exception.ConstraintViolationException: could not insert: [com.redhat.rhn.domain.errata.impl.PublishedErrataFile] at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:71) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2267) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2660) at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:56) at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:250) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:234) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:141) at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298) at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27) at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000) at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338) at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106) at com.redhat.rhn.common.hibernate.ConnectionManager.commitTransaction(ConnectionManager.java:233) at com.redhat.rhn.common.hibernate.HibernateFactory.commitTransaction(HibernateFactory.java:331) at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:299) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) at java.lang.Thread.run(Thread.java:701) Caused by: java.sql.SQLIntegrityConstraintViolationException: ORA-00001: unique constraint (SPACEWALK.RHN_ERRATAFILE_EID_FILE_UQ) violated Unfortunately I'm not able to try the workaround.