Bug 873747

Summary: Spacewalk 1.8 | Error after "spacewalk-schema-upgrade"
Product: [Community] Spacewalk Reporter: Holger Schmitt <holger.schmitt>
Component: InstallationAssignee: Jan Pazdziora <jpazdziora>
Status: CLOSED NOTABUG QA Contact: Red Hat Satellite QA List <satqe-list>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 1.8CC: jpazdziora, tlestach
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-11-14 11:24:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1484117    
Attachments:
Description Flags
Tracebacks none

Description Holger Schmitt 2012-11-06 15:51:10 UTC
Created attachment 639420 [details]
Tracebacks

Description of problem:
After spacewalk update and schema upgrade I get "Internal Server Error".


Version-Release number of selected component (if applicable):


How reproducible:
always


Steps to Reproduce:
Clicking on detail pages:
- https://myserver/rhn/systems/details/Overview.do?sid=1000010052
- https://myserver/rhn/channels/ChannelDetail.do?cid=108)
- https://myserver/rhn/channels/manage/Edit.do?cid=101
- ...

  
Actual results:
"Internal Server Error"


Expected results:
Show detail pages


Additional info:
For tracebacks see attachment.


Software-Versions:
[root@myserver schema-upgrade]# uname -a
Linux myserver.adm-group.com 2.6.32-279.9.1.el6.x86_64 #1 SMP Tue Sep 25 14:55:44 CDT 2012 x86_64 x86_64 x86_64 GNU/Linux


[root@myserver schema-upgrade]# rpm -qa|grep spacewalk|sort
spacewalk-admin-1.8.6-1.el6.noarch
spacewalk-backend-1.8.85-1.el6.noarch
spacewalk-backend-app-1.8.85-1.el6.noarch
spacewalk-backend-applet-1.8.85-1.el6.noarch
spacewalk-backend-config-files-1.8.85-1.el6.noarch
spacewalk-backend-config-files-common-1.8.85-1.el6.noarch
spacewalk-backend-config-files-tool-1.8.85-1.el6.noarch
spacewalk-backend-iss-1.8.85-1.el6.noarch
spacewalk-backend-iss-export-1.8.85-1.el6.noarch
spacewalk-backend-libs-1.9.2-1.el6.noarch
spacewalk-backend-package-push-server-1.8.85-1.el6.noarch
spacewalk-backend-server-1.8.85-1.el6.noarch
spacewalk-backend-sql-1.8.85-1.el6.noarch
spacewalk-backend-sql-postgresql-1.8.85-1.el6.noarch
spacewalk-backend-tools-1.8.85-1.el6.noarch
spacewalk-backend-xml-export-libs-1.8.85-1.el6.noarch
spacewalk-backend-xmlrpc-1.8.85-1.el6.noarch
spacewalk-base-1.8.49-1.el6.noarch
spacewalk-base-minimal-1.8.49-1.el6.noarch
spacewalk-branding-1.8.7-1.el6.noarch
spacewalk-certs-tools-1.9.1-1.el6.noarch
spacewalk-client-repo-1.9-1.el6.noarch
spacewalk-common-1.8.6-1.el6.noarch
spacewalk-config-1.8.6-1.el6.noarch
spacewalk-doc-indexes-1.1.1-1.el6.noarch
spacewalk-grail-1.8.49-1.el6.noarch
spacewalk-html-1.8.49-1.el6.noarch
spacewalk-java-1.8.180-1.el6.noarch
spacewalk-java-config-1.8.180-1.el6.noarch
spacewalk-java-lib-1.8.180-1.el6.noarch
spacewalk-java-postgresql-1.8.180-1.el6.noarch
spacewalk-jpp-workaround-1.0.4-1.el6.noarch
spacewalk-monitoring-1.4.1-1.el6.noarch
spacewalk-monitoring-selinux-1.8.4-1.el6.noarch
spacewalk-postgresql-1.8.6-1.el6.noarch
spacewalk-pxt-1.8.49-1.el6.noarch
spacewalk-repo-1.8-4.el6.noarch
spacewalk-schema-1.8.88-1.el6.noarch
spacewalk-search-1.8.6-1.el6.noarch
spacewalk-selinux-1.8.2-1.el6.noarch
spacewalk-setup-1.8.24-1.el6.noarch
spacewalk-setup-jabberd-1.8.7-1.el6.noarch
spacewalk-slf4j-1.6.1-1.el6.noarch
spacewalk-sniglets-1.8.49-1.el6.noarch
spacewalk-taskomatic-1.8.180-1.el6.noarch


root@myserver schema-upgrade]# spacewalk-schema-upgrade
Schema upgrade: [spacewalk-schema-1.8.88-1.el6] -> [spacewalk-schema-1.8.88-1.el6]
Your database schema already matches the schema package version [spacewalk-schema-1.8.88-1.el6].


[root@myserver schema-upgrade]# rpm -qa|grep postgres|sort
postgresql-8.4.13-1.el6_3.x86_64
postgresql-contrib-8.4.13-1.el6_3.x86_64
postgresql-jdbc-8.4.701-8.el6.noarch
postgresql-libs-8.4.13-1.el6_3.x86_64
postgresql-server-8.4.13-1.el6_3.x86_64
spacewalk-backend-sql-postgresql-1.8.85-1.el6.noarch
spacewalk-java-postgresql-1.8.180-1.el6.noarch
spacewalk-postgresql-1.8.6-1.el6.noarch


[root@myserver schema-upgrade]# rpm -qa|grep tomcat
apache-tomcat-apis-0.1-1.el6.noarch
tomcat5-jsp-2.0-api-5.5.27-7.jpp5.noarch
tomcat6-el-2.1-api-6.0.24-45.el6.noarch
tomcat6-servlet-2.5-api-6.0.24-45.el6.noarch
tomcat6-lib-6.0.24-45.el6.noarch
tomcat6-6.0.24-45.el6.noarch
tomcat6-jsp-2.1-api-6.0.24-45.el6.noarch
tomcat5-servlet-2.4-api-5.5.27-7.jpp5.noarch

Comment 1 Jan Pazdziora 2012-11-06 16:33:32 UTC
What Spacewalk version was this installation upgraded from?

Comment 2 Holger Schmitt 2012-11-06 16:43:04 UTC
(In reply to comment #1)
> What Spacewalk version was this installation upgraded from?

Always from the latest (http://spacewalk.redhat.com/yum/latest/RHEL/6/x86_64/)

Maybe there is a problem with the db schema. Is it possible to repair it, without losing data?

Comment 3 Jan Pazdziora 2012-11-06 17:20:42 UTC
(In reply to comment #2)
> (In reply to comment #1)
> > What Spacewalk version was this installation upgraded from?
> 
> Always from the latest
> (http://spacewalk.redhat.com/yum/latest/RHEL/6/x86_64/)
> 
> Maybe there is a problem with the db schema. Is it possible to repair it,
> without losing data?

Restoring your 1.7 schema from backup and re-runing the schema upgrade would be the correct way.

But we would also like to know what caused this -- what is in your schema upgrade logs, etc. The missing rhnabrtinfo table sounds like something non-trivially wrong happened there.

Comment 4 Holger Schmitt 2012-11-06 18:00:14 UTC
Here the recent files in "/var/log/spacewalk/schema-upgrade/"

Do you need any more files?


--------------------------------------------------------

[root@myserver ~]# cat /var/log/spacewalk/schema-upgrade/20121106-161051-script.sql 
update rhnVersionInfo
       set label = 'schema-from-' || to_char(created, 'YYYYMMDD-HH24MISS'),
           modified = current_timestamp
       where label = 'schema';
commit;
-- oracle equivalent source sha1 cb07e6926fb5f79d7645be270e632f9168631964
--
-- Copyright (c) 2010--2012 Red Hat, Inc.
--
-- This software is licensed to you under the GNU General Public License,
-- version 2 (GPLv2). There is NO WARRANTY for this software, express or
-- implied, including the implied warranties of MERCHANTABILITY or FITNESS
-- FOR A PARTICULAR PURPOSE. You should have received a copy of GPLv2
-- along with this software; if not, see
-- http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt.
--
-- Red Hat trademarks are not licensed under GPLv2. No permission is
-- granted to use or replicate Red Hat trademarks that are incorporated
-- in this software or its documentation.
--

--
-- empty varchars are not allowed for the oracle-postgres compatibility
-- create constraints on all varchar columns (all tables for the current user)
-- and returns number of errors during processing (if not 0 then
-- check the pgsql log -- usually /var/lib/pgsql/data/pg_log -- for errors)
--
create or replace function create_varnull_constriants() returns integer as $$
declare
    tabs record;
    total integer default 0;
begin

    for tabs in select 
        c.relname as "tab",
        a.attname as "col"
    from
        pg_catalog.pg_attribute a
        left outer join pg_catalog.pg_class c on a.attrelid = c.oid 
    where
        -- skip system columns
        a.attnum > 0
        -- skip dropped columns
        and not a.attisdropped
        -- filter only varchars
        and a.atttypid = 1043
        -- skip cols that already has this constraint
        and not exists (
            select 1 from pg_catalog.pg_constraint 
            where conname = 'vn_' || c.relname || '_' || a.attname
        )
        -- filter only tables owned by current user
        and a.attrelid in (
            select c.oid from pg_catalog.pg_class c
            where relkind = 'r' and pg_catalog.pg_table_is_visible(c.oid) and relowner = (
                select oid from pg_catalog.pg_authid where rolname = current_user
            )
        ) loop
        begin
            -- create constraint
            execute 'alter table ' || tabs.tab || ' add constraint vn_' ||
                tabs.tab || '_' || tabs.col || ' check (' || tabs.col || ' <> '''')';
            -- count them
        exception when others then
            total = total + 1;
            raise warning '% unable to create constraint for %.%', now(), tabs.tab, tabs.col;
        end;
    end loop;

    return total;
end;
$$ language plpgsql;

select create_varnull_constriants();


insert into rhnVersionInfo
       ( label, name_id, evr_id, created, modified )
       values ('schema', lookup_package_name('spacewalk-schema'),
               lookup_evr(null, '1.8.88' , '1.el6' ),
               current_timestamp, current_timestamp );
commit;


--------------------------------------------------------


[root@myserver ~]# cat /var/log/spacewalk/schema-upgrade/20121106-161051-to-spacewalk-schema-1.8.log 
UPDATE 1
COMMIT
CREATE FUNCTION
 create_varnull_constriants 
----------------------------
                          0
(1 row)

INSERT 0 1
COMMIT

Comment 5 Jan Pazdziora 2012-11-06 18:46:36 UTC
(In reply to comment #4)
> Here the recent files in "/var/log/spacewalk/schema-upgrade/"
> 
> Do you need any more files?
> 
> 
> --------------------------------------------------------
> 
> [root@myserver ~]# cat
> /var/log/spacewalk/schema-upgrade/20121106-161051-script.sql 

Yes, the previous ones. This ones are from the run when the schema version was already 1.8.88-1.el6. Which is however strange.

Did you by any chance migrate your Spacewalk server installation from (say) EL5 to EL6?

Comment 6 Jan Pazdziora 2012-11-06 18:48:57 UTC
Could you please paste output of

select label, created, ( select evr from rhnpackageevr where evr_id = id ) as evr from rhnVersionInfo order by created ;

when run in your psql?

Comment 7 Jan Pazdziora 2012-11-10 16:23:19 UTC
Also, could you show a list of databases in your PostgreSQL?

Command

\l

in psql.

Comment 8 Holger Schmitt 2012-11-13 16:34:27 UTC
Meanwhile I set up a new 1.8 installation, because we need a working spacewalk server.

Here are the informations from the database, I backed up before. 



spaceschema=# \l
                                   List of databases
    Name     |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
-------------+----------+----------+-------------+-------------+-----------------------
 postgres    | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 spaceschema | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0   | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
             |          |          |             |             | postgres=CTc/postgres
 template1   | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
             |          |          |             |             | postgres=CTc/postgres
(4 rows)


spaceschema=# select label, created, ( select evr from rhnpackageevr where evr_id = id ) as evr from rhnVersionInfo order by created ;
            label            |            created            |       evr       
-----------------------------+-------------------------------+-----------------
 schema-from-20120524-175710 | 2012-05-24 17:57:10.916031+02 | (,1.8.39,1.el6)
 schema-from-20120525-095231 | 2012-05-25 09:52:31.403195+02 | (,1.8.40,1.el6)
 schema                      | 2012-11-06 16:11:18.546448+01 | (,1.8.88,1.el6)
(3 rows)

Comment 9 Jan Pazdziora 2012-11-14 11:24:38 UTC
(In reply to comment #8)
> Meanwhile I set up a new 1.8 installation, because we need a working
> spacewalk server.

And I assume it works fine.

> spaceschema=# select label, created, ( select evr from rhnpackageevr where
> evr_id = id ) as evr from rhnVersionInfo order by created ;
>             label            |            created            |       evr    
> 
> -----------------------------+-------------------------------+---------------
> --
>  schema-from-20120524-175710 | 2012-05-24 17:57:10.916031+02 |
> (,1.8.39,1.el6)
>  schema-from-20120525-095231 | 2012-05-25 09:52:31.403195+02 |
> (,1.8.40,1.el6)
>  schema                      | 2012-11-06 16:11:18.546448+01 |
> (,1.8.88,1.el6)
> (3 rows)

This means the original installation was a Spacewalk nightly.

We don't support upgrades from Spacewalk nightly to next released release.

Comment 10 Eric Herget 2017-09-28 17:56:14 UTC
This BZ closed some time during 2.5, 2.6 or 2.7.  Adding to 2.7 tracking bug.