Bug 604175

See also SR2024138

Hello,

sending additional investigation details to complete Description and comment #1.

satellite-sync doesn't print the custom channels as imported, because it considers as imported only Red Hat channels. The Red Hat channels have a NULL as an organization id in Satellite db and sattelite-sync considers channels as imported only when org-id is NULL (where the other custom/cloned channels have their org-id set to 1 or greater).

Hard to tell whether this is an intended behaviour or bug, but printing a channel as not imported if we just imported it is a bug, although, rather cosmetic one as the channel itself gets properly imported by satellite-sync.

The current behaviour:

~]# satellite-sync -c testicek -m ./testicek/
...
    Import complete:
        Begin time: Wed Feb 23 06:18:08 2011
        End time:   Wed Feb 23 06:18:10 2011
        Elapsed:    0 hours, 0 minutes, 1 seconds
        
~]#

note '.' next to channel name 'testicek', there should be 'p' instead of '.':

~]# satellite-sync -c testicek -m ./testicek/ -l
...
06:40:34    p = previously imported/synced channel
06:40:34    . = channel not yet imported/synced
06:40:34    base-channels:
06:40:34       . testicek                                    1       full import from Wed Feb 23 00:51:30 2011
...
~]#

The above is caused by a sql call which gets channels with NULL org-id only and related conditions which decide whether the channels are imported or not. Possible solution would be to search also for channels with non-NULL org-id:

=== <snip> ===
--- spacewalk-backend-0.5.28/satellite_tools/satsync.py.orig    2011-02-22 16:42:06.946528032 +0100
+++ spacewalk-backend-0.5.28/satellite_tools/satsync.py 2011-02-22 16:42:30.137439447 +0100
@@ -2104,7 +2104,7 @@ def _getImportedChannels():
     "Retrieves the channels already imported in the satellite's database"
 
     try:
-        h = rhnSQL.prepare("""select label from rhnChannel where org_id is null""")
+        h = rhnSQL.prepare("""select label from rhnChannel""")
         h.execute()
         return map(lambda x: x['label'], h.fetchall_dict() or [])
     except (SQLError, SQLSchemaError, SQLConnectError), e:
=== </snip> ===

but I epxect it breaks some background intention as it seems to be intended to search only for Red Hat content, or should there be some flag implemented in db which would tell the channel was imported or not instead of using org-id for decision?

Anyway, if you try the above changes, satellite-sync starts to print correctly 'p' insted of '.' and other functionality seems to be untouched.



The following text and code snippets describe codepath leading to the problem:

_compute_channel_request() (line #655) internally calls a function which counts&filters all the channels and a result is passed through method's internal variables (_available,_requested,_imported) into _printChannelTree() (#663) which finally prints all the channels (-l option):

=== <snip> ===
spacewalk-backend-0.5.28/satellite_tools/satsync.py:
 630     def process_channels(self):
...
 653         # clean up the channel request and populate self._channel_request
 654         # This essentially determines which channels are to be imported
 655         self._compute_channel_request()         <<<--- 1
 656 
 657         # print out the relevant channel tree
 658         #3/6/06 wregglej 183213 Don't print out the end-of-service message if 
 659         #satellite-sync is running with the --mount-point (-m) option. If it
 660         #did, it would incorrectly list channels as end-of-service if they had been 
 661         #synced already but aren't in the channel dump.
 662         if self.mountpoint:
 663             self._printChannelTree(doEOSYN=0)   <<<--- 2
 664         else:
 665             self._printChannelTree(doEOSYN=1)
 666 
 667         if self.listChannelsYN:
 668             # We're done here
 669             return
=== </snip> ===

The following function is only "frontend" calling a function (line #780) which obtains all the imported channels (through the mentioned sql) where only channels with org-id = NULL get parsed, then there is a line #796 compute() - this is the function which decides according to results from sql which channels are imported, avaliable to be imported or end-of-life:

=== <snip> ===
spacewalk-backend-0.5.28/satellite_tools/satsync.py:
 773     def _compute_channel_request(self):
 774         """ channels request is verify and categorized.
 775 
 776             NOTE: self.channel_req *will be* initialized by this method
 777         """
 778 
 779         # channels already imported, and all channels
 780         importedChannels = _getImportedChannels()    <<<--- 1.1
 781         availableChannels = self._channel_collection.get_channel_labels()
 782         log(6, 'XXX: imported channels: %s' % importedChannels, 1)
 783         log(6, 'XXX:   cached channels: %s' % availableChannels, 1)
 784 
 785         # if requested a channel list, we are requesting all channels
 786         if self.listChannelsYN:
 787             requested_channels = availableChannels
 788             log(6, 'XXX: list channels called', 1)
 789         else:
 790             requested_channels = self._requested_channels
 791 
 792         rc = req_channels.RequestedChannels(requested_channels)
 793         rc.set_available(availableChannels)
 794         rc.set_imported(importedChannels)
 795         # rc does all the logic of doing intersections and stuff
 796         rc.compute()   <<<--- 1.2
=== </snip> ===

=== <snip> ===
spacewalk-backend-0.5.28/satellite_tools/satsync.py:
2103 def _getImportedChannels():
2104     "Retrieves the channels already imported in the satellite's database"
2105 
2106     try:
2107         h = rhnSQL.prepare("""select label from rhnChannel where org_id is null""")  <<<--- 1.1.1 sql which considers channels as imported according to NULL org-id
2108         h.execute()
2109         return map(lambda x: x['label'], h.fetchall_dict() or [])
=== </snip> ===

Results obtained by the above sql:

SQL> select label from rhnChannel where org_id is null;

LABEL
--------------------------------------------------------------------------------
rhel-x86_64-server-5
rhn-tools-rhel-x86_64-server-5
rhel-i386-server-5
rhn-tools-rhel-i386-server-5
rhel-x86_64-server-6
rhn-tools-rhel-x86_64-server-6
redhat-rhn-proxy-5.4-server-x86_64-5
rhel-i386-server-6
rhn-tools-rhel-i386-server-6

9 rows selected.

SQL> select label from rhnChannel;

LABEL
--------------------------------------------------------------------------------
...
testicek
...

18 rows selected.

SQL>

The following is the function which decides whether channel is requested - line #109 (-c option), or whether is available to be imported - #110 (through local xml or -m option) and whether it is imported already - #113 (list/array of channels obtained from the sql):

=== <snip> ===
spacewalk-backend-0.5.28/satellite_tools/req_channels.py:
105     def compute(self):
106         self._reset_computed()
107         available = self._available.copy()
108         imported = self._imported.copy()
109         for c in self._requested.keys():  <<<--- requested?
110             if self._available.has_key(c): <<<--- available?
111                 del available[c]
112                 # Channel exists
113                 if self._imported.has_key(c):  <<<--- imported?
114                     del imported[c]
115                     self._requested_imported.append(c) <<<--- if yes, then print 'p'
116                     continue
117                 self._requested_new.append(c) <<<--- if no, print '.'
118                 continue
119             # Requested channel not available
120             if self._imported.has_key(c):
121                 del imported[c]
122                 self._end_of_service.append(c)
123                 continue
124             # Typo
125             self._typos.append(c)
126 
127         for c in available.keys():
128             if imported.has_key(c):
129                 # Available, already imported
130                 del imported[c]
131             # Available, not imported
132             self._not_requested.append(c)
133 
134         # The rest are channels that were once imported, but now are
135         # unavailable
136         self._end_of_service.extend(imported.keys())
137 
138         self._requested_channels.extend(self._requested_new)
139         self._requested_channels.extend(self._requested_imported)
140 
141         # Sort the lists
142         for name in self.__lists:
143             getattr(self, name).sort()
144         return self
=== </snip> ===

Best regards,
-Martin

Created attachment 483961 [details]
temporary patch, which fixes the problem, but can break some background intention, see comment #2

Taking.

(In reply to comment #0)
> 
> When syncing a custom channel from a master to a slave satellite with
> "satellite-sync -c some-channel", satellite sync does not remember that the
> channel has been synced previously (flag "p" in satellite-sync -l) 

The problem with this bug report / RFE is -- Satellite does not remember for *any* channel that it was synced previously, not even for Red Hat channels.

For Red Hat channels, the only way they could get to the Satellite is via satellite-sync. So for any Red Hat channel (== null organization) it is safe to assume that it was satellite-sync that brought it in.

For custom channels, Satellite initially did not have a way to sync them. Then we added the possibility of exporting and syncing them ... but the fact that there is a custom channel does not mean that it was satellite-sync'ed. In fact, you can easily create custom channel test-1 in Satellite you export from and put some packages into it, and export it into a dump (possibly together with some Red Hat channel). And you can create (via WebUI, not via satellite-sync) custom channel test-1 in the second Satellite where dumps are synced, and put some content to it.

Since Satellite does not keep track of how the channel came into existence, whether via satellite-sync or by being created manually, it has no way of knowing whether it should put the test-1 content from the dump to the test-1 channel on the second Satellite (which was *not* created by satellite-sync in the first place) or not.

In Spacewalk master, 2dfcfc43a7a38183a76d29985a501ec9f621f5aa, new option --include-custom-channels was added which will remove the "where org_id is
null" option when used, thus syncing all custom channel from the dump / ISS as well.

As noted in comment 11 thou, it will also sync custom channel that were *not* created via satellite-sync in the past.

Will this new option be a solution for issues raised by this bugzilla?

Seems to be a way to go. I tested it, works as expected. 

Can we please also have a short parameter "-i"? 

Thanks

(In reply to comment #14)
> Seems to be a way to go. I tested it, works as expected.

Thank you for the confirmation.

> Can we please also have a short parameter "-i"?

I prefer not to change the list of short options for minor release. Looking at the output of

  satellite-sync -h

we only have very few options with short parameter anyway.

Verified w/ spacewalk-backend-1.2.13-50.

1. I created a custom channel on satellite A, pushed 2 packages into and allowed iss mastering
2. I set up a iss slave on satellite B and synced the custom channel
3. I pushed another packages in custom channel on satellite A
4. Running 
# satellite-sync --include-custom-channels
on satellite B synced the custom channel content without specifying any custom channel label

spacewalk-backend-tools-1.2.13-52.el6sat.noarch

Stage validated with Satellite-5.4.1-RHEL6-re20110531.2

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

https://rhn.redhat.com/errata/RHEA-2011-0875.html