Bug 1794647 - [RFE] Support POWER kickstart tree
Summary: [RFE] Support POWER kickstart tree
Status: NEW
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Provisioning
Version: 6.4
Hardware: Unspecified
OS: Unspecified
low vote
Target Milestone: Unspecified
Assignee: Lukas Zapletal
QA Contact: Roman Plevka
Depends On:
TreeView+ depends on / blocked
Reported: 2020-01-24 08:49 UTC by Bryan Kearney
Modified: 2020-09-04 01:20 UTC (History)
11 users (show)

Fixed In Version: foreman-2.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed:
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Foreman Issue Tracker 28846 Normal Closed Support PPC64 PXE files location 2020-09-07 08:20:55 UTC
Foreman Issue Tracker 30763 Normal Ready For Testing Define architecture on abstract provider 2020-09-07 08:20:55 UTC
Pulp Redmine 7303 Normal CLOSED - NOTABUG Power kickstart does not sync PXE files 2020-09-02 13:05:09 UTC

Description Bryan Kearney 2020-01-24 08:49:47 UTC
It looks like PXE images for Red Hat systems are stored in a different subdirectory than on Intel. Foreman then tries to find them under wrong URL which leads to 404 when provisioning PPC64 machines.

Comment 1 Bryan Kearney 2020-01-24 08:49:52 UTC
Created from redmine issue https://projects.theforeman.org/issues/28846

Comment 5 Bryan Kearney 2020-02-10 15:02:26 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/28846 has been resolved.

Comment 6 Roman Plevka 2020-07-29 14:37:03 UTC
on sat6.8.0-9

I can't make this work.
I sysnced the PPCle kickstart repos and created a new host record for PPC arch. the orchestration seems to only deploy empty files:

[root@dhcp-2-25 boot]# pwd
[root@dhcp-2-25 boot]# ll
total 63148
drwxr-xr-x. 2 foreman-proxy root               139 Jul 28 12:55 fdi-image
-rw-r--r--. 1 foreman-proxy foreman-proxy 57894232 Jul 28 07:38 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-8-1-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6762800 Jul 28 07:38 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-8-1-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy        0 Jul 29 10:28 red-hat-enterprise-linux-8-for-power-little-endian-baseos-beta-kickstart-4-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy        0 Jul 29 10:28 red-hat-enterprise-linux-8-for-power-little-endian-baseos-beta-kickstart-4-vmlinuz

I thought this may be the deferred-download thing so I resynced the repo with "immediate" download policy.

I don't have a ppc machine handy, but i used x86_64 machine just for the sake of picking these files up via pxe boot. Still no luck - the files stay empty.

@lzap, am i doing anything wrong or is this a valid bug?

Comment 7 Lukas Zapletal 2020-08-06 11:05:59 UTC
Roman, can you please paste the proxy.log lines where those two files gets downloaded? I am interested in the URLs.

Can you try to reach those URLs? Are they 404? Can you tell if the files or paths are different for RHEL 8?

Some platforms have different paths and RHEL8 might changed it.

Comment 9 Lukas Zapletal 2020-08-07 13:10:11 UTC
Hello, this turns out to be bug in pulp perhaps.

I can see that RHEL8 PPC64LE KS is synced properly however kickstart (PXE) files are missing: /content/beta/rhel8/8/ppc64le/baseos/kickstart/images is empty as well as /content/beta/rhel8/8/ppc64le/baseos/kickstart/images/ppc/ppc64.

For the record this is the treeinfo file:

images/boot.iso = sha256:a407c97fd193c66c921fd4be13b6a3f1f123c1e43e083579df5cbd90a28cb20b
images/install.img = sha256:f2ebf358b516d3e17b0f64a8884c85d3f048c7d9e7bb35dd591ac10953019e2e
ppc/ppc64/initrd.img = sha256:badab8b80aa1bdeadbc6c45401207a1923df6da780067f6e8cfedeeebad0e7d7
ppc/ppc64/vmlinuz = sha256:d597d603b240e3a4077ee77af358a1f1c6044c2302be82c5dd902eb5101004a3

; WARNING.0 = This section provides compatibility with pre-productmd treeinfos.
; WARNING.1 = Read productmd documentation for details about new format.
arch = ppc64le
family = Red Hat Enterprise Linux
name = Red Hat Enterprise Linux 8.3
packagedir = Packages
platforms = ppc64le
repository = .
timestamp = 1593612907
variant = BaseOS
variants = BaseOS
version = 8.3

type = productmd.treeinfo
version = 1.2

boot.iso = images/boot.iso
initrd = ppc/ppc64/initrd.img
kernel = ppc/ppc64/vmlinuz

name = Red Hat Enterprise Linux
short = RHEL
version = 8.3

mainimage = images/install.img

arch = ppc64le
build_timestamp = 1593612907
platforms = ppc64le
variants = BaseOS

id = BaseOS
name = BaseOS
packages = Packages
repository = .
type = variant
uid = BaseOS

Comment 11 Bryan Kearney 2020-08-10 16:05:58 UTC
Upstream bug assigned to lzap@redhat.com

Comment 12 David Davis 2020-08-10 18:15:44 UTC
I agree this sounds like a Pulp bug. Could we get an upstream Pulp issue filed with reproducer steps including a kickstart we could use for testing?

Comment 13 Lukas Zapletal 2020-08-11 07:51:55 UTC
Before we do that, Roman can you do one thing? I remember that there was a problem with OnDemand sync, can you switch this to Immediate? Do we even have this option?

Comment 14 Lukas Zapletal 2020-08-11 07:53:52 UTC
BZ does not let me to link the issue, how do I do it?


Comment 15 pulp-infra@redhat.com 2020-08-11 12:06:09 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 16 pulp-infra@redhat.com 2020-08-11 12:06:12 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 17 Roman Plevka 2020-08-24 09:09:28 UTC
(In reply to Lukas Zapletal from comment #13)
> Before we do that, Roman can you do one thing? I remember that there was a
> problem with OnDemand sync, can you switch this to Immediate? Do we even
> have this option?

I already tried that with no luck

Comment 19 Grant Gainey 2020-09-01 17:23:16 UTC
I am working with Centos8 kickstart currently, and here's what I see on Pulp2, 2-master for pulp and pulp_rpm:

  #!/bin/bash -v                                                                               
  pulp-admin rpm repo create --serve-http=true --repo-id=$BASE --relative-url=$BASE --feed=$REMOTE1 --download-policy immediate
  pulp-admin rpm repo sync run --repo-id=$BASE 
                Synchronizing Repository [centos8-baseos-ks]
  ... completed

  Task Succeeded

If I look at the published trees I see the kickstart files where I would expect them:

  [root@pulp2 yum]# tree | grep -v rpm
  ├── http
  │   └── repos
  │       ├── centos8-baseos-ks -> /var/lib/pulp/published/yum/master/yum_distributor/centos8-baseos-ks/1598980046.23
  │       └── listing
  ├── https
  │   └── repos
  │       ├── centos8-baseos-ks -> /var/lib/pulp/published/yum/master/yum_distributor/centos8-baseos-ks/1598980046.23
  │       └── listing
  └── master
      └── yum_distributor
          └── centos8-baseos-ks
              └── 1598980046.23
                  ├── images
                  │   ├── boot.iso -> /var/lib/pulp/content/units/distribution/7b/27380410b6008e50444acb8fdd69e351b7cbd4e4f398fa410457f39157e0b9/images/boot.iso
                  │   └── install.img -> /var/lib/pulp/content/units/distribution/7b/27380410b6008e50444acb8fdd69e351b7cbd4e4f398fa410457f39157e0b9/images/install.img
                  ├── Packages
                  │   ├── a
                  │   ...
                  │   ├── z
                  ├── ppc
                  │   └── ppc64
                  │       ├── initrd.img -> /var/lib/pulp/content/units/distribution/7b/27380410b6008e50444acb8fdd69e351b7cbd4e4f398fa410457f39157e0b9/ppc/ppc64/initrd.img
                  │       └── vmlinuz -> /var/lib/pulp/content/units/distribution/7b/27380410b6008e50444acb8fdd69e351b7cbd4e4f398fa410457f39157e0b9/ppc/ppc64/vmlinuz
                  └── repodata
                      ├── 4bce5e49d718a0f08800330740a8b6ae60e72f494ee4a096793abbdefffc93aa-filelists.xml.gz
                      ├── 549bdf78ee50c7c8d91304bdde84ac8d9e76d0b83fc66353725d4aee2b673e09-comps.xml
                      ├── 7b91eeafeffd01bd9f7cad8a64a2f1609ecfed9c6697ec9113b8d84c2949485f-updateinfo.xml.gz
                      ├── a99296cc5691e272ddc74415f3341891330300d001f6bba57e1bedff23b695e9-primary.xml.gz
                      ├── d797275dc5994c926ade087a3d435c23d5eaf52e96924ab0b7d3e91c6347d907-other.xml.gz
                      └── repomd.xml

  1 directories, 2528 files
  [root@pulp2 yum]# 

And when I try to access them, they get delivered:

  [root@pulp2 yum]# wget --no-check-certificate https://pulp2.dev/pulp/repos/centos8-baseos-ks/images/install.img
  Saving to: ‘install.img’
  100%[================================================================>] 527,413,248  451MB/s   in 1.1s   
  2020-09-01 17:19:44 (451 MB/s) - ‘install.img’ saved [527413248/527413248]
  [root@pulp2 yum]# wget --no-check-certificate http://pulp2.dev/pulp/repos/centos8-baseos-ks/images/install.img
  Saving to: ‘install.img.1’
  100%[================================================================>] 527,413,248  768MB/s   in 0.7s   
  2020-09-01 17:19:48 (768 MB/s) - ‘install.img.1’ saved [527413248/527413248]
  [vagrant@pulp2 1794647]

I am going to reset and retry with a RHEL8 kickstart repo, assuming I can find/gain access to such a thing - but so far, this looks ok from Pulp2's side.

@lzap, @rplevka, am I missing something here?

Comment 20 Grant Gainey 2020-09-01 19:18:47 UTC
Pulp2 directory structure looks fine (i.e., "matches what I expect from looking at the treeinfo file") after synching RHEL8-baseos-kickstart repo from https://cdn.redhat.com/content/dist/rhel8/8.2/ppc64le/baseos/kickstart/ as well:

  #!/bin/bash -v                                                                               
  pulp-admin rpm repo create --serve-http=true --repo-id=$BASE --relative-url=$BASE --feed=$REMOTE1 --download-policy immediate --feed-key ./cdn.pem --feed-cert ./cdn.pem --feed-ca-cert ./redhat-uep.pem
  pulp-admin rpm repo sync run --repo-id=$BASE

  [root@pulp2 yum]# pwd

  [root@pulp2 yum]# tree . | grep -v rpm
  ├── http
  │   └── repos
  │       ├── centos8-baseos-ks -> /var/lib/pulp/published/yum/master/yum_distributor/centos8-baseos-ks/1598980046.23
  │       ├── listing
  │       └── rhel8-baseos-ks -> /var/lib/pulp/published/yum/master/yum_distributor/rhel8-baseos-ks/1598986776.13
  ├── https
  │   └── repos
  │       ├── centos8-baseos-ks -> /var/lib/pulp/published/yum/master/yum_distributor/centos8-baseos-ks/1598980046.23
  │       ├── listing
  │       └── rhel8-baseos-ks -> /var/lib/pulp/published/yum/master/yum_distributor/rhel8-baseos-ks/1598986776.13
  └── master
      └── yum_distributor
          ├── centos8-baseos-ks
          │   └── 1598980046.23
          │       ...
          └── rhel8-baseos-ks
              └── 1598986776.13
                  ├── images
                  │   ├── boot.iso -> /var/lib/pulp/content/units/distribution/51/695f7a47d2bb8f43cfee8eb12a711e735c5093c52b7260e105f764cbf13e67/images/boot.iso
                  │   └── install.img -> /var/lib/pulp/content/units/distribution/51/695f7a47d2bb8f43cfee8eb12a711e735c5093c52b7260e105f764cbf13e67/images/install.img
                  ├── install.img
                  ├── Packages
                  │   ├── a
                  │   ├── z
                  ├── ppc
                  │   └── ppc64
                  │       ├── initrd.img -> /var/lib/pulp/content/units/distribution/51/695f7a47d2bb8f43cfee8eb12a711e735c5093c52b7260e105f764cbf13e67/ppc/ppc64/initrd.img
                  │       └── vmlinuz -> /var/lib/pulp/content/units/distribution/51/695f7a47d2bb8f43cfee8eb12a711e735c5093c52b7260e105f764cbf13e67/ppc/ppc64/vmlinuz
                  ├── repodata
                  │   ├── 3ea0cdd6c686aac3c2c3862b81ea7d9fa9a9d1e77a8c411ea4eb9f5a9dfc4991-filelists.xml.gz
                  │   ├── 4df08de286551a837c0da1088bda7f4c6c08bd7745ef2bf5173cbbf7a0a5f2fb-other.xml.gz
                  │   ├── 9b28aa4f2e4d9e018e13997182e1dd6a19f714f943605245cd45cd6e4c1eedc0-comps.xml
                  │   ├── 9cbea1d8ad5317e27521d78f729c68c1909b59f059b1b90f095eac83a657f895-productid.gz
                  │   ├── a41c1ebf9624fbd07f7d8e5b9ba91a3c60ae359d22de36db577333b8e4775abd-updateinfo.xml.gz
                  │   ├── cdf308b88cc57d936afcd37e6da6e70cbd97eef8c55d9628052c6507935f34c3-primary.xml.gz
                  │   └── repomd.xml
                  └── treeinfo -> /var/lib/pulp/content/units/distribution/51/695f7a47d2bb8f43cfee8eb12a711e735c5093c52b7260e105f764cbf13e67/treeinfo
  76 directories, 5071 files
  [root@pulp2 yum]# wget --no-check-certificate http://pulp2.dev/pulp/repos/rhel8-baseos-ks/images/install.img
  Length: 527704064 (503M) [application/octet-stream]
  Saving to: ‘install.img’
  100%[================================================================>] 527,704,064  839MB/s   in 0.6s   
  2020-09-01 19:14:55 (839 MB/s) - ‘install.img’ saved [527704064/527704064]
  [root@pulp2 yum]# wget --no-check-certificate https://pulp2.dev/pulp/repos/rhel8-baseos-ks/images/install.img
  Length: 527704064 (503M) [application/octet-stream]
  Saving to: ‘install.img.1’
  100%[================================================================>] 527,704,064  253MB/s   in 2.0s   
  2020-09-01 19:15:04 (253 MB/s) - ‘install.img.1’ saved [527704064/527704064]
  [root@pulp2 yum]#
Unless someone can show that Pulp is putting content in the wrong place, I'm going to say that whatever the problem is here, it isn't a Pulp2 problem.

Comment 21 pulp-infra@redhat.com 2020-09-01 20:06:12 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 23 Grant Gainey 2020-09-02 12:19:47 UTC
The entire point of the foreman change (which is what this bug is actually about) is that you can't look for pxeboot files, for ppc64le, under /pxeboot. See


from the PR that is attempting to address this bug.

Whatever is generating the link for "/content/beta/rhel8/8/ppc64le/baseos/kickstart//images/pxeboot/vmlinuz" is wrong - either the foreman PR we're linked to here isn't on the system-under-test or the fix is incorrect.

Pulp is placing the files where PPC64LE wants them to be. Pulp doesn't generate the kickstart profiles, which is where the error is happening. 

(One note - the ks-files are in ppc/ppc64, but the arch is actually called ppc64le - not sure if that might be why the code in the PR isn't working? because it's looking for arch/ppc64 instead of ppc64le?)

Comment 24 Lukas Zapletal 2020-09-02 12:37:10 UTC
Yes, as you wrote the comment I was looking into this and made a patch and tested it on Roman's instance:


Apologies for the noise, this turned out to be an issue in Foreman core.

Foreman puts "architecture" there, whatever user defines. Roman, update your architecture to be ppc64le so the URL is correct. I have applied the patch on your instance so you can continue testing.

Comment 25 pulp-infra@redhat.com 2020-09-02 13:05:11 UTC
The Pulp upstream bug status is at CLOSED - NOTABUG. Updating the external tracker on this bug.

Comment 26 Grant Gainey 2020-09-02 13:10:09 UTC
Returning this to Provisioning

Note You need to log in before you can comment on or make changes to this bug.