Created from redmine issue https://projects.theforeman.org/issues/28846

for docs, see https://community.theforeman.org/t/netbooting-ibm-power9-via-petitboot/17104

Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/28846 has been resolved.

FAILED QA
on sat6.8.0-9

I can't make this work.
I sysnced the PPCle kickstart repos and created a new host record for PPC arch. the orchestration seems to only deploy empty files:

[root@dhcp-2-25 boot]# pwd
/var/lib/tftpboot/boot
[root@dhcp-2-25 boot]# ll
total 63148
drwxr-xr-x. 2 foreman-proxy root               139 Jul 28 12:55 fdi-image
-rw-r--r--. 1 foreman-proxy foreman-proxy 57894232 Jul 28 07:38 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-8-1-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6762800 Jul 28 07:38 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-8-1-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy        0 Jul 29 10:28 red-hat-enterprise-linux-8-for-power-little-endian-baseos-beta-kickstart-4-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy        0 Jul 29 10:28 red-hat-enterprise-linux-8-for-power-little-endian-baseos-beta-kickstart-4-vmlinuz

I thought this may be the deferred-download thing so I resynced the repo with "immediate" download policy.

I don't have a ppc machine handy, but i used x86_64 machine just for the sake of picking these files up via pxe boot. Still no luck - the files stay empty.

@lzap, am i doing anything wrong or is this a valid bug?

Roman, can you please paste the proxy.log lines where those two files gets downloaded? I am interested in the URLs.

Can you try to reach those URLs? Are they 404? Can you tell if the files or paths are different for RHEL 8?

Some platforms have different paths and RHEL8 might changed it.

Hello, this turns out to be bug in pulp perhaps.

I can see that RHEL8 PPC64LE KS is synced properly however kickstart (PXE) files are missing: /content/beta/rhel8/8/ppc64le/baseos/kickstart/images is empty as well as /content/beta/rhel8/8/ppc64le/baseos/kickstart/images/ppc/ppc64.

For the record this is the treeinfo file:

[checksums]
images/boot.iso = sha256:a407c97fd193c66c921fd4be13b6a3f1f123c1e43e083579df5cbd90a28cb20b
images/install.img = sha256:f2ebf358b516d3e17b0f64a8884c85d3f048c7d9e7bb35dd591ac10953019e2e
ppc/ppc64/initrd.img = sha256:badab8b80aa1bdeadbc6c45401207a1923df6da780067f6e8cfedeeebad0e7d7
ppc/ppc64/vmlinuz = sha256:d597d603b240e3a4077ee77af358a1f1c6044c2302be82c5dd902eb5101004a3

[general]
; WARNING.0 = This section provides compatibility with pre-productmd treeinfos.
; WARNING.1 = Read productmd documentation for details about new format.
arch = ppc64le
family = Red Hat Enterprise Linux
name = Red Hat Enterprise Linux 8.3
packagedir = Packages
platforms = ppc64le
repository = .
timestamp = 1593612907
variant = BaseOS
variants = BaseOS
version = 8.3

[header]
type = productmd.treeinfo
version = 1.2

[images-ppc64le]
boot.iso = images/boot.iso
initrd = ppc/ppc64/initrd.img
kernel = ppc/ppc64/vmlinuz

[release]
name = Red Hat Enterprise Linux
short = RHEL
version = 8.3

[stage2]
mainimage = images/install.img

[tree]
arch = ppc64le
build_timestamp = 1593612907
platforms = ppc64le
variants = BaseOS

[variant-BaseOS]
id = BaseOS
name = BaseOS
packages = Packages
repository = .
type = variant
uid = BaseOS

Upstream bug assigned to lzap

I agree this sounds like a Pulp bug. Could we get an upstream Pulp issue filed with reproducer steps including a kickstart we could use for testing?

Before we do that, Roman can you do one thing? I remember that there was a problem with OnDemand sync, can you switch this to Immediate? Do we even have this option?

BZ does not let me to link the issue, how do I do it?

https://pulp.plan.io/issues/7303

The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

(In reply to Lukas Zapletal from comment #13)
> Before we do that, Roman can you do one thing? I remember that there was a
> problem with OnDemand sync, can you switch this to Immediate? Do we even
> have this option?

I already tried that with no luck

I am working with Centos8 kickstart currently, and here's what I see on Pulp2, 2-master for pulp and pulp_rpm:

  #!/bin/bash -v                                                                               
  BASE='centos8-baseos-ks'                                                                     
  REMOTE1='http://centos.mirror.rafal.ca/8.2.2004/BaseOS/ppc64le/kickstart/'                   
  pulp-admin rpm repo create --serve-http=true --repo-id=$BASE --relative-url=$BASE --feed=$REMOTE1 --download-policy immediate
  pulp-admin rpm repo sync run --repo-id=$BASE 
  +----------------------------------------------------------------------+
                Synchronizing Repository [centos8-baseos-ks]
  +----------------------------------------------------------------------+
  blahblahblah
  ... completed

  Task Succeeded

If I look at the published trees I see the kickstart files where I would expect them:

  [root@pulp2 yum]# tree | grep -v rpm
  .
  ├── http
  │   └── repos
  │       ├── centos8-baseos-ks -> /var/lib/pulp/published/yum/master/yum_distributor/centos8-baseos-ks/1598980046.23
  │       └── listing
  ├── https
  │   └── repos
  │       ├── centos8-baseos-ks -> /var/lib/pulp/published/yum/master/yum_distributor/centos8-baseos-ks/1598980046.23
  │       └── listing
  └── master
      └── yum_distributor
          └── centos8-baseos-ks
              └── 1598980046.23
                  ├── images
                  │   ├── boot.iso -> /var/lib/pulp/content/units/distribution/7b/27380410b6008e50444acb8fdd69e351b7cbd4e4f398fa410457f39157e0b9/images/boot.iso
                  │   └── install.img -> /var/lib/pulp/content/units/distribution/7b/27380410b6008e50444acb8fdd69e351b7cbd4e4f398fa410457f39157e0b9/images/install.img
                  ├── Packages
                  │   ├── a
                  │   ...
                  │   ├── z
                  ├── ppc
                  │   └── ppc64
                  │       ├── initrd.img -> /var/lib/pulp/content/units/distribution/7b/27380410b6008e50444acb8fdd69e351b7cbd4e4f398fa410457f39157e0b9/ppc/ppc64/initrd.img
                  │       └── vmlinuz -> /var/lib/pulp/content/units/distribution/7b/27380410b6008e50444acb8fdd69e351b7cbd4e4f398fa410457f39157e0b9/ppc/ppc64/vmlinuz
                  └── repodata
                      ├── 4bce5e49d718a0f08800330740a8b6ae60e72f494ee4a096793abbdefffc93aa-filelists.xml.gz
                      ├── 549bdf78ee50c7c8d91304bdde84ac8d9e76d0b83fc66353725d4aee2b673e09-comps.xml
                      ├── 7b91eeafeffd01bd9f7cad8a64a2f1609ecfed9c6697ec9113b8d84c2949485f-updateinfo.xml.gz
                      ├── a99296cc5691e272ddc74415f3341891330300d001f6bba57e1bedff23b695e9-primary.xml.gz
                      ├── d797275dc5994c926ade087a3d435c23d5eaf52e96924ab0b7d3e91c6347d907-other.xml.gz
                      └── repomd.xml

  1 directories, 2528 files
  [root@pulp2 yum]# 


And when I try to access them, they get delivered:

  [root@pulp2 yum]# wget --no-check-certificate https://pulp2.dev/pulp/repos/centos8-baseos-ks/images/install.img
  Saving to: ‘install.img’
  100%[================================================================>] 527,413,248  451MB/s   in 1.1s   
  2020-09-01 17:19:44 (451 MB/s) - ‘install.img’ saved [527413248/527413248]
  [root@pulp2 yum]# wget --no-check-certificate http://pulp2.dev/pulp/repos/centos8-baseos-ks/images/install.img
  Saving to: ‘install.img.1’
  100%[================================================================>] 527,413,248  768MB/s   in 0.7s   
  2020-09-01 17:19:48 (768 MB/s) - ‘install.img.1’ saved [527413248/527413248]
  [vagrant@pulp2 1794647]


I am going to reset and retry with a RHEL8 kickstart repo, assuming I can find/gain access to such a thing - but so far, this looks ok from Pulp2's side.

@lzap, @rplevka, am I missing something here?

Pulp2 directory structure looks fine (i.e., "matches what I expect from looking at the treeinfo file") after synching RHEL8-baseos-kickstart repo from https://cdn.redhat.com/content/dist/rhel8/8.2/ppc64le/baseos/kickstart/ as well:

  #!/bin/bash -v                                                                               
  BASE='rhel8-baseos-ks'                                                                       
  REMOTE1=https://cdn.redhat.com/content/dist/rhel8/8.2/ppc64le/baseos/kickstart/              
  pulp-admin rpm repo create --serve-http=true --repo-id=$BASE --relative-url=$BASE --feed=$REMOTE1 --download-policy immediate --feed-key ./cdn.pem --feed-cert ./cdn.pem --feed-ca-cert ./redhat-uep.pem
  pulp-admin rpm repo sync run --repo-id=$BASE

  [root@pulp2 yum]# pwd
  /var/www/pub/yum

  [root@pulp2 yum]# tree . | grep -v rpm
  .
  ├── http
  │   └── repos
  │       ├── centos8-baseos-ks -> /var/lib/pulp/published/yum/master/yum_distributor/centos8-baseos-ks/1598980046.23
  │       ├── listing
  │       └── rhel8-baseos-ks -> /var/lib/pulp/published/yum/master/yum_distributor/rhel8-baseos-ks/1598986776.13
  ├── https
  │   └── repos
  │       ├── centos8-baseos-ks -> /var/lib/pulp/published/yum/master/yum_distributor/centos8-baseos-ks/1598980046.23
  │       ├── listing
  │       └── rhel8-baseos-ks -> /var/lib/pulp/published/yum/master/yum_distributor/rhel8-baseos-ks/1598986776.13
  └── master
      └── yum_distributor
          ├── centos8-baseos-ks
          │   └── 1598980046.23
          │       ...
          └── rhel8-baseos-ks
              └── 1598986776.13
                  ├── images
                  │   ├── boot.iso -> /var/lib/pulp/content/units/distribution/51/695f7a47d2bb8f43cfee8eb12a711e735c5093c52b7260e105f764cbf13e67/images/boot.iso
                  │   └── install.img -> /var/lib/pulp/content/units/distribution/51/695f7a47d2bb8f43cfee8eb12a711e735c5093c52b7260e105f764cbf13e67/images/install.img
                  ├── install.img
                  ├── Packages
                  │   ├── a
                   ...
                  │   ├── z
                  ├── ppc
                  │   └── ppc64
                  │       ├── initrd.img -> /var/lib/pulp/content/units/distribution/51/695f7a47d2bb8f43cfee8eb12a711e735c5093c52b7260e105f764cbf13e67/ppc/ppc64/initrd.img
                  │       └── vmlinuz -> /var/lib/pulp/content/units/distribution/51/695f7a47d2bb8f43cfee8eb12a711e735c5093c52b7260e105f764cbf13e67/ppc/ppc64/vmlinuz
                  ├── repodata
                  │   ├── 3ea0cdd6c686aac3c2c3862b81ea7d9fa9a9d1e77a8c411ea4eb9f5a9dfc4991-filelists.xml.gz
                  │   ├── 4df08de286551a837c0da1088bda7f4c6c08bd7745ef2bf5173cbbf7a0a5f2fb-other.xml.gz
                  │   ├── 9b28aa4f2e4d9e018e13997182e1dd6a19f714f943605245cd45cd6e4c1eedc0-comps.xml
                  │   ├── 9cbea1d8ad5317e27521d78f729c68c1909b59f059b1b90f095eac83a657f895-productid.gz
                  │   ├── a41c1ebf9624fbd07f7d8e5b9ba91a3c60ae359d22de36db577333b8e4775abd-updateinfo.xml.gz
                  │   ├── cdf308b88cc57d936afcd37e6da6e70cbd97eef8c55d9628052c6507935f34c3-primary.xml.gz
                  │   └── repomd.xml
                  └── treeinfo -> /var/lib/pulp/content/units/distribution/51/695f7a47d2bb8f43cfee8eb12a711e735c5093c52b7260e105f764cbf13e67/treeinfo
  76 directories, 5071 files
  [root@pulp2 yum]# wget --no-check-certificate http://pulp2.dev/pulp/repos/rhel8-baseos-ks/images/install.img
  Length: 527704064 (503M) [application/octet-stream]
  Saving to: ‘install.img’
  100%[================================================================>] 527,704,064  839MB/s   in 0.6s   
  2020-09-01 19:14:55 (839 MB/s) - ‘install.img’ saved [527704064/527704064]
  [root@pulp2 yum]# wget --no-check-certificate https://pulp2.dev/pulp/repos/rhel8-baseos-ks/images/install.img
  Length: 527704064 (503M) [application/octet-stream]
  Saving to: ‘install.img.1’
  100%[================================================================>] 527,704,064  253MB/s   in 2.0s   
  2020-09-01 19:15:04 (253 MB/s) - ‘install.img.1’ saved [527704064/527704064]
  [root@pulp2 yum]#
 
Unless someone can show that Pulp is putting content in the wrong place, I'm going to say that whatever the problem is here, it isn't a Pulp2 problem.

The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

The entire point of the foreman change (which is what this bug is actually about) is that you can't look for pxeboot files, for ppc64le, under /pxeboot. See

  https://github.com/theforeman/foreman/pull/7381/files#diff-fd59a92ae681a1fd981d71c04679dd9dR28

from the PR that is attempting to address this bug.

Whatever is generating the link for "/content/beta/rhel8/8/ppc64le/baseos/kickstart//images/pxeboot/vmlinuz" is wrong - either the foreman PR we're linked to here isn't on the system-under-test or the fix is incorrect.

Pulp is placing the files where PPC64LE wants them to be. Pulp doesn't generate the kickstart profiles, which is where the error is happening. 

(One note - the ks-files are in ppc/ppc64, but the arch is actually called ppc64le - not sure if that might be why the code in the PR isn't working? because it's looking for arch/ppc64 instead of ppc64le?)

Yes, as you wrote the comment I was looking into this and made a patch and tested it on Roman's instance:

https://projects.theforeman.org/issues/30763

Apologies for the noise, this turned out to be an issue in Foreman core.

Foreman puts "architecture" there, whatever user defines. Roman, update your architecture to be ppc64le so the URL is correct. I have applied the patch on your instance so you can continue testing.

The Pulp upstream bug status is at CLOSED - NOTABUG. Updating the external tracker on this bug.

Returning this to Provisioning

Thanks for attaching the case, this will help PMs to identify which features to implement.

We do not currently support other platforms than Intel for provisioning, but some workflows are known to work. For example IBM POWER over BOOTP does work, we do not test it tho. There was a regression recently which was fixed in 6.8: https://projects.theforeman.org/issues/28846

I am going to close this bug, we do not test or support IBM POWER provisioning at the moment, however everything needed for modern IBM POWER (OpenPOWER) systems to boot should be ready in Satellite 6.8.