Bug 2065727

Created attachment 1866623 [details]
metal3-ironic-conductor.log

Created attachment 1866623 [details]
metal3-ironic-conductor.log

Description of problem:

I'm having problems when scaling down a node in an hypershift cluster, instead of booting again assisted-installer's discovery ISO, the BMH is being shutdown and set into maintenance mode

Long explanation:
I set up a HUB cluster using dev-scripts, and installed there assisted-service, hive and hypershift
I create an infraenv, and create BMH with infraenvs.agent-install.openshift.io: myinfraenv label
Then I create an agent hypershift cluster, and scale it up
The problem appears when I try to scale it down
The expected behaviour would be that a worker node is removed from the cluster, and then BMH is reprovisioned using assisted's discovery ISO
What actually happens is that the node is shutdown


How reproducible:
Always

Steps to Reproduce:
Start a HUB cluster using dev-scripts. This is my config_$USER.sh file:

--- config_$USER.sh ---
export WORKING_DIR=/home/dev-scripts
#Go to https://console-openshift-console.apps.ci.l2s4.p1.openshiftapps.com/, click on your name in the top right, and use the token from the login command
export CI_TOKEN='mysecrettoken'

export CLUSTER_NAME="hub"
export BASE_DOMAIN="redhat.com"

export OPENSHIFT_VERSION=4.9.17
export OPENSHIFT_RELEASE_TYPE=ga
export IP_STACK=v4
export BMC_DRIVER=redfish-virtualmedia
export REDFISH_EMULATOR_IGNORE_BOOT_DEVICE=True

export NUM_WORKERS=0
export MASTER_VCPU=8
export MASTER_MEMORY=20000
export MASTER_DISK=60

export NUM_EXTRA_WORKERS=2
export WORKER_VCPU=8
export WORKER_MEMORY=35000
export WORKER_DISK=150

export VM_EXTRADISKS=true
export VM_EXTRADISKS_LIST="vda vdb"
export VM_EXTRADISKS_SIZE=30G
--- END FILE ---

Steps to reproduce:

NAMESPACE=mynamespace
CLUSTERNAME=mycluster
INFRAENV=myinfraenv
BASEDOMAIN=redhat.com
SSHKEY=~/.ssh/id_rsa.pub
HYPERSHIFT_IMAGE=quay.io/hypershift/hypershift-operator:latest

make all assisted
cp ocp/hub/auth/kubeconfig ~/.kube/config
oc patch storageclass assisted-service -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'                                                              
oc patch provisioning provisioning-configuration --type merge -p '{"spec":{"watchAllNamespaces": true}}'                                                                                      

alias hypershift="podman run --net host --rm --entrypoint /usr/bin/hypershift -e KUBECONFIG=/working_dir/ocp/hub/auth/kubeconfig -v $HOME/.ssh:/root/.ssh -v $(pwd):/working_dir $HYPERSHIFT_IMAGE"
hypershift install --hypershift-image $HYPERSHIFT_IMAGE

oc create namespace $NAMESPACE

# Create pull secret
cat << EOF | oc apply -n $NAMESPACE -f -
apiVersion: v1
kind: Secret
type: kubernetes.io/dockerconfigjson
metadata:
  name: my-pull-secret
data:
  .dockerconfigjson: $(cat pull_secret.json | base64 -w0)
EOF

# Create infraenv
cat << EOF | oc apply -n $NAMESPACE -f -
apiVersion: agent-install.openshift.io/v1beta1
kind: InfraEnv
metadata:
  name: $INFRAENV
spec:
  pullSecretRef:
    name: my-pull-secret
  sshAuthorizedKey: $(cat $SSHKEY)
EOF

# Create BMH
cat  ocp/hub/saved-assets/assisted-installer-manifests/06-extra-host-manifests.yaml | sed "s/assisted-installer/$NAMESPACE/g; s/myinfraenv/$INFRAENV/g" | oc apply -f -

# Create hypershift cluster
INFRAID=$(oc get infraenv $INFRAENV -n $NAMESPACE -o yaml| yq -r .status.isoDownloadURL | awk -F/ '{print $NF}'| cut -d \? -f 1)
hypershift create cluster agent --name $CLUSTERNAME --base-domain $BASEDOMAIN --pull-secret /working_dir/pull_secret.json  --ssh-key $SSHKEY --agent-namespace $NAMESPACE --namespace $NAMESPACE --infra-id=$INFRAID --control-plane-operator-image=$HYPERSHIFT_IMAGE
KUBECONFIG=ocp/hub/auth/kubeconfig:<(hypershift create kubeconfig) kubectl config view --flatten > ~/.kube/config

# Scale up
oc scale nodepool/$CLUSTERNAME -n $NAMESPACE --replicas=2

# Force EFI next-boot to be CD
ssh core@extraworker-0 sudo efibootmgr -n1
ssh core@extraworker-1 sudo efibootmgr -n1


# Scale down
oc scale nodepool/$CLUSTERNAME -n $NAMESPACE --replicas=1


Actual results:
Scaled down node is shutdown, and in ironic it appears as in maintenance mode

Expected results:
Scaled down node is rebooted into assisted service's discovery ISO

Additional info:

Related slack threads
https://coreos.slack.com/archives/C02N0RJR170/p1647513139893809
https://coreos.slack.com/archives/CFP6ST0A3/p1647449263684629

I also attach logs from metal3-ironic-conductor, metal3-baremetal-operator, cluster-api-agent-provider and assisted-service

I'll also attach an affected BareMetalHost CR