Bug 1850629

Today my understanding is Premium disks are the only globally available storage solution available for installer. If the user in a region with different storage support we should provide documentation on how to upgrade these machines for IPI. Moving to machine-api for input on support.

# machine-api team
What stops users form rolling out new machines 1 by one with different storage? Is it supported? 

+ Abhinav(installer) for observability

>What stops users form rolling out new machines 1 by one with different storage? Is it supported? 

Nothing stops you from the machine API pov.
Doing it in a way that etcd quorum is maintained is the only constraint.
The only difference between machine "roles" is the software that runs on it.
May be this just needs to be documented in the same line of https://docs.openshift.com/container-platform/4.4/backup_and_restore/replacing-unhealthy-etcd-member.html#replacing-unhealthy-etcd-member.

Note solving this should not be tied to environments where the machine API is backing the instances but to any flavour of the product. This bz and doing a manual rollout of the control plane members will have similar implications for environments where the instances are not managed by machine API resources.

fwiw vertical rolling upgrades for the control plane are going towards a more automated solution with a machine API managed control plane via [1] and etcd operator supporting scaling down.

As for the installer it already supports user input for the DiskType [2] though it defaults to Premium_LRS [3].

[1] https://github.com/openshift/enhancements/pull/292
[2] https://github.com/openshift/installer/blob/8303d48c3d47df46a88ad32d922ea6fa5ec71a2c/pkg/types/azure/machinepool.go#L35
[3] https://github.com/openshift/installer/blob/f18b009b1295d13dde4d181235a188240e44940f/pkg/asset/machines/azure/machines.go#L93-L95

If the solution for https://bugzilla.redhat.com/show_bug.cgi?id=1802768 was concluded as using a different diskType I think I'll move this ticket to docs to better document the procedure to bumping the disk type for the control plane. I'll leave some time for everyone to chime in before doing so.

(In reply to Alberto from comment #2)
> >What stops users form rolling out new machines 1 by one with different storage? Is it supported? 
> 
> Nothing stops you from the machine API pov.
> Doing it in a way that etcd quorum is maintained is the only constraint.
> The only difference between machine "roles" is the software that runs on it.
> May be this just needs to be documented in the same line of
> https://docs.openshift.com/container-platform/4.4/backup_and_restore/
> replacing-unhealthy-etcd-member.html#replacing-unhealthy-etcd-member.
> 
> Note solving this should not be tied to environments where the machine API
> is backing the instances but to any flavour of the product. This bz and
> doing a manual rollout of the control plane members will have similar
> implications for environments where the instances are not managed by machine
> API resources.
> 
> fwiw vertical rolling upgrades for the control plane are going towards a
> more automated solution with a machine API managed control plane via [1] and
> etcd operator supporting scaling down.
> 
> As for the installer it already supports user input for the DiskType [2]
> though it defaults to Premium_LRS [3].
> 
> [1] https://github.com/openshift/enhancements/pull/292
> [2]
> https://github.com/openshift/installer/blob/
> 8303d48c3d47df46a88ad32d922ea6fa5ec71a2c/pkg/types/azure/machinepool.go#L35
> [3]
> https://github.com/openshift/installer/blob/
> f18b009b1295d13dde4d181235a188240e44940f/pkg/asset/machines/azure/machines.
> go#L93-L95
> 
> If the solution for https://bugzilla.redhat.com/show_bug.cgi?id=1802768 was
> concluded as using a different diskType I think I'll move this ticket to
> docs to better document the procedure to bumping the disk type for the
> control plane. I'll leave some time for everyone to chime in before doing so.


Actually, This is about the Raft runtime changes which landed in 4.4, because etcd was performing very bad on Azure and for which the following bugzilla was created -
https://bugzilla.redhat.com/show_bug.cgi?id=1806700

My cusutomer upgraded to 4.4 and still with the changed Raft Runtime, they are still facing the issue. Changing the disktype is not a solution here and hence I think this should not be moved as a DOc Bug.

>My cusutomer upgraded to 4.4 and still with the changed Raft Runtime, they are still facing the issue. Changing the disktype is not a solution here and hence I think this should not be moved as a DOc Bug.

Can you share must gather logs that back this statement to help etcd team with farther triaging. I'm moving back to etcd team to re-evaluate based on this feedback.

This might be a dup of https://bugzilla.redhat.com/show_bug.cgi?id=1850057

(In reply to Alberto from comment #4)
> >My cusutomer upgraded to 4.4 and still with the changed Raft Runtime, they are still facing the issue. Changing the disktype is not a solution here and hence I think this should not be moved as a DOc Bug.
> 
> Can you share must gather logs that back this statement to help etcd team
> with farther triaging. I'm moving back to etcd team to re-evaluate based on
> this feedback.
> 
> This might be a dup of https://bugzilla.redhat.com/show_bug.cgi?id=1850057

The customer has provided the sos-report, please check on the supportshell -
$ ssh <kerberosid>@supportshell.cee.redhat.com
$ yank -y 02640065
$ ls ~/02640065/0180-must-gather.local.212582227496092128.zip/

Do let me know if anything else is required.