Bug 1704348

+++ This bug was initially created as a clone of Bug #1685814 +++

How reproducible:
fence_gce --action=reboot --zone=<zone-name> --project=<Google Project> -n <Nodename> -v
or manually execute the python code from fence_gce 

Steps to Reproduce:
  1) Add the following to your /root/.bash_profile 
       export GOOGLE_APPLICATION_CREDENTIALS="/root/account.json"
  2) Close terminal and start a new session.
  3) Verify that variable exists
       # echo $GOOGLE_APPLICATION_CREDENTIALS
  4) Manually test fencing to verify that fencing works with no errors.
       # fence_gce --action=reboot --zone=<zone-details> --project=<Google Project> -n <Nodename> -v
  5) If fencing works with no errors, then verify that it works within the pacemaker
     by fencing a cluster node which should reboot the node. 
       # pcs stonith fence <node_name>
       
also here is the python snippet from fence_gce which can simulate the failure situation 

import googleapiclient.discovery
credentials = None
conn = googleapiclient.discovery.build('compute', 'v1', credentials=credentials)  ====> This should fail 
instance = conn.instances().get(project="<Enter Google project details>",zone="<Enter you zone details>",instance="<Enter node to fence>").execute()
print(json.dumps(instance.get('status'),indent=4, sort_keys=True))

Here is the python snippet from fence_gce which works 

import os
import googleapiclient.discovery
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "<Filename /location for the file for account.json>"
credentials = None
conn = googleapiclient.discovery.build('compute', 'v1', credentials=credentials)  
conn.instances().stop(project="<Enter Google project details>",zone="<Enter you zone details>",instance="<Enter node to fence>").execute()
instance = conn.instances().get(project="<Enter Google project details>",zone="<Enter you zone details>",instance="<Enter node to fence>").execute()
instance = conn.instances().get(project="<Enter Google project details>",zone="<Enter you zone details>",instance="<Enter node to fence>").execute()
print(json.dumps(instance.get('status'),indent=4, sort_keys=True)) ==>This should result in terminated for successfull fence 

Actual results:
Insufficient privilege, 403 error while accessing googleapis for node resets 

Expected results:
It should reset the node by successfully authenticating using the GOOGLE_APPLICATION_CREDENTIALS or get successful authorization against the Oauth , There is no good documentation on this 
or while configuring the stonith there is no way that we could mention from where to source the variable GOOGLE_APPLICATION_CREDENTIALS for authentication except setting them manually in bash profile or hardcoding the environment variable in fence_gce 

Additional info:
Reference internal redhat case 02326309

--- Additional comment from RHEL Product and Program Management on 2019-03-06 02:56:59 EST ---

Since this bug report was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release.

--- Additional comment from Shane Bradley on 2019-03-06 11:33:49 EST ---

The customer that opened bz has a ticket opened as well: 02326309

They stated they help anyway they can and I am sure would be open to using a test/devel version of fence_gce if we create a solution to this issue.  


Here is doc that describes that the env GOOGLE_APPLICATION_CREDENTIALS  should be declared for scripts. In some of links there is example on how to create creds object instead of letting it autocreate when that env GOOGLE_APPLICATION_CREDENTIALS  exists.
  - Setting Up Authentication for Server to Server Production Applications  |  Authentication  |  Google Cloud 
    https://cloud.google.com/docs/authentication/production

----

I believe we could improve fence_gce in a couple ways.
• Error checking if GOOGLE_APPLICATION_CREDENTIALS is not declared.
• Possible new attribute for GOOGLE_APPLICATION_CREDENTIALS on the fencing agent. 
  The API i believe would allow us to take a path to creds and not need GOOGLE_APPLICATION_CREDENTIALS 
  declared in our fencing script or in env. We could create
  a oauth2client.Credentials or google.auth.credentials.Credentials and pass to the
  build. I know i have done something in similar in script I wrote to 
  access the google gdrive.

--- Additional comment from Shane Bradley on 2019-04-15 10:15:44 EDT ---

Just to clear up what the issue is:

In nutshell by manually setting GOOGLE_APPLICATION_CREDENTIALS works for fence_gce (at command prompt or by including configure env in source path like /root/.bash_profile) but doesn't work while you initiate a stonith operation:
  # pcs stonith fence <node name> 

The reason is that when you fence_gce with stonith_admin or pcs the env variable is not sourced in. This cause the google api we use to error out. Currently the only way that the customer can get fence_gce to work is by hardcoding the fencing agent to source in the path (or declaring env in fence agent) as shown in comment #0.

--- Additional comment from Brandon Perkins on 2019-04-23 12:26:48 EDT ---

AFAICT it's working as designed.  They are attempting to use the fence agent and resource agent in a way that it was never designed to function.  As far as I know (with the exception of OCF variables), the fence and resource agents never use environment variables and all configuration is done within explicitly defined parameters in pacemaker.  I presume we could add this as an RFE, but I would want Oyvind (and potentially Chris Feist to see if we have any precedent for doing this in any other agent we ship.                                                                                                                                           

There is no setting of environment variables in the instructions in the documentation:

https://access.redhat.com/articles/3479821

Oyvind may need to correct me here, but my recollection is that the credentials for the fence agent are implicitly specified via the creation and configuration of the base GCP instance:

https://access.redhat.com/articles/3479821#create-and-configure-a-base-gcp-instance-7

and the service account is created in the "Before you start" section:

https://access.redhat.com/articles/3479821#header51

As for the resource agent, those credentials are specified when running "gcloud-ra init" as documented in:

https://access.redhat.com/articles/3479821#configure-gcp-node-authorization-14

--- Additional comment from Shane Bradley on 2019-04-23 13:57:56 EDT ---

I did find the following command, but do not believe it tells us if the user and project have the path to token configured correctly.
  # gcloud config configurations list


• Is there a way to test if you have it properly configured (like the path to token is setup correctly)? 
• Do we believe that the "gcloud-ra init" is what should make sure that google tools will find the token
  file (that is used for creds)?

--- Additional comment from Brandon Perkins on 2019-04-23 14:27:43 EDT ---

When speaking about the fence agent, the gcloud command is irrelevant as it is using implicit credentials from the service account dictated by the environment.  That is, there is no token or credentials file to speak of.

For the resource agent, we don't want customers using 'gcloud', only 'gcloud-ra' as it's the subset of commands which we support (i.e. credential and network setup only).  When 'gcloud-ra init' is run, it should create:

==========================================================================
.config/gcloud/.metricsUUID
.config/gcloud/gce
.config/gcloud/configurations/config_default
.config/gcloud/active_config
.config/gcloud/config_sentinel
.config/gcloud/credentials.db
.config/gcloud/access_tokens.db
==========================================================================

in the home directory.  And to verify that works, you could run something like:

==========================================================================
# gcloud-ra compute networks list   
NAME                          SUBNET_MODE  BGP_ROUTING_MODE  IPV4_RANGE  GATEWAY_IPV4
bperkinsrhel80-vpc-private    CUSTOM       REGIONAL
bperkinsrhel80-vpc-protected  CUSTOM       REGIONAL
bperkinsrhel80-vpc-public     CUSTOM       REGIONAL
default                       AUTO         REGIONAL
==========================================================================

--- Additional comment from Oyvind Albrigtsen on 2019-04-29 11:11:09 EDT ---

We should add a parameter for setting GOOGLE_APPLICATION_CREDENTIALS and possibly other profile info to the resource-agents and fence-agents.