Bug 1582186

Let's use the contents of /etc/machine-id as an ID in the nfs module parameter an /etc/modprobe.d/nfs.conf file.  The nfs.config service can check to see if modprobe.d/nfs.conf exists with that parameter, and add it if it is not there.  We'll need a way to detect if an admin has already set that parameter as well, perhaps by prepending the value with something static, or marking the config option with a comment.

See the man page MACHINE-ID(5) for information on the contents of /etc/machine-id, it seems to be exactly we want.

Looking in the nfs client code, 

'migration' mount option:
    'nfs4_unique_id' is set:
        string is: Linux NFSv<MAJOR_VERSION>.<MINOR_VERSION> <UNIQUE_STRING>/<CLIENT_HOSTNAME>
    'nfs4_unique_id' is not set:
        string is: Linux NFSv<MAJOR_VERSION>.<MINOR_VERSION> <CLIENT_HOSTNAME>/<SERVER_IP>
'nomigration' mount option:
    'nfs4_unique_id' is set:
        string is: Linux NFSv4.0 <CLIENT_HOSTNAME>/<UNIQUE_STRING>/<SERVER_IP>
    'nfs4_unique_id' is not set:
        string is: Linux NFSv4.0 <CLIENT_HOSTNAME>

So the <CLIENT_HOSTNAME> gets used in all cases, and would be pointless in the nfs4_unique_id as a result.  <SERVER_IP> is also already included in the 'nomigration + nfs4_unique_id' case, an would obviously be pointless anyway, since the problem is already with the client string not being unique when viewed by the server (so would not help in distinguishing between clients).

The /etc/machine-id sounds like a good option, however the problem is typically seen between nfs clients that have been cloned from the same master, and the /etc/machine-id is not unique (obviously it should be, but in practice isn't).



What about something like a short nfs-utils program to generate (and set?) the nfs4_unique_id based on a template in /etc/nfs.conf?  something like:

[general]
# current default - empty
# nfs4-unique-id=

# nfs4-unique-id=some static string

# from /etc/machine-id
# nfs4-unique-id=%{machine_id}

# generated completely randomly from /proc/sys/kernel/random/uuid
# nfs4-unique-id=%{random_uuid}

# use a script/program to 
# nfs4-unique-id=%{exec=/path/to/script}

# use current time
# nfs4-unique-id=%{epoch}


(or some combination of the above)

(In reply to Frank Sorenson from comment #7)
> The /etc/machine-id sounds like a good option, however the problem is
> typically seen between nfs clients that have been cloned from the same
> master, and the /etc/machine-id is not unique (obviously it should be, but
> in practice isn't).
> 
> 
> 
> What about something like a short nfs-utils program to generate (and set?)
> the nfs4_unique_id based on a template in /etc/nfs.conf?  something like:

How does it persist across boots?  If we're storing the result in the filesystem,
what happens if someone then clones that filesystem?

Surely we must not be the only subsystem with this problem.  I don't see how we can get around fixing /etc/machine-id.

(In reply to J. Bruce Fields from comment #8)
> (In reply to Frank Sorenson from comment #7)

> > What about something like a short nfs-utils program to generate (and set?)
> > the nfs4_unique_id based on a template in /etc/nfs.conf?  something like:
> 
> How does it persist across boots?  If we're storing the result in the
> filesystem,
> what happens if someone then clones that filesystem?

Yes, if the template for nfs4_unique_id only includes elements read directly from the filesystem, then they'll experience the same issue; hence the suggestions that the template could use the epoch seconds or a completely unique value instead, or as well.


> Surely we must not be the only subsystem with this problem.

No, we're definitely not.  A quick search of bugzilla shows a number of places where either machine-id is not becoming uniquely set, or where that causes a problem as a result.


> I don't see how
> we can get around fixing /etc/machine-id.

Sure, but currently there's nothing that sets /sys/module/nfs/parameters/nfs4_unique_id to machine-id, or to anything else.  If we add something to set nfs4_unique_id to machine-id (which I agree is a reasonable solution), is there a case where the customer might want an alternate string?  How do we detect that condition?  The drawback to setting nfs4_unique_id to machine-id (unless it's already populated) is that we then have nfs-related configuration items that are configured outside of nfs-utils.

The template suggestion was intended to make that configuration easier, and all from within nfs-utils.  But I'm not stuck on the idea.  Just a thought on how we could make setting nfs4_unique_id easily configurable.

(In reply to Frank Sorenson from comment #9)
> Sure, but currently there's nothing that sets
> /sys/module/nfs/parameters/nfs4_unique_id to machine-id, or to anything
> else.  If we add something to set nfs4_unique_id to machine-id (which I
> agree is a reasonable solution), is there a case where the customer might
> want an alternate string?

OK, got it, so we could use your proposal to default to machine-id while still allowing for some exceptions, that makes sense.

Though I worry that the typical user experience won't be great: clone a bunch of VMs; find they don't work; realize its because NFS mounts are erroring out; stare at network traces for a day, finally figure out the right google keywords to realize it's something to do with nfs4_unique_id and find documentation pointing to /etc/nfs.conf.  A single clone app that everyone knows to use and can fix up stuff like machine-id would be nice.  But I assume people have already tried that.

Maybe I'm too worried about persistence.  What actually happens if every NFS client just generates a new random ID every time it boots?

I think the only effect for now is that everybody has to wait a lease period for the rebooted client's locks to get revoked.  Whereas if it boots back up with the same ID in less than a lease period, it can tell the server to release its old state early.  Maybe that's not a huge deal.

There's also client delegation reclaim--in theory a client that keeps cached data on disk can try to reclaim its old delegations on reboot to avoid throwing out its cache.  We don't implement that, but maybe we will some day.

Hi,

Will the discussed solution be included into the next RHEL 7.7?

Regards,
Ikarashi

Customers running upgrading from earlier rhel7 setups to rhel7.6 are running into this, marking as regression.

Looks reasonable to me.  So, "nfs4_unique_id = ${machine_id}" would be the default on a new install?

If this can't make into 7.7, we have to defer it to RHEL8.

Thanks.

Yes, I think we'd want to default to "nfs4_unique_id = ${machine_id}", rather than the empty string, as with current behavior.


At this point, moving this to RHEL 8 is probably appropriate.

Created attachment 1588467 [details]
potential nfs4_unique_id implementation V1

this patch adds the nfs4_unique_id program, which can be used to parse and potentially expand the [general]/unique_id setting in the nfs.conf (or as given on the command line).  The program will then set the nfs4 module's nfs4_unique_id parameter to the expanded string.


usage: nfs4_unique_id [options]
      -h, -?, --help      Print this help screen.
      -s                  Set the nfs4_unique_id string.
      -u <unique_id_string>, --unique_id=<unique_id_string>
                          String to parse.
      -n, --null          Null out the system's nfs4_unique_id string.
      -x, --hex           Dump the expanded nfs4_unique_id string in hex.
      -v, --verbose       Display verbose information about the expansion.
      -V, --version       Display version information and exit.


expansion rules and keywords:
        input string is null-terminated
        expanded string is not terminated (0x00 is a valid nfs4_unique_id character)

        input string:
                empty string - empty string is returned

                string beginning with $ - replaced by the value of an environment variable

                regular characters - copied to epanded as-is
                        (including spaces but not a trailing \n, unless escaped or quoted)
                %{keyword} - expanded/replaced as defined
                        %{machine_id} - replaced by hash of contents of /etc/machine-id
                        %{net_ns} - replaced by the network namespace pointed to by /proc/self/ns/net
                        %{file=/path/to/file} - replaced by contents of the given file
                        %{exec=/path/to/executable} - execute the given program/script, and use the output
                        %{env=ENVIRONMENT_VAR} - replaced by the value of an environment variable

                        %{epoch} - replaced by the epoch time in seconds
                        %{epoch_ns} - replaced by the epoch time in nanoseconds
                        %{random_uuid} - replaced by uuid generated by /proc/sys/kernel/random/uuid

                        %{hostname} - expands to hostname; does not add uniqueness

                        broken %{... without terminating } - copy literal
                        unknown %{keyword} - copy literal

                % (without the next character being {) - literal %
                \% - literal %
                \" - literal "
                \\ - literal \
                \{ - literal {
                \x## - replace this hex sequence with the 8-bit character value

Created attachment 1588469 [details]
program to test expansion of unique_id strings

this patch adds a test_expansions program to test the various expansions of the nfs4_unique_id program

I'm assuming that we think this is a reasonable target for 7.8 at this point in time? So I've set devel ack. Alice, feel free to assign this bug to yourself if this is the one that you mentioned that you were working on the other day.

Frank has done all the work, I'm just chipping in with some suggestions wrt the nfs.conf parts and springfield integration

Alice suggested extending conffile/nfsconf settings with this expansion capability, rather than building it directly into a tool to set just the nfs4_unique_id, which makes complete sense, and which I had totally missed.

I'm attaching new patches which do just this.  The first adds the expansion capabilities to the configuration settings, and the second adds the nfs4_unique_id-specific pieces.

Created attachment 1591153 [details]
conffile expansions patch v2

Created attachment 1591155 [details]
nfs4_unique_id patch v2

add the nfs4_unique_id utility

Hey Frank - are you wanting review or should you just submit your patches to linux-nfs?

CLOSING WONTFIX for RHEL7 and will open clone to RHEL8

RHEL8 bug https://bugzilla.redhat.com/show_bug.cgi?id=1801326