Description of problem:

It would appear, though I have not traced every possible code path, that if the size of the stored metadata associated with an object grows to use multiple metadata keys, and then shrinks to use fewer keys, those higher numbered keys end up stranded, causing metadata read operations for all keys even though they are not used.

As of the latest code at the time of this writing, plugins/utils.py:write_metadata() pickles the metadata dictionary into a string, and then writes that string in key-value pairs as follows (depending on the length):

  user.swift.metadata: string[0-253]
  user.swift.metadata1: string[254-507]
  user.swift.metadata2: string[508-761]

So if the pickled size of the metadata is initially 400 bytes long, then it will be stored using two keys, user.swift.metadata and user.swift.metadata1. Should the pickled size of the metadata for that object shrink to a length less than 255 bytes, it will now be stored using only one key.

However, the second key, user.swift.metadata1, is not removed.

Currently, plugins/utils.py:read_metadata() is coded to keep looking for keys, incrementing the trailing number, appending the strings together to form the pickled string, doing so until it is unable to find a key. It then unpickles the resulting object.

The pickle code ignores any trailing data that is not part of the pickle, so the metadata is stored properly still.

With the second key now stranded, it will be continually added to the value of the first key before unpickling. Until that second key is removed, multiple system calls are then required to fetch all the metadata.


Version-Release number of selected component (if applicable):

 RHS 2.0

How reproducible:

 Every time

Steps to Reproduce:

See attached reproducer, test-metadata.sh, which creates three containers, where the first one has minimal metadata fitting in one xattr key/value pair, the second has big enough metadata requiring 3 xattr key/value pairs, and the third container adds and removes enough metadata so that it once required 3 xattr key/value pairs, but no longer does. As a result, looking at the xattrs on the file system shows the keys, but only one is required.
  
Actual results:

 Three keys created, two get stranded.

Expected results:

 One key exists.

Additional info:

This code was originally copied from the OpenStack Swift project, swift/obj/server.py, where metadata on a file is never rewritten, so they never encounter this problem. They create temporary '*.meta' data files that they rename to replace the existing one when they update data. We can encounter the problem because we use rewrite the metadata on the existing file.

Note that this code relies on pyxattr, which performs two getxattr system calls to read each key: one to find the size of the value stored for that key, the second to fetch the actual value once a suitably sized buffer to hold it has been allocated.

It should be noted that XFS has a 64 KB limit per key for xattr values.

Also note that Swift has a default HTTP REST protocol metadata (key lengths + values) limit of 4096. Note that the defaults for the individual length of metadata values is 256, with a limit of 90 such values, for a total of 23,040 bytes storeable by default. Since we use pickle to store these bytes, it adds some amount bytes of overhead.