Update Mountpoint's semantics to enable opening a new file handle on an inode if all the existing open handles have been flushed

[mansi153]

Update Mountpoint's semantics to enable opening a new file handle on an inode if all the existing open handles have been flushed.

This allows for the filesystem to not depend on a release request to complete pending (delayed) upload and cleanup the internal state of the inode within Mountpoint; alleviating situations where an open request made immediately after a close(flush) request leads to a race condition between the release following the close and the new open, which is sometimes an issue when the upload to S3 has been delayed until release.

Does this change impact existing behavior?

The semantics continue to allow only one writer OR one/many readers concurrently active for a file; however, now a new handle can be opened which can override the current active handles (despite not being officially released) if all the active handles are marked "flushed".
As part of opening the new handle, Mountpoint will also attempt uploading any pending data written for the previous file handle. All the requests to the overridden handle(s) will then start to fail or be no-op.
Multiple concurrent writers or concurrent readers and writers are still not allowed.

A handle is marked "flushed" when a close/flush is called on a file descriptor mapped to that handle. A following read/write request will revert that flushed state and signify that the handle is actively in use and can not be overridden. We maintain this information at the individual handle level and also in an inode-locked map of handles.

Breaking changes:

• Requests made to a duplicate file descriptor for a flushed file handle will start to fail (for e.g. a read/write would fail with EBADF: file handle has been invalidated by a newer handle opened) or be no-op (for e.g. flush, release) if a new open has overridden the flushed handle(s).
• A race condition can occur between a read/write request for a duplicate file descriptor on an existing (flushed) handle and multiple concurrent open requests, and any of them might succeed due to parallel processing of FUSE requests within Mountpoint. However, only one of them will ever succeed and there cannot be two concurrent writers or reader+writer for the inode at any point in time.
• An open request for an inode might fail if the pending upload to S3 fails. This is independent of whether the file has been truncated in the second open.

Does this change need a changelog entry? Does it require a version change?

Yes, it needs a changelog entry and update to the semantics.md. Yes it also requires a version change.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

[passaro]

Here are a few notes on the latest changes. We should add specific tests around the new lookup on open behavior (local vs remote).

[passaro]

Looks good. Only a couple of style suggestions.