Fix restore --staged for certain vfs ops

[tyrielv]

When using virtualfilesystem, Running git restore --staged after git cherry-pick -n or git reset --soft would result in an incorrect state - the modified files would still have their modified contents, but git would no longer recognize them as being modified; and the added files would be deleted from disk instead of just unstaged.

See microsoft/VFSForGit#1855 for more details.

This commit fixes these issues with two changes:

Add a new flag to the index state structure, vfs_check_added_entries_for_clear_skip_worktree. This flag is set during cherry-pick -n before calculating the new index state, and causes newly added entries to have their SKIP_WORKTREE bit cleared. This ensures that newly added files are added to both the index and the worktree in the next step. Otherwise, sparse-checkout would prevent them from being added to the worktree.

Set the existing flag updated_skipworktree on the index when running a checkout index (aka restore --staged) operation. These operations already will clear SKIP_WORKTREE bits for modified files, but without this flag set the virtualfilesystem hook notification does not indicate that they changed, causing the VFS to not update its state correctly.

Thanks for taking the time to contribute to Git!

This fork contains changes specific to monorepo scenarios. If you are an
external contributor, then please detail your reason for submitting to
this fork:

• This is an early version of work already under review upstream.
• This change only applies to interactions with Azure DevOps and the
  GVFS Protocol.
• This change only applies to the virtualization hook and VFS for Git.

[dscho]

I have to admit that I'm certain that I fail to understand the deep detail here, why VFS has to be special-cased (and, say, Scalar-based checkouts are fine).

Do you think that there may be an easy way to test this in a new test case? That might jog my brain to understand what bug is fixed here.

[dscho]

How about moving the istate->vfs_... condition to the front, so that non-VFS scenarios are affected minimally?

[dscho]

Let's leave that empty line there.

[tyrielv]

I have to admit that I'm certain that I fail to understand the deep detail here, why VFS has to be special-cased (and, say, Scalar-based checkouts are fine).

Do you think that there may be an easy way to test this in a new test case? That might jog my brain to understand what bug is fixed here.

I've got a test case proposed for VFSForGit functional tests

I'll look into what happens with Scalar in this same case.

[tyrielv]

I have to admit that I'm certain that I fail to understand the deep detail here, why VFS has to be special-cased (and, say, Scalar-based checkouts are fine).
Do you think that there may be an easy way to test this in a new test case? That might jog my brain to understand what bug is fixed here.

I've got a test case proposed for VFSForGit functional tests

I'll look into what happens with Scalar in this same case.

@dscho You're right, this does not appear to be handled entirely properly in Scalar either, though it's not as bad.

For cherry-pick -n followed by restore --staged, when the file path is not in sparse checkout (Scalar) or hydrated (VFSForGit):

Scenario	Scalar Expected	Scalar Actual	VFSForGit Expected	VFSForGit Actual
File added	??? Either 1. unstaged, no longer shows in git status, not present on disk, or 2.unstaged, present on disk, warning about files modified outside sparse checkout definition	File remains added in index. Warning about sparse-checkout not matching. Can be cleaned up with git rm --sparse	File is unstaged but git status shows as an untracked file and it is projected in file system.	File remains staged and it is projected in file system
File modified	??? same as added	File is unstaged, no longer shows in git status, not on disk	File is unstaged, shows in git status as modified file, file system projects modified file	File is unstaged, does not show in git status, file system projects modified file. Because this is not detected in git status, all git operations (checkout/switch/reset/etc) behave as though the unmodified file were projected, even though the modified file is - including if you switch to a branch which has a a different version of the file: it remains projected with the modified version from the cherry-pick.
File deleted	File unstaged and no longer shows in git status	File unstaged and no longer shows in git status	File unstaged, shows as deleted in git status	File unstaged, does not show in git status

If I were to expand the proposed fix to not isolate to virtual file system it would make Scalar break expectations, placing files outside the sparse checkout definition and possibly turning off skip-worktree despite the definition.

I propose we set the expectation for Scalar on restore --staged of Added/Modified files outside the sparse checkout definition is to unstage and no longer show in git status. That matches the current behavior for modified and deleted files, so we just need to fix restore --staged of added files. That fix could probably go into upstream git.