renameDirectory fails once

[pjljvandelaar]

General summary/comments (optional)

On windows, renameDirectory fails when installing new ghc version (only once).

Steps to reproduce

1. Change your stack.yaml file to use a ghc configuration that has never been used before (e.g. move to the new lts-24.2).
2. Run command stack test on windows.

Expected

The following behaviour is expected

PS C:\TorXakis> stack test
Preparing to install GHC to an isolated location. This will not interfere with any system-level installation.
Already downloaded ghc-9.10.2.
Already downloaded 7z.dll.
Already downloaded 7z.exe.
Decompressing ghc-9.10.2.tar.xz...

7-Zip 22.01 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-07-15

Scanning the drive for archives:
1 file, 348217276 bytes (333 MiB)

Extracting archive: C:\Users\laarpjljvd\AppData\Local\Programs\stack\x86_64-windows\ghc-9.10.2.tar.xz
--
Path = C:\Users\laarpjljvd\AppData\Local\Programs\stack\x86_64-windows\ghc-9.10.2.tar.xz
Type = xz
Physical Size = 348217276
Method = LZMA2:26 CRC64
Streams = 1
Blocks = 18
Cluster Size = 201326592
Characteristics = BlockPackSize BlockUnpackSize

Everything is Ok

Size:       3446179840
Compressed: 348217276
Extracting ghc-9.10.2.tar...
Extracted total of 21 files from ghc-9.10.2.tar
GHC installed to C:\Users\laarpjljvd\AppData\Local\Programs\stack\x86_64-windows\ghc-9.10.2\.

Actual

This behaviour is observed

PS C:\TorXakis> stack test
Preparing to install GHC to an isolated location. This will not interfere with any system-level installation.
Downloaded ghc-9.10.2.
Already downloaded 7z.dll.
Already downloaded 7z.exe.
Decompressing ghc-9.10.2.tar.xz...

7-Zip 22.01 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-07-15

Scanning the drive for archives:
1 file, 348217276 bytes (333 MiB)

Extracting archive: C:\Users\laarpjljvd\AppData\Local\Programs\stack\x86_64-windows\ghc-9.10.2.tar.xz
--
Path = C:\Users\laarpjljvd\AppData\Local\Programs\stack\x86_64-windows\ghc-9.10.2.tar.xz
Type = xz
Physical Size = 348217276
Method = LZMA2:26 CRC64
Streams = 1
Blocks = 18
Cluster Size = 201326592
Characteristics = BlockPackSize BlockUnpackSize

Everything is Ok

Size:       3446179840
Compressed: 348217276
Extracting ghc-9.10.2.tar...
Extracted total of 21 files from ghc-9.10.2.tar
C:\Users\laarpjljvd\AppData\Local\Temp\stack-tmp-1f079d91757d6e41\ghc-9.10.2-x86_64-unknown-mingw32: renameDirectory:renamePath:MoveFileEx "\\\\?\\C:\\Users\\laarpjljvd\\AppData\\Local\\Temp\\stack-tmp-1f079d91757d6e41\\ghc-9.10.2-x86_64-unknown-mingw32" Just "\\\\?\\C:\\Users\\laarpjljvd\\AppData\\Local\\Programs\\stack\\x86_64-windows\\ghc-9.10.2": permission denied (Access is denied.)

Note by running PS C:\TorXakis> stack test again, I get the desired / expected behaviour.

Stack version

PS C:\TorXakis> stack --version
Version 3.7.1, Git revision cfae313a06ea9f7278e0ac55a58341fa0440f1aa x86_64 hpack-0.38.1

Method of installation

• Official binary, downloaded via haskellstack.org or from Stack's repository

Platform

Windows

[mpilgrem]

@pjljvandelaar, thanks for reporting. I have not been able to reproduce. Are you experiencing this repeatedly?

The relevant code is the final renameDir line at the end of Stack.Setup.withUnpackedTarball7z:

  withRunInIO $ \run ->
    withTempDir' tmpName $ \tmpDir ->
      run $ do
        liftIO $ ignoringAbsence (removeDirRecur destDir)
        run7z tmpDir archiveFile
        run7z tmpDir (tmpDir </> tarFile)
        absSrcDir <- expectSingleUnpackedDir archiveFile tmpDir
        -- On Windows, 'renameDir' does not work across drives. However, we have
        -- ensured that 'tmpDir' has the same drive as 'destDir'.
        renameDir absSrcDir destDir

That is Path.IO.rename from the path-io package. It documents:

On Win32 platforms, renameDir fails if the new directory already exists.

However, I don't think that is true. (I could not get it to fail for that reason on my Windows 11 system.)

I am wondering what could be preventing access to: \\\\?\\C:\\Users\\laarpjljvd\\AppData\\Local\\Programs\\stack\\x86_64-windows\\ghc-9.10.2. This maybe clutching at straws, but could something like the Haskell extension for VS Code be trying to install GHC 9.10.2 at the same time as stack test is trying to install it?

[pjljvandelaar]

It is a pity that you couldn't reproduce this issue.

I am experiencing it repeatedly. In fact, that was the trigger for me to report it...

The only hypothesis I have is that the new directory is accessed before its creation is completed.
This would explain why the second time everything goes fine.

[mpilgrem]

@pjljvandelaar, your hypothesis is similar to my own. I was trying to think of what else might be trying to access the directory at the same time as the Stack that you are using at the command line.

On Windows, I think the code behind renameDir is in package directory at System.Directory.Internal.Windows.renamePathInternal:

renamePathInternal :: OsPath -> OsPath -> IO ()
renamePathInternal opath npath =
  (`ioeSetOsPath` opath) `modifyIOError` do
    opath' <- furnishPath opath
    npath' <- furnishPath npath
    Win32.moveFileEx opath' (Just npath') Win32.mOVEFILE_REPLACE_EXISTING

-- package Win32
moveFileEx :: String -> Maybe String -> MoveFileFlag -> IO ()
moveFileEx src dest flags =
  withFilePath src $ \ c_src ->
  maybeWith withFilePath dest $ \ c_dest ->
  failIfFalseWithRetry_ (unwords ["MoveFileEx",show src,show dest]) $
    c_MoveFileEx c_src c_dest flags

foreign import WINDOWS_CCONV unsafe "windows.h MoveFileExW"
  c_MoveFileEx :: LPCTSTR -> LPCTSTR -> MoveFileFlag -> IO Bool

Which would be consistent with your error message:

C:\Users\laarpjljvd\AppData\Local\Temp\stack-tmp-1f079d91757d6e41\ghc-9.10.2-x86_64-unknown-mingw32:
  renameDirectory:
    renamePath:
      MoveFileEx 
        "\\\\?\\C:\\Users\\laarpjljvd\\AppData\\Local\\Temp\\stack-tmp-1f079d91757d6e41\\ghc-9.10.2-x86_64-unknown-mingw32"
        Just "\\\\?\\C:\\Users\\laarpjljvd\\AppData\\Local\\Programs\\stack\\x86_64-windows\\ghc-9.10.2":
          permission denied (Access is denied.)

I need to read up on that part of the Windows API:

• https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-movefileexw

[mpilgrem]

@pjljvandelaar, can we explore whether the problem is specific to that location. What happens if you (temporarily) specify a different Stack 'programs' directory?

local-programs-path: <Absolute_path_to_a_programs_directory>

[pjljvandelaar]

As suggested, I added local-programs-path: C:\temp\srp to my stack.yaml file.

Running stack test resulted in

Downloaded msys2-20240727.
Downloaded 7z.dll.
Downloaded 7z.exe.
Decompressing msys2-20240727.tar.xz...

7-Zip 22.01 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-07-15

Scanning the drive for archives:
1 file, 71943628 bytes (69 MiB)

Extracting archive: C:\temp\srp\x86_64-windows\msys2-20240727.tar.xz
--
Path = C:\temp\srp\x86_64-windows\msys2-20240727.tar.xz
Type = xz
Physical Size = 71943628
Method = LZMA2:24 CRC32
Streams = 1
Blocks = 5
Cluster Size = 67108864
Characteristics = BlockPackSize BlockUnpackSize

Everything is Ok

Size:       331865088
Compressed: 71943628
Extracting msys2-20240727.tar...
Extracted total of 21 files from msys2-20240727.tar
C:\Users\laarpjljvd\AppData\Local\Temp\stack-tmp-99dce20923e2ddf8\msys64: renameDirectory:renamePath:MoveFileEx "\\\\?\\C:\\Users\\laarpjljvd\\AppData\\Local\\Temp\\stack-tmp-99dce20923e2ddf8\\msys64" Just "\\\\?\\C:\\temp\\srp\\x86_64-windows\\msys2-20240727": permission denied (Access is denied.)

running stack test again resulted in

Already downloaded msys2-20240727.
Already downloaded 7z.dll.
Already downloaded 7z.exe.
Decompressing msys2-20240727.tar.xz...

7-Zip 22.01 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-07-15

Scanning the drive for archives:
1 file, 71943628 bytes (69 MiB)

Extracting archive: C:\temp\srp\x86_64-windows\msys2-20240727.tar.xz
--
Path = C:\temp\srp\x86_64-windows\msys2-20240727.tar.xz
Type = xz
Physical Size = 71943628
Method = LZMA2:24 CRC32
Streams = 1
Blocks = 5
Cluster Size = 67108864
Characteristics = BlockPackSize BlockUnpackSize

Everything is Ok

Size:       331865088
Compressed: 71943628
Extracting msys2-20240727.tar...
Extracted total of 21 files from msys2-20240727.tar
Preparing to install GHC to an isolated location. This will not interfere with any system-level installation.
Downloaded ghc-9.10.2.
Already downloaded 7z.dll.
Already downloaded 7z.exe.
Decompressing ghc-9.10.2.tar.xz...

7-Zip 22.01 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-07-15

Scanning the drive for archives:
1 file, 348217276 bytes (333 MiB)

Extracting archive: C:\temp\srp\x86_64-windows\ghc-9.10.2.tar.xz
--
Path = C:\temp\srp\x86_64-windows\ghc-9.10.2.tar.xz
Type = xz
Physical Size = 348217276
Method = LZMA2:26 CRC64
Streams = 1
Blocks = 18
Cluster Size = 201326592
Characteristics = BlockPackSize BlockUnpackSize

Everything is Ok

Size:       3446179840
Compressed: 348217276
Extracting ghc-9.10.2.tar...
Extracted total of 21 files from ghc-9.10.2.tar
GHC installed to C:\temp\srp\x86_64-windows\ghc-9.10.2\.
List                > configure
...

Summarizing, I still get the reported error (during the installation of msys2),
but I don't get the error during the ghc installation.
So the suggestion seems to have some effects.

[pjljvandelaar]

To determine that this issue is really dependent on the location,
I restored the stack.yaml file and
I removed the C:\Users\laarpjljvd\AppData\Local\Programs\stack directory.

Running stack test resulted in

Downloaded msys2-20240727.
Downloaded 7z.dll.
Downloaded 7z.exe.
Decompressing msys2-20240727.tar.xz...

7-Zip 22.01 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-07-15

Scanning the drive for archives:
1 file, 71943628 bytes (69 MiB)

Extracting archive: C:\Users\laarpjljvd\AppData\Local\Programs\stack\x86_64-windows\msys2-20240727.tar.xz
--
Path = C:\Users\laarpjljvd\AppData\Local\Programs\stack\x86_64-windows\msys2-20240727.tar.xz
Type = xz
Physical Size = 71943628
Method = LZMA2:24 CRC32
Streams = 1
Blocks = 5
Cluster Size = 67108864
Characteristics = BlockPackSize BlockUnpackSize

Everything is Ok

Size:       331865088
Compressed: 71943628
Extracting msys2-20240727.tar...
Extracted total of 21 files from msys2-20240727.tar
Preparing to install GHC to an isolated location. This will not interfere with any system-level installation.
Downloaded ghc-9.10.2.
Already downloaded 7z.dll.
Already downloaded 7z.exe.
Decompressing ghc-9.10.2.tar.xz...

7-Zip 22.01 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-07-15

Scanning the drive for archives:
1 file, 348217276 bytes (333 MiB)

Extracting archive: C:\Users\laarpjljvd\AppData\Local\Programs\stack\x86_64-windows\ghc-9.10.2.tar.xz
--
Path = C:\Users\laarpjljvd\AppData\Local\Programs\stack\x86_64-windows\ghc-9.10.2.tar.xz
Type = xz
Physical Size = 348217276
Method = LZMA2:26 CRC64
Streams = 1
Blocks = 18
Cluster Size = 201326592
Characteristics = BlockPackSize BlockUnpackSize

Everything is Ok

Size:       3446179840
Compressed: 348217276
Extracting ghc-9.10.2.tar...
Extracted total of 21 files from ghc-9.10.2.tar
GHC installed to C:\Users\laarpjljvd\AppData\Local\Programs\stack\x86_64-windows\ghc-9.10.2\.
...

So

1. The issue doesn't always occur (as both msys2 and ghc installation now went fine).
2. Location might not be relevant.
3. I was very unlucky the first time - both failed which triggered me to write this issue.