From 0c8424c259b417c6aadc23f5398e55edd7b047a2 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Thu, 2 Apr 2026 08:51:18 +0200 Subject: [PATCH 1/5] t: work around multibyte bug in quoted heredocs with Dash v0.5.13 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When executing our test suite with Dash v0.5.13.2 one can observe several test failures that all have the same symptoms: we have a quoted heredoc that contains multibyte characters, but the final data does not match what we actually wanted to write. One such example is in t0300, where we see the diffs like the following: --- expect-stdout 2026-04-01 07:25:45.249919440 +0000 +++ stdout 2026-04-01 07:25:45.254919509 +0000 @@ -1,5 +1,5 @@ protocol=https host=example.com -path=perú.git +path=perú.git username=foo password=bar While seemingly the same, the data that we've written via the heredoc contains some invisible bytes. The expected hex representation of the string is: 7065 72c3 ba2e 6769 74 per...git But what we actually get instead is this string: 7065 7285 02c3 ba02 852e 6769 74 per.......git What's important to note here is that the multibyte character exists in both versions. But in the broken version we see that the bytes are wrapped in a sequence of "85 02" and "02 85". This is the CTLMBCHAR byte sequence of Dash, which it uses internally to quote multibyte sequences. As it turns out, this bug was introduced in c5bf970 (expand: Add multi-byte support to pmatch, 2024-06-02), which adds multibyte support to more contexts of Dash. One of these contexts seems to be in heredocs, and Dash _does_ correctly unquote these multibyte sequences when using an unquoted heredoc. But the bug seems to be that this unquoting does not happen in quoted heredocs, and the bug still exists on the latest "master" branch. For now, work around the bug by using unquoted heredocs instead. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- t/t0300-credentials.sh | 4 +++- t/t3430-rebase-merges.sh | 6 ++++-- t/t3902-quoted.sh | 16 +++++++++------- t/t4014-format-patch.sh | 16 ++++++++++++---- t/t4201-shortlog.sh | 4 +++- t/t9001-send-email.sh | 12 +++++++++--- 6 files changed, 40 insertions(+), 18 deletions(-) diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh index 07aa834d33e248..64ead1571ae1e1 100755 --- a/t/t0300-credentials.sh +++ b/t/t0300-credentials.sh @@ -675,7 +675,9 @@ test_expect_success 'match percent-encoded values' ' test_expect_success 'match percent-encoded UTF-8 values in path' ' test_config credential.https://example.com.useHttpPath true && test_config credential.https://example.com/perú.git.helper "$HELPER" && - check fill <<-\EOF + # NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs + # that contain multibyte chars. + check fill <<-EOF url=https://example.com/per%C3%BA.git -- protocol=https diff --git a/t/t3430-rebase-merges.sh b/t/t3430-rebase-merges.sh index cc627e34a741e6..84b2d0e664e58f 100755 --- a/t/t3430-rebase-merges.sh +++ b/t/t3430-rebase-merges.sh @@ -507,9 +507,11 @@ test_expect_success 'octopus merges' ' git rebase -i --force-rebase -r HEAD^^ && test "Hank" = "$(git show -s --format=%an HEAD)" && test "$before" != $(git rev-parse HEAD) && - test_cmp_graph HEAD^^.. <<-\EOF + # NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs + # that contain multibyte chars. + test_cmp_graph HEAD^^.. <<-EOF *-. Tüntenfüsch - |\ \ + |\\ \\ | | * three | * | two | |/ diff --git a/t/t3902-quoted.sh b/t/t3902-quoted.sh index f528008c363c68..8660ec5cb0acdb 100755 --- a/t/t3902-quoted.sh +++ b/t/t3902-quoted.sh @@ -60,16 +60,18 @@ With SP in it "\346\277\261\351\207\216\347\264\224" EOF -cat >expect.raw <<\EOF +# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs +# that contain multibyte chars. +cat >expect.raw <expect <<'EOF' +# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs +# that contain multibyte chars. +cat >expect < @@ -1300,7 +1302,9 @@ test_expect_success 'format-patch wraps extremely long from-header (non-ASCII wi test_cmp expect actual ' -cat >expect <<'EOF' +# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs +# that contain multibyte chars. +cat >expect <expect <<'EOF' +# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs +# that contain multibyte chars. +cat >expect <patch && - cat >expect <<-\EOF && + # NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs + # that contain multibyte chars. + cat >expect <<-EOF && From: C O Mitter Content-Type: text/plain; charset=UTF-8 diff --git a/t/t4201-shortlog.sh b/t/t4201-shortlog.sh index 5f23fc147bb88d..9f41d56d9a556c 100755 --- a/t/t4201-shortlog.sh +++ b/t/t4201-shortlog.sh @@ -105,7 +105,9 @@ test_expect_success 'output from user-defined format is re-wrapped' ' ' test_expect_success !MINGW,ICONV 'shortlog wrapping' ' - cat >expect <<\EOF && + # NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs + # that contain multibyte chars. + cat >expect <email-using-8bit <<\EOF +# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs +# that contain multibyte chars. +cat >email-using-8bit < From: author@example.com @@ -1735,7 +1737,9 @@ test_expect_success $PREREQ '--8bit-encoding overrides sendemail.8bitEncoding' ' ' test_expect_success $PREREQ 'setup expect' ' - cat >email-using-8bit <<-\EOF + # NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs + # that contain multibyte chars. + cat >email-using-8bit <<-EOF From fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001 Message-ID: From: author@example.com @@ -1764,7 +1768,9 @@ test_expect_success $PREREQ '--8bit-encoding also treats subject' ' ' test_expect_success $PREREQ 'setup expect' ' - cat >email-using-8bit <<-\EOF + # NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs + # that contain multibyte chars. + cat >email-using-8bit <<-EOF From fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001 Message-ID: From: A U Thor From d48c5d5a4c801dfe9acd5dc4a3c1b94430883f52 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Thu, 2 Apr 2026 08:51:19 +0200 Subject: [PATCH 2/5] t9300: work around partial read bug in Dash v0.5.13 When executing t9300 with Dash v0.5.13.1 we can see that the test hangs completely with the following (condensed) trace: git fast-import + error=1 + read output + cat input + echo checkpoint + echo progress checkpoint + test rogress checkpoint = progress checkpoint + test rogress checkpoint = UNEXPECTED + echo cruft: rogress checkpoint cruft: rogress checkpoint + read output + test = progress checkpoint + test = UNEXPECTED + echo cruft: cruft: + read output Basically, what's happening here is that we spawn git-fast-import(1) and wait for it to output a certain string, "progress checkpoint". Curiously though, what we end up reading is "rogress checkpoint" -- so the first byte of the expected string is missing. Same as in the preceding commit, this seems to be a bug in Dash itself that bisects to c5bf970 (expand: Add multi-byte support to pmatch, 2024-06-02). But other than in the preceding commit, this bug has already been fixed upstream in 079059a (input: Fix heap-buffer-overflow in preadbuffer on long lines, 2026-02-11), which is part of v0.5.13.2. For now though, work around the bug by waiting for the expected output in a different way. There is no good reason why one version should work better than the other, but at least the new version doesn't exhibit the bug. And, if you ask me, it's also slightly easier to read. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- t/t9300-fast-import.sh | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-) diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh index 5685cce6fea7da..479437760bae88 100755 --- a/t/t9300-fast-import.sh +++ b/t/t9300-fast-import.sh @@ -3635,25 +3635,21 @@ background_import_then_checkpoint () { echo "progress checkpoint" ) >&8 & - error=1 ;# assume the worst - while read output <&9 - do - if test "$output" = "progress checkpoint" - then - error=0 - break - elif test "$output" = "UNEXPECTED" - then - break - fi - # otherwise ignore cruft - echo >&2 "cruft: $output" - done + last=$( + while read output <&9 + do + if test "$output" = "progress checkpoint" || test "$output" = "UNEXPECTED" + then + echo "$output" + break + else + # otherwise ignore cruft + echo >&2 "cruft: $output" + fi + done + ) - if test $error -eq 1 - then - false - fi + test "$last" = "progress checkpoint" } background_import_still_running () { From 89152af176ea94ea8f3249115b6e00827fbbeb70 Mon Sep 17 00:00:00 2001 From: Johannes Schindelin Date: Fri, 3 Apr 2026 08:55:02 +0000 Subject: [PATCH 3/5] cmake: use writev(3p) wrapper as needed This is a companion patch of 3b9b2c2a29a (compat/posix: introduce writev(3p) wrapper, 2026-03-13) where support for using the `writev()` wrapper was introduced in the `Makefile` and the Meson-based build, but the CMake build still needs that treatment, too. Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano --- contrib/buildsystems/CMakeLists.txt | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 28877feb9d1707..ad58c320be14f8 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -374,7 +374,7 @@ endif() #function checks set(function_checks strcasestr memmem strlcpy strtoimax strtoumax strtoull - setenv mkdtemp poll pread memmem) + setenv mkdtemp poll pread memmem writev) #unsetenv,hstrerror are incompatible with windows build if(NOT WIN32) @@ -419,6 +419,10 @@ if(NOT HAVE_MEMMEM) list(APPEND compat_SOURCES compat/memmem.c) endif() +if(NOT HAVE_WRITEV) + list(APPEND compat_SOURCES compat/writev.c) +endif() + if(NOT WIN32) if(NOT HAVE_UNSETENV) list(APPEND compat_SOURCES compat/unsetenv.c) From c664ee2001a1ea0ecbc6448b24303687e6caf1cb Mon Sep 17 00:00:00 2001 From: Johannes Schindelin Date: Fri, 3 Apr 2026 09:56:23 +0000 Subject: [PATCH 4/5] mingw: use strftime() directly in UCRT builds The `mingw_strftime()` wrapper exists to work around msvcrt.dll's incomplete `strftime()` implementation by dynamically loading the version from ucrtbase.dll at runtime via `LoadLibrary()` + `GetProcAddress()`. When the binary is already linked against UCRT (i.e. when building in the UCRT64 environment), the linked-in `strftime()` is the ucrtbase.dll version, making the dynamic loading needless churn: It's calling the very same code. Simply guard both the declaration and implementation so that the unnecessary work-around is skipped in UCRT builds. Signed-off-by: Johannes Schindelin Signed-off-by: Junio C Hamano --- compat/mingw.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/compat/mingw.c b/compat/mingw.c index c667a2dcda7ac7..338ec3535edf41 100644 --- a/compat/mingw.c +++ b/compat/mingw.c @@ -1394,6 +1394,9 @@ int mingw_utime (const char *file_name, const struct utimbuf *times) size_t mingw_strftime(char *s, size_t max, const char *format, const struct tm *tm) { +#ifdef _UCRT + size_t ret = strftime(s, max, format, tm); +#else /* a pointer to the original strftime in case we can't find the UCRT version */ static size_t (*fallback)(char *, size_t, const char *, const struct tm *) = strftime; size_t ret; @@ -1404,6 +1407,7 @@ size_t mingw_strftime(char *s, size_t max, ret = strftime(s, max, format, tm); else ret = fallback(s, max, format, tm); +#endif if (!ret && errno == EINVAL) die("invalid strftime format: '%s'", format); From 2855562ca6a9c6b0e7bc780b050c1e83c9fcfbd0 Mon Sep 17 00:00:00 2001 From: Junio C Hamano Date: Fri, 3 Apr 2026 15:26:34 -0700 Subject: [PATCH 5/5] A couple more on top of -rc0 Signed-off-by: Junio C Hamano --- Documentation/RelNotes/2.54.0.adoc | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/RelNotes/2.54.0.adoc b/Documentation/RelNotes/2.54.0.adoc index 629e603f4336e8..04c038f035ae76 100644 --- a/Documentation/RelNotes/2.54.0.adoc +++ b/Documentation/RelNotes/2.54.0.adoc @@ -277,6 +277,12 @@ Performance, Internal Implementation, Development Support etc. * In case homebrew breaks REG_ENHANCED again, leave a in-code comment to suggest use of our replacement regex as a workaround. + * MinGW build updates. + + * The way dash 0.5.13 handles non-ASCII contents in here-doc + is buggy and breaks our existing tests, which unfortunately + have been rewritten to avoid triggering the bug. + Fixes since v2.53 -----------------