Skip to content

Switch number parsing to std::from_chars / extended locale strto*_l if available#5237

Open
jgreitemann wants to merge 4 commits into
nlohmann:developfrom
jgreitemann:issue5198
Open

Switch number parsing to std::from_chars / extended locale strto*_l if available#5237
jgreitemann wants to merge 4 commits into
nlohmann:developfrom
jgreitemann:issue5198

Conversation

@jgreitemann

Copy link
Copy Markdown

Fixes #5198. The issue describes a TOCTOU bug when changes to the global C locale race against the use of localeconv/strto* by the lexer. Even discounting the TOCTOU issue, using locale-dependent functions while switching locale concurrently is undefined behavior.

Implements a two-tiered approach to avoid locale dependent number parsing:

  • When available, the C++17 std::from_chars provides an inherently locale-independent API.
  • Otherwise, use extended locale APIs (strto*_l on POSIX, _strto*_l on Windows) and pass them an instance of the "C" locale. This isolates them from the effects of the global locale.
  • Otherwise, fall back to the locale-dependent strto* functions. The bug is not fixed in this case.

All approaches are encapsulated in terms of a nlohmann::detail::from_chars API that is more or less identical to that of std::from_chars.

Caveats / Availability:

  • std::from_chars support is detected based on the __cpp_lib_to_chars feature-test macro. Implementation support for std::from_chars has not been great. For example, libc++ only added support for float and double in Clang 20 and to this day does not support long double. Hence, libc++ does not set __cpp_lib_to_chars and we won't be using std::from_chars, even for float and double. If a future version of libc++ adds support for long double, the PR would start using it, though.
  • Standard library implementations diverge in their handling of floating-point under-/overflow. P4168 does a good job in describing their disagreement and proposes the behavior that would actually be more useful as it allows distinguishing under-/overflow towards ±0 vs. ±∞. This PR implements a shim around std::from_chars which inspects the result in case of result_out_of_range and ensures behavior consistent with what P4168 proposes (and which libc++ and MS STL already implement), even on libstdc++ (which currently leaves the value unset). If P4168 is accepted as a defect report and implementations converge in the future, this shim should no longer be exercised (though it would have to stay in to still support older toolchains).
  • Extended locale functions, i.e. non-standard variants which accept an additional locale argument, cannot be assumed universally available on POSIX platforms and live outside of the POSIX standard. That being said, I'm not aware of any mainstream platform that doesn't provide the extended locale functions. This PR uses a trait class template with a SFINAE-based detection of the availability of strtof_l. On Windows, _strtof_l and friends have been available since at least Visual Studio 2005, so I think we don't need to bother with SFINAE in that case.
  • AFAICT, musl doesn't do locale, in the sense that all locales are essentially identical to the "C" locale. Curiously, the extended locale functions still exist for compatibility with software expecting them but they simply ignore the locale argument.

A last-resort mitigation (in case extended locale APIs are not available) would potentially see us using uselocale (POSIX) and _configthreadlocale (Windows) to set (and unset) a thread-local locale. This approach comes with degraded performance, even for users who do not need to mitigate the issue because they don't change locale. In practice, the absence of the two other mitigations should be quite rare, so this approach is not implemented by this PR (see discussion in issue). An open question is whether this is still a sufficiently important issue to add a note to the documentation.

  • The changes are described in detail, both the what and why.
  • If applicable, an existing issue is referenced.
  • The Code coverage remained at 100%. A test case for every new line of code.
  • If applicable, the documentation is updated.
  • The source code is amalgamated by running make amalgamate.

In debug builds, this test fails with `SIGABRT` due to an assertion.
In release builds, truncation of the fractional part will be detected.

Sometimes, this test may also cause memory violation (`SIGSEGV`) due to
it being undefined behavior to set locale and use locale-dependent
functions concurrently. On Windows, this does not happen since the
global locale is protected by a mutex.

This test assumes that a locale called "de_DE.UTF-8" exists on the host.
If it does not, the test bails out but does not fail. The locale name is
hard-coded because it is impossible to query available locales portably.
"de_DE.UTF-8" is an accepted format on Ubuntu, macOS, and Windows.

Signed-off-by: Jonas Greitemann <jgreitemann@gmail.com>
When an integer JSON token exceeds the range of the underlying 64-bit
unsigned/integer types, the lexer instead identifies the token type as
`value_float`. When the value exceeds the range of the underlying
floating-point type, the token type remains as `value_float`.

This was previously not tested explicitly for the edge case values and
subtly breaking this behavior right around the edge case was not caught
by any other test.

Signed-off-by: Jonas Greitemann <jgreitemann@gmail.com>
…herwise

`std::from_chars` is locale-independent and thus avoids the TOCTOU
issue nlohmann#5198 when it's available.

This commit introduces a `from_chars` utility in the `detail` namespace
which merely wraps `std::from_chars` if it is available and emulates its
API and behavior using the `strto*` family of functions when it is not.
As of this commit, the `strto*`-based fallback is still locale-dependent
and only matches `std::from_chars` when the "C" locale is used.

Availability of `std::from_chars` for floating point numbers is varied.
For instance, as of libc++ 22, `std::from_chars` for `long double` isn't
supported and, consequently, the feature-test macro `__cpp_lib_to_chars`
is not set, while the overloads for `float` and `double` are available.
To avoid complicating matters further, the feature-test macro is taken
as the authoritative indicator for (full) `std::from_chars` support and
the fallback is used otherwise.

If `std::from_chars` is available, our wrapper additionally tweaks its
output in the case of floating-point out-of-range results. The behavior
in this case is poorly (under-)specified and existing implementations do
not agree. The wrapper's tweaks are intended to ensure consistent
behavior across all toolchains in this edge case. In practice, this
should only take effect when parsing out-of-range floats on libstdc++.

The unit tests for the `from_chars` helper are intentionally compiled in
both C++11 and C++17 modes. This ensures that the test expectations
align with the actual behavior we're trying to model. The tests are not
aiming to cover all of floating-point parsing comprehensively, as
`std::from_chars` is specified to behave mostly the same as `strto*`
and instead focuses on the edge cases where the two differ:
- leading whitespace is not tolerated
- `+` sign is not allowed (outside of the exponent)
- out-parameter is left unchanged in case of `invalid_argument` and
  integral `result_out_of_range` errors

Signed-off-by: Jonas Greitemann <jgreitemann@gmail.com>
On Windows, `_strtof_l` et al. are always available. POSIX on the other
hand only covers `newlocale` and `freelocale` but the extended locale
functions `strtof_l` provided by glibc or `<xlocale.h>` on BSDs cannot
be assumed to be universally available. For this reason, the trait
`has_extended_locale_support` uses SFINAE to detect whether `strtof_l`
is defined and falls back to the locale-dependent standard functions if
it isn't.

Signed-off-by: Jonas Greitemann <jgreitemann@gmail.com>

@nlohmann nlohmann left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some observations (thanks to Claude Code):

  1. The std::from_chars-tier P4168 shim overflows a plain int when re-parsing the exponent, causing an assertion abort (debug builds) or a silently wrong result (release builds). E.g., "1e-99999999999999999999" incorrectly parses to inf instead of the correct 0.0.

  2. The POSIX extended_locale_traits SFINAE probe only tests strtof_l, but the selected specialization unconditionally uses strtoll_l/strtoull_l/isspace_l too, which musl doesn't declare.

@nlohmann

nlohmann commented Jul 8, 2026

Copy link
Copy Markdown
Owner

Hi @jgreitemann, thanks for your patch! I had a look at the CI failures:

  1. Widespread linker failures ("undefined reference to pthread_create") — this is the single biggest cluster (gcc, gcc_old, clang 3.9+, icpc, noexceptions, and more). The new regression test for TOCTOU race between lexer construction and locale changes causes float truncation #5198 spawns a std::thread, but tests/CMakeLists.txt never links against Threads::Threads/pthread. It happened to link on some toolchains only because their driver auto-adds -pthread. Fix: add find_package(Threads REQUIRED) + link Threads::Threads for the test targets.
  2. #if defined(__has_include) && __has_include(<version>) breaks old preprocessors (from_chars.hpp:13, :29) — GCC 4.8 errors with missing binary operator before token "(", and MSVC (AppVeyor VS2015 jobs) warns C4067 unexpected tokens following preprocessor directive (promoted to error under /WX). The rest of the codebase avoids this by nesting the checks instead of combining them on one line, e.g. macro_scope.hpp:61-62 (#ifdef __has_include / #if __has_include(<version>)). Same fix should apply here.
  3. -Wfloat-equal -Werror (gcc) at from_chars.hpp:116 and :418 — the exact-equality checks (value == orig_value, value == T{}/infinity()) are intentional, but need the same #pragma GCC diagnostic push / ignored "-Wfloat-equal" / pop wrapping already used in to_chars.hpp:1078-1090, json.hpp:3761/3865, and binary_writer.hpp:1779.
  4. clang-tidy misc-const-correctness at from_chars.hpp:260 (c_locale can be const) and :485 (pointee of ptr can be const) — straightforward const additions.
  5. MinGW build fails: _strtoll_l/_strtoull_l/_strtof_l/_strtold_l are "not declared in this scope". The comment "on Windows, _strtof_l et al. are always available" doesn't hold for MinGW's CRT (only real MSVC). Might need a MinGW-specific carve-out that falls back to the locale-dependent path, similar to the musl case.
  6. MSVC VS2015 jobs (AppVeyor): error C2131: expression did not evaluate to a constant at from_chars.hpp:379/387/395/403/411 — the from_chars_traits<T> specializations initialize static constexpr function-pointer members (&extended_locale_traits<>::strtoll etc.) at class-definition time, which VS2015's constexpr support apparently can't handle. Worth checking if VS2015 is still a hard requirement for this project's CI matrix; if so this may need a non-constexpr indirection for that toolset.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TOCTOU race between lexer construction and locale changes causes float truncation

2 participants