Draft replace rapidjson with simdjson by rok · Pull Request #47 · rok/arrow

rok · 2026-01-09T19:19:39Z

This is a discussion PR. It will likely be broken up in multiple PRs.

rok · 2026-01-28T18:56:51Z

@Taepper this currently in a bit of a cmake limbo :).

Taepper · 2026-01-28T18:58:51Z

@Taepper this currently in a bit of a cmake limbo :).

Hehe :) it is quite the big change, I will also take a look tomorrow!

WillAyd

Cool nice work. Some suggestions to improve the meson integration, which hopefully can help while the cmake issues get figured out (?)

WillAyd · 2026-02-03T15:43:05Z

cpp/src/arrow/meson.build

-        'RapidJSON',
-        fallback: ['rapidjson', 'rapidjson_dep'],
-    )
+    simdjson_dep = dependency('simdjson', fallback: ['simdjson', 'simdjson_dep'])


Suggested change

simdjson_dep = dependency('simdjson', fallback: ['simdjson', 'simdjson_dep'])

simdjson_dep = dependency('simdjson')

The fallback as written is not correct (the variable name in the simdjson module is actually still just simdjson), but I also don't think its required in this case

WillAyd · 2026-02-03T15:44:46Z

cpp/subprojects/simdjson.wrap

-source_hash = b9290a9a6d444c8e049bd589ab804e0ccf2b05dc5984a19ed5ae75d090064806
-source_fallback_url = https://apache.jfrog.io/artifactory/arrow/thirdparty/7.0.0/rapidjson-232389d4f1012dddec4ef84861face2d2ba85709.tar.gz
-patch_directory = rapidjson
+directory = simdjson-3.12.3


There's a wrap entry for 4.2.2 at this point - maybe worth using instead of 3.12? Or, if we wanted to stay in the 3 series, there's also 3.13

WillAyd · 2026-02-03T15:46:25Z

cpp/subprojects/packagefiles/simdjson/meson.build

 # under the License.

-project('rapidjson', 'cpp')
+project('simdjson', 'cpp', version: '3.12.3', default_options: ['cpp_std=c++17'])


You should be able to remove this file altogether. subprojects/packagefiles contain user-specific overrides for packages (i.e. you want to create a meson.build file that is "better" than what you get upstream).

In the case of simdjson, we should probably trust the configuration provided by the wrapdb, unless we know better

rok · 2026-02-04T00:23:04Z

Cool nice work. Some suggestions to improve the meson integration, which hopefully can help while the cmake issues get figured out (?)

Thanks @WillAyd ! I've not had a chance to properly look into meson config yet and this helps!

Taepper

A few comments – impressive how fast you went through the code-base! I would have thought that it would first only replace a few rapidjson usages.

Generally, we can choose whether to use the simdjson::dom or simdjson::ondemand frontend for arrow.

simdjson::ondemand can be faster and less memory-intensive. [1]

Its api is a little more difficult to use, but might be fine for our use-case, as we do not intend to parse multiple values in the document in parallel.

I see that you are using ondemand frontend for json/parser.cc, and dom for all other use-cases. It seems sensible to me at first glance, but it might make sense to document the reasoning for this decision

[1] https://arxiv.org/pdf/2312.17149

Taepper · 2026-02-03T16:06:48Z

ci/docker/alpine-linux-3.22-cpp.dockerfile

        py3-numpy-dev \
        python3-dev \
-        rapidjson-dev \
+        simdjson-dev \


Is this required when you bundle simdjson in all cases?

Taepper · 2026-02-03T16:09:09Z

cpp/examples/arrow/CMakeLists.txt

-  add_arrow_example(rapidjson_row_converter EXTRA_LINK_LIBS RapidJSON)
-  add_arrow_example(from_json_string_example EXTRA_LINK_LIBS RapidJSON)
-endif()
+add_arrow_example(from_json_string_example)


simdjson is not always enabled. Should this stay conditional?

if(ARROW_JSON OR ARROW_FLIGHT_SQL_ODBC) set(ARROW_WITH_SIMDJSON ON) endif()

Taepper · 2026-02-03T16:12:59Z

cpp/cmake_modules/ThirdpartyToolchain.cmake

+  )

-  externalproject_add(rapidjson_ep
+  set(SIMDJSON_CMAKE_ARGS


We might want to set this option to OFF:
option(SIMDJSON_EXCEPTIONS "Enable simdjson's exception-throwing interface" ON)
https://github.com/simdjson/simdjson/blob/6fffc99dac4029a67dd5a794d2f59ec6ba84a1b1/cmake/exception-flags.cmake#L1

This disables any exception-throwing API-functions from simdjson

Taepper · 2026-02-04T08:40:20Z

ci/scripts/r_revdepcheck.sh

  python3-rados \
  rados-objclass-dev \
-  rapidjson-dev \
+  libsimdjson-dev \


Taepper · 2026-02-04T08:41:13Z

cpp/examples/minimal_build/system_dependency.dockerfile

      pkg-config \
      protobuf-compiler \
-      rapidjson-dev \
+      libsimdjson-dev \


simdjson is bundled again, right?

rok · 2026-02-04T10:39:47Z

Thanks for the review @Taepper ! As a general comment, I thought I would first create a PR here and work on a POC "in secret" to see if it's feasible and then split it up into manageable PR against main.
I'm travelling today and will probably have time to work on this by end of week.

github-actions bot added Component: FlightRPC Component: C++ Component: Parquet awaiting committer review Component: Python Component: Documentation CI: Extra: C++ labels Jan 9, 2026

rok force-pushed the simdjson branch 2 times, most recently from 5af4403 to bb71ad8 Compare January 20, 2026 12:03

github-actions bot added Component: R Component: Gandiva Component: MATLAB Component: Ruby Component: GLib labels Jan 20, 2026

rok force-pushed the simdjson branch from 53ee4e6 to 8ea924d Compare January 20, 2026 12:34

Repository owner deleted a comment from github-actions bot Jan 20, 2026

github-actions bot removed Component: R Component: Gandiva Component: MATLAB Component: Ruby Component: GLib labels Jan 20, 2026

rok force-pushed the simdjson branch from aafaaeb to 5f1261a Compare January 21, 2026 16:25

github-actions bot added Component: R Component: Ruby labels Jan 21, 2026

rok force-pushed the simdjson branch from 5f1261a to bfee058 Compare January 21, 2026 16:27

rok mentioned this pull request Jan 21, 2026

[C++] can we use simdjson to replace rapidjson apache/arrow#35460

Open

github-actions bot removed Component: R Component: Ruby labels Jan 21, 2026

rok force-pushed the simdjson branch from f54ab81 to 934c5cf Compare January 23, 2026 12:03

github-actions bot added the Component: R label Jan 23, 2026

rok added 20 commits January 26, 2026 12:28

cmake

61296f1

cmake

343dda0

cmake

7cf6df0

cmake

63d8f3e

cmake

28ae1db

cmake

fb52c90

lint

cf1f66c

fix simdjson version in conan

355a9f2

simplification

84af758

fix

bb6f94c

cmake

e696611

cmake

06cb2ce

cmake

bc16df5

cmake

734da24

cmake

d8f55d5

cmake

56f0185

cmake

142193c

cmake

017cdec

cmake

ae9b2e7

cmake and lint

76ee292

rok force-pushed the simdjson branch from 0420e36 to 76ee292 Compare January 28, 2026 12:01

Taepper mentioned this pull request Jan 28, 2026

GH-35460: Use simdjson instead of RapidJSON apache/arrow#49048

Draft

rok added 2 commits January 29, 2026 02:29

cmake

f4386b0

cmake

19f136b

WillAyd reviewed Feb 3, 2026

View reviewed changes

Taepper reviewed Feb 4, 2026

View reviewed changes

	simdjson_dep = dependency('simdjson', fallback: ['simdjson', 'simdjson_dep'])
	simdjson_dep = dependency('simdjson')

Conversation

rok commented Jan 9, 2026

Uh oh!

rok commented Jan 28, 2026

Uh oh!

Taepper commented Jan 28, 2026

Uh oh!

WillAyd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Feb 4, 2026

Uh oh!

Taepper left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Feb 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants