Build ODBC statically on macOS

* Link ODBC library statically on unix

Change unit tests to re-use code. Testing on windows is needed.

Now seeing error
```
Library not loaded: /usr/local/opt/grpc/lib/libgrpc++.1.76.dylib
```
on Excel.

Attempt to unlink arrow_flight.proto

Now I see ODBC.dylib has 100MB, and error from Excel is
```
Library not loaded: /usr/local/opt/libiodbc/lib/libiodbc.2.dylib
```
instead of looking for flightsql.

`iodbctest` command outside of Excel works.

in-progress link arrow statically for `flightsql-odbc`

* both ODBC driver and `arrow_odbc_spi_impl` need to link Arrow statically for tests to work.
* getting error
`File already exists in database: Flight.proto`
since the ODBC layer is linking dynamically while I successfully got `arrow_odbc_spi_impl` to link to Arrow statically

Draft for resolving `File already exists in database: FlightSql.proto` error

Note: make ODBC available on Excel first, test with `iodbctest`, worry about ODBC test executable later.

Add unix build (same as Windows build)

* Attempted to link static libraries with `arrow_odbc_spi_impl`, failed as it caused linking errors and rendered the ODBC driver unusable.

Undo changes to resolve broken driver

Resolves dependency errors

Attempt to switch to static build

Still getting error
```
[iODBC][Driver Manager]dlopen(/Library/ODBC/arrow-odbc/libarrow_flight_sql_odbc.2300.0.0.dylib, 0x0006): Library not loaded: @rpath/libarrow_flight_sql.2300.dylib
```

In-progress attempt to build ODBC statically on unix systems

In-progress changes to enable static odbc_impl library

pushing the changes just for saving my code. Currently odbc test still has double registration issue, work on this next. I think it should be solvable by either adding library flags or just using odbc spi shared. Because `odbc_spi_impl_test` works. And the ODBC tests worked just fine before I switched to static odbc spi impl.

Fix for gtest missing issue

Attempting to fix for issue:
IMPORTANT NOTICE - DO NOT IGNORE:
This test program did NOT call testing::InitGoogleTest() before calling RUN_ALL_TESTS().
This is INVALID. Soon Google Test will start to enforce the valid usage.
Please fix it ASAP, or IT WILL START TO FAIL.

Issue fixed with adding `${ARROW_TEST_LINK_LIBS}`. It is needed alongside `arrow_flight_testing_shared`.

IN-PROGRESS build static test on macOS

* Trying to have 2 builds. One `arrow_odbc_spi_impl_static` for static build and one `arrow_odbc_spi_impl_shared` for a shared lib that links Arrow dynamically
---
only committing the approach that worked.
Trying to work from `flight_sql_odbc_test` to debug the real issue.

Link 3rd party dependencies statically

Fix 3rd party headers cannot find issue for:
- boost xpressive and beast headers
- rapidjson headers

Resolve unneeded link for ODBC

----
Moving `set(APPEND CMAKE_FIND_LIBRARY_SUFFIXES ".a")` up didn't seem to have any effect

----
Fix for static gRPC connection issue:
export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/etc/ssl/cert.pem

Fix cares `cannot modify alias target` issue

Make arrow_odbc_spi_impl static lib

In progress for getting gprc linked statically

Add debug msgs in add_arrow_lib BuildUtils.cmake

Remove commented out code in odbc lib

Fix build issue from odbc impl tests

Added draft code for Link Arrow libs statically on Unix for unit tests

Add commented out code`ARROW_DEPENDENCY_USE_SHARED` in ODBC cpp yml

Uncomment when it is verified to work.

In-progress Fix ODBC test build and make it run.

Build executable from scratch

Revert "Build executable from scratch"

This reverts commit a51dc39.

Dummy test

dummy test without using ODBC passed without double registration issue

Make ODBC test run

Issue to resolve:
[libprotobuf FATAL /path/to/arrow/cpp/debug-build/_deps/protobuf-src/src/google/protobuf/extension_set.cc:100] Multiple extension registrations for type "google.protobuf.MessageOptions", field number 1000.
unknown file: Failure
C++ exception with description "Multiple extension registrations for type "google.protobuf.MessageOptions", field number 1000." thrown in SetUp().

Fix: only link test executable directly with ODBC shared dylib. Do not link test executable with odbc spi impl dylib. And change ODBC to link odbc spi impl dylib publicly instead of privately

in-progress changes to accomodate macos CI

Test ODBC CI - to be reverted

Can revert later

Only install ODBC dependencies in mac

As dependencies are static and built from source

Uninstall absl as attempt to resolve build on Intel

Add C++ standard 20

Remove absl header from intel

Use static linking in dependency install step

Use bundled boost for static linking

In-progress Fix Windows build

Fix macOS (after Windows build)

Remove unneeded ODBC link

Resolves the issue of ODBC on macOS being dynamically linked to ODBC

Fix ODBC double proto issue on macOS

Need to check if Windows CI passes

Clean up PR

- Remove dummy tests
- Remove unneeded changes

Revert "Test ODBC CI - to be reverted"

This reverts commit 711b3e0.

Remove unneeded code

Attempt to revert c-ares change

Fix `set_target_properties can not be used on an ALIAS target.` error.

Example of failed run in forked repo:
https://github.com/Bit-Quill/arrow/actions/runs/21459770235/job/61809740997?pr=153#step:7:540

Fix descriptor pointer tests on static macOS

Fix mimalloc issue

Fixes issue:
```
mimalloc: assertion failed: at "arrow/cpp/debug-build/mimalloc_ep-prefix/src/mimalloc_ep/include/mimalloc/internal.h":658, _mi_ptr_page
  assertion: "p==NULL || mi_is_in_heap_region(p)"
zsh: abort      debug/arrow-flight-sql-odbc-test
```

Enable ODBC tests for static build

Add iodbc link to macOS build on ODBC layer

Add iodbc link to macOS build on ODBC Test

Add iodbc link to macOS build on ODBC layer did not resolve issue of unixodbc being picked in CI

* Fix segfault at SQLError

* work on Justin's comments

Author: alinaliBQ

.github/workflows/cpp_extra.yml

@@ -357,6 +357,10 @@ jobs:
       ARROW_BUILD_TESTS: ON
       ARROW_FLIGHT_SQL_ODBC: ON
       ARROW_HOME: /tmp/local
+      ARROW_DEPENDENCY_USE_SHARED: OFF
+      ARROW_DEPENDENCY_SOURCE: BUNDLED
+      ARROW_MIMALLOC: OFF
+      CMAKE_CXX_STANDARD: "20"
     steps:
       - name: Checkout Arrow
         uses: actions/checkout@v6.0.1
@@ -366,6 +370,22 @@ jobs:
       - name: Install Dependencies
         run: |
           brew bundle --file=cpp/Brewfile
+          
+          # We want to use bundled RE2 for static linking. If
+          # Homebrew's RE2 is installed, its header file may be used.
+          # We uninstall Homebrew's RE2 to ensure using bundled RE2.
+          brew uninstall grpc || : # gRPC depends on RE2
+          brew uninstall grpc@1.54 || : # gRPC 1.54 may be installed too
+          brew uninstall re2
+
+          # We want to use bundled Protobuf for static linking. If
+          # Homebrew's Protobuf is installed, its library file may be
+          # used on test  We uninstall Homebrew's Protobuf to ensure using
+          # bundled Protobuf.
+          brew uninstall protobuf
+
+          # We want to use bundled Boost for static linking.
+          brew uninstall boost
       - name: Setup ccache
         run: |
           ci/scripts/ccache_setup.sh
@@ -395,6 +415,10 @@ jobs:
           export ARROW_CMAKE_ARGS="-DODBC_INCLUDE_DIR=$ODBC_INCLUDE_DIR"
           export CXXFLAGS="$CXXFLAGS -I$ODBC_INCLUDE_DIR"
           ci/scripts/cpp_build.sh $(pwd) $(pwd)/build
+      - name: Register Flight SQL ODBC Driver
+        run: |
+          chmod +x cpp/src/arrow/flight/sql/odbc/install/mac/install_odbc.sh
+          sudo cpp/src/arrow/flight/sql/odbc/install/mac/install_odbc.sh $(pwd)/build/cpp/debug/libarrow_flight_sql_odbc.dylib
       - name: Test
         shell: bash
         run: |
@@ -431,6 +455,7 @@ jobs:
       CMAKE_INSTALL_PREFIX: /usr
       VCPKG_BINARY_SOURCES: 'clear;nugettimeout,600;nuget,GitHub,readwrite'
       VCPKG_DEFAULT_TRIPLET: x64-windows
+      CMAKE_CXX_STANDARD: "20"
     steps:
       - name: Disable Crash Dialogs
         run: |

ci/scripts/cpp_test.sh

@@ -59,7 +59,6 @@ case "$(uname)" in
     ;;
   Darwin)
     n_jobs=$(sysctl -n hw.ncpu)
-    exclude_tests+=("arrow-flight-sql-odbc-test")
     # TODO: https://github.com/apache/arrow/issues/40410
     exclude_tests+=("arrow-s3fs-test")
     ;;

cpp/cmake_modules/ThirdpartyToolchain.cmake

@@ -1114,6 +1114,11 @@ function(build_boost)
   else()
     list(APPEND BOOST_EXCLUDE_LIBRARIES uuid)
   endif()
+  if(ARROW_FLIGHT_SQL_ODBC)
+    list(APPEND BOOST_INCLUDE_LIBRARIES beast xpressive)
+  else()
+    list(APPEND BOOST_EXCLUDE_LIBRARIES beast xpressive)
+  endif()
   set(BOOST_SKIP_INSTALL_RULES ON)
   if(NOT ARROW_ENABLE_THREADING)
     set(BOOST_UUID_LINK_LIBATOMIC OFF)
@@ -3023,8 +3028,8 @@ function(build_cares)
   if(APPLE)
     # libresolv must be linked from c-ares version 1.16.1
     find_library(LIBRESOLV_LIBRARY NAMES resolv libresolv REQUIRED)
-    set_target_properties(c-ares::cares PROPERTIES INTERFACE_LINK_LIBRARIES
-                                                   "${LIBRESOLV_LIBRARY}")
+    set_target_properties(c-ares PROPERTIES INTERFACE_LINK_LIBRARIES
+                                            "${LIBRESOLV_LIBRARY}")
   endif()
 
   set(ARROW_BUNDLED_STATIC_LIBS

cpp/src/arrow/flight/sql/odbc/CMakeLists.txt

@@ -19,6 +19,11 @@
 # GH-44792: Arrow will switch to C++ 20
 set(CMAKE_CXX_STANDARD 20)
 
+if(APPLE)
+  # CMAKE_FIND_LIBRARY_SUFFIXES.
+  set(APPEND CMAKE_FIND_LIBRARY_SUFFIXES ".a")
+endif()
+
 if(WIN32)
   if(MSVC_VERSION GREATER_EQUAL 1900)
     set(ODBCINST legacy_stdio_definitions odbccp32 shlwapi)
@@ -61,33 +66,56 @@ if(WIN32)
   list(APPEND ARROW_FLIGHT_SQL_ODBC_SRCS odbc.def install/versioninfo.rc)
 endif()
 
-add_arrow_lib(arrow_flight_sql_odbc
-              CMAKE_PACKAGE_NAME
-              ArrowFlightSqlOdbc
-              PKG_CONFIG_NAME
-              arrow-flight-sql-odbc
-              OUTPUTS
-              ARROW_FLIGHT_SQL_ODBC_LIBRARIES
-              SOURCES
-              ${ARROW_FLIGHT_SQL_ODBC_SRCS}
-              DEPENDENCIES
-              arrow_flight_sql
-              DEFINITIONS
-              UNICODE
-              SHARED_LINK_FLAGS
-              ${ARROW_VERSION_SCRIPT_FLAGS} # Defined in cpp/arrow/CMakeLists.txt
-              SHARED_LINK_LIBS
-              arrow_flight_sql_shared
-              SHARED_INSTALL_INTERFACE_LIBS
-              ArrowFlight::arrow_flight_sql_shared
-              STATIC_LINK_LIBS
-              arrow_flight_sql_static
-              STATIC_INSTALL_INTERFACE_LIBS
-              ArrowFlight::arrow_flight_sql_static
-              SHARED_PRIVATE_LINK_LIBS
-              ODBC::ODBC
-              ${ODBCINST}
-              arrow_odbc_spi_impl)
+# On Windows, dynmaic build for ODBC is supported.
+# On unix systems, static build for ODBC is supported, all libraries are linked statically on unix.
+if(WIN32)
+  add_arrow_lib(arrow_flight_sql_odbc
+                CMAKE_PACKAGE_NAME
+                ArrowFlightSqlOdbc
+                PKG_CONFIG_NAME
+                arrow-flight-sql-odbc
+                OUTPUTS
+                ARROW_FLIGHT_SQL_ODBC_LIBRARIES
+                SOURCES
+                ${ARROW_FLIGHT_SQL_ODBC_SRCS}
+                DEFINITIONS
+                UNICODE
+                SHARED_LINK_FLAGS
+                ${ARROW_VERSION_SCRIPT_FLAGS} # Defined in cpp/arrow/CMakeLists.txt
+                SHARED_LINK_LIBS
+                arrow_flight_sql_shared
+                arrow_odbc_spi_impl
+                SHARED_INSTALL_INTERFACE_LIBS
+                ArrowFlight::arrow_flight_sql_shared
+                STATIC_LINK_LIBS
+                arrow_flight_sql_static
+                STATIC_INSTALL_INTERFACE_LIBS
+                ArrowFlight::arrow_flight_sql_static
+                SHARED_PRIVATE_LINK_LIBS
+                ODBC::ODBC
+                ${ODBCINST})
+else()
+  # Unix
+  add_arrow_lib(arrow_flight_sql_odbc
+                CMAKE_PACKAGE_NAME
+                ArrowFlightSqlOdbc
+                PKG_CONFIG_NAME
+                arrow-flight-sql-odbc
+                OUTPUTS
+                ARROW_FLIGHT_SQL_ODBC_LIBRARIES
+                SOURCES
+                ${ARROW_FLIGHT_SQL_ODBC_SRCS}
+                DEFINITIONS
+                UNICODE
+                SHARED_LINK_FLAGS
+                ${ARROW_VERSION_SCRIPT_FLAGS} # Defined in cpp/arrow/CMakeLists.txt
+                STATIC_LINK_LIBS
+                iodbc
+                ODBC::ODBC
+                ${ODBCINST}
+                SHARED_LINK_LIBS
+                arrow_odbc_spi_impl)
+endif()
 
 foreach(LIB_TARGET ${ARROW_FLIGHT_SQL_ODBC_LIBRARIES})
   target_compile_definitions(${LIB_TARGET} PRIVATE ARROW_FLIGHT_SQL_ODBC_EXPORTING)

cpp/src/arrow/flight/sql/odbc/odbc_impl/CMakeLists.txt

@@ -15,7 +15,7 @@
 # specific language governing permissions and limitations
 # under the License.
 
-add_library(arrow_odbc_spi_impl
+add_library(arrow_odbc_spi_impl STATIC
             accessors/binary_array_accessor.cc
             accessors/binary_array_accessor.h
             accessors/boolean_array_accessor.cc
@@ -133,8 +133,11 @@ endif()
 if(APPLE)
   target_include_directories(arrow_odbc_spi_impl SYSTEM BEFORE PUBLIC ${ODBC_INCLUDE_DIR})
   target_link_libraries(arrow_odbc_spi_impl
-                        PUBLIC arrow_flight_sql_shared arrow_compute_shared Boost::locale
-                               iodbc)
+                        PUBLIC arrow_flight_sql_static
+                               arrow_compute_static
+                               Boost::locale
+                               Boost::headers
+                               RapidJSON)
 else()
   find_package(ODBC REQUIRED)
   target_include_directories(arrow_odbc_spi_impl PUBLIC ${ODBC_INCLUDE_DIR})
@@ -143,14 +146,6 @@ else()
                                ${ODBCINST})
 endif()
 
-set_target_properties(arrow_odbc_spi_impl
-                      PROPERTIES ARCHIVE_OUTPUT_DIRECTORY
-                                 ${CMAKE_BINARY_DIR}/$<CONFIG>/lib
-                                 LIBRARY_OUTPUT_DIRECTORY
-                                 ${CMAKE_BINARY_DIR}/$<CONFIG>/lib
-                                 RUNTIME_OUTPUT_DIRECTORY
-                                 ${CMAKE_BINARY_DIR}/$<CONFIG>/lib)
-
 # CLI
 add_executable(arrow_odbc_spi_impl_cli main.cc)
 set_target_properties(arrow_odbc_spi_impl_cli
@@ -159,6 +154,16 @@ set_target_properties(arrow_odbc_spi_impl_cli
 target_link_libraries(arrow_odbc_spi_impl_cli arrow_odbc_spi_impl)
 
 # Unit tests
+
+# On Windows, dynamic linking ODBC is supported.
+# On unix systems, static linking ODBC is supported, thus the library linking is static.
+if(WIN32)
+  set(ODBC_SPI_IMPL_TEST_LINK_LIBS arrow_flight_testing_shared)
+else()
+  # unix
+  set(ODBC_SPI_IMPL_TEST_LINK_LIBS arrow_flight_testing_static)
+endif()
+
 add_arrow_test(odbc_spi_impl_test
                SOURCES
                accessors/boolean_array_accessor_test.cc
@@ -177,4 +182,4 @@ add_arrow_test(odbc_spi_impl_test
                util_test.cc
                EXTRA_LINK_LIBS
                arrow_odbc_spi_impl
-               arrow_flight_testing_shared)
+               ${ODBC_SPI_IMPL_TEST_LINK_LIBS})

cpp/src/arrow/flight/sql/odbc/odbc_impl/config/connection_string_parser.h

@@ -19,7 +19,7 @@
 
 #include <string>
 
-#include "config/configuration.h"
+#include "arrow/flight/sql/odbc/odbc_impl/config/configuration.h"
 
 namespace arrow::flight::sql::odbc {
 namespace config {

cpp/src/arrow/flight/sql/odbc/odbc_impl/odbc_statement.cc

@@ -30,7 +30,6 @@
 #include <sql.h>
 #include <sqlext.h>
 #include <sqltypes.h>
-#include <boost/variant.hpp>
 #include <optional>
 #include <utility>
 

cpp/src/arrow/flight/sql/odbc/odbc_impl/spi/connection.h

@@ -18,7 +18,6 @@
 #pragma once
 
 #include <boost/algorithm/string.hpp>
-#include <boost/variant.hpp>
 #include <functional>
 #include <map>
 #include <optional>

cpp/src/arrow/flight/sql/odbc/odbc_impl/ui/add_property_window.cc

@@ -15,7 +15,7 @@
 // specific language governing permissions and limitations
 // under the License.
 
-#include "ui/add_property_window.h"
+#include "arrow/flight/sql/odbc/odbc_impl/ui/add_property_window.h"
 
 #include <Windowsx.h>
 
@@ -25,8 +25,8 @@
 #include <commctrl.h>
 
 #include "arrow/flight/sql/odbc/odbc_impl/exceptions.h"
-#include "ui/custom_window.h"
-#include "ui/window.h"
+#include "arrow/flight/sql/odbc/odbc_impl/ui/custom_window.h"
+#include "arrow/flight/sql/odbc/odbc_impl/ui/window.h"
 
 namespace arrow::flight::sql::odbc {
 namespace config {

cpp/src/arrow/flight/sql/odbc/odbc_impl/ui/custom_window.cc

@@ -29,7 +29,7 @@
 #include <sstream>
 
 #include "arrow/flight/sql/odbc/odbc_impl/exceptions.h"
-#include "ui/custom_window.h"
+#include "arrow/flight/sql/odbc/odbc_impl/ui/custom_window.h"
 
 namespace arrow::flight::sql::odbc {
 namespace config {