Ability to get binary response body (e.g., for images)

[kinghuang]

Is there a way to get the http_get function to return a binary response body? For example, let's say I want to get a PNG image. Right now, it results in an Invalid Input Error.

D select
    http_get('https://httpbin.org/image', headers => MAP {'accept': 'image/png'}, params => MAP {}) as data
  ;
Invalid Input Error: Invalid unicode (byte sequence mismatch) detected in segment statistics update

The http_get function currently returns a JSON object with the response status, reason, and body as a string. This doesn't work well with non-text response bodies.

[lmangani]

Hello @kinghuang
I guess our focus was mostly data so far - but why not, we could add other datatypes for BLOB destinations.
We can try add this to the roadmap in best-effort mode or accept a PR quite happily!

[kinghuang]

I appreciate the best-effort mode! I haven't written any C++ code in ages, but I'll consider that later if I find some time!

[lmangani]

We can either do some magic detections in the mime-types returned by a request, or simply add a new function dedicated to binary types... a rough slop sketch 👇

// Add this new function
static void HTTPGetBlobRequestFunction(DataChunk &args, ExpressionState &state, Vector &result) {
    D_ASSERT(args.data.size() == 1);

    UnaryExecutor::Execute<string_t, string_t>(args.data[0], result, args.size(), [&](string_t input) {
        std::string url = input.GetString();

        // Setup client
        auto client_and_path = SetupHttpClient(url);
        auto &client = client_and_path.first;
        auto &path = client_and_path.second;

        // GET request
        auto res = client.Get(path.c_str());
        if (res) {
            // Return body as BLOB
            auto &body = res->body;
            return StringVector::AddBlob(result, body.data(), body.size());
        } else {
            // On error, return NULL
            return StringVector::AddNull(result);
        }
    });
}

// In LoadInternal, register the new function
ScalarFunctionSet http_get_blob("http_get_blob");
http_get_blob.AddFunction(ScalarFunction({LogicalType::VARCHAR}, LogicalType::BLOB(), HTTPGetBlobRequestFunction));
ExtensionUtil::RegisterFunction(instance, http_get_blob);

[kinghuang]

Yeah, that would work!