[Q&A] Not able to send messages from fluentd to kafka with zstd compression codec

[RamHaridas]

What is a problem?

I am trying send messages from fluentd to kafka with compression codec zstd.
But the message passing fails with an error saying that libzstd not available at build time. The error is thrown by rdkafka plugin.
I have installed the gem from fluent-package-v5.0.7 package.

Describe the configuration of Fluentd

I am using below config with compression codec set to zstd with @type rdkafk2

   <source>
      @type exec
      tag kafka.message
      command echo '{"message":"Hello there [rdkafka2 - GSSAPI - test3 zstd]!"}'
      run_interval 10s
      <parse>
        @type json
      </parse>
    </source>
    <match kafka.message>
      @type rdkafka2
      brokers kf-test-kafka-0.kf-test-kafka-headless.preeti-krb-system.svc.cluster.local:9092
      topic_key topic
      default_topic fluentd-zstd
      use_event_time true
      principal <<krb_principle>>
      keytab /etc/kerberos/secret/keytab
      compression_codec zstd
      ssl_ca_cert /etc/kerberos/kafka/tls.crt
      ssl_client_cert /etc/kerberos/kafka/tls.crt
      ssl_client_cert_key /etc/kerberos/kafka/tls.key
      ssl_client_cert_key_password test1234
      # ssl_verify_hostname false
      <format>
        @type json
      </format>
      <buffer topic>
        @type file
        path /var/log/fluentd-buffers/kafka
        flush_mode interval
        flush_interval 5s
        chunk_limit_size 1m
        queue_limit_length 32
        retry_max_interval 30
        retry_forever true
      </buffer>
      log_level debug
    </match>

Describe the logs of Fluentd

Fluentd throws below error logs

#0 Send exception occurred: Unsupported value "zstd" for configuration property "compression.codec": libzstd not available at build time at /opt/fluent/lib/ruby/gems/3.2.0/gems/rdkafka-0.12.0/lib/rdkafka/config.rb:226:in `block (2 levels) in native_config'getting this error when i am trying to send messages to kafka using rdkafka plugin and zstd compression codec

Now I understand that the error outcome is a result of the fact that the rdkafka 0.12.0 (coming from fluent-package-v5.0.7) has not been built with zstd support.
Because when I switch to explicit installation of the rdkafka gem with libzstd-devel package, it works and does not throw the above mentioned error.

I wanted to confirm if that is the case and the rdkafka gem present in fluent-package-v5.0.7 does not support zstd by default?

Environment

- Fluentd version: 1.16.9
- Fluent Package version: v5.0.7
- Operating system: Rocky Linux 8.10 (Green Obsidian)
- Kernel version: 5.14
- Rdkafka gem: 0.12.0

Update: I have tested the same scenario with fluent package v6.0.0 as well. The rdkafka gem failed with same error in that version as well.

[Watson1978]

I answered at fluent/fluent-plugin-kafka#539