Fix "JSON.generate: UTF-8 string passed as BINARY" warning for text attachments

[andreaslillebo]

Fix "JSON.generate: UTF-8 string passed as BINARY" warning for text attachments

Problem

Sending a text attachment with non-ASCII content trips a deprecation warning that becomes a hard error in json 3.0:

File.write('hei.txt', 'Hei på deg')

chat = RubyLLM.chat
chat.ask 'What does this say?', with: 'hei.txt'
# warning: JSON.generate: UTF-8 string passed as BINARY, this will raise an encoding error in json 3.0

Cause

The path the bytes take from hei.txt to JSON.generate:

# lib/ruby_llm/attachment.rb
def load_content_from_path
  @content = File.binread(@source)                                          # => ASCII-8BIT
end

# lib/ruby_llm/attachment.rb
def for_llm
  case type
  when :text
    "<file name='#{filename}' mime_type='#{mime_type}'>#{content}</file>"   # => ASCII-8BIT
  else
    "data:#{mime_type};base64,#{encoded}"
  end
end

# Faraday's :json request middleware calls JSON.generate on the request body.
# json/common.rb:445: warning: JSON.generate: UTF-8 string passed as BINARY, ...

File.binread reads bytes as ASCII-8BIT. The tag rides through for_llm's string interpolation into the request body, where JSON.generate rejects it. The other three content loaders (load_content_from_io, load_content_from_active_storage, fetch_content) hit the same chain.

Solution

Re-tag the content as UTF-8 once the mime type is known to be text-like:

@content&.force_encoding(Encoding::UTF_8) if text?

Test

Adds a non-ASCII fixture (spec/fixtures/multilingual.txt) and a regression test:

it 'serializes non-ASCII text/* attachment content as JSON' do
  attachment = RubyLLM::Attachment.new('spec/fixtures/multilingual.txt')

  expect { JSON.generate(text: attachment.content) }.not_to output.to_stderr
end

The test fails before the fix with the json gem's deprecation warning, and passes after.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.05%. Comparing base (4942d6c) to head (3485a5f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #762   +/-   ##
=======================================
  Coverage   87.05%   87.05%           
=======================================
  Files         119      119           
  Lines        5594     5595    +1     
  Branches     1407     1408    +1     
=======================================
+ Hits         4870     4871    +1     
  Misses        724      724           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.