Tag path params as UTF-8 instead of leaving them binary

[ericproulx]

Summary

Mustermann decodes path captures out of PATH_INFO, which Rack hands over tagged ASCII-8BIT, and nothing re-tagged the result. Query and body params arrive UTF-8 because Rack tags those itself — so the same value reached the endpoint with a different encoding depending on where it came from.

That made an API's own declarations disagree with themselves, since a binary string never equals the UTF-8 literal it was written as:

params { requires :id, type: String, values: ['café'] }

request	before	after
GET /?id=café (query)	200	200
GET /café (path)	400 "id does not have a valid value"	200

The same held for same_as, except_values, and any comparison an endpoint made against a non-ASCII literal. It also leaked into serialization: a non-ASCII path param rendered into a JSON response drew warning: JSON.generate: UTF-8 string passed as BINARY, which the json gem says will become an error in json 3.0.

Approach

Re-tag in Route#params_for, the single funnel for path-extracted values.

Why UTF-8, given the path is octets. Nothing obliges a client to send UTF-8: HTTP treats the request target as octets, and Rack's SPEC has CGI keys carry non-ASCII as ASCII-8BIT — which is exactly why PATH_INFO arrives binary in the first place. UTF-8 is the convention rather than a guarantee: it is what browsers percent-encode with, what an IRI maps to, and what Rails settles on — ActionDispatch::Journey::Router force-encodes every path capture to UTF-8 after unescaping (action_dispatch/journey/router.rb:79, actionpack 8.1.3.1):

val = val.include?("%") ? CGI.unescapeURIComponent(val) : val
val.force_encoding(::Encoding::UTF_8)
path_parameters[name.to_sym] = val

Only the encoding changes — the bytes are untouched, so octets that are not UTF-8 stay detectably invalid and are still caught downstream rather than being silently scrubbed into something the client never sent. Unnamed splats capture into an Array ({"splat" => ["a", "b"]}), so those are walked too.

Backward compatibility

UPGRADING entry added. Comparisons against pure-ASCII strings are unaffected, which is the overwhelming majority of code. Code that relied on a path param being binary — concatenating one with genuinely binary data — can now raise Encoding::CompatibilityError and should call .b on the param. An app that already worked around this with its own force_encoding needs no change; that call becomes a no-op.

For an app with too many such sites to patch individually, UPGRADING also documents a wholesale opt-out — a before filter at the top of the root API:

class API < Grape::API
  before do
    env['grape.routing_args']&.each_key do |key|
      next if key == :route_info

      value = params[key]
      params[key] = value.b if value.is_a?(String)
      params[key] = value.map(&:b) if value.is_a?(Array)
    end
  end

  # ... mounts and routes
end

env['grape.routing_args'] is populated from params_for alone, so this touches only path params and leaves query and body params alone; :route_info is skipped because it is the Route object sharing that hash. before runs ahead of validation (endpoint.rb:167 vs :178), so declarations see the binary strings too — verified by values: ['café'] going back to 400 on GET /café with the filter installed. Declared in the root API it covers mounted APIs as well, and the Array line matters for splats, where params[:splat] is an Array<String>.

It is framed there as a migration aid rather than a setting, since it reinstates exactly the inconsistency this PR fixes.

Perf

The re-tag is unconditional and in place — Mustermann's Pattern#params builds a fresh Hash of fresh, unfrozen strings on every call and skips its own Match cache, so nothing here is shared or frozen and force_encoding mutates directly. The tagging itself is therefore allocation-free.

Since every capture has to be visited anyway, params_for now does the whole job in one pass, building the result Hash directly instead of allocating the two intermediates that compact and symbolize_keys produced. Net, this branch allocates less than master while doing strictly more work.

Measured against 22d79756, same benchmark both sides.

Route#params_for in isolation — two captures, binary input:

	master	this branch
allocations	9.0 obj/call	8.0 obj/call
ASCII path	607k i/s	566k i/s (−6.8%)
non-ASCII path	206k i/s	202k i/s (−2.2%)

Full request — four route shapes:

route	master	this branch	objects/request
two captures	79–81k i/s	79–81k i/s	35 → 34
no capture	92.7k i/s	93.7k i/s	31 → 30
splat	80.0k i/s	79.8k i/s	33 → 32
non-ASCII	58.1k i/s	57.7k i/s	71 → 70

Allocations drop by exactly one object per request on every shape — an integer count with no variance, so it is the trustworthy signal here. The isolated 6.8% is the force_encoding per capture, but params_for is ~1.65 µs of a ~12.4 µs request, so it works out to ~0.9% of the request, below the noise floor. I ran the request-level comparison interleaved three times to confirm: every difference stayed inside the ±4–6% error bars.

One note for anyone reproducing this. The obvious one-pass form, each_with_object({}), is slower than the chain it replaces — Hash#each_with_object packs every entry into a pair Array before yielding, so it costs 4 objects/call against the chain's 2, and filter_map { … }.to_h is worse still at 6. Building into an explicit Hash with a two-arity each block skips the pair allocation and is the only form that actually wins.

Test plan

• Six new examples in api_spec.rb covering single/multiple/splat/unnamed-splat captures, the values: equality case, and a byte-preservation invariant; verified the behavioural ones fail without the lib/ change.
• Full RSpec suite passes locally (2591 examples, 0 failures).
• RuboCop clean.
• CI green.

🤖 Generated with Claude Code

[github-actions]

Danger Report

No issues found.

View run

[dblock]

See comments below, but LGTM at a high level.

[dblock]

Add some code that shows how to restore old behavior in UPGRADING and this is good to go. I wouldn't add an option.