Add support for auxiliary "tools" column in data processing pipelines

[kryvokhyzha]

Description

Add support for a tools column in the SFT data pipeline, enabling function-calling / tool-use training.

When train_data_columns: ['messages', 'tools'] is set, the pipeline extracts the tools column and forwards it as tools= to every tokenizer.apply_chat_template() call. This lets the chat template render tool definitions into the prompt - required for training on tool-use datasets.

Config

train_data_columns: ['messages', 'tools']
eval_data_columns: ['messages', 'tools']

Why do we need deserialization?

Parquet (Arrow) requires a fixed schema for nested columns. But messages has varying structure - some messages contain tool_calls, others don't; tools has varying parameter schemas per function. Arrow can't represent that as a native nested type.

So I serialize them as JSON strings when writing to Parquet. On the read side, the pipeline gets plain strings like "[{\"role\": \"user\", ...}]" instead of Python lists/dicts. apply_chat_template expects actual Python objects, so we json.loads() them before passing to the template.

Dataset Example

tools

messages

[{"type": "function", "function": {"name": "set_smart_light_color", "description": "Sets the color and brightness of smart lights in the specified room.", "parameters": {"type": "object", "properties": {"room": {"type": "string", "description": "A room with smart lamps."}, "brightness": {"type": "string", "description": "Desired brightness level.", "enum": ["bright", "dim", "off"]}, "color": {"type": "string", "description": "Desired color setting.", "enum": ["warm", "cool", "daylight"]}}, "required": ["room", "brightness", "color"]}}}, {"type": "function", "function": {"name": "sync_lights_with_automation_system", "description": "Syncs smart lights with your home automation system for a specified room.", "parameters": {"type": "object", "properties": {"room": {"type": "string", "description": "A room with smart lamps."}}, "required": ["room"]}}}]

[{"role": "user", "content": "I'm preparing my living room for a cozy movie night and I want to set the right mood with the lighting. Could you please set the smart lights in the living room to emit a warm light with a low brightness level? Also, I need the lights to be synchronized with my home automation system to work seamlessly. To invoke the function, please use the living room as the specified room, set the brightness to 'dim' and the color to 'warm'. Then make sure the lights are well integrated with the automation system in the living room."}, {"role": "assistant", "content": "", "tool_calls": [{"type": "function", "function": {"name": "set_smart_light_color", "arguments": {"room": "living room", "brightness": "dim", "color": "warm"}}}, {"type": "function", "function": {"name": "sync_lights_with_automation_system", "arguments": {"room": "living room"}}}]}]

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.