Add a source-level row limit (record_limit / limit pushdown) to bound decoding work when sampling

[lokm01]

Is your feature request related to a problem? Please describe.

When sampling a large EBCDIC dataset (e.g. to profile schema or preview data), there is currently no way to bound the decoding work at the source. df.limit(N) does not help: Cobrix's spark-cobol source is a Spark DataSource V1 relation — DefaultSource is a RelationProvider/SchemaRelationProvider returning a CobolRelation extends BaseRelation with TableScan. Because V1 TableScan has no limit-pushdown hook (and the relation does not implement the V2 SupportsPushDownLimit mixin), a limit(N) is planned as a GlobalLimitExec/LocalLimitExec operator above the scan and is never communicated to the reader.

Decoding is lazy per record (getRowIterator consumed via flatMap/mapPartitions), so limit only stops the iterator being pulled — but every RDD partition Spark schedules is still opened and decoded. Reading a directory is worse: buildScanForFixedLength unions the binaryRecords RDDs of all matched files, so all files in the glob are scheduled and decoded regardless of N.

Concrete impact. We profile mainframe datasets on Spark via Cobrix. On a 20-file / 1.7 GB / 6.2M-row HIDAM dataset, adding df.limit(500_000) for a "sample" gave essentially no speedup — Cobrix still decoded all 20 files (~3.5 min). Our only effective lever was to bypass the glob and hand Cobrix a single representative file (~320k rows, ~10 s). That works for schema discovery, but it is a workaround outside Cobrix and cannot express "first N rows across the dataset".

Describe the solution you'd like

A source-level row cap that stops decoding once the target is reached — ideally either or both of:

1. A read option, e.g. .option("record_limit", N) (and/or max_records), that caps rows read per file split (or globally), so partitions stop decoding early and, where possible, files beyond the cap are not scheduled. Today the only offset controls are file_start_offset / file_end_offset, which are byte offsets, not row limits.
2. (Longer term) Migrating the relation to DataSource V2 and implementing SupportsPushDownLimit, so a plain df.limit(N) pushes down automatically.

Option 1 alone would cover the sampling/profiling use case and looks like a smaller, self-contained change.

Describe alternatives you've considered

• df.limit(N) — no effect on decode (explained above).
• Reading one file from the directory manually — works for schema, but is external to Cobrix and cannot sample across files.
• file_end_offset — byte-based, so you cannot ask for "N records" without knowing the record layout / positioning per file.

Additional context

Verified against master: DefaultSource.scala, CobolRelation.scala (extends BaseRelation with TableScan), scanners/CobolScanners.scala (buildScanForFixedLength unions per-file binaryRecords RDDs), and the README "List of all available options" (no row-cap option today).

Thanks @yruslan — happy to help test a prototype against our HIDAM/fixed-length datasets if useful.