Simplify byte_pair_merge

[hauntsaninja]

Based on suggestion in #239 (specifically 8f5dd7d)

Like that commit, this:

• Does the init in a single loop and saves a loop if there are no merges
• Simplifies get_rank and no longer uses it in init (so you don't need multiple skip values)

Unlike that commit:

• We drop optimisations enabled by ignoring single tokens. These didn't show any benefit on benchmarks for me (this makes sense given typical piece sizes, but let me know if that's unexpected!). Given this, I opted for the simpler version.
• I preserve some of the comments from the original that I think are still useful

Let me know what you think! Once we figure this one out, we'll look at the linearithmic fix (I have some thoughts there, still doing some benchmarking).

Co-authored-by: @paplorinc

[l0rinc]

did you see any effect of the inlining here?
I didn't, and even the linter complained, this being a closure inheriting some paramters

[hauntsaninja]

Good question, I dimly remember it being useful in #31 (but it was also used in an additional place then). I can double check :-) Which linter?

[l0rinc]

We drop optimisations enabled by ignoring single tokens.

the parts.len() > 3 means that once we're down to 2 tokens, we don't need more iterations, we don't have to try to merge it into a single token since we've already filtered those out - but that won't show up in the benchmarks, since that's basically constant time, so I agree, the code is simpler this way :)

Thanks for adding the comments back, please see my inline comments.
If you can, please add me as a coauthor here.

Thanks!

[hauntsaninja]

Thank you for the original change and follow-up review! I've marked you as co-author on the commit :-)