Stringapalooza

[StefanKarpinski]

0.5 Major tasks

Round 1

• Base: merge ASCIIString, UTF8String and ByteString into String, replace ASCIIString & UTF8String with String #16058
• Compat: add String, String: alias for new String or old UTF8String Compat.jl#192

Round 2

• Base: replace utf8, bytestring and string with String, replace bytestring with String #16453, deprecate utf8 for String #16469
• Base: replace s = ascii(s) with s = String(s); isascii(s) || error(...), ascii: only support checking String for pure ASCIIness #16396

Round 3

• Base: cleanup conversion mess to and from string types String constructor can segfault #16470, String constructor is bad #16713, restore non-copying String behavior, add unsafe_string*, ... #16731
• Base: provide a way to interact with Windows APIs, via Cwstring deprecate WString and wstring #16975, Base.transcode to replace utf8to16 and utf16to8 #16974
• make package with ASCII, Latin-1, UTF-8, UTF-16 and UTF-32 string types (LegacyStrings).
• Base: remove UTF16String and UTF32String remove UTF-16 and UTF-32 stuff #16590

Cleanup tasks

• figure out LineEdit test change, replace ASCIIString & UTF8String with String #16058 (comment), fix: fix mystery REPL test change introduced by 5de52cf9c9343cfcf50be4c #16198
• delete redundant String inner constructor, replace ASCIIString & UTF8String with String #16058 (comment)
• undelete "redundant" String inner constructor, replace ASCIIString & UTF8String with String #16058 (comment)
• replace jl_is_utf8_string and jl_is_byte_string with jl_is_string, replace ASCIIString & UTF8String with String #16058 (comment)
• figure out code coverage drop, replace ASCIIString & UTF8String with String #16058 (comment)
• use ascii in uv_getaddrinfo to error on non-ASCII domain names, replace ASCIIString & UTF8String with String #16058 (comment)
• use ascii in SuiteSparse to error on non-ASCII inputs,
  replace ASCIIString & UTF8String with String #16058 (comment)
• remove special case printing for String type, 5de52cf#commitcomment-17373792, Fix printing of TypeVar with lower bound. #16221
• make sure tests pass with inlining off, replace ASCIIString & UTF8String with String #16058 (comment)
• add deprecation for readdlm's ignore_invalid_chars option, replace ASCIIString & UTF8String with String #16058 (comment), e95f5f2
• remove documentation for readdlm's ignore_invalid_chars option 969d61b
• NEWS.md entries
• fix signature of connect in docs,
• update doc/manual/interacting-with-julia.rst (with doctests), replace ASCIIString & UTF8String with String #16058 (comment)
• add iconv tests JuliaStrings/LegacyStrings.jl#4

---

0.6 Major tasks

Round 4

• Base: change Char representation (allow lossless string processing of any data)
• Base: remove RepString (moved to LegacyStrings)
• Base: remove RevString (move to package?)
• Base: merge SubString and String (add offset field to String)

Cleanup tasks

• make prevind("ll", 5) and such errors, replace ASCIIString & UTF8String with String #16058 (comment)
  replace ASCIIString & UTF8String with String #16058 (comment)
• improve isspace implementation, replace ASCIIString & UTF8String with String #16058 (comment)
• figure out why removing seemingly redundant convert method breaks bootstrap, replace ASCIIString & UTF8String with String #16058 (comment)
• simplify takebuf API, replace ASCIIString & UTF8String with String #16058 (comment), Simplify takebuf() API #19088
• move docstrings inline

[JeffBezanson]

Great list.

What about windows APIs that use utf-16?

[StefanKarpinski]

What about windows APIs that use utf-16?

Already taken care of: #15033.

Two more potential rounds:

• removing RepString and RevString
• merging SubString and String

Might not get to those until the next release though.

[quinnj]

What about:

• String iteration: making this faster, based on your experiments here in the past
• Decide on String indexing: Restrict indexing into strings to a special ByteIndex or StringIndex type #9297

I definitely think there's a lot to play with around merging Substring and String, but it certainly feels like 0.6 material.

[tkelman]

It was just added, but now that readstring(io) is just String(read(io)) it maybe doesn't even need a separate name forever

[lobingera]

just for my curiosity: Why should this be part of 0.5? The title of the release had something to do with Arrays ... not Strings?

[nalimilan]

+1 for including as much of this plan as possible into 0.5. Breakage better happen soon.

As regards moving ASCIIString, etc. to a package (round 3, bullet 2), see discussion on an implementation for any encoding in StringEncodings.jl here.

As regards changing Char's underlying representation(round 4, bullet 1), another step would also be to introduce AbstractChar and use it in method signatures to allow e.g. the ASCIIString replacement to implement ASCIIChar <: AbstractChar as a UInt8. This would allow people working with ASCII data to actually enjoy higher performance than before.

EDIT: Finally, I'd like to see a discussion regarding the opportunity of ensuring that String only holds valid UTF-8 data or not, and how to handle file paths (special string type or not). But I don't want to derail this already rich thread, so maybe better open a separate issue for that?

[StefanKarpinski]

just for my curiosity: Why should this be part of 0.5? The title of the release had something to do with Arrays ... not Strings?

Because I've been working on this for months and it's ready to go. It won't hold up the release anyway.

[lobingera]

I somehow agree, it will not hold the release of the language. But more syntax changes in 0.5 give some impact in porting packages to 0.5; maybe i'm just wrong that this causes effort ...

[nalimilan]

I somehow agree, it will not hold the release of the language. But more syntax changes in 0.5 give some impact in porting packages to 0.5; maybe i'm just wrong that this causes effort ...

And more syntax changes in 0.6 will cause effort for even more packages. And given the growth rate of the package ecosystem...

[stevengj]

#15033 didn't fully provide a path for external packages to call Windows APIs, without calling internal functions.