Diagnose RescueGroups response encoding by adding encoding scheme in log And Repair Mojibake

[patrickZWY]

No description provided.

[patrickZWY]

Work in progress… will see better whether funky word is caused by no encoding scheme and resorting to default Latin-1

[patrickZWY]

temporarily removed debug flag to try logging from real rescue api, currently api is down. try it at another time.

[patrickZWY]

From log, we get:
2026-08-05 02:13:42 [DEBUG] adoption_sources.rescue_groups - RescueGroups response decoding: Content-Type='application/vnd.api+json' requests_encoding=None
2026-08-05 02:13:42 [DEBUG] adoption_sources.rescue_groups - RescueGroups response.json() matches explicit UTF-8 parse: True

and:

                                'from 1:00PM â\x80\x93 6:00PM, Tuesdays by '
                                'appointment only from 1:00PM â\x80\x93 '
                                '6:00PM, closed Mondays & Holidays.  We '

and:

                                'hours: Wednesdays-Sundays from 1:00PM '
                                'â\x80\x93 6:00PM, Tuesdays by appointment '
                                'only from 1:00PM â\x80\x93 6:00PM, closed '

It seems that text is already corrupted when we received it. Do minimal repair instead.

[patrickZWY]

Added general mojibake repair, behavior tests, regression tests, comments for mastodon impl details, sanity check logging post repair. Will need to switch dev setting back to normal when finished. Also more testing of ftfy capabilities (for example its fix and explain may be useful for logging)

[patrickZWY]

As the log shows in descriptionHtml and descriptionText: the text is already corrupted somewhere upstream before requested by us, so the chain of events likely is:
correct chars -> bad encoding -> mojibake -> utf-8 serialization -> decoded as utf-8 -> original mojibake
(-------------------------- upstream ---------------------------) (------------- us ----------------------)

[patrickZWY]

changed to a more conservative way of using mojibake by choosing when and which to repair and not so that stuff like valid à and  text remains unchanged.

[patrickZWY]

under current conservative rule, these will not be repaired: missing byte replaced by a space such as voilà le travail will not become violà le travail. if clean and corrupted text mixed on the same line, will not repair. lossy corruption that cannot be recovered reliably. so repair is for complete, recognizable encoding mistakes, but anything that requires guessing will remain the same.

[patrickZWY]

Things that may be correct but may be over-corrected by ftfy: “hello” because of curly quotes become straight quotes. HTML entities remaining in the text like """. CJK text normalized. typographical stuff like "fl".

In [8]: ftfy.fix_text("✔ no problems")
Out[8]: '✔ no problems'

In [9]: ftfy.fix_text('from 1:00PM â\x80\x93 6:00PM, Tuesdays by ')
Out[9]: 'from 1:00PM – 6:00PM, Tuesdays by '

In [10]: ftfy.fix_text('voilà le travail')
Out[10]: 'voilà le travail'

In [11]: ftfy.fix_text("“hello”")
Out[11]: '"hello"'

In [12]: ftfy.fix_text("&quot")
Out[12]: '&quot'

In [13]: ftfy.fix_text(" ")
Out[13]: '\xa0'

In [14]: ftfy.fix_text(""")
Out[14]: '"'

In [15]: ftfy.fix_text("fl")
Out[15]: 'fl'

In [16]: ftfy.fix_text("Ｈｅｌｌｏ")
Out[16]: 'Hello'

[patrickZWY]

After second thoughts, fix and explain feature can be used more widely as long as it doesn’t touch anything beyond the textual content of the post.