Skip to content

feat: add Portuguese (Brazil) / Portuguese (Portugal) distinction#245

Open
tuliovianna-lab wants to merge 1 commit into
hydropix:mainfrom
tuliovianna-lab:feat/pt-br-pt-pt-upstream
Open

feat: add Portuguese (Brazil) / Portuguese (Portugal) distinction#245
tuliovianna-lab wants to merge 1 commit into
hydropix:mainfrom
tuliovianna-lab:feat/pt-br-pt-pt-upstream

Conversation

@tuliovianna-lab

Copy link
Copy Markdown

Summary

The UI previously offered a single "Portuguese" option that did not distinguish between Brazilian and European Portuguese. This PR adds two separate options throughout the application.

Changes

UI

  • translation_interface.html: 6 dropdowns now show "Portuguese (Brazil)" and "Portuguese (Portugal)" instead of a single "Portuguese"
  • index.js: TTS voice selects (Edge-TTS and Chatterbox) split into pt-BR (FranciscaNeural) and pt-PT (RaquelNeural)
  • form-manager.js: browser language detection now maps pt-BR to "Portuguese (Brazil)" and pt-PT to "Portuguese (Portugal)"

Backend

  • src/utils/lang_normalize.py: new helper that strips regional suffixes for dict lookups ("Portuguese (Brazil)" -> "portuguese") so example texts, pricing ratios, and context optimization are shared across variants
  • prompts.py, examples/helpers.py, placeholder_examples.py: use normalize_lang_key() for example lookups; the full name with region is still passed to the LLM prompt so the model knows which variant to use
  • epub/lang_support.py: new _REGIONAL_NAME_TO_CODE dict maps the two variants to BCP47 tags pt-BR / pt-PT for the EPUB lang/xml:lang attributes (kept separate from LANGUAGE_NAME_TO_CODE to not break the BCP47 primary-subtag conformance test)
  • tts_config.py: added "portuguese (brazil)" and "portuguese (portugal)" voice mappings
  • pricing/estimator.py, context_optimizer.py: use normalize_lang_key() so both variants share the existing "portuguese" ratio entries

Tests

  • test_epub_xhtml_lang_attributes.py: added parametrized cases for "Portuguese (Brazil)" -> pt-BR and "Portuguese (Portugal)" -> pt-PT
  • All existing tests pass

Design rationale

The key design decision: the full name with region (e.g. "Portuguese (Brazil)") goes to the LLM prompt so the model knows which variant to produce, but internal lookups (example texts, pricing ratios, token estimation) normalize to the base "portuguese" key — so both variants share the same tables without duplication. This avoids maintenance burden while still giving the LLM the regional context it needs.

The regional BCP47 codes (pt-BR, pt-PT) are kept in a separate _REGIONAL_NAME_TO_CODE dict so the existing BCP47 conformance test (which validates primary subtags only) continues to pass unchanged.

Previously the UI offered a single "Portuguese" option that did not
distinguish between Brazilian and European Portuguese. This adds two
separate options throughout the application:

UI changes:
- translation_interface.html: 6 dropdowns now show "Portuguese (Brazil)"
  and "Portuguese (Portugal)" instead of a single "Portuguese"
- index.js: TTS voice selects (Edge-TTS and Chatterbox) split into pt-BR
  (FranciscaNeural) and pt-PT (RaquelNeural)
- form-manager.js: browser language detection now maps pt-BR to
  "Portuguese (Brazil)" and pt-PT to "Portuguese (Portugal)"

Backend changes:
- src/utils/lang_normalize.py: new helper that strips regional suffixes
  for dict lookups ("Portuguese (Brazil)" -> "portuguese") so example
  texts, pricing ratios, and context optimization are shared across
  variants
- prompts.py, examples/helpers.py, placeholder_examples.py: use
  normalize_lang_key() for example lookups; the full name with region is
  still passed to the LLM prompt so the model knows which variant to use
- epub/lang_support.py: new _REGIONAL_NAME_TO_CODE dict maps the two
  variants to BCP47 tags pt-BR / pt-PT for the EPUB lang/xml:lang
  attributes (kept separate from LANGUAGE_NAME_TO_CODE to not break the
  BCP47 primary-subtag conformance test)
- tts_config.py: added "portuguese (brazil)" and "portuguese (portugal)"
  voice mappings
- pricing/estimator.py, context_optimizer.py: use normalize_lang_key()
  so both variants share the existing "portuguese" ratio entries

Tests:
- test_epub_xhtml_lang_attributes.py: added parametrized cases for
  "Portuguese (Brazil)" -> pt-BR and "Portuguese (Portugal)" -> pt-PT
- All 881 unit/integration tests pass.
@tuliovianna-lab

Copy link
Copy Markdown
Author

Why this distinction matters

Portuguese is the 6th most spoken language in the world (~260M speakers), split between two major variants that differ significantly in grammar, vocabulary, spelling, and tone:

  • Brazilian Portuguese (pt-BR) — ~215M speakers in Brazil. Uses "você" for informal "you", different verb conjugations, and has distinct slang/idioms.
  • European Portuguese (pt-PT) — ~45M speakers across Portugal and Lusophone Africa. Uses "tu" for informal "you", different syntax, and a more formal register. African variants (Angola, Mozambique, Cape Verde) align closer to pt-PT.

Concrete differences a translator must handle

English Brazilian Portuguese European Portuguese
"You speak" "Você fala" "Tu falas"
"The screen" "A tela" "O ecrã"
"The train" "O trem" "O comboio"
"Cool!" "Legal!" "Fixe!"

Impact on translation quality

A model told to translate to generic "Portuguese" will produce inconsistent output — sometimes Brazilian, sometimes European — which reads as awkward to native speakers of either variant. Brazilian readers notice European constructions immediately (and vice versa), especially in:

  • Novels and fiction (this tool's primary use case): dialogue must sound natural in the target variant, or immersion breaks.
  • TTS narration: pronunciation differs significantly (e.g., final vowels are open in pt-BR but often dropped in pt-PT).
  • EPUB readers: the lang attribute drives hyphenation, dictionary lookup, and font selection — pt-BR and pt-PT give different results.

What this PR does

By passing the full variant name (e.g. "Portuguese (Brazil)") to the LLM prompt, the model receives explicit regional context and produces consistent output. Internal lookups (example texts, pricing, token estimation) normalize to the shared "portuguese" key, so there's no duplication or maintenance burden. The BCP47 tags (pt-BR, pt-PT) flow through to EPUB metadata and TTS voice selection automatically.

This follows the same pattern already used for English (en-US/en-GB) and Spanish (es-MX) in the TTS config, extending it to the translation pipeline itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant