Skip to content

feat: add use_representation_model option to hierarchical_topics() for parent node labeling#2513

Open
pidefrem wants to merge 1 commit into
MaartenGr:masterfrom
pidefrem:pr09-hierarchy-labeling
Open

feat: add use_representation_model option to hierarchical_topics() for parent node labeling#2513
pidefrem wants to merge 1 commit into
MaartenGr:masterfrom
pidefrem:pr09-hierarchy-labeling

Conversation

@pidefrem

@pidefrem pidefrem commented Jul 8, 2026

Copy link
Copy Markdown

What does this PR do?

feat: add use_representation_model option to hierarchical_topics() for parent node labeling

hierarchical_topics() builds parent topic names by concatenating the top 5 c-TF-IDF keywords (e.g., "wear_safety_PPE_worker_work"), while leaf topics get rich labels from the representation model (e.g., "PPE Non-Compliance Incidents" via an LLM). This creates a jarring inconsistency in visualize_hierarchy().

Root cause: _extract_words_per_topic is called with calculate_aspects=False for parent nodes.

Changes:

Add use_representation_model: bool = False parameter to hierarchical_topics(). When enabled, a post-processing step runs the full representation pipeline on all parent topics in a single batch call — not inline during the merge loop (which would make N-1 separate LLM calls).

# Before: parent nodes show "wear_safety_PPE_worker_work"
hierarchy = topic_model.hierarchical_topics(docs)

# After: parent nodes get proper labels from the representation model
hierarchy = topic_model.hierarchical_topics(docs, use_representation_model=True)

Default is False — existing behavior unchanged. The batch approach keeps LLM cost bounded.

Fixes #2503

Note: Cleaner if #2497 (shared helpers refactoring) lands first, but works standalone.

Before submitting

  • This PR fixes a typo or improves the docs (if yes, ignore all other checks!).
  • Did you read the contributor guideline?
  • Was this discussed/approved via a Github issue? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes (if applicable)?
  • Did you write any new necessary tests?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hierarchical_topics() parent nodes show raw c-TF-IDF keywords instead of representation model labels

2 participants