Skip to content

feat: add top-down (divisive) strategy for hierarchical_topics()#2512

Open
pidefrem wants to merge 1 commit into
MaartenGr:masterfrom
pidefrem:pr14-divisive-hierarchy
Open

feat: add top-down (divisive) strategy for hierarchical_topics()#2512
pidefrem wants to merge 1 commit into
MaartenGr:masterfrom
pidefrem:pr14-divisive-hierarchy

Conversation

@pidefrem

@pidefrem pidefrem commented Jul 8, 2026

Copy link
Copy Markdown

What does this PR do?

feat: add top-down (divisive) strategy for hierarchical_topics()

hierarchical_topics() builds hierarchies exclusively bottom-up using scipy's agglomerative linkage. While this works for small topic counts, merge quality degrades at higher levels and it has a known bug when 3+ topics have identical c-TF-IDF distances (#1907).

Changes:

Add a strategy parameter to hierarchical_topics():

# Current behavior (default)
hierarchy = topic_model.hierarchical_topics(docs, strategy="agglomerative")

# New: top-down recursive splitting
hierarchy = topic_model.hierarchical_topics(docs, strategy="divisive")

The divisive path recursively splits topics using c-TF-IDF weighted NMF decomposition, building a tree where each parent-child relationship reflects a meaningful topic subdivision. Each split is locally optimal, avoiding the degradation of late merges in agglomerative linkage.

Default is "agglomerative" — existing behavior unchanged.

Fixes #2508

Coordination note: this adds a new strategy option rather than changing the default, so existing hierarchies are untouched. It introduces a top-down NMF code path, which is a meaningful addition to maintain — if you'd like to discuss scope (e.g. land it as experimental first) before reviewing, I'm happy to. It also pairs naturally with the parent-node labeling in #2503.

Before submitting

  • This PR fixes a typo or improves the docs (if yes, ignore all other checks!).
  • Did you read the contributor guideline?
  • Was this discussed/approved via a Github issue? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes (if applicable)?
  • Did you write any new necessary tests?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add top-down (divisive) strategy for hierarchical_topics()

2 participants