Add News dataset simulator generation notebook

[Gitanaskhan26]

Summary

Adds news_nytimes_lda.npz to the pgmpy/example_datasets repo. This file contains the pre-computed Latent Dirichlet Allocation (LDA) arrays and sparse word counts that pgmpy.datasets.NewsDataset loads on instantiation to simulate the causal inference benchmark from Johansson et al. (2016).

Related Issue : pgmpy/pgmpy#3522

Provenance

• Source: The raw bag-of-words corpus of the New York Times (docword.nytimes.txt.gz and vocab.nytimes.txt) from the UCI Machine Learning Repository.
• Only the pre-computed topic distributions ($K=50$ and $K=200$), the LDA components (topic-word distributions), and the sparse matrices are kept. Treatment assignments ($t$) and factual/counterfactual outcomes ($y$) are deliberately excluded NewsDataset acts as a simulator and regenerates the response surfaces dynamically based on user-defined parameters ($\kappa$ and $C$).
• Generation Script: News_Dataset_Generation.ipynb (included in this PR), which acts as a clean, reproducible pipeline to download the UCI data, fit the LDA models, select the vocabulary, and package the compressed sparse arrays.

Validated properties

• Shape/Size: The total uncompressed raw corpus is 300,000 documents $\times$ 102,660 unique words. The final packaged .npz file is ~245 MB.
• LDA Models: Contains pre-fitted document-topic distributions for $K=50$ (the baseline used in the original paper) and $K=200$. The $K=200$ model allows NewsDataset to dynamically merge topics (via agglomerative clustering) to fulfill user requests for any $K \le 200$ instantly, avoiding 30-minute training bottlenecks.
• Vocabulary Setup: The baseline vocabulary size in this specific implementation is 2,388 words. This is exactly the union of the top 100 most heavily weighted words across all 50 baseline topics in our specific seed=42 LDA initialization.

Why news_nytimes_lda.npz instead of the Raw NYT Corpus?

Because NewsDataset treats this as a simulator, the covariates are derived from real NYT text. However, running LDA inference on 300,000 documents takes ~30-60 minutes to compute. By hosting the pre-computed topic distributions as an .npz file, users can instantly load the high-dimensional covariates without expensive LDA training, while retaining the flexibility to request custom topic sizes and vocabulary counts on the fly.

What the NPZ arrays actually are

Array Key	Shape	Meaning
topic_distributions_k50	(300000, 50)	The document-topic distribution $z(x)$ for the 50-topic baseline model.
topic_distributions_k200	(300000, 200)	The document-topic distribution for the 200-topic model (used for dynamic merge-down).
lda_components_k50 / _k200	(K, 102660)	The topic-word distributions used for vocabulary selection.
vocab_3477	(2388,)	The string vocabulary for the default baseline setup.
word_counts_3477_*	CSR components	The sparse document-term matrix restricted to the default 2388 vocabulary.
full_vocab & word_counts_full_*	(102660,)	The complete vocabulary and CSR matrix, kept to allow NewsDataset to scale the vocabulary if the user requests it.

Checklist

• File uploaded to pgmpy/example_datasets on HuggingFace
• Confirmed accessible via _get_raw_data("news_nytimes_lda.npz")
• News_Dataset_Generation.ipynb included in this PR for reproducibility
• Byte-identical regeneration confirmed (running the notebook with SEED=42 reproduces this exact .npz file)