Skip to content

Performance/parallelise search index generation#675

Open
d7919 wants to merge 9 commits intoFortran-FOSS-Programmers:masterfrom
d7919:performance/parallelise_search_index_generation
Open

Performance/parallelise search index generation#675
d7919 wants to merge 9 commits intoFortran-FOSS-Programmers:masterfrom
d7919:performance/parallelise_search_index_generation

Conversation

@d7919
Copy link
Copy Markdown
Contributor

@d7919 d7919 commented Nov 13, 2024

Builds on #674 to reduce conflicts although the changes should be effectively orthogonal.

Uses process based parallelism to accelerate the search index generation. Tests of thread based parallelism didn't seem to show much gain but it should be easy to switch from process to thread here.

@d7919
Copy link
Copy Markdown
Contributor Author

d7919 commented Nov 13, 2024

We may also want to consider adding lxml as a dependency -- this can be installed from pypi and seems to give a decent speedup to the beautiful soup parsing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants