Skip to content

Improve export feature#9

Merged
juliobguedes merged 7 commits intomasterfrom
julio/improve-export
Sep 23, 2025
Merged

Improve export feature#9
juliobguedes merged 7 commits intomasterfrom
julio/improve-export

Conversation

@juliobguedes
Copy link
Contributor

What is the purpose of this pull request?

To improve the export feature, allowing accounts with a huge number of redirects to work (e.g. Americanas)

What problem is this solving?

Currently, Americanas has over 7M registered redirects. However, this plugin stored the redirects in memory, to only save them to a file in the end, causing OOM issues that can kill the OS. This PR changes the process to enqueue the paginated requests and write them to a file concurrently to requesting next pages, which allows a better memory management and avoids OOM issues.

Types of changes

  • Refactor (non-breaking change that only makes the code better)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Requires change to documentation, which has been updated accordingly.

Chores checklist

  • Update CHANGELOG.md

Copy link
Contributor

@silvadenisaraujo silvadenisaraujo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Working with async writes works well and in my perspective solves the memory issue that we have today for large exports, I added comments if you want to go beyond and we could add a mechanism to handle retries and throttling efficiently.

Copy link

@filafb filafb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just a comment about handling error in a more clear way, but not a blocker.

Copy link

@filafb filafb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just a comment about error handling, but not a blocker at all.

@juliobguedes juliobguedes merged commit 6e3ee84 into master Sep 23, 2025
5 checks passed
@juliobguedes juliobguedes deleted the julio/improve-export branch September 23, 2025 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants