-
Notifications
You must be signed in to change notification settings - Fork 11
feat: create sql generator file for data export #641
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #641 +/- ##
==========================================
- Coverage 93.79% 93.36% -0.44%
==========================================
Files 1291 1292 +1
Lines 46878 47119 +241
Branches 1517 1567 +50
==========================================
+ Hits 43971 43992 +21
- Misses 2598 2818 +220
Partials 309 309
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
The data export is going to need some SQL at the end of the day, as we can't just do a sql dump since we're doing specific models + data. You have to manually create that dump file by specifying your requirements, including which models you want to export data of, which fields within those models to exclude, etc.
https://linear.app/getsentry/issue/CCMRG-1994/create-sql-generator-file
Legal Boilerplate
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. In 2022 this entity acquired Codecov and as result Sentry is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.
Note
Introduces a new
sql_generator.pyto produce additive, batched PostgreSQLINSERT ... ON CONFLICTstatements for owner-scoped data exports.ExportContextprefetches hierarchy IDs and applies ownership/date filtering (with full exports for select models and hierarchy-based filtering for others)generate_upsert_sqlbuilds batched UPSERTs per model usingBATCH_SIZE, honoring composite conflict keys (e.g.,core.Branch,timeseries.Measurement)serialize_valuehandles enums, JSON, UUIDs, timestamps, bytea, and float/Decimal edge cases (NaN/Infinity)generate_full_exportwrites a complete SQL script with header, per-model batches, andBEGIN/COMMITgenerate_sequence_resetsappends sequence reset statements for auto-increment PKs;generate_timescale_exporttargetsTIMESCALE_MODELSWritten by Cursor Bugbot for commit bdd8028. This will update automatically on new commits. Configure here.