listenbrainz: add option to import listens from ListenBrainz data export zip#6606
Open
Maxr1998 wants to merge 5 commits intobeetbox:masterfrom
Open
listenbrainz: add option to import listens from ListenBrainz data export zip#6606Maxr1998 wants to merge 5 commits intobeetbox:masterfrom
Maxr1998 wants to merge 5 commits intobeetbox:masterfrom
Conversation
|
Thank you for the PR! The changelog has not been updated, so here is a friendly reminder to check if you need to add an entry. |
Contributor
There was a problem hiding this comment.
Pull request overview
grug see PR try make lbimport faster and nicer to ListenBrainz server. it add way to read listens from ListenBrainz export zip, then update play counts in beets.
Changes:
- add
lbimport --export-file PATHto read listens from ListenBrainz data export zip instead of API - add zip/jsonl import helper and wire it into
_lbupdate - tweak MBID pick logic to handle
mbid_mapping: nulland prefer explicit MBID when present
Comment on lines
+152
to
+170
| def import_listenbrainz_data_export(self, log, export_file): | ||
| """Import ListenBrainz data from a .zip file.""" | ||
| export_file = syspath(normpath(export_file)) | ||
|
|
||
| all_listens = [] | ||
|
|
||
| with zipfile.ZipFile(export_file, "r") as zip_file: | ||
| for file_name in zip_file.namelist(): | ||
| if file_name.startswith("listens/") and file_name.endswith( | ||
| ".jsonl" | ||
| ): | ||
| log.info("Reading listens from {}", file_name) | ||
| with zip_file.open(file_name) as file: | ||
| data = file.read() | ||
| listens = [ | ||
| json.loads(line) for line in data.splitlines() | ||
| ] | ||
| all_listens.extend(listens) | ||
| return all_listens |
Comment on lines
+50
to
+56
| lbupdate_cmd.parser.add_option( | ||
| "--export-file", | ||
| dest="export_file", | ||
| metavar="PATH", | ||
| default=None, | ||
| help="path to a ListenBrainz data export .zip file (instead of fetching from the API)", | ||
| ) |
| """Update play counts from ListenBrainz listening history.""" | ||
| if export_file is not None: | ||
| log.info("Importing ListenBrainz data from {}...", export_file) | ||
| listens = self.import_listenbrainz_data_export(log, export_file) |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6606 +/- ##
==========================================
- Coverage 72.07% 71.99% -0.08%
==========================================
Files 159 159
Lines 20633 20664 +31
Branches 3273 3279 +6
==========================================
+ Hits 14871 14878 +7
- Misses 5053 5076 +23
- Partials 709 710 +1
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Since importing listens from the API is rather slow and puts unnecessary load on the servers, importing a dump exported from https://listenbrainz.org/settings/export/ should also be possible. This change introduces the
export-fileparameter to thelbimportcommand, which takes all listens from the zipfile to update the play counts.I also fixed an issue where
mbid_mappingcould beNoneif explicitly set tonullin the JSON (which happens for these exports sometimes), and changed the recording mbid logic to prefer one explicitly supplied by the player in theadditional_data, instead of relying on the (sometimes inaccurate) mapper.Since the
--maxparameter isn't documented either, I'm unsure whether I should add the parameter to the plugin docs. In theory, the--helpparameter already tells you enough.I'm also not sure if the
--maxparameter should apply to exported files, especially since the order of zip file entries isn't deterministic.To Do
docs/to describe it.)docs/changelog.rstto the bottom of one of the lists near the top of the document.)