Target branch: release-v0.8.x
Current behavior
On a very large database, syncing 363k buffer records, and 728k related record max counters, took a significant amount of time to transfer data to another local instance. On average, a transfer of just 200 buffers took 10 seconds. During each buffer chunk, significant CPU usage was observed. Analyzing the process, it was observed that the code involved queries related to buffer serialization.
Upon investigation, the BufferSerializer creates an N+1 query problem when serializing a list of buffers. This is because the serialization of each buffer accesses rmcb_list on the buffer, creating a query for the counters, meaning 200 buffers creates 201 queries.
Desired behavior
The queries for the RecordMaxCounterBuffer records should be consolidated into on query that can operate in bulk, to minimize the overhead of numerous queries to the DB.
Value add
Speeds up transfer when syncing an instance with a large SQLite DB.
Possible tradeoffs
We've observed this issue in use of DRF within Kolibri and Kolibri Studio. We've developed architectural patterns in those projects that help navigate these issues, but those are not available to Morango at this time. For the time being, a targeted fix is likely the best approach.
AI usage
AI was used to verify the issue that was identified through manual investigation.
Target branch: release-v0.8.x
Current behavior
On a very large database, syncing 363k buffer records, and 728k related record max counters, took a significant amount of time to transfer data to another local instance. On average, a transfer of just 200 buffers took 10 seconds. During each buffer chunk, significant CPU usage was observed. Analyzing the process, it was observed that the code involved queries related to buffer serialization.
Upon investigation, the
BufferSerializercreates an N+1 query problem when serializing a list of buffers. This is because the serialization of each buffer accessesrmcb_liston the buffer, creating a query for the counters, meaning 200 buffers creates 201 queries.Desired behavior
The queries for the
RecordMaxCounterBufferrecords should be consolidated into on query that can operate in bulk, to minimize the overhead of numerous queries to the DB.Value add
Speeds up transfer when syncing an instance with a large SQLite DB.
Possible tradeoffs
We've observed this issue in use of DRF within Kolibri and Kolibri Studio. We've developed architectural patterns in those projects that help navigate these issues, but those are not available to Morango at this time. For the time being, a targeted fix is likely the best approach.
AI usage
AI was used to verify the issue that was identified through manual investigation.