Remove perseus files from storage calculations

[ozer550]

Summary

Perseus exercise export files are now excluded from user storage calculations. These files are auto-generated during publishing and aren't user-uploaded content, so they shouldn't count against a user's storage quota.

References

closes #3306

Reviewer guidance

• Copy an exercise from the public library into a channel multiple times.
• Publish the channel
• Check the user's storage usage the generated .perseus file should not be counted.

[rtibbles]

Not an immediate concern, but just a heads that when QTI assessments are more broadly available, and we are generating QTI ZIP files, then we may need to filter these too (and it would need to be on the format preset, rather than the file format id, because the format id would be 'zip'!)

[ozer550]

This was the analysis after latest changes:

Unique  (cost=7.20..1007.44 rows=1 width=33)
        (actual time=1.205..1.213 rows=0 loops=1)
  Output: contentcuration_file.checksum

  -> Nested Loop  (cost=7.20..1007.44 rows=1 width=33)
                  (actual time=1.205..1.212 rows=0 loops=1)
        Output: contentcuration_file.checksum
        Inner Unique: true

        -> Merge Anti Join  (cost=7.05..1002.77 rows=1 width=66)
                             (actual time=1.204..1.211 rows=0 loops=1)
              Output: contentcuration_file.checksum,
                      contentcuration_file.contentnode_id
              Merge Cond:
                ((contentcuration_file.checksum)::text =
                 (user_files.checksum)::text)

              -> Sort  (cost=3.53..3.59 rows=25 width=66)
                        (actual time=0.056..0.059 rows=25 loops=1)
                    Output: contentcuration_file.checksum,
                            contentcuration_file.contentnode_id
                    Sort Key: contentcuration_file.checksum
                    Sort Method: quicksort  Memory: 28kB

                    -> Seq Scan on contentcuration_file
                          (cost=0.00..2.94 rows=25 width=66)
                          (actual time=0.010..0.029 rows=25 loops=1)
                          Filter:
                            (file_format_id IS NOT NULL
                             AND file_format_id <> 'perseus'
                             AND uploaded_by_id = 1)
                          Rows Removed by Filter: 38

              -> Unique  (cost=3.53..998.72 rows=12 width=33)
                           (actual time=1.138..1.144 rows=1 loops=1)
                    Output: user_files.checksum

                    -> Subquery Scan on user_files
                          (cost=3.53..998.69 rows=12 width=33)
                          (actual time=1.137..1.143 rows=1 loops=1)
                          Output: user_files.checksum
                          Filter:
                            (alternatives: SubPlan 1 or hashed SubPlan 2)

                          -> Unique  (cost=3.53..3.78 rows=24 width=72)
                                       (actual time=0.067..0.068 rows=1 loops=1)
                                Output:
                                  contentcuration_file_1.checksum,
                                  contentcuration_file_1.contentnode_id,
                                  contentcuration_file_1.file_format_id

                                -> Sort  (cost=3.53..3.59 rows=25 width=72)
                                            (actual time=0.067 rows=1 loops=1)
                                      Output:
                                        contentcuration_file_1.checksum,
                                        contentcuration_file_1.contentnode_id,
                                        contentcuration_file_1.file_format_id
                                      Sort Key:
                                        contentcuration_file_1.checksum,
                                        contentcuration_file_1.contentnode_id,
                                        contentcuration_file_1.file_format_id
                                      Sort Method: quicksort  Memory: 28kB

                                      -> Seq Scan on contentcuration_file
                                            contentcuration_file_1
                                            (cost=0.00..2.94 rows=25 width=72)
                                            (actual time=0.003..0.019
                                             rows=25 loops=1)
                                            Filter:
                                              (file_format_id IS NOT NULL
                                               AND file_format_id <> 'perseus'
                                               AND uploaded_by_id = 1)
                                            Rows Removed by Filter: 38

                          SubPlan 1
                            -> Nested Loop  (cost=33.37..41.44 rows=1 width=0)
                                  (never executed)
                                  ...

                          SubPlan 2
                            -> Hash Join  (cost=33.28..47.16 rows=17 width=32)
                                          (actual time=0.523..1.028
                                           rows=58 loops=1)
                                  Output: u0_1.id
                                  Hash Cond:
                                    (u0_1.tree_id =
                                     contentcuration_contentnode_2.tree_id)

                                  -> Seq Scan on contentcuration_contentnode
                                        u0_1
                                        (cost=0.00..13.42 rows=142 width=37)
                                        (actual time=0.219..0.695
                                         rows=143 loops=1)

                                  -> Hash  (cost=33.26..33.26 rows=2 width=4)
                                            (actual time=0.282..0.284
                                             rows=5 loops=1)
                                        Output:
                                          contentcuration_contentnode_2.tree_id

                                        -> Unique
                                             (cost=33.23..33.24
                                              rows=2 width=4)
                                             (actual time=0.273..0.277
                                              rows=5 loops=1)

                                              -> Sort
                                                   (cost=33.23..33.23
                                                    rows=2 width=4)
                                                   (actual time=0.272..0.274
                                                    rows=5 loops=1)

                                                    -> Nested Loop Left Join
                                                         (cost=4.32..33.22
                                                          rows=2 width=4)
                                                         (actual time=0.239..0.261
                                                          rows=5 loops=1)

                                                          -> Nested Loop
                                                               (cost=4.17..21.61
                                                                rows=2 width=82)
                                                               (actual time=0.226..0.235
                                                                rows=5 loops=1)
                                                               Join Filter:
                                                                 (channel.id =
                                                                  channel_editors.channel_id)
                                                               Rows Removed by Join Filter: 10

                                                               -> Seq Scan on
                                                                    contentcuration_channel
                                                                    (cost=0.00..10.10
                                                                     rows=5 width=164)
                                                                    (actual time=0.007..0.010
                                                                     rows=5 loops=1)
                                                                    Filter: (NOT deleted)

                                                               -> Materialize
                                                                    -> Bitmap Heap Scan on
                                                                         contentcuration_channel_editors
                                                                         (cost=4.17..11.28
                                                                          rows=3 width=82)
                                                                         (actual time=0.206..0.208
                                                                          rows=5 loops=1)
                                                                         Recheck Cond:
                                                                           (user_id = 1)

                                                                         -> Bitmap Index Scan on
                                                                              contentcuration_channel_editors_user_id_446ae41b
                                                                              (cost=0.00..4.17
                                                                               rows=3 width=0)
                                                                              (actual time=0.015
                                                                               rows=5 loops=1)

                                                          -> Index Scan on
                                                               contentcuration_contentnode
                                                               contentcuration_contentnode_2
                                                               (cost=0.14..5.76
                                                                rows=1 width=37)
                                                               (actual time=0.004
                                                                rows=1 loops=5)

        -> Index Scan using
             contentcuration_contentnode_id_2b2d9339_like
             on contentcuration_contentnode
             (cost=0.14..5.76 rows=1 width=33)
             (actual time=0.007..0.007 rows=0 loops=0)

Planning Time: 2.860 ms
Execution Time: 1.085 ms

[bjester]

This was the analysis after latest changes:

That was for the check_channel_space queries?

[bjester]

Overall, this is looking pretty great! Just some small comments

[bjester]

Code looks great, and testing confirms calculations are working, including for a deploying a cheffed channel