Some api/opensearch optimizations

[mbertrand]

What are the relevant tickets?

Partially addresses https://github.com/mitodl/hq/issues/5101

Description (What does it do?)

• Removes calculation of user-specific field values for user_list_parents and learning_path_parents from learning_resources_search and learning_resources serializers. I kept those fields in the serializers as empty lists to maintain backwards API compatibility. The frontend is making separate API calls to get the required data for these now.
• Adds a custom caching function to ensure that all users hit the same cache for public read-only API endpoints. While testing with the original cache functions I noticed that a cache key was created for each user (both anon and authenticated) for the same url, which kind of defeats the purpose.
• Adds rate limits to more opensearch tasks, values assigned via settings
• Uses dfs_query_then_fetch only when there is a q parameter, with default sorting.
• Omits a has_child opensearch subquery if content_file_score_weight is 0 (but it typically is not).

How can this be tested?

Same cache for all users

main branch

• Run docker compose exec redis redis-cli -n 4 FLUSHDB to clear all view caches
• Run docker compose exec redis redis-cli -n 4 KEYS ":1:views.decorators.cache.cache_page.search.GET*" to verify there are no cache keys
• As an anonymous user, go to http://open.odl.local:8065/api/v1/learning_resources_search/?q=biological
• Check redis cache keys again . You should see something like this:

  1) 2) ":1:views.decorators.cache.cache_page.search.GET.f9a6aadf431833a2289d1030793a64b9.f632a43d08880965d0bbf771c71a15a3.en-us.UTC"
  ``
  

• As an anonymous user in an incognito tab or another browser app, go to the same URL as above.
• Check redis cache keys again. You should see something like this:

  2) ":1:views.decorators.cache.cache_page.search.GET.f9a6aadf431833a2289d1030793a64b9.f632a43d08880965d0bbf771c71a15a3.en-us.UTC"
  3) ":1:views.decorators.cache.cache_page.search.GET.f9a6aadf431833a2289d1030793a64b9.52498d91dc09f347723aa735b400b2c8.en-us.UTC"
  

• The two keys above have the same prefix but different suffixes (one for each anon session). The page is being cached for each anonymous user
• Log in then hit the same api url. Check the redis cache keys again, there should not be any additional ones because the page is not cached for authenticated users.

this branch

• Run docker compose exec redis redis-cli -n 4 FLUSHDB to clear all view caches

• Run docker compose exec redis redis-cli -n 4 KEYS ":1:views.decorators.cache.cache_page.search.GET*" to verify there are no cache keys

• Log in as an authenticated user and go to http://open.odl.local:8065/api/v1/learning_resources_search/?q=biological

• Check the redis cache keys again, you should see something like this:

  ":1:views.decorators.cache.cache_page.search.GET.8db48075a2d9458a571e75697e6e5e2b"
  

• Log in as another user in an incognito tab or other browser, and go to http://open.odl.local:8065/api/v1/learning_resources_search/?q=biological

• Check the redis cache keys again, it should be the same as before, no additional keys.

• You can repeat the above with an anonmyous user, there should still be no additional keys.

• For each logged in user, go to the search page, add some courses to your Favorites (and learning paths if an admin), refresh the page. Each user should see their own favorite courses marked as such.

• As an admin, go to http://open.odl.local:8062/learningpaths. You should still be able to see your learning paths, the list of resources in each, etc.

• As an admin, go to http://open.odl.local:8065/api/v1/articles - you should see both published and unpublished articles (create some of each if you don't have any at http://open.odl.local:8062/articles/new/

• As an anon user or other non-admin user, go to the same url above, you should only see published articles.

Performance Testing

You can use Locust for this. View the README under the load_testing folder to get it up & running. To test with the same user, check "Authenticated User" near the top, and under "Custom Options", copy the value of "sessionid" from your browser cookie.

compared to main (copy the locustfile.py and uwsgi changes there for a fair comparison):

Additional Context

I think SessionMiddleware is responsible for adding the Vary: Cookie header everywhere based on what I read online. Seemed easier to create a custom caching function to ignore that than mess around with the middleware, but I can be persuaded otherwise.

@rhysyngsun would you mind taking a glance at this as a sanity check? Particularly the custom caching function to ensure all users hit the same cache

[shanbady]

works as intended. leaving my approve here. 👍