Start deleting loose nodes while packing (and commit pack index) in a safe way

[giovannipizzi]

If the dostore container is repacked, typically all objects are first packed, and then one has to call separately the clean_storage function to remove loose objects. However, if the container is big (say 100GB+), this has a high change to end disk space.

One could provide an option to commit the sqlite DB and clean the storage as soon as a pack file is completed, so the additional required space at any time is max ~4GB (and then use it from AiiDA). Pinging @unkcpz as we discussed this together

[giovannipizzi]

As a note, one has to be careful, however to do operations correctly, so there is no risk that a concurrent read process does not find an object at some point in time (eg, one has to commit the sqlite DB and make sure it's written to disk, before attempting any cleaning). Probably calling clean_storage is safe enough, after a commit

[GeigerJ2]

Note to self, the issues for Mac and Windows should be fixed:

disk-objectstore/disk_objectstore/container.py

Lines 1456 to 1464 in e8df2f3

	# If we are here, things should be guaranteed by SQLite to be written to disk.
	# Then, it would be safe to already do some clean up of loose objects that are now packed,
	# and by doing it here we would do it after each pack.
	# This would mean keeping track of the loose objects added to packs, and removing them.
	# HOWEVER, while this would work fine on Linux, there are concurrency issues both
	# on Mac and on Windows (see issues #37 and #43). Therefore, I do NOT delete them,
	# and deletion is deferred to a manual clean-up operation.
	if callback:
	callback(action='close', value=None)

Add option to clean all loose objects that are already in a packed, expose to user through aiida.

Notes:

• Call clean_storage periodically might be the simplest but not the fastest.
• See if "# Force reload" via session closing is necessary.
• Code checks if it can find the file in packed, if it doesn't find it, checks in loose, then again, if it doesn't find it another time, in packed again.
• Linux will first write a file to RAM, then first fast cache on disk, and then finally to disk. For reading, one of these caches, will be used so it will be very quick. So run command to flush disk, for these tests, to make them fair. In real-world scenario, files are probably big and some time old, so not in the cache anymore.
• In the callback, function inside function, with open in the inner function.
• with open filehandle, define callback which takes the filehandle, functools.partial, fix filehandle, and pass to callback
• functools.partial(f, y=True)

Todos:

• Add benchmark, manually: pack-file of 1mb or 100kb, push 10k file objects (different ones), so many that there are 100 packs -> Then, run packing. This triggers the deletion enough times to see if it has some performance implications. Old code + clean_storage at the end, compared to the new implementation with frequently cleaning.
• Check that files are indeed disappearing.
• Unit test: Add one that packs, pack size to sthg small, approx. 20 packs. Run functionality with new flag on. Bc I know the count of the objects, check how many packs I get. Check that files get deleted one by one, or in groups, to make sure that the new feature is warking -> Check callback, could be there is a function that can count the number of loose objects, then one can use that.
• Test that checks concurrency. Few writers, few readers, one packer. Trying that all the time writing and reading, while packer is working. See the settings of this test, e.g., large pack size. If it's already 4gb, probably reduce that size. Have a run of that test, such that many packs are created (these tests will be very different between MacOS and Windows).
• Test with Packer running, but one process that keeps one loose file open, e.g., via context manager, read the file if it's correct; then, run the functionality -> In Windows, this file won't be deleted, as Windows has stricter locks on files during usage; vs. for MacOS and Linux, file will be deleted; but because the file was previously opened and the pointer is still in use, the file contents are still on disk. Inquire in which pack the file was put, then check if the file is still there. try-except, see if there is still this file
• sig0 hash, see if the hash is what we expect