Add offline file-size optimisation

[cschwan]

Possible optimisations:

• symmetrisation of the initial states if the hadronic initial states are the same
• better file format, which doesn't save zeros; if the optimisation is offline, we can try several sparse matrix ideas like DOK, LIL, COO, CSR, BSR, ...
• optimise luminosity function:
  • strip empty entries
  • strip numerical zeros, see also Add CMS_ZPT_8TEV dataset pinecards#107
  • Redefine channels such that the number of non-zero subgrids is minimal #199
  • strip entries with factor = 0.0 (why are they there in the first place?)
  • Implement subgrid equality detection #246
• static scale detection; remove interpolation of Q^2 in this case
• Add single- and half- (?) precision float subgrid types #119
• use bincode's varint encoding, part of Release version v1 of the file format #118
• Strip out out empty orders of grids #117
• Remove empty unused x1 and x2 grid values in subgrids #151
• Optimize all subgrids #202

[cschwan]

Pull request #48 implements a more efficient data structure.

[cschwan]

I've tested PR #48 with the complete ATLAS DY 3D grid, before (LagrangeSubgridV1) and after (LagrangeSparseSubgridV1) the optimisation, and both with and without LZ4 compression. Here's the table:

Compression	LagrangeSubgridV1	LagrangeSparseSubgridV1
none	3.1 GB	497 MB
LZ4	377 MB	364 MB

The numbers before compression are basically the memory requirements when loading the grid (for convolutions, etc.). Due to the smaller size also convolutions of the grid with PDFs are faster: from 22 seconds down to 16 seconds, where the LZ4 compression virtually makes no difference.

[cschwan]

To optimise a grid, simply run pineappl optimize input.pineappl optimized.pineappl; no re-generation of the grid is needed.

[cschwan]

Here a comparison against APPLgrid (using the converter from PR #17 and CMS_SINGLETOP_TCH_R_7TEV_T.root):

Setup	File size
APPLgrid	2.1 MB
converted	1.9 MB
converted+optimized	1.5 MB
converted+optimized+compressed	1.4 MB
converted+optimized+symmetrized+compressed	683K

[cschwan]

Commit 1ca0a55 further optimizes the file sizes of grids for initial-state symmetric processes (for instance proton-proton collisions) by making use of the symmetry of the double-sum over the interpolated x1 and x2.

• Up to numerical noise the results are unchanged.
• The size optimization is a real reduction of information, so that both the compressed and uncompressed sizes of the grids are halved (see also table above).
• Convolution time is also twice as fast.
• If FK tables are generated from the optimized grids their size should also be cut in half.

[cschwan]

Commits 098fe5d and a0f32fc further decrease the size of all grids that have a static scale (different static scales in different bins are also optimised). The size improvement is a factor of four by default (the interpolation degree plus one).

This optimisation modifies the numerical value of the convolution, since the PDFs are no longer evaluated at multiple q2 grid points, but instead at the single static scale; however, in general the result should be more accurate, because one interpolation dimension is removed.

[cschwan]

Using ./appl2pine (see also #17) I converted all grids (except two) into PineAPPL grids. The following are my observations:

• in the worst case scenario the PineAPPL grid is 0.3% percent larger than the APPLgrid. This happens 4 times out of 489
• in the best case scenario the PineAPPL grid is 98.5% percent smaller, however in this case the APPLgrid is empty
• the total size of all APPLgrids is 5.8 GB, the converted PineAPPL grids are 3.7 GB large. That's a reduction of 36% in size

[cschwan]

Commit fce09e1 removes empty luminosity entries, which is primarily required for generating smaller FK tables.

[felixhekhorn]

@scarlehoff just came across this one: "strip numerical zeros" - maybe we can increase the priority?