small update to implementation of mutation storing

treerec/implementation.md

@@ -182,13 +182,15 @@ And, these operations have the following properties:
 
 ### Mutation metadata in SLiM 6.0
 
-In SLiM 6.0 (presently the `multitrait` branch of the SLiM repo), metadata about mutations is no longer kept in the tree sequence mutation table.  That was inconvenient because the same mutation can be recorded multiple times, in multiple derived states, due to stacking.  That led to duplicated metadata for the same mutation, recorded at different times and thus potentially different from recording to recording.  To rememdy this, and makes things generally more coherent and efficient, we now record mutation metadata in a top-level metadata binary table.  Each mutation is recorded there only once.  The table is generated at save time, with the final metadata state for each mutation, so it is all up-to-date and correct.
+In SLiM 6.0, metadata about mutations is no longer kept in the tree sequence mutation table.  That was inconvenient because the same mutation can be recorded multiple times, in multiple derived states, due to stacking.  That led to duplicated metadata for the same mutation, recorded at different times and thus potentially different from recording to recording.  To rememdy this, and makes things generally more coherent and efficient, we now record mutation metadata in a binary chunk of top-level metadata.  Each mutation is recorded there only once.  The table is generated at save time, with the final metadata state for each mutation, so it is all up-to-date and correct. So, the tskit mutation table contains information about tskit mutations, the derived state of which may contain more than one SLiM mutation; and information about SLiM mutations is stored in top-level metadata, under "SLiM_mutation_list".
 
-An entry is written out for each segregating mutation, and for each substitution object.  In addition, some mutations need to have their information written out even though they are not segregating/fixed.  These are referred to as "retained by the tree".  The design is that mutations get "retained by the tree" only in cases where they could end up being in ancestral derived states after simplification even though they have no descendants.  That happens in three cases.  One is with stacking policy "l", where a new mutation replaces an existing mutation rather than stacking; in that situation, the ancestral mutation might remain ancestral to an extant clade but not have any descendants.  Two is when the user explicitly removes a mutation from a haplosome in script.  Three is when an individual is remembered with treeSeqRememberIndividuals(); all of its mutations are then retained by the tree.  In those three cases, muts_retained_by_treeseq_ is used to slap a retain on the mutation(s) in question, and they are persisted in the mutation table at save time even if they don't exist in any extant individuals.  (You can search for muts_retained_by_treeseq_ to see all the code in the project that uses this; there's only about 20-25, including comments, it is quite straightforward.)  When a .trees file is read back in, and mutations referenced by a derived state entry that do not get turned into segregating mutations or substitutions get instantiated instead as mutation objects that are "retained by the tree", so that they continue to be persisted across multiple save/load cycles.
+At save time, we record information about all mutations (past and present) whose IDs are referenced in the "derived state" column of the mutation table (and only these mutations). To do this, we need to make sure to maintain any mutation objects that we might need, and then when writing out the tree sequence we simply loop through the derived state column, recording information for any referenced mutations.
 
-For most models I expect that few mutations will get marked in this way in the first place, and so the savings in memory usage seem unlikely to be worth the time spent doing a sweep through all derived mutation lists across the whole tree sequence.  Unless there is really a large number of them, the entries just sit in the table taking up a bit of space and doing no harm.  But I have in fact implemented a cleaning mechanism already.  In Species::SimplifyAllTreeSequences() the code does a pass through all derived states and releases mutations retained by the tree if they are no longer referenced.  It only does this when (muts_retained_by_treeseq_.size() > 10000) though; if the number of retained mutations is small, the code deems it not worth the effort.  10,000 is about 360K of table space, which seems negligible to me; and if a model doesn't get to 10,000 retained mutations, it is probably actually close to zero retained mutations in most cases.  Broadly speaking, either a model's design is such that it generates a lot of retained mutations for some reason, or (the common case) it generates few or none.
+We make sure we have the correct set of mutation objects as follows: we will definitely use all segregating mutation, and all substitutions; these are present anyhow.  In addition, some mutations need to have their information written out even though they are not segregating/fixed.  These are referred to as "retained by the tree".  The design is that mutations get "retained by the tree" only in cases where they could end up being in ancestral derived states after simplification even though they have no descendants.  That happens in three cases.  One is with stacking policy "l", where a new mutation replaces an existing mutation rather than stacking; in that situation, the ancestral mutation might remain ancestral to an extant clade but not have any descendants.  Two is when the user explicitly removes a mutation from a haplosome in script.  Three is when an individual is remembered with treeSeqRememberIndividuals(); all of its mutations are then retained by the tree.  In those three cases, "retained_by_treeseq_" is used to slap a retain on the mutation(s) in question, and they are persisted even if they don't exist in any extant individuals.  (You can search for muts_retained_by_treeseq_ to see all the code in the project that uses this; there's only about 20-25, including comments, it is quite straightforward.)
 
-The other case where mutations get removed from the "retained by the tree" list is if they get turned into a Substitution object. All substitutions are listed in the mutation metadata table, always; so then there is no longer a need for the original mutation to be listed.
+When a .trees file is read back in, and mutations referenced by a derived state entry that do not get turned into segregating mutations or substitutions get instantiated instead as mutation objects that are "retained by the tree", so that they continue to be persisted across multiple save/load cycles. (Note, however, that only mutations referenced by a derived state entry will be instantiated.)
+
+This mechanism retains a certain number of unneeded mutations, which are cleaned out as follows. For most models I expect that few mutations will get marked in this way in the first place, and so the savings in memory usage seem unlikely to be worth the time spent doing a sweep through all derived mutation lists across the whole tree sequence.  Unless there is really a large number of them, the entries just sit in the table taking up a bit of space and doing no harm.  In Species::SimplifyAllTreeSequences() the code does a pass through all derived states and releases mutations retained by the tree if they are no longer referenced.  It only does this when (muts_retained_by_treeseq_.size() > 10000) though; if the number of retained mutations is small, the code deems it not worth the effort.  10,000 is about 360K of table space, which seems negligible to me; and if a model doesn't get to 10,000 retained mutations, it is probably actually close to zero retained mutations in most cases.  Broadly speaking, either a model's design is such that it generates a lot of retained mutations for some reason, or (the common case) it generates few or none. Mutations also get removed from the "retained by the tree" list is if they get turned into a Substitution object. All substitutions are listed in the mutation metadata table, always; so then there is no longer a need for the original mutation to be listed.
 
 
 ## Population table state