[Bug]: Dreyfus purge error

[nickva]

Version

unknown

Describe the problem you're encountering

Purging a dreyfus (clouseau) text index when the min purge infos limit is > 0 and then querying is broken and raises this kind of an error:

Querying search index
{
  "error": "{invalid_start_purge_seq,0,3}",
  "reason": "[{couch_bt_engine,fold_purge_infos,5,\n                  [{file,\"src/couch_bt_engine.erl\"},{line,619}]},\n {dreyfus_index_updater,purge_index,3,\n                        [{file,\"src/dreyfus_index_updater.erl\"},{line,116}]},\n {dreyfus_index_updater,update,2,\n                        [{file,\"src/dreyfus_index_updater.erl\"},{line,52}]}]"
}

Expected Behaviour

Dreyfus/clouseau indexes should handle purges even after they've been compacted and start at a value > 0

Steps to Reproduce

Purge greater than purge infos limit number of docs, then compact, then create a clouseau/dreyfus search index

Your Environment

No response

Additional Context

No response

[nickva]

One of the causes is using this 0 as the starting/ending PurgeSeq in

couchdb/src/dreyfus/src/dreyfus_index_updater.erl

Lines 116 to 118 in 25e1269

	{ok, {ExcludeList, NewPurgeSeq}} = couch_db:fold_purge_infos(
	Db, IdxPurgeSeq, FoldFun, {[], 0}, []
	),

If there are no purges present, the fold will end up with a 0 and both set the new index purge sequence to 0 (which may not be true) and also set the purge checkpoint local doc to 0 as well. Next time the fold will start from 0 and that's will trigger the invalid_start_purge_seq since it looks like index had skipped a bunch of purges.

[nickva]

Another issue is how purge_seq is handled by clouseau/dreyfus indexes.

When we initialize the purge checkpoint in

couchdb/src/dreyfus/src/dreyfus_index.erl

Line 126 in 25e1269

dreyfus_util:maybe_create_local_purge_doc(Db, Pid, Index),

we also set the purge seq in the index:

couchdb/src/dreyfus/src/dreyfus_util.erl

Lines 352 to 354 in 25e1269

	DbPurgeSeq = couch_db:get_purge_seq(Db),
	clouseau_rpc:set_purge_seq(IndexPid, DbPurgeSeq),
	DocContent = dreyfus_util:get_local_purge_doc_body(

That is done with the idea that later on in

couchdb/src/dreyfus/src/dreyfus_index_updater.erl

Lines 90 to 91 in 25e1269

	purge_index(Db, IndexPid, Index) ->
	{ok, IdxPurgeSeq} = clouseau_rpc:get_purge_seq(IndexPid),

we can read that purge_seq from the index and pass that as the initial start purge sequence. However setting a purge_seq on an index then immediately reading it back, we won't get the value we set. Until it commits we'll get the old committed value. That is incorrect behavior as far how purging logic is concerned because of this sequence of steps when the purge sequence is non-0 for a db.

For example is db purge seq is 42, the minimum value may be 12

1. When index is first created we set clouseau's purge seq = 42
2. In purge_index

   couchdb/src/dreyfus/src/dreyfus_index_updater.erl

   Line 91 in 25e1269

   {ok, IdxPurgeSeq} = clouseau_rpc:get_purge_seq(IndexPid),

   we read it right back from the index but instead of 42 we'll get 0
3. Folding purges starting from 0 will fail, because the minimum purge seq is 12, and to the purge checkpoint logic it looks like the index just skipped a bunch of purges and is now out of sync.

Maybe instead we should use the local doc checkpoints to keep track of the index purge seq instead of using the index. We already have to track them in the checkpoints anyway. Or, we have to force the index to properly commit and a set_purge_seq checkpoint is initialized (I am not sure if we a synchronous commit on purge_seq set though...)

[nickva]

A third issue, slightly tangential to the above is dreyfus/clouseau index don't have a the reset functionality on purge seq inconsistency detection as we do for main view indexes. Basically there is no equivalent to:

couchdb/src/couch_index/src/couch_index_updater.erl

Lines 218 to 222 in 25e1269

	error:{invalid_start_purge_seq, StartSeqErr, MinSeqErr} ->
	DbName = couch_db:name(Db),
	ErrMsg = "~p : ~p db:~p resetting index min purge seq: ~p start seq:~p",
	couch_log:error(ErrMsg, [?MODULE, Mod, DbName, MinSeqErr, StartSeqErr]),
	exit({reset, self()})

[nickva]

Pr to fix the first issue #5918

[nickva]

The second issue is fixed in cloudant-labs/clouseau#146

And we decided we won't add the reset functionality in for now.

Closing this issue as fixed.