[Bug]: used space do not freed after docs deleted and pruned

[sergey-safarov]

Version

3.5.0

Describe the problem you're encountering

In my case CouchDB database used as configuration storage.
Here about 1000 configuration docs and some of them actively created, deleted and then pruned.
I found issue where all configuration docs required about 1Mb on filesystem but shard size on more then 500Mb.

To troubleshoot this I have deleted all documents in the database, compacted but still use more then 500 Mb on the filesystem.
Here is database info

[root@host bin]# curl -s http://admin:xxxx@127.0.0.1:5984/test| jq
{
  "instance_start_time": "1768133824",
  "db_name": "test",
  "purge_seq": "4463449-g2wAAAABaANkAB9jb3VjaGRiQGVzcnAtZGIwYS5waC5uZ2E5MTEuY29tbAAAAAJhAG4EAP____9qYgBEG1lq",
  "update_seq": "8971555-g1AAAABueJzLYWBgYMxgTmGQT84vTc5ISXJILS4q0E1JMkjUK8jQy0tPtDQ01EvOz80BKmRKZMhjYfgPBFkZzEkMHU-Vc4Gi7OYGRilGxpaEzcgCABfAIew",
  "sizes": {
    "file": 1322799314,
    "external": 0,
    "active": 508996585
  },
  "props": {},
  "doc_del_count": 0,
  "doc_count": 0,
  "disk_format_version": 8,
  "compact_running": true,
  "cluster": {
    "q": 1,
    "n": 3,
    "w": 2,
    "r": 2
  }
}

Please be aware here about 8 971 555 document changes and 4 463 449 purge docs.
To compact used command

[root@host bin]# curl -s -X POST -H "Content-Type: application/json" http://admin:xxxxx@127.0.0.1:5984/test/_compact| jq
{
  "ok": true
}

On file system empty database use about 650 Mb

[root@host bin]# ls -l /var/lib/couchdb/shards/00000000-ffffffff/test.1768133824.couch
-rw-r--r--. 1 couchdb couchdb 557266 Mar 21 16:38 /var/lib/couchdb/shards/00000000-ffffffff/test.1768133824.couch
[root@host bin]# du -hs /var/lib/couchdb/.shards/00000000-ffffffff/test.1768133824_design
157M	/var/lib/couchdb/.shards/00000000-ffffffff/test.1768133824_design

Shards info

[root@host bin]# curl -s http://admin:xxxxx@127.0.0.1:5984/test/_shards| jq
{
  "shards": {
    "00000000-ffffffff": [
      "couchdb@node0.example.com",
      "couchdb@node1.example.com",
      "couchdb@node2.example.com"
    ]
  }
}

Shard location

[root@node bin]# curl -s http://admin:xxxxx@127.0.0.1:5984/_node/_local/_dbs/test| jq
{
  "_id": "test",
  "_rev": "1-b562eb40b3a9250a1cee5d08d337a567",
  "shard_suffix": [
    46,
    49,
    55,
    54,
    56,
    49,
    51,
    51,
    56,
    50,
    52
  ],
  "changelog": [
    [
      "add",
      "00000000-ffffffff",
      "couchdb@node0.example.com"
    ],
    [
      "add",
      "00000000-ffffffff",
      "couchdb@node1.example.com"
    ],
    [
      "add",
      "00000000-ffffffff",
      "couchdb@node2.example.com"
    ]
  ],
  "by_node": {
    "couchdb@node0.example.com": [
      "00000000-ffffffff"
    ],
    "couchdb@node1.example.com": [
      "00000000-ffffffff"
    ],
    "couchdb@node2.example.com": [
      "00000000-ffffffff"
    ]
  },
  "by_range": {
    "00000000-ffffffff": [
      "couchdb@node0.example.com",
      "couchdb@node1.example.com",
      "couchdb@node2.example.com"
    ]
  },
  "props": {}
}

Cluster status

[root@host bin]# curl -s http://admin:xxxx@127.0.0.1:5984/_membership| jq
{
  "all_nodes": [
    "couchdb@node0.example.com",
    "couchdb@node1.example.com",
    "couchdb@node2.example.com"
  ],
  "cluster_nodes": [
    "couchdb@node0.example.com",
    "couchdb@node1.example.com",
    "couchdb@node2.example.com"
  ]
}

Could you suggest how to troubleshoot issue with user space cleanup when all docs deleted and pruned.

Expected Behaviour

When all documents deleted from database shard files should be compacted and shard file size should be not more then 3Mb.

Steps to Reproduce

It looks like issue related to cluster deployments and many documents created, deleted and then pruned.

Your Environment

Three node cluster with CouchdB 3.5.0

[root@node bin]# curl -s http://admin:xxxx@127.0.0.1:5984| jq
{
  "couchdb": "Welcome",
  "version": "3.5.0",
  "git_sha": "11f0d36",
  "uuid": "3212a9b297b73cfe646f2c45dd7b8049",
  "features": [
    "access-ready",
    "partitioned",
    "pluggable-storage-engines",
    "reshard",
    "scheduler"
  ],
  "vendor": {
    "name": "The Apache Software Foundation"
  }
}

Additional Context

No response

[sergey-safarov]

Closing this.
Database freed while I creating ticket.

[root@host bin]# curl -s http://admin:xxxxx@127.0.0.1:5984/test| jq
{
  "instance_start_time": "1768133824",
  "db_name": "test",
  "purge_seq": "4463449-g2wAAAABaANkAB9jb3VjaGRiQGVzcnAtZGIwYS5waC5uZ2E5MTEuY29tbAAAAAJhAG4EAP____9qYgBEG1lq",
  "update_seq": "8971555-g1AAAABueJzLYWBgYMxgTmGQT84vTc5ISXJILS4q0E1JMkjUK8jQy0tPtDQ01EvOz80BKmRKZMhjYfgPBFkZzEkMHU-Vc4Gi7OYGRilGxpaEzcgCABfAIew",
  "sizes": {
    "file": 557266,
    "external": 0,
    "active": 138096
  },
  "props": {},
  "doc_del_count": 0,
  "doc_count": 0,
  "disk_format_version": 8,
  "compact_running": false,
  "cluster": {
    "q": 1,
    "n": 3,
    "w": 2,
    "r": 2
  }
}

[sergey-safarov]

I just reproduced issue again.
For now I have do not deleted all docs from the database.
Current database status

[user@host bin]$ curl -s 'http://admin:xxxxx@127.0.0.1:5984/test' | jq
{
  "instance_start_time": "1768133824",
  "db_name": "test",
  "purge_seq": "6101027-g2wAAAABaANkAB9jb3VjaGRiQGVzcnAtZGIwYS5waC5uZ2E5MTEuY29tbAAAAAJhAG4EAP____9qYgBdGCNq",
  "update_seq": "12211142-g1AAAABueJzLYWBgYMxgTmGQT84vTc5ISXJILS4q0E1JMkjUK8jQy0tPtDQ01EvOz80BKmRKZMhjYfgPBFkZzEkMu4KP5QJF2c0NjFKMjC0Jm5EFACQiIi8",
  "sizes": {
    "file": 190083335,
    "external": 657156,
    "active": 188159324
  },
  "props": {},
  "doc_del_count": 1584,
  "doc_count": 844,
  "disk_format_version": 8,
  "compact_running": false,
  "cluster": {
    "q": 1,
    "n": 3,
    "w": 2,
    "r": 2
  }
}

On file system database use about 250 Mb

[user@host bin]$ ls -l /var/lib/couchdb/shards/00000000-ffffffff/test.1768133824.couch
-rw-r--r--. 1 couchdb couchdb 190083335 Apr 13 09:06 /var/lib/couchdb/shards/00000000-ffffffff/test.1768133824.couch
[user@host bin]$ du -hs /var/lib/couchdb/.shards/00000000-ffffffff/test.1768133824_design
74M	/var/lib/couchdb/.shards/00000000-ffffffff/test.1768133824_design

If I dump all db content to text file it will take about 800 kb.

[user@host bin]$ curl -s 'http://admin:xxxxx@127.0.0.1:5984/test/_all_docs?include_docs=true&attachments=true' > /tmp/dump.json
[user@host bin]$ ls -l /tmp/dump.json
-rw-rw-r--. 1 user user 829276 Apr 13 10:19 /tmp/dump.json

Could you help troubleshoot why db size on disk more then 190 timer bigger then expected and cannot be compacted.

[rnewson]

please manually invoke compaction for the database, confirm this was successfully completed, and report the sizes again, then we can investigate.

[rnewson]

and manually compact all view groups too

[sergey-safarov]

please manually invoke compaction for the database, confirm this was successfully completed, and report the sizes again, then we can investigate.

Here is results

[user@host bin]# curl -X POST \
>   -H "Content-Type: application/json" \
>   'http://admin:xxxxx@127.0.0.1:5984/test/_compact'
{"ok":true}
[user@host bin]# curl -s 'http://admin:xxxxx@127.0.0.1:5984/test' | jq
{
  "instance_start_time": "1768133824",
  "db_name": "test",
  "purge_seq": "6101027-g2wAAAABaANkAB9jb3VjaGRiQGVzcnAtZGIwYS5waC5uZ2E5MTEuY29tbAAAAAJhAG4EAP____9qYgBdGCNq",
  "update_seq": "12211142-g1AAAABueJzLYWBgYMxgTmGQT84vTc5ISXJILS4q0E1JMkjUK8jQy0tPtDQ01EvOz80BKmRKZMhjYfgPBFkZzEkMu4KP5QJF2c0NjFKMjC0Jm5EFACQiIi8",
  "sizes": {
    "file": 190021895,
    "external": 657156,
    "active": 188157519
  },
  "props": {},
  "doc_del_count": 1584,
  "doc_count": 844,
  "disk_format_version": 8,
  "compact_running": true,
  "cluster": {
    "q": 1,
    "n": 3,
    "w": 2,
    "r": 2
  }
}

and manually compact all view groups too
Here is procedure

# get design docs
views=$(curl -s --noproxy '*' 'http://admin:xxxxx@127.0.0.1:5984/test/_all_docs' |  jq -r '.rows | map(.id) | map(select(startswith("_design"))) | .[]')

# compact views
set -x
for view in ${views}; do
view_stripped=$(echo "${view}" | sed -e 's|_design/||')
curl -X POST \
  -H "Content-Type: application/json" \
  "http://admin:xxxxx@127.0.0.1:5984/test/_compact/${view_stripped}"
done

After compaction database status

[user@host bin]# curl -s http://admin:xxxxx@127.0.0.1:5984/test/| jq
{
  "instance_start_time": "1768133824",
  "db_name": "test",
  "purge_seq": "6101027-g2wAAAABaANkAB9jb3VjaGRiQGVzcnAtZGIwYS5waC5uZ2E5MTEuY29tbAAAAAJhAG4EAP____9qYgBdGCNq",
  "update_seq": "12211142-g1AAAABueJzLYWBgYMxgTmGQT84vTc5ISXJILS4q0E1JMkjUK8jQy0tPtDQ01EvOz80BKmRKZMhjYfgPBFkZzEkMu4KP5QJF2c0NjFKMjC0Jm5EFACQiIi8",
  "sizes": {
    "file": 190021895,
    "external": 657156,
    "active": 188157519
  },
  "props": {},
  "doc_del_count": 1584,
  "doc_count": 844,
  "disk_format_version": 8,
  "compact_running": false,
  "cluster": {
    "q": 1,
    "n": 3,
    "w": 2,
    "r": 2
  }
}

For it looks like after compaction issue still here.

[big-r81]

Are there any running tasks? Does your log file show anything obvious?

[rnewson]

for the database "compact_running": true, means it's still going, you don't get any space back until it finishes. monitor /_active_tasks until all the compaction jobs have finished, check your log file to make sure they completed successfully, and then check your disk space usage.

[rnewson]

sorry, I see your 'after' status. the file size is the same as before (190021895). That pretty much has to mean the compaction task is not completely successfully. check your log.

[sergey-safarov]

Are there any running tasks? Does your log file show anything obvious?

Here is output

[user@host bin# curl 'http://admin:xxxxx@127.0.0.1:5984/_active_tasks' 
[]

When starting compaction using command

[user@host bin]# curl -X POST \
>   -H "Content-Type: application/json" \
>   'http://admin:xxxxx@127.0.0.1:5984/test/_compact'
{"ok":true}

In CouchDB logs I see a lot of messages like

not updated in 7987469 seconds. Expected update interval is at least 86400 seconds

As I understanding default compaction interval configured for 1 day (86400 seconds). And in current example compaction not running 92 days (7987469 seconds). And manual call does not triggers compaction. I have executed "compaction" command several timers.

Full couchdb logs follows.

Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.236874151.6> -------- Starting compaction for db "shards/00000000-ffffffff/test.1768133824" at 12210749
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-17d6f419d28e73c696e0387c1a708051 not updated in 3201950 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-1a17e88c928130d58ac08b679a0d6d5b not updated in 380824 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-1d6c3e5997ef68eb1e3bfb850d37d24c not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-1e82cd03b50adbbc82a804b1cb64c3f2 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-20fef36bdec4b109a7cdc9ea2f3dfd70 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-24d66490155943d5ce5f8ba41c8f2cb6 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-27fcce55829ac4f09ab0f195d43ad529 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-2b65a5cdc3a494fa17cc1f645768def1 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-32266a2c7ee2f84ac7e384dce0086b62 not updated in 303408 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-34d7a15781b2915621611836b7e4ebd1 not updated in 2932127 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-462e71ffa4b5d3b38b811d08ad72396c not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-4fcb65e3071ca7c224d2ad2bfb0c4ba5 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-5ff8c2e39476fa5f8599fec47ae51e28 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-61afbb8b4289ba4cade0e41718010571 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-638c9523ae7c1e00fb22a48fa2eeb9d7 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-66d760f898ee98e42442c7d60470a7fa not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-6ca561964a095a15f145d0aac875a6da not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-78179a23e773efa34428d4ec9699d546 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-8e934f1da05d42acc4f18ae06e229bbc not updated in 380972 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-93cd52debd470f1a3463645e4d9225fc not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-94b03392130cc67dda8b6b56c27f9d7a not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-a413f1fd8747be194bd9a49a35aebcae not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-a6f995f290a08406e7b011d773a348b9 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-b50f2c6516787a57733d498dd67f3af7 not updated in 134479 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-b8d70e49452187e758d54b6387a7a718 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-b9a49bb8b24a083e07c436f17397fb54 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-c5dba9bbf7c3518033bc5d831e6f3eed not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-c994df8441b8f634949d2abcb0247cd6 not updated in 379251 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-d30620bd01fae48b31b30ecc821d76d3 not updated in 3202271 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-d338be563c1cba2f710f697553a19afb not updated in 381425 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-d46e5ec17c9392dd065d5cc4d21c6555 not updated in 381425 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-d7c66c68ec860660a8b01e99e2580ce4 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-dc04c69d4c0410ad3a7d7ab5cf7a1453 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-dccaf47c49f9992e74b0174177c4770a not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-dd51edd8663a1cc90af2f8215cd797be not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-e963fa424dfcd3a1903eb5d9c50ba41a not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-efec41fb4e3d3778431ffd14ed0a0ee6 not updated in 301785 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-f99e863b45527504b0efe0847be824c9 not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-fd55255f67347619d4c1970e433773a5 not updated in 381425 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- couch_db : Purge checkpoint <<"shards/00000000-ffffffff/test.1768133824">> : _local/purge-mrview-fd96aadee2e2f27c57c72ac19945bf7f not updated in 7987469 seconds. Expected update interval is at least 86400 seconds
Apr 14 05:37:03 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248713179.6> -------- The purge sequence for 'shards/00000000-ffffffff/test.1768133824' exceeds configured threshold
Apr 14 05:37:04 host0.example.org couchdb[2141995]: couchdb@host0.example.org <0.248644112.6> fd1d212a34 undefined 2605:84c0:40:3::3b89 undefined GET /_up 200 ok 1

[rnewson]

There's no compaction interval, compaction is triggered whenever needed, unless you've configured smoosh so it's not allowed to run.

So, we at least know that compaction is not finishing, which is why disk space is not being released. Additionally it seems you do a lot of purging and it's implied that you haven't been compacting for quite some time.

For today I suggest you just replicate this database to a new database and make sure the compaction daemon is configured to run (the defaults are fine). Compact that new database manually and you should not see the errors you just showed. Delete the old database once you've switched over to the new database.

[sergey-safarov]

Thanks @rnewson for suggetion.
Yes this works, after first replication database size about 1Mb as expected.
But on production server it required to switch client traffic between database current and replicated.
For this we use HAproxy daemon and some injected rules in HAproxy config file.
But this requires some skills and understanding.
This works for us and we use this as workaround for now.
This ticket I have created to troubleshot root of issue.

There's no compaction interval, compaction is triggered whenever needed, unless you've configured smoosh so it's not allowed to run.

We use this smoosh settings

[smoosh]
state_dir = ./data
compaction_log_level = info

[smoosh.ratio_dbs]
priority = ratio
min_priority = 2.0
from = 08:00
to = 10:00

[smoosh.ratio_views]
priority = ratio
min_priority = 2.0
from = 08:00
to = 10:00

[smoosh.slack_dbs]
priority = slack
min_priority = 536870912
from = 08:00
to = 10:00

[smoosh.slack_views]
priority = slack
min_priority = 536870912
from = 08:00
to = 10:00

We use this settings to define day interval when we prefer compaction run.