BitArchiveReader extractTo memory slow than extractTo Dir

[Changeyuy]

在使用bit7z库时 遇到一个问题 当我使用BitArchiveReader类进行内存解压时， 在一些7z大文件上要比直接解压到磁盘上慢很多。

When using the bit7z library, I encountered a problem. When I use the BitArchiveReader class for memory decompression, it is much slower on some large 7z files compared to decompressing directly to disk.

try {
using namespace bit7z;
const tstring lib_path = "./7z.so";
Bit7zLibrary lib{lib_path};
BitArchiveReader reader{lib, "bigfile.7z", BitFormat::SevenZip};
vector item_info = reader.items();
for(uint32_t i=0;i<item_info.size();i++) {
if(item_info[i].isDir()) {
continue;
}
std::cout << item_info[i].index() << '\n';
std::cout << item_info[i].nativePath() << std::endl;
std::cout << item_info[i].name() << std::endl;
std::cout << item_info[i].size() << std::endl;
std::cout << std::endl;

  std::vector<unsigned char> buffer;
  reader.extractTo(buffer, i);
}

} catch ( const bit7z::BitException& ex ) {
if(ex.code() == std::errc::operation_not_permitted) {
std::cout << ex.posixCode() << std::endl;
}
std::cout << ex.what() << std::endl;
}

[Changeyuy]

https://github.com/rikyoz/bit7z/issues/261I found a related issue that seems to be a similar problem.

[rikyoz]

Hi!

There are two issues with your code.

First, you are using reader.items() to get a vector of all the items in the archive. Since the archive is large, this is an expensive operation because you're reading the entire archive and storing all the properties of each item in the vector.

The items() method is meant for checking the items of an archive after the reader object's lifecycle ends (e.g., when it exits the scope and it closes the archive).

In your case, I suggest simply using a for-each loop over the archive itself. For example:

for (const auto& item : reader) {
   //...
}

#261 I found a related issue that seems to be a similar problem.

The second problem is that 7z archives are usually created using solid compression, as I mentioned in the related issue.
With this type of archive, extracting individual files by index, as you do in your loop, is usually significantly slower.

There's not much bit7z can do in this case; the operation itself is slow, not an issue of the library's overhead.

I usually recommend performing the extraction as a single operation. In your case, I would use the extractTo method overload that outputs to a std::map of buffers, as it is usually faster than multiple single-file extractions.

There's no utility for printing the metadata of each item while you're extracting them.
If you only need to print the path of the item being extracted, you might use a FileCallback:

reader.setFileCallback([&]( const tstring& path ) {
    std::cout << path << '\n';
});
std::map<tstring, buffer_t> bufferMap; // The key is the path of the item within the archive
reader.extractTo(bufferMap);

In the next v4.1 I'll provide some improvements and utilities/callbacks that should help use cases like yours.

[Changeyuy]

Thank you very much for taking the time to answer this question！

[rikyoz]

You're welcome!