Apparently this check fails
if (_deflatedStream.CanSeek)
_deflatedStream.Seek(6, SeekOrigin.Current);
Even though the stream claims it supports seek, it actually does not, resulting in NotImplementedException being thrown.
I suspect it has something to do with the different implementation of streams in background agents, since the streams in the main assembly work as expected.
As a workaround, I removed CanSeek check and always do 6 byte reads and it seems to work even for background agents.