Invoke-DbaDbDecryptObject - Add NoDAC to decrypt without a dedicated admin connection

[howarthcd]

(do Invoke-DbaDbDecryptObject)

Adds -NoDAC, which reads the encrypted definition straight from the raw data pages with DBCC PAGE instead of opening a dedicated admin connection and altering each object inside a rolled back transaction. Nothing is written to the database on this path. Omitting the switch keeps the original behaviour.

The reader lives in four new private functions. Get-EncryptedObjectImageValue is the engine, seeking the sysobjvalues clustered index and falling back to a page scan; ConvertFrom-DbccPageDump, ConvertFrom-EncryptedObjectChunk and Get-EncryptedObjectKeystream are split out so they can be unit tested without an instance.

Also fixed while here:

• A dedicated admin connection this command opens is now closed even when the run fails. The instance loop body is wrapped in try/finally, because an instance allows only one and a leaked session blocked every later run.
• Encrypted INSTEAD OF triggers on views are now found, and can only be decrypted with -NoDAC. The default method derives a known plaintext by rewriting the trigger as AFTER INSERT, which a view rejects.
• Multi database runs no longer carry objects between databases.
• Trigger discovery no longer costs one query per table, and IsEncrypted is added to the SMO init fields, which takes the test suite from 490s to 92s.
• -EncodingType warns when bound with -NoDAC, because it is ignored there.

Type of Change

• Bug fix (non-breaking change)
• New feature (non-breaking change, adds functionality)
• Breaking change (affects multiple commands or functionality, fixes # )
• Ran manual Pester test and has passed (Invoke-ManualPester -Path <command> -ScriptAnalyzer -Compliance)
• Adding code coverage to existing functionality
• Pester test is included
• If new file reference added for test, has is been added to github.com/dataplat/appveyor-lab ?
• Unit test is included
• Documentation
• Build system

Purpose

The existing command can only reach an encrypted definition through a dedicated admin connection, and it obtains the known plaintext it needs by altering every object inside a transaction that is rolled back. That rules the command out where a DAC is unavailable or where writing to the database, even transiently, is unacceptable, and an instance allows only one DAC at a time.

It also cannot decrypt an encrypted INSTEAD OF trigger defined on a view at all, because the known plaintext it builds rewrites the object as an AFTER trigger and a view rejects that.

Approach

-NoDAC derives the RC4 key from public metadata rather than obtaining a known plaintext, so it needs no dedicated admin connection and writes nothing. The scheme is set out under Learning below.

The ciphertext lives in sys.sysobjvalues, which is DAC-only through T-SQL, so the raw pages are read with DBCC PAGE ... WITH TABLERESULTS and the family GUID with DBCC DBINFO WITH TABLERESULTS. Both need sysadmin, checked up front so the failure is a clear message rather than a permission error midway through reading pages.

Rows are found by seeking the sysobjvalues clustered index, about five page reads whatever the size of the database, with a full page scan as the fallback and as the test oracle.

Comment-based help was updated throughout, including a note that a view trigger requires -NoDAC.

Commands to test

The help examples cover the normal paths. Beyond those:

# read the definitions without a DAC and without writing anything
Invoke-DbaDbDecryptObject -SqlInstance sql01 -Database db1 -NoDAC

# the case the default method cannot do at all
Invoke-DbaDbDecryptObject -SqlInstance sql01 -Database db1 -ObjectName MyViewTrigger -NoDAC

# the default method still behaves exactly as before
Invoke-DbaDbDecryptObject -SqlInstance sql01 -Database db1 -ObjectName MyProc

Learning

The part worth writing down is the obfuscation scheme itself. WITH ENCRYPTION is widely described as "not really encryption", but the actual construction does not appear to be written up anywhere, so it was reverse engineered for this change and is documented here in case it is useful to anyone else.

SQL Server stores the module text in sys.sysobjvalues.imageval, keyed on the object id with valclass = 1. The bytes are the UCS-2 (UTF-16LE) source text XORed with an RC4 keystream. There is no secret: the RC4 key is a SHA1 over 22 bytes of metadata that any sysadmin can already read.

seed(22)  = familyGuid(16) + objectId(4, little endian) + colId(2, little endian)
key       = SHA1(seed)
keystream = RC4(key)
plaintext = ciphertext XOR keystream          then decode as UTF-16LE

Four details are load bearing, and each of them fails in a way that is quiet rather than obvious:

• The GUID byte order is the .NET System.Guid layout, where the first three fields are little endian, not the order the GUID prints in. Using the string order produces a valid looking key and complete garbage. The value is dbi_familyGUID from DBCC DBINFO, and it is a property of the database family rather than of the object.
• colId is an input to the key, so a definition that spans more than one sysobjvalues row needs a separate keystream per row. Deriving one keystream for the whole object leaves the first chunk perfectly readable and everything after it mojibake, which reads like an encoding bug rather than a key bug. The chunks also have to be concatenated in colId order rather than in the order the rows were read.
• Because the key includes the object id, two objects with identical source text produce completely different ciphertext. That rules out any approach based on recognising repeated ciphertext.
• RC4 is a stream cipher, so the keystream depends only on the key and not on the data. Exactly as many bytes are generated as there is ciphertext, and the ciphertext length must be even, because UCS-2 is two bytes per character. An odd length silently decodes with the trailing byte dropped, which produces a plausible looking definition that is subtly truncated, so it is refused instead.

This is also why the existing method works at all. It never derives the key: it alters the object to a placeholder of exactly the same length inside a transaction that is rolled back, which yields a known plaintext and its matching ciphertext, and XORing the three values together recovers the original. That is a clever way around not knowing the key, but it costs a DAC, a write, and it cannot be applied to an object whose definition it is unable to legally rewrite, which is exactly the INSTEAD OF trigger on a view case. Deriving the key directly removes all three constraints.