RFC 8702: RSA-PSS-SHAKE128/256 and ECDSA-SHAKE128/256

[karel-m]

Implemented RFC 8702 support for:

• RSA-PSS-SHAKE128 / RSA-PSS-SHAKE256
• ECDSA-SHAKE128 / ECDSA-SHAKE256

At first this looked like a small change, but the RSA-PSS part turned out to be less clean than expected.

Added SHAKE128/256 support through:

• sha3_shake128_init / sha3_shake128_process / sha3_shake128_done
• sha3_shake256_init / sha3_shake256_process / sha3_shake256_done

Also added the corresponding descriptors: shake128_desc, shake256_desc (these descriptors have fixed digest lengths set according to RFC 8702 - shake128 outputs 32 bytes, shake256 outputs 64 bytes).

The ECDSA-SHAKE128/256 implementation was fairly straightforward.

The RSA-PSS-SHAKE128/256 implementation required a few workarounds because SHAKE is also used directly for MGF [not like MGF1(shakeNNN)]. Therefore I added a new direct_shake_mgf flag to struct ltc_rsa_parameters and a few IFs in the code.

The result works but the implementation is not as clean as I would have liked.

Checklist

• documentation is added or updated
• tests are added or updated

[sjaeckel]

Thanks a lot! Bascially LGTM, besides that one remark.

[sjaeckel]

I don't like the fact that it's now ambiguous and one could also do mgf1_hash_idx = find_hash("shakeXXX"); and the result of the operation would be different. I believe it'd be better if this were resolved internally, even if it makes the implementation more complicated than via this approach.

[karel-m]

I agree that the whole logic around mgf1_hash_idx and direct_shake_mgf is hard to follow and full of unexpected surprises.

I expected this PR to be a simple/easy addition. As I mentioned, I am not satisfied with the result either.

[karel-m]

UPDATE: removed direct_shake_mgf, all "magic" now takes place inside ltc_pkcs_1_mgf1(..) - which is now slightly misleadingly named as it is not always MGF1 (but it is only an internal function, not part of the public API)