Wycheproof failures

[karel-m]

In Project Wycheproof there are currently 340 JSON files with test vectors.

I have:

• used current https://github.com/C2SP/wycheproof
• executed 65883 vectors (with libtomcrypt develop branch)
• skipped 78428 vectors (mostly the given algo not implemented in libtomcrypt)

From the 65883 executed tests, we have 308 unique failed tcIds (0.5%):

  JSON file                            Fail/Total
  ------------------------------------ ------------
  aes_cbc_pkcs5_test.json                 3/216
  camellia_cbc_pkcs5_test.json            3/216
  ec_prime_order_curves_test.json         1/26
  ecdh_brainpoolP224r1_test.json          5/787
  ecdh_brainpoolP256r1_test.json          5/804
  ecdh_brainpoolP320r1_test.json          3/771
  ecdh_brainpoolP384r1_test.json          5/936
  ecdh_brainpoolP512r1_test.json          5/794
  ecdh_secp224r1_pem_test.json            3/714
  ecdh_secp224r1_test.json                3/714
  ecdh_secp256k1_test.json                3/752
  ecdh_secp256r1_ecpoint_test.json        9/355
  ecdh_secp256r1_pem_test.json           12/612
  ecdh_secp256r1_test.json               12/612
  ecdh_secp256r1_webcrypto_test.json      9/353
  ecdh_secp384r1_ecpoint_test.json       13/790
  ecdh_secp384r1_pem_test.json           16/1047
  ecdh_secp384r1_test.json               16/1047
  ecdh_secp384r1_webcrypto_test.json     13/794
  ecdh_secp521r1_ecpoint_test.json       12/661
  ecdh_secp521r1_pem_test.json           15/916
  ecdh_secp521r1_test.json               15/916
  ecdh_secp521r1_webcrypto_test.json     12/655
  ed448_test.json                         3/86
  x448_asn_test.json                     28/529
  x448_jwk_test.json                     28/523
  x448_pem_test.json                     28/510
  x448_test.json                         28/510

Do not take the numbers as exact; they may vary slightly, as I may have done some simplifications.

For now just this information, I will try to submit separate PRs to fix some of them.

[sjaeckel]

Would it maybe make sense to create a single PR with all the fixes as separate commits and link it to this issue?

[karel-m]

not sure, the needed fixes are very different

[karel-m]

With these fixes:

• wycheproof - fix failing ASN.1 BIT STRING related test #750
• Ed448/X448 failing wycheproof tests #749
• ECC failing wycheproof tests #748
• padding_depad - reject zero-length input (except for LTC_PAD_ZERO) #747
• add FRP256v1 - French governmental ECC curve #746

I am able to run wycheproof tests with these results:

testcases total=144311
- unsupported=77789 (not run, skipped)
- run=66522
- failed = 0 (which is great)
- acceptable.rejected=3159 (this is good)
- acceptable.tolerated=2163 (this is not that good but tolerated)

Just FYI skipped tests are:

# wycheproof2 unsupported:    475  unsupported AEAD algorithm AEGIS128
# wycheproof2 unsupported:    479  unsupported AEAD algorithm AEGIS128L
# wycheproof2 unsupported:    472  unsupported AEAD algorithm AEGIS256
# wycheproof2 unsupported:    202  unsupported AEAD algorithm AES-GCM-SIV
# wycheproof2 unsupported:    552  unsupported AEAD algorithm ARIA-CCM
# wycheproof2 unsupported:    311  unsupported AEAD algorithm ARIA-GCM
# wycheproof2 unsupported:    192  unsupported AEAD algorithm ASCON128
# wycheproof2 unsupported:    192  unsupported AEAD algorithm ASCON128A
# wycheproof2 unsupported:    192  unsupported AEAD algorithm ASCON80PQ
# wycheproof2 unsupported:    916  unsupported AEAD algorithm MORUS1280
# wycheproof2 unsupported:    458  unsupported AEAD algorithm MORUS640
# wycheproof2 unsupported:    267  unsupported ECDH curve sect283k1
# wycheproof2 unsupported:    260  unsupported ECDH curve sect283r1
# wycheproof2 unsupported:    265  unsupported ECDH curve sect409k1
# wycheproof2 unsupported:    258  unsupported ECDH curve sect409r1
# wycheproof2 unsupported:    267  unsupported ECDH curve sect571k1
# wycheproof2 unsupported:    257  unsupported ECDH curve sect571r1
# wycheproof2 unsupported:   2189  unsupported ECDSA hash SHAKE128
# wycheproof2 unsupported:   2562  unsupported ECDSA hash SHAKE256
# wycheproof2 unsupported:    123  unsupported IND-CPA algorithm AES-XTS
# wycheproof2 unsupported:    216  unsupported IND-CPA algorithm ARIA-CBC-PKCS5
# wycheproof2 unsupported:    311  unsupported MAC algorithm ARIA-CMAC
# wycheproof2 unsupported:    174  unsupported MAC algorithm HMACSM3
# wycheproof2 unsupported:    174  unsupported MAC algorithm KMAC128
# wycheproof2 unsupported:    261  unsupported MAC algorithm KMAC256
# wycheproof2 unsupported:     40  unsupported MAC algorithm SipHash-1-3
# wycheproof2 unsupported:     40  unsupported MAC algorithm SipHash-2-4
# wycheproof2 unsupported:     40  unsupported MAC algorithm SipHash-4-8
# wycheproof2 unsupported:     40  unsupported MAC algorithm SipHashX-2-4
# wycheproof2 unsupported:     40  unsupported MAC algorithm SipHashX-4-8
# wycheproof2 unsupported:   1528  unsupported MAC-with-IV algorithm VMAC-AES
# wycheproof2 unsupported:    780  unsupported RSA PSS hash
# wycheproof2 unsupported:    110  unsupported RSA private key
# wycheproof2 unsupported:   1091  unsupported RSA public key
# wycheproof2 unsupported:     19  unsupported schema bls_aggregate_verify_schema.json
# wycheproof2 unsupported:     34  unsupported schema bls_hash_to_g2_schema.json
# wycheproof2 unsupported:    114  unsupported schema bls_sig_verify_schema.json
# wycheproof2 unsupported:  31567  unsupported schema fpe_list_test_schema.json
# wycheproof2 unsupported:  25679  unsupported schema fpe_str_test_schema.json
# wycheproof2 unsupported:     83  unsupported schema json_web_crypto_schema_v1.json
# wycheproof2 unsupported:    139  unsupported schema json_web_encryption_schema_v1.json
# wycheproof2 unsupported:    401  unsupported schema json_web_signature_schema_v1.json
# wycheproof2 unsupported:   1024  unsupported schema keywrap_test_schema_v1.json
# wycheproof2 unsupported:    217  unsupported schema mldsa_sign_noseed_schema.json
# wycheproof2 unsupported:    281  unsupported schema mldsa_sign_seed_schema.json
# wycheproof2 unsupported:    485  unsupported schema mldsa_verify_schema.json
# wycheproof2 unsupported:    795  unsupported schema mlkem_encaps_test_schema.json
# wycheproof2 unsupported:    300  unsupported schema mlkem_keygen_seed_test_schema.json
# wycheproof2 unsupported:     21  unsupported schema mlkem_semi_expanded_decaps_test_schema.json
# wycheproof2 unsupported:    579  unsupported schema mlkem_test_schema.json
# wycheproof2 unsupported:    317  unsupported schema primality_test_schema_v1.json

Some algorithms (SM3, ML-KEM and ML-DSA) are waiting as pull requests.

Missing algorithms are clear, but there are some cases where we are pretty close, especially RSA and ECC combined with SHAKE128/256. We do support SHAKE128/256, but not in the form of an ltc_hash_descriptor that can be passed to e.g. rsa_sign_hash_ex. Maybe two simple helpers sha3_shake128_init + sha3_shake256_init (usable for ltc_hash_descriptor) based on sha3_shake_init would do the trick. I am not sure whether it is worth it.