Add ML-DSA certificate support for TLS 1.3

[ygkat]

Problem:

AWS-LC currently supports ML-DSA (FIPS 204) algorithm for signing and verification operations via EVP API, but TLS/SSL layer does not support ML-DSA certificates. When attempting to load an ML-DSA certificate into an SSL context, the operation fails with SSL_R_UNKNOWN_CERTIFICATE_TYPE error.

Reproduction steps:

1. Generate ML-DSA-65 certificate and private key (e.g., using OpenSSL + oqs-provider)
2. Attempt to load certificate into SSL_CTX:

SSL_CTX *ctx = SSL_CTX_new(TLS_server_method());
SSL_CTX_use_certificate_file(ctx, "mldsa_cert.pem", SSL_FILETYPE_PEM);  // Fails
SSL_CTX_use_PrivateKey_file(ctx, "mldsa_key.pem", SSL_FILETYPE_PEM);    // Fails

Error:

error:100000e4:SSL routines:OPENSSL_internal:UNKNOWN_CERTIFICATE_TYPE:ssl/ssl_cert.cc:361

Root cause:

The ssl_is_key_type_supported() function in ssl/ssl_privkey.cc only recognizes RSA, EC, and Ed25519 key types:

bool ssl_is_key_type_supported(int key_type) {
  return key_type == EVP_PKEY_RSA || key_type == EVP_PKEY_EC ||
         key_type == EVP_PKEY_ED25519;
}

ML-DSA keys (EVP_PKEY_PQDSA, NID 993) are not included, despite the algorithm being fully implemented in crypto/fipsmodule/ml_dsa/.

Why this matters:

• NIST has standardized ML-DSA (FIPS 204) for post-quantum digital signatures
• RFC 9881 defines X.509 certificate algorithm identifiers for ML-DSA
• AWS Private CA now supports ML-DSA certificates for private PKI use cases
• Users cannot utilize ML-DSA certificates for TLS authentication (mTLS, server auth) with AWS-LC

Solution:

Add ML-DSA (PQDSA) support to the TLS certificate handling code. The following files require modifications:

1. ssl/internal.h - Add new certificate slot:

#define SSL_PKEY_RSA 0
#define SSL_PKEY_ECC 1
#define SSL_PKEY_ED25519 2
#define SSL_PKEY_PQDSA 3      // New
#define SSL_PKEY_SIZE 4       // Updated from 3

2. ssl/ssl_privkey.cc - Update key type check:

bool ssl_is_key_type_supported(int key_type) {
  return key_type == EVP_PKEY_RSA || key_type == EVP_PKEY_EC ||
         key_type == EVP_PKEY_ED25519 || key_type == EVP_PKEY_PQDSA;
}

3. ssl/ssl_cipher.cc - Update certificate slot index:

int ssl_get_certificate_slot_index(const EVP_PKEY *pkey) {
  switch (EVP_PKEY_id(pkey)) {
    case EVP_PKEY_RSA:
      return SSL_PKEY_RSA;
    case EVP_PKEY_EC:
      return SSL_PKEY_ECC;
    case EVP_PKEY_ED25519:
      return SSL_PKEY_ED25519;
    case EVP_PKEY_PQDSA:
      return SSL_PKEY_PQDSA;
    default:
      return -1;
  }
}

4. ssl/ssl_cipher.cc - Update auth mask for key:

uint32_t ssl_cipher_auth_mask_for_key(const EVP_PKEY *key) {
  switch (EVP_PKEY_id(key)) {
    case EVP_PKEY_RSA:
      return SSL_aRSA;
    case EVP_PKEY_EC:
    case EVP_PKEY_ED25519:
      return SSL_aECDSA;
    case EVP_PKEY_PQDSA:
      return SSL_aGENERIC;  // TLS 1.3 only
    default:
      return 0;
  }
}

Design decisions:

• ML-DSA certificates should only be used with TLS 1.3, as TLS 1.2 signature algorithms do not include ML-DSA
• Using SSL_aGENERIC for auth mask aligns with TLS 1.3's cipher-agnostic authentication model
• A dedicated certificate slot (SSL_PKEY_PQDSA) allows servers to configure multiple certificate types for algorithm negotiation

• Does this change any public APIs? No public API changes. Internal SSL functions are modified to recognize additional key type.
• Which algorithm(s) will this impact? ML-DSA-44, ML-DSA-65, ML-DSA-87 (all variants under EVP_PKEY_PQDSA)

Requirements / Acceptance Criteria:

What must a solution address in order to solve the problem? How do we know the solution is complete?

• SSL_CTX_use_certificate_file() successfully loads ML-DSA certificates
• SSL_CTX_use_PrivateKey_file() successfully loads ML-DSA private keys
• SSL_CTX_check_private_key() correctly validates ML-DSA key pairs
• TLS 1.3 handshake completes successfully with ML-DSA server certificate
• TLS 1.3 mTLS works with ML-DSA client certificates
• ML-DSA certificate rejected for TLS 1.2 and below (appropriate error)
• Multiple certificate types (RSA + ECDSA + ML-DSA) can be configured simultaneously

• RFC links:

  • RFC 9881 - Internet X.509 Public Key Infrastructure: Algorithm Identifiers for ML-DSA
  • FIPS 204 - Module-Lattice-Based Digital Signature Standard
  • draft-tls-westerbaan-mldsa - ML-DSA for TLS 1.3

• Related Issues:

  • aws/aws-lc-rs#964 - ML-DSA stabilization
  • aws/aws-lc-rs#773 - Add support for ML-DSA

• Will the Usage Guide or other documentation need to be updated?
  Yes, documentation should be updated to include:

  • ML-DSA certificate configuration examples
  • TLS 1.3 requirement for ML-DSA certificates
  • Multi-certificate configuration for hybrid deployments

• Testing: How will this change be tested? Call out new integration tests, functional tests, or particularly interesting/important unit tests.

  • Unit tests for ssl_is_key_type_supported() with PQDSA key type

  • Unit tests for ssl_get_certificate_slot_index() with PQDSA

  • Integration tests: TLS 1.3 handshake with ML-DSA-44/65/87 certificates

  • Integration tests: mTLS with ML-DSA client certificates

  • Negative tests: ML-DSA certificate rejection in TLS 1.2

  • Interoperability tests with OQS-OpenSSL (if available)

  • Will this change trigger AWS LibCrypto Formal Verification changes? No, this change is in the SSL layer, not the cryptographic implementation.

  • Should this change be fuzz tested? Yes, ML-DSA certificate parsing in TLS handshake should be fuzz tested as it handles untrusted input from peers.

Out of scope:

• Hybrid certificates (e.g., ML-DSA + ECDSA composite): This request focuses on pure ML-DSA certificates only
• TLS 1.2 support: ML-DSA is intended for TLS 1.3 only per current standards
• Certificate chain validation changes: Assumes existing X.509 ML-DSA parsing works correctly
• Performance optimization: Initial implementation focuses on correctness
• FIPS certification: ML-DSA TLS usage may require separate FIPS validation process

[nebeid]

We greatly appreciate not only your interest, but the details and proposed solution you put in the issue. We will review it in January.