TLS & Custom Elliptic Curves

TL;DR

While TLS can technically support custom elliptic curves, it’s rarely done in practice due to security risks, compatibility issues, and the complexity of implementation. Standard curves are almost always preferred.

Understanding Elliptic Curves in TLS

TLS (Transport Layer Security) uses elliptic curve cryptography (ECC) for key exchange and digital signatures. The specific curves used are defined within the TLS protocol. These curves determine the strength of the encryption and the speed of operations.

Why Use Standard Elliptic Curves?

• Security: Standard curves have been extensively vetted by cryptographers for weaknesses. Using a custom curve introduces potential vulnerabilities if it hasn’t undergone similar rigorous analysis.
• Compatibility: Most TLS libraries and servers are configured to support only a limited set of well-known, standard curves (like NIST P-256, P-384, P-521, Curve25519).
• Performance: Standard curves often have optimized implementations in hardware and software.

Can You Use Custom Curves?

Yes, technically. TLS allows for the negotiation of elliptic curves using the elliptic_curves extension (defined in RFC 8422). However, this requires both the client and server to explicitly support the custom curve.

How to Attempt Using a Custom Curve (Advanced)

1. Curve Definition: You need to define your elliptic curve mathematically. This includes specifying the coefficients of the equation that defines the curve, the base point, and the order of the group.
2. Library Support: Your TLS library (e.g., OpenSSL) must support adding custom curves. This often involves modifying source code or using a less common build configuration.
3. Server Configuration: Configure your server to advertise support for your custom curve in the elliptic_curves extension. This usually involves editing TLS configuration files (e.g., Apache’s ssl.conf, Nginx’s configuration). The exact method varies greatly depending on the webserver and TLS library used.
4. Client Configuration: Configure your client to prefer or allow the custom curve during TLS handshake. This also depends heavily on the client software.

Example (Conceptual OpenSSL – Highly Simplified)

Warning: This is a very simplified illustration and likely requires significant modification for practical use. Directly adding curves to OpenSSL without deep understanding can compromise security.

/* Add the custom curve definition in your OpenSSL configuration file (openssl.cnf) */
[elliptic_curves]
my_custom_curve = my_custom_curve_name_nid=some_unique_nid,prime=your_prime,a=your_a,b=your_b,generator=your_generator,order=your_order

Then rebuild OpenSSL with the custom curve definitions included.

Risks and Considerations

• Security Audits: Have your custom curve thoroughly audited by experienced cryptographers.
• Implementation Errors: Incorrect implementation can lead to severe security vulnerabilities.
• Forward Secrecy: Ensure that the key exchange protocol used with the custom curve provides forward secrecy (e.g., ECDHE).
• Compatibility Hell: Expect significant difficulties getting clients and servers to negotiate your custom curve.

Alternatives

Instead of using a custom curve, consider these alternatives:

• Use Standard Curves: This is the recommended approach for most applications.
• Curve25519: A modern, high-performance elliptic curve that’s gaining wider support.