Updated 20th July 2017 to clarify notation for the point of infinity. A previous version used the symbol 0 (zero) rather than O, which may have been confusing.
Updated 28th May 2020: in step 4 of the full validation check, n is the order of the prime sub-group defined by the generator point G, not the order of the curve itself. This is critical for security if you are performing this check because small-order points will satisfy the order of the curve (which is h * n), but not the order of G.
In the wake of the recent critical security vulnerabilities in some JOSE/JWT libraries around ECDH public key validation, a number of implementations scrambled to implement specific validation of public keys to eliminate these attacks. But how do we know whether these checks are sufficient? Is there any guidance on what checks should be performed? The answer is yes, but it can be a bit hard tracking down exactly what validation needs to be done in which cases. For modern elliptic curve schemes like X25519 and Ed25519, there is some debate over whether validation should be performed at all in the basic primitive implementations, as the curve eliminates some of the issues while high-level protocols can be designed to eliminate others. However, for the NIST standard curves used in JOSE, the question is more clear cut: it is absolutely critical that public keys are correctly validated, as evidenced by the linked security alert.
Continue reading “So how *do* you validate (NIST) ECDH public keys?”