[ruby/openssl] Fix modular square root test with LibreSSL >= 3.8

If x is a modular square root of a (mod p) then so is (p - x). Both
answers are valid. In particular, both 2 and 3 are valid square roots
of 4 (mod 5). Do not assume that a particular square root is chosen by
the algorithm. Indeed, the algorithm in OpenSSL and LibreSSL <= 3.7
returns a non-deterministic answer in many cases. LibreSSL 3.8 and
later will always return the smaller of the two possible answers. This
breaks the current test case.

Instead of checking for a particular square root, check that the square
of the claimed square root is the given value. This is always true. Add
the simplest test case where the answer is indeed non-deterministic.

https://github.com/ruby/openssl/commit/93548ae959

1 files changed, 3 insertions, 1 deletions

diff --git a/test/openssl/test_bn.rb b/test/openssl/test_bn.rb
index 77af14091e..ea88ff06ce 100644
--- a/test/openssl/test_bn.rb
+++ b/test/openssl/test_bn.rb
@@ -175,7 +175,9 @@ class OpenSSL::TestBN < OpenSSL::TestCase
   end
 
   def test_mod_sqrt
-    assert_equal(3, 4.to_bn.mod_sqrt(5))
+    assert_equal(4, 4.to_bn.mod_sqrt(5).mod_sqr(5))
+    # One of 189484 or 326277 is returned as a square root of 2 (mod 515761).
+    assert_equal(2, 2.to_bn.mod_sqrt(515761).mod_sqr(515761))
     assert_equal(0, 5.to_bn.mod_sqrt(5))
     assert_raise(OpenSSL::BNError) { 3.to_bn.mod_sqrt(5) }
   end