[ruby/openssl] Raise an exception if the IO object passed to SSLSocket isn't a file

SSLSocket#connect eventually calls `GetOpenFile` in order to get the
underlying file descriptor for the IO object passed in on
initialization.  `GetOpenFile` assumes that the Ruby object passed in is
a T_FILE object and just casts it to a T_FILE without any checks.  If
you pass an object that *isn't* a T_FILE to that function, the program
will segv.

Since we assume the IO object is a file in the `connect` method, this
commit adds a `CheckType` in the initialize method to ensure that the IO
object is actually a T_FILE.  If the object *isn't* a T_FILE, this class
will segv on `connect`, so I think this is a backwards compatible
change.

https://github.com/ruby/openssl/commit/919fa44ec2

1 files changed, 11 insertions, 0 deletions

diff --git a/test/openssl/test_ssl.rb b/test/openssl/test_ssl.rb
index 6412250c86..15e8d7e3f3 100644
--- a/test/openssl/test_ssl.rb
+++ b/test/openssl/test_ssl.rb
@@ -4,6 +4,17 @@ require_relative "utils"
 if defined?(OpenSSL)
 
 class OpenSSL::TestSSL < OpenSSL::SSLTestCase
+  def test_bad_socket
+    bad_socket = Struct.new(:sync).new
+    assert_raises TypeError do
+      socket = OpenSSL::SSL::SSLSocket.new bad_socket
+      # if the socket is not a T_FILE, `connect` will segv because it tries
+      # to get the underlying file descriptor but the API it calls assumes
+      # the object type is T_FILE
+      socket.connect
+    end
+  end
+
   def test_ctx_options
     ctx = OpenSSL::SSL::SSLContext.new