digest: remove OpenSSL engine

The OpenSSL engine of Digest uses the low-level API of OpenSSL, whose
use has been discouraged for years for multiple reasons.

A long-standing issue on a FIPS-enabled system is that using ::Digest
results in crashing the Ruby process, because the low-level API lacks
the mechanism to report an error (the policy violation) and thus kills
the process as a last resort[1][2]. Also, the upcoming OpenSSL 3.0 will
deprecate it for future removal[3]. Compiling with
-Wdeprecated-declarations will start to emit warnings.

A proper fix for this is to make it use the EVP API instead. This is a
non-trivial work as it requires backwards-incompatible changes to the
framework interface of Digest::Base and rb_digest_metadata_t.

It is more than 15 years ago that the openssl library became part of the
standard library. It has implemented the exactly same functionality as
OpenSSL::Digest, in fact, as a subclass of Digest::Class. There is not
much point in having an identical code in the digest library. Let's
just get rid of OpenSSL within digest. This leaves the C implementations
and the CommonCrypto engine for Apple systems.

A patch is being prepared for the openssl library to provide ::Digest
constants for better performance[4].

[1] https://bugs.ruby-lang.org/issues/6946
[2] https://bugs.ruby-lang.org/issues/13681
[3] https://www.openssl.org/docs/OpenSSL300Design.html
[4] https://github.com/ruby/openssl/pull/377

4 files changed, 1 insertions, 26 deletions

diff --git a/ext/digest/rmd160/depend b/ext/digest/rmd160/depend
index 424aa43b0c..ce5dcdb871 100644
--- a/ext/digest/rmd160/depend
+++ b/ext/digest/rmd160/depend
@@ -329,5 +329,4 @@ rmd160init.o: $(srcdir)/../defs.h
 rmd160init.o: $(srcdir)/../digest.h
 rmd160init.o: rmd160.h
 rmd160init.o: rmd160init.c
-rmd160init.o: rmd160ossl.h
 # AUTOGENERATED DEPENDENCIES END
diff --git a/ext/digest/rmd160/extconf.rb b/ext/digest/rmd160/extconf.rb
index a02ba56169..ffa70ee803 100644
--- a/ext/digest/rmd160/extconf.rb
+++ b/ext/digest/rmd160/extconf.rb
@@ -10,7 +10,7 @@ $defs << "-DNDEBUG" << "-DHAVE_CONFIG_H"
 
 $objs = [ "rmd160init.#{$OBJEXT}" ]
 
-digest_conf("rmd160", "ripemd", "RIPEMD160")
+digest_conf("rmd160")
 
 have_header("sys/cdefs.h")
 
diff --git a/ext/digest/rmd160/rmd160init.c b/ext/digest/rmd160/rmd160init.c
index a2c0a023c0..2c44d83df6 100644
--- a/ext/digest/rmd160/rmd160init.c
+++ b/ext/digest/rmd160/rmd160init.c
@@ -3,11 +3,7 @@
 
 #include <ruby/ruby.h>
 #include "../digest.h"
-#if defined(RMD160_USE_OPENSSL)
-#include "rmd160ossl.h"
-#else
 #include "rmd160.h"
-#endif
 
 static const rb_digest_metadata_t rmd160 = {
     RUBY_DIGEST_API_VERSION,
diff --git a/ext/digest/rmd160/rmd160ossl.h b/ext/digest/rmd160/rmd160ossl.h
deleted file mode 100644
index e6bf5ea8d0..0000000000
--- a/ext/digest/rmd160/rmd160ossl.h
+++ /dev/null
@@ -1,20 +0,0 @@
-/* $Id$ */
-
-#ifndef RMD160OSSL_H_INCLUDED
-#define RMD160OSSL_H_INCLUDED
-
-#include <stddef.h>
-#include <openssl/ripemd.h>
-
-#define RMD160_CTX	RIPEMD160_CTX
-
-#define RMD160_Init	RIPEMD160_Init
-#define RMD160_Update	RIPEMD160_Update
-
-#define RMD160_BLOCK_LENGTH		RIPEMD160_CBLOCK
-#define RMD160_DIGEST_LENGTH		RIPEMD160_DIGEST_LENGTH
-
-static DEFINE_FINISH_FUNC_FROM_FINAL(RIPEMD160)
-#define RMD160_Finish rb_digest_RIPEMD160_finish
-
-#endif