[Bug #20650] Fix memory leak in Regexp capture group when timeout (#11244)

Fix memory leak in Regexp capture group when timeout

[Bug #20650]

The capture group allocates memory that is leaked when it times out.

For example:

    re = Regexp.new("^#{"(a*)" * 10_000}x$", timeout: 0.000001)
    str = "a" * 1000000 + "x"

    10.times do
      100.times do
        re =~ str
      rescue Regexp::TimeoutError
      end

      puts `ps -o rss= -p #{$$}`
    end

Before:

    34688
    56416
    78288
    100368
    120784
    140704
    161904
    183568
    204320
    224800

After:

    16288
    16288
    16880
    16896
    16912
    16928
    16944
    17184
    17184
    17200

1 files changed, 1 insertions, 0 deletions

diff --git a/include/ruby/onigmo.h b/include/ruby/onigmo.h
index d233336316..db290cd47a 100644
--- a/include/ruby/onigmo.h
+++ b/include/ruby/onigmo.h
@@ -636,6 +636,7 @@ ONIG_EXTERN const OnigSyntaxType*   OnigDefaultSyntax;
 #define ONIGERR_PARSE_DEPTH_LIMIT_OVER                        -16
 #define ONIGERR_DEFAULT_ENCODING_IS_NOT_SET                   -21
 #define ONIGERR_SPECIFIED_ENCODING_CANT_CONVERT_TO_WIDE_CHAR  -22
+#define ONIGERR_TIMEOUT                                       -23
 /* general error */
 #define ONIGERR_INVALID_ARGUMENT                              -30
 /* syntax error */