Backport 3.3: YJIT memory leak fix with additional CI fixes (#9841)

merge revision(s) 2cc7a56e,b0711b1,db5d9429: [Backport #20209]

	YJIT: Avoid leaks by skipping objects with a singleton class

	For receiver with a singleton class, there are multiple vectors YJIT can
	end up retaining the object. There is a path in jit_guard_known_klass()
	that bakes the receiver into the code, and the object could also be kept
	alive indirectly through a path starting at the CME object baked into
	the code.

	To avoid these leaks, avoid compiling calls on objects with a singleton
	class.

	See: https://github.com/Shopify/ruby/issues/552

	[Bug #20209]
	---
	 yjit/bindgen/src/main.rs       |  1 +
	 yjit/src/codegen.rs            | 17 +++++++++++++++++
	 yjit/src/cruby_bindings.inc.rs |  1 +
	 yjit/src/stats.rs              |  2 ++
	 4 files changed, 21 insertions(+)

	YJIT: Fix tailcall and JIT entry eating up FINISH frames (#9729)

	Suppose YJIT runs a rb_vm_opt_send_without_block()
	fallback and the control frame stack looks like:

	```
	will_tailcall_bar [FINISH]
	caller_that_used_fallback
	```

	will_tailcall_bar() runs in the interpreter and sets up a tailcall.
	Right before JIT_EXEC() in the `send` instruction, the stack will look like:

	```
	bar [FINISH]
	caller_that_used_fallback
	```

	Previously, JIT_EXEC() ran bar() in JIT code, which caused the `FINISH`
	flag to return to the interpreter instead of to the JIT code running
	caller_that_used_fallback(), causing code to run twice and probably
	crash. Recent flaky failures on CI about "each stub expects a particular
	iseq" are probably due to leaving methods twice in
	`test_optimizations.rb`.

	Only run JIT code from the interpreter if a new frame is pushed.
	---
	 test/ruby/test_optimization.rb | 11 +++++++++++
	 vm_exec.h                      |  3 ++-
	 2 files changed, 13 insertions(+), 1 deletion(-)

	YJIT: No need to RESTORE_REG now that we reject tailcalls

	Thanks to Kokubun for noticing.

	Follow-up: b0711b1cf152afad0a480ee2f9bedd142a0d24ac
	---
	 vm_exec.h | 1 -
	 1 file changed, 1 deletion(-)

1 files changed, 17 insertions, 0 deletions

diff --git a/yjit/src/codegen.rs b/yjit/src/codegen.rs
index f360c33bcb..75986dedb1 100644
--- a/yjit/src/codegen.rs
+++ b/yjit/src/codegen.rs
@@ -7132,6 +7132,17 @@ fn gen_send_general(
     assert_eq!(RUBY_T_CLASS, comptime_recv_klass.builtin_type(),
         "objects visible to ruby code should have a T_CLASS in their klass field");
 
+    // Don't compile calls through singleton classes to avoid retaining the receiver.
+    // Make an exception for class methods since classes tend to be retained anyways.
+    // Also compile calls on top_self to help tests.
+    if VALUE(0) != unsafe { FL_TEST(comptime_recv_klass, VALUE(RUBY_FL_SINGLETON as usize)) }
+        && comptime_recv != unsafe { rb_vm_top_self() }
+        && !unsafe { RB_TYPE_P(comptime_recv, RUBY_T_CLASS) }
+        && !unsafe { RB_TYPE_P(comptime_recv, RUBY_T_MODULE) } {
+        gen_counter_incr(asm, Counter::send_singleton_class);
+        return None;
+    }
+
     // Points to the receiver operand on the stack
     let recv = asm.stack_opnd(recv_idx);
     let recv_opnd: YARVOpnd = recv.into();
@@ -7887,6 +7898,12 @@ fn gen_invokesuper_specialized(
         return None;
     }
 
+    // Don't compile `super` on objects with singleton class to avoid retaining the receiver.
+    if VALUE(0) != unsafe { FL_TEST(comptime_recv.class_of(), VALUE(RUBY_FL_SINGLETON as usize)) } {
+        gen_counter_incr(asm, Counter::invokesuper_singleton_class);
+        return None;
+    }
+
     // Do method lookup
     let cme = unsafe { rb_callable_method_entry(comptime_superclass, mid) };
     if cme.is_null() {