YJIT: Make block invalidation more robust

This commit adds an entry_exit field to block_t for use in
invalidate_block_version(). By patching the start of the block while
invalidating it, invalidate_block_version() can function correctly
while there is no executable memory left for new branch stubs.

This change additionally fixes correctness for situations where we
cannot patch incoming jumps to the invalidated block. In situations
such as Shopify/yjit#226, the address to the start of the block
is saved and used later, possibly after the block is invalidated.

The assume_* family of function now generate block->entry_exit before
remembering blocks for invalidation.

RubyVM::YJIT.simulate_oom! is introduced for testing out of memory
conditions. The test for it is disabled for now because OOM triggers
other failure conditions not addressed by this commit.

Fixes Shopify/yjit#226

1 files changed, 74 insertions, 0 deletions

diff --git a/bootstraptest/test_yjit.rb b/bootstraptest/test_yjit.rb
index f1900f9f3f..8286be74e7 100644
--- a/bootstraptest/test_yjit.rb
+++ b/bootstraptest/test_yjit.rb
@@ -2381,3 +2381,77 @@ assert_equal '{:foo=>2}', %q{
   foo
   foo
 }
+
+# block invalidation edge case
+assert_equal 'undef', %q{
+  class A
+    def foo(arg)
+      arg.times { A.remove_method(:bar) }
+      self
+    end
+
+    def bar
+      4
+    end
+
+    def use(arg)
+      # two consecutive sends. When bar is removed, the return address
+      # for calling it is already on foo's control frame
+      foo(arg).bar
+    rescue NoMethodError
+      :undef
+    end
+  end
+
+  A.new.use 0
+  A.new.use 0
+  A.new.use 1
+}
+
+# block invalidation edge case
+assert_equal 'ok', %q{
+  class A
+    Good = :ng
+    def foo(arg)
+      arg.times { A.const_set(:Good, :ok) }
+      self
+    end
+
+    def id(arg)
+      arg
+    end
+
+    def use(arg)
+      # send followed by an opt_getinlinecache.
+      # The return address remains on the control frame
+      # when opt_getinlinecache is invalidated.
+      foo(arg).id(Good)
+    end
+  end
+
+  A.new.use 0
+  A.new.use 0
+  A.new.use 1
+}
+
+# block invalidation while out of memory
+assert_equal 'new', %q{
+  def foo
+    :old
+  end
+
+  def test
+    foo
+  end
+
+  test
+  test
+
+  RubyVM::YJIT.simulate_oom! if defined?(RubyVM::YJIT)
+
+  def foo
+    :new
+  end
+
+  test
+} if false # disabled for now since OOM crashes in the test harness