Rust YJIT

In December 2021, we opened an [issue] to solicit feedback regarding the
porting of the YJIT codebase from C99 to Rust. There were some
reservations, but this project was given the go ahead by Ruby core
developers and Matz. Since then, we have successfully completed the port
of YJIT to Rust.

The new Rust version of YJIT has reached parity with the C version, in
that it passes all the CRuby tests, is able to run all of the YJIT
benchmarks, and performs similarly to the C version (because it works
the same way and largely generates the same machine code). We've even
incorporated some design improvements, such as a more fine-grained
constant invalidation mechanism which we expect will make a big
difference in Ruby on Rails applications.

Because we want to be careful, YJIT is guarded behind a configure
option:

```shell
./configure --enable-yjit # Build YJIT in release mode
./configure --enable-yjit=dev # Build YJIT in dev/debug mode
```

By default, YJIT does not get compiled and cargo/rustc is not required.
If YJIT is built in dev mode, then `cargo` is used to fetch development
dependencies, but when building in release, `cargo` is not required,
only `rustc`. At the moment YJIT requires Rust 1.60.0 or newer.

The YJIT command-line options remain mostly unchanged, and more details
about the build process are documented in `doc/yjit/yjit.md`.

The CI tests have been updated and do not take any more resources than
before.

The development history of the Rust port is available at the following
commit for interested parties:
https://github.com/Shopify/ruby/commit/1fd9573d8b4b65219f1c2407f30a0a60e537f8be

Our hope is that Rust YJIT will be compiled and included as a part of
system packages and compiled binaries of the Ruby 3.2 release. We do not
anticipate any major problems as Rust is well supported on every
platform which YJIT supports, but to make sure that this process works
smoothly, we would like to reach out to those who take care of building
systems packages before the 3.2 release is shipped and resolve any
issues that may come up.

[issue]: https://bugs.ruby-lang.org/issues/18481

Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Co-authored-by: Noah Gibbs <the.codefolio.guy@gmail.com>
Co-authored-by: Kevin Newton <kddnewton@gmail.com>

2 files changed, 467 insertions, 1 deletions

diff --git a/bootstraptest/test_yjit.rb b/bootstraptest/test_yjit.rb
index a84a9e035a..b0c17229c5 100644
--- a/bootstraptest/test_yjit.rb
+++ b/bootstraptest/test_yjit.rb
@@ -2057,7 +2057,6 @@ assert_equal '[:itself]', %q{
     itself
   end
 
-
   tracing_ractor = Ractor.new do
     # 1: start tracing
     events = []
@@ -2806,3 +2805,48 @@ assert_equal '', %q{
   foo
   foo
 }
+
+# Make sure we're correctly reading RStruct's as.ary union for embedded RStructs
+assert_equal '3,12', %q{
+  pt_struct = Struct.new(:x, :y)
+  p = pt_struct.new(3, 12)
+  def pt_inspect(pt)
+    "#{pt.x},#{pt.y}"
+  end
+
+  # Make sure pt_inspect is JITted
+  10.times { pt_inspect(p) }
+
+  # Make sure it's returning '3,12' instead of e.g. '3,false'
+  pt_inspect(p)
+}
+
+# Regression test for deadlock between branch_stub_hit and ractor_receive_if
+assert_equal '10', %q{
+  r = Ractor.new Ractor.current do |main|
+    main << 1
+    main << 2
+    main << 3
+    main << 4
+    main << 5
+    main << 6
+    main << 7
+    main << 8
+    main << 9
+    main << 10
+  end
+
+  a = []
+  a << Ractor.receive_if{|msg| msg == 10}
+  a << Ractor.receive_if{|msg| msg == 9}
+  a << Ractor.receive_if{|msg| msg == 8}
+  a << Ractor.receive_if{|msg| msg == 7}
+  a << Ractor.receive_if{|msg| msg == 6}
+  a << Ractor.receive_if{|msg| msg == 5}
+  a << Ractor.receive_if{|msg| msg == 4}
+  a << Ractor.receive_if{|msg| msg == 3}
+  a << Ractor.receive_if{|msg| msg == 2}
+  a << Ractor.receive_if{|msg| msg == 1}
+
+  a.length
+}
diff --git a/bootstraptest/test_yjit_rust_port.rb b/bootstraptest/test_yjit_rust_port.rb
new file mode 100644
index 0000000000..e399e0e49e
--- /dev/null
+++ b/bootstraptest/test_yjit_rust_port.rb
@@ -0,0 +1,422 @@
+# Simple tests that we know we can pass
+# To keep track of what we got working during the Rust port
+# And avoid breaking/losing functionality
+#
+# Say "Thread" here to dodge WASM CI check. We use ractors here
+# which WASM doesn't support and it only greps for "Thread".
+
+# Test for opt_mod
+assert_equal '2', %q{
+  def mod(a, b)
+    a % b
+  end
+
+  mod(7, 5)
+  mod(7, 5)
+}
+
+# Test for opt_mult
+assert_equal '12', %q{
+  def mult(a, b)
+    a * b
+  end
+
+  mult(6, 2)
+  mult(6, 2)
+}
+
+# Test for opt_div
+assert_equal '3', %q{
+  def div(a, b)
+    a / b
+  end
+
+  div(6, 2)
+  div(6, 2)
+}
+
+assert_equal '5', %q{
+  def plus(a, b)
+    a + b
+  end
+
+  plus(3, 2)
+}
+
+assert_equal '1', %q{
+  def foo(a, b)
+    a - b
+  end
+
+  foo(3, 2)
+}
+
+assert_equal 'true', %q{
+  def foo(a, b)
+    a < b
+  end
+
+  foo(2, 3)
+}
+
+# Bitwise left shift
+assert_equal '4', %q{
+  def foo(a, b)
+    1 << 2
+  end
+
+  foo(1, 2)
+}
+
+assert_equal '-7', %q{
+  def foo(a, b)
+    -7
+  end
+
+  foo(1, 2)
+}
+
+# Putstring
+assert_equal 'foo', %q{
+  def foo(a, b)
+    "foo"
+  end
+
+  foo(1, 2)
+}
+
+assert_equal '-6', %q{
+  def foo(a, b)
+    a + -7
+  end
+
+  foo(1, 2)
+}
+
+assert_equal 'true', %q{
+  def foo(a, b)
+    a == b
+  end
+
+  foo(3, 3)
+}
+
+assert_equal 'true', %q{
+  def foo(a, b)
+    a < b
+  end
+
+  foo(3, 5)
+}
+
+assert_equal '777', %q{
+  def foo(a)
+    if a
+      777
+    else
+      333
+    end
+  end
+
+  foo(true)
+}
+
+assert_equal '5', %q{
+  def foo(a, b)
+    while a < b
+      a += 1
+    end
+    a
+  end
+
+  foo(1, 5)
+}
+
+# opt_aref
+assert_equal '2', %q{
+  def foo(a, b)
+    a[b]
+  end
+
+  foo([0, 1, 2], 2)
+}
+
+# Simple function calls with 0, 1, 2 arguments
+assert_equal '-2', %q{
+  def bar()
+    -2
+  end
+
+  def foo(a, b)
+    bar()
+  end
+
+  foo(3, 2)
+}
+assert_equal '2', %q{
+  def bar(a)
+    a
+  end
+
+  def foo(a, b)
+    bar(b)
+  end
+
+  foo(3, 2)
+}
+assert_equal '1', %q{
+  def bar(a, b)
+    a - b
+  end
+
+  def foo(a, b)
+    bar(a, b)
+  end
+
+  foo(3, 2)
+}
+
+# Regression test for assembler bug
+assert_equal '1', %q{
+  def check_index(index)
+    if 0x40000000 < index
+        return -1
+    end
+    1
+  end
+
+  check_index 2
+}
+
+# Setivar test
+assert_equal '2', %q{
+  class Klass
+    attr_accessor :a
+
+    def set()
+        @a = 2
+    end
+
+    def get()
+        @a
+    end
+  end
+
+  o = Klass.new
+  o.set()
+  o.a
+}
+
+# Regression for putobject bug
+assert_equal '1.5', %q{
+  def foo(x)
+    x
+  end
+
+  def bar
+    foo(1.5)
+  end
+
+  bar()
+}
+
+# Getivar with an extended ivar table
+assert_equal '3', %q{
+  class Foo
+    def initialize
+      @x1 = 1
+      @x2 = 1
+      @x3 = 1
+      @x4 = 3
+    end
+
+    def bar
+      @x4
+    end
+  end
+
+  f = Foo.new
+  f.bar
+}
+
+assert_equal 'true', %q{
+  x = [[false, true]]
+  for i, j in x
+    ;
+  end
+  j
+}
+
+# Regression for getivar
+assert_equal '[nil]', %q{
+  [TrueClass].each do |klass|
+    klass.class_eval("def foo = @foo")
+  end
+
+  [true].map do |instance|
+    instance.foo
+  end
+}
+
+# Regression for send
+assert_equal 'ok', %q{
+  def bar(baz: 2)
+    baz
+  end
+
+  def foo
+    bar(1, baz: 123)
+  end
+
+  begin
+    foo
+    foo
+  rescue ArgumentError => e
+    print "ok"
+  end
+}
+
+# Array access regression test
+assert_equal '[0, 1, 2, 3, 4, 5]', %q{
+  def expandarray_useless_splat
+    arr = [0, 1, 2, 3, 4, 5]
+    a, * = arr
+  end
+
+  expandarray_useless_splat
+}
+
+# Make sure we're correctly reading RStruct's as.ary union for embedded RStructs
+assert_equal '3,12', %q{
+  pt_struct = Struct.new(:x, :y)
+  p = pt_struct.new(3, 12)
+  def pt_inspect(pt)
+    "#{pt.x},#{pt.y}"
+  end
+
+  # Make sure pt_inspect is JITted
+  10.times { pt_inspect(p) }
+
+  # Make sure it's returning '3,12' instead of e.g. '3,false'
+  pt_inspect(p)
+}
+
+assert_equal '2', %q{
+  def foo(s)
+    s.foo
+  end
+
+  S = Struct.new(:foo)
+  foo(S.new(1))
+  foo(S.new(2))
+}
+
+# Try to compile new method while OOM
+assert_equal 'ok', %q{
+  def foo
+    :ok
+  end
+
+  RubyVM::YJIT.simulate_oom! if defined?(RubyVM::YJIT)
+
+  foo
+}
+
+# test hitting a branch stub when out of memory
+assert_equal 'ok', %q{
+  def nimai(jita)
+    if jita
+      :ng
+    else
+      :ok
+    end
+  end
+
+  nimai(true)
+  nimai(true)
+
+  RubyVM::YJIT.simulate_oom! if defined?(RubyVM::YJIT)
+
+  nimai(false)
+}
+
+# Ractor.current returns a current ractor
+assert_equal 'Ractor', %q{
+  Ractor.current.class
+}
+
+# Ractor.new returns new Ractor
+assert_equal 'Ractor', %q{
+  Ractor.new{}.class
+}
+
+# Ractor.allocate is not supported
+assert_equal "[:ok, :ok]", %q{
+  rs = []
+  begin
+    Ractor.allocate
+  rescue => e
+    rs << :ok if e.message == 'allocator undefined for Ractor'
+  end
+
+  begin
+    Ractor.new{}.dup
+  rescue
+    rs << :ok if e.message == 'allocator undefined for Ractor'
+  end
+
+  rs
+}
+
+# A return value of a Ractor block will be a message from the Ractor.
+assert_equal 'ok', %q{
+  # join
+  r = Ractor.new do
+    'ok'
+  end
+  r.take
+}
+
+# Passed arguments to Ractor.new will be a block parameter
+# The values are passed with Ractor-communication pass.
+assert_equal 'ok', %q{
+  # ping-pong with arg
+  r = Ractor.new 'ok' do |msg|
+    msg
+  end
+  r.take
+}
+
+# Pass multiple arguments to Ractor.new
+assert_equal 'ok', %q{
+  # ping-pong with two args
+  r =  Ractor.new 'ping', 'pong' do |msg, msg2|
+    [msg, msg2]
+  end
+  'ok' if r.take == ['ping', 'pong']
+}
+
+# Ractor#send passes an object with copy to a Ractor
+# and Ractor.receive in the Ractor block can receive the passed value.
+assert_equal 'ok', %q{
+  r = Ractor.new do
+    msg = Ractor.receive
+  end
+  r.send 'ok'
+  r.take
+}
+
+assert_equal '[1, 2, 3]', %q{
+  def foo(arr)
+    arr << 1
+    arr << 2
+    arr << 3
+    arr
+  end
+
+  def bar()
+    foo([])
+  end
+
+  bar()
+}