| Age | Commit message (Collapse) | Author |
|---|
|
Refactor gen_opt_mod in YJIT
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
This fails if there are any unused rust-bindgen "allow" entries. For
that target we turn on Rust warnings (there are a lot) and grep for the
ones that correspond to unused allow entries.
I've added check-yjit-bindgen-unused as a dependency of
check-yjit-bindings, so unused allow entries will now fail CI.
This change also removes our single unused allow entry (VM_CALL.*) which
was known to be bad.
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
Also slightly broaden the cases where << on two strings will generate
specialised code rather than a plain method call.
Notes:
Merged: https://github.com/ruby/ruby/pull/6022
|
|
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/5643
|
|
This commit makes YJIT allocate memory for generated code gradually as
needed. Previously, YJIT allocates all the memory it needs on boot in
one go, leading to higher than necessary resident set size (RSS) and
time spent on boot initializing the memory with a large memset().
Users should no longer need to search for a magic number to pass to
`--yjit-exec-mem` since physical memory consumption should now more
accurately reflect the requirement of the workload.
YJIT now reserves a range of addresses on boot. This region start out
with no access permission at all so buggy attempts to jump to the region
crashes like before this change. To get this hardening at finer
granularity than the page size, we fill each page with trapping
instructions when we first allocate physical memory for the page.
Most of the time applications don't need 256 MiB of executable code, so
allocating on-demand ends up doing less total work than before. Case in
point, a simple `ruby --yjit-call-threshold=1 -eitself` takes about
half as long after this change. In terms of memory consumption, here is
a table to give a rough summary of the impact:
| Peak RSS in MiB | -eitself example | railsbench once |
| :-------------: | ---------------: | --------------: |
| before | 265 | 377 |
| after | 11 | 143 |
| no YJIT | 10 | 101 |
A new module is introduced to handle allocation bookkeeping.
`CodePtr` is moved into the module since it has a close relationship
with the new `VirtualMemory` struct. This new interface has a slightly
smaller surface than before in that marking a region as writable is no
longer a public operation.
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
This way YJIT has to match CRuby for each of them.
Remove unused string_p() Rust function
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
but true at this point and we don't usually check the returned value. (#6000)
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
When running with `--yjit-stats` turned on, yjit can inform the user
what the most common exits are. While this is useful information it
doesn't tell you the source location of the code that exited or what the
code that exited looks like. This change intends to fix that.
To use the feature, run yjit with the `--yjit-trace-exits` option,
which will record the backtrace for every exit that occurs. This functionality
requires the stats feature to be turned on. Calling `--yjit-trace-exits`
will automatically set the `--yjit-stats` option.
Users must call `RubyVM::YJIT.dump_exit_locations(filename)` which will
Marshal dump the contents of `RubyVM::YJIT.exit_locations` into a file
based on the passed filename.
*Example usage:*
Given the following script, we write to a file called
`concat_array.dump` the results of `RubyVM::YJIT.exit_locations`.
```ruby
def concat_array
["t", "r", *x = "u", "e"].join
end
1000.times do
concat_array
end
RubyVM::YJIT.dump_exit_locations("concat_array.dump")
```
When we run the file with this branch and the appropriate flags the
stacktrace will be recorded. Note Stackprof needs to be installed or you
need to point to the library directly.
```
./ruby --yjit --yjit-call-threshold=1 --yjit-trace-exits -I/Users/eileencodes/open_source/stackprof/lib test.rb
```
We can then read the dump file with Stackprof:
```
./ruby -I/Users/eileencodes/open_source/stackprof/lib/ /Users/eileencodes/open_source/stackprof/bin/stackprof --text concat_array.dump
```
Results will look similar to the following:
```
==================================
Mode: ()
Samples: 1817 (0.00% miss rate)
GC: 0 (0.00%)
==================================
TOTAL (pct) SAMPLES (pct) FRAME
1001 (55.1%) 1001 (55.1%) concatarray
335 (18.4%) 335 (18.4%) invokeblock
178 (9.8%) 178 (9.8%) send
140 (7.7%) 140 (7.7%) opt_getinlinecache
...etc...
```
Simply inspecting the `concatarray` method will give `SOURCE
UNAVAILABLE` because the source is insns.def.
```
./ruby -I/Users/eileencodes/open_source/stackprof/lib/ /Users/eileencodes/open_source/stackprof/bin/stackprof --text concat_array.dump --method concatarray
```
Result:
```
concatarray (nonexistent.def:1)
samples: 1001 self (55.1%) / 1001 total (55.1%)
callers:
1000 ( 99.9%) Object#concat_array
1 ( 0.1%) Gem.suffixes
callees (0 total):
code:
SOURCE UNAVAILABLE
```
However if we go deeper to the callee we can see the exact
source of the `concatarray` exit.
```
./ruby -I/Users/eileencodes/open_source/stackprof/lib/ /Users/eileencodes/open_source/stackprof/bin/stackprof --text concat_array.dump --method Object#concat_array
```
```
Object#concat_array (/Users/eileencodes/open_source/rust_ruby/test.rb:1)
samples: 0 self (0.0%) / 1000 total (55.0%)
callers:
1000 ( 100.0%) block in <main>
callees (1000 total):
1000 ( 100.0%) concatarray
code:
| 1 | def concat_array
1000 (55.0%) | 2 | ["t", "r", *x = "u", "e"].join
| 3 | end
```
The `--walk` option is recommended for this feature as it make it
easier to traverse the tree of exits.
*Goals of this feature:*
This feature is meant to give more information when working on YJIT.
The idea is that if we know what code is exiting we can decide what
areas to prioritize when fixing exits. In some cases this means adding
prioritizing avoiding certain exits in yjit. In more complex cases it
might mean changing the Ruby code to be more performant when run with
yjit. Ultimately the more information we have about what code is exiting
AND why, the better we can make yjit.
*Known limitations:*
* Due to tracing exits, running this on large codebases like Rails
can be quite slow.
* On complex methods it can still be difficult to pinpoint the exact cause of
an exit.
* Stackprof is a requirement to to view the backtrace information from
the dump file.
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
Constants that can't be imported via bindgen should have
a comment saying why not.
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
|
|
Also add jhawthorn's test to for this bug.
Fix String#to_s invalidation test
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
We want to make it convenient for people to build YJIT and Rust version 1.58.1
or above is available on Ubuntu Jammy, Debian testing, and Fedora 36 through
the usual package manager on those systems. This saves the need to install
`rustup` for some people.
Our code is already 1.58.1 compatible so this commit simply tweaks CI to make
sure that we keep supporting that version. We still test against the latest Rust
version in `--enable-yjit=dev` builds through the Rust version available in
GitHub's CI image.
Rust versions older than 1.58.1 might build YJIT today, but we might make
incompatible changes in the future.
Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
Notes:
Merged: https://github.com/ruby/ruby/pull/5951
Merged-By: XrXr
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/5948
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/5930
|
|
runtime guard-checks for String#to_s, making some blocks too
short to invalidate later. Add NOPs in those cases to reserve space.
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
This implements the getblockparam instruction.
There are two cases we need to handle depending on whether or not
VM_FRAME_FLAG_MODIFIED_BLOCK_PARAM is set in the environment flag.
When the modified flag is unset, we need to call rb_vm_bh_to_procval to
get a proc from our passed block, save the proc in the environment, and
set the modified flag.
In the case that the modified flag is set we are able to just use the
existing proc in the environment.
One quirk of this is that we need to call jit_prepare_routine_call early
and ensure we update PC and SP regardless of the branch taken, so that
we have a consistent SP offset at the start of the next instruction.
We considered using a chain guard to generate these two paths
separately, but decided against it because it's very common to see both
and the modified case is basically a subset of the instructions in the
unmodified case.
This includes tests for both getblockparam and getblockparamproxy which
was previously missing a test.
Notes:
Merged: https://github.com/ruby/ruby/pull/5881
|
|
Before this commit we were accidentally clobbering RAX. Additionally,
since this function had RAX hardcoded then the function may not have
worked with registers other than RAX.
Co-authored-by: John Hawthorn <john@hawthorn.email>
Notes:
Merged-By: jhawthorn <john@hawthorn.email>
|
|
For string concat, see if compile-time encoding of strings matches.
If so, use simple buffer string concat at runtime. Otherwise, use
encoding-checking string concat.
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
Turned out to be a one-character fix :)
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
Follow up https://github.com/ruby/ruby/commit/0514d81
Rust YJIT requires Rust 1.60.0 or later. So, `extern crate` looks unnecessary
because it can use the following Rust 2018 edition feature:
https://doc.rust-lang.org/stable/edition-guide/rust-2018/path-changes.html#no-more-extern-crate
It passes the following tests.
```console
% cd yjit
% cargo test --features asm_comments,disasm
(snip)
test result: ok. 56 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
```
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
`rustc` performs in depth dead code analysis and issues warning
even for things like unused struct fields and unconstructed enum
variants. This was annoying for us during the port but hopefully
they are less of an issue now.
This patch enables all the unused warnings we disabled and address
all the warnings we previously ignored. Generally, the approach I've
taken is to use `cfg!` instead of using the `cfg` attribute and
to delete code where it makes sense. I've put `#[allow(unused)]`
on things we intentionally keep around for printf style debugging
and on items that are too annoying to keep warning-free in all
build configs.
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
This adopts most suggestions that rust-clippy is confident enough to
auto apply. The manual changes mostly fix manual if-lets and take
opportunities to use the `Default` trait on standard collections.
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
Notes:
Merged: https://github.com/ruby/ruby/pull/5853
|
|
is disabled (#5863)
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
* YJIT: replace BLOCKID_NULL with Option<BlockId>, more idiomatic
* Update yjit/src/core.rs
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
* Update yjit/src/core.rs
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
Notes:
Merged-By: maximecb <maximecb@ruby-lang.org>
|
|
|
|
Thanks to suggestion from bjorn3 on GitHub.
Co-authored-by: bjorn3 <bjorn3@users.noreply.github.com>
Notes:
Merged: https://github.com/ruby/ruby/pull/5826
|
|
Thanks to suggestions from Stranger6667 on GitHub.
Co-authored-by: Dmitry Dygalo <dmitry@dygalo.dev>
Notes:
Merged: https://github.com/ruby/ruby/pull/5826
|
|
In December 2021, we opened an [issue] to solicit feedback regarding the
porting of the YJIT codebase from C99 to Rust. There were some
reservations, but this project was given the go ahead by Ruby core
developers and Matz. Since then, we have successfully completed the port
of YJIT to Rust.
The new Rust version of YJIT has reached parity with the C version, in
that it passes all the CRuby tests, is able to run all of the YJIT
benchmarks, and performs similarly to the C version (because it works
the same way and largely generates the same machine code). We've even
incorporated some design improvements, such as a more fine-grained
constant invalidation mechanism which we expect will make a big
difference in Ruby on Rails applications.
Because we want to be careful, YJIT is guarded behind a configure
option:
```shell
./configure --enable-yjit # Build YJIT in release mode
./configure --enable-yjit=dev # Build YJIT in dev/debug mode
```
By default, YJIT does not get compiled and cargo/rustc is not required.
If YJIT is built in dev mode, then `cargo` is used to fetch development
dependencies, but when building in release, `cargo` is not required,
only `rustc`. At the moment YJIT requires Rust 1.60.0 or newer.
The YJIT command-line options remain mostly unchanged, and more details
about the build process are documented in `doc/yjit/yjit.md`.
The CI tests have been updated and do not take any more resources than
before.
The development history of the Rust port is available at the following
commit for interested parties:
https://github.com/Shopify/ruby/commit/1fd9573d8b4b65219f1c2407f30a0a60e537f8be
Our hope is that Rust YJIT will be compiled and included as a part of
system packages and compiled binaries of the Ruby 3.2 release. We do not
anticipate any major problems as Rust is well supported on every
platform which YJIT supports, but to make sure that this process works
smoothly, we would like to reach out to those who take care of building
systems packages before the 3.2 release is shipped and resolve any
issues that may come up.
[issue]: https://bugs.ruby-lang.org/issues/18481
Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com>
Co-authored-by: Noah Gibbs <the.codefolio.guy@gmail.com>
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
Notes:
Merged: https://github.com/ruby/ruby/pull/5826
|
