summaryrefslogtreecommitdiff
path: root/yjit/src/asm/mod.rs
AgeCommit message (Collapse)Author
2025-09-10YJIT: Tiny refactors (#14505)Stan Lo
Addressed some suggestions from clippy that made sense to me.
2025-07-14YJIT: Move RefCell one level downKunshan Wang
This is the second part of making YJIT work with parallel GC. During GC, `rb_yjit_iseq_mark` and `rb_yjit_iseq_update_references` need to resolve offsets in `Block::gc_obj_offsets` into absolute addresses before reading or updating the fields. This needs the base address stored in `VirtualMemory::region_start` which was previously behind a `RefCell`. When multiple GC threads scan multiple iseq simultaneously (which is possible for some GC modules such as MMTk), it will panic because the `RefCell` is already borrowed. We notice that some fields of `VirtualMemory`, such as `region_start`, are never modified once `VirtualMemory` is constructed. We change the type of the field `CodeBlock::mem_block` from `Rc<RefCell<T>>` to `Rc<T>`, and push the `RefCell` into `VirtualMemory`. We extract mutable fields of `VirtualMemory` into a dedicated struct `VirtualMemoryMut`, and store them in a field `VirtualMemory::mutable` which is a `RefCell<VirtualMemoryMut>`. After this change, methods that access immutable fields in `VirtualMemory`, particularly `base_ptr()` which reads `region_start`, will no longer need to borrow any `RefCell`. Methods that access mutable fields will need to borrow `VirtualMemory::mutable`, but the number of borrowing operations becomes strictly fewer than before because borrowing operations previously done in callers (such as `CodeBlock::write_mem`) are moved into methods of `VirtualMemory` (such as `VirtualMemory::write_bytes`).
2025-07-14YJIT: Set code mem permissions in bulkKunshan Wang
Some GC modules, notably MMTk, support parallel GC, i.e. multiple GC threads work in parallel during a GC. Currently, when two GC threads scan two iseq objects simultaneously when YJIT is enabled, both threads will attempt to borrow `CodeBlock::mem_block`, which will result in panic. This commit makes one part of the change. We now set the YJIT code memory to writable in bulk before the reference-updating phase, and reset it to executable in bulk after the reference-updating phase. Previously, YJIT lazily sets memory pages writable while updating object references embedded in JIT-compiled machine code, and sets the memory back to executable by calling `mark_all_executable`. This approach is inherently unfriendly to parallel GC because (1) it borrows `CodeBlock::mem_block`, and (2) it sets the whole `CodeBlock` as executable which races with other GC threads that are updating other iseq objects. It also has performance overhead due to the frequent invocation of system calls. We now set the permission of all the code memory in bulk before and after the reference updating phase. Multiple GC threads can now perform raw memory writes in parallel. We should also see performance improvement during moving GC because of the reduced number of `mprotect` system calls.
2025-01-29YJIT: A64: Remove assert that trips when OOM at page boundaryAlan Wu
With a well-timed OOM around a page switch in the backend, it can return RetryOnNextPage twice and crash due to the assert. (More places can signal OOM now since VirtualMem tracks Rust malloc heap size for --yjit-mem-size.) Return error in these cases instead of crashing. Fixes: https://github.com/Shopify/ruby/issues/566 Notes: Merged: https://github.com/ruby/ruby/pull/12668
2024-10-07YJIT: Add --yjit-mem-size option (#11810)Takashi Kokubun
* YJIT: Add --yjit-mem-size option * Improve --help * s/the region/this virtual memory region/ Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com> --------- Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com> Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2024-09-30Return an Iterator Instead of a Vector in `addrs_to_pages` Method (#11725)whtsht
* Returning an iterator instead of a vec * Avoid changing the meaning of end_page --------- Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com> Notes: Merged-By: k0kubun <takashikkbn@gmail.com>
2024-07-08YJIT: `dump-disasm`: Print comments and bytes in release buildsAlan Wu
This change implements a fallback mode for the `--yjit-dump-disasm` development command-line option to make it usable in release builds. Previously, using the option with release builds of YJIT yielded only a warning asking the user to build with `--enable-yjit=dev`. While builds that use the `disasm` feature still give the best output, just having the comments is useful enough for many kinds of debugging. Having it usable in release builds is nice for new hackers, too, since this allows for tinkering without having to learn how to build YJIT in development mode. Sample output on A64: ``` # regenerate_branch # Insn: 0001 opt_send_without_block (stack_size: 1) # guard known object with singleton class 0x11f7e0034: 4b 00 00 58 03 00 00 14 08 ce 9c 04 01 00 00 0x11f7e0043: 00 3f 00 0b eb 81 06 01 54 1f 20 03 d5 # RUBY_VM_CHECK_INTS(ec) 0x11f7e0050: 8b 02 42 b8 cb 07 01 35 # stack overflow check 0x11f7e0058: ab 62 02 91 7f 02 0b eb 69 07 01 54 # save PC to CFP 0x11f7e0064: 0b 3b 9a d2 2b 2f a0 f2 0b 00 cc f2 6b 02 00 0x11f7e0073: f8 ab 82 00 91 ``` To ensure this feature doesn't incur too much cost when running without the `--yjit-dump-disasm` option, I checked that there is no significant impact to compile time and memory usage with the `compile_time_ns` and `yjit_alloc_size` entry in `RubyVM::YJIT.runtime_stats`. For each sample, I ran 3 iterations of the `lobsters` YJIT benchmark. The statistics summary and done with the `summary` function in R. Compile time, sample size of 60, lower is better: ``` Before After Min. :2.054e+09 Min. :2.028e+09 1st Qu.:2.069e+09 1st Qu.:2.044e+09 Median :2.081e+09 Median :2.060e+09 Mean :2.089e+09 Mean :2.066e+09 3rd Qu.:2.109e+09 3rd Qu.:2.085e+09 Max. :2.146e+09 Max. :2.144e+09 ``` Allocation size, sample size of 20, lower is better: ``` Before After Min. :21804742 Min. :21794082 1st Qu.:21826682 1st Qu.:21816282 Median :21844042 Median :21826814 Mean :21960664 Mean :22026291 3rd Qu.:21861228 3rd Qu.:22040439 Max. :22587426 Max. :22930614 ``` The `yjit_alloc_size` samples are noisy, but since the average increased by only 0.3%, and the median is lower, I feel safe saying that there is no significant change.
2023-12-05YJIT: Assert code pages are not partially in-boundsAlan Wu
Helps understand page switching
2023-12-05YJIT: Simplify code page switching logic, remove an assertAlan Wu
We have received a report of `assert!( !cb.has_dropped_bytes())` in set_page() failing. The only explanation for this seems to be memory allocation failing in write_byte(). The if condition implies that `current_write_pos < dst_pos < mem_size`, which rules out failing to encode the relative jump. The has_capacity() assert above not tripping implies that we were in a place in the page where write_byte() did attempt to write the byte and potentially made a syscall in the process. Remove the assert, since memory allocation could fail. Also, return failure if the destination is outside of the code region to detect that out-of-memory situation quicker.
2023-11-07YJIT: Use u32 for CodePtr to save 4 bytes eachAlan Wu
We've long had a size restriction on the code memory region such that a u32 could refer to everything. This commit capitalizes on this restriction by shrinking the size of `CodePtr` to be 4 bytes from 8. To derive a full raw pointer from a `CodePtr`, one needs a base pointer. Both `CodeBlock` and `VirtualMemory` can be used for this purpose. The base pointer is readily available everywhere, except for in the case of the `jit_return` "branch". Generalize lea_label() to lea_jump_target() in the IR to delay deriving the `jit_return` address until `compile()`, when the base pointer is available. On railsbench, this yields roughly a 1% reduction to `yjit_alloc_size` (58,397,765 to 57,742,248).
2023-10-18YJIT: Add --yjit-perf (#8697)Takashi Kokubun
Co-authored-by: Alan Wu <alansi.xingwu@shopify.com>
2023-09-13YJIT: Skip adding past_page_bytes for past pages (#8433)Takashi Kokubun
YJIT: Skip adding past_pages_bytes for past pages Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-09-06YJIT: Make compiled_* stats available by default (#8379)Takashi Kokubun
* YJIT: Make compiled_* stats available by default * Update comment about default counters [ci skip] Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com> --------- Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com> Notes: Merged-By: k0kubun <takashikkbn@gmail.com>
2023-04-11YJIT: Fix build on A64Alan Wu
Typo fix for the last commit (1432b37)
2023-04-11YJIT: Fix a compilation warning in x86_64Takashi Kokubun
This is used only for arm64's cb.jmp_ptr_bytes().
2023-04-11YJIT: Reduce paddings if --yjit-exec-mem-size <= 128 on arm64 (#7671)Takashi Kokubun
* YJIT: Reduce paddings if --yjit-exec-mem-size <= 128 on arm64 * YJIT: Define jmp_ptr_bytes on CodeBlock Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-04-05YJIT: Count the number of actually written bytes (#7658)Takashi Kokubun
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-03-29YJIT: code_gc(): Assert self is inline to avoid other_cb()Alan Wu
The derived `&mut` from `other_cb()` overlapped with the parameter `ocb`. Use `cfg!()` instead of `#[cfg...]` to avoid unused warnings. Notes: Merged: https://github.com/ruby/ruby/pull/7611
2023-03-29YJIT: Fix overlapping &mut in Assembler::code_gc()Alan Wu
Making overlapping `&mut`s triggers Undefined Bahavior. This function previously had them through `cb` and `ocb` aliasing with `self` or live references in the caller. To fix the overlap, take `ocb` as a parameter and don't use `get_inline_cb()` in the body of the function. Notes: Merged: https://github.com/ruby/ruby/pull/7611
2023-03-03YJIT: Fix a cargo test warning on x86_64 (#7428)Takashi Kokubun
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-03-02YJIT: Delete stale `frozen_bytes` related code (#7423)Alan Wu
The code and comments in there have been disabled by comments for a long time. The issues that the counter used to solve are now solved more comprehensively by "runningness" [tracking][1] introduced by Code GC and [delayed deallocation][2]. Having a single counter doesn't fit our current model where code pages that could be touched or not are interleaved, anyway. Just delete the code. [1]: e7c71c6c9271b0c29f210769159090e17128e740 [2]: a0b0365e905e1ac51998ace7e6fc723406a2f157 Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-02-20YJIT: Fix assertion for partially mapped last pages (#7337)Takashi Kokubun
Follows up [Bug #19400] Notes: Merged-By: k0kubun <takashikkbn@gmail.com>
2023-02-09YJIT: Use the system page size when the code page size is too small (#7267)Alan Wu
Previously on ARM64 Linux systems that use 64 KiB pages (`CONFIG_ARM64_64K_PAGES=y`), YJIT was panicking on boot due to a failed assertion. The assertion was making sure that code GC can free the last code page that YJIT manages without freeing unrelated memory. YJIT prefers picking 16 KiB as the granularity at which to free code memory, but when the system can only free at 64 KiB granularity, that is not possible. The fix is to use the system page size as the code page size when the system page size is 64 KiB. Continue to use 16 KiB as the code page size on common systems that use 16/4 KiB pages. Add asserts to code_gc() and free_page() about code GC's assumptions. Fixes [Bug #19400] Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-02-02Fix typos in YJIT [ci skip]Alan Wu
2023-02-02YJIT: other_cb is None in testsAlan Wu
Since the other cb is in CodegenGlobals, and we want Rust tests to be self-contained. Notes: Merged: https://github.com/ruby/ruby/pull/7227
2023-02-02YJIT: Move CodegenGlobals::freed_pages into an RcAlan Wu
This allows for supplying a freed_pages vec in Rust tests. We need it so we can test scenarios that occur after code GC. Notes: Merged: https://github.com/ruby/ruby/pull/7227
2023-01-12Enable `clippy` checks for yjit in CI (#7093)Ian Ker-Seymer
* Add job to check clippy lints in CI * Address all remaining clippy lints * Check lints on arm64 as well * Apply latest clippy lints * Do not exit 0 on clippy warnings Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-01-10YJIT: Fix a compilation warning with release build (#7092)Takashi Kokubun
warning: unused variable: `start_addr` --> ../yjit/src/asm/mod.rs:359:39 | 359 | pub fn remove_comments(&mut self, start_addr: CodePtr, end_addr: CodePtr) { | ^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_start_addr` | = note: `#[warn(unused_variables)]` on by default warning: unused variable: `end_addr` --> ../yjit/src/asm/mod.rs:359:60 | 359 | pub fn remove_comments(&mut self, start_addr: CodePtr, end_addr: CodePtr) { | Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-01-09YJIT: Remove old comments for regenerated branches (#7083)Takashi Kokubun
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-12-05YJIT: Remove --yjit-code-page-size (#6865)Alan Wu
Certain code page sizes don't work and can cause crashes, so having this value available as a command-line option is a bit dangerous. Remove it and turn it into a constant instead. Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-11-30YJIT: Fix IseqPayload::pages memory bloatAlan Wu
HashSet::clear() doesn't deallocate the backing buffer and shrink the capacity. Replace with a 0-capcity set instead so we reclaim some memory each code GC. Notes: Merged: https://github.com/ruby/ruby/pull/6833
2022-11-23YJIT: Use NonNull pointer for CodePtr (#6792)Takashi Kokubun
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-11-15YJIT: Include actual memory region size in stats (#6736)Takashi Kokubun
Notes: Merged-By: k0kubun <takashikkbn@gmail.com>
2022-11-08YJIT: Reset dropped_bytes when patching codeAlan Wu
We switch to a new page when we detect dropped_bytes flipping from false to true. Previously, when we patch code for invalidation during code gc, we start with the flag being set to true, so we failed to apply patches that straddle pages. We would write out jumps half way and then stop, which left the code corrupted. Reset the flag before patching so we patch across pages properly. Notes: Merged: https://github.com/ruby/ruby/pull/6686
2022-11-07YJIT: Free pages after ObjectSpace API usages (#6676)Takashi Kokubun
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-11-03YJIT: Make Code GC metrics available for non-stats builds (#6665)Takashi Kokubun
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-11-03YJIT: Stop incrementing write_pos if cb.has_dropped_bytes (#6664)Takashi Kokubun
Co-Authored-By: Alan Wu <alansi.xingwu@shopify.com> Co-authored-by: Alan Wu <alansi.xingwu@shopify.com> Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-11-02YJIT: Avoid accumulating freed pages in the payload (#6657)Takashi Kokubun
Co-Authored-By: Alan Wu <alansi.xingwu@shopify.com> Co-Authored-By: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com> Co-authored-by: Alan Wu <alansi.xingwu@shopify.com> Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com> Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-10-31YJIT: Add RubyVM::YJIT.code_gc (#6644)Takashi Kokubun
* YJIT: Add RubyVM::YJIT.code_gc * Rename compiled_page_count to live_page_count Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-10-25YJIT: GC and recompile all code pages (#6406)Takashi Kokubun
when it fails to allocate a new page. Co-authored-by: Alan Wu <alansi.xingwu@shopify.com> Notes: Merged-By: k0kubun <takashikkbn@gmail.com>
2022-10-21YJIT: Fix page rounding for icache bustingAlan Wu
Previously, we found the current page by rounding the current pointer to the closest smaller page size. This is incorrect because pages are relative to the start of the address we reserve. For example, if the starting address is 12KiB modulo the 16KiB page size, once we have more than 4KiB of code, calculating with the address would incorrectly give us page 1 when we're actually still on page 0. Previously, I can reproduce crashes with: make btest RUN_OPTS=--yjit-code-page-size=32 on ARM64 macOS, where system page sizes are 16KiB. Notes: Merged: https://github.com/ruby/ruby/pull/6607 Merged-By: XrXr
2022-10-19YJIT: Skip dumping code for the other cb on --yjit-dump-disasm (#6592)Takashi Kokubun
YJIT: Skip dumping code for the other cb on --yjit-dump-disasm Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-10-19YJIT: fold the "asm_comments" feature into "disasm" (#6591)Alan Wu
Previously, enabling only "disasm" didn't actually build. Since these two features are closely related and we don't really use one without the other, let's simplify and merge the two features together. Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-10-18Code clean around unused code for some architectures or features (#6581)Jimmy Miller
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-10-17YJIT: Interleave inline and outlined code blocks (#6460)Takashi Kokubun
Co-authored-by: Alan Wu <alansi.xingwu@shopify.com> Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com> Notes: Merged-By: k0kubun <takashikkbn@gmail.com>
2022-10-13fixes more clippy warnings (#6543)Jimmy Miller
* fixes more clippy warnings * Fix x86 c_callable to have doc_strings Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-09-30A bunch of clippy auto fixes for yjit (#6476)Jimmy Miller
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-09-27YJIT: add assertion wrt label names (#6459)Maxime Chevalier-Boisvert
Add assertion wrt label names Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-09-01Let --yjit-dump-disasm=all dump ocb code as well (#6309)Takashi Kokubun
* Let --yjit-dump-disasm=all dump ocb code as well * Use an enum instead * Add a None Option to DumpDisasm (#444) * Add a None Option to DumpDisasm * Update yjit/src/asm/mod.rs Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com> * Fix a build failure * Use only a single name * Only None will be a disabled case * Fix cargo test * Fix --yjit-dump-disasm=all to print outlined cb Co-authored-by: Jimmy Miller <jimmyhmiller@gmail.com> Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com> Notes: Merged-By: k0kubun <takashikkbn@gmail.com>
2022-08-29Add --yjit-dump-disasm to dump every compiled code ↵Takashi Kokubun
(https://github.com/Shopify/ruby/pull/430) * Add --yjit-dump-disasm to dump every compiled code * Just use get_option * Carve out disasm_from_addr * Avoid push_str with format! * Share the logic through asm.compile * This seems to negatively impact the compilation speed Notes: Merged: https://github.com/ruby/ruby/pull/6289
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-16 10:11:18 +0000