summaryrefslogtreecommitdiff
path: root/yjit/src/asm/mod.rs
AgeCommit message (Collapse)Author
2023-12-05YJIT: Assert code pages are not partially in-boundsAlan Wu
Helps understand page switching
2023-12-05YJIT: Simplify code page switching logic, remove an assertAlan Wu
We have received a report of `assert!( !cb.has_dropped_bytes())` in set_page() failing. The only explanation for this seems to be memory allocation failing in write_byte(). The if condition implies that `current_write_pos < dst_pos < mem_size`, which rules out failing to encode the relative jump. The has_capacity() assert above not tripping implies that we were in a place in the page where write_byte() did attempt to write the byte and potentially made a syscall in the process. Remove the assert, since memory allocation could fail. Also, return failure if the destination is outside of the code region to detect that out-of-memory situation quicker.
2023-11-07YJIT: Use u32 for CodePtr to save 4 bytes eachAlan Wu
We've long had a size restriction on the code memory region such that a u32 could refer to everything. This commit capitalizes on this restriction by shrinking the size of `CodePtr` to be 4 bytes from 8. To derive a full raw pointer from a `CodePtr`, one needs a base pointer. Both `CodeBlock` and `VirtualMemory` can be used for this purpose. The base pointer is readily available everywhere, except for in the case of the `jit_return` "branch". Generalize lea_label() to lea_jump_target() in the IR to delay deriving the `jit_return` address until `compile()`, when the base pointer is available. On railsbench, this yields roughly a 1% reduction to `yjit_alloc_size` (58,397,765 to 57,742,248).
2023-10-18YJIT: Add --yjit-perf (#8697)Takashi Kokubun
Co-authored-by: Alan Wu <alansi.xingwu@shopify.com>
2023-09-13YJIT: Skip adding past_page_bytes for past pages (#8433)Takashi Kokubun
YJIT: Skip adding past_pages_bytes for past pages Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-09-06YJIT: Make compiled_* stats available by default (#8379)Takashi Kokubun
* YJIT: Make compiled_* stats available by default * Update comment about default counters [ci skip] Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com> --------- Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com> Notes: Merged-By: k0kubun <takashikkbn@gmail.com>
2023-04-11YJIT: Fix build on A64Alan Wu
Typo fix for the last commit (1432b37)
2023-04-11YJIT: Fix a compilation warning in x86_64Takashi Kokubun
This is used only for arm64's cb.jmp_ptr_bytes().
2023-04-11YJIT: Reduce paddings if --yjit-exec-mem-size <= 128 on arm64 (#7671)Takashi Kokubun
* YJIT: Reduce paddings if --yjit-exec-mem-size <= 128 on arm64 * YJIT: Define jmp_ptr_bytes on CodeBlock Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-04-05YJIT: Count the number of actually written bytes (#7658)Takashi Kokubun
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-03-29YJIT: code_gc(): Assert self is inline to avoid other_cb()Alan Wu
The derived `&mut` from `other_cb()` overlapped with the parameter `ocb`. Use `cfg!()` instead of `#[cfg...]` to avoid unused warnings. Notes: Merged: https://github.com/ruby/ruby/pull/7611
2023-03-29YJIT: Fix overlapping &mut in Assembler::code_gc()Alan Wu
Making overlapping `&mut`s triggers Undefined Bahavior. This function previously had them through `cb` and `ocb` aliasing with `self` or live references in the caller. To fix the overlap, take `ocb` as a parameter and don't use `get_inline_cb()` in the body of the function. Notes: Merged: https://github.com/ruby/ruby/pull/7611
2023-03-03YJIT: Fix a cargo test warning on x86_64 (#7428)Takashi Kokubun
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-03-02YJIT: Delete stale `frozen_bytes` related code (#7423)Alan Wu
The code and comments in there have been disabled by comments for a long time. The issues that the counter used to solve are now solved more comprehensively by "runningness" [tracking][1] introduced by Code GC and [delayed deallocation][2]. Having a single counter doesn't fit our current model where code pages that could be touched or not are interleaved, anyway. Just delete the code. [1]: e7c71c6c9271b0c29f210769159090e17128e740 [2]: a0b0365e905e1ac51998ace7e6fc723406a2f157 Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-02-20YJIT: Fix assertion for partially mapped last pages (#7337)Takashi Kokubun
Follows up [Bug #19400] Notes: Merged-By: k0kubun <takashikkbn@gmail.com>
2023-02-09YJIT: Use the system page size when the code page size is too small (#7267)Alan Wu
Previously on ARM64 Linux systems that use 64 KiB pages (`CONFIG_ARM64_64K_PAGES=y`), YJIT was panicking on boot due to a failed assertion. The assertion was making sure that code GC can free the last code page that YJIT manages without freeing unrelated memory. YJIT prefers picking 16 KiB as the granularity at which to free code memory, but when the system can only free at 64 KiB granularity, that is not possible. The fix is to use the system page size as the code page size when the system page size is 64 KiB. Continue to use 16 KiB as the code page size on common systems that use 16/4 KiB pages. Add asserts to code_gc() and free_page() about code GC's assumptions. Fixes [Bug #19400] Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-02-02Fix typos in YJIT [ci skip]Alan Wu
2023-02-02YJIT: other_cb is None in testsAlan Wu
Since the other cb is in CodegenGlobals, and we want Rust tests to be self-contained. Notes: Merged: https://github.com/ruby/ruby/pull/7227
2023-02-02YJIT: Move CodegenGlobals::freed_pages into an RcAlan Wu
This allows for supplying a freed_pages vec in Rust tests. We need it so we can test scenarios that occur after code GC. Notes: Merged: https://github.com/ruby/ruby/pull/7227
2023-01-12Enable `clippy` checks for yjit in CI (#7093)Ian Ker-Seymer
* Add job to check clippy lints in CI * Address all remaining clippy lints * Check lints on arm64 as well * Apply latest clippy lints * Do not exit 0 on clippy warnings Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-01-10YJIT: Fix a compilation warning with release build (#7092)Takashi Kokubun
warning: unused variable: `start_addr` --> ../yjit/src/asm/mod.rs:359:39 | 359 | pub fn remove_comments(&mut self, start_addr: CodePtr, end_addr: CodePtr) { | ^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_start_addr` | = note: `#[warn(unused_variables)]` on by default warning: unused variable: `end_addr` --> ../yjit/src/asm/mod.rs:359:60 | 359 | pub fn remove_comments(&mut self, start_addr: CodePtr, end_addr: CodePtr) { | Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2023-01-09YJIT: Remove old comments for regenerated branches (#7083)Takashi Kokubun
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-12-05YJIT: Remove --yjit-code-page-size (#6865)Alan Wu
Certain code page sizes don't work and can cause crashes, so having this value available as a command-line option is a bit dangerous. Remove it and turn it into a constant instead. Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-11-30YJIT: Fix IseqPayload::pages memory bloatAlan Wu
HashSet::clear() doesn't deallocate the backing buffer and shrink the capacity. Replace with a 0-capcity set instead so we reclaim some memory each code GC. Notes: Merged: https://github.com/ruby/ruby/pull/6833
2022-11-23YJIT: Use NonNull pointer for CodePtr (#6792)Takashi Kokubun
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-11-15YJIT: Include actual memory region size in stats (#6736)Takashi Kokubun
Notes: Merged-By: k0kubun <takashikkbn@gmail.com>
2022-11-08YJIT: Reset dropped_bytes when patching codeAlan Wu
We switch to a new page when we detect dropped_bytes flipping from false to true. Previously, when we patch code for invalidation during code gc, we start with the flag being set to true, so we failed to apply patches that straddle pages. We would write out jumps half way and then stop, which left the code corrupted. Reset the flag before patching so we patch across pages properly. Notes: Merged: https://github.com/ruby/ruby/pull/6686
2022-11-07YJIT: Free pages after ObjectSpace API usages (#6676)Takashi Kokubun
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-11-03YJIT: Make Code GC metrics available for non-stats builds (#6665)Takashi Kokubun
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-11-03YJIT: Stop incrementing write_pos if cb.has_dropped_bytes (#6664)Takashi Kokubun
Co-Authored-By: Alan Wu <alansi.xingwu@shopify.com> Co-authored-by: Alan Wu <alansi.xingwu@shopify.com> Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-11-02YJIT: Avoid accumulating freed pages in the payload (#6657)Takashi Kokubun
Co-Authored-By: Alan Wu <alansi.xingwu@shopify.com> Co-Authored-By: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com> Co-authored-by: Alan Wu <alansi.xingwu@shopify.com> Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com> Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-10-31YJIT: Add RubyVM::YJIT.code_gc (#6644)Takashi Kokubun
* YJIT: Add RubyVM::YJIT.code_gc * Rename compiled_page_count to live_page_count Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-10-25YJIT: GC and recompile all code pages (#6406)Takashi Kokubun
when it fails to allocate a new page. Co-authored-by: Alan Wu <alansi.xingwu@shopify.com> Notes: Merged-By: k0kubun <takashikkbn@gmail.com>
2022-10-21YJIT: Fix page rounding for icache bustingAlan Wu
Previously, we found the current page by rounding the current pointer to the closest smaller page size. This is incorrect because pages are relative to the start of the address we reserve. For example, if the starting address is 12KiB modulo the 16KiB page size, once we have more than 4KiB of code, calculating with the address would incorrectly give us page 1 when we're actually still on page 0. Previously, I can reproduce crashes with: make btest RUN_OPTS=--yjit-code-page-size=32 on ARM64 macOS, where system page sizes are 16KiB. Notes: Merged: https://github.com/ruby/ruby/pull/6607 Merged-By: XrXr
2022-10-19YJIT: Skip dumping code for the other cb on --yjit-dump-disasm (#6592)Takashi Kokubun
YJIT: Skip dumping code for the other cb on --yjit-dump-disasm Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-10-19YJIT: fold the "asm_comments" feature into "disasm" (#6591)Alan Wu
Previously, enabling only "disasm" didn't actually build. Since these two features are closely related and we don't really use one without the other, let's simplify and merge the two features together. Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-10-18Code clean around unused code for some architectures or features (#6581)Jimmy Miller
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-10-17YJIT: Interleave inline and outlined code blocks (#6460)Takashi Kokubun
Co-authored-by: Alan Wu <alansi.xingwu@shopify.com> Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com> Notes: Merged-By: k0kubun <takashikkbn@gmail.com>
2022-10-13fixes more clippy warnings (#6543)Jimmy Miller
* fixes more clippy warnings * Fix x86 c_callable to have doc_strings Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-09-30A bunch of clippy auto fixes for yjit (#6476)Jimmy Miller
Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-09-27YJIT: add assertion wrt label names (#6459)Maxime Chevalier-Boisvert
Add assertion wrt label names Notes: Merged-By: maximecb <maximecb@ruby-lang.org>
2022-09-01Let --yjit-dump-disasm=all dump ocb code as well (#6309)Takashi Kokubun
* Let --yjit-dump-disasm=all dump ocb code as well * Use an enum instead * Add a None Option to DumpDisasm (#444) * Add a None Option to DumpDisasm * Update yjit/src/asm/mod.rs Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com> * Fix a build failure * Use only a single name * Only None will be a disabled case * Fix cargo test * Fix --yjit-dump-disasm=all to print outlined cb Co-authored-by: Jimmy Miller <jimmyhmiller@gmail.com> Co-authored-by: Maxime Chevalier-Boisvert <maximechevalierb@gmail.com> Notes: Merged-By: k0kubun <takashikkbn@gmail.com>
2022-08-29Add --yjit-dump-disasm to dump every compiled code ↵Takashi Kokubun
(https://github.com/Shopify/ruby/pull/430) * Add --yjit-dump-disasm to dump every compiled code * Just use get_option * Carve out disasm_from_addr * Avoid push_str with format! * Share the logic through asm.compile * This seems to negatively impact the compilation speed Notes: Merged: https://github.com/ruby/ruby/pull/6289
2022-08-29Fix code invalidation while OOM and OOM simulation ↵Alan Wu
(https://github.com/Shopify/ruby/pull/395) `YJIT.simulate_oom!` used to leave one byte of space in the code block, so our test didn't expose a problem with asserting that the write position is in bounds in `CodeBlock::set_pos`. We do the following when patching code: 1. save current write position 2. seek to middle of the code block and patch 3. restore old write position The bounds check fails on (3) when the code block is already filled up. Leaving one byte of space also meant that when we write that byte, we need to fill the entire code region with trapping instruction in `VirtualMem`, which made the OOM tests unnecessarily slow. Remove the incorrect bounds check and stop leaving space in the code block when simulating OOM.
2022-08-29Assert that the # of bytes matches for label refs ↵Kevin Newton
(https://github.com/Shopify/ruby/pull/316)
2022-08-29Better label refs (https://github.com/Shopify/ruby/pull/310)Kevin Newton
Previously we were using a `Box<dyn FnOnce>` to support patching the code when jumping to labels. We needed to do this because some of the closures that were being used to patch needed to capture local variables (on both X86 and ARM it was the type of condition for the conditional jumps). To get around that, we can instead use const generics since the condition codes are always known at compile-time. This means that the closures go from polymorphic to monomorphic, which means they can be represented as an `fn` instead of a `Box<dyn FnOnce>`, which means they can fall back to a plain function pointer. This simplifies the storage of the `LabelRef` structs and should hopefully be a better default going forward.
2022-08-29More Arm64 lowering/backend work (https://github.com/Shopify/ruby/pull/307)Kevin Newton
* More Arm64 lowering/backend work * We now have encoding support for the LDR instruction for loading a PC-relative memory location * You can now call add/adds/sub/subs with signed immediates, which switches appropriately based on sign * We can now load immediates into registers appropriately, attempting to keep the minimal number of instructions: * If it fits into 16 bytes, we use just a single movz. * Else if it can be encoded into a bitmask immediate, we use a single mov. * Otherwise we use a movz, a movk, and then optionally another one or two movks. * Fixed a bunch of code to do with the Op::Load opcode. * We now handle GC-offsets properly for Op::Load by skipping around them with a jump instruction. (This will be made better by constant pools in the future.) * Op::Lea is doing what it's supposed to do now. * Fixed a bug in the backend tests to do with not using the result of an Op::Add. * Fix the remaining tests for Arm64 * Move split loads logic into each backend
2022-08-29Arm64 progress (https://github.com/Shopify/ruby/pull/304)Kevin Newton
* Get initial wiring up * Split IncrCounter instruction * Breakpoints in Arm64 * Support for ORR * MOV instruction encodings * Implement JmpOpnd and CRet * Add ORN * Add MVN * PUSH, POP, CCALL for Arm64 * Some formatting and implement Op::Not for Arm64 * Consistent constants when working with the Arm64 SP * Allow OR-ing values into the memory buffer * Test lowering Arm64 ADD * Emit unconditional jumps consistently in Arm64 * Begin emitting conditional jumps for A64 * Back out some labelref changes * Remove label API that no longer exists * Use a trait for the label encoders * Encode nop * Add in nops so jumps are the same width no matter what on Arm64 * Op::Jbe for CodePtr * Pass src_addr and dst_addr instead of calculated offset to label refs * Even more jump work for Arm64 * Fix up jumps to use consistent assertions * Handle splitting Add, Sub, and Not insns for Arm64 * More Arm64 splits and various fixes * PR feedback for Arm64 support * Split up jumps and conditional jump logic
2022-08-29MOVK, MOVZ, BR (https://github.com/Shopify/ruby/pull/296)Kevin Newton
* MOVK instruction * More tests for the A64 entrypoints * Finish testing entrypoints * MOVZ * BR instruction
2022-08-29Port gen_leave_exit(), add support for labels to backendMaxime Chevalier-Boisvert
generated by cgit v1.2.3 (git 2.25.1) at 2026-05-13 16:49:28 +0000