diff options
Diffstat (limited to 'doc/contributing')
| -rw-r--r-- | doc/contributing/bug_triaging.rdoc | 79 | ||||
| -rw-r--r-- | doc/contributing/building_ruby.md | 300 | ||||
| -rw-r--r-- | doc/contributing/concurrency_guide.md | 154 | ||||
| -rw-r--r-- | doc/contributing/contributing.md | 35 | ||||
| -rw-r--r-- | doc/contributing/documentation_guide.md | 322 | ||||
| -rw-r--r-- | doc/contributing/dtrace_probes.rdoc | 184 | ||||
| -rw-r--r-- | doc/contributing/glossary.md | 48 | ||||
| -rw-r--r-- | doc/contributing/making_changes_to_stdlibs.md | 16 | ||||
| -rw-r--r-- | doc/contributing/memory_view.md | 167 | ||||
| -rw-r--r-- | doc/contributing/reporting_issues.md | 37 | ||||
| -rw-r--r-- | doc/contributing/testing_ruby.md | 123 | ||||
| -rw-r--r-- | doc/contributing/vm_stack_and_frames.md | 163 |
12 files changed, 1430 insertions, 198 deletions
diff --git a/doc/contributing/bug_triaging.rdoc b/doc/contributing/bug_triaging.rdoc new file mode 100644 index 0000000000..83fe88cabe --- /dev/null +++ b/doc/contributing/bug_triaging.rdoc @@ -0,0 +1,79 @@ += Bug Triaging Guide + +This guide discusses recommendations for triaging bugs in Ruby's bug tracker. + +== Bugs with Reproducible Examples + +These are the best bug reports. First, consider whether the bug reported is +actually an issue or if it is expected Ruby behavior. If it is expected Ruby +behavior, update the issue with why the behavior is expected, and set the +status to Rejected. + +If the bug reported appears to be an actual bug, try reproducing the bug with +the master branch. If you are not able to reproduce the issue on the master +branch, try reproducing it on the latest version for the branch the bug was +reported on. If you cannot reproduce the issue in either case, update +the issue stating you cannot reproduce the issue, ask the reporter if they +can reproduce the issue with either the master branch or a later release, +and set the status to Feedback. + +If you can reproduce the example with the master branch, try to figure out +what is causing the issue. If you feel comfortable, try working on a +patch for the issue, update the issue, and attach the patch. Try to figure +out which committer should be assigned to the issue, and set them as the +assignee, and set the status to Assigned. + +If you cannot reproduce the example with the master branch, but can reproduce +the issue on the latest version for the branch, then it is likely the bug has +already been fixed, but it has not been backported yet. Try to determine +which commit fixed it, and update the issue noting that the issue has been +fixed but not yet backported. If the Ruby version is in the security +maintenance phase or no longer supported, change the status to Closed. This +change can be made without adding a note to avoid spamming the mailing list. + +For issues that may require backwards incompatible changes or may benefit from +general committer attention or discussion, consider adding them as agenda +items for the next committer meeting (https://bugs.ruby-lang.org/issues/14770). + +== Crash Bugs Without Reproducers + +Many bugs reported have little more than a crash report, often with no way to +reproduce the issue. These bugs are difficult to triage as they often do not +contain enough information. + +For these bugs, if the Ruby version is the master branch or is the latest +release for the branch and the branch is in normal maintenance phase, look at +the backtrace and see if you can determine what could be causing the issue. +If you can guess what could be causing the issue, see if you can put together +a reproducible example (this is in general quite difficult). If you cannot +guess what could be causing the issue, or cannot put together a reproducible +example yourself, please ask the reporter to provide a reproducible example, +and change the status to Feedback. + +If the Ruby version is no longer current (e.g. 2.5.0 when the latest version +on the Ruby 2.5 branch is 2.5.5), add a note to the issue asking the reporter +to try the latest Ruby version for the branch and report back, and change the +status to Feedback. If the Ruby version is in the security maintenance phase +or no longer supported, change the status to Closed. This change can be made +without adding a note. + +== Crash Bugs With 3rd Party C Extensions + +If the crash happens inside a 3rd party C extension, try to figure out inside +which C extension it happens, and add a note to the issue to report the +issue to that C extension, and set the status to Third Party's Issue. + +== Non-Bug reports + +Any issues in the bug tracker that are not reports of problems should have +the tracker changed from Bug to either Feature (new features or performance +improvements) or Misc. This change can be made without adding a note. + +== Stale Issues + +There are many issues that are stale, with no updates in months or even years. +For stale issues in Feedback state, where the feedback has not been received, +you can change the status to Closed without adding a note. For stale issues +in Assigned state, you can reach out to the assignee and see if they can update +the issue. If the assignee is no longer an active committer, remove them +as the assignee and change the status to Open. diff --git a/doc/contributing/building_ruby.md b/doc/contributing/building_ruby.md index 1ac7a375f8..286bd1f116 100644 --- a/doc/contributing/building_ruby.md +++ b/doc/contributing/building_ruby.md @@ -1,80 +1,190 @@ # Building Ruby -## Quick start guide +## Dependencies 1. Install the prerequisite dependencies for building the CRuby interpreter: * C compiler - * autoconf - 2.67 or later - * bison - 3.0 or later - * gperf - 3.0.3 or later - * ruby - 2.7 or later + + For RubyGems, you will also need: + + * [OpenSSL] 1.1.x or 3.0.x / [LibreSSL] + * [libyaml] 0.1.7 or later + * [zlib] + + If you want to build from the git repository, you will also need: + + * [autoconf] - 2.67 or later + * [gperf] - 3.1 or later + * Usually unneeded; only if you edit some source files using gperf + * ruby - 3.1 or later + * We can upgrade this version to system ruby version of the latest + Ubuntu LTS. + * git - 2.32 or later + * Anterior versions may work; 2.32 or later will prevent build + errors in case your system `.gitconfig` uses `$HOME` paths. 2. Install optional, recommended dependencies: - * OpenSSL/LibreSSL - * readline/editline (libedit) - * zlib - * libffi - * libyaml - * libexecinfo (FreeBSD) - * rustc - 1.58.0 or later (if you wish to build [YJIT](/doc/yjit/yjit.md)) + * [libffi] (to build fiddle) + * [gmp] (if you wish to accelerate Bignum operations) + * [rustc] - 1.58.0 or later, if you wish to build + [YJIT](rdoc-ref:RubyVM::YJIT). + + If you want to link the libraries (e.g., gmp) installed into other than + the OS default place, typically using Homebrew on macOS, pass the + `--with-opt-dir` (or `--with-gmp-dir` for gmp) option to `configure`. - If you installed the libraries needed for extensions (openssl, readline, libyaml, zlib) into other than the OS default place, - typically using Homebrew on macOS, add `--with-EXTLIB-dir` options to `CONFIGURE_ARGS` environment variable. + ```sh + configure --with-opt-dir=$(brew --prefix gmp):$(brew --prefix jemalloc) + ``` + + As for the libraries needed for particular extensions only and not for + Ruby (openssl, readline, libyaml, zlib), you can add `--with-EXTLIB-dir` + options to the command line or to `CONFIGURE_ARGS` environment variable. + The command line options will be embedded in `rbconfig.rb`, while the + latter environment variable is not embedded and is only used when + building the extension libraries. - ``` shell + ```sh export CONFIGURE_ARGS="" for ext in openssl readline libyaml zlib; do CONFIGURE_ARGS="${CONFIGURE_ARGS} --with-$ext-dir=$(brew --prefix $ext)" done ``` -3. Checkout the CRuby source code: +[OpenSSL]: https://www.openssl.org +[LibreSSL]: https://www.libressl.org +[libyaml]: https://github.com/yaml/libyaml/ +[zlib]: https://www.zlib.net +[autoconf]: https://www.gnu.org/software/autoconf/ +[gperf]: https://www.gnu.org/software/gperf/ +[libffi]: https://sourceware.org/libffi/ +[gmp]: https://gmplib.org +[rustc]: https://www.rust-lang.org - ``` shell - git clone https://github.com/ruby/ruby.git - ``` +## Quick start guide -4. Generate the configure file: +1. Download ruby source code: - ``` shell - ./autogen.sh - ``` + Select one of the below. + + 1. Build from the tarball: -5. Create a `build` directory outside of the source directory: + Download the latest tarball from [Download Ruby] page and extract + it. Example for Ruby 3.0.2: - ``` shell + ```sh + tar -xzf ruby-3.0.2.tar.gz + cd ruby-3.0.2 + ``` + + 2. Build from the git repository: + + Checkout the CRuby source code: + + ```sh + git clone https://github.com/ruby/ruby.git + cd ruby + ``` + + Run the GNU Autoconf script (which generates the `configure` script): + + ```sh + ./autogen.sh + ``` + +2. Create a `build` directory inside the repository directory: + + ```sh mkdir build && cd build ``` - While it's not necessary to build in a separate directory, it's good practice to do so. + While it's not necessary to build in a dedicated directory like this, it's good + practice to do so. -6. We'll install Ruby in `~/.rubies/ruby-master`, so create the directory: +3. We'll eventually install our new Ruby in `~/.rubies/ruby-master`, so we'll create that directory: - ``` shell + ```sh mkdir ~/.rubies ``` -7. Run configure: +4. Run the `configure` script (which generates the `Makefile`): - ``` shell + ```sh ../configure --prefix="${HOME}/.rubies/ruby-master" ``` - - If you are frequently building Ruby, add the `--disable-install-doc` flag to not build documentation which will speed up the build process. + - Also `-C` (or `--config-cache`) would reduce time to configure from the + next time. + +5. Build Ruby: + + ```sh + make + ``` + +6. [Run tests](testing_ruby.md) to confirm your build succeeded. -8. Build Ruby: +7. Install our newly-compiled Ruby into `~/.rubies/ruby-master`: - ``` shell + ```sh make install ``` -9. [Run tests](testing_ruby.md) to confirm your build succeeded. + - If you need to run `make install` with `sudo` and want to avoid document + generation with different permissions, you can use `make SUDO=sudo + install`. + +8. You can then try your new Ruby out, for example: + + ```sh + ~/.rubies/ruby-master/bin/ruby -e "puts 'Hello, World!'" + ``` + +By the end, your repo will look like this: + +```text +ruby +├── autogen.sh # Pre-existing Autoconf script, used in step 1 +├── configure # Generated in step 1, which generates the `Makefile` in step 4 +├── build # Created in step 2 and populated in step 4 +│ ├── GNUmakefile # Generated by `../configure` +│ ├── Makefile # Generated by `../configure` +│ ├── object.o # Compiled object file, built my `make` +│ └── ... other compiled `.o` object files +│ +│ # Other interesting files: +├── include +│ └── ruby.h # The main public header +├── internal +│ ├── object.h +│ └── ... other header files used by the `.c` files in the repo root. +├── lib +│ └── # Default gems, like `bundler`, `erb`, `set`, `yaml`, etc. +├── spec +│ └── # A mirror of the Ruby specification from github.com/ruby/spec +├── test +│ ├── ruby +│ └── ... +├── object.c +└── ... other `.c` files +``` + +[Download Ruby]: https://www.ruby-lang.org/en/downloads/ ### Unexplainable Build Errors -If you are having unexplainable build errors, after saving all your work, try running `git clean -xfd` in the source root to remove all git ignored local files. If you are working from a source directory that's been updated several times, you may have temporary build artifacts from previous releases which can cause build failures. +If you are having unexplainable build errors, after saving all your work, try +running `git clean -xfd` in the source root to remove all git ignored local +files. If you are working from a source directory that's been updated several +times, you may have temporary build artifacts from previous releases which can +cause build failures. + +## Building on Windows + +The documentation for building on Windows can be found in [the separated +file](../distribution/windows.md). ## More details @@ -83,18 +193,21 @@ about Ruby's build to help out. ### Running make scripts in parallel -In GNU make and BSD make implementations, to run a specific make script in parallel, pass the flag `-j<number of processes>`. For instance, -to run tests on 8 processes, use: +In GNU make[^caution-gmake-3] and BSD make implementations, to run a specific make script in +parallel, pass the flag `-j<number of processes>`. For instance, to run tests +on 8 processes, use: -``` shell +```sh make test-all -j8 ``` We can also set `MAKEFLAGS` to run _all_ `make` commands in parallel. -Having the right `--jobs` flag will ensure all processors are utilized when building software projects. To do this effectively, you can set `MAKEFLAGS` in your shell configuration/profile: +Having the right `--jobs` flag will ensure all processors are utilized when +building software projects. To do this effectively, you can set `MAKEFLAGS` in +your shell configuration/profile: -``` shell +```sh # On macOS with Fish shell: export MAKEFLAGS="--jobs "(sysctl -n hw.ncpu) @@ -108,20 +221,25 @@ export MAKEFLAGS="--jobs "(nproc) export MAKEFLAGS="--jobs $(nproc)" ``` +[^caution-gmake-3]: **CAUTION**: GNU make 3 is missing some features for parallel execution, we +recommend to upgrade to GNU make 4 or later. + ### Miniruby vs Ruby -Miniruby is a version of Ruby which has no external dependencies and lacks certain features. -It can be useful in Ruby development because it allows for faster build times. Miniruby is -built before Ruby. A functional Miniruby is required to build Ruby. To build Miniruby: +Miniruby is a version of Ruby which has no external dependencies and lacks +certain features. It can be useful in Ruby development because it allows for +faster build times. Miniruby is built before Ruby. A functional Miniruby is +required to build Ruby. To build Miniruby: -``` shell +```sh make miniruby ``` ## Debugging -You can use either lldb or gdb for debugging. Before debugging, you need to create a `test.rb` -with the Ruby script you'd like to run. You can use the following make targets: +You can use either lldb or gdb for debugging. Before debugging, you need to +create a `test.rb` with the Ruby script you'd like to run. You can use the +following make targets: * `make run`: Runs `test.rb` using Miniruby * `make lldb`: Runs `test.rb` using Miniruby in lldb @@ -130,25 +248,92 @@ with the Ruby script you'd like to run. You can use the following make targets: * `make lldb-ruby`: Runs `test.rb` using Ruby in lldb * `make gdb-ruby`: Runs `test.rb` using Ruby in gdb +For VS Code users, you can set up editor-based debugging experience by running: + +```shell +cp -r misc/.vscode .vscode +``` + +This will add launch configurations for debugging Ruby itself by running `test.rb` with `lldb`. + +**Note**: if you build Ruby under the `./build` folder, you'll need to update `.vscode/launch.json`'s program entry accordingly to: `"${workspaceFolder}/build/ruby"` + +### Compiling for Debugging + +You can compile Ruby with the `RUBY_DEBUG` macro to enable debugging on some +features. One example is debugging object shapes in Ruby with +`RubyVM::Shape.of(object)`. + +Additionally Ruby can be compiled to support the `RUBY_DEBUG` environment +variable to enable debugging on some features. An example is using +`RUBY_DEBUG=gc_stress` to debug GC-related issues. + +There is also support for the `RUBY_DEBUG_LOG` environment variable to log a +lot of information about what the VM is doing, via the `USE_RUBY_DEBUG_LOG` +macro. + +You should also configure Ruby without optimization and other flags that may +interfere with debugging by changing the optimization flags. + +Bringing it all together: + +```sh +./configure cppflags="-DRUBY_DEBUG=1 -DUSE_RUBY_DEBUG_LOG=1" --enable-debug-env optflags="-O0 -fno-omit-frame-pointer" +``` + ### Building with Address Sanitizer -Using the address sanitizer is a great way to detect memory issues. +Using the address sanitizer (ASAN) is a great way to detect memory issues. It +can detect memory safety issues in Ruby itself, and also in any C extensions +compiled with and loaded into a Ruby compiled with ASAN. -``` shell +```sh ./autogen.sh mkdir build && cd build -export ASAN_OPTIONS="halt_on_error=0:use_sigaltstack=0:detect_leaks=0" -../configure cppflags="-fsanitize=address -fno-omit-frame-pointer" optflags=-O0 LDFLAGS="-fsanitize=address -fno-omit-frame-pointer" +../configure CC=clang-18 cflags="-fsanitize=address -fno-omit-frame-pointer -DUSE_MN_THREADS=0" # and any other options you might like make ``` -On Linux it is important to specify `-O0` when debugging. This is especially true for ASAN which sometimes works incorrectly at higher optimisation levels. +The compiled Ruby will now automatically crash with a report and a backtrace +if ASAN detects a memory safety issue. To run Ruby's test suite under ASAN, +issue the following command. Note that this will take quite a long time (over +two hours on my laptop); the `RUBY_TEST_TIMEOUT_SCALE` and +`SYNTAX_SUGEST_TIMEOUT` variables are required to make sure tests don't +spuriously fail with timeouts when in fact they're just slow. + +```sh +RUBY_TEST_TIMEOUT_SCALE=5 SYNTAX_SUGGEST_TIMEOUT=600 make check +``` + +Please note, however, the following caveats! + +* Due to [Bug #20243], Clang generates code for threadlocal variables which + doesn't work with M:N threading. Thus, it's necessary to disable M:N + threading support at build time for now (with the `-DUSE_MN_THREADS=0` + configure argument). +* ASAN will only work when using Clang version 18 or later - it requires + [llvm/llvm-project#75290] related to multithreaded `fork`. +* ASAN has only been tested so far with Clang on Linux. It may or may not work + with other compilers or on other platforms - please file an issue on + [Ruby Issue Tracking System] if you run into problems with such configurations + (or, to report that they actually work properly!) +* In particular, although I have not yet tried it, I have reason to believe + ASAN will _not_ work properly on macOS yet - the fix for the multithreaded + fork issue was actually reverted for macOS (see [llvm/llvm-project#75659]). + Please open an issue on [Ruby Issue Tracking System] if this is a problem for + you. + +[Revision 9d0a5148]: https://bugs.ruby-lang.org/projects/ruby-master/repository/git/revisions/9d0a5148ae062a0481a4a18fbeb9cfd01dc10428 +[Bug #20243]: https://bugs.ruby-lang.org/issues/20243 +[llvm/llvm-project#75290]: https://github.com/llvm/llvm-project/pull/75290 +[llvm/llvm-project#75659]: https://github.com/llvm/llvm-project/pull/75659#issuecomment-1861584777 +[Ruby Issue Tracking System]: https://bugs.ruby-lang.org ## How to measure coverage of C and Ruby code You need to be able to use gcc (gcov) and lcov visualizer. -``` shell +```sh ./autogen.sh ./configure --enable-gcov make @@ -159,11 +344,12 @@ make lcov open lcov-out/index.html ``` -If you need only C code coverage, you can remove `COVERAGE=true` from the above process. -You can also use `gcov` command directly to get per-file coverage. +If you need only C code coverage, you can remove `COVERAGE=true` from the +above process. You can also use `gcov` command directly to get per-file +coverage. -If you need only Ruby code coverage, you can remove `--enable-gcov`. -Note that `test-coverage.dat` accumulates all runs of `make test-all`. -Make sure that you remove the file if you want to measure one test run. +If you need only Ruby code coverage, you can remove `--enable-gcov`. Note +that `test-coverage.dat` accumulates all runs of `make test-all`. Make sure +that you remove the file if you want to measure one test run. You can see the coverage result of CI: https://rubyci.org/coverage diff --git a/doc/contributing/concurrency_guide.md b/doc/contributing/concurrency_guide.md new file mode 100644 index 0000000000..1fb58f7203 --- /dev/null +++ b/doc/contributing/concurrency_guide.md @@ -0,0 +1,154 @@ +# Concurrency Guide + +This is a guide to thinking about concurrency in the cruby source code, whether that's contributing to Ruby +by writing C or by contributing to one of the JITs. This does not touch on native extensions, only the core +language. It will go over: + +* What needs synchronizing? +* How to use the VM lock, and what you can and can't do when you've acquired this lock. +* What you can and can't do when you've acquired other native locks. +* The difference between the VM lock and the GVL. +* What a VM barrier is and when to use it. +* The lock ordering of some important locks. +* How ruby interrupt handling works. +* The timer thread and what it's responsible for. + +## What needs synchronizing? + +Before ractors, only one ruby thread could run at once. That didn't mean you could forget about concurrency issues, though. The timer thread +is a native thread that interacts with other ruby threads and changes some VM internals, so if these changes can be done in parallel by both the timer +thread and a ruby thread, they need to be synchronized. + +When you add ractors to the mix, it gets more complicated. However, ractors allow you to forget about synchronization for non-shareable objects because +they aren't used across ractors. Only one ruby thread can touch the object at once. For shareable objects, they are deeply frozen so there isn't any +mutation on the objects themselves. However, something like reading/writing constants across ractors does need to be synchronized. In this case, ruby threads need to +see a consistent view of the VM. If publishing the update takes 2 steps or even two separate instructions, like in this case, synchronization is required. + +Most synchronization is to protect VM internals. These internals include structures for the thread scheduler on each ractor, the global ractor scheduler, the +coordination between ruby threads and ractors, global tables (for `fstrings`, encodings, symbols and global vars), etc. Anything that can be mutated by a ractor +that can also be read or mutated by another ractor at the same time requires proper synchronization. + +## The VM Lock + +There's only one VM lock and it is for critical sections that can only be entered by one ractor at a time. +Without ractors, the VM lock is useless. It does not stop all ractors from running, as ractors can run +without trying to acquire this lock. If you're updating global (shared) data between ractors and aren't using +atomics, you need to use a lock and this is a convenient one to use. Unlike other locks, you can allocate ruby-managed +memory with it held. When you take the VM lock, there are things you can and can't do during your critical section: + +You can (as long as no other locks are also held before the VM lock): + +* Create ruby objects, call `ruby_xmalloc`, etc. + +You can't: + +* Context switch to another ruby thread or ractor. This is important, as many things can cause ruby-level context switches including: + + * Calling any ruby method through, for example, `rb_funcall`. If you execute ruby code, a context switch could happen. + This also applies to ruby methods defined in C, as they can be redefined in Ruby. Things that call ruby methods such as + `rb_obj_respond_to` are also disallowed. + + * Calling `rb_raise`. This will call `initialize` on the new exception object. With the VM lock + held, nothing you call should be able to raise an exception. `NoMemoryError` is allowed, however. + + * Calling `rb_nogvl` or a ruby-level mechanism that can context switch like `rb_mutex_lock`. + + * Enter any blocking operation managed by ruby. This will context switch to another ruby thread using `rb_nogvl` or + something equivalent. A blocking operation is one that blocks the thread's progress, such as `sleep` or `IO#read`. + +Internally, the VM lock is the `vm->ractor.sync.lock`. + +You need to be on a ruby thread to take the VM lock. You also can't take it inside any functions that could be called during sweeping, as MMTK sweeps +on another thread and you need a valid `ec` to grab the lock. For this same reason (among others), you can't take it from the timer thread either. + +## Other Locks + +All native locks that aren't the VM lock share a more strict set of rules for what's allowed during the critical section. By native locks, we mean +anything that uses `rb_native_mutex_lock`. Some important locks include the `interrupt_lock`, the ractor scheduling lock (protects global scheduling data structures), +the thread scheduling lock (local to each ractor, protects per-ractor scheduling data structures) and the ractor lock (local to each ractor, protects ractor data structures). + +When you acquire one of these locks, + +You can: + +* Allocate memory though non-ruby allocation such as raw `malloc` or the standard library. But be careful, some functions like `strdup` use +ruby allocation through the use of macros! + +* Use `ccan` lists, as they don't allocate. + +* Do the usual things like set variables or struct fields, manipulate linked lists, signal condition variables etc. + +You can't: + +* Allocate ruby-managed memory. This includes creating ruby objects or using `ruby_xmalloc` or `st_insert`. The reason this +is disallowed is if that allocation causes a GC, then all other ruby threads must join a VM barrier as soon as possible +(when they next check interrupts or acquire the VM lock). This is so that no other ractors are running during GC. If a ruby thread +is waiting (blocked) on this same native lock, it can't join the barrier and a deadlock occurs because the barrier will never finish. + +* Raise exceptions. You also can't use `EC_JUMP_TAG` if it jumps out of the critical section. + +* Context switch. See the `VM Lock` section for more info. + +## Difference Between VM Lock and GVL + +The VM Lock is a particular lock in the source code. There is only one VM Lock. The GVL, on the other hand, is more of a combination of locks. +It is "acquired" when a ruby thread is about to run or is running. Since many ruby threads can run at the same time if they're in different ractors, +there are many GVLs (1 per `SNT` + 1 for the main ractor). It can no longer be thought of as a "Global VM Lock" like it once was before ractors. + +## VM Barriers + +Sometimes, taking the VM Lock isn't enough and you need a guarantee that all ractors have stopped. This happens when running `GC`, for instance. +To get a barrier, you take the VM Lock and call `rb_vm_barrier()`. For the duration that the VM lock is held, no other ractors will be running. It's not used +often as taking a barrier slows ractor performance down considerably, but it's useful to know about and is sometimes the only solution. + +## Lock Orderings + +It's a good idea to not hold more than 2 locks at once on the same thread. Locking multiple locks can introduce deadlocks, so do it with care. When locking +multiple locks at once, follow an ordering that is consistent across the program, otherwise you can introduce deadlocks. Here are the orderings of some important locks: + +* VM lock before ractor_sched_lock +* thread_sched_lock before ractor_sched_lock +* interrupt_lock before timer_th.waiting_lock +* timer_th.waiting_lock before ractor_sched_lock + +These orderings are subject to change, so check the source if you're not sure. On top of this: + +* During each `ubf` (unblock) function, the VM lock can be taken around it in some circumstances. This happens during VM shutdown, for example. +See the "Interrupt Handling" section for more details. + +## Ruby Interrupt Handling + +When the VM runs ruby code, ruby's threads intermittently check ruby-level interrupts. These software interrupts +are for various things in ruby and they can be set by other ruby threads or the timer thread. + +* Ruby threads check when they should give up their timeslice. The native thread switches to another ruby thread when their time is up. +* The timer thread sends a "trap" interrupt to the main thread if any ruby-level signal handlers are pending. +* Ruby threads can have other ruby threads run tasks for them by sending them an interrupt. For instance, ractors send +the main thread an interrupt when they need to `require` a file so that it's done on the main thread. They wait for the +main thread's result. +* During VM shutdown, a "terminate" interrupt is sent to all ractor main threads top stop them asap. +* When calling `Thread#raise`, the caller sends an interrupt to that thread telling it which exception to raise. +* Unlocking a mutex sends the next waiter (if any) an interrupt telling it to grab the lock. +* Signalling or broadcasting on a condition variable tells the waiter(s) to wake up. + +This isn't a complete list. + +When sending an interrupt to a ruby thread, the ruby thread can be blocked. For example, it could be in the middle of a `TCPSocket#read` call. If so, +the receiving thread's `ubf` (unblock function) gets called from the thread (ruby thread or timer thread) that sent the interrupt. +Each ruby thread has a `ubf` that is set when it enters a blocking operation and is unset after returning from it. By default, this `ubf` function sends a +`SIGVTALRM` to the receiving thread to try to unblock it from the kernel so it can check its interrupts. There are other `ubfs` that +aren't associated with a syscall, such as when calling `Ractor#join` or `sleep`. All `ubfs` are called with the `interrupt_lock` held, +so take that into account when using locks inside `ubfs`. + +Remember, `ubfs` can be called from the timer thread so you cannot assume an `ec` inside them. The `ec` (execution context) is only set on ruby threads. + +## The Timer Thread + +The timer thread has a few functions. They are: + +* Send interrupts to ruby threads that have run for their whole timeslice. +* Wake up M:N ruby threads (threads in non-main ractors) blocked on IO or after a specified timeout. This +uses `kqueue` or `epoll`, depending on the OS, to receive IO events on behalf of the threads. +* Continue calling the `SIGVTARLM` signal if a thread is still blocked on a syscall after the first `ubf` call. +* Signal native threads (`SNT`) waiting on a ractor if there are ractors waiting in the global run queue. +* Create more `SNT`s if some are blocked, like on IO or on `Ractor#join`. diff --git a/doc/contributing/contributing.md b/doc/contributing/contributing.md new file mode 100644 index 0000000000..a2ed00ab90 --- /dev/null +++ b/doc/contributing/contributing.md @@ -0,0 +1,35 @@ +# Contributing to Ruby + +## Ruby Issues + +To report an issue in the Ruby core: + +* [Report issues](reporting_issues.md). + +## Ruby Core + +To contribute to the Ruby core functionality, +you'll need initially to: + +* [Build Ruby](building_ruby.md) on your system. +* [Test Ruby](testing_ruby.md), to make sure the build is correct. + +Then: + +* [Make changes to Ruby](making_changes_to_ruby.md). + +And possibly: + +* [Benchmark Ruby](https://github.com/ruby/ruby/tree/master/benchmark#make-benchmark). + +## Ruby Documentation + +To contribute to the Ruby core documentation, see: + +* [Making changes to the Ruby documentation](documentation_guide.md). + +## Ruby Standard Library + +To contribute to the Ruby Standard Library, see: + +* [Making changes to the Ruby Standard Library](making_changes_to_stdlibs.md). diff --git a/doc/contributing/documentation_guide.md b/doc/contributing/documentation_guide.md index 9cfd59d629..4b1e2ac9ad 100644 --- a/doc/contributing/documentation_guide.md +++ b/doc/contributing/documentation_guide.md @@ -7,7 +7,7 @@ in the Ruby core and in the Ruby standard library. ## Generating documentation Most Ruby documentation lives in the source files and is written in -[RDoc format](rdoc-ref:RDoc::Markup). +[RDoc format](https://ruby.github.io/rdoc/RDoc/MarkupReference.html). Some pages live under the `doc` folder and can be written in either `.rdoc` or `.md` format, determined by the file extension. @@ -20,14 +20,16 @@ build directory: make html ``` +If you don't have a build directory, follow the [quick start +guide](building_ruby.md#label-Quick+start+guide) up to step 4. + Then you can preview your changes by opening `{build folder}/.ext/html/index.html` file in your browser. - ## Goal The goal of Ruby documentation is to impart the most important -and relevant in the shortest time. +and relevant information in the shortest time. The reader should be able to quickly understand the usefulness of the subject code and how to use it. @@ -41,16 +43,16 @@ Use your judgment about what the user needs to know. - Write short declarative or imperative sentences. - Group sentences into (ideally short) paragraphs, each covering a single topic. -- Organize material with [headers](rdoc-ref:RDoc::Markup@Headers). +- Organize material with + [headings]. - Refer to authoritative and relevant sources using - [links](rdoc-ref:RDoc::Markup@Links). + [links](https://ruby.github.io/rdoc/RDoc/MarkupReference.html#class-RDoc::MarkupReference-label-Links). - Use simple verb tenses: simple present, simple past, simple future. - Use simple sentence structure, not compound or complex structure. - Avoid: - - Excessive comma-separated phrases; - consider a [list](rdoc-ref:RDoc::Markup@Simple+Lists). + - Excessive comma-separated phrases; consider a [list]. - Idioms and culture-specific references. - - Overuse of headers. + - Overuse of headings. - Using US-ASCII-incompatible characters in C source files; see [Characters](#label-Characters) below. @@ -59,7 +61,7 @@ Use your judgment about what the user needs to know. Use only US-ASCII-compatible characters in a C source file. (If you use other characters, the Ruby CI will gently let you know.) -If want to put ASCII-incompatible characters into the documentation +If you want to put ASCII-incompatible characters into the documentation for a C-coded class, module, or method, there are workarounds involving new files `doc/*.rdoc`: @@ -72,7 +74,7 @@ involving new files `doc/*.rdoc`: class Foo; end ``` -- Similarly, for module `Bar` (defined in file `bar.c`, +- Similarly, for module `Bar` (defined in file `bar.c`), create file `doc/bar.rdoc`, declare `module Bar; end`, and place the module documentation above that declaration: @@ -92,7 +94,7 @@ involving new files `doc/*.rdoc`: Example: - ``` + ```c /* * call-seq: * each_byte {|byte| ... } -> self @@ -107,7 +109,7 @@ involving new files `doc/*.rdoc`: Ruby is documented using RDoc. For information on \RDoc syntax and features, see the -[RDoc Markup Reference](rdoc-ref:RDoc::Markup@RDoc+Markup+Reference). +[RDoc Markup Reference](https://ruby.github.io/rdoc/RDoc/MarkupReference.html). ### Output from `irb` @@ -124,17 +126,23 @@ a.shuffle! #=> [2, 3, 1] a #=> [2, 3, 1] ``` -### Headers +### Headings -Organize a long discussion with [headers](rdoc-ref:RDoc::Markup@Headers). +Organize a long discussion for a class or module with [headings]. + +Do not use formal headings in the documentation for a method or constant. + +In the rare case where heading-like structures are needed +within the documentation for a method or constant, use +[bold text](https://ruby.github.io/rdoc/RDoc/MarkupReference.html#class-RDoc::MarkupReference-label-Bold) +as pseudo-headings. ### Blank Lines A blank line begins a new paragraph. -A [code block](rdoc-ref:RDoc::Markup@Paragraphs+and+Verbatim) -or [list](rdoc-ref:RDoc::Markup@Simple+Lists) -should be preceded by and followed by a blank line. +A [code block](https://ruby.github.io/rdoc/RDoc/MarkupReference.html#class-RDoc::MarkupReference-label-Code+Blocks) +or [list] should be preceded by and followed by a blank line. This is unnecessary for the HTML output, but helps in the `ri` output. ### \Method Names @@ -150,19 +158,146 @@ For a method name in text: or a hash mark for an instance method: <tt>Foo.bar</tt>, <tt>Foo#baz</tt>. +### Embedded Code and Commands + +Code or commands embedded in running text (i.e., not in a code block) +should marked up as +[monofont]. + +Code that is a simple string should include the quote marks. + ### Auto-Linking +Most often, the name of a class, module, or method +is auto-linked: + +```rdoc +- Float. +- Enumerable. +- File.new +- File#read. +``` + +renders as: + +> - Float. +> - Enumerable. +> - File.new +> - File#read. + In general, \RDoc's auto-linking should not be suppressed. -For example, we should write `Array`, not `\Array`. +For example, we should write just plain _Float_ (which is auto-linked): + +```rdoc +Returns a Float. +``` + +which renders as: + +> Returns a Float. + +However, _do_ suppress auto-linking when the word in question +does not refer to a Ruby entity (e.g., some uses of _Class_ or _English_): + +```rdoc +Class variables can be tricky. +``` + +renders as: + +> Class variables can be tricky. + +Also, _do_ suppress auto-linking when the word in question +refers to the current document +(e.g., _Float_ in the documentation for class Float). + +In this case you may consider forcing the name to +[monofont], +which suppresses auto-linking, and also emphasizes that the word is a class name: + +```rdoc +A +Float+ object represents .... +``` + +renders as: + +> A `Float` object represents .... + +For a _very_ few, _very_ often-discussed classes, +you might consider avoiding the capitalized class name altogether. +For example, for some mentions of arrays, +you might write simply the lowercase _array_. + +Instead of: + +```rdoc +For an empty Array, .... +``` + +which renders as: + +> For an empty Array, .... + +you might write: + +```rdoc +For an empty array, .... +``` + +which renders as: + +> For an empty array, .... + +This more casual usage avoids both auto-linking and distracting font changes, +and is unlikely to cause confusion. + +This principle may be usefully applied, in particular, for: + +- An array. +- An integer. +- A hash. +- A string. + +However, it should be applied _only_ when referring to an _instance_ of the class, +and _never_ when referring to the class itself. + +### Explicit Links + +When writing an explicit link, follow these guidelines. -We might consider whether to suppress when: +#### `rdoc-ref` Scheme -- The word in question does not refer to a Ruby entity - (e.g., some uses of _Class_ or _English_). -- The reference is to the current class document - (e.g., _Array_ in the documentation for class `Array`). -- The same reference is repeated many times - (e.g., _RDoc_ on this page). +Use the `rdoc-ref` scheme for: + +- A link in core documentation to other core documentation. +- A link in core documentation to documentation in a standard library package. +- A link in a standard library package to other documentation in that same + standard library package. + +See section "`rdoc-ref` Scheme" in [links]. + +#### URL-Based Link + +Use a full URL-based link for: + +- A link in standard library documentation to documentation in the core. +- A link in standard library documentation to documentation in a different + standard library package. + +Doing so ensures that the link will be valid even when the package documentation +is built independently (separately from the core documentation). + +The link should lead to a target in https://docs.ruby-lang.org/en/master/. + +Also use a full URL-based link for a link to an off-site document. + +### Variable Names + +The name of a variable (as specified in its call-seq) should be marked up as +[monofont]. + +Also, use monofont text for the name of a transient variable +(i.e., one defined and used only in the discussion, such as `n`). ### HTML Tags @@ -175,16 +310,31 @@ may not render them properly. In particular, avoid building tables with HTML tags (<tt><table></tt>, etc.). -Alternatives are: +Alternatives: + +- A {verbatim text block}[https://ruby.github.io/rdoc/RDoc/MarkupReference.html#class-RDoc::MarkupReference-label-Verbatim+Text+Blocks], + using spaces and punctuation to format the text; + note that {text markup}[https://ruby.github.io/rdoc/RDoc/MarkupReference.html#class-RDoc::MarkupReference-label-Text+Markup] + will not be honored: -- The GFM (GitHub Flavored Markdown) table extension, - which is enabled by default. See - {GFM tables extension}[https://github.github.com/gfm/#tables-extension-]. + - Example {source}[https://github.com/ruby/ruby/blob/34d802f32f00df1ac0220b62f72605827c16bad8/file.c#L6570-L6596]. + - Corresponding {output}[rdoc-ref:File@Read-2FWrite+Mode]. -- A {verbatim text block}[rdoc-ref:RDoc::MarkupReference@Verbatim+Text+Blocks], - using spaces and punctuation to format the text. - Note that {text markup}[rdoc-ref:RDoc::MarkupReference@Text+Markup] - will not be honored. +- (Markdown format only): A {Github Flavored Markdown (GFM) table}[https://github.github.com/gfm/#tables-extension-], + using special formatting for the text: + + - Example {source}[https://github.com/ruby/ruby/blob/34d802f32f00df1ac0220b62f72605827c16bad8/doc/contributing/glossary.md?plain=1]. + - Corresponding {output}[https://docs.ruby-lang.org/en/master/contributing/glossary_md.html]. + +### Languages in Examples + +For symbols and strings in documentation examples: + +- Prefer \English in \English documentation: <tt>'Hello'</tt>. +- Prefer Japanese in Japanese documentation: <tt>'こんにちは'</tt>. +- If a second language is needed (as, for example, characters with different byte-sizes), + prefer Japanese in \English documentation and \English in Japanese documentation. +- Use other languages examples only as necessary: see String#capitalize. ## Documenting Classes and Modules @@ -214,9 +364,11 @@ Guidelines: - The section title is `What's Here`. - Consider listing the parent class and any included modules; consider - [links](rdoc-ref:RDoc::Markup@Links) - to their "What's Here" sections if those exist. -- List methods as a bullet list: + [links] to their "What's Here" sections if those exist. +- All methods mentioned in the left-pane table of contents + should be listed (including any methods extended from another class). +- Attributes (which are not included in the TOC) may also be listed. +- Display methods as items in one or more bullet lists: - Begin each item with the method name, followed by a colon and a short description. @@ -224,9 +376,9 @@ Guidelines: (and do not list the aliases separately). - Check the rendered documentation to determine whether \RDoc has recognized the method and linked to it; if not, manually insert a - [link](rdoc-ref:RDoc::Markup@Links). + [link](https://ruby.github.io/rdoc/RDoc/MarkupReference.html#class-RDoc::MarkupReference-label-Links). -- If there are numerous entries, consider grouping them into subsections with headers. +- If there are numerous entries, consider grouping them into subsections with headings. - If there are more than a few such subsections, consider adding a table of contents just below the main section title. @@ -238,10 +390,10 @@ The general structure of the method documentation should be: - Calling sequence (for methods written in C). - Synopsis (short description). +- In-brief examples (optional) - Details and examples. - Argument description (if necessary). - Corner cases and exceptions. -- Aliases. - Related methods (optional). ### Calling Sequence (for methods written in C) @@ -250,17 +402,17 @@ For methods written in Ruby, \RDoc documents the calling sequence automatically. For methods written in C, \RDoc cannot determine what arguments the method accepts, so those need to be documented using \RDoc directive -[`call-seq:`](rdoc-ref:RDoc::Markup@Method+arguments). +[`call-seq:`](https://ruby.github.io/rdoc/RDoc/MarkupReference.html#class-RDoc::MarkupReference-label-Directives+for+Method+Documentation). For a singleton method, use the form: -``` +```rdoc class_name.method_name(method_args) {|block_args| ... } -> return_type ``` Example: -``` +```rdoc * call-seq: * Hash.new(default_value = nil) -> new_hash * Hash.new {|hash, key| ... } -> new_hash @@ -269,23 +421,32 @@ Example: For an instance method, use the form (omitting any prefix, just as RDoc does for a Ruby-coded method): -``` +```rdoc method_name(method_args) {|block_args| ... } -> return_type ``` + For example, in Array, use: -``` +```rdoc * call-seq: * count -> integer * count(obj) -> integer * count {|element| ... } -> integer ``` -``` -* call-seq: +```rdoc +* call-seq: * <=> other -> -1, 0, 1, or nil ``` +For a binary-operator style method (e.g., Array#&), +cite `self` in the call-seq (not, e.g., `array` or `receiver`): + +```rdoc +* call-seq: +* self & other_array -> new_array +``` + Arguments: - If the method does not accept arguments, omit the parentheses. @@ -297,15 +458,17 @@ Arguments: or an explicit argument, use a `call-seq` with optional arguments. For example, use: - ``` - respond_to?(symbol, include_all = false) -> true or false + ```rdoc + * call-seq: + * respond_to?(symbol, include_all = false) -> true or false ``` - If the behavior is different with an omitted or an explicit argument, use a `call-seq` with separate lines. For example, in Enumerable, use: - ``` + ```rdoc + * call-seq: * max -> element * max(n) -> array ``` @@ -315,20 +478,32 @@ Block: - If the method does not accept a block, omit the block. - If the method accepts a block, the `call-seq` should have `{|args| ... }`, not `{|args| block }` or `{|args| code }`. +- If the method accepts a block, but returns an Enumerator when the block is omitted, + the `call-seq` should show both forms: + + ```rdoc + * call-seq: + * array.select {|element| ... } -> new_array + * array.select -> new_enumerator + ``` Return types: - If the method can return multiple different types, separate the types with "or" and, if necessary, commas. -- If the method can return multiple types, use +object+. -- If the method returns the receiver, use +self+. +- If the method can return multiple types, use `object`. +- If the method returns the receiver, use `self`. - If the method returns an object of the same class, - prefix `new_` if an only if the object is not +self+; + prefix `new_` if and only if the object is not `self`; example: `new_array`. Aliases: -- Omit aliases from the `call-seq`, but mention them near the end (see below). +- Omit aliases from the `call-seq`, unless the alias is an + operator method. If listing both a regular method and an + operator method in the `call-seq`, explain in the details and + examples section when it is recommended to use the regular method + and when it is recommended to use the operator method. ### Synopsis @@ -339,14 +514,22 @@ an entire paragraph. For `Array#count`, the synopsis is: -``` -Returns a count of specified elements. -``` +> Returns a count of specified elements. This is great as it is short and descriptive. Avoid documenting too much in the synopsis, stick to the most important information for the benefit of the reader. +### In-Brief Examples + +For a method whose documentation is lengthy, +consider adding an "in-brief" passage, +showing examples that summarize the method's uses. + +The passage may answer some users' questions +(without their having to read long documentation); +see Array#[] and Array#[]=. + ### Details and Examples Most non-trivial methods benefit from examples, as well as details @@ -369,6 +552,15 @@ do not add an example if it provides the same information given in the synopsis or details. The purpose of examples is not to prove what the details are stating. +Many methods that can take an optional block call the block if it is given, +but return a new Enumerator if the block is not given; +in that case, do not provide an example, +but do state the fact (with the auto-linking uppercase Enumerator): + +```rdoc +* With no block given, returns a new Enumerator. +``` + ### Argument Description (if necessary) For methods that require arguments, if not obvious and not explicitly @@ -385,7 +577,7 @@ argument passed if it is not obvious, not explicitly mentioned in the details, and not implicitly shown in the examples. If there is more than one argument or block argument, use a -[labeled list](rdoc-ref:RDoc::Markup@Labeled+Lists). +[labeled list](https://ruby.github.io/rdoc/RDoc/MarkupReference.html#class-RDoc::MarkupReference-label-Labeled+Lists). ### Corner Cases and Exceptions @@ -398,14 +590,6 @@ you do not need to document that a `TypeError` is raised if a non-integer is passed. Do not provide examples of exceptions being raised unless that is a common case, such as `Hash#fetch` raising a `KeyError`. -### Aliases - -Mention aliases in the form - -``` -// Array#find_index is an alias for Array#index. -``` - ### Related Methods (optional) In some cases, it is useful to document which methods are related to @@ -414,7 +598,7 @@ mention `Hash#fetch` as a related method, and `Hash#merge` might mention `Hash#merge!` as a related method. - Consider which methods may be related - to the current method, and if you think the reader would benefit it, + to the current method, and if you think the reader would benefit from it, at the end of the method documentation, add a line starting with "Related: " (e.g. "Related: #fetch."). - Don't list more than three related methods. @@ -423,7 +607,7 @@ mention `Hash#fetch` as a related method, and `Hash#merge` might mention - Consider adding: - A phrase suggesting how the related method is similar to, - or different from,the current method. + or different from, the current method. See an example at Time#getutc. - Example code that illustrates the similarities and differences. See examples at Time#ctime, Time#inspect, Time#to_s. @@ -433,3 +617,7 @@ mention `Hash#fetch` as a related method, and `Hash#merge` might mention For methods that accept multiple argument types, in some cases it can be useful to document the different argument types separately. It's best to use a separate paragraph for each case you are discussing. + +[headings]: https://ruby.github.io/rdoc/RDoc/MarkupReference.html#class-RDoc::MarkupReference-label-Headings +[list]: https://ruby.github.io/rdoc/RDoc/MarkupReference.html#class-RDoc::MarkupReference-label-Lists +[monofont]: https://ruby.github.io/rdoc/RDoc/MarkupReference.html#class-RDoc::MarkupReference-label-Monofont diff --git a/doc/contributing/dtrace_probes.rdoc b/doc/contributing/dtrace_probes.rdoc new file mode 100644 index 0000000000..1b20597ab4 --- /dev/null +++ b/doc/contributing/dtrace_probes.rdoc @@ -0,0 +1,184 @@ += DTrace Probes + +A list of DTrace probes and their functionality. "Module" and "Function" cannot +be defined in user defined probes (known as USDT), so they will not be +specified. Probe definitions are in the format of: + + provider:module:function:name(arguments) + +Since module and function cannot be specified, they will be blank. An example +probe definition for Ruby would then be: + + ruby:::method-entry(class name, method name, file name, line number) + +Where "ruby" is the provider name, module and function names are blank, the +probe name is "method-entry", and the probe takes four arguments: + +* class name +* method name +* file name +* line number + +== Probes List + +=== Stability + +Before we list the specific probes, let's talk about stability. Probe stability +is declared in the probes.d file at the bottom on the #pragma D attributes +lines. Here is a description of each of the stability declarations. + +[Provider name stability] + The provider name of "ruby" has been declared as stable. It is unlikely that + we will change the provider name from "ruby" to something else. + +[Module and Function stability] + Since we are not allowed to provide values for the module and function name, + the values we have provided (no value) is declared as stable. + +[Probe name stability] + The probe names are likely to change in the future, so they are marked as + "Evolving". Consumers should not depend on these names to be stable. + +[Probe argument stability] + The parameters passed to the probes are likely to change in the future, so + they are marked as "Evolving". Consumers should not depend on these to be + stable. + +=== Declared probes + +Probes are defined in the probes.d file. Here are the declared probes along +with when they are fired and the arguments they take: + +[ruby:::method-entry(classname, methodname, filename, lineno);] + This probe is fired just before a method is entered. + + classname:: name of the class (a string) + methodname:: name of the method about to be executed (a string) + filename:: the file name where the method is _being called_ (a string) + lineno:: the line number where the method is _being called_ (an int) + + *NOTE*: will only be fired if tracing is enabled, e.g. with: <code>TracePoint.new{}.enable</code>. + See Feature#14104[https://bugs.ruby-lang.org/issues/14104] for more details. + +[ruby:::method-return(classname, methodname, filename, lineno);] + This probe is fired just after a method has returned. The arguments are the + same as "ruby:::method-entry". + + *NOTE*: will only be fired if tracing is enabled, e.g. with: <code>TracePoint.new{}.enable</code>. + See Feature#14104[https://bugs.ruby-lang.org/issues/14104] for more details. + +[ruby:::cmethod-entry(classname, methodname, filename, lineno);] + This probe is fired just before a C method is entered. The arguments are the + same as "ruby:::method-entry". + +[ruby:::cmethod-return(classname, methodname, filename, lineno);] + This probe is fired just before a C method returns. The arguments are the + same as "ruby:::method-entry". + +[ruby:::require-entry(requiredfile, filename, lineno);] + This probe is fired on calls to rb_require_safe (when a file is required). + + requiredfile:: the name of the file to be required (string). + filename:: the file that called "+require+" (string). + lineno:: the line number where the call to require was made (int). + +[ruby:::require-return(requiredfile, filename, lineno);] + This probe is fired just before rb_require_safe (when a file is required) + returns. The arguments are the same as "ruby:::require-entry". This probe + will not fire if there was an exception during file require. + +[ruby:::find-require-entry(requiredfile, filename, lineno);] + This probe is fired right before search_required is called. search_required + determines whether the file has already been required by searching loaded + features (<code>$"</code>), and if not, figures out which file must be loaded. + + requiredfile:: the file to be required (string). + filename:: the file that called "require" (string). + lineno:: the line number where the call to require was made (int). + +[ruby:::find-require-return(requiredfile, filename, lineno);] + This probe is fired right after search_required returns. See the + documentation for "ruby:::find-require-entry" for more details. Arguments for + this probe are the same as "ruby:::find-require-entry". + +[ruby:::load-entry(loadedfile, filename, lineno);] + This probe is fired when calls to "load" are made. The arguments are the same + as "ruby:::require-entry". + +[ruby:::load-return(loadedfile, filename, lineno);] + This probe is fired when "load" returns. The arguments are the same as + "ruby:::load-entry". + +[ruby:::raise(classname, filename, lineno);] + This probe is fired when an exception is raised. + + classname:: the class name of the raised exception (string) + filename:: the name of the file where the exception was raised (string) + lineno:: the line number in the file where the exception was raised (int) + +[ruby:::object-create(classname, filename, lineno);] + This probe is fired when an object is about to be allocated. + + classname:: the class of the allocated object (string) + filename:: the name of the file where the object is allocated (string) + lineno:: the line number in the file where the object is allocated (int) + +[ruby:::array-create(length, filename, lineno);] + This probe is fired when an Array is about to be allocated. + + length:: the size of the array (long) + filename:: the name of the file where the array is allocated (string) + lineno:: the line number in the file where the array is allocated (int) + +[ruby:::hash-create(length, filename, lineno);] + This probe is fired when a Hash is about to be allocated. + + length:: the size of the hash (long) + filename:: the name of the file where the hash is allocated (string) + lineno:: the line number in the file where the hash is allocated (int) + +[ruby:::string-create(length, filename, lineno);] + This probe is fired when a String is about to be allocated. + + length:: the size of the string (long) + filename:: the name of the file where the string is allocated (string) + lineno:: the line number in the file where the string is allocated (int) + +[ruby:::symbol-create(str, filename, lineno);] + This probe is fired when a Symbol is about to be allocated. + + str:: the contents of the symbol (string) + filename:: the name of the file where the string is allocated (string) + lineno:: the line number in the file where the string is allocated (int) + +[ruby:::parse-begin(sourcefile, lineno);] + Fired just before parsing and compiling a source file. + + sourcefile:: the file being parsed (string) + lineno:: the line number where the source starts (int) + +[ruby:::parse-end(sourcefile, lineno);] + Fired just after parsing and compiling a source file. + + sourcefile:: the file being parsed (string) + lineno:: the line number where the source ended (int) + +[ruby:::gc-mark-begin();] + Fired at the beginning of a mark phase. + +[ruby:::gc-mark-end();] + Fired at the end of a mark phase. + +[ruby:::gc-sweep-begin();] + Fired at the beginning of a sweep phase. + +[ruby:::gc-sweep-end();] + Fired at the end of a sweep phase. + +[ruby:::method-cache-clear(class, sourcefile, lineno);] + Fired when the method cache is cleared. + + class:: the classname being cleared, or "global" (string) + sourcefile:: the file being parsed (string) + lineno:: the line number where the source ended (int) + diff --git a/doc/contributing/glossary.md b/doc/contributing/glossary.md new file mode 100644 index 0000000000..3ec9796147 --- /dev/null +++ b/doc/contributing/glossary.md @@ -0,0 +1,48 @@ +# Ruby Internals Glossary + +Just a list of acronyms I've run across in the Ruby source code and their meanings. + +| Term | Definition | +| --- | -----------| +| `bmethod` | Method defined by `define_method() {}` (a Block that runs as a Method). | +| `BIN` | Basic Instruction Name. Used as a macro to reference the YARV instruction. Converts pop into YARVINSN_pop. | +| `bop` | Basic Operator. Relates to methods like `Integer` plus and minus which can be optimized as long as they haven't been redefined. | +| `cc` | Call Cache. An inline cache structure for the call site. Stored in the `cd` | +| `cd` | Call Data. A data structure that points at the `ci` and the `cc`. `iseq` objects points at the `cd`, and access call information and call caches via this structure | +| CFG | Control Flow Graph. Representation of the program where all control-flow and data dependencies have been made explicit by unrolling the stack and local variables. | +| `cfp`| Control Frame Pointer. Represents a Ruby stack frame. Calling a method pushes a new frame (cfp), returning pops a frame. Points at the `pc`, `sp`, `ep`, and the corresponding `iseq`| +| `ci` | Call Information. Refers to an `rb_callinfo` struct. Contains call information about the call site, including number of parameters to be passed, whether they are keyword arguments or not, etc. Used in conjunction with the `cc` and `cd`. | +| `cme` | Callable Method Entry. Refers to the `rb_callable_method_entry_t` struct, the internal representation of a Ruby method that has `defined_class` and `owner` set and is ready for dispatch. | +| `cref` | Class reference. A structure pointing to the class reference where `klass_or_self`, visibility scope, and refinements are stored. It also stores a pointer to the next class in the hierarchy referenced by `rb_cref_struct * next`. The Class reference is lexically scoped. | +| CRuby | Reference implementation of Ruby written in C | +| `cvar` | Class Variable. Refers to a Ruby class variable like `@@foo` | +| `dvar` | Dynamic Variable. Used by the parser to refer to local variables that are defined outside of the current lexical scope. For example `def foo; bar = 1; -> { p bar }; end` the "bar" inside the block is a `dvar` | +| `ec` | Execution Context. The top level VM context, points at the current `cfp` | +| `ep` | Environment Pointer. Local variables, including method parameters are stored in the `ep` array. The `ep` is pointed to by the `cfp` | +| GC | Garbage Collector | +| `gvar` | Global Variable. Refers to a Ruby global variable like `$$`, etc | +| `ICLASS` | Internal Class. When a module is included, the target class gets a new superclass which is an instance of an `ICLASS`. The `ICLASS` represents the module in the inheritance chain. | +| `ifunc` | Internal FUNCtion. A block implemented in C. | +| `iseq` | Instruction Sequence. Usually "iseq" in the C code will refer to an `rb_iseq_t` object that holds a reference to the actual instruction sequences which are executed by the VM. The object also holds information about the code, like the method name associated with the code. | +| `insn` | Instruction. Refers to a YARV instruction. | +| `insns` | Instructions. Usually an array of YARV instructions. | +| `ivar` | Instance Variable. Refers to a Ruby instance variable like `@foo` | +| `imemo` | Internal Memo. A tagged struct whose memory is managed by Ruby's GC, but contains internal information and isn't meant to be exposed to Ruby programs. Contains various information depending on the type. See the `imemo_type` enum for different types. | +| `IVC` | Instance Variable Cache. Cache specifically for instance variable access | +| JIT | Just In Time compiler | +| `lep` | Local Environment Pointer. An `ep` which is tagged `VM_ENV_FLAG_LOCAL`. Usually this is the `ep` of a method (rather than a block, whose `ep` isn't "local") | +| `local` | Local. Refers to a local variable. | +| `me` | Method Entry. Refers to an `rb_method_entry_t` struct, the internal representation of a Ruby method. | +| MRI | Matz's Ruby Implementation | +| `pc` | Program Counter. Usually the instruction that will be executed _next_ by the VM. Pointed to by the `cfp` and incremented by the VM | +| `snt` | Shared Native Thread. OS thread on which many ruby threads can run. Ruby threads from different ractors can even run on the same SNT. Ruby threads can switch SNTs when they context switch. SNTs are used in the M:N threading model. By default, non-main ractors use this model. +| `dnt` | Dedicated Native Thread. OS thread on which only one ruby thread can run. The ruby thread always runs on that same OS thread. DNTs are used in the 1:1 threading model. By default, the main ractor uses this model. +| `sp` | Stack Pointer. The top of the stack. The VM executes instructions in the `iseq` and instructions will push and pop values on the stack. The VM updates the `sp` on the `cfp` to point at the top of the stack| +| ST table | ST table is the main C implementation of a hash (smaller Ruby hashes may be backed by AR tables). | +| `svar` | Special Variable. Refers to special local variables like `$~` and `$_`. See the `getspecial` instruction in `insns.def` | +| `VALUE` | VALUE is a pointer to a ruby object from the Ruby C code. | +| VM | Virtual Machine. In MRI's case YARV (Yet Another Ruby VM) +| WB | Write Barrier. To do with GC write barriers | +| WC | Wild Card. As seen in instructions like `getlocal_WC_0`. It means this instruction takes a "wild card" for the parameter (in this case an index for a local) | +| YARV | Yet Another Ruby VM. The virtual machine that CRuby uses | +| ZOMBIE | A zombie object. An object that has a finalizer which hasn't been executed yet. The object has been collected, so is "dead", but the finalizer hasn't run yet so it's still somewhat alive. | diff --git a/doc/contributing/making_changes_to_stdlibs.md b/doc/contributing/making_changes_to_stdlibs.md index ef3811ea12..2ceb2e6075 100644 --- a/doc/contributing/making_changes_to_stdlibs.md +++ b/doc/contributing/making_changes_to_stdlibs.md @@ -4,17 +4,17 @@ Everything in the [lib](https://github.com/ruby/ruby/tree/master/lib) directory If you'd like to make contributions to standard libraries, do so in the standalone repositories, and the changes will be automatically mirrored into the Ruby repository. -For example, CSV lives in [a separate repository](https://github.com/ruby/csv) and is mirrored into [Ruby](https://github.com/ruby/ruby/tree/master/lib/csv). +For example, ERB lives in [a separate repository](https://github.com/ruby/erb) and is mirrored into [Ruby](https://github.com/ruby/ruby/tree/master/lib/erb). ## Maintainers -You can find the list of maintainers [here](https://docs.ruby-lang.org/en/master/maintainers_rdoc.html#label-Maintainers). +You can find the list of maintainers [here](https://docs.ruby-lang.org/en/master/maintainers_md.html#label-Maintainers). ## Build First, install its dependencies using: -``` +```shell bundle install ``` @@ -22,7 +22,7 @@ bundle install If the library has a `/ext` directory, it has C files that you need to compile with: -``` +```shell bundle exec rake compile ``` @@ -32,18 +32,18 @@ All standard libraries use [test-unit](https://github.com/test-unit/test-unit) a To run all tests: -``` +```shell bundle exec rake test ``` To run a single test file: -``` +```shell bundle exec rake test TEST="test/test_foo.rb" ``` To run a single test case: -``` -bundle exec rake test TEST="test/test_foo.rb" TESTOPS="--name=/test_mytest/" +```shell +bundle exec rake test TEST="test/test_foo.rb" TESTOPTS="--name=/test_mytest/" ``` diff --git a/doc/contributing/memory_view.md b/doc/contributing/memory_view.md new file mode 100644 index 0000000000..0b1369163d --- /dev/null +++ b/doc/contributing/memory_view.md @@ -0,0 +1,167 @@ +# MemoryView + +MemoryView provides the features to share multidimensional homogeneous arrays of +fixed-size element on memory among extension libraries. + +## Disclaimer + +* This feature is still experimental. The specification described here can be changed in the future. + +* This document is under construction. Please refer the master branch of ruby for the latest version of this document. + +## Overview + +We sometimes deal with certain kinds of objects that have arrays of the same typed fixed-size elements on a contiguous memory area as its internal representation. +Numo::NArray in numo-narray and Magick::Image in rmagick are typical examples of such objects. +MemoryView plays the role of the hub to share the internal data of such objects without copy among such libraries. + +Copy-less sharing of data is very important in some field such as data analysis, machine learning, and image processing. In these field, people need to handle large amount of on-memory data with several libraries. If we are forced to copy to exchange large data among libraries, a large amount of the data processing time must be occupied by copying data. You can avoid such wasting time by using MemoryView. + +MemoryView has two categories of APIs: + +1. Producer API + + Classes can register own MemoryView entry which allows objects of that classes to expose their MemoryView + +2. Consumer API + + Consumer API allows us to obtain and manage the MemoryView of an object + +## MemoryView structure + +A MemoryView structure, `rb_memory_view_t`, is used for exporting objects' MemoryView. +This structure contains the reference of the object, which is the owner of the MemoryView, the pointer to the head of exported memory, and the metadata that describes the structure of the memory. The metadata can describe multidimensional arrays with strides. + +### The member of MemoryView structure + +The MemoryView structure consists of the following members. + +- `VALUE obj` + + The reference to the original object that has the memory exported via the MemoryView. + + RubyVM manages the reference count of the MemoryView-exported objects to guard them from the garbage collection. The consumers do not have to struggle to guard this object from GC. + +- `void *data` + + The pointer to the head of the exported memory. + +- `ssize_t byte_size` + + The number of bytes in the memory pointed by `data`. + +- `bool readonly` + + `true` for readonly memory, `false` for writable memory. + +- `const char *format` + + A string to describe the format of an element, or NULL for unsigned byte. + +- `ssize_t item_size` + + The number of bytes in each element. + +- `const rb_memory_view_item_component_t *item_desc.components` + + The array of the metadata of the component in an element. + +- `size_t item_desc.length` + + The number of items in `item_desc.components`. + +- `ssize_t ndim` + + The number of dimensions. + +- `const ssize_t *shape` + + A `ndim` size array indicating the number of elements in each dimension. + This can be `NULL` when `ndim` is 1. + +- `const ssize_t *strides` + + A `ndim` size array indicating the number of bytes to skip to go to the next element in each dimension. + This can be `NULL` when `ndim` is 1. + +- `const ssize_t *sub_offsets` + + A `ndim` size array consisting of the offsets in each dimension when the MemoryView exposes a nested array. + This can be `NULL` when the MemoryView exposes a flat array. + +- `void *private_data` + + The private data that MemoryView provider uses internally. + This can be `NULL` when any private data is unnecessary. + +## MemoryView APIs + +### For consumers + +- `bool rb_memory_view_available_p(VALUE obj)` + + Return `true` if `obj` supports to export a MemoryView. Return `false` otherwise. + + If this function returns `true`, it doesn't mean the function `rb_memory_view_get` will succeed. + +- `bool rb_memory_view_get(VALUE obj, rb_memory_view_t *view, int flags)` + + If the given `obj` supports to export a MemoryView that conforms the given `flags`, this function fills `view` by the information of the MemoryView and returns `true`. In this case, the reference count of `obj` is increased. + + If the given combination of `obj` and `flags` cannot export a MemoryView, this function returns `false`. The content of `view` is not touched in this case. + + The exported MemoryView must be released by `rb_memory_view_release` when the MemoryView is no longer needed. + +- `bool rb_memory_view_release(rb_memory_view_t *view)` + + Release the given MemoryView `view` and decrement the reference count of `view->obj`. + + Consumers must call this function when the MemoryView is no longer needed. Missing to call this function leads memory leak. + +- `ssize_t rb_memory_view_item_size_from_format(const char *format, const char **err)` + + Calculate the number of bytes occupied by an element. + + When the calculation fails, the failed location in `format` is stored into `err`, and returns `-1`. + +- `void *rb_memory_view_get_item_pointer(rb_memory_view_t *view, const ssize_t *indices)` + + Calculate the location of the item indicated by the given `indices`. + The length of `indices` must equal to `view->ndim`. + This function initializes `view->item_desc` if needed. + +- `VALUE rb_memory_view_get_item(rb_memory_view_t *view, const ssize_t *indices)` + + Return the Ruby object representation of the item indicated by the given `indices`. + The length of `indices` must equal to `view->ndim`. + This function uses `rb_memory_view_get_item_pointer`. + +- `rb_memory_view_init_as_byte_array(rb_memory_view_t *view, VALUE obj, void *data, const ssize_t len, const bool readonly)` + + Fill the members of `view` as an 1-dimensional byte array. + +- `void rb_memory_view_fill_contiguous_strides(const ssize_t ndim, const ssize_t item_size, const ssize_t *const shape, const bool row_major_p, ssize_t *const strides)` + + Fill the `strides` array with byte-Strides of a contiguous array of the given shape with the given element size. + +- `void rb_memory_view_prepare_item_desc(rb_memory_view_t *view)` + + Fill the `item_desc` member of `view`. + +- `bool rb_memory_view_is_contiguous(const rb_memory_view_t *view)` + + Return `true` if the data in the MemoryView `view` is row-major or column-major contiguous. + + Return `false` otherwise. + +- `bool rb_memory_view_is_row_major_contiguous(const rb_memory_view_t *view)` + + Return `true` if the data in the MemoryView `view` is row-major contiguous. + + Return `false` otherwise. + +- `bool rb_memory_view_is_column_major_contiguous(const rb_memory_view_t *view)` + + Return `true` if the data in the MemoryView `view` is column-major contiguous. + + Return `false` otherwise. diff --git a/doc/contributing/reporting_issues.md b/doc/contributing/reporting_issues.md index 25516ffc6b..a1a2295712 100644 --- a/doc/contributing/reporting_issues.md +++ b/doc/contributing/reporting_issues.md @@ -52,19 +52,30 @@ your feature it could help persuade Ruby core. Here is a template you can use for a feature proposal: -``` -[Abstract] - Briefly summarize your feature -[Background] - Describe current behavior -[Proposal] - Describe your feature in detail -[Use cases] - Give specific example uses of your feature -[Discussion] - Describe why this feature is necessary and better than using existing features -[See also] - Link to other related resources (such as implementations in other languages) +```markdown +# Abstract + +Briefly summarize your feature + +# Background + +Describe current behavior + +# Proposal + +Describe your feature in detail + +# Use cases + +Give specific example uses of your feature + +# Discussion + +Describe why this feature is necessary and better than using existing features + +# See also + +Link to other related resources (such as implementations in other languages) ``` ## Backport requests diff --git a/doc/contributing/testing_ruby.md b/doc/contributing/testing_ruby.md index 6247686efc..4c7ce7f6a8 100644 --- a/doc/contributing/testing_ruby.md +++ b/doc/contributing/testing_ruby.md @@ -1,5 +1,9 @@ # Testing Ruby +All the commands below assume that you're running them from the `build/` directory made during [Building Ruby](building_ruby.md). + +Most commands below should work with [GNU make](https://www.gnu.org/software/make/) (the default on Linux and macOS), [BSD make](https://man.freebsd.org/cgi/man.cgi?make(1)) and [NMAKE](https://learn.microsoft.com/en-us/cpp/build/reference/nmake-reference), except where indicated otherwise. + ## Test suites There are several test suites in the Ruby codebase: @@ -8,131 +12,144 @@ We can run any of the make scripts [in parallel](building_ruby.md#label-Running+ 1. [bootstraptest/](https://github.com/ruby/ruby/tree/master/bootstraptest) - This is a small test suite that runs on Miniruby (see [building Ruby](building_ruby.md#label-Miniruby+vs+Ruby)). We can run it with: + This is a small test suite that runs on [Miniruby](building_ruby.md#label-Miniruby+vs+Ruby). We can run it with: - ``` + ```sh make btest ``` - To run it with logs, we can use: + To run individual bootstrap tests, we can either specify a list of filenames or use the `--sets` flag in the variable `BTESTS`: - ``` - make btest OPTS=-v + ```sh + make btest BTESTS="../bootstraptest/test_string.rb ../bootstraptest/test_class.rb" + make btest BTESTS="--sets=string,class" ``` - To run individual bootstrap tests, we can either specify a list of filenames or use the `--sets` flag in the variable `BTESTS`: + To run these tests with verbose logging, we can add `-v` to the `OPTS`: - ``` - make btest BTESTS="bootstraptest/test_fork.rb bootstraptest/tes_gc.rb" - make btest BTESTS="--sets=fork,gc" + ```sh + make btest OPTS="--sets=string,class -v" ``` If we want to run the bootstrap test suite on Ruby (not Miniruby), we can use: - ``` + ```sh make test ``` - To run it with logs, we can use: + To run these tests with verbose logging, we can add `-v` to the `OPTS`: - ``` + ```sh make test OPTS=-v ``` - To run a file or directory with GNU make, we can use: + (GNU make only) To run a specific file, we can use: + ```sh + make ../test/ruby/test_string.rb ``` - make test/ruby/test_foo.rb - make test/ruby/test_foo.rb TESTOPTS="-n /test_bar/" + + You can use the `-n` test option to run a specific test with a regex: + + ```sh + make ../test/ruby/test_string.rb TESTOPTS="-n /test_.*_to_s/" ``` 2. [test/](https://github.com/ruby/ruby/tree/master/test) This is a more comprehensive test suite that runs on Ruby. We can run it with: - ``` + ```sh make test-all ``` - We can run a specific test directory in this suite using the `TESTS` option, for example: + We can run a specific test file or directory in this suite using the `TESTS` option, for example: - ``` - make test-all TESTS=test/rubygems - ``` - - We can run a specific test file in this suite by also using the `TESTS` option, for example: - - ``` - make test-all TESTS=test/ruby/test_array.rb + ```sh + make test-all TESTS="../test/ruby/" + make test-all TESTS="../test/ruby/test_string.rb" ``` We can run a specific test in this suite using the `TESTS` option, specifying first the file name, and then the test name, prefixed with `--name`. For example: - ``` - make test-all TESTS="../test/ruby/test_alias.rb --name=/test_alias_with_zsuper_method/" + ```sh + make test-all TESTS="../test/ruby/test_string.rb --name=TestString#test_to_s" ``` - To run these specs with logs, we can use: + To run these tests with verbose logging, we can add `-v` to `TESTS`: - ``` + ```sh make test-all TESTS=-v ``` - If we would like to run both the `test/` and `bootstraptest/` test suites, we can run + We can display the help of the `TESTS` option: + ```sh + make test-all TESTS=--help ``` + + We can run all the tests in `test/`, `bootstraptest/` and `spec/` (the `spec/` is explained in a later section) all together with: + + ```sh make check ``` 3. [spec/ruby](https://github.com/ruby/ruby/tree/master/spec/ruby) - This is a test suite that exists in [the Ruby spec repository](https://github.com/ruby/spec) and is mirrored into the `spec/ruby` directory in the Ruby repository. It tests the behavior of the Ruby programming language. We can run this using: + This is a test suite defined in [the Ruby spec repository](https://github.com/ruby/spec), and is periodically mirrored into the `spec/ruby` directory of this repository. It tests the behavior of the Ruby programming language. We can run this using: - ``` + ```sh make test-spec ``` - To run a specific directory, we can use `MSPECOPT` to specify the directory: + We can run a specific test file or directory in this suite using the `SPECOPTS` option, for example: - ``` - make test-spec MSPECOPT=spec/ruby/core/array - ``` - - To run a specific file, we can also use `MSPECOPT` to specify the file: - - ``` - make test-spec MSPECOPT=spec/ruby/core/array/any_spec.rb + ```sh + make test-spec SPECOPTS="../spec/ruby/core/string/" + make test-spec SPECOPTS="../spec/ruby/core/string/to_s_spec.rb" ``` To run a specific test, we can use the `--example` flag to match against the test name: - ``` - make test-spec MSPECOPT="../spec/ruby/core/array/any_spec.rb --example='is false if the array is empty'" + ```sh + make test-spec SPECOPTS="../spec/ruby/core/string/to_s_spec.rb --example='returns self when self.class == String'" ``` - To run these specs with logs, we can use: + To run these specs with verbose logging, we can add `-v` to the `SPECOPTS`: - ``` - make test-spec MSPECOPT=-Vfs + ```sh + make test-spec SPECOPTS="../spec/ruby/core/string/to_s_spec.rb -Vfs" ``` - To run a ruby-spec file or directory with GNU make, we can use + (GNU make only) To run a ruby-spec file or directory, we can use - ``` - make spec/ruby/core/foo/bar_spec.rb + ```sh + make ../spec/ruby/core/string/to_s_spec.rb ``` 4. [spec/bundler](https://github.com/ruby/ruby/tree/master/spec/bundler) - The bundler test suite exists in [the RubyGems repository](https://github.com/rubygems/rubygems/tree/master/bundler/spec) and is mirrored into the `spec/bundler` directory in the Ruby repository. We can run this using: + The bundler test suite is defined in [the RubyGems repository](https://github.com/rubygems/rubygems/tree/master/bundler/spec), and is periodically mirrored into the `spec/ruby` directory of this repository. We can run this using: - ``` + ```sh make test-bundler ``` To run a specific bundler spec file, we can use `BUNDLER_SPECS` as follows: + ```sh + make test-bundler BUNDLER_SPECS=commands/exec_spec.rb ``` - $ make test-bundler BUNDLER_SPECS=commands/exec_spec.rb - ``` + +## Troubleshooting + +### Running test suites on s390x CPU Architecture + +If we see failing tests related to the zlib library on s390x CPU architecture, we can run the test suites with `DFLTCC=0` to pass: + +```sh +DFLTCC=0 make check +``` + +The failures can happen with the zlib library applying the patch [madler/zlib#410](https://github.com/madler/zlib/pull/410) to enable the deflate algorithm producing a different compressed byte stream. We manage this issue at [[ruby-core:114942][Bug #19909]](https://bugs.ruby-lang.org/issues/19909). diff --git a/doc/contributing/vm_stack_and_frames.md b/doc/contributing/vm_stack_and_frames.md new file mode 100644 index 0000000000..c7dc59db16 --- /dev/null +++ b/doc/contributing/vm_stack_and_frames.md @@ -0,0 +1,163 @@ +# Ruby VM Stack and Frame Layout + +This document explains the Ruby VM stack architecture, including how the value +stack (SP) and control frames (CFP) share a single contiguous memory region, +and how individual frames are structured. + +## VM Stack Architecture + +The Ruby VM uses a single contiguous stack (`ec->vm_stack`) with two different +regions growing toward each other. Understanding this requires distinguishing +the overall architecture (how CFPs and values share one stack) from individual +frame internals (how values are organized for one single frame). + +```text +High addresses (ec->vm_stack + ec->vm_stack_size) + ↓ + [CFP region starts here] ← RUBY_VM_END_CONTROL_FRAME(ec) + [CFP - 1] New frame pushed here (grows downward) + [CFP - 2] Another frame + ... + + (Unused space - stack overflow when they meet) + + ... Value stack grows UP toward higher addresses + [SP + n] Values pushed here + [ec->cfp->sp] Current executing frame's stack pointer + ↑ +Low addresses (ec->vm_stack) +``` + +The "unused space" represents free space available for new frames and values. When this gap closes (CFP meets SP), stack overflow occurs. + +### Stack Growth Directions + +**Control Frames (CFP):** + +- Start at `ec->vm_stack + ec->vm_stack_size` (high addresses) +- Grow **downward** toward lower addresses as frames are pushed +- Each new frame is allocated at `cfp - 1` (lower address) +- The `rb_control_frame_t` structure itself moves downward + +**Value Stack (SP):** + +- Starts at `ec->vm_stack` (low addresses) +- Grows **upward** toward higher addresses as values are pushed +- Each frame's `cfp->sp` points to the top of its value stack + +### Stack Overflow + +When recursive calls push too many frames, CFP grows downward until it collides +with SP growing upward. The VM detects this with `CHECK_VM_STACK_OVERFLOW0`, +which computes `const rb_control_frame_struct *bound = (void *)&sp[margin];` +and raises if `cfp <= &bound[1]`. + +## Understanding Individual Frame Value Stacks + +Each frame has its own portion of the overall VM stack, called its "VM value stack" +or simply "value stack". This space is pre-allocated when the frame is created, +with size determined by: + +- `local_size` - space for local variables +- `stack_max` - maximum depth for temporary values during execution + +The frame's value stack grows upward from its base (where self/arguments/locals +live) toward `cfp->sp` (the current top of temporary values). + +## Visualizing How Frames Fit in the VM Stack + +The left side shows the overall VM stack with CFP metadata separated from frame +values. The right side zooms into one frame's value region, revealing its internal +structure. + +```text +Overall VM Stack (ec->vm_stack): Zooming into Frame 2's value stack: + +High addr (vm_stack + vm_stack_size) High addr (cfp->sp) + ↓ ┌ + [CFP 1 metadata] │ [Temporaries] + [CFP 2 metadata] ─────────┐ │ [Env: Flags/Block/CME] ← cfp->ep + [CFP 3 metadata] │ │ [Locals] + ──────────────── │ ┌─┤ [Arguments] + (unused space) │ │ │ [self] + ──────────────── │ │ └ + [Frame 3 values] │ │ Low addr (frame base) + [Frame 2 values] <────────┴───────┘ + [Frame 1 values] + ↑ +Low addr (vm_stack) +``` + +## Examining a Single Frame's Value Stack + +Now let's walk through a concrete Ruby program to see how a single frame's +value stack is structured internally: + +```ruby +def foo(x, y) + z = x.casecmp(y) +end + +foo(:one, :two) +``` + +First, after arguments are evaluated and right before the `send` to `foo`: + +```text + ┌────────────┐ + putself │ :two │ + putobject :one 0x2 ├────────────┤ + putobject :two │ :one │ +► send <:foo, argc:2> 0x1 ├────────────┤ + leave │ self │ + 0x0 └────────────┘ +``` + +The `put*` instructions have pushed 3 items onto the stack. It's now time to +add a new control frame for `foo`. The following is the shape of the stack +after one instruction in `foo`: + +```text + cfp->sp=0x8 at this point. + 0x8 ┌────────────┐◄──Stack space for temporaries + │ :one │ live above the environment. + 0x7 ├────────────┤ + getlocal x@0 │ < flags > │ foo's rb_control_frame_t +► getlocal y@1 0x6 ├────────────┤◄──has cfp->ep=0x6 + send <:casecmp, argc:1> │ <no block> │ + dup 0x5 ├────────────┤ The flags, block, and CME triple + setlocal z@2 │ <CME: foo> │ (VM_ENV_DATA_SIZE) form an + leave 0x4 ├────────────┤ environment. They can be used to + │ z (nil) │ figure out what local variables + 0x3 ├────────────┤ are below them. + │ :two │ + 0x2 ├────────────┤ Notice how the arguments, now + │ :one │ locals, never moved. This layout + 0x1 ├────────────┤ allows for argument transfer + │ self │ without copying. + 0x0 └────────────┘ +``` + +Given that locals have lower address than `cfp->ep`, it makes sense then that +`getlocal` in `insns.def` has `val = *(vm_get_ep(GET_EP(), level) - idx);`. +When accessing variables in the immediate scope, where `level=0`, it's +essentially `val = cfp->ep[-idx];`. + +Note that this EP-relative index has a different basis than the index that comes +after "@" in disassembly listings. The "@" index is relative to the 0th local +(`x` in this case). + +### Q&A + +Q: It seems that the receiver is always at an offset relative to EP, + like locals. Couldn't we use EP to access it instead of using `cfp->self`? + +A: Not all calls put the `self` in the callee on the stack. Two + examples are `Proc#call`, where the receiver is the Proc object, but `self` + inside the callee is `Proc#receiver`, and `yield`, where the receiver isn't + pushed onto the stack before the arguments. + +Q: Why have `cfp->ep` when it seems that everything is below `cfp->sp`? + +A: In the example, `cfp->ep` points to the stack, but it can also point to the + GC heap. Blocks can capture and evacuate their environment to the heap. |