<feed xmlns='http://www.w3.org/2005/Atom'>
<title>ruby.git/internal/parse.h, branch v3_3_11</title>
<subtitle>The Ruby Programming Language</subtitle>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/'/>
<entry>
<title>Shorten `rb_strterm_literal_t` members</title>
<updated>2023-10-14T02:08:43+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2023-10-13T10:13:26+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=5fc9810bf3a0ac6eb9964ff30b0e1060270ab597'/>
<id>5fc9810bf3a0ac6eb9964ff30b0e1060270ab597</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Manage `rb_strterm_t` without imemo</title>
<updated>2023-10-14T02:08:43+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2023-10-13T10:07:31+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=a075c55d0c29fa5c99c5a3b6f25540cf14ddaefd'/>
<id>a075c55d0c29fa5c99c5a3b6f25540cf14ddaefd</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Remove unions in `rb_strterm` structs for alignment</title>
<updated>2023-10-14T02:08:43+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2023-10-12T18:33:48+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=cb06b6632a0d6e33522d75cab7836ff457a452b5'/>
<id>cb06b6632a0d6e33522d75cab7836ff457a452b5</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Move SCRIPT_LINES__ away from parse.y</title>
<updated>2023-08-25T09:23:05+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2023-08-25T07:53:29+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=6aa16f9ec14a296f549dc955774ad2293d1c54d6'/>
<id>6aa16f9ec14a296f549dc955774ad2293d1c54d6</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Fix duplicate symbol errors when statically linking ripper</title>
<updated>2023-06-12T11:22:01+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2023-06-12T11:06:19+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=81836c6cb9fceb4cc8e9e95317a6cbb3d0ad181a'/>
<id>81836c6cb9fceb4cc8e9e95317a6cbb3d0ad181a</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[Feature #19719] Universal Parser</title>
<updated>2023-06-12T09:23:48+00:00</updated>
<author>
<name>yui-knk</name>
<email>spiketeika@gmail.com</email>
</author>
<published>2023-05-28T11:00:20+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=b481b673d753339204290d7582dbb91a6e14447a'/>
<id>b481b673d753339204290d7582dbb91a6e14447a</id>
<content type='text'>
Introduce Universal Parser mode for the parser.
This commit includes these changes:

* Introduce `UNIVERSAL_PARSER` macro. All of CRuby related functions
  are passed via `struct rb_parser_config_struct` when this macro is enabled.
* Add CI task with 'cppflags=-DUNIVERSAL_PARSER' for ubuntu.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Introduce Universal Parser mode for the parser.
This commit includes these changes:

* Introduce `UNIVERSAL_PARSER` macro. All of CRuby related functions
  are passed via `struct rb_parser_config_struct` when this macro is enabled.
* Add CI task with 'cppflags=-DUNIVERSAL_PARSER' for ubuntu.
</pre>
</div>
</content>
</entry>
<entry>
<title>Revert "reuse open(2) from rb_file_load_ok on POSIX-like system"</title>
<updated>2023-02-27T17:24:45+00:00</updated>
<author>
<name>Takashi Kokubun</name>
<email>takashikkbn@gmail.com</email>
</author>
<published>2023-02-27T17:24:45+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=526111290b2e01e798f436dfe4acc3bf10c6bbd1'/>
<id>526111290b2e01e798f436dfe4acc3bf10c6bbd1</id>
<content type='text'>
This reverts commit 35136e1e9c232ad7a03407b992b2e86b6df43f63.

test-spec has been failing since this revision.

.github/workflows/compilers.yml:82
https://github.com/ruby/ruby/actions/runs/4276884159/jobs/7445299562

```
env:
  # Minimal flags to pass the check.
  default_cc: 'gcc-11 -fcf-protection -Wa,--generate-missing-build-notes=yes'
  optflags: '-O2'
  LDFLAGS: '-Wl,-z,now'
  # FIXME: Drop skipping options
  # https://bugs.ruby-lang.org/issues/18061
  # https://sourceware.org/annobin/annobin.html/Test-pie.html
  TEST_ANNOCHECK_OPTS: "--skip-pie --skip-gaps"
```

Failure:

```
  1)
  An exception occurred during: Kernel#require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:317
  Kernel#require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
  LeakError: Leaked file descriptor: 8 : #&lt;File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `&lt;top (required)&gt;'

  2)
  An exception occurred during: Kernel#require ($LOADED_FEATURES) stores an absolute path
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:330
  Kernel#require ($LOADED_FEATURES) stores an absolute path ERROR
  LeakError: Closed file descriptor: 8
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `&lt;top (required)&gt;'

  3)
  An exception occurred during: Kernel#require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:535
  Kernel#require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded ERROR
  LeakError: Leaked file descriptor: 8 : #&lt;File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `&lt;top (required)&gt;'

  4)
  An exception occurred during: Kernel#require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:551
  Kernel#require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded ERROR
  LeakError: Leaked file descriptor: 9 : #&lt;File:../code/load_fixture.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `&lt;top (required)&gt;'

  5)
  An exception occurred during: Kernel#require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:563
  Kernel#require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required ERROR
  LeakError: Closed file descriptor: 8
  Closed file descriptor: 9
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `&lt;top (required)&gt;'

  6)
  An exception occurred during: Kernel.require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:317
  Kernel.require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
  LeakError: Leaked file descriptor: 8 : #&lt;File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `&lt;top (required)&gt;'

  7)
  An exception occurred during: Kernel.require ($LOADED_FEATURES) stores an absolute path
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:330
  Kernel.require ($LOADED_FEATURES) stores an absolute path ERROR
  LeakError: Closed file descriptor: 8
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `&lt;top (required)&gt;'

  8)
  An exception occurred during: Kernel.require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:535
  Kernel.require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded ERROR
  LeakError: Leaked file descriptor: 8 : #&lt;File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `&lt;top (required)&gt;'

  9)
  An exception occurred during: Kernel.require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:551
  Kernel.require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded ERROR
  LeakError: Leaked file descriptor: 9 : #&lt;File:../code/load_fixture.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `&lt;top (required)&gt;'

  10)
  An exception occurred during: Kernel.require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:563
  Kernel.require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required ERROR
  LeakError: Closed file descriptor: 8
  Closed file descriptor: 9
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `&lt;top (required)&gt;'

  11)
  An exception occurred during: Kernel#require_relative with a relative path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:197
  Kernel#require_relative with a relative path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
  LeakError: Leaked file descriptor: 8 : #&lt;File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:4:in `&lt;top (required)&gt;'

  12)
  An exception occurred during: Kernel#require_relative with a relative path ($LOADED_FEATURES) stores an absolute path
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:205
  Kernel#require_relative with a relative path ($LOADED_FEATURES) stores an absolute path ERROR
  LeakError: Closed file descriptor: 8
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:4:in `&lt;top (required)&gt;'

  13)
  An exception occurred during: Kernel#require_relative with an absolute path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:399
  Kernel#require_relative with an absolute path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
  LeakError: Leaked file descriptor: 8 : #&lt;File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:277:in `&lt;top (required)&gt;'

  14)
  An exception occurred during: Kernel#require_relative with an absolute path ($LOAD_FEATURES) stores an absolute path
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:407
  Kernel#require_relative with an absolute path ($LOAD_FEATURES) stores an absolute path ERROR
  LeakError: Closed file descriptor: 8
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:277:in `&lt;top (required)&gt;'
```
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This reverts commit 35136e1e9c232ad7a03407b992b2e86b6df43f63.

test-spec has been failing since this revision.

.github/workflows/compilers.yml:82
https://github.com/ruby/ruby/actions/runs/4276884159/jobs/7445299562

```
env:
  # Minimal flags to pass the check.
  default_cc: 'gcc-11 -fcf-protection -Wa,--generate-missing-build-notes=yes'
  optflags: '-O2'
  LDFLAGS: '-Wl,-z,now'
  # FIXME: Drop skipping options
  # https://bugs.ruby-lang.org/issues/18061
  # https://sourceware.org/annobin/annobin.html/Test-pie.html
  TEST_ANNOCHECK_OPTS: "--skip-pie --skip-gaps"
```

Failure:

```
  1)
  An exception occurred during: Kernel#require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:317
  Kernel#require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
  LeakError: Leaked file descriptor: 8 : #&lt;File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `&lt;top (required)&gt;'

  2)
  An exception occurred during: Kernel#require ($LOADED_FEATURES) stores an absolute path
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:330
  Kernel#require ($LOADED_FEATURES) stores an absolute path ERROR
  LeakError: Closed file descriptor: 8
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `&lt;top (required)&gt;'

  3)
  An exception occurred during: Kernel#require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:535
  Kernel#require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded ERROR
  LeakError: Leaked file descriptor: 8 : #&lt;File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `&lt;top (required)&gt;'

  4)
  An exception occurred during: Kernel#require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:551
  Kernel#require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded ERROR
  LeakError: Leaked file descriptor: 9 : #&lt;File:../code/load_fixture.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `&lt;top (required)&gt;'

  5)
  An exception occurred during: Kernel#require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:563
  Kernel#require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required ERROR
  LeakError: Closed file descriptor: 8
  Closed file descriptor: 9
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `&lt;top (required)&gt;'

  6)
  An exception occurred during: Kernel.require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:317
  Kernel.require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
  LeakError: Leaked file descriptor: 8 : #&lt;File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `&lt;top (required)&gt;'

  7)
  An exception occurred during: Kernel.require ($LOADED_FEATURES) stores an absolute path
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:330
  Kernel.require ($LOADED_FEATURES) stores an absolute path ERROR
  LeakError: Closed file descriptor: 8
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `&lt;top (required)&gt;'

  8)
  An exception occurred during: Kernel.require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:535
  Kernel.require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded ERROR
  LeakError: Leaked file descriptor: 8 : #&lt;File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `&lt;top (required)&gt;'

  9)
  An exception occurred during: Kernel.require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:551
  Kernel.require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded ERROR
  LeakError: Leaked file descriptor: 9 : #&lt;File:../code/load_fixture.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `&lt;top (required)&gt;'

  10)
  An exception occurred during: Kernel.require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required
  /__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:563
  Kernel.require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required ERROR
  LeakError: Closed file descriptor: 8
  Closed file descriptor: 9
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `&lt;top (required)&gt;'

  11)
  An exception occurred during: Kernel#require_relative with a relative path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:197
  Kernel#require_relative with a relative path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
  LeakError: Leaked file descriptor: 8 : #&lt;File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:4:in `&lt;top (required)&gt;'

  12)
  An exception occurred during: Kernel#require_relative with a relative path ($LOADED_FEATURES) stores an absolute path
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:205
  Kernel#require_relative with a relative path ($LOADED_FEATURES) stores an absolute path ERROR
  LeakError: Closed file descriptor: 8
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:4:in `&lt;top (required)&gt;'

  13)
  An exception occurred during: Kernel#require_relative with an absolute path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:399
  Kernel#require_relative with an absolute path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
  LeakError: Leaked file descriptor: 8 : #&lt;File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb&gt;
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:277:in `&lt;top (required)&gt;'

  14)
  An exception occurred during: Kernel#require_relative with an absolute path ($LOAD_FEATURES) stores an absolute path
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:407
  Kernel#require_relative with an absolute path ($LOAD_FEATURES) stores an absolute path ERROR
  LeakError: Closed file descriptor: 8
  /__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:277:in `&lt;top (required)&gt;'
```
</pre>
</div>
</content>
</entry>
<entry>
<title>reuse open(2) from rb_file_load_ok on POSIX-like system</title>
<updated>2023-02-26T20:39:41+00:00</updated>
<author>
<name>Eric Wong</name>
<email>e@80x24.org</email>
</author>
<published>2023-02-24T18:05:36+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=35136e1e9c232ad7a03407b992b2e86b6df43f63'/>
<id>35136e1e9c232ad7a03407b992b2e86b6df43f63</id>
<content type='text'>
When loading Ruby source files, we can save the result of
successful opens as open(2)/openat(2) are a fairly expensive
syscalls.  This also avoids a time-of-check-to-time-of-use
(TOCTTOU) problem.

This reduces open(2) syscalls during `require'; but should be
most apparent when users have a small $LOAD_PATH.  Users with
large $LOAD_PATH will benefit less since there'll be more
open(2) failures due to ENOENT.

With `strace -c -e openat ruby -e exit' under Linux, this
results in a ~14% reduction of openat(2) syscalls
(glibc uses openat(2) to implement open(2)).

 % time     seconds  usecs/call     calls    errors syscall
 ------ ----------- ----------- --------- --------- ----------------
   0.00    0.000000           0       296       110 openat
   0.00    0.000000           0       254       110 openat

Additionally, the introduction of `struct ruby_file_load_state'
may make future optimizations more apparent.

This change cannot benefit binary (.so) loading since the
dlopen(3) API requires a filename and I'm not aware of an
alternative that takes a pre-existing FD.  In typical
situations, Ruby source files outnumber the mount of .so
files.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
When loading Ruby source files, we can save the result of
successful opens as open(2)/openat(2) are a fairly expensive
syscalls.  This also avoids a time-of-check-to-time-of-use
(TOCTTOU) problem.

This reduces open(2) syscalls during `require'; but should be
most apparent when users have a small $LOAD_PATH.  Users with
large $LOAD_PATH will benefit less since there'll be more
open(2) failures due to ENOENT.

With `strace -c -e openat ruby -e exit' under Linux, this
results in a ~14% reduction of openat(2) syscalls
(glibc uses openat(2) to implement open(2)).

 % time     seconds  usecs/call     calls    errors syscall
 ------ ----------- ----------- --------- --------- ----------------
   0.00    0.000000           0       296       110 openat
   0.00    0.000000           0       254       110 openat

Additionally, the introduction of `struct ruby_file_load_state'
may make future optimizations more apparent.

This change cannot benefit binary (.so) loading since the
dlopen(3) API requires a filename and I'm not aware of an
alternative that takes a pre-existing FD.  In typical
situations, Ruby source files outnumber the mount of .so
files.
</pre>
</div>
</content>
</entry>
<entry>
<title>Enhance keep_tokens option for RubyVM::AbstractSyntaxTree parsing methods</title>
<updated>2022-11-21T00:01:34+00:00</updated>
<author>
<name>yui-knk</name>
<email>spiketeika@gmail.com</email>
</author>
<published>2022-09-23T13:40:02+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=d8601621edcf29e3323b90dcf04b774edd9fb45e'/>
<id>d8601621edcf29e3323b90dcf04b774edd9fb45e</id>
<content type='text'>
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.

This commit adds these methods.

* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.

[Feature #19070]

Impacts on memory usage and performance are below:

Memory usage:

```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)

$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb

# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb

# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```

Performance:

```
$ cat ../ast_keep_tokens.yml
prelude: |
  src = &lt;&lt;~SRC
    module M
      class C
        def m1(a, b)
          1 + a + b
        end
      end
    end
  SRC
benchmark:
  without_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
  with_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)

$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
            --executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
            --executables="built-ruby::./miniruby -I../lib -I. -I.ext/common  ../tool/runruby.rb --extout=.ext  -- --disable-gems --disable-gem" \
            --output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..

|                     |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens  |     21.659k|   21.303k|
|                     |       1.02x|         -|
|with_keep_tokens     |      6.220k|    5.691k|
|                     |       1.09x|         -|
```
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.

This commit adds these methods.

* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.

[Feature #19070]

Impacts on memory usage and performance are below:

Memory usage:

```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)

$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb

# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb

# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```

Performance:

```
$ cat ../ast_keep_tokens.yml
prelude: |
  src = &lt;&lt;~SRC
    module M
      class C
        def m1(a, b)
          1 + a + b
        end
      end
    end
  SRC
benchmark:
  without_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
  with_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)

$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
            --executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
            --executables="built-ruby::./miniruby -I../lib -I. -I.ext/common  ../tool/runruby.rb --extout=.ext  -- --disable-gems --disable-gem" \
            --output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..

|                     |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens  |     21.659k|   21.303k|
|                     |       1.02x|         -|
|with_keep_tokens     |      6.220k|    5.691k|
|                     |       1.09x|         -|
```
</pre>
</div>
</content>
</entry>
<entry>
<title>Add error_tolerant option to RubyVM::AST</title>
<updated>2022-10-08T08:59:11+00:00</updated>
<author>
<name>yui-knk</name>
<email>spiketeika@gmail.com</email>
</author>
<published>2022-09-25T08:53:44+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=fbbdbdd8911ffb24d98bb71c7c33d24609ce7dfe'/>
<id>fbbdbdd8911ffb24d98bb71c7c33d24609ce7dfe</id>
<content type='text'>
If this option is enabled, SyntaxError is not raised and Node is
returned even if passed script is broken.

[Feature #19013]
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
If this option is enabled, SyntaxError is not raised and Node is
returned even if passed script is broken.

[Feature #19013]
</pre>
</div>
</content>
</entry>
</feed>
