<feed xmlns='http://www.w3.org/2005/Atom'>
<title>ruby.git/test/ruby/test_ast.rb, branch v3_2_11</title>
<subtitle>The Ruby Programming Language</subtitle>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/'/>
<entry>
<title>merge revision(s) 979dd02e2f0a70f3f30d47c6115780d8fac6147e: [Backport #19262]</title>
<updated>2023-01-19T02:05:29+00:00</updated>
<author>
<name>NARUSE, Yui</name>
<email>naruse@airemix.jp</email>
</author>
<published>2023-01-19T02:05:29+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=1a2447ac9925e7794a7c5aa07960ad4de08e6aca'/>
<id>1a2447ac9925e7794a7c5aa07960ad4de08e6aca</id>
<content type='text'>
	Check if the argument is Thread::Backtrace::Location object

	[Bug #19262]
	---
	 ast.c                 | 5 +++++
	 test/ruby/test_ast.rb | 6 ++++++
	 2 files changed, 11 insertions(+)
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
	Check if the argument is Thread::Backtrace::Location object

	[Bug #19262]
	---
	 ast.c                 | 5 +++++
	 test/ruby/test_ast.rb | 6 ++++++
	 2 files changed, 11 insertions(+)
</pre>
</div>
</content>
</entry>
<entry>
<title>Disallow mixed usage of ... and */**</title>
<updated>2022-12-15T09:56:24+00:00</updated>
<author>
<name>Shugo Maeda</name>
<email>shugo@ruby-lang.org</email>
</author>
<published>2022-12-15T06:12:33+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=2581de112c1957dc4b5852e54337551dc8972c99'/>
<id>2581de112c1957dc4b5852e54337551dc8972c99</id>
<content type='text'>
[Feature #19134]
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
[Feature #19134]
</pre>
</div>
</content>
</entry>
<entry>
<title>Add a test case for argument forwarding</title>
<updated>2022-11-29T09:23:00+00:00</updated>
<author>
<name>yui-knk</name>
<email>spiketeika@gmail.com</email>
</author>
<published>2022-11-29T02:42:53+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=67ae3e97388bc9083cb3cc2f22ba8abc28d6fda3'/>
<id>67ae3e97388bc9083cb3cc2f22ba8abc28d6fda3</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Enhance keep_tokens option for RubyVM::AbstractSyntaxTree parsing methods</title>
<updated>2022-11-21T00:01:34+00:00</updated>
<author>
<name>yui-knk</name>
<email>spiketeika@gmail.com</email>
</author>
<published>2022-09-23T13:40:02+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=d8601621edcf29e3323b90dcf04b774edd9fb45e'/>
<id>d8601621edcf29e3323b90dcf04b774edd9fb45e</id>
<content type='text'>
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.

This commit adds these methods.

* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.

[Feature #19070]

Impacts on memory usage and performance are below:

Memory usage:

```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)

$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb

# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb

# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```

Performance:

```
$ cat ../ast_keep_tokens.yml
prelude: |
  src = &lt;&lt;~SRC
    module M
      class C
        def m1(a, b)
          1 + a + b
        end
      end
    end
  SRC
benchmark:
  without_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
  with_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)

$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
            --executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
            --executables="built-ruby::./miniruby -I../lib -I. -I.ext/common  ../tool/runruby.rb --extout=.ext  -- --disable-gems --disable-gem" \
            --output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..

|                     |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens  |     21.659k|   21.303k|
|                     |       1.02x|         -|
|with_keep_tokens     |      6.220k|    5.691k|
|                     |       1.09x|         -|
```
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.

This commit adds these methods.

* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.

[Feature #19070]

Impacts on memory usage and performance are below:

Memory usage:

```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)

$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb

# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb

# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```

Performance:

```
$ cat ../ast_keep_tokens.yml
prelude: |
  src = &lt;&lt;~SRC
    module M
      class C
        def m1(a, b)
          1 + a + b
        end
      end
    end
  SRC
benchmark:
  without_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
  with_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)

$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
            --executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
            --executables="built-ruby::./miniruby -I../lib -I. -I.ext/common  ../tool/runruby.rb --extout=.ext  -- --disable-gems --disable-gem" \
            --output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..

|                     |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens  |     21.659k|   21.303k|
|                     |       1.02x|         -|
|with_keep_tokens     |      6.220k|    5.691k|
|                     |       1.09x|         -|
```
</pre>
</div>
</content>
</entry>
<entry>
<title>Make anonymous rest arg (*) and block arg (&amp;) accessible from ARGS node</title>
<updated>2022-11-18T09:25:42+00:00</updated>
<author>
<name>yui-knk</name>
<email>spiketeika@gmail.com</email>
</author>
<published>2022-11-17T14:43:21+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=f0ce1186620273a1182e6084559765143099eb88'/>
<id>f0ce1186620273a1182e6084559765143099eb88</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Allow anonymous keyword rest parameter with other keyword parameters</title>
<updated>2022-11-18T09:23:06+00:00</updated>
<author>
<name>Shugo Maeda</name>
<email>shugo@ruby-lang.org</email>
</author>
<published>2022-11-16T14:03:39+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=ddd62fadaf91418cd259593285bc59358fb0b166'/>
<id>ddd62fadaf91418cd259593285bc59358fb0b166</id>
<content type='text'>
Fixes [Bug #19132]
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Fixes [Bug #19132]
</pre>
</div>
</content>
</entry>
<entry>
<title>Add `node_id_for_backtrace_location` function</title>
<updated>2022-10-31T04:39:56+00:00</updated>
<author>
<name>eileencodes</name>
<email>eileencodes@gmail.com</email>
</author>
<published>2022-10-19T17:23:53+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=3391c51effcd61f9a718adf59740661d99f57b5b'/>
<id>3391c51effcd61f9a718adf59740661d99f57b5b</id>
<content type='text'>
We want to use error highlight with eval'd code, specifically ERB
templates. We're able to recover the generated code for eval'd templates
and can get a parse tree for the ERB generated code, but we don't have a
way to get the node id from the backtrace location. So we can't pass the
right node into error highlight.

This patch gives us an API to get the node id from the backtrace
location so we can find the node in the AST.

Error Highlight PR: https://github.com/ruby/error_highlight/pull/26

Co-authored-by: Aaron Patterson &lt;tenderlove@ruby-lang.org&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
We want to use error highlight with eval'd code, specifically ERB
templates. We're able to recover the generated code for eval'd templates
and can get a parse tree for the ERB generated code, but we don't have a
way to get the node id from the backtrace location. So we can't pass the
right node into error highlight.

This patch gives us an API to get the node id from the backtrace
location so we can find the node in the AST.

Error Highlight PR: https://github.com/ruby/error_highlight/pull/26

Co-authored-by: Aaron Patterson &lt;tenderlove@ruby-lang.org&gt;
</pre>
</div>
</content>
</entry>
<entry>
<title>Suppress warnings in test/ruby/test_ast.rb</title>
<updated>2022-10-12T12:48:28+00:00</updated>
<author>
<name>Yusuke Endoh</name>
<email>mame@ruby-lang.org</email>
</author>
<published>2022-10-12T12:48:28+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=9cfc45d6e52cf4ffde0375f8d3f2271c36e9e91c'/>
<id>9cfc45d6e52cf4ffde0375f8d3f2271c36e9e91c</id>
<content type='text'>
The tests for error tolerance printed some warnings. This change
suppresses them.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The tests for error tolerance printed some warnings. This change
suppresses them.
</pre>
</div>
</content>
</entry>
<entry>
<title>Do not set `$!` to `SyntaxError` when error tolerant</title>
<updated>2022-10-09T10:07:21+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2022-10-09T08:43:16+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=546566d34b37fb5a028bcbe6bafa8248877f06ca'/>
<id>546566d34b37fb5a028bcbe6bafa8248877f06ca</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Extract `assert_error_tolerant`</title>
<updated>2022-10-09T10:05:31+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2022-10-09T09:01:54+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=d9e2ef6000a7a7621e110a7092f09e273eb81fcb'/>
<id>d9e2ef6000a7a7621e110a7092f09e273eb81fcb</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
</feed>
