<feed xmlns='http://www.w3.org/2005/Atom'>
<title>ruby.git/ast.rb, branch v4.0.3</title>
<subtitle>The Ruby Programming Language</subtitle>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/'/>
<entry>
<title>[DOC] Fix typo in related class reference</title>
<updated>2024-12-22T06:07:56+00:00</updated>
<author>
<name>zverok</name>
<email>zverok.offline@gmail.com</email>
</author>
<published>2024-12-21T18:23:43+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=e76d2fddad7829ad5e2d36455b68e1025bbfbf33'/>
<id>e76d2fddad7829ad5e2d36455b68e1025bbfbf33</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Update recommendations regarding parsing Ruby code now that Prism is stable</title>
<updated>2024-09-27T14:45:36+00:00</updated>
<author>
<name>Benoit Daloze</name>
<email>eregontp@gmail.com</email>
</author>
<published>2024-09-27T14:45:36+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=b873787a42e4ec23dab3dbc5dded95c4804472d2'/>
<id>b873787a42e4ec23dab3dbc5dded95c4804472d2</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[Feature #20624] Enhance `RubyVM::AbstractSyntaxTree::Node#locations`</title>
<updated>2024-07-23T03:36:00+00:00</updated>
<author>
<name>yui-knk</name>
<email>spiketeika@gmail.com</email>
</author>
<published>2024-07-10T13:28:22+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=f23485a8d693cb69fd7b84c1ab93cb4198ecfe4a'/>
<id>f23485a8d693cb69fd7b84c1ab93cb4198ecfe4a</id>
<content type='text'>
This commit introduce `RubyVM::AbstractSyntaxTree::Node#locations` method
and `RubyVM::AbstractSyntaxTree::Location` class.

Ruby AST node will hold multiple locations information.
`RubyVM::AbstractSyntaxTree::Node#locations` provides a way to access
these locations information.

`RubyVM::AbstractSyntaxTree::Location` is a class which holds these location information:

* `#first_lineno`
* `#first_column`
* `#last_lineno`
* `#last_column`
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This commit introduce `RubyVM::AbstractSyntaxTree::Node#locations` method
and `RubyVM::AbstractSyntaxTree::Location` class.

Ruby AST node will hold multiple locations information.
`RubyVM::AbstractSyntaxTree::Node#locations` provides a way to access
these locations information.

`RubyVM::AbstractSyntaxTree::Location` is a class which holds these location information:

* `#first_lineno`
* `#first_column`
* `#last_lineno`
* `#last_column`
</pre>
</div>
</content>
</entry>
<entry>
<title>ast.rb: Fix bug for source of multibyte characters</title>
<updated>2023-11-04T15:54:22+00:00</updated>
<author>
<name>alpaca-tc</name>
<email>alpaca-tc@alpaca.tc</email>
</author>
<published>2023-11-04T14:04:51+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=8ed733f8f22cdc56a9d2694078871cfc401cb029'/>
<id>8ed733f8f22cdc56a9d2694078871cfc401cb029</id>
<content type='text'>
first_column and last_column return byte positions, but existing implementations
did not consider multibyte.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
first_column and last_column return byte positions, but existing implementations
did not consider multibyte.
</pre>
</div>
</content>
</entry>
<entry>
<title>Move SCRIPT_LINES__ away from parse.y</title>
<updated>2023-08-25T09:23:05+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2023-08-25T07:53:29+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=6aa16f9ec14a296f549dc955774ad2293d1c54d6'/>
<id>6aa16f9ec14a296f549dc955774ad2293d1c54d6</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Fix spelling (#7389)</title>
<updated>2023-02-27T17:56:06+00:00</updated>
<author>
<name>John Bampton</name>
<email>jbampton@users.noreply.github.com</email>
</author>
<published>2023-02-27T17:56:06+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=2f7270c68150d9f49b48c8ff995f4f343d216cee'/>
<id>2f7270c68150d9f49b48c8ff995f4f343d216cee</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[DOC] Document AST.parse's keyword arguments</title>
<updated>2022-12-24T01:52:20+00:00</updated>
<author>
<name>zverok</name>
<email>zverok.offline@gmail.com</email>
</author>
<published>2022-12-22T21:29:44+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=640dca9ae72e55028af812cc041bdaed1f5a2af4'/>
<id>640dca9ae72e55028af812cc041bdaed1f5a2af4</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[DOC] Fix AST documents</title>
<updated>2022-11-25T10:42:47+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2022-11-25T10:42:47+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=e9d6d2a57949541fd8775832b9ee55cf0df67006'/>
<id>e9d6d2a57949541fd8775832b9ee55cf0df67006</id>
<content type='text'>
- Fix indents of `tokens`, to make the contents of Token a list
- Move the example of `tokens` to separate from the above list
- Markup keyword argument and method name
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
- Fix indents of `tokens`, to make the contents of Token a list
- Move the example of `tokens` to separate from the above list
- Markup keyword argument and method name
</pre>
</div>
</content>
</entry>
<entry>
<title>Enhance keep_tokens option for RubyVM::AbstractSyntaxTree parsing methods</title>
<updated>2022-11-21T00:01:34+00:00</updated>
<author>
<name>yui-knk</name>
<email>spiketeika@gmail.com</email>
</author>
<published>2022-09-23T13:40:02+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=d8601621edcf29e3323b90dcf04b774edd9fb45e'/>
<id>d8601621edcf29e3323b90dcf04b774edd9fb45e</id>
<content type='text'>
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.

This commit adds these methods.

* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.

[Feature #19070]

Impacts on memory usage and performance are below:

Memory usage:

```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)

$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb

# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb

# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```

Performance:

```
$ cat ../ast_keep_tokens.yml
prelude: |
  src = &lt;&lt;~SRC
    module M
      class C
        def m1(a, b)
          1 + a + b
        end
      end
    end
  SRC
benchmark:
  without_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
  with_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)

$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
            --executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
            --executables="built-ruby::./miniruby -I../lib -I. -I.ext/common  ../tool/runruby.rb --extout=.ext  -- --disable-gems --disable-gem" \
            --output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..

|                     |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens  |     21.659k|   21.303k|
|                     |       1.02x|         -|
|with_keep_tokens     |      6.220k|    5.691k|
|                     |       1.09x|         -|
```
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.

This commit adds these methods.

* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.

[Feature #19070]

Impacts on memory usage and performance are below:

Memory usage:

```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)

$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb

# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb

# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```

Performance:

```
$ cat ../ast_keep_tokens.yml
prelude: |
  src = &lt;&lt;~SRC
    module M
      class C
        def m1(a, b)
          1 + a + b
        end
      end
    end
  SRC
benchmark:
  without_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
  with_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)

$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
            --executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
            --executables="built-ruby::./miniruby -I../lib -I. -I.ext/common  ../tool/runruby.rb --extout=.ext  -- --disable-gems --disable-gem" \
            --output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..

|                     |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens  |     21.659k|   21.303k|
|                     |       1.02x|         -|
|with_keep_tokens     |      6.220k|    5.691k|
|                     |       1.09x|         -|
```
</pre>
</div>
</content>
</entry>
<entry>
<title>Add `node_id_for_backtrace_location` function</title>
<updated>2022-10-31T04:39:56+00:00</updated>
<author>
<name>eileencodes</name>
<email>eileencodes@gmail.com</email>
</author>
<published>2022-10-19T17:23:53+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=3391c51effcd61f9a718adf59740661d99f57b5b'/>
<id>3391c51effcd61f9a718adf59740661d99f57b5b</id>
<content type='text'>
We want to use error highlight with eval'd code, specifically ERB
templates. We're able to recover the generated code for eval'd templates
and can get a parse tree for the ERB generated code, but we don't have a
way to get the node id from the backtrace location. So we can't pass the
right node into error highlight.

This patch gives us an API to get the node id from the backtrace
location so we can find the node in the AST.

Error Highlight PR: https://github.com/ruby/error_highlight/pull/26

Co-authored-by: Aaron Patterson &lt;tenderlove@ruby-lang.org&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
We want to use error highlight with eval'd code, specifically ERB
templates. We're able to recover the generated code for eval'd templates
and can get a parse tree for the ERB generated code, but we don't have a
way to get the node id from the backtrace location. So we can't pass the
right node into error highlight.

This patch gives us an API to get the node id from the backtrace
location so we can find the node in the AST.

Error Highlight PR: https://github.com/ruby/error_highlight/pull/26

Co-authored-by: Aaron Patterson &lt;tenderlove@ruby-lang.org&gt;
</pre>
</div>
</content>
</entry>
</feed>
