<feed xmlns='http://www.w3.org/2005/Atom'>
<title>ruby.git/ext/ripper/eventids2.c, branch v4.0.3</title>
<subtitle>The Ruby Programming Language</subtitle>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/'/>
<entry>
<title>Remove unused token definitions for `tRPAREN` in Ripper and parser files</title>
<updated>2025-01-06T22:17:55+00:00</updated>
<author>
<name>ydah</name>
<email>t.yudai92@gmail.com</email>
</author>
<published>2024-12-29T02:45:54+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=de8deb5f9c7aef899bbad6652e86ab1031a25965'/>
<id>de8deb5f9c7aef899bbad6652e86ab1031a25965</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Verify that events2table.c was generated successfully</title>
<updated>2023-12-28T09:07:49+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2023-12-28T09:07:49+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=7016ab873eaa68d1dfe1af50198c157e451c784b'/>
<id>7016ab873eaa68d1dfe1af50198c157e451c784b</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[Feature #19719] Universal Parser</title>
<updated>2023-06-12T09:23:48+00:00</updated>
<author>
<name>yui-knk</name>
<email>spiketeika@gmail.com</email>
</author>
<published>2023-05-28T11:00:20+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=b481b673d753339204290d7582dbb91a6e14447a'/>
<id>b481b673d753339204290d7582dbb91a6e14447a</id>
<content type='text'>
Introduce Universal Parser mode for the parser.
This commit includes these changes:

* Introduce `UNIVERSAL_PARSER` macro. All of CRuby related functions
  are passed via `struct rb_parser_config_struct` when this macro is enabled.
* Add CI task with 'cppflags=-DUNIVERSAL_PARSER' for ubuntu.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Introduce Universal Parser mode for the parser.
This commit includes these changes:

* Introduce `UNIVERSAL_PARSER` macro. All of CRuby related functions
  are passed via `struct rb_parser_config_struct` when this macro is enabled.
* Add CI task with 'cppflags=-DUNIVERSAL_PARSER' for ubuntu.
</pre>
</div>
</content>
</entry>
<entry>
<title>Enhance keep_tokens option for RubyVM::AbstractSyntaxTree parsing methods</title>
<updated>2022-11-21T00:01:34+00:00</updated>
<author>
<name>yui-knk</name>
<email>spiketeika@gmail.com</email>
</author>
<published>2022-09-23T13:40:02+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=d8601621edcf29e3323b90dcf04b774edd9fb45e'/>
<id>d8601621edcf29e3323b90dcf04b774edd9fb45e</id>
<content type='text'>
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.

This commit adds these methods.

* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.

[Feature #19070]

Impacts on memory usage and performance are below:

Memory usage:

```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)

$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb

# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb

# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```

Performance:

```
$ cat ../ast_keep_tokens.yml
prelude: |
  src = &lt;&lt;~SRC
    module M
      class C
        def m1(a, b)
          1 + a + b
        end
      end
    end
  SRC
benchmark:
  without_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
  with_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)

$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
            --executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
            --executables="built-ruby::./miniruby -I../lib -I. -I.ext/common  ../tool/runruby.rb --extout=.ext  -- --disable-gems --disable-gem" \
            --output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..

|                     |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens  |     21.659k|   21.303k|
|                     |       1.02x|         -|
|with_keep_tokens     |      6.220k|    5.691k|
|                     |       1.09x|         -|
```
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.

This commit adds these methods.

* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.

[Feature #19070]

Impacts on memory usage and performance are below:

Memory usage:

```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)

$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb

# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb

# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```

Performance:

```
$ cat ../ast_keep_tokens.yml
prelude: |
  src = &lt;&lt;~SRC
    module M
      class C
        def m1(a, b)
          1 + a + b
        end
      end
    end
  SRC
benchmark:
  without_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
  with_keep_tokens: |
    RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)

$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
            --executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
            --executables="built-ruby::./miniruby -I../lib -I. -I.ext/common  ../tool/runruby.rb --extout=.ext  -- --disable-gems --disable-gem" \
            --output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..

|                     |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens  |     21.659k|   21.303k|
|                     |       1.02x|         -|
|with_keep_tokens     |      6.220k|    5.691k|
|                     |       1.09x|         -|
```
</pre>
</div>
</content>
</entry>
<entry>
<title>Suppress warnings by gcc 10.1.0-RC-20200430</title>
<updated>2020-05-04T03:28:24+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2020-05-04T03:10:04+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=b7e1eda932c74196d58e6b63644200b764b5453e'/>
<id>b7e1eda932c74196d58e6b63644200b764b5453e</id>
<content type='text'>
* Folding results should not be empty.

  If `OnigCodePointCount(to-&gt;n)` were 0, `for` loop using `fn`
  wouldn't execute and `ncs` elements are not initialized.

  ```
  enc/unicode.c:557:21: warning: 'ncs[0]' may be used uninitialized in this function [-Wmaybe-uninitialized]
    557 |  for (i = 0; i &lt; ncs[0]; i++) {
        |                  ~~~^~~
  ```

* Cast to `enum yytokentype`

  Additional enums for scanner events by ripper are not included
  in `yytokentype`.

  ```
  ripper.y:7274:28: warning: implicit conversion from 'enum &lt;anonymous&gt;' to 'enum yytokentype' [-Wenum-conversion]
  ```
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* Folding results should not be empty.

  If `OnigCodePointCount(to-&gt;n)` were 0, `for` loop using `fn`
  wouldn't execute and `ncs` elements are not initialized.

  ```
  enc/unicode.c:557:21: warning: 'ncs[0]' may be used uninitialized in this function [-Wmaybe-uninitialized]
    557 |  for (i = 0; i &lt; ncs[0]; i++) {
        |                  ~~~^~~
  ```

* Cast to `enum yytokentype`

  Additional enums for scanner events by ripper are not included
  in `yytokentype`.

  ```
  ripper.y:7274:28: warning: implicit conversion from 'enum &lt;anonymous&gt;' to 'enum yytokentype' [-Wenum-conversion]
  ```
</pre>
</div>
</content>
</entry>
<entry>
<title>Revert "Method reference operator"</title>
<updated>2019-11-12T08:24:48+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2019-10-23T16:06:59+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=fb6a489af2765a3b56e301adf0019af6bbad6156'/>
<id>fb6a489af2765a3b56e301adf0019af6bbad6156</id>
<content type='text'>
This reverts commit 67c574736912003c377218153f9d3b9c0c96a17b.
[Feature #16275]
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This reverts commit 67c574736912003c377218153f9d3b9c0c96a17b.
[Feature #16275]
</pre>
</div>
</content>
</entry>
<entry>
<title>* expand tabs. [ci skip]</title>
<updated>2019-10-21T02:50:44+00:00</updated>
<author>
<name>git</name>
<email>svn-admin@ruby-lang.org</email>
</author>
<published>2019-10-21T02:50:44+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=215e4b8721d968b83c66c253e6da15b90520f679'/>
<id>215e4b8721d968b83c66c253e6da15b90520f679</id>
<content type='text'>
Tabs were expanded because previously the file did not have any tab indentation.
Please update your editor config, and use misc/expand_tabs.rb in the pre-commit hook.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Tabs were expanded because previously the file did not have any tab indentation.
Please update your editor config, and use misc/expand_tabs.rb in the pre-commit hook.
</pre>
</div>
</content>
</entry>
<entry>
<title>Look up event ID offsets by token as index</title>
<updated>2019-10-21T02:45:54+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2019-10-21T02:45:54+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=44da5a1c872879b05acab045263d0d4f906f04ba'/>
<id>44da5a1c872879b05acab045263d0d4f906f04ba</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Also BDOT2 and BDOT3 are operators</title>
<updated>2019-10-20T11:45:56+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2019-10-20T11:45:56+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=ca7de0334d26b28eb8fd731b130404beb5cf5c12'/>
<id>ca7de0334d26b28eb8fd731b130404beb5cf5c12</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Changed numbered parameter prefix</title>
<updated>2019-09-24T12:57:54+00:00</updated>
<author>
<name>Nobuyoshi Nakada</name>
<email>nobu@ruby-lang.org</email>
</author>
<published>2019-09-03T15:07:50+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=ea68bb914a3c806a1c5188993b96791a76ab0849'/>
<id>ea68bb914a3c806a1c5188993b96791a76ab0849</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
</feed>
