| Age | Commit message (Collapse) | Author |
|---|
|
This was an optimization for versions prior to 1.9 that traverse the
AST at runtime.
|
|
|
|
|
|
|
|
This patch suggests relocating the code dealing with `SCRIPT_LINES__` from ast.c to ruby_parser.c.
## Background
- I guess `AbstractSyntaxTree.of` method used to use `SCRIPT_LINES__` internally for some reason before
- However, now it appears `SCRIPT_LINES__` is no longer used meaningfully by the method
- As evidence of this, (and as my patch shows,) removing the function call of `rb_script_lines_for()` from `ast_s_of()` does not affect the result of `test/ruby/test_ast.rb`
Given the above, I think two possibilities can be considered:
- (A) `AbstractSyntaxTree.of` has not needed `SCRIPT_LINES__` already (I pick this)
- (B) We lack a test case of `AbstractSyntaxTree.of` that needs to use `SCRIPT_LINES__`
## Besides,
The current implementation causes strange behavior:
```console
ruby -e"SCRIPT_LINES__ = {__FILE__ => []}; puts RubyVM::AbstractSyntaxTree.of(->{ 1 + 2 }, keep_script_lines: true).script_lines"
=> `-e:1:in '<main>': undefined method 'script_lines' for nil (NoMethodError)`
```
I think this is a bug because `AbstractSyntaxTree.of` is not supposed to return `nil` even in this case.
This happens due to the ast.c's dependence on `SCRIPT_LINES__`.
And at the end of the `ast_s_of()`, `node_find()` can not find the target child node obviously because it doesn't make sense to look for a corresponding node made from the parameter of `AbstractSyntaxTree.of` in the AST tree made from the value of `{__FILE__ => []}`
## Solution
Since I think it's good enough `SCRIPT_LINES__` to be only referred by ruby.c, I chose the possibility "(A)" and wrote this patch which moves `rb_script_lines_for()` from ast.c to ruby_parser.c.
So as the result:
- `ast_s_of()` function no longer look up `SCRIPT_LINES__`
- Even so, this patched code passes the existing tests
- The strange behavior above no longer happens (I also added a test for it)
Please correct me if I miss something🙏
|
|
This allows ... argument forwarding to benefit from Allocationless
Anonymous Splat Forwarding, allowing the `f` call below to not
allocate an array or a hash.
```ruby
a = [1]
kw = {b: 2}
def c(a, b:)
end
def f(...)
c(...)
end
f(*a, **kw)
```
This temporarily skips prism locals tests until prism is changed
to use * and ** for ..., instead of using ruby2_keywords.
Ignore failures in rbs bundled gems tests, since they fail due
to this change.
|
|
`:sym` was managed by `NODE_LIT` with `Symbol` object.
This commit introduces `NODE_SYM` so that
1. Symbol literal is detectable from AST Node
2. Reduce dependency on ruby object
|
|
|
|
first_column and last_column return byte positions, but existing implementations
did not consider multibyte.
|
|
|
|
Restore `rescue`-context from the outer context.
`retry` targets the next outer block except for between `rescue` and
`else` or `ensure`, otherwise, if there is no enclosing block, it
should be syntax error.
|
|
|
|
|
|
|
|
|
|
|
|
The token just after BOM needs to position at column 0, so that the
indentation matches closing line.
Notes:
Merged: https://github.com/ruby/ruby/pull/8281
|
|
This token is exposed only when `RubyVM::AbstractSyntaxTree` with
`keep_tokens` option.
Notes:
Merged: https://github.com/ruby/ruby/pull/7622
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/7077
|
|
[Bug #19262]
Notes:
Merged: https://github.com/ruby/ruby/pull/7029
|
|
[Feature #19134]
Notes:
Merged: https://github.com/ruby/ruby/pull/6934
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6828
|
|
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.
This commit adds these methods.
* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.
[Feature #19070]
Impacts on memory usage and performance are below:
Memory usage:
```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb
# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb
# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```
Performance:
```
$ cat ../ast_keep_tokens.yml
prelude: |
src = <<~SRC
module M
class C
def m1(a, b)
1 + a + b
end
end
end
SRC
benchmark:
without_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
with_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)
$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
--executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
--executables="built-ruby::./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems --disable-gem" \
--output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..
| |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens | 21.659k| 21.303k|
| | 1.02x| -|
|with_keep_tokens | 6.220k| 5.691k|
| | 1.09x| -|
```
Notes:
Merged: https://github.com/ruby/ruby/pull/6770
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6760
|
|
Fixes [Bug #19132]
Notes:
Merged: https://github.com/ruby/ruby/pull/6743
|
|
We want to use error highlight with eval'd code, specifically ERB
templates. We're able to recover the generated code for eval'd templates
and can get a parse tree for the ERB generated code, but we don't have a
way to get the node id from the backtrace location. So we can't pass the
right node into error highlight.
This patch gives us an API to get the node id from the backtrace
location so we can find the node in the AST.
Error Highlight PR: https://github.com/ruby/error_highlight/pull/26
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
Notes:
Merged: https://github.com/ruby/ruby/pull/6593
|
|
The tests for error tolerance printed some warnings. This change
suppresses them.
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6517
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6519
|
|
`PrettyPrint` results depend on the `COLUMNS` environment variable by
default.
|
|
So that "IF" node is kept in the case below
```
def m
if
end
```
[Feature #19013]
Notes:
Merged: https://github.com/ruby/ruby/pull/6512
|
|
By this change, syntax error is recovered smaller units.
In the case below, "DEFN :bar" is same level with "CLASS :Foo"
now.
```
module Z
class Foo
foo.
end
def bar
end
end
```
[Feature #19013]
Notes:
Merged: https://github.com/ruby/ruby/pull/6512
|
|
"end" after "." or "::" is treated as local variable or method,
see `EXPR_DOT_bit` for detail.
However this "changes" where `bar` method is defined. In the example
below it is not module Z but class Foo.
```
module Z
class Foo
foo.
end
def bar
end
end
```
[Feature #19013]
Notes:
Merged: https://github.com/ruby/ruby/pull/6512
|
|
but "end" tokens are needed for correct language.
[Feature #19013]
Notes:
Merged: https://github.com/ruby/ruby/pull/6512
|
|
If this option is enabled, SyntaxError is not raised and Node is
returned even if passed script is broken.
[Feature #19013]
Notes:
Merged: https://github.com/ruby/ruby/pull/6512
|
|
|
|
Do not override the input string encoding at the time of preparation,
the source encoding is not determined from the input yet.
Notes:
Merged: https://github.com/ruby/ruby/pull/6015
|
|
ISeqs from ISeq#to_binary/load do not support `keep_script_lines`
now so some tests should be pending tests with
`RUBY_ISEQ_DUMP_DEBUG=to_binary`
Notes:
Merged: https://github.com/ruby/ruby/pull/5508
|
|
[Bug #18434]
Notes:
Merged: https://github.com/ruby/ruby/pull/5347
|
|
|
|
Now the following code works without an exception.
```
RubyVM.keep_script_lines = true
eval(<<END)
def foo
end
END
p RubyVM::AbstractSyntaxTree.of(method(:foo))
```
|
|
This check is needed to fix a bug of error_highlight when NameError
occurred in eval'ed code.
https://github.com/ruby/error_highlight/pull/16
The same check for proc/method has been already introduced since
64ac984129a7a4645efe5ac57c168ef880b479b2.
|
|
|
|
[Bug #18178]
|
|
... as per ko1's preference. He is preparing to extend this feature to
ISeq for his new debugger. He prefers "keep" to "save" for this wording.
This API is internal and not included in any released version, so I
change it in advance.
|
|
This changes Thread::Location::Backtrace#absolute_path to return
nil for methods/procs defined in eval. If the realpath of an iseq
is nil, that indicates it was defined in eval, in which case you
cannot use RubyVM::AbstractSyntaxTree.of.
Fixes [Bug #16983]
Co-authored-by: Koichi Sasada <ko1@atdot.net>
Notes:
Merged: https://github.com/ruby/ruby/pull/4519
|
|
This option makes the parser keep the original source as an array of
the original code lines. This feature exploits the mechanism of
`SCRIPT_LINES__` but records only the specified code that is passed to
RubyVM::AST.of or .parse, instead of recording all parsed program texts.
Notes:
Merged: https://github.com/ruby/ruby/pull/4581
|
|
|
|
|
|
`RubyVM::AbstractSyntaxTree`.
Notes:
Merged: https://github.com/ruby/ruby/pull/3298
|
