Age | Commit message (Collapse) | Author |
|
```
1) Failure:
TestAst#test_ranges:test/fiber/scheduler.rb [/home/runner/work/ruby/ruby/src/test/ruby/test_ast.rb:122]:
<[]> expected but was
<[{:type=>:max_validation_error,
:max=>
#<RubyVM::AbstractSyntaxTree::Node::CodePosition:0x00007f80d630b598
@column=20,
@lineno=203>,
:end_pos=>
#<RubyVM::AbstractSyntaxTree::Node::CodePosition:0x00007f80d630b778
@column=19,
@lineno=203>,
:node=>
(BLOCK_PASS@203:15-203:19
(ARGSPUSH@203:15-203:20 (SPLAT@203:16-203:19 (LVAR@203:16-203:19 :*))
(HASH@203:16-203:19
(LIST@203:16-203:19 nil (LVAR@203:16-203:19 :**) nil)))
(LVAR@203:16-203:19 :&))}]>.
```
|
|
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6700
|
|
This was introduced by b609bdeb5307e280137b4b2838af0fe4e4b46f1c
to suppress warnings. However these warngins were deleted by
beae6cbf0fd8b6619e5212552de98022d4c4d4d4. Therefore these codes
are not needed anymore.
Notes:
Merged: https://github.com/ruby/ruby/pull/6826
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6818
|
|
[Feature #19134]
Notes:
Merged: https://github.com/ruby/ruby/pull/6818
|
|
If the rescue clause has only exc_var and not exc_list, use the
exc_var position instead of the rescue body position.
This issue appears to have been introduced in
688169fd83b24564b653c03977c168cea50ccd35 when "opt_list" was split
into "exc_list exc_var".
Fixes [Bug #18974]
Notes:
Merged: https://github.com/ruby/ruby/pull/6283
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6776
|
|
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.
This commit adds these methods.
* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.
[Feature #19070]
Impacts on memory usage and performance are below:
Memory usage:
```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb
# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb
# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```
Performance:
```
$ cat ../ast_keep_tokens.yml
prelude: |
src = <<~SRC
module M
class C
def m1(a, b)
1 + a + b
end
end
end
SRC
benchmark:
without_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
with_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)
$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
--executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
--executables="built-ruby::./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems --disable-gem" \
--output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..
| |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens | 21.659k| 21.303k|
| | 1.02x| -|
|with_keep_tokens | 6.220k| 5.691k|
| | 1.09x| -|
```
Notes:
Merged: https://github.com/ruby/ruby/pull/6770
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6760
|
|
Fixes [Bug #19132]
Notes:
Merged: https://github.com/ruby/ruby/pull/6743
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6721
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6698
|
|
Before:
```
Reducing stack by rule 639 (line 5062):
$1 = token "integer literal" (1.0-1.1: 1)
-> $$ = nterm simple_numeric (1.0-1.1: )
```
After:
```
Reducing stack by rule 641 (line 5078):
$1 = token "integer literal" (1.0-1.1: 1)
-> $$ = nterm simple_numeric (1.0-1.1: NODE_LIT)
```
`"<*>"` is supported by Bison 2.3b (2008-05-27) or later.
https://git.savannah.gnu.org/cgit/bison.git/commit/?id=12e3584054c16ab255672c07af0ffc7bb220e8bc
Therefore developers need to install Bison 2.3b+ to build ruby from
source codes if their Bison is older.
Minimum version requirement for Bison is changed to 3.0.
See: https://bugs.ruby-lang.org/issues/19068 [Feature #19068]
Notes:
Merged: https://github.com/ruby/ruby/pull/6579
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6517
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6515
|
|
Assign internal_id to semantic value so that dump parsetree option
can render the tree for these codes without SEGV.
* `def m(&); end`
* `def m(*); end`
* `def m(**); end`
Notes:
Merged: https://github.com/ruby/ruby/pull/6514
|
|
So that "IF" node is kept in the case below
```
def m
if
end
```
[Feature #19013]
Notes:
Merged: https://github.com/ruby/ruby/pull/6512
|
|
By this change, syntax error is recovered smaller units.
In the case below, "DEFN :bar" is same level with "CLASS :Foo"
now.
```
module Z
class Foo
foo.
end
def bar
end
end
```
[Feature #19013]
Notes:
Merged: https://github.com/ruby/ruby/pull/6512
|
|
"end" after "." or "::" is treated as local variable or method,
see `EXPR_DOT_bit` for detail.
However this "changes" where `bar` method is defined. In the example
below it is not module Z but class Foo.
```
module Z
class Foo
foo.
end
def bar
end
end
```
[Feature #19013]
Notes:
Merged: https://github.com/ruby/ruby/pull/6512
|
|
but "end" tokens are needed for correct language.
[Feature #19013]
Notes:
Merged: https://github.com/ruby/ruby/pull/6512
|
|
If this option is enabled, SyntaxError is not raised and Node is
returned even if passed script is broken.
[Feature #19013]
Notes:
Merged: https://github.com/ruby/ruby/pull/6512
|
|
The following warning appears without this fix:
```
parse.y:78:1: warning: unknown warning group '-Wpsabi', ignored
[-Wunknown-warning-option]
RBIMPL_WARNING_IGNORED(-Wpsabi)
^
./include/ruby/internal/warning_push.h:103:39: note: expanded from macro
'RBIMPL_WARNING_IGNORED'
^
./include/ruby/internal/warning_push.h:99:39: note: expanded from macro
'RBIMPL_WARNING_PRAGMA2'
^
./include/ruby/internal/warning_push.h:98:39: note: expanded from macro
'RBIMPL_WARNING_PRAGMA1'
^
./include/ruby/internal/warning_push.h:97:39: note: expanded from macro
'RBIMPL_WARNING_PRAGMA0'
^
<scratch space>:49:27: note: expanded from here
clang diagnostic ignored "-Wpsabi"
^
1 warning generated.
```
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6373
Merged-By: nobu <nobu@ruby-lang.org>
|
|
matching
[Bug #18990]
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6292
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6263
Merged-By: nobu <nobu@ruby-lang.org>
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/5977
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6073
|
|
`Ripper::Lexer#parse` re-parses the source code with syntax errors
when `raise_errors: false`.
Co-Authored-By: tompng <tomoyapenguin@gmail.com>
Notes:
Merged: https://github.com/ruby/ruby/pull/6233
|
|
* Fix some UBSAN false positives.
* ruby tool/update-deps --fix
Notes:
Merged-By: jhawthorn <john@hawthorn.email>
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6087
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6065
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6064
Merged-By: nobu <nobu@ruby-lang.org>
|
|
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/5867
|
|
Do not override the input string encoding at the time of preparation,
the source encoding is not determined from the input yet.
Notes:
Merged: https://github.com/ruby/ruby/pull/6015
|
|
`no_blockarg` is called for non-null `node` only.
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6016
|
|
Invalid escapes are handled at multiple levels. The first level
is in parse.y, so skip invalid unicode escape checks for regexps
in parse.y.
Make rb_reg_preprocess and unescape_nonascii accept the regexp
options. In unescape_nonascii, if the regexp is an extended
regexp, when "#" is encountered, ignore all characters until the
end of line or end of regexp.
Unfortunately, in extended regexps, you can use "#" as a non-comment
character inside a character class, so also parse "[" and "]"
specially for extended regexps, and only skip comments if "#" is
not inside a character class. Handle nested character classes as well.
This issue doesn't just affect extended regexps, it also affects
"(#?" comments inside all regexps. So for those comments, scan
until trailing ")" and ignore content inside.
I'm not sure if there are other corner cases not handled. A
better fix would be to redesign the regexp parser so that it
unescaped during parsing instead of before parsing, so you already
know the current parsing state.
Fixes [Bug #18294]
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Notes:
Merged: https://github.com/ruby/ruby/pull/5721
Merged-By: jeremyevans <code@jeremyevans.net>
|
|
|
|
|
|
Record block ID before vtable_pop, so the incorrect one doesn't
override it.
Fixes [Bug #18673]
Notes:
Merged: https://github.com/ruby/ruby/pull/5761
|
|
|
|
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/5545
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/5540
|
|
|
|
Get rid of depending on using $/ internally in String#chomp!, and
chomp the separator at once.
Notes:
Merged: https://github.com/ruby/ruby/pull/5403
|
|
Get rid of depending on using $/ internally in ARGF.gets.
Notes:
Merged: https://github.com/ruby/ruby/pull/5403
|