summaryrefslogtreecommitdiff
path: root/parse.y
AgeCommit message (Collapse)Author
2024-07-21merge revision(s) b15e88e0fcccb03b9cc5e4c1478ec9b10e26c961: [Backport #19619]nagachika
[Bug #19619] Preserve numbered parameters context Preserve numbered parameters context across method definitions
2024-07-15merge revision(s) 05553cf22d43dd78b8f30cc4591230b5c000c538: [Backport #20517]nagachika
[Bug #20517] Make a multibyte character one token at meta escape
2024-07-15merge revision(s) d503e1b95a40e45d7767e0175de60092de4ba54e: [Backport #20030]nagachika
[Bug #20030] dispatch invalid escaped character without ignoring it
2024-02-25Revert "merge revision(s) c8d162c889008028b148437d02f36f4edaa749fd: ↵nagachika
[Backport #19973]" This reverts commit 24dd529750c08b5603fb418a4f34998b9bf6c8f9.
2024-02-25merge revision(s) c8d162c889008028b148437d02f36f4edaa749fd: [Backport #19973]nagachika
[Bug #19973] Warn duplicated keyword arguments after keyword splat --- parse.y | 11 +++++++---- test/ruby/test_syntax.rb | 6 ++++++ 2 files changed, 13 insertions(+), 4 deletions(-)
2023-11-10merge revision(s) 17b0643392749f45b7aacb64fc1c1bd704d42b4c: [Backport #19924]nagachika
[Bug #19924] Source code should be unsigned char stream Use `peekc` or `nextc` to fetch the next character, instead of reading from `lex.pcur` directly, for compilers that plain char is signed. --- parse.y | 10 +++++----- test/ruby/test_parse.rb | 2 ++ 2 files changed, 7 insertions(+), 5 deletions(-)
2023-09-09merge revision(s) 382678d4112f4afc6272244c22924d2b004274b1: [Backport #19788]nagachika
[Bug #19788] Use the result of `tCOLON2` event --- parse.y | 16 ++++++++-------- test/ripper/test_parser_events.rb | 17 +++++++++++++++++ test/ripper/test_scanner_events.rb | 5 +++++ 3 files changed, 30 insertions(+), 8 deletions(-)
2023-08-13merge revision(s) 0b8f15575a440f85ac686f5b0eae8f8b7c2b72e7: [Backport #19836]nagachika
Fix memory leak for incomplete lambdas [Bug #19836] The parser does not free the chain of `struct vtable`, which causes memory leaks. The following script reproduces this issue: ``` 10.times do 100_000.times do Ripper.parse("-> {") end puts `ps -o rss= -p #{$$}` end ``` --- parse.y | 24 ++++++++++++++---------- test/ripper/test_ripper.rb | 7 +++++++ 2 files changed, 21 insertions(+), 10 deletions(-)
2023-08-13merge revision(s) 5bc8fceca8d47ed1ef9c603c6531a408de36b60c: [Backport #19835]nagachika
Fix memory leak in parser for incomplete tokens [Bug #19835] The parser does not free the `tbl` of the `struct vtable` when there are leftover `lvtbl` in the parser. This causes a memory leak. The following script reproduces this issue: ``` 10.times do 100_000.times do Ripper.parse("class Foo") end puts `ps -o rss= -p #{$$}` end ``` --- parse.y | 42 ++++++++++++++++++++++++++++-------------- test/ripper/test_ripper.rb | 7 +++++++ 2 files changed, 35 insertions(+), 14 deletions(-)
2023-07-22merge revision(s) 91c004885fc75a93cadf0094fa86ec3bd0ec25f5: [Backport #19025]nagachika
[Bug #19025] Numbered parameter names are always local variables --- parse.y | 2 +- test/ruby/test_syntax.rb | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
2023-07-17merge revision(s) 1bc8838d60ef3fc6812d3b64ed87caaf0ae943d9: [Backport #19750]nagachika
Handle unterminated unicode escapes in regexps This fixes an infinite loop possible after ec3542229b29ec93062e9d90e877ea29d3c19472. For \u{} escapes in regexps, skip validation in the parser, and rely on the regexp code to handle validation. This is necessary so that invalid unicode escapes in comments in extended regexps are allowed. Fixes [Bug #19750] Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org> --- parse.y | 97 ++++++++++++++++++++++++++++++++----------------- test/ruby/test_parse.rb | 16 ++++++++ 2 files changed, 79 insertions(+), 34 deletions(-)
2022-12-15Disallow mixed usage of ... and */**Shugo Maeda
[Feature #19134] Notes: Merged: https://github.com/ruby/ruby/pull/6934
2022-12-13[Bug #19195] Allow optional newlines before closing parenthesisNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6916
2022-12-05Use loc to fix a failure of test_ast.rbShugo Maeda
``` 1) Failure: TestAst#test_ranges:test/fiber/scheduler.rb [/home/runner/work/ruby/ruby/src/test/ruby/test_ast.rb:122]: <[]> expected but was <[{:type=>:max_validation_error, :max=> #<RubyVM::AbstractSyntaxTree::Node::CodePosition:0x00007f80d630b598 @column=20, @lineno=203>, :end_pos=> #<RubyVM::AbstractSyntaxTree::Node::CodePosition:0x00007f80d630b778 @column=19, @lineno=203>, :node=> (BLOCK_PASS@203:15-203:19 (ARGSPUSH@203:15-203:20 (SPLAT@203:16-203:19 (LVAR@203:16-203:19 :*)) (HASH@203:16-203:19 (LIST@203:16-203:19 nil (LVAR@203:16-203:19 :**) nil))) (LVAR@203:16-203:19 :&))}]>. ```
2022-12-05Should use argsloc for the last argument for arg_append()Shugo Maeda
2022-12-02Introduce encoding check macroS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/6700
2022-11-29Remove ruby2_keywords related to args forwardingyui-knk
This was introduced by b609bdeb5307e280137b4b2838af0fe4e4b46f1c to suppress warnings. However these warngins were deleted by beae6cbf0fd8b6619e5212552de98022d4c4d4d4. Therefore these codes are not needed anymore. Notes: Merged: https://github.com/ruby/ruby/pull/6826
2022-11-29Use idFWD_* instead of ANON_*_IDShugo Maeda
Notes: Merged: https://github.com/ruby/ruby/pull/6818
2022-11-29Allow ** in def foo(...)Shugo Maeda
[Feature #19134] Notes: Merged: https://github.com/ruby/ruby/pull/6818
2022-11-24Fix the position of rescue clause without exc_listJeremy Evans
If the rescue clause has only exc_var and not exc_list, use the exc_var position instead of the rescue body position. This issue appears to have been introduced in 688169fd83b24564b653c03977c168cea50ccd35 when "opt_list" was split into "exc_list exc_var". Fixes [Bug #18974] Notes: Merged: https://github.com/ruby/ruby/pull/6283
2022-11-21Refactor to use has_delayed_token macroyui-knk
Notes: Merged: https://github.com/ruby/ruby/pull/6776
2022-11-21Enhance keep_tokens option for RubyVM::AbstractSyntaxTree parsing methodsyui-knk
Implementation for Language Server Protocol (LSP) sometimes needs token information. For example both `m(1)` and `m(1, )` has same AST structure other than node locations then it's impossible to check the existence of `,` from AST. However in later case, it might be better to suggest variables list for the second argument. Token information is important for such case. This commit adds these methods. * Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of` * Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes. * Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node. [Feature #19070] Impacts on memory usage and performance are below: Memory usage: ``` $ cat test.rb root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true) $ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux] 11408kb # keep_tokens :false $ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb 17508kb # keep_tokens :true $ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb 30960kb ``` Performance: ``` $ cat ../ast_keep_tokens.yml prelude: | src = <<~SRC module M class C def m1(a, b) 1 + a + b end end end SRC benchmark: without_keep_tokens: | RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false) with_keep_tokens: | RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true) $ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml /home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \ --executables="compare-ruby::./ruby -I.ext/common --disable-gem" \ --executables="built-ruby::./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems --disable-gem" \ --output=markdown --output-compare -v ../ast_keep_tokens.yml compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux] built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux] warming up.. | |compare-ruby|built-ruby| |:--------------------|-----------:|---------:| |without_keep_tokens | 21.659k| 21.303k| | | 1.02x| -| |with_keep_tokens | 6.220k| 5.691k| | | 1.09x| -| ``` Notes: Merged: https://github.com/ruby/ruby/pull/6770
2022-11-18Make anonymous rest arg (*) and block arg (&) accessible from ARGS nodeyui-knk
Notes: Merged: https://github.com/ruby/ruby/pull/6760
2022-11-18Allow anonymous keyword rest parameter with other keyword parametersShugo Maeda
Fixes [Bug #19132] Notes: Merged: https://github.com/ruby/ruby/pull/6743
2022-11-16Using UNDEF_P macroS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/6721
2022-11-09Now bison 3.0 or later is requiredNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6698
2022-11-08Set default %printer for NODE ntermsyui-knk
Before: ``` Reducing stack by rule 639 (line 5062): $1 = token "integer literal" (1.0-1.1: 1) -> $$ = nterm simple_numeric (1.0-1.1: ) ``` After: ``` Reducing stack by rule 641 (line 5078): $1 = token "integer literal" (1.0-1.1: 1) -> $$ = nterm simple_numeric (1.0-1.1: NODE_LIT) ``` `"<*>"` is supported by Bison 2.3b (2008-05-27) or later. https://git.savannah.gnu.org/cgit/bison.git/commit/?id=12e3584054c16ab255672c07af0ffc7bb220e8bc Therefore developers need to install Bison 2.3b+ to build ruby from source codes if their Bison is older. Minimum version requirement for Bison is changed to 3.0. See: https://bugs.ruby-lang.org/issues/19068 [Feature #19068] Notes: Merged: https://github.com/ruby/ruby/pull/6579
2022-10-09Do not set `$!` to `SyntaxError` when error tolerantNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6517
2022-10-08Fix typosyui-knk
Notes: Merged: https://github.com/ruby/ruby/pull/6515
2022-10-08Fix SEGV of dump parsetreeyui-knk
Assign internal_id to semantic value so that dump parsetree option can render the tree for these codes without SEGV. * `def m(&); end` * `def m(*); end` * `def m(**); end` Notes: Merged: https://github.com/ruby/ruby/pull/6514
2022-10-08"expr_value" can be erroryui-knk
So that "IF" node is kept in the case below ``` def m if end ``` [Feature #19013] Notes: Merged: https://github.com/ruby/ruby/pull/6512
2022-10-08Move `error` from top_stmts and top_stmt to stmtyui-knk
By this change, syntax error is recovered smaller units. In the case below, "DEFN :bar" is same level with "CLASS :Foo" now. ``` module Z class Foo foo. end def bar end end ``` [Feature #19013] Notes: Merged: https://github.com/ruby/ruby/pull/6512
2022-10-08Treat "end" as reserved word with consideration of indentyui-knk
"end" after "." or "::" is treated as local variable or method, see `EXPR_DOT_bit` for detail. However this "changes" where `bar` method is defined. In the example below it is not module Z but class Foo. ``` module Z class Foo foo. end def bar end end ``` [Feature #19013] Notes: Merged: https://github.com/ruby/ruby/pull/6512
2022-10-08Generates "end" tokens if parser hits end of inputyui-knk
but "end" tokens are needed for correct language. [Feature #19013] Notes: Merged: https://github.com/ruby/ruby/pull/6512
2022-10-08Add error_tolerant option to RubyVM::ASTyui-knk
If this option is enabled, SyntaxError is not raised and Node is returned even if passed script is broken. [Feature #19013] Notes: Merged: https://github.com/ruby/ruby/pull/6512
2022-09-26Suppress a warning on clangShugo Maeda
The following warning appears without this fix: ``` parse.y:78:1: warning: unknown warning group '-Wpsabi', ignored [-Wunknown-warning-option] RBIMPL_WARNING_IGNORED(-Wpsabi) ^ ./include/ruby/internal/warning_push.h:103:39: note: expanded from macro 'RBIMPL_WARNING_IGNORED' ^ ./include/ruby/internal/warning_push.h:99:39: note: expanded from macro 'RBIMPL_WARNING_PRAGMA2' ^ ./include/ruby/internal/warning_push.h:98:39: note: expanded from macro 'RBIMPL_WARNING_PRAGMA1' ^ ./include/ruby/internal/warning_push.h:97:39: note: expanded from macro 'RBIMPL_WARNING_PRAGMA0' ^ <scratch space>:49:27: note: expanded from here clang diagnostic ignored "-Wpsabi" ^ 1 warning generated. ```
2022-09-14Reuse `opt_arg_append` functionS.H
Notes: Merged: https://github.com/ruby/ruby/pull/6373 Merged-By: nobu <nobu@ruby-lang.org>
2022-09-09Fix unexpected "duplicated key name" error in paren-less one line pattern ↵Kazuki Tsujimoto
matching [Bug #18990]
2022-08-28[Bug #18963] Separate string contents by here document terminatorNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6292
2022-08-22Reuse `nonlocal_var` patternsS.H
Notes: Merged: https://github.com/ruby/ruby/pull/6263 Merged-By: nobu <nobu@ruby-lang.org>
2022-08-19Reuse opt_nl ruleS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/5977
2022-08-19Repalce to NIL_P macroS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/6073
2022-08-12[Bug #18962] Do not read again once reached EOFNobuyoshi Nakada
`Ripper::Lexer#parse` re-parses the source code with syntax errors when `raise_errors: false`. Co-Authored-By: tompng <tomoyapenguin@gmail.com> Notes: Merged: https://github.com/ruby/ruby/pull/6233
2022-07-12Fix some UBSAN false positives (#6115)Kevin Backhouse
* Fix some UBSAN false positives. * ruby tool/update-deps --fix Notes: Merged-By: jhawthorn <john@hawthorn.email>
2022-07-06[Bug #18890] newline should be insignificant after pattern labelNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6087
2022-06-30[Bug #18877] Let `lex_ctxt` not to eat escaped whitespaceNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6065
2022-06-29[Bug #18884] `class` cannot be just followed by modifiersNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6064 Merged-By: nobu <nobu@ruby-lang.org>
2022-06-23Suppress notes for old gccNobuyoshi Nakada
2022-06-17Using is_ascii_string to check encodingS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/5867
2022-06-17Respect the encoding of the source [Bug #18827]Nobuyoshi Nakada
Do not override the input string encoding at the time of preparation, the source encoding is not determined from the input yet. Notes: Merged: https://github.com/ruby/ruby/pull/6015
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-15 09:41:39 +0000