summaryrefslogtreecommitdiff
path: root/parse.y
AgeCommit message (Collapse)Author
2023-02-01Extract body rules from endless method definitionsNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/7222
2023-01-25Remove not used argument from tokenize_identyui-knk
This has not been used since 5e59be3eddd2ce28610ec8d1b42845cde0080833 Notes: Merged: https://github.com/ruby/ruby/pull/7179
2023-01-07Fix format specifiers for pointer differencesNobuyoshi Nakada
2023-01-06[Bug #19312] Return end-of-input at `__END__`Nobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/7077
2023-01-02[Bug #19291] Rewind to the previous lineNobuyoshi Nakada
When rewinding looking ahead after newline token, also reset the last line string, the pointers to it, and the location, not only the line number. Notes: Merged: https://github.com/ruby/ruby/pull/7054
2022-12-26EXPR_DOT is set when next token is tANDDOT ("&.") [ci skip]yui-knk
Notes: Merged: https://github.com/ruby/ruby/pull/7028
2022-12-15Disallow mixed usage of ... and */**Shugo Maeda
[Feature #19134] Notes: Merged: https://github.com/ruby/ruby/pull/6934
2022-12-13[Bug #19195] Allow optional newlines before closing parenthesisNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6916
2022-12-05Use loc to fix a failure of test_ast.rbShugo Maeda
``` 1) Failure: TestAst#test_ranges:test/fiber/scheduler.rb [/home/runner/work/ruby/ruby/src/test/ruby/test_ast.rb:122]: <[]> expected but was <[{:type=>:max_validation_error, :max=> #<RubyVM::AbstractSyntaxTree::Node::CodePosition:0x00007f80d630b598 @column=20, @lineno=203>, :end_pos=> #<RubyVM::AbstractSyntaxTree::Node::CodePosition:0x00007f80d630b778 @column=19, @lineno=203>, :node=> (BLOCK_PASS@203:15-203:19 (ARGSPUSH@203:15-203:20 (SPLAT@203:16-203:19 (LVAR@203:16-203:19 :*)) (HASH@203:16-203:19 (LIST@203:16-203:19 nil (LVAR@203:16-203:19 :**) nil))) (LVAR@203:16-203:19 :&))}]>. ```
2022-12-05Should use argsloc for the last argument for arg_append()Shugo Maeda
2022-12-02Introduce encoding check macroS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/6700
2022-11-29Remove ruby2_keywords related to args forwardingyui-knk
This was introduced by b609bdeb5307e280137b4b2838af0fe4e4b46f1c to suppress warnings. However these warngins were deleted by beae6cbf0fd8b6619e5212552de98022d4c4d4d4. Therefore these codes are not needed anymore. Notes: Merged: https://github.com/ruby/ruby/pull/6826
2022-11-29Use idFWD_* instead of ANON_*_IDShugo Maeda
Notes: Merged: https://github.com/ruby/ruby/pull/6818
2022-11-29Allow ** in def foo(...)Shugo Maeda
[Feature #19134] Notes: Merged: https://github.com/ruby/ruby/pull/6818
2022-11-24Fix the position of rescue clause without exc_listJeremy Evans
If the rescue clause has only exc_var and not exc_list, use the exc_var position instead of the rescue body position. This issue appears to have been introduced in 688169fd83b24564b653c03977c168cea50ccd35 when "opt_list" was split into "exc_list exc_var". Fixes [Bug #18974] Notes: Merged: https://github.com/ruby/ruby/pull/6283
2022-11-21Refactor to use has_delayed_token macroyui-knk
Notes: Merged: https://github.com/ruby/ruby/pull/6776
2022-11-21Enhance keep_tokens option for RubyVM::AbstractSyntaxTree parsing methodsyui-knk
Implementation for Language Server Protocol (LSP) sometimes needs token information. For example both `m(1)` and `m(1, )` has same AST structure other than node locations then it's impossible to check the existence of `,` from AST. However in later case, it might be better to suggest variables list for the second argument. Token information is important for such case. This commit adds these methods. * Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of` * Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes. * Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node. [Feature #19070] Impacts on memory usage and performance are below: Memory usage: ``` $ cat test.rb root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true) $ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux] 11408kb # keep_tokens :false $ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb 17508kb # keep_tokens :true $ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb 30960kb ``` Performance: ``` $ cat ../ast_keep_tokens.yml prelude: | src = <<~SRC module M class C def m1(a, b) 1 + a + b end end end SRC benchmark: without_keep_tokens: | RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false) with_keep_tokens: | RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true) $ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml /home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \ --executables="compare-ruby::./ruby -I.ext/common --disable-gem" \ --executables="built-ruby::./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems --disable-gem" \ --output=markdown --output-compare -v ../ast_keep_tokens.yml compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux] built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux] warming up.. | |compare-ruby|built-ruby| |:--------------------|-----------:|---------:| |without_keep_tokens | 21.659k| 21.303k| | | 1.02x| -| |with_keep_tokens | 6.220k| 5.691k| | | 1.09x| -| ``` Notes: Merged: https://github.com/ruby/ruby/pull/6770
2022-11-18Make anonymous rest arg (*) and block arg (&) accessible from ARGS nodeyui-knk
Notes: Merged: https://github.com/ruby/ruby/pull/6760
2022-11-18Allow anonymous keyword rest parameter with other keyword parametersShugo Maeda
Fixes [Bug #19132] Notes: Merged: https://github.com/ruby/ruby/pull/6743
2022-11-16Using UNDEF_P macroS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/6721
2022-11-09Now bison 3.0 or later is requiredNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6698
2022-11-08Set default %printer for NODE ntermsyui-knk
Before: ``` Reducing stack by rule 639 (line 5062): $1 = token "integer literal" (1.0-1.1: 1) -> $$ = nterm simple_numeric (1.0-1.1: ) ``` After: ``` Reducing stack by rule 641 (line 5078): $1 = token "integer literal" (1.0-1.1: 1) -> $$ = nterm simple_numeric (1.0-1.1: NODE_LIT) ``` `"<*>"` is supported by Bison 2.3b (2008-05-27) or later. https://git.savannah.gnu.org/cgit/bison.git/commit/?id=12e3584054c16ab255672c07af0ffc7bb220e8bc Therefore developers need to install Bison 2.3b+ to build ruby from source codes if their Bison is older. Minimum version requirement for Bison is changed to 3.0. See: https://bugs.ruby-lang.org/issues/19068 [Feature #19068] Notes: Merged: https://github.com/ruby/ruby/pull/6579
2022-10-09Do not set `$!` to `SyntaxError` when error tolerantNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6517
2022-10-08Fix typosyui-knk
Notes: Merged: https://github.com/ruby/ruby/pull/6515
2022-10-08Fix SEGV of dump parsetreeyui-knk
Assign internal_id to semantic value so that dump parsetree option can render the tree for these codes without SEGV. * `def m(&); end` * `def m(*); end` * `def m(**); end` Notes: Merged: https://github.com/ruby/ruby/pull/6514
2022-10-08"expr_value" can be erroryui-knk
So that "IF" node is kept in the case below ``` def m if end ``` [Feature #19013] Notes: Merged: https://github.com/ruby/ruby/pull/6512
2022-10-08Move `error` from top_stmts and top_stmt to stmtyui-knk
By this change, syntax error is recovered smaller units. In the case below, "DEFN :bar" is same level with "CLASS :Foo" now. ``` module Z class Foo foo. end def bar end end ``` [Feature #19013] Notes: Merged: https://github.com/ruby/ruby/pull/6512
2022-10-08Treat "end" as reserved word with consideration of indentyui-knk
"end" after "." or "::" is treated as local variable or method, see `EXPR_DOT_bit` for detail. However this "changes" where `bar` method is defined. In the example below it is not module Z but class Foo. ``` module Z class Foo foo. end def bar end end ``` [Feature #19013] Notes: Merged: https://github.com/ruby/ruby/pull/6512
2022-10-08Generates "end" tokens if parser hits end of inputyui-knk
but "end" tokens are needed for correct language. [Feature #19013] Notes: Merged: https://github.com/ruby/ruby/pull/6512
2022-10-08Add error_tolerant option to RubyVM::ASTyui-knk
If this option is enabled, SyntaxError is not raised and Node is returned even if passed script is broken. [Feature #19013] Notes: Merged: https://github.com/ruby/ruby/pull/6512
2022-09-26Suppress a warning on clangShugo Maeda
The following warning appears without this fix: ``` parse.y:78:1: warning: unknown warning group '-Wpsabi', ignored [-Wunknown-warning-option] RBIMPL_WARNING_IGNORED(-Wpsabi) ^ ./include/ruby/internal/warning_push.h:103:39: note: expanded from macro 'RBIMPL_WARNING_IGNORED' ^ ./include/ruby/internal/warning_push.h:99:39: note: expanded from macro 'RBIMPL_WARNING_PRAGMA2' ^ ./include/ruby/internal/warning_push.h:98:39: note: expanded from macro 'RBIMPL_WARNING_PRAGMA1' ^ ./include/ruby/internal/warning_push.h:97:39: note: expanded from macro 'RBIMPL_WARNING_PRAGMA0' ^ <scratch space>:49:27: note: expanded from here clang diagnostic ignored "-Wpsabi" ^ 1 warning generated. ```
2022-09-14Reuse `opt_arg_append` functionS.H
Notes: Merged: https://github.com/ruby/ruby/pull/6373 Merged-By: nobu <nobu@ruby-lang.org>
2022-09-09Fix unexpected "duplicated key name" error in paren-less one line pattern ↵Kazuki Tsujimoto
matching [Bug #18990]
2022-08-28[Bug #18963] Separate string contents by here document terminatorNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6292
2022-08-22Reuse `nonlocal_var` patternsS.H
Notes: Merged: https://github.com/ruby/ruby/pull/6263 Merged-By: nobu <nobu@ruby-lang.org>
2022-08-19Reuse opt_nl ruleS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/5977
2022-08-19Repalce to NIL_P macroS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/6073
2022-08-12[Bug #18962] Do not read again once reached EOFNobuyoshi Nakada
`Ripper::Lexer#parse` re-parses the source code with syntax errors when `raise_errors: false`. Co-Authored-By: tompng <tomoyapenguin@gmail.com> Notes: Merged: https://github.com/ruby/ruby/pull/6233
2022-07-12Fix some UBSAN false positives (#6115)Kevin Backhouse
* Fix some UBSAN false positives. * ruby tool/update-deps --fix Notes: Merged-By: jhawthorn <john@hawthorn.email>
2022-07-06[Bug #18890] newline should be insignificant after pattern labelNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6087
2022-06-30[Bug #18877] Let `lex_ctxt` not to eat escaped whitespaceNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6065
2022-06-29[Bug #18884] `class` cannot be just followed by modifiersNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6064 Merged-By: nobu <nobu@ruby-lang.org>
2022-06-23Suppress notes for old gccNobuyoshi Nakada
2022-06-17Using is_ascii_string to check encodingS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/5867
2022-06-17Respect the encoding of the source [Bug #18827]Nobuyoshi Nakada
Do not override the input string encoding at the time of preparation, the source encoding is not determined from the input yet. Notes: Merged: https://github.com/ruby/ruby/pull/6015
2022-06-15Remove unnecessary conditionNobuyoshi Nakada
`no_blockarg` is called for non-null `node` only.
2022-06-14ripper: Check if anonymous parameters defined [Bug #18828]Nobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/6016
2022-06-06Ignore invalid escapes in regexp commentsJeremy Evans
Invalid escapes are handled at multiple levels. The first level is in parse.y, so skip invalid unicode escape checks for regexps in parse.y. Make rb_reg_preprocess and unescape_nonascii accept the regexp options. In unescape_nonascii, if the regexp is an extended regexp, when "#" is encountered, ignore all characters until the end of line or end of regexp. Unfortunately, in extended regexps, you can use "#" as a non-comment character inside a character class, so also parse "[" and "]" specially for extended regexps, and only skip comments if "#" is not inside a character class. Handle nested character classes as well. This issue doesn't just affect extended regexps, it also affects "(#?" comments inside all regexps. So for those comments, scan until trailing ")" and ignore content inside. I'm not sure if there are other corner cases not handled. A better fix would be to redesign the regexp parser so that it unescaped during parsing instead of before parsing, so you already know the current parsing state. Fixes [Bug #18294] Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org> Notes: Merged: https://github.com/ruby/ruby/pull/5721 Merged-By: jeremyevans <code@jeremyevans.net>
2022-04-21Private local variables should shadow outer variables [Bug #18629]Nobuyoshi Nakada
2022-04-18Fix the wrong index of the previous component [Bug #18739]Nobuyoshi Nakada
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-01 08:09:44 +0000