summaryrefslogtreecommitdiff
path: root/parse.y
AgeCommit message (Collapse)Author
2024-09-05Implement AND/OR NODE operator locationsydah
Notes: Merged: https://github.com/ruby/ruby/pull/11543
2024-09-04Implement VALIAS NODE keyword locationsydah
Notes: Merged: https://github.com/ruby/ruby/pull/11531
2024-09-03Implement ALIAS NODE keyword locationsydah
Notes: Merged: https://github.com/ruby/ruby/pull/11530
2024-09-03Implement UNDEF NODE keyword locationsydah
Notes: Merged: https://github.com/ruby/ruby/pull/11523
2024-09-03[Bug #20695] Do not create needless string object in parseryui-knk
`set_parser_s_value` does nothing in parser therefore no need to create string object in parser `set_yylval_node`. # Object allocation Run `ruby benchmarks/lobsters/benchmark.rb` with the patch ```diff diff --git a/benchmarks/lobsters/benchmark.rb b/benchmarks/lobsters/benchmark.rb index 240c50c..6cdd0ac 100644 --- a/benchmarks/lobsters/benchmark.rb +++ b/benchmarks/lobsters/benchmark.rb @@ -7,6 +7,8 @@ Dir.chdir __dir__ use_gemfile require_relative 'config/environment' +printf "allocated_after_load=%d\n", GC.stat(:total_allocated_objects) +exit require_relative "route_generator" # For an in-mem DB, we need to load all data on every boot ``` ## Before ``` ruby 3.4.0dev (2024-08-31T18:30:25Z master d6fc8f3d57) [arm64-darwin21] ... allocated_after_load=2143519 ``` ## After ``` ruby 3.4.0dev (2024-09-01T00:40:04Z fix_bugs_20695 d1bae52f75) [arm64-darwin21] ... allocated_after_load=1579662 ``` ## Ruby 3.3.0 for reference ``` ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin21] ... allocated_after_load=1732702 ``` Notes: Merged: https://github.com/ruby/ruby/pull/11522
2024-08-25[Bug #20680] `ensure` block is always void contextNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/11451
2024-08-20Make same structures sameNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/11411
2024-08-07Fix leak of token_info when Ripper#warn jumpsPeter Zhu
For example, the following code leaks: class MyRipper < Ripper def initialize(src, &blk) super(src) @blk = blk end def warn(msg, *args) = @blk.call(msg) end $VERBOSE = true def call_parse = MyRipper.new("if true\n end\n") { |msg| return msg }.parse 10.times do 500_000.times do call_parse end puts `ps -o rss= -p #{$$}` end Before: 37536 53744 70064 86448 102576 119120 135248 151216 167744 183824 After: 19280 19696 19728 20336 20448 21408 21616 21616 21824 21840 Notes: Merged: https://github.com/ruby/ruby/pull/11289
2024-08-06Fix leak in warning of duplicate keys when Ripper#warn jumpsPeter Zhu
For example, the following code leaks: class MyRipper < Ripper def initialize(src, &blk) super(src) @blk = blk end def warn(msg, *args) = @blk.call(msg) end $VERBOSE = true def call_parse = MyRipper.new("if true\n end\n") { |msg| return msg }.parse 10.times do 500_000.times do call_parse end puts `ps -o rss= -p #{$$}` end Before: 34832 51952 69760 88048 105344 123040 141152 159152 176656 194272 After: 18400 20256 20272 20272 20272 20304 20368 20368 20368 20400 Notes: Merged: https://github.com/ruby/ruby/pull/11288
2024-08-02Refactor to use `tokenize_ident` instead of `TOK_INTERN` and `set_yylval_name`yui-knk
Notes: Merged: https://github.com/ruby/ruby/pull/11294
2024-07-31Fix leak of AST when Ripper#compile_error jumpsPeter Zhu
For example, the following script leaks: class MyRipper < Ripper def initialize(src, &blk) super(src) @blk = blk end def compile_error(msg) = @blk.call(msg) end def call_parse = MyRipper.new("/") { |msg| return msg }.parse 10.times do 100_000.times do call_parse end puts `ps -o rss= -p #{$$}` end Before: 93952 169040 244224 318784 394432 468224 544048 618560 693776 768384 After: 19776 19776 20352 20880 20912 21408 21328 21152 21472 20944 Notes: Merged: https://github.com/ruby/ruby/pull/11287
2024-07-26Change RESBODY Node structureyui-knk
Extracrt exception variable into `nd_exc_var` field to keep the original grammar structure. For example: ``` begin rescue Error => e1 end ``` Before: ``` @ NODE_RESBODY (id: 8, line: 2, location: (2,0)-(2,18)) +- nd_args: | @ NODE_LIST (id: 2, line: 2, location: (2,7)-(2,12)) | +- as.nd_alen: 1 | +- nd_head: | | @ NODE_CONST (id: 1, line: 2, location: (2,7)-(2,12)) | | +- nd_vid: :Error | +- nd_next: | (null node) +- nd_body: | @ NODE_BLOCK (id: 6, line: 2, location: (2,13)-(2,18)) | +- nd_head (1): | | @ NODE_LASGN (id: 3, line: 2, location: (2,13)-(2,18)) | | +- nd_vid: :e1 | | +- nd_value: | | @ NODE_ERRINFO (id: 5, line: 2, location: (2,13)-(2,18)) | +- nd_head (2): | @ NODE_BEGIN (id: 4, line: 2, location: (2,18)-(2,18)) | +- nd_body: | (null node) +- nd_next: (null node) ``` After: ``` @ NODE_RESBODY (id: 6, line: 2, location: (2,0)-(2,18)) +- nd_args: | @ NODE_LIST (id: 2, line: 2, location: (2,7)-(2,12)) | +- as.nd_alen: 1 | +- nd_head: | | @ NODE_CONST (id: 1, line: 2, location: (2,7)-(2,12)) | | +- nd_vid: :Error | +- nd_next: | (null node) +- nd_exc_var: | @ NODE_LASGN (id: 3, line: 2, location: (2,13)-(2,18)) | +- nd_vid: :e1 | +- nd_value: | @ NODE_ERRINFO (id: 5, line: 2, location: (2,13)-(2,18)) +- nd_body: | @ NODE_BEGIN (id: 4, line: 2, location: (2,18)-(2,18)) | +- nd_body: | (null node) +- nd_next: (null node) ``` Notes: Merged: https://github.com/ruby/ruby/pull/11243
2024-07-24[Bug #20647] Disallow `return` directly within a singleton classNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/11234
2024-07-23Fix memory leak in parser when loading non-ASCII filePeter Zhu
When loading a non-ASCII compatible file, an error is raised which causes memory leak. For example: require "tempfile" Tempfile.create do |f| f.write("# -*- coding: UTF-16BE -*-") f.flush 10.times do 20_000.times do begin load(f.path) rescue end end puts `ps -o rss= -p #{$$}` end end Before: 33904 49072 64528 79216 94576 109504 124768 139536 154928 170256 After: 19568 21296 21664 21728 22192 22256 22416 22272 22272 22272 Notes: Merged: https://github.com/ruby/ruby/pull/11220
2024-07-23Implement UNLESS NODE keyword locationsyui-knk
Notes: Merged: https://github.com/ruby/ruby/pull/11227
2024-07-21Remove unneeded local variableNobuyoshi Nakada
`$5`, `brace_block` is no longer assigned in this action.
2024-07-21Fix SEGV on method call with empty args and brace block for do block command ↵yui-knk
call Notes: Merged: https://github.com/ruby/ruby/pull/11215
2024-07-20Include `undef` keyword into UNDEF NODE locationyui-knk
For example: ``` undef a, b ``` Before: ``` @ NODE_UNDEF (id: 1, line: 1, location: (1,6)-(1,10))* ``` After: ``` @ NODE_UNDEF (id: 1, line: 1, location: (1,0)-(1,10))* ``` Notes: Merged: https://github.com/ruby/ruby/pull/11214
2024-07-20Change UNDEF Node structureyui-knk
Change UNDEF Node to hold their items to keep the original grammar structure. For example: ``` undef a, b ``` Before: ``` @ NODE_BLOCK (id: 4, line: 1, location: (1,6)-(1,10))* +- nd_head (1): | @ NODE_UNDEF (id: 1, line: 1, location: (1,6)-(1,7)) | +- nd_undef: | @ NODE_SYM (id: 0, line: 1, location: (1,6)-(1,7)) | +- string: :a +- nd_head (2): @ NODE_UNDEF (id: 3, line: 1, location: (1,9)-(1,10)) +- nd_undef: @ NODE_SYM (id: 2, line: 1, location: (1,9)-(1,10)) +- string: :b ``` After: ``` @ NODE_UNDEF (id: 1, line: 1, location: (1,6)-(1,10))* +- nd_undefs: +- length: 2 +- element (0): | @ NODE_SYM (id: 0, line: 1, location: (1,6)-(1,7)) | +- string: :a +- element (1): @ NODE_SYM (id: 2, line: 1, location: (1,9)-(1,10)) +- string: :b ``` Notes: Merged: https://github.com/ruby/ruby/pull/11213
2024-07-18Free `data` of `struct rb_parser_ary` in `rb_parser_ary_free`yui-knk
For example: 10.times do 100_000.times do RubyVM::AbstractSyntaxTree.parse("x = 1 + 2 +", keep_tokens: true) rescue SyntaxError end puts `ps -o rss= -p #{$$}` end Before: 28944 44816 60720 76496 92336 108160 123968 139808 155648 171408 After: 11984 12704 12816 12832 13072 13088 13088 13136 13136 13152 Notes: Merged: https://github.com/ruby/ruby/pull/11193
2024-06-26Change `enum rb_parser_ary_data_type` default value to 1 for easy debugyui-knk
We face `[BUG] unexpected rb_parser_ary_data_type (0) for script lines` on master branch recently. This commit changes `enum rb_parser_ary_data_type` to start with `1` and `0` to be invalid then it makes clear `rb_parser_ary_data_type (0)` is not intentional.
2024-06-25[Bug #20457] Do not remove final `return` nodeNobuyoshi Nakada
This was an optimization for versions prior to 1.9 that traverse the AST at runtime.
2024-06-25Parenthesize `nd_fl_newline` macro expressionsNobuyoshi Nakada
2024-06-18Optimized forwarding callers and calleesAaron Patterson
This patch optimizes forwarding callers and callees. It only optimizes methods that only take `...` as their parameter, and then pass `...` to other calls. Calls it optimizes look like this: ```ruby def bar(a) = a def foo(...) = bar(...) # optimized foo(123) ``` ```ruby def bar(a) = a def foo(...) = bar(1, 2, ...) # optimized foo(123) ``` ```ruby def bar(*a) = a def foo(...) list = [1, 2] bar(*list, ...) # optimized end foo(123) ``` All variants of the above but using `super` are also optimized, including a bare super like this: ```ruby def foo(...) super end ``` This patch eliminates intermediate allocations made when calling methods that accept `...`. We can observe allocation elimination like this: ```ruby def m x = GC.stat(:total_allocated_objects) yield GC.stat(:total_allocated_objects) - x end def bar(a) = a def foo(...) = bar(...) def test m { foo(123) } end test p test # allocates 1 object on master, but 0 objects with this patch ``` ```ruby def bar(a, b:) = a + b def foo(...) = bar(...) def test m { foo(1, b: 2) } end test p test # allocates 2 objects on master, but 0 objects with this patch ``` How does it work? ----------------- This patch works by using a dynamic stack size when passing forwarded parameters to callees. The caller's info object (known as the "CI") contains the stack size of the parameters, so we pass the CI object itself as a parameter to the callee. When forwarding parameters, the forwarding ISeq uses the caller's CI to determine how much stack to copy, then copies the caller's stack before calling the callee. The CI at the forwarded call site is adjusted using information from the caller's CI. I think this description is kind of confusing, so let's walk through an example with code. ```ruby def delegatee(a, b) = a + b def delegator(...) delegatee(...) # CI2 (FORWARDING) end def caller delegator(1, 2) # CI1 (argc: 2) end ``` Before we call the delegator method, the stack looks like this: ``` Executing Line | Code | Stack ---------------+---------------------------------------+-------- 1| def delegatee(a, b) = a + b | self 2| | 1 3| def delegator(...) | 2 4| # | 5| delegatee(...) # CI2 (FORWARDING) | 6| end | 7| | 8| def caller | -> 9| delegator(1, 2) # CI1 (argc: 2) | 10| end | ``` The ISeq for `delegator` is tagged as "forwardable", so when `caller` calls in to `delegator`, it writes `CI1` on to the stack as a local variable for the `delegator` method. The `delegator` method has a special local called `...` that holds the caller's CI object. Here is the ISeq disasm fo `delegator`: ``` == disasm: #<ISeq:delegator@-e:1 (1,0)-(1,39)> local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] "..."@0 0000 putself ( 1)[LiCa] 0001 getlocal_WC_0 "..."@0 0003 send <calldata!mid:delegatee, argc:0, FCALL|FORWARDING>, nil 0006 leave [Re] ``` The local called `...` will contain the caller's CI: CI1. Here is the stack when we enter `delegator`: ``` Executing Line | Code | Stack ---------------+---------------------------------------+-------- 1| def delegatee(a, b) = a + b | self 2| | 1 3| def delegator(...) | 2 -> 4| # | CI1 (argc: 2) 5| delegatee(...) # CI2 (FORWARDING) | cref_or_me 6| end | specval 7| | type 8| def caller | 9| delegator(1, 2) # CI1 (argc: 2) | 10| end | ``` The CI at `delegatee` on line 5 is tagged as "FORWARDING", so it knows to memcopy the caller's stack before calling `delegatee`. In this case, it will memcopy self, 1, and 2 to the stack before calling `delegatee`. It knows how much memory to copy from the caller because `CI1` contains stack size information (argc: 2). Before executing the `send` instruction, we push `...` on the stack. The `send` instruction pops `...`, and because it is tagged with `FORWARDING`, it knows to memcopy (using the information in the CI it just popped): ``` == disasm: #<ISeq:delegator@-e:1 (1,0)-(1,39)> local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] "..."@0 0000 putself ( 1)[LiCa] 0001 getlocal_WC_0 "..."@0 0003 send <calldata!mid:delegatee, argc:0, FCALL|FORWARDING>, nil 0006 leave [Re] ``` Instruction 001 puts the caller's CI on the stack. `send` is tagged with FORWARDING, so it reads the CI and _copies_ the callers stack to this stack: ``` Executing Line | Code | Stack ---------------+---------------------------------------+-------- 1| def delegatee(a, b) = a + b | self 2| | 1 3| def delegator(...) | 2 4| # | CI1 (argc: 2) -> 5| delegatee(...) # CI2 (FORWARDING) | cref_or_me 6| end | specval 7| | type 8| def caller | self 9| delegator(1, 2) # CI1 (argc: 2) | 1 10| end | 2 ``` The "FORWARDING" call site combines information from CI1 with CI2 in order to support passing other values in addition to the `...` value, as well as perfectly forward splat args, kwargs, etc. Since we're able to copy the stack from `caller` in to `delegator`'s stack, we can avoid allocating objects. I want to do this to eliminate object allocations for delegate methods. My long term goal is to implement `Class#new` in Ruby and it uses `...`. I was able to implement `Class#new` in Ruby [here](https://github.com/ruby/ruby/pull/9289). If we adopt the technique in this patch, then we can optimize allocating objects that take keyword parameters for `initialize`. For example, this code will allocate 2 objects: one for `SomeObject`, and one for the kwargs: ```ruby SomeObject.new(foo: 1) ``` If we combine this technique, plus implement `Class#new` in Ruby, then we can reduce allocations for this common operation. Co-Authored-By: John Hawthorn <john@hawthorn.email> Co-Authored-By: Alan Wu <XrXr@users.noreply.github.com>
2024-06-14[Bug #20579] ripper: Dispatch spaces at END-OF-INPUT without newlineNobuyoshi Nakada
2024-06-14Include `__LINE__` in `add_delayed_token` macroNobuyoshi Nakada
2024-06-14[Bug #20578] ripper: Fix dispatching part at invalid escapesNobuyoshi Nakada
2024-06-12Introduce `ident_or_const` inline ruleS-H-GAMELINKS
2024-06-12ripper: Unify `dispatch_end`Nobuyoshi Nakada
2024-06-09Use `dllexport` as `RUBY_FUNC_EXPORTED` on WindowsNobuyoshi Nakada
2024-06-08ripper: Unify formal argument error handlingNobuyoshi Nakada
2024-06-08ripper: Unify backref error handlingNobuyoshi Nakada
2024-06-08ripper: Introduce `RIPPER_ID` macro instead of `ripper_id_` macrosNobuyoshi Nakada
2024-06-07ripper: Fix excess `compile_error` at simple backref op_asgnNobuyoshi Nakada
Fix up 89cfc1520717257073012ec07105c551e4b8af7c.
2024-06-06Remove circular parameter syntax errorKevin Newton
https://bugs.ruby-lang.org/issues/20478
2024-06-06[Bug #20521] ripper: Clean up strtermNobuyoshi Nakada
2024-06-02Ditto for NODE_DOT2 and NODE_DOT3Nobuyoshi Nakada
2024-06-02Use `RNode_DREGX` variable for debuggersNobuyoshi Nakada
At least LLDB needs an actual variable not only casts to access the type in debugger sessions.
2024-06-02Make interchangeable NODE types aliasesNobuyoshi Nakada
2024-06-01Get rid of type-punning pointer castsNobuyoshi Nakada
2024-06-01[Bug #20517] Make a multibyte character one token at meta escapeNobuyoshi Nakada
2024-05-31Make error messages clear blocks/keywords are disallowed in index assignmentJeremy Evans
Blocks and keywords are allowed in regular index. Also update NEWS to make this more clear. Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
2024-05-31Revert 528c4501f46fbe1e06028d673a777ef124d29829Yusuke Endoh
Recently, `TestRubyLiteral#test_float` fails randomly. ``` 1) Error: TestRubyLiteral#test_float: ArgumentError: SyntaxError#path changed: "(eval at /home/chkbuild/chkbuild/tmp/build/20240527T050036Z/ruby/test/ruby/test_literal.rb:642)"->"(eval at /home/chkbuild/chkbuild/tmp/build/20240527T050036Z/ruby/test/ruby/test_literal.rb:642)" ``` https://rubyci.s3.amazonaws.com/s390x/ruby-master/log/20240527T050036Z.fail.html.gz According to Launchable, the first failure was on Apr 30. This is just when 528c4501f46fbe1e06028d673a777ef124d29829 was committed. I don't know if the change is really the cause, but I want to revert it once to see if the random failure disappears.
2024-05-24Update duplicated when clause warning messageKevin Newton
2024-05-23Remove dead codeNobuyoshi Nakada
Since 140512d2225e6fd046ba1bdbcd1a27450f55c233, `else` without `rescue` has been a syntax error.
2024-05-23Add RB_GC_GUARD for rb_str_to_parser_stringYusuke Endoh
I think this fixes the following random test failure that could not be fixed for a long time: ``` 1) Failure: TestSymbol#test_inspect_under_gc_compact_stress [/home/chkbuild/chkbuild/tmp/build/20240522T003003Z/ruby/test/ruby/test_symbol.rb:126]: <":testing"> expected but was <":\"\\x00\\x00\\x00\\x00\\x00\\x00\\x00\"">. ``` The value passed to this function is the return value of `rb_id2str`, so it is never collected. However, if auto_compact is enabled, the string may move and `RSTRING_PTR(str)` became invalid. This change prevents the string from being moved by RB_GC_GUARD.
2024-05-21ripper: Splat find patternsNobuyoshi Nakada
2024-05-21ripper: Splat hash patternsNobuyoshi Nakada
2024-05-21ripper: Splat array patterns with `pre_arg`Nobuyoshi Nakada
2024-05-21ripper: Splat `$:opt_args_tail` for `params!`Nobuyoshi Nakada
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-18 07:39:27 +0000