| Age | Commit message (Collapse) | Author |
|---|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11716
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11702
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11713
|
|
Since #11698, `parser_str_new` makes `rb_parser_string_t` and `VALUE`
but discards the former, and then `STR_NEW3` makes the same thing
again.
Notes:
Merged: https://github.com/ruby/ruby/pull/11710
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11710
|
|
Changed to use `rb_parser_is_ascii_string` function instead of `is_ascii_string` function
Notes:
Merged: https://github.com/ruby/ruby/pull/11698
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11701
|
|
|
|
The allocated parser string is never freed, which causes a memory leak.
The following code leaks memory:
Ripper.sexp_raw(DATA.read)
__END__
<<~EOF
a
#{1}
a
EOF
Notes:
Merged: https://github.com/ruby/ruby/pull/11669
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11663
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11589
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11661
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11661
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11661
|
|
This change is reduce Ruby C API dependency for Universal Parser.
Reuse dedent_string functions in rb_ruby_ripper_dedent_string functions and remove dependencies on rb_str_modify and rb_str_set_len from the parser.
Notes:
Merged: https://github.com/ruby/ruby/pull/11658
|
|
optimized away
In cases where break/next/redo are not valid syntax, they should
raise a SyntaxError even if inside a conditional block that is
optimized away.
Fixes [Bug #20597]
Co-authored-by: Kevin Newton <kddnewton@gmail.com>
Notes:
Merged: https://github.com/ruby/ruby/pull/11099
Merged-By: jeremyevans <code@jeremyevans.net>
|
|
Fixes [Bug #20720]
Notes:
Merged: https://github.com/ruby/ruby/pull/11565
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11584
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11579
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11553
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11543
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11531
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11530
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11523
|
|
`set_parser_s_value` does nothing in parser therefore no need to
create string object in parser `set_yylval_node`.
# Object allocation
Run `ruby benchmarks/lobsters/benchmark.rb` with the patch
```diff
diff --git a/benchmarks/lobsters/benchmark.rb b/benchmarks/lobsters/benchmark.rb
index 240c50c..6cdd0ac 100644
--- a/benchmarks/lobsters/benchmark.rb
+++ b/benchmarks/lobsters/benchmark.rb
@@ -7,6 +7,8 @@ Dir.chdir __dir__
use_gemfile
require_relative 'config/environment'
+printf "allocated_after_load=%d\n", GC.stat(:total_allocated_objects)
+exit
require_relative "route_generator"
# For an in-mem DB, we need to load all data on every boot
```
## Before
```
ruby 3.4.0dev (2024-08-31T18:30:25Z master d6fc8f3d57) [arm64-darwin21]
...
allocated_after_load=2143519
```
## After
```
ruby 3.4.0dev (2024-09-01T00:40:04Z fix_bugs_20695 d1bae52f75) [arm64-darwin21]
...
allocated_after_load=1579662
```
## Ruby 3.3.0 for reference
```
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin21]
...
allocated_after_load=1732702
```
Notes:
Merged: https://github.com/ruby/ruby/pull/11522
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11451
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11411
|
|
For example, the following code leaks:
class MyRipper < Ripper
def initialize(src, &blk)
super(src)
@blk = blk
end
def warn(msg, *args) = @blk.call(msg)
end
$VERBOSE = true
def call_parse = MyRipper.new("if true\n end\n") { |msg| return msg }.parse
10.times do
500_000.times do
call_parse
end
puts `ps -o rss= -p #{$$}`
end
Before:
37536
53744
70064
86448
102576
119120
135248
151216
167744
183824
After:
19280
19696
19728
20336
20448
21408
21616
21616
21824
21840
Notes:
Merged: https://github.com/ruby/ruby/pull/11289
|
|
For example, the following code leaks:
class MyRipper < Ripper
def initialize(src, &blk)
super(src)
@blk = blk
end
def warn(msg, *args) = @blk.call(msg)
end
$VERBOSE = true
def call_parse = MyRipper.new("if true\n end\n") { |msg| return msg }.parse
10.times do
500_000.times do
call_parse
end
puts `ps -o rss= -p #{$$}`
end
Before:
34832
51952
69760
88048
105344
123040
141152
159152
176656
194272
After:
18400
20256
20272
20272
20272
20304
20368
20368
20368
20400
Notes:
Merged: https://github.com/ruby/ruby/pull/11288
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11294
|
|
For example, the following script leaks:
class MyRipper < Ripper
def initialize(src, &blk)
super(src)
@blk = blk
end
def compile_error(msg) = @blk.call(msg)
end
def call_parse = MyRipper.new("/") { |msg| return msg }.parse
10.times do
100_000.times do
call_parse
end
puts `ps -o rss= -p #{$$}`
end
Before:
93952
169040
244224
318784
394432
468224
544048
618560
693776
768384
After:
19776
19776
20352
20880
20912
21408
21328
21152
21472
20944
Notes:
Merged: https://github.com/ruby/ruby/pull/11287
|
|
Extracrt exception variable into `nd_exc_var` field
to keep the original grammar structure.
For example:
```
begin
rescue Error => e1
end
```
Before:
```
@ NODE_RESBODY (id: 8, line: 2, location: (2,0)-(2,18))
+- nd_args:
| @ NODE_LIST (id: 2, line: 2, location: (2,7)-(2,12))
| +- as.nd_alen: 1
| +- nd_head:
| | @ NODE_CONST (id: 1, line: 2, location: (2,7)-(2,12))
| | +- nd_vid: :Error
| +- nd_next:
| (null node)
+- nd_body:
| @ NODE_BLOCK (id: 6, line: 2, location: (2,13)-(2,18))
| +- nd_head (1):
| | @ NODE_LASGN (id: 3, line: 2, location: (2,13)-(2,18))
| | +- nd_vid: :e1
| | +- nd_value:
| | @ NODE_ERRINFO (id: 5, line: 2, location: (2,13)-(2,18))
| +- nd_head (2):
| @ NODE_BEGIN (id: 4, line: 2, location: (2,18)-(2,18))
| +- nd_body:
| (null node)
+- nd_next:
(null node)
```
After:
```
@ NODE_RESBODY (id: 6, line: 2, location: (2,0)-(2,18))
+- nd_args:
| @ NODE_LIST (id: 2, line: 2, location: (2,7)-(2,12))
| +- as.nd_alen: 1
| +- nd_head:
| | @ NODE_CONST (id: 1, line: 2, location: (2,7)-(2,12))
| | +- nd_vid: :Error
| +- nd_next:
| (null node)
+- nd_exc_var:
| @ NODE_LASGN (id: 3, line: 2, location: (2,13)-(2,18))
| +- nd_vid: :e1
| +- nd_value:
| @ NODE_ERRINFO (id: 5, line: 2, location: (2,13)-(2,18))
+- nd_body:
| @ NODE_BEGIN (id: 4, line: 2, location: (2,18)-(2,18))
| +- nd_body:
| (null node)
+- nd_next:
(null node)
```
Notes:
Merged: https://github.com/ruby/ruby/pull/11243
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11234
|
|
When loading a non-ASCII compatible file, an error is raised which
causes memory leak.
For example:
require "tempfile"
Tempfile.create do |f|
f.write("# -*- coding: UTF-16BE -*-")
f.flush
10.times do
20_000.times do
begin
load(f.path)
rescue
end
end
puts `ps -o rss= -p #{$$}`
end
end
Before:
33904
49072
64528
79216
94576
109504
124768
139536
154928
170256
After:
19568
21296
21664
21728
22192
22256
22416
22272
22272
22272
Notes:
Merged: https://github.com/ruby/ruby/pull/11220
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11227
|
|
`$5`, `brace_block` is no longer assigned in this action.
|
|
call
Notes:
Merged: https://github.com/ruby/ruby/pull/11215
|
|
For example:
```
undef a, b
```
Before:
```
@ NODE_UNDEF (id: 1, line: 1, location: (1,6)-(1,10))*
```
After:
```
@ NODE_UNDEF (id: 1, line: 1, location: (1,0)-(1,10))*
```
Notes:
Merged: https://github.com/ruby/ruby/pull/11214
|
|
Change UNDEF Node to hold their items to keep the original grammar
structure.
For example:
```
undef a, b
```
Before:
```
@ NODE_BLOCK (id: 4, line: 1, location: (1,6)-(1,10))*
+- nd_head (1):
| @ NODE_UNDEF (id: 1, line: 1, location: (1,6)-(1,7))
| +- nd_undef:
| @ NODE_SYM (id: 0, line: 1, location: (1,6)-(1,7))
| +- string: :a
+- nd_head (2):
@ NODE_UNDEF (id: 3, line: 1, location: (1,9)-(1,10))
+- nd_undef:
@ NODE_SYM (id: 2, line: 1, location: (1,9)-(1,10))
+- string: :b
```
After:
```
@ NODE_UNDEF (id: 1, line: 1, location: (1,6)-(1,10))*
+- nd_undefs:
+- length: 2
+- element (0):
| @ NODE_SYM (id: 0, line: 1, location: (1,6)-(1,7))
| +- string: :a
+- element (1):
@ NODE_SYM (id: 2, line: 1, location: (1,9)-(1,10))
+- string: :b
```
Notes:
Merged: https://github.com/ruby/ruby/pull/11213
|
|
For example:
10.times do
100_000.times do
RubyVM::AbstractSyntaxTree.parse("x = 1 + 2 +", keep_tokens: true)
rescue SyntaxError
end
puts `ps -o rss= -p #{$$}`
end
Before:
28944
44816
60720
76496
92336
108160
123968
139808
155648
171408
After:
11984
12704
12816
12832
13072
13088
13088
13136
13136
13152
Notes:
Merged: https://github.com/ruby/ruby/pull/11193
|
|
We face `[BUG] unexpected rb_parser_ary_data_type (0) for script lines`
on master branch recently.
This commit changes `enum rb_parser_ary_data_type` to start with `1`
and `0` to be invalid then it makes clear `rb_parser_ary_data_type (0)`
is not intentional.
|
|
This was an optimization for versions prior to 1.9 that traverse the
AST at runtime.
|
|
|
|
This patch optimizes forwarding callers and callees. It only optimizes methods that only take `...` as their parameter, and then pass `...` to other calls.
Calls it optimizes look like this:
```ruby
def bar(a) = a
def foo(...) = bar(...) # optimized
foo(123)
```
```ruby
def bar(a) = a
def foo(...) = bar(1, 2, ...) # optimized
foo(123)
```
```ruby
def bar(*a) = a
def foo(...)
list = [1, 2]
bar(*list, ...) # optimized
end
foo(123)
```
All variants of the above but using `super` are also optimized, including a bare super like this:
```ruby
def foo(...)
super
end
```
This patch eliminates intermediate allocations made when calling methods that accept `...`.
We can observe allocation elimination like this:
```ruby
def m
x = GC.stat(:total_allocated_objects)
yield
GC.stat(:total_allocated_objects) - x
end
def bar(a) = a
def foo(...) = bar(...)
def test
m { foo(123) }
end
test
p test # allocates 1 object on master, but 0 objects with this patch
```
```ruby
def bar(a, b:) = a + b
def foo(...) = bar(...)
def test
m { foo(1, b: 2) }
end
test
p test # allocates 2 objects on master, but 0 objects with this patch
```
How does it work?
-----------------
This patch works by using a dynamic stack size when passing forwarded parameters to callees.
The caller's info object (known as the "CI") contains the stack size of the
parameters, so we pass the CI object itself as a parameter to the callee.
When forwarding parameters, the forwarding ISeq uses the caller's CI to determine how much stack to copy, then copies the caller's stack before calling the callee.
The CI at the forwarded call site is adjusted using information from the caller's CI.
I think this description is kind of confusing, so let's walk through an example with code.
```ruby
def delegatee(a, b) = a + b
def delegator(...)
delegatee(...) # CI2 (FORWARDING)
end
def caller
delegator(1, 2) # CI1 (argc: 2)
end
```
Before we call the delegator method, the stack looks like this:
```
Executing Line | Code | Stack
---------------+---------------------------------------+--------
1| def delegatee(a, b) = a + b | self
2| | 1
3| def delegator(...) | 2
4| # |
5| delegatee(...) # CI2 (FORWARDING) |
6| end |
7| |
8| def caller |
-> 9| delegator(1, 2) # CI1 (argc: 2) |
10| end |
```
The ISeq for `delegator` is tagged as "forwardable", so when `caller` calls in
to `delegator`, it writes `CI1` on to the stack as a local variable for the
`delegator` method. The `delegator` method has a special local called `...`
that holds the caller's CI object.
Here is the ISeq disasm fo `delegator`:
```
== disasm: #<ISeq:delegator@-e:1 (1,0)-(1,39)>
local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
[ 1] "..."@0
0000 putself ( 1)[LiCa]
0001 getlocal_WC_0 "..."@0
0003 send <calldata!mid:delegatee, argc:0, FCALL|FORWARDING>, nil
0006 leave [Re]
```
The local called `...` will contain the caller's CI: CI1.
Here is the stack when we enter `delegator`:
```
Executing Line | Code | Stack
---------------+---------------------------------------+--------
1| def delegatee(a, b) = a + b | self
2| | 1
3| def delegator(...) | 2
-> 4| # | CI1 (argc: 2)
5| delegatee(...) # CI2 (FORWARDING) | cref_or_me
6| end | specval
7| | type
8| def caller |
9| delegator(1, 2) # CI1 (argc: 2) |
10| end |
```
The CI at `delegatee` on line 5 is tagged as "FORWARDING", so it knows to
memcopy the caller's stack before calling `delegatee`. In this case, it will
memcopy self, 1, and 2 to the stack before calling `delegatee`. It knows how much
memory to copy from the caller because `CI1` contains stack size information
(argc: 2).
Before executing the `send` instruction, we push `...` on the stack. The
`send` instruction pops `...`, and because it is tagged with `FORWARDING`, it
knows to memcopy (using the information in the CI it just popped):
```
== disasm: #<ISeq:delegator@-e:1 (1,0)-(1,39)>
local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
[ 1] "..."@0
0000 putself ( 1)[LiCa]
0001 getlocal_WC_0 "..."@0
0003 send <calldata!mid:delegatee, argc:0, FCALL|FORWARDING>, nil
0006 leave [Re]
```
Instruction 001 puts the caller's CI on the stack. `send` is tagged with
FORWARDING, so it reads the CI and _copies_ the callers stack to this stack:
```
Executing Line | Code | Stack
---------------+---------------------------------------+--------
1| def delegatee(a, b) = a + b | self
2| | 1
3| def delegator(...) | 2
4| # | CI1 (argc: 2)
-> 5| delegatee(...) # CI2 (FORWARDING) | cref_or_me
6| end | specval
7| | type
8| def caller | self
9| delegator(1, 2) # CI1 (argc: 2) | 1
10| end | 2
```
The "FORWARDING" call site combines information from CI1 with CI2 in order
to support passing other values in addition to the `...` value, as well as
perfectly forward splat args, kwargs, etc.
Since we're able to copy the stack from `caller` in to `delegator`'s stack, we
can avoid allocating objects.
I want to do this to eliminate object allocations for delegate methods.
My long term goal is to implement `Class#new` in Ruby and it uses `...`.
I was able to implement `Class#new` in Ruby
[here](https://github.com/ruby/ruby/pull/9289).
If we adopt the technique in this patch, then we can optimize allocating
objects that take keyword parameters for `initialize`.
For example, this code will allocate 2 objects: one for `SomeObject`, and one
for the kwargs:
```ruby
SomeObject.new(foo: 1)
```
If we combine this technique, plus implement `Class#new` in Ruby, then we can
reduce allocations for this common operation.
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Co-Authored-By: Alan Wu <XrXr@users.noreply.github.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
