| Age | Commit message (Collapse) | Author |
|---|
|
|
|
[Feature #21084]
# Summary
The current way of marking weak references uses `rb_gc_mark_weak(VALUE *ptr)`.
This presents challenges because Ruby's GC is incremental, meaning that if the
`ptr` changes (e.g. realloc'd or free'd), then we could have an invalid memory
access. This also overwrites `*ptr = Qundef` if `*ptr` is dead, which prevents
any cleanup to be run (e.g. freeing memory or deleting entries from hash
tables). This ticket proposes `rb_gc_declare_weak_references` which declares
that an object has weak references and calls a cleanup function after marking,
allowing the object to clean up any memory for dead objects.
# Introduction
In [[Feature #19783]](https://bugs.ruby-lang.org/issues/19783), I introduced an
API allowing objects to mark weak references, the function signature looks like
this:
```c
void rb_gc_mark_weak(VALUE *ptr);
```
`rb_gc_mark_weak` is called during the marking phase of the GC to specify that
the memory at `ptr` holds a pointer to a Ruby object that is weakly referenced.
`rb_gc_mark_weak` appends this pointer to a list that is processed after the
marking phase of the GC. If the object at `*ptr` is no longer alive, then it
overwrites the object reference with a special value (`*ptr = Qundef`).
However, this API resulted in two challenges:
1. Ruby's default GC is incremental, which means that the GC is not ran in one
phase, but rather split into chunks of work that interleaves with Ruby
execution. The `ptr` passed into `rb_gc_mark_weak` could be on the malloc
heap, and that memory could be realloc'd or even free'd. We had to use
workarounds such as `rb_gc_remove_weak` to ensure that there were no illegal
memory accesses. This made `rb_gc_mark_weak` difficult to use, impacted
runtime performance, and increased memory usage.
2. When an object dies, `rb_gc_mark_weak` only overwites the reference with
`Qundef`. This means that if we want to do any cleanup (e.g. free a piece of
memory or delete a hash table entry), we could not do that and had to defer
this process elsewhere (e.g. during marking or runtime).
In this ticket, I'm proposing a new API for weak references. Instead of an
object marking its weak references during the marking phase, the object declares
that it has weak references using the `rb_gc_declare_weak_references` function.
This declaration occurs during runtime (e.g. after the object has been created)
rather than during GC.
After an object declares that it has weak references, it will have its callback
function called after marking as long as that object is alive. This callback
function can then call a special function `rb_gc_handle_weak_references_alive_p`
to determine whether its references are alive. This will allow the callback
function to do whatever it wants on the object, allowing it to perform any
cleanup work it needs.
This significantly simplifies the code for `ObjectSpace::WeakMap` and
`ObjectSpace::WeakKeyMap` because it no longer needs to have the workarounds for
the limitations of `rb_gc_mark_weak`.
# Performance
The performance results below demonstrate that `ObjectSpace::WeakMap#[]=` is now
about 60% faster because the implementation has been simplified and the number
of allocations has been reduced. We can see that there is not a significant
impact on the performance of `ObjectSpace::WeakMap#[]`.
Base:
```
ObjectSpace::WeakMap#[]=
4.620M (± 6.4%) i/s (216.44 ns/i) - 23.342M in 5.072149s
ObjectSpace::WeakMap#[]
30.967M (± 1.9%) i/s (32.29 ns/i) - 154.998M in 5.007157s
```
Branch:
```
ObjectSpace::WeakMap#[]=
7.336M (± 2.8%) i/s (136.31 ns/i) - 36.755M in 5.013983s
ObjectSpace::WeakMap#[]
30.902M (± 5.4%) i/s (32.36 ns/i) - 155.901M in 5.064060s
```
Code:
```
require "bundler/inline"
gemfile do
source "https://rubygems.org"
gem "benchmark-ips"
end
wmap = ObjectSpace::WeakMap.new
key = Object.new
val = Object.new
wmap[key] = val
Benchmark.ips do |x|
x.report("ObjectSpace::WeakMap#[]=") do |times|
i = 0
while i < times
wmap[Object.new] = Object.new
i += 1
end
end
x.report("ObjectSpace::WeakMap#[]") do |times|
i = 0
while i < times
wmap[key]
wmap[val] # does not exist
i += 1
end
end
end
```
# Alternative designs
Currently, `rb_gc_declare_weak_references` is designed to be an internal-only
API. This allows us to assume the object types that call
`rb_gc_declare_weak_references`. In the future, if we want to open up this API
to third parties, we may want to change this function to something like:
```c
void rb_gc_add_cleaner(VALUE obj, void (*callback)(VALUE obj));
```
This will allow the third party to implement a custom `callback` that gets
called after the marking phase of GC to clean up any dead references. I chose
not to implement this design because it is less efficient as we would need to
store a mapping from `obj` to `callback`, which requires extra memory.
|
|
[Feature #20408]
|
|
[Feature #20408]
|
|
[Feature #20408]
|
|
We shouldn't run any ruby code with the VM lock held.
|
|
* See https://bugs.ruby-lang.org/issues/21613
|
|
This reverts commit c1c0b32445c66e343c136faa28d7a0f0f46d96a2.
This test is clearly not reliable:
https://github.com/ruby/ruby/actions/runs/17446920961/job/49543543423
I want to skip this unstable test on GitHub Actions to make sure this
test doesn't prevent PRs from getting merged. We can continue to monitor
the state of this test on RubyCI and ci.rvm.jp.
|
|
This fails way too often across many environments. I don't think this
test is healthy.
https://github.com/ruby/ruby/actions/runs/17343611722/job/49240735401
Before we give up on it, let me see if this helps.
|
|
|
|
https://github.com/ruby/ruby/actions/runs/16969921157/job/48103809963
https://github.com/ruby/ruby/actions/runs/16969655024/job/48102876839
|
|
gc_config_set returned rb_gc_impl_config_get, but gc_config_get also added
the implementation key to the return value. This caused the return value
of GC.config to differ depending on whether the optional hash argument is
provided or not.
|
|
The test sometimes fails with "Expected 2062788 to be < 2000000" because
heap 0 has not been cleared yet by GC. This commit fixes it to run GC
before the assertion to ensure that it does not flake.
|
|
|
|
Using an enumerator does not resolve the intermittent failures: 100+
failures in 10,000 iterations.
|
|
I can reproduce timeout failure with the high load machine:
```
$ RUBY_MAX_CPU=100 ruby -e '100.times{Ractor.new{loop{}}}; sleep' &
$ while make test-all -o exts -o encs TESTS='ruby/gc -n /test_interrupt_in_finalizer/ --repeat-count=100'; do date; done
....
Finished(93/100) tests in 0.653434s, 1.5304 tests/s, 7.6519 assertions/s.
Finished(94/100) tests in 0.614422s, 1.6275 tests/s, 8.1377 assertions/s.
[1/1] TestGc#test_interrupt_in_finalizer = 11.08 s
1) Timeout:
TestGc#test_interrupt_in_finalizer
```
|
|
To debug flaky failures on i686 that look like:
1) Failure:
TestGc#test_heaps_grow_independently [test/ruby/test_gc.rb:704]:
Expected 2061929 to be < 2000000.
Notes:
Merged: https://github.com/ruby/ruby/pull/13140
|
|
`RUBY_DEBUG=gc_stress ./miniruby -e0` crashes because of this
marking miss.
BTW, to use `RUBY_DEBUG=gc_stress` we need to specify
`--enable-debug-env` configure option. This is why I couldn't repro
on my environments.
see c0c94ab183d0d428595ccb74ae71ee945f1afd45
Notes:
Merged: https://github.com/ruby/ruby/pull/13506
|
|
```
1) Failure:
TestGc#test_gc_stress_at_startup [/home/chkbuild/chkbuild/tmp/build/20250530T213003Z/ruby/test/ruby/test_gc.rb:799]:
[Bug #15784]
pid 1748790 killed by SIGSEGV (signal 11) (core dumped).
1. [3/3] Assertion for "success?"
| Expected #<Process::Status: pid 1748790 SIGSEGV (signal 11) (core dumped)> to be success?.
```
|
|
Clear the ary variable before setting it to nil. Otherwise, if
the previous ary value was somewhere on the stack, all references
in it would be considered live, and the wmap size would be 10000.
|
|
[Bug #21214]
If we allocate objects where one heap holds transient objects and another
holds long lived objects, then the heap with transient objects will grow
along the heap with long lived objects, causing higher memory usage.
For example, we can see this issue in this script:
def allocate_small_object = []
def allocate_large_object = Array.new(10)
arys = Array.new(1_000_000) do
# Allocate 10 small transient objects
10.times { allocate_small_object }
# Allocate 1 large object that is persistent
allocate_large_object
end
pp GC.stat
pp GC.stat_heap
Before this change:
heap_live_slots: 2837243
{0 =>
{slot_size: 40,
heap_eden_pages: 1123,
heap_eden_slots: 1838807},
2 =>
{slot_size: 160,
heap_eden_pages: 2449,
heap_eden_slots: 1001149},
}
After this change:
heap_live_slots: 1094474
{0 =>
{slot_size: 40,
heap_eden_pages: 58,
heap_eden_slots: 94973},
2 =>
{slot_size: 160,
heap_eden_pages: 2449,
heap_eden_slots: 1001149},
}
Notes:
Merged: https://github.com/ruby/ruby/pull/13061
|
|
https://bugs.ruby-lang.org/issues/21203
TestGc#test_gc_parameter_init_slots is a flaky test that fails intermittently. Until the issue with flakiness is resolved, I will skip it.
Notes:
Merged-By: ono-max <onoto1998@gmail.com>
|
|
The test could flake because a major GC could be triggered due to allocation
for caches or other things, which would cause the test to fail.
|
|
The test fails sometimes with:
TestGc#test_latest_gc_info_weak_references_count [test/ruby/test_gc.rb:421]:
Expected 2 to be <= 1.
Notes:
Merged: https://github.com/ruby/ruby/pull/12894
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/12740
|
|
Instead of silently ignoring the key, we should raise to clearly tell
the user that this key is read-only.
Notes:
Merged: https://github.com/ruby/ruby/pull/11872
|
|
And a default and readonly key to the GC.config hash that names the
current GC implementation.
This is provided by each implementation by the API function
rb_gc_impl_active_gc_name
Notes:
Merged: https://github.com/ruby/ruby/pull/11872
|
|
I caught a reproduction of this test failing under rr, and was able to
replay it to isolate the failure. The call to
`before_stat_heap = GC.stat_heap` is itself allocating a hash, which in
unlucky circumstances can result in a new page being allocated and thus
`before_stats[:heap_allocated_pages]` no longer equals
`after_stats[:heap_allocated_pages]`.
The solution is to use the form of GC.stat/stat_heap which takes a hash
as an argument, and thus needs to perform no Ruby allocations itself.
Notes:
Merged: https://github.com/ruby/ruby/pull/11997
|
|
GC.config is always defined.
Notes:
Merged: https://github.com/ruby/ruby/pull/11867
|
|
|
|
Now that we've inlined the eden_heap into the size_pool, we should
rename the size_pool to heap. So that Ruby contains multiple heaps, with
different sized objects.
The term heap as a collection of memory pages is more in memory
management nomenclature, whereas size_pool was a name chosen out of
necessity during the development of the Variable Width Allocation
features of Ruby.
The concept of size pools was introduced in order to facilitate
different sized objects (other than the default 40 bytes). They wrapped
the eden heap and the tomb heap, and some related state, and provided a
reasonably simple way of duplicating all related concerns, to provide
multiple pools that all shared the same structure but held different
objects.
Since then various changes have happend in Ruby's memory layout:
* The concept of tomb heaps has been replaced by a global free pages list,
with each page having it's slot size reconfigured at the point when it
is resurrected
* the eden heap has been inlined into the size pool itself, so that now
the size pool directly controls the free_pages list, the sweeping
page, the compaction cursor and the other state that was previously
being managed by the eden heap.
Now that there is no need for a heap wrapper, we should refer to the
collection of pages containing Ruby objects as a heap again rather than
a size pool
Notes:
Merged: https://github.com/ruby/ruby/pull/11771
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/11635
|
|
This adds keys heap_empty_pages and heap_allocatable_slots to GC.stat.
|
|
|
|
It's possible for a GC to run between the calls of GC.latest_gc_info,
which would cause the test to fail. We can disable GC so that GC only
triggers manually.
|
|
When assertions are enabled, the following code triggers an assertion
error:
GC.disable
GC.start(immediate_mark: false, immediate_sweep: false)
10_000_000.times { Object.new }
This is because the GC.start ignores that the GC is disabled and will
start incremental marking and lazy sweeping. But the assertions in
gc_marks_continue and gc_sweep_continue assert that GC is not disabled.
This commit changes it for the assertion to pass if the GC was triggered
from a method.
Notes:
Merged: https://github.com/ruby/ruby/pull/11391
|
|
`RBOOL` is a macro to convert C boolean to Ruby boolean.
Notes:
Merged: https://github.com/ruby/ruby/pull/11351
|
|
|
|
This feature provides a new method `GC.config` that configures internal
GC configuration variables provided by an individual GC implementation.
Implemented in this PR is the option `full_mark`: a boolean value that
will determine whether the Ruby GC is allowed to run a major collection
while the process is running.
It has the following semantics
This feature configures Ruby's GC to only run minor GC's. It's designed
to give users relying on Out of Band GC complete control over when a
major GC is run. Configuring `full_mark: false` does two main things:
* Never runs a Major GC. When the heap runs out of space during a minor
and when a major would traditionally be run, instead we allocate more
heap pages, and mark objspace as needing a major GC.
* Don't increment object ages. We don't promote objects during GC, this
will cause every object to be scanned on every minor. This is an
intentional trade-off between minor GC's doing more work every time,
and potentially promoting objects that will then never be GC'd.
The intention behind not aging objects is that users of this feature
should use a preforking web server, or some other method of pre-warming
the oldgen (like Nakayoshi fork)before disabling Majors. That way most
objects that are going to be old will have already been promoted.
This will interleave major and minor GC collections in exactly the same
what that the Ruby GC runs in versions previously to this. This is the
default behaviour.
* This new method has the following extra semantics:
- `GC.config` with no arguments returns a hash of the keys of the
currently configured GC
- `GC.config` with a key pair (eg. `GC.config(full_mark: true)` sets
the matching config key to the corresponding value and returns the
entire known config hash, including the new values. If the key does
not exist, `nil` is returned
* When a minor GC is run, Ruby sets an internal status flag to determine
whether the next GC will be a major or a minor. When `full_mark:
false` this flag is ignored and every GC will be a minor.
This status flag can be accessed at
`GC.latest_gc_info(:needs_major_by)`. Any value other than `nil` means
that the next collection would have been a major.
Thus it's possible to use this feature to check at a predetermined
time, whether a major GC is necessary and run one if it is. eg. After
a request has finished processing.
```ruby
if GC.latest_gc_info(:needs_major_by)
GC.start(full_mark: true)
end
```
[Feature #20443]
|
|
We only collect GC.stat_heap(nil, stat_heap_all)
once, outside of the loop, but assert_equal could
allocate objects which can cause a GC to run and
cause stat_heap_all to be out-of-sync.
|
|
This commit splits gc.c into two files:
- gc.c now only contains code not specific to Ruby GC. This includes
code to mark objects (which the GC implementation may choose not to
use) and wrappers for internal APIs that the implementation may need
to use (e.g. locking the VM).
- gc_impl.c now contains the implementation of Ruby's GC. This includes
marking, sweeping, compaction, and statistics. Most importantly,
gc_impl.c only uses public APIs in Ruby and a limited set of functions
exposed in gc.c. This allows us to build gc_impl.c independently of
Ruby and plug Ruby's GC into itself.
|
|
|
|
This reverts commit 3680981c7b71df8c3a426164787ccefe5296bb25.
|
|
(maybe) from 9cf754b the test fails on some environments:
https://rubyci.s3.amazonaws.com/icc-x64/ruby-master/log/20240325T200004Z.fail.html.gz
```
1) Failure:
TestGc#test_gc_stress_at_startup [/home/chkbuild/chkbuild/tmp/build/20240325T200004Z/ruby/test/ruby/test_gc.rb:771]:
[Bug #15784]
pid 1087168 killed by SIGSEGV (signal 11) (core dumped).
1. [3/3] Assertion for "success?"
| Expected #<Process::Status: pid 1087168 SIGSEGV (signal 11) (core dumped)> to be success?.
```
https://rubyci.s3.amazonaws.com/freebsd14/ruby-master/log/20240325T203002Z.fail.html.gz
https://rubyci.s3.amazonaws.com/osx1200arm-no-yjit/ruby-master/log/20240325T195003Z.fail.html.gz
https://rubyci.s3.amazonaws.com/s390x/ruby-master/log/20240325T210003Z.fail.html.gz
...
so just skipt it until it works.
|
|
|
|
When error on finalizer, the exception will be ignored.
To restart the code, we need to restore the stack pointer.
fix [Bug #20042]
|
|
The test sometimes fails with:
1) Failure:
TestGc#test_stat_heap [/tmp/ruby/src/trunk-repeat50/test/ruby/test_gc.rb:169]:
Expected 33434403 to be <= 33434354.
|
|
It randomly fails like this:
https://github.com/ruby/ruby/actions/runs/7191443542/job/19586164973
|
|
|
|
|
