Fix use-after-free in constant cache
[Bug #20921]
When we create a cache entry for a constant, the following sequence of
events could happen:
- vm_track_constant_cache is called to insert a constant cache.
- In vm_track_constant_cache, we first look up the ST table for the ID
of the constant. Assume the ST table exists because another iseq also
holds a cache entry for this ID.
- We then insert into this ST table with the iseq_inline_constant_cache.
- However, while inserting into this ST table, it allocates memory, which
could trigger a GC. Assume that it does trigger a GC.
- The GC frees the one and only other iseq that holds a cache entry for
this ID.
- In remove_from_constant_cache, it will appear that the ST table is now
empty because there are no more iseq with cache entries for this ID, so
we free the ST table.
- We complete GC and continue our st_insert. However, this ST table has
been freed so we now have a use-after-free.
This issue is very hard to reproduce, because it requires that the GC runs
at a very specific time. However, we can make it show up by applying this
patch which runs GC right before the st_insert to mimic the st_insert
triggering a GC:
diff --git a/vm_insnhelper.c b/vm_insnhelper.c
index 3cb23f06f0..a93998136a 100644
--- a/vm_insnhelper.c
+++ b/vm_insnhelper.c
@@ -6338,6 +6338,10 @@ vm_track_constant_cache(ID id, void *ic)
rb_id_table_insert(const_cache, id, (VALUE)ics);
}
+ if (id == rb_intern("MyConstant")) rb_gc();
+
st_insert(ics, (st_data_t) ic, (st_data_t) Qtrue);
}
And if we run this script:
Object.const_set("MyConstant", "Hello!")
my_proc = eval("-> { MyConstant }")
my_proc.call
my_proc = eval("-> { MyConstant }")
my_proc.call
We can see that ASAN outputs a use-after-free error:
==36540==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000049528 at pc 0x000102f3ceac bp 0x00016d607a70 sp 0x00016d607a68
READ of size 8 at 0x606000049528 thread T0
#0 0x102f3cea8 in do_hash st.c:321
#1 0x102f3ddd0 in rb_st_insert st.c:1132
#2 0x103140700 in vm_track_constant_cache vm_insnhelper.c:6345
#3 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
#4 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
#5 0x1030bc1e0 in vm_exec_core insns.def:263
#6 0x1030b55fc in rb_vm_exec vm.c:2585
#7 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
#8 0x102a82588 in rb_ec_exec_node eval.c:281
#9 0x102a81fe0 in ruby_run_node eval.c:319
#10 0x1027f3db4 in rb_main main.c:43
#11 0x1027f3bd4 in main main.c:68
#12 0x183900270 (<unknown module>)
0x606000049528 is located 8 bytes inside of 56-byte region [0x606000049520,0x606000049558)
freed by thread T0 here:
#0 0x104174d40 in free+0x98 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54d40)
#1 0x102ada89c in rb_gc_impl_free default.c:8183
#2 0x102ada7dc in ruby_sized_xfree gc.c:4507
#3 0x102ac4d34 in ruby_xfree gc.c:4518
#4 0x102f3cb34 in rb_st_free_table st.c:663
#5 0x102bd52d8 in remove_from_constant_cache iseq.c:119
#6 0x102bbe2cc in iseq_clear_ic_references iseq.c:153
#7 0x102bbd2a0 in rb_iseq_free iseq.c:166
#8 0x102b32ed0 in rb_imemo_free imemo.c:564
#9 0x102ac4b44 in rb_gc_obj_free gc.c:1407
#10 0x102af4290 in gc_sweep_plane default.c:3546
#11 0x102af3bdc in gc_sweep_page default.c:3634
#12 0x102aeb140 in gc_sweep_step default.c:3906
#13 0x102aeadf0 in gc_sweep_rest default.c:3978
#14 0x102ae4714 in gc_sweep default.c:4155
#15 0x102af8474 in gc_start default.c:6484
#16 0x102afbe30 in garbage_collect default.c:6363
#17 0x102ad37f0 in rb_gc_impl_start default.c:6816
#18 0x102ad3634 in rb_gc gc.c:3624
#19 0x1031406ec in vm_track_constant_cache vm_insnhelper.c:6342
#20 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
#21 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
#22 0x1030bc1e0 in vm_exec_core insns.def:263
#23 0x1030b55fc in rb_vm_exec vm.c:2585
#24 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
#25 0x102a82588 in rb_ec_exec_node eval.c:281
#26 0x102a81fe0 in ruby_run_node eval.c:319
#27 0x1027f3db4 in rb_main main.c:43
#28 0x1027f3bd4 in main main.c:68
#29 0x183900270 (<unknown module>)
previously allocated by thread T0 here:
#0 0x104174c04 in malloc+0x94 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54c04)
#1 0x102ada0ec in rb_gc_impl_malloc default.c:8198
#2 0x102acee44 in ruby_xmalloc gc.c:4438
#3 0x102f3c85c in rb_st_init_table_with_size st.c:571
#4 0x102f3c900 in rb_st_init_table st.c:600
#5 0x102f3c920 in rb_st_init_numtable st.c:608
#6 0x103140698 in vm_track_constant_cache vm_insnhelper.c:6337
#7 0x1030b91d8 in vm_ic_track_const_chain vm_insnhelper.c:6356
#8 0x1030b8cf8 in rb_vm_opt_getconstant_path vm_insnhelper.c:6424
#9 0x1030bc1e0 in vm_exec_core insns.def:263
#10 0x1030b55fc in rb_vm_exec vm.c:2585
#11 0x1030fe0ac in rb_iseq_eval_main vm.c:2851
#12 0x102a82588 in rb_ec_exec_node eval.c:281
#13 0x102a81fe0 in ruby_run_node eval.c:319
#14 0x1027f3db4 in rb_main main.c:43
#15 0x1027f3bd4 in main main.c:68
#16 0x183900270 (<unknown module>)
This commit fixes this bug by adding a inserting_constant_cache_id field
to the VM, which stores the ID that is currently being inserted and, in
remove_from_constant_cache, we don't free the ST table for ID equal to
this one.
Co-Authored-By: Alan Wu <alanwu@ruby-lang.org>
[Bug #19592] Fix ext/Setup support After [1], using ext/Setup to link some, but not all extensions failed during linking. I did not know about this option, and had assumed that only `--with-static-linked-ext` builds can include statically linked extensions. Include the support code for statically linked extensions in all configurations like before [1]. Initialize the table lazily to minimize footprint on builds that have no statically linked extensions. [1]: 790cf4b6d0475614afb127b416e87cfa39044d67 "Fix autoload status of statically linked extensions" --- load.c | 28 +++++++++++++++++----------- vm.c | 3 --- vm_core.h | 6 ------ 3 files changed, 17 insertions(+), 20 deletions(-)
Speed up rebuilding the loaded feature index Rebuilding the loaded feature index slowed down with the bug fix for #17885 in 79a4484a072e9769b603e7b4fbdb15b1d7eccb15. The slowdown was extreme if realpath emulation was used, but even when not emulated, it could be about 10x slower. This adds loaded_features_realpath_map to rb_vm_struct. This is a hidden hash mapping loaded feature paths to realpaths. When rebuilding the loaded feature index, look at this hash to get cached realpath values, and skip calling rb_check_realpath if a cached value is found. Fixes [Bug #19246] --- load.c | 27 +++++++++++++++++++++++---- vm.c | 2 ++ vm_core.h | 1 + 3 files changed, 26 insertions(+), 4 deletions(-)
This reverts commit 788b03d5ba82fd8b35ce1fe2618ce6bacc648333.
* Re-apply "Ruby 3.2 - Speed up rebuilding the loaded feature index and realpath cache (#8023)" * [CI] mingw.yml - remove IBM437 encoding for test-all, use cmd shell for test & test-all * Skip failing test on mingw with readline.so Co-authored-by: nagachika <nagachika@ruby-lang.org> --------- Co-authored-by: nagachika <nagachika@ruby-lang.org>
This reverts commit 8346d1630b8193eef1ec9dd537b16de74afdc2e8. It seems break CI on MinGW. https://github.com/ruby/ruby/actions/runs/5698775895/job/15447455988
* Speed up rebuilding the loaded feature index Rebuilding the loaded feature index slowed down with the bug fix for #17885 in 79a4484a072e9769b603e7b4fbdb15b1d7eccb15. The slowdown was extreme if realpath emulation was used, but even when not emulated, it could be about 10x slower. This adds loaded_features_realpath_map to rb_vm_struct. This is a hidden hash mapping loaded feature paths to realpaths. When rebuilding the loaded feature index, look at this hash to get cached realpath values, and skip calling rb_check_realpath if a cached value is found. Fixes [Bug #19246] * Add a realpath cache to reduce number of syscalls. Number of lstat and stat syscalls for each 'require'd file is doubled, because rb_realpath_internal is called from two places with the same arguments in require_internal; once for checking the realpaths cache, and once in load_iseq_eval when iseq is not found. Introduce rb_realpath_internal_cached function to reuse the realpath_map cache which memoizes rb_realpath_internal function, leading to less syscalls and increased startup performance depending on the cost of the syscalls in a particular environment. --------- Co-authored-by: Jeremy Evans <code@jeremyevans.net>
Avoid checking interrupt when loading iseq MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The interrupt check will unintentionally release the VM lock when loading an iseq. And this will cause issues with the `debug` gem's [`ObjectSpace.each_iseq` method](https://github.com/ruby/debug/blob/0fcfc28acae33ec1c08068fb7c33703cfa681fa7/ext/debug/iseq_collector.c#L61-L67), which wraps iseqs with a wrapper and exposes their internal states when they're actually not ready to be used. And when that happens, errors like this would occur and kill the `debug` gem's thread: ``` DEBUGGER: ReaderThreadError: uninitialized InstructionSequence ┃ DEBUGGER: Disconnected. ┃ ["/opt/rubies/ruby-3.2.0/lib/ruby/gems/3.2.0/gems/debug-1.7.1/lib/debug/breakpoint.rb:247:in `absolute_path'", ┃ "/opt/rubies/ruby-3.2.0/lib/ruby/gems/3.2.0/gems/debug-1.7.1/lib/debug/breakpoint.rb:247:in `block in iterate_iseq'", ┃ "/opt/rubies/ruby-3.2.0/lib/ruby/gems/3.2.0/gems/debug-1.7.1/lib/debug/breakpoint.rb:246:in `each_iseq'", ... ``` A way to reproduce the issue is to satisfy these conditions at the same time: 1. `debug` gem calling `ObjectSpace.each_iseq` (e.g. [activating a `LineBreakpoint`](https://github.com/ruby/debug/blob/0fcfc28acae33ec1c08068fb7c33703cfa681fa7/lib/debug/breakpoint.rb#L246)). 2. A large amount of iseq being loaded from another thread (possibly through the `bootsnap` gem). 3. 1 and 2 iterating through the same iseq(s) at the same time. Because this issue requires external dependencies and a rather complicated timing setup to reproduce, I wasn't able to write a test case for it. But here's some pseudo code to help reproduce it: ```rb require "debug/session" Thread.new do 100.times do ObjectSpace.each_iseq do |iseq| iseq.absolute_path end end end sleep 0.1 load_a_bunch_of_iseq possibly_through_bootsnap ``` [Bug #19348] Co-authored-by: Peter Zhu <peter@peterzhu.ca> --- compile.c | 2 +- vm_core.h | 1 + vm_insnhelper.c | 11 +++++++++++ 3 files changed, 13 insertions(+), 1 deletion(-)