Allow ivars movement in too_complex RCLASSes to fix crash

Previously, because gc_update_object_references() did not update the
VALUEs in the too_complex ivar st_table for T_CLASS and T_MODULE
objects, GC compaction could finish with corrupted objects.

 - start with `klass`, not too_complex
 - GC incremental step marks `klass` and its ivars
 - ruby code makes `klass` too_complex
 - GC compaction runs and move `klass` ivars, but because `klass` is
   too_complex, its ivars are not updated by gc_update_object_references(),
   leaving T_NONE or T_MOVED objects in the ivar table.

Co-authored-by: Peter Zhu <peter@peterzhu.ca>

1 files changed, 5 insertions, 2 deletions

diff --git a/gc.c b/gc.c
index 5321470d35..3f992604b2 100644
--- a/gc.c
+++ b/gc.c
@@ -7269,7 +7269,7 @@ gc_mark_children(rb_objspace_t *objspace, VALUE obj)
         mark_cvc_tbl(objspace, obj);
         cc_table_mark(objspace, obj);
         if (rb_shape_obj_too_complex(obj)) {
-            mark_tbl(objspace, (st_table *)RCLASS_IVPTR(obj));
+            mark_tbl_no_pin(objspace, (st_table *)RCLASS_IVPTR(obj));
         }
         else {
             for (attr_index_t i = 0; i < RCLASS_IV_COUNT(obj); i++) {
@@ -10610,7 +10610,10 @@ gc_update_object_references(rb_objspace_t *objspace, VALUE obj)
         update_cvc_tbl(objspace, obj);
         update_superclasses(objspace, obj);
 
-        if (!rb_shape_obj_too_complex(obj)) {
+        if (rb_shape_obj_too_complex(obj)) {
+            gc_ref_update_table_values_only(objspace, RCLASS_IV_HASH(obj));
+        }
+        else {
             for (attr_index_t i = 0; i < RCLASS_IV_COUNT(obj); i++) {
                 UPDATE_IF_MOVED(objspace, RCLASS_IVPTR(obj)[i]);
             }