Initialize VM stack if VM_CHECK_MODE

Lately there has been a few flaky YJIT CI failures where a new Ruby
thread is finding the canary on the VM stack. For example:

https://github.com/ruby/ruby/actions/runs/8287357784/job/22679508482#step:14:109

After checking a local rr recording, it's clear that the canary was
written there when YJIT was using a temporary malloc region, and then
later handed to the new Ruby thread. Previously, the VM stack was
uninitialized, so it can have stale values in it, like the canary.

Though unlikely, this can happen without YJIT too. Initialize the stack
if we're spawning canaries.

1 files changed, 4 insertions, 0 deletions

diff --git a/vm.c b/vm.c
index 33cd0fa0c8..0f7d986d4e 100644
--- a/vm.c
+++ b/vm.c
@@ -3560,6 +3560,10 @@ rb_ec_initialize_vm_stack(rb_execution_context_t *ec, VALUE *stack, size_t size)
 {
     rb_ec_set_vm_stack(ec, stack, size);
 
+#if VM_CHECK_MODE > 0
+    MEMZERO(stack, VALUE, size); // malloc memory could have the VM canary in it
+#endif
+
     ec->cfp = (void *)(ec->vm_stack + ec->vm_stack_size);
 
     vm_push_frame(ec,