Use ident hash for top-level recursion check

We track recursion in order to not infinite loop in ==, inspect, and
similar methods by keeping a thread-local 1 or 2 level hash. This allows
us to track when we have seen the same object (ex. using inspect) or
same pair of objects (ex. using ==) in this stack before and to treat
that differently.

Previously both levels of this Hash used the object's memory_id as a key
(using object_id would be slow and wasteful). Unfortunately, prettyprint
(pp.rb) uses this thread local variable to "pretend" to be inspect and
inherit its same recursion behaviour.

This commit changes the top-level hash to be an identity hash and to use
objects as keys instead of their object_ids.

I'd like to have also converted the 2nd level hash to an ident hash, but
it would have prevented an optimization which avoids allocating a 2nd
level hash for only a single element, which we want to keep because it's
by far the most common case.

So the new format of this hash is:

{ object => true } (not paired)
{ lhs_object => rhs_object_memory_id } (paired, single object)
{ lhs_object => { rhs_object_memory_id => true, ... } } (paired, many objects)

We must also update pp.rb to match this (using identity hashes).

1 files changed, 6 insertions, 8 deletions

diff --git a/lib/pp.rb b/lib/pp.rb
index 3e17c7ffaf..de4b79c4de 100644
--- a/lib/pp.rb
+++ b/lib/pp.rb
@@ -106,17 +106,17 @@ class PP < PrettyPrint
     # and preserves the previous set of objects being printed.
     def guard_inspect_key
       if Thread.current[:__recursive_key__] == nil
-        Thread.current[:__recursive_key__] = {}.taint
+        Thread.current[:__recursive_key__] = {}.compare_by_identity.taint
       end
 
       if Thread.current[:__recursive_key__][:inspect] == nil
-        Thread.current[:__recursive_key__][:inspect] = {}.taint
+        Thread.current[:__recursive_key__][:inspect] = {}.compare_by_identity.taint
       end
 
       save = Thread.current[:__recursive_key__][:inspect]
 
       begin
-        Thread.current[:__recursive_key__][:inspect] = {}.taint
+        Thread.current[:__recursive_key__][:inspect] = {}.compare_by_identity.taint
         yield
       ensure
         Thread.current[:__recursive_key__][:inspect] = save
@@ -149,18 +149,16 @@ class PP < PrettyPrint
     # Object#pretty_print_cycle is used when +obj+ is already
     # printed, a.k.a the object reference chain has a cycle.
     def pp(obj)
-      id = obj.object_id
-
-      if check_inspect_key(id)
+      if check_inspect_key(obj)
         group {obj.pretty_print_cycle self}
         return
       end
 
       begin
-        push_inspect_key(id)
+        push_inspect_key(obj)
         group {obj.pretty_print self}
       ensure
-        pop_inspect_key(id) unless PP.sharing_detection
+        pop_inspect_key(obj) unless PP.sharing_detection
       end
     end