Separate RDoc::TokenStream#add_tokens and #add_token

The old version of `add_tokens` accepts an array of tokens, and
multiple arguments of tokens by using `Array#flatten`.
And `add_token` was an alias to `add_tokens`.

I think it is unnecessarily flexible; in fact, all callsites of
`add_tokens` (except test) passes only an array of tokens.
And the code created a lot of temporal arrays.

This change makes `add_tokens` accept only one array of tokens,
and does `add_token` accept one token.  It is a bit faster (about
1 second in Ruby's `make rdoc`), and it ls also cleaner in my point of
view.

2 files changed, 10 insertions, 5 deletions

diff --git a/lib/rdoc/token_stream.rb b/lib/rdoc/token_stream.rb
index dbe6c5ae85..f428e2400c 100644
--- a/lib/rdoc/token_stream.rb
+++ b/lib/rdoc/token_stream.rb
@@ -74,11 +74,16 @@ module RDoc::TokenStream
   ##
   # Adds +tokens+ to the collected tokens
 
-  def add_tokens(*tokens)
-    tokens.flatten.each { |token| @token_stream << token }
+  def add_tokens(tokens)
+    @token_stream.concat(tokens)
   end
 
-  alias add_token add_tokens
+  ##
+  # Adds one +token+ to the collected tokens
+
+  def add_token(token)
+    @token_stream.push(token)
+  end
 
   ##
   # Starts collecting tokens
diff --git a/test/rdoc/test_rdoc_any_method.rb b/test/rdoc/test_rdoc_any_method.rb
index 6dd46b0b46..615789dfb3 100644
--- a/test/rdoc/test_rdoc_any_method.rb
+++ b/test/rdoc/test_rdoc_any_method.rb
@@ -78,7 +78,7 @@ method(a, b) { |c, d| ... }
     ]
 
     @c2_a.collect_tokens
-    @c2_a.add_tokens(*tokens)
+    @c2_a.add_tokens(tokens)
 
     expected = '<span class="ruby-constant">CONSTANT</span>'
 
@@ -96,7 +96,7 @@ method(a, b) { |c, d| ... }
     ]
 
     @c2_a.collect_tokens
-    @c2_a.add_tokens(*tokens)
+    @c2_a.add_tokens(tokens)
 
     assert_equal <<-EXPECTED.chomp, @c2_a.markup_code
 <span class="ruby-comment"># File xref_data.rb, line 1</span>