Separate RDoc::TokenStream#add_tokens and #add_token

The old version of `add_tokens` accepts an array of tokens, and
multiple arguments of tokens by using `Array#flatten`.
And `add_token` was an alias to `add_tokens`.

I think it is unnecessarily flexible; in fact, all callsites of
`add_tokens` (except test) passes only an array of tokens.
And the code created a lot of temporal arrays.

This change makes `add_tokens` accept only one array of tokens,
and does `add_token` accept one token.  It is a bit faster (about
1 second in Ruby's `make rdoc`), and it ls also cleaner in my point of
view.

1 files changed, 8 insertions, 3 deletions

diff --git a/lib/rdoc/token_stream.rb b/lib/rdoc/token_stream.rb
index dbe6c5ae85..f428e2400c 100644
--- a/lib/rdoc/token_stream.rb
+++ b/lib/rdoc/token_stream.rb
@@ -74,11 +74,16 @@ module RDoc::TokenStream
   ##
   # Adds +tokens+ to the collected tokens
 
-  def add_tokens(*tokens)
-    tokens.flatten.each { |token| @token_stream << token }
+  def add_tokens(tokens)
+    @token_stream.concat(tokens)
   end
 
-  alias add_token add_tokens
+  ##
+  # Adds one +token+ to the collected tokens
+
+  def add_token(token)
+    @token_stream.push(token)
+  end
 
   ##
   # Starts collecting tokens