https://github.com/ruby/rdoc/commit/04f75d8516

Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>

The change in #1055 might be a breaking change.
So, just simply wrap `token_stream` with `Array`

https://github.com/ruby/rdoc/commit/d8c19d7fa1

Co-authored-by: Jonathan Hefner <jonathan@hefner.pro>

This reverts commit https://github.com/ruby/rdoc/commit/4a1c74bc0a09.

Since RDoc still supports ruby 2.6 which has not deprecated `$,`, the
argument of `Array#join` is not unnecessary yet.

https://github.com/ruby/rdoc/commit/72897d32ed

https://github.com/ruby/rdoc/commit/4a1c74bc0a

Calling `tokens_to_s` gets an error if `token_stream` is nil:

```
undefined method `compact' for nil:NilClass (NoMethodError)
```

So, fall back to an empty array if `@token_stream` is nil.

https://github.com/ruby/rdoc/commit/452e4a2600

https://github.com/ruby/rdoc/commit/4b68c0728a

The old version of `add_tokens` accepts an array of tokens, and
multiple arguments of tokens by using `Array#flatten`.
And `add_token` was an alias to `add_tokens`.

I think it is unnecessarily flexible; in fact, all callsites of
`add_tokens` (except test) passes only an array of tokens.
And the code created a lot of temporal arrays.

This change makes `add_tokens` accept only one array of tokens,
and does `add_token` accept one token.  It is a bit faster (about
1 second in Ruby's `make rdoc`), and it ls also cleaner in my point of
view.

  It fixed the several bugs that was found after RDoc 6 releasing.

From: SHIBATA Hiroshi <hsbt@ruby-lang.org>

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62924 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

  It version applied `frozen_string_literal: true`

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60920 b2dd03c8-39d4-4d8f-98ff-823fe69b080e