Refactor and improve performance of RDoc::Markup::Parser

This change introduces a wrapper of StringScanner that is aware of the
current position (column and lineno).
It has two advantages: faster and more modular.

The old code frequently runs `@input.byteslice(0, byte_offset).length`
to get the current position, but it was painfully slow.  This change
keeps track of the position at each scan, which reduces about half of
time of "Generating RI format into ..." in Ruby's `make rdoc`
(5.5 sec -> 3.0 sec).

And the old code used four instance variables (`@input`, `@line`,
`@line_pos`, and `@s`) to track the position.  This change factors them
out into MyStringScanner, so now only one variable (`@s`) is needed.

1 files changed, 0 insertions, 18 deletions

diff --git a/test/rdoc/test_rdoc_markup_parser.rb b/test/rdoc/test_rdoc_markup_parser.rb
index 344d67df39..b9705e19d1 100644
--- a/test/rdoc/test_rdoc_markup_parser.rb
+++ b/test/rdoc/test_rdoc_markup_parser.rb
@@ -22,15 +22,6 @@ class TestRDocMarkupParser < RDoc::TestCase
     assert_equal @RM::Heading.new(3, 'heading three'), parser.build_heading(3)
   end
 
-  def test_char_pos
-    parser = @RMP.new
-    s = parser.setup_scanner 'cät'
-
-    s.scan(/\S+/)
-
-    assert_equal 3, parser.char_pos(s.pos)
-  end
-
   def test_get
     parser = util_parser
 
@@ -1647,15 +1638,6 @@ Example heading:
     assert_equal expected, @RMP.tokenize(str)
   end
 
-  def test_token_pos
-    parser = @RMP.new
-    s = parser.setup_scanner 'cät'
-
-    s.scan(/\S+/)
-
-    assert_equal [3, 0], parser.token_pos(s.pos)
-  end
-
   # HACK move to Verbatim test case
   def test_verbatim_normalize
     v = @RM::Verbatim.new "foo\n", "\n", "\n", "bar\n"