Backport lib/reline, and lib/irb for 3.0.1 2nd (#4157)

* [ruby/irb] Stub a screen size for tests

https://github.com/ruby/irb/commit/6663057083

* [ruby/irb] Support GitHub Actions

https://github.com/ruby/irb/commit/8e9e6c4037

* [ruby/irb] Stub a screen size for test_context

http://ci.rvm.jp/logfiles/brlog.trunk-random1.20210119-074232

https://github.com/ruby/irb/commit/ea87592d4a

* [ruby/irb] Use a real screen size for pp by default

https://github.com/ruby/irb/commit/9b9300dec2

* [ruby/irb] Rescue Errno::EINVAL on IRB pp

http://rubyci.s3.amazonaws.com/solaris11-gcc/ruby-master/log/20210119T070008Z.log.html.gz
is caused by:

/export/home/chkbuild/chkbuild-gcc/tmp/build/20210119T150010Z/ruby/lib/reline/ansi.rb:157:in `winsize': Invalid argument - <STDIN> (Errno::EINVAL)
        from /export/home/chkbuild/chkbuild-gcc/tmp/build/20210119T150010Z/ruby/lib/reline/ansi.rb:157:in `get_screen_size'
        from /export/home/chkbuild/chkbuild-gcc/tmp/build/20210119T150010Z/ruby/lib/reline.rb:168:in `get_screen_size'
        from /export/home/chkbuild/chkbuild-gcc/tmp/build/20210119T150010Z/ruby/lib/forwardable.rb:238:in `get_screen_size'
        from /export/home/chkbuild/chkbuild-gcc/tmp/build/20210119T150010Z/ruby/lib/irb/color_printer.rb:7:in `pp'
        from -e:1:in `<main>'

https://github.com/ruby/irb/commit/1719514598

* [ruby/irb] Split test files for IRB::Color and IRB::ColorPrinter

https://github.com/ruby/irb/commit/d95e8daab3

* [ruby/irb] Undefine unused constants

https://github.com/ruby/irb/commit/eea9c16804

* [ruby/irb] Remove pp-specific stub from TestColor

because it was for TestColorPrinter

https://github.com/ruby/irb/commit/7569206fd4

* [ruby/irb] Delete a doodle-level memo comment...

https://github.com/ruby/irb/commit/fc3e1d9e0c

* [ruby/irb] Indent correctly with keyword "for" and "in"

https://github.com/ruby/irb/commit/47c83ea724

* [ruby/irb] Indent correctly with method calling with receiver

https://github.com/ruby/irb/commit/e7c68e74a0

* [ruby/irb] add `IRB::FileInputMethod.open` to ensure closing associated File

* tweak some methods not to raise exception after `#close`
* use it in `IRB::IrbLoader#{source_file,load_file}

https://github.com/ruby/irb/commit/ec2947acbd

* [ruby/irb] use `RubyLex::TerminateLineInput` appropriately [Bug #17564]

* using the appropriciate exception instead of `break` so that the session
  can be continue after the `irb_source` and `irb_load` commands
* suppress extra new line due to one more `#prompt` call

https://github.com/ruby/irb/commit/bdefaa7cfd

* [ruby/irb] specify the `VERBOSE` to `false` and fix tests to fit

https://github.com/ruby/irb/commit/502c590925

* In test, need to pass a context to IRB::WorkSpace.new explicitly

* Fix absolute path predicate on Windows

A path starts with '/' is not an absolute path on Windows, because
of drive letter or UNC.

* [ruby/irb] follow up the actual line number

https://github.com/ruby/irb/commit/7aed8fe3b1

* [ruby/irb] Add info.rb to gemspec

https://github.com/ruby/irb/commit/adbba19adf

* [ruby/irb] Allow "measure" command to take block

https://github.com/ruby/irb/commit/20f1ca23e9

* [ruby/irb] Enable to reassign a new block with "measure" command

https://github.com/ruby/irb/commit/b444573aa2

* [ruby/reline] Cache pasting state in processing a key

Because it's too slow.

The rendering time in IRB has been reduced as follows:

  start = Time.now

  def each_top_level_statement
    initialize_input
    catch(:TERM_INPUT) do
      loop do
        begin
          prompt
          unless l = lex
            throw :TERM_INPUT if @line == ''
          else
            @line_no += l.count("\n")
            next if l == "\n"
            @line.concat l
            if @code_block_open or @ltype or @continue or @indent > 0
              next
            end
          end
          if @line != "\n"
            @line.force_encoding(@io.encoding)
            yield @line, @exp_line_no
          end
          break if @io.eof?
          @line = ''
          @exp_line_no = @line_no

          @indent = 0
        rescue TerminateLineInput
          initialize_input
          prompt
        end
      end
    end
  end

  puts "Duration: #{Time.now - start} seconds"

0.22sec -> 0.14sec

https://github.com/ruby/reline/commit/b8b3dd52c0

* [ruby/reline] Initialize uninitialized variables in tests

https://github.com/ruby/reline/commit/25af4bb64b

* [ruby/reline] Remove an unused variable

https://github.com/ruby/reline/commit/123ea51166

* [ruby/reline] Scroll down when ^C is pressed

https://github.com/ruby/reline/commit/6877a7e3f5

* [ruby/reline] Show all lines higher than the screen when finished

On Unix-like OSes, logs prior to the screen are not editable. When the code
is higher than the screen, the code is only shown on the screen until input
is finished, but when it is finished, all lines are outputted.

https://github.com/ruby/reline/commit/8cd9132a39

* [ruby/reline] Handle past logs correctly when the code is higher than the screen

https://github.com/ruby/reline/commit/f197139b4a

* [ruby/reline] Update cursor info by inserting newline even if not in pasting

https://github.com/ruby/reline/commit/92d314f514

* [ruby/reline] Move cursor just after the last line when finished

https://github.com/ruby/reline/commit/ba06e4c480

* [ruby/reline] The vi_histedit supports multiline

This closes ruby/reline#253.

https://github.com/ruby/reline/commit/f131f86d71

* [ruby/reline] Autowrap correctly when inserting chars in the middle of a line

https://github.com/ruby/reline/commit/ebaf37255f

* [ruby/reline] Terminate correctly in the middle of lines higher than the screen

https://github.com/ruby/reline/commit/e1d9240ada

* [ruby/irb] Version 1.3.3

https://github.com/ruby/irb/commit/4c87035b7c

* [ruby/reline] Version 0.2.3

https://github.com/ruby/reline/commit/b26c7d60c8

Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com>
Co-authored-by: Nobuhiro IMAI <nov@yo.rim.or.jp>
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: ima1zumi <mariimaizumi5@gmail.com>

6 files changed, 319 insertions, 25 deletions

diff --git a/test/irb/test_cmd.rb b/test/irb/test_cmd.rb
index b2246dfff5..7219473e4c 100644
--- a/test/irb/test_cmd.rb
+++ b/test/irb/test_cmd.rb
@@ -275,5 +275,98 @@ module TestIRB
       assert_empty err
       assert_match(/\A=> 3\nCUSTOM is added\.\n=> nil\ncustom processing time: .+\n=> 3\n=> nil\n=> 3\n/, out)
     end
+
+    def test_measure_with_proc
+      IRB.init_config(nil)
+      IRB.conf[:PROMPT] = {
+        DEFAULT: {
+          PROMPT_I: '> ',
+          PROMPT_S: '> ',
+          PROMPT_C: '> ',
+          PROMPT_N: '> '
+        }
+      }
+      IRB.conf[:VERBOSE] = false
+      IRB.conf[:PROMPT_MODE] = :DEFAULT
+      IRB.conf[:MEASURE] = false
+      input = TestInputMethod.new([
+        "3\n",
+        "measure { |context, code, line_no, &block|\n",
+        "  result = block.()\n",
+        "  puts 'aaa' if IRB.conf[:MEASURE]\n",
+        "  result\n",
+        "}\n",
+        "3\n",
+        "measure { |context, code, line_no, &block|\n",
+        "  result = block.()\n",
+        "  puts 'bbb' if IRB.conf[:MEASURE]\n",
+        "  result\n",
+        "}\n",
+        "3\n",
+        "measure :off\n",
+        "3\n",
+      ])
+      c = Class.new(Object)
+      irb = IRB::Irb.new(IRB::WorkSpace.new(c.new), input)
+      irb.context.return_format = "=> %s\n"
+      out, err = capture_output do
+        irb.eval_input
+      end
+      assert_empty err
+      assert_match(/\A=> 3\nBLOCK is added\.\n=> nil\naaa\n=> 3\nBLOCK is added.\naaa\n=> nil\nbbb\n=> 3\n=> nil\n=> 3\n/, out)
+      assert_empty(c.class_variables)
+    end
+
+    def test_irb_source
+      IRB.init_config(nil)
+      File.write("#{@tmpdir}/a.rb", "a = 'hi'\n")
+      input = TestInputMethod.new([
+          "a = 'bug17564'\n",
+          "a\n",
+          "irb_source '#{@tmpdir}/a.rb'\n",
+          "a\n",
+        ])
+      IRB.conf[:VERBOSE] = false
+      IRB.conf[:PROMPT_MODE] = :SIMPLE
+      irb = IRB::Irb.new(IRB::WorkSpace.new(self), input)
+      IRB.conf[:MAIN_CONTEXT] = irb.context
+      out, err = capture_output do
+        irb.eval_input
+      end
+      assert_empty err
+      assert_pattern_list([
+          /=> "bug17564"\n/,
+          /=> "bug17564"\n/,
+          /   => "hi"\n/,
+          /   => nil\n/,
+          /=> "hi"\n/,
+        ], out)
+    end
+
+    def test_irb_load
+      IRB.init_config(nil)
+      File.write("#{@tmpdir}/a.rb", "a = 'hi'\n")
+      input = TestInputMethod.new([
+          "a = 'bug17564'\n",
+          "a\n",
+          "irb_load '#{@tmpdir}/a.rb'\n",
+          "a\n",
+        ])
+      IRB.conf[:VERBOSE] = false
+      IRB.conf[:PROMPT_MODE] = :SIMPLE
+      irb = IRB::Irb.new(IRB::WorkSpace.new(self), input)
+      IRB.conf[:MAIN_CONTEXT] = irb.context
+      out, err = capture_output do
+        irb.eval_input
+      end
+      assert_empty err
+      assert_pattern_list([
+          /=> "bug17564"\n/,
+          /=> "bug17564"\n/,
+          /   => "hi"\n/,
+          /   => nil\n/,
+          /=> "bug17564"\n/,
+        ], out)
+    end
   end
 end
diff --git a/test/irb/test_color.rb b/test/irb/test_color.rb
index d035e443a8..9976008124 100644
--- a/test/irb/test_color.rb
+++ b/test/irb/test_color.rb
@@ -1,7 +1,6 @@
 # frozen_string_literal: false
 require 'test/unit'
 require 'irb/color'
-require 'irb/color_printer'
 require 'rubygems'
 require 'stringio'
 
@@ -153,23 +152,6 @@ module TestIRB
       end
     end
 
-    IRBTestColorPrinter = Struct.new(:a)
-
-    def test_color_printer
-      unless ripper_lexer_scan_supported?
-        skip 'Ripper::Lexer#scan is supported in Ruby 2.7+'
-      end
-      {
-        1 => "#{BLUE}#{BOLD}1#{CLEAR}\n",
-        IRBTestColorPrinter.new('test') => "#{GREEN}#<struct TestIRB::TestColor::IRBTestColorPrinter#{CLEAR} a#{GREEN}=#{CLEAR}#{RED}#{BOLD}\"#{CLEAR}#{RED}test#{CLEAR}#{RED}#{BOLD}\"#{CLEAR}#{GREEN}>#{CLEAR}\n",
-        Ripper::Lexer.new('1').scan => "[#{GREEN}#<Ripper::Lexer::Elem:#{CLEAR} on_int@1:0 END token: #{RED}#{BOLD}\"#{CLEAR}#{RED}1#{CLEAR}#{RED}#{BOLD}\"#{CLEAR}#{GREEN}>#{CLEAR}]\n",
-        Class.new{define_method(:pretty_print){|q| q.text("[__FILE__, __LINE__, __ENCODING__]")}}.new => "[#{CYAN}#{BOLD}__FILE__#{CLEAR}, #{CYAN}#{BOLD}__LINE__#{CLEAR}, #{CYAN}#{BOLD}__ENCODING__#{CLEAR}]\n",
-      }.each do |object, result|
-        actual = with_term { IRB::ColorPrinter.pp(object, '') }
-        assert_equal(result, actual, "Case: IRB::ColorPrinter.pp(#{object.inspect}, '')")
-      end
-    end
-
     def test_inspect_colorable
       {
         1 => true,
@@ -202,10 +184,6 @@ module TestIRB
       Gem::Version.new(RUBY_VERSION) >= Gem::Version.new('2.7.0')
     end
 
-    def ripper_lexer_scan_supported?
-      Gem::Version.new(RUBY_VERSION) >= Gem::Version.new('2.7.0')
-    end
-
     def with_term
       stdout = $stdout
       io = StringIO.new
diff --git a/test/irb/test_color_printer.rb b/test/irb/test_color_printer.rb
new file mode 100644
index 0000000000..1b28837658
--- /dev/null
+++ b/test/irb/test_color_printer.rb
@@ -0,0 +1,67 @@
+# frozen_string_literal: false
+require 'test/unit'
+require 'irb/color_printer'
+require 'rubygems'
+require 'stringio'
+
+module TestIRB
+  class TestColorPrinter < Test::Unit::TestCase
+    CLEAR     = "\e[0m"
+    BOLD      = "\e[1m"
+    RED       = "\e[31m"
+    GREEN     = "\e[32m"
+    BLUE      = "\e[34m"
+    CYAN      = "\e[36m"
+
+    def setup
+      @get_screen_size = Reline.method(:get_screen_size)
+      Reline.instance_eval { undef :get_screen_size }
+      def Reline.get_screen_size
+        [36, 80]
+      end
+    end
+
+    def teardown
+      Reline.instance_eval { undef :get_screen_size }
+      Reline.define_singleton_method(:get_screen_size, @get_screen_size)
+    end
+
+    IRBTestColorPrinter = Struct.new(:a)
+
+    def test_color_printer
+      unless ripper_lexer_scan_supported?
+        skip 'Ripper::Lexer#scan is supported in Ruby 2.7+'
+      end
+      {
+        1 => "#{BLUE}#{BOLD}1#{CLEAR}\n",
+        IRBTestColorPrinter.new('test') => "#{GREEN}#<struct TestIRB::TestColorPrinter::IRBTestColorPrinter#{CLEAR} a#{GREEN}=#{CLEAR}#{RED}#{BOLD}\"#{CLEAR}#{RED}test#{CLEAR}#{RED}#{BOLD}\"#{CLEAR}#{GREEN}>#{CLEAR}\n",
+        Ripper::Lexer.new('1').scan => "[#{GREEN}#<Ripper::Lexer::Elem:#{CLEAR} on_int@1:0 END token: #{RED}#{BOLD}\"#{CLEAR}#{RED}1#{CLEAR}#{RED}#{BOLD}\"#{CLEAR}#{GREEN}>#{CLEAR}]\n",
+        Class.new{define_method(:pretty_print){|q| q.text("[__FILE__, __LINE__, __ENCODING__]")}}.new => "[#{CYAN}#{BOLD}__FILE__#{CLEAR}, #{CYAN}#{BOLD}__LINE__#{CLEAR}, #{CYAN}#{BOLD}__ENCODING__#{CLEAR}]\n",
+      }.each do |object, result|
+        actual = with_term { IRB::ColorPrinter.pp(object, '') }
+        assert_equal(result, actual, "Case: IRB::ColorPrinter.pp(#{object.inspect}, '')")
+      end
+    end
+
+    private
+
+    def ripper_lexer_scan_supported?
+      Gem::Version.new(RUBY_VERSION) >= Gem::Version.new('2.7.0')
+    end
+
+    def with_term
+      stdout = $stdout
+      io = StringIO.new
+      def io.tty?; true; end
+      $stdout = io
+
+      env = ENV.to_h.dup
+      ENV['TERM'] = 'xterm-256color'
+
+      yield
+    ensure
+      $stdout = stdout
+      ENV.replace(env) if env
+    end
+  end
+end
diff --git a/test/irb/test_context.rb b/test/irb/test_context.rb
index f3d0626caa..71e8ad1c0d 100644
--- a/test/irb/test_context.rb
+++ b/test/irb/test_context.rb
@@ -42,6 +42,17 @@ module TestIRB
       IRB.conf[:VERBOSE] = false
       workspace = IRB::WorkSpace.new(Object.new)
       @context = IRB::Context.new(nil, workspace, TestInputMethod.new)
+
+      @get_screen_size = Reline.method(:get_screen_size)
+      Reline.instance_eval { undef :get_screen_size }
+      def Reline.get_screen_size
+        [36, 80]
+      end
+    end
+
+    def teardown
+      Reline.instance_eval { undef :get_screen_size }
+      Reline.define_singleton_method(:get_screen_size, @get_screen_size)
     end
 
     def test_last_value
@@ -447,7 +458,7 @@ module TestIRB
         irb.eval_input
       end
       assert_empty err
-      if '2.5.0' <= RUBY_VERSION && RUBY_VERSION < '3.0.0'
+      if '2.5.0' <= RUBY_VERSION && RUBY_VERSION < '3.0.0' && STDOUT.tty?
         expected = [
           :*, /Traceback \(most recent call last\):\n/,
           :*, /\t 2: from \(irb\):1:in `<main>'\n/,
@@ -477,7 +488,7 @@ module TestIRB
         irb.eval_input
       end
       assert_empty err
-      if '2.5.0' <= RUBY_VERSION && RUBY_VERSION < '3.0.0'
+      if '2.5.0' <= RUBY_VERSION && RUBY_VERSION < '3.0.0' && STDOUT.tty?
         expected = [
           :*, /Traceback \(most recent call last\):\n/,
           :*, /\t 2: from \(irb\):1:in `<main>'\n/,
@@ -513,7 +524,7 @@ module TestIRB
         irb.eval_input
       end
       assert_empty err
-      if '2.5.0' <= RUBY_VERSION && RUBY_VERSION < '3.0.0'
+      if '2.5.0' <= RUBY_VERSION && RUBY_VERSION < '3.0.0' && STDOUT.tty?
         expected = [
           :*, /Traceback \(most recent call last\):\n/,
           :*, /\t... 5 levels...\n/,
@@ -561,5 +572,26 @@ module TestIRB
     ensure
       $VERBOSE = verbose
     end
+
+    def test_lineno
+      input = TestInputMethod.new([
+        "\n",
+        "__LINE__\n",
+        "__LINE__\n",
+        "\n",
+        "\n",
+        "__LINE__\n",
+      ])
+      irb = IRB::Irb.new(IRB::WorkSpace.new(Object.new), input)
+      out, err = capture_output do
+        irb.eval_input
+      end
+      assert_empty err
+      assert_pattern_list([
+          :*, /\b2\n/,
+          :*, /\b3\n/,
+          :*, /\b6\n/,
+        ], out)
+    end
   end
 end
diff --git a/test/irb/test_ruby_lex.rb b/test/irb/test_ruby_lex.rb
index ed4944afc6..a45ca668b9 100644
--- a/test/irb/test_ruby_lex.rb
+++ b/test/irb/test_ruby_lex.rb
@@ -333,6 +333,102 @@ module TestIRB
       end
     end
 
+    def test_corresponding_syntax_to_keyword_for
+      input_with_correct_indents = [
+        Row.new(%q(for i in [1]), nil, 2, 1),
+        Row.new(%q(  puts i), nil, 2, 1),
+        Row.new(%q(end), 0, 0, 0),
+      ]
+
+      lines = []
+      input_with_correct_indents.each do |row|
+        lines << row.content
+        assert_indenting(lines, row.current_line_spaces, false)
+        assert_indenting(lines, row.new_line_spaces, true)
+        assert_nesting_level(lines, row.nesting_level)
+      end
+    end
+
+    def test_corresponding_syntax_to_keyword_for_with_do
+      input_with_correct_indents = [
+        Row.new(%q(for i in [1] do), nil, 2, 1),
+        Row.new(%q(  puts i), nil, 2, 1),
+        Row.new(%q(end), 0, 0, 0),
+      ]
+
+      lines = []
+      input_with_correct_indents.each do |row|
+        lines << row.content
+        assert_indenting(lines, row.current_line_spaces, false)
+        assert_indenting(lines, row.new_line_spaces, true)
+        assert_nesting_level(lines, row.nesting_level)
+      end
+    end
+
+    def test_bracket_corresponding_to_times
+      input_with_correct_indents = [
+        Row.new(%q(3.times { |i|), nil, 2, 1),
+        Row.new(%q(  puts i), nil, 2, 1),
+        Row.new(%q(}), 0, 0, 0),
+      ]
+
+      lines = []
+      input_with_correct_indents.each do |row|
+        lines << row.content
+        assert_indenting(lines, row.current_line_spaces, false)
+        assert_indenting(lines, row.new_line_spaces, true)
+        assert_nesting_level(lines, row.nesting_level)
+      end
+    end
+
+    def test_do_corresponding_to_times
+      input_with_correct_indents = [
+        Row.new(%q(3.times do |i|), nil, 2, 1),
+        #Row.new(%q(  puts i), nil, 2, 1),
+        #Row.new(%q(end), 0, 0, 0),
+      ]
+
+      lines = []
+      input_with_correct_indents.each do |row|
+        lines << row.content
+        assert_indenting(lines, row.current_line_spaces, false)
+        assert_indenting(lines, row.new_line_spaces, true)
+        assert_nesting_level(lines, row.nesting_level)
+      end
+    end
+
+    def test_bracket_corresponding_to_loop
+      input_with_correct_indents = [
+        Row.new(%q(loop {), nil, 2, 1),
+        Row.new(%q(  3), nil, 2, 1),
+        Row.new(%q(}), 0, 0, 0),
+      ]
+
+      lines = []
+      input_with_correct_indents.each do |row|
+        lines << row.content
+        assert_indenting(lines, row.current_line_spaces, false)
+        assert_indenting(lines, row.new_line_spaces, true)
+        assert_nesting_level(lines, row.nesting_level)
+      end
+    end
+
+    def test_do_corresponding_to_loop
+      input_with_correct_indents = [
+        Row.new(%q(loop do), nil, 2, 1),
+        Row.new(%q(  3), nil, 2, 1),
+        Row.new(%q(end), 0, 0, 0),
+      ]
+
+      lines = []
+      input_with_correct_indents.each do |row|
+        lines << row.content
+        assert_indenting(lines, row.current_line_spaces, false)
+        assert_indenting(lines, row.new_line_spaces, true)
+        assert_nesting_level(lines, row.nesting_level)
+      end
+    end
+
     def test_heredoc_with_indent
       input_with_correct_indents = [
         Row.new(%q(<<~Q), nil, 0, 0),
diff --git a/test/reline/yamatanooroti/test_rendering.rb b/test/reline/yamatanooroti/test_rendering.rb
index b583f8ddac..0ccc331efd 100644
--- a/test/reline/yamatanooroti/test_rendering.rb
+++ b/test/reline/yamatanooroti/test_rendering.rb
@@ -678,6 +678,34 @@ begin
       EOC
     end
 
+    def test_autowrap_in_the_middle_of_a_line
+      start_terminal(5, 20, %W{ruby -I#{@pwd}/lib #{@pwd}/bin/multiline_repl}, startup_message: 'Multiline REPL.')
+      write("def abcdefg; end\C-b\C-b\C-b\C-b\C-b")
+      %w{h i}.each do |c|
+        write(c)
+      end
+      close
+      assert_screen(<<~EOC)
+        Multiline REPL.
+        prompt> def abcdefgh
+        i; end
+      EOC
+    end
+
+    def test_terminate_in_the_middle_of_lines
+      start_terminal(5, 20, %W{ruby -I#{@pwd}/lib #{@pwd}/bin/multiline_repl}, startup_message: 'Multiline REPL.')
+      write("def hoge\n  1\n  2\n  3\n  4\nend\n")
+      write("\C-p\C-p\C-p\C-e\n")
+      close
+      assert_screen(<<~EOC)
+        prompt>   3
+        prompt>   4
+        prompt> end
+        => :hoge
+        prompt>
+      EOC
+    end
+
     private def write_inputrc(content)
       File.open(@inputrc_file, 'w') do |f|
         f.write content