Backport lib/reline, ext/readline, and lib/irb for 3.0.1 (#4085)

* Get rid of inconsistent dll linkages against vcpkg readline

* [ruby/irb] Enhance colored inspect output

https://github.com/ruby/irb/commit/dffcdb5269

* [ruby/irb] Add color_printer.rb to gemspec

https://github.com/ruby/irb/commit/b4df0fd8b2

* [ruby/irb] Fix failing tests

https://github.com/ruby/irb/commit/7723ade899

* irb: add more syntax errors colorizing support (#3967)

* [ruby/irb] Do not colorize partially-correct inspect

This is to prevent a yellow-mixed output for ActiveSupport::TimeWithZone.

Follows up https://github.com/ruby/irb/pull/159 and https://github.com/ruby/ruby/pull/3967.

https://github.com/ruby/irb/commit/a5804c3560bb1de3ea8e40002635bff87f6a2825

* [ruby/irb] Remove unnecessary ignore_error in dispatch_seq

Just forgotten in https://github.com/ruby/irb/commit/a5804c3560bb1de3ea8e40002635bff87f6a2825

https://github.com/ruby/irb/commit/e42e548793

* Increase timeout for reline with --jit-wait

for failures like:
http://ci.rvm.jp/logfiles/brlog.trunk-mjit-wait.20201229-130509
http://ci.rvm.jp/logfiles/brlog.trunk-mjit-wait.20201229-165132
http://ci.rvm.jp/logfiles/brlog.trunk-mjit-wait.20201228-015519

* [ruby/irb] Stringify when a non-object is passed to PP#text

If a nested object is passed to #pp, it may be sometimes passed to the #text
method as an object without being stringified.

This is fixed on the Ruby main repository;
https://github.com/ruby/ruby/commit/433a3be86a811de0b4adbb92e054ee3a6fc6b4d8
but it was a bug of Ripper so still needs this workaround for using irb
as a gem on Ruby 3.0.0 or earlier.

Co-authored-by: k0kubun <takashikkbn@gmail.com>

https://github.com/ruby/irb/commit/8d13df22ee

* [ruby/irb] Newline in oneliner def doesn't reset indent

This closes ruby/irb#132.

https://github.com/ruby/irb/commit/43456dcf5e

* [ruby/irb] Escape invalid byte sequence in Exception

This fixes ruby/irb#141.

https://github.com/ruby/irb/commit/0815317d42

* [ruby/irb] Handle indentations related to keyword "do" correctly

This fixes ruby/irb#158.

https://github.com/ruby/irb/commit/964643400b

* [ruby/irb] Heredoc may contain multiple newlines in a single token

Use the start token as the indentation criteria so that it works properly in
heredoc.

ref. https://github.com/ruby/reline/pull/242

https://github.com/ruby/irb/commit/9704808dfd

* [ruby/irb] Use Ripper::Lexer#scan to take broken tokens

ref. https://github.com/ruby/reline/pull/242

https://github.com/ruby/irb/commit/54f90cb6c9

* [ruby/irb] Use error tokens if there are no correct tokens in the same place

For example, the broken code "%www" will result in only one error token.

https://github.com/ruby/irb/commit/9fa39a7cf3

* [ruby/irb] Ensure to restore $VERBOSE

https://github.com/ruby/irb/commit/cef474a76a

* 600x larger timeout for Reline

I didn't notice it's msec. 2.5s is too short.
http://ci.rvm.jp/results/trunk-mjit-wait@phosphorus-docker/3311385

* [ruby/irb] fix typo in `IRB::Irb#convert_invalid_byte_sequence`

https://github.com/ruby/irb/commit/d09d3c3d68

* [ruby/irb] do not escape a predicate method for doc namespace

* Fixes #88

https://github.com/ruby/irb/commit/d431a30af4

* [ruby/irb] refactoring an error handling in `IRB::Inspector`

* moved rescue clause to `#inspect_value` to catch all failures in inspectors
* test with all (currently five kind of) inspect modes
  - tweaked the input due to only `Marshal` can inspect(dump) a `BasicObject`

https://github.com/ruby/irb/commit/9d112fab8e

* [ruby/irb] Use Exception#full_message to show backtrace in the correct order

[Bug #17466]

https://github.com/ruby/irb/commit/1c76845cca

* [ruby/irb] Fix BACK_TRACE_LIMIT logic

https://github.com/ruby/irb/commit/30dc5d43fe

* irb: Drop lines from backtrace for tests in Ruby repository

* [ruby/reline] Update cursor correctly when just cursor moving

This fixes ruby/reline#236 and ruby/reline#239.

https://github.com/ruby/reline/commit/3e3c89d00b

* [ruby/reline] Correct var names in Reline were different from vi-*-mode-string

https://github.com/ruby/reline/commit/8255fc93b9

* [ruby/reline] Remove debug print

https://github.com/ruby/reline/commit/d7fbaedc6a

* [ruby/reline] Suppress crashing when auto_indent_proc returns broken indent info

Co-authored-by: Juanito Fatas <me@juanitofatas.com>

https://github.com/ruby/reline/commit/7c24276275

* [ruby/reline] Suppress crashing when dynamic_prompt_proc returns a broken prompt list

Co-authored-by: Juanito Fatas <me@juanitofatas.com>

https://github.com/ruby/reline/commit/558f7be168

* [ruby/reline] Suppress auto indent for adding newlines in pasting

Co-authored-by: Juanito Fatas <me@juanitofatas.com>

https://github.com/ruby/reline/commit/074bb017a7

* [ruby/reline] Add acknowledgments and license for rb-readline

https://github.com/ruby/reline/commit/19df59b916

* [ruby/irb] Fix comment, irb gem supports 2.5.0 or older

https://github.com/ruby/irb/commit/36118015ba

* should use `assert_include` here.

Random ordering test can introduce antoher candidate so it should be
`assert_include`.

* [ruby/irb] Add missing require

This is useful if you want to use IRB::ColorPrinter as a library like:

```
begin
  require 'irb/color_printer'
  IRB::ColorPrinter.pp(obj)
rescue LoadError
  pp(obj)
end
```

https://github.com/ruby/irb/commit/f8461691c7

* [ruby/irb] Make IRB::ColorPrinter.pp compatible with PP.pp

The incompatible interface is not helpful, again if you want to use it
as a standalone library, falling it back to PP.

Original PP.pp also ends with `out << "\n"`.

https://github.com/ruby/irb/commit/4c74c7d84c

* Suppress constant redefinition warnings

* Fix the failing test with XDG_CONFIG_HOME

* [ruby/irb] Version 1.3.1

https://github.com/ruby/irb/commit/c230d08911

* [ruby/reline] Handle ed_search_{prev,next}_history in multiline correctly

The current line was being handled incorrectly when displaying the hit
history, so it has been fixed to be correct.

https://github.com/ruby/reline/commit/a3df4343b3

* [ruby/reline] Move the cursor correctly when deleting at eol

This fixes ruby/reline#246.

https://github.com/ruby/reline/commit/07a73ba601

* [ruby/reline] Version 0.2.1

https://github.com/ruby/reline/commit/a3b3c6ee60

* [ruby/reline] Initialize a variable just in case

https://github.com/ruby/reline/commit/29b10f6e98

* [ruby/reline] Tests with yamatanooroti don't need chdir

Because of chdir, log files ware created in temporary directries on Windows.

https://github.com/ruby/reline/commit/200b469a68

* [ruby/reline] Windows needs more times to wait rendering

https://github.com/ruby/reline/commit/53ff2b09c7

* [ruby/reline] Support for change in Windows-specific behavior at eol

The behavior of automatically moving the cursor to the next line when
displaying a char at the eol on Windows suddenly disappeared.

https://github.com/ruby/reline/commit/cad4de6ee8

* [ruby/reline] Reline::Windows.erase_after_cursor erases attributes too

https://github.com/ruby/reline/commit/68b961dfc7

* [ruby/irb] [ruby/irb] [ruby/reline] Version 0.2.2

https://github.com/ruby/reline/commit/dfb710946f

https://github.com/ruby/irb/commit/1a1cdf9628

https://github.com/ruby/irb/commit/fe99faf8bd

* [ruby/irb] handle `__ENCODING__` as a keyword as well

https://github.com/ruby/irb/commit/a6a33d908f

* [ruby/irb] handle repeated exception separately

https://github.com/ruby/irb/commit/fcf6b34bc5

* [ruby/irb] skip a failling test on TruffleRuby

* due to the difference of backtrace pointed out by @aycabta

https://github.com/ruby/irb/commit/5e00a0ae61

* [ruby/irb] Version 1.3.2

https://github.com/ruby/irb/commit/a7699026cc

Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com>
Co-authored-by: Nobuhiro IMAI <nov@yo.rim.or.jp>
Co-authored-by: Koichi Sasada <ko1@atdot.net>
Co-authored-by: Hiroshi SHIBATA <hsbt@ruby-lang.org>

5 files changed, 50 insertions, 43 deletions

diff --git a/lib/reline/config.rb b/lib/reline/config.rb
index 4141031912..63ab7b7402 100644
--- a/lib/reline/config.rb
+++ b/lib/reline/config.rb
@@ -34,8 +34,8 @@ class Reline::Config
     show-all-if-unmodified
     visible-stats
     show-mode-in-prompt
-    vi-cmd-mode-icon
-    vi-ins-mode-icon
+    vi-cmd-mode-string
+    vi-ins-mode-string
     emacs-mode-string
     enable-bracketed-paste
     isearch-terminators
@@ -56,8 +56,8 @@ class Reline::Config
     @key_actors[:emacs] = Reline::KeyActor::Emacs.new
     @key_actors[:vi_insert] = Reline::KeyActor::ViInsert.new
     @key_actors[:vi_command] = Reline::KeyActor::ViCommand.new
-    @vi_cmd_mode_icon = '(cmd)'
-    @vi_ins_mode_icon = '(ins)'
+    @vi_cmd_mode_string = '(cmd)'
+    @vi_ins_mode_string = '(ins)'
     @emacs_mode_string = '@'
     # https://tiswww.case.edu/php/chet/readline/readline.html#IDX25
     @history_size = -1 # unlimited
@@ -270,9 +270,9 @@ class Reline::Config
         @show_mode_in_prompt = false
       end
     when 'vi-cmd-mode-string'
-      @vi_cmd_mode_icon = retrieve_string(value)
+      @vi_cmd_mode_string = retrieve_string(value)
     when 'vi-ins-mode-string'
-      @vi_ins_mode_icon = retrieve_string(value)
+      @vi_ins_mode_string = retrieve_string(value)
     when 'emacs-mode-string'
       @emacs_mode_string = retrieve_string(value)
     when *VARIABLE_NAMES then
diff --git a/lib/reline/line_editor.rb b/lib/reline/line_editor.rb
index d4075c0934..92ea42fffb 100644
--- a/lib/reline/line_editor.rb
+++ b/lib/reline/line_editor.rb
@@ -68,24 +68,24 @@ class Reline::LineEditor
     end
   end
 
-  private def check_mode_icon
-    mode_icon = nil
+  private def check_mode_string
+    mode_string = nil
     if @config.show_mode_in_prompt
       if @config.editing_mode_is?(:vi_command)
-        mode_icon = @config.vi_cmd_mode_icon
+        mode_string = @config.vi_cmd_mode_string
       elsif @config.editing_mode_is?(:vi_insert)
-        mode_icon = @config.vi_ins_mode_icon
+        mode_string = @config.vi_ins_mode_string
       elsif @config.editing_mode_is?(:emacs)
-        mode_icon = @config.emacs_mode_string
+        mode_string = @config.emacs_mode_string
       else
-        mode_icon = '?'
+        mode_string = '?'
       end
     end
-    if mode_icon != @prev_mode_icon
+    if mode_string != @prev_mode_string
       @rerender_all = true
     end
-    @prev_mode_icon = mode_icon
-    mode_icon
+    @prev_mode_string = mode_string
+    mode_string
   end
 
   private def check_multiline_prompt(buffer, prompt)
@@ -99,8 +99,8 @@ class Reline::LineEditor
       prompt = @prompt
     end
     if simplified_rendering?
-      mode_icon = check_mode_icon
-      prompt = mode_icon + prompt if mode_icon
+      mode_string = check_mode_string
+      prompt = mode_string + prompt if mode_string
       return [prompt, calculate_width(prompt, true), [prompt] * buffer.size]
     end
     if @prompt_proc
@@ -112,6 +112,7 @@ class Reline::LineEditor
           use_cached_prompt_list = true
         end
       end
+      use_cached_prompt_list = false if @rerender_all
       if use_cached_prompt_list
         prompt_list = @cached_prompt_list
       else
@@ -119,15 +120,21 @@ class Reline::LineEditor
         @prompt_cache_time = Time.now.to_f
       end
       prompt_list.map!{ prompt } if @vi_arg or @searching_prompt
-      mode_icon = check_mode_icon
-      prompt_list = prompt_list.map{ |pr| mode_icon + pr } if mode_icon
+      mode_string = check_mode_string
+      prompt_list = prompt_list.map{ |pr| mode_string + pr } if mode_string
       prompt = prompt_list[@line_index]
       prompt = prompt_list[0] if prompt.nil?
+      prompt = prompt_list.last if prompt.nil?
+      if buffer.size > prompt_list.size
+        (buffer.size - prompt_list.size).times do
+          prompt_list << prompt_list.last
+        end
+      end
       prompt_width = calculate_width(prompt, true)
       [prompt, prompt_width, prompt_list]
     else
-      mode_icon = check_mode_icon
-      prompt = mode_icon + prompt if mode_icon
+      mode_string = check_mode_string
+      prompt = mode_string + prompt if mode_string
       prompt_width = calculate_width(prompt, true)
       [prompt, prompt_width, nil]
     end
@@ -218,7 +225,7 @@ class Reline::LineEditor
     @eof = false
     @continuous_insertion_buffer = String.new(encoding: @encoding)
     @scroll_partial_screen = nil
-    @prev_mode_icon = nil
+    @prev_mode_string = nil
     @drop_terminate_spaces = false
     reset_line
   end
@@ -370,6 +377,7 @@ class Reline::LineEditor
     end
     new_highest_in_this = calculate_height_by_width(prompt_width + calculate_width(@line.nil? ? '' : @line))
     # FIXME: end of logical line sometimes breaks
+    rendered = false
     if @add_newline_to_end_of_buffer
       rerender_added_newline
       @add_newline_to_end_of_buffer = false
@@ -471,7 +479,7 @@ class Reline::LineEditor
         calculate_height_by_lines(@buffer_of_lines[0..(@line_index - 1)], prompt_list || prompt)
       end
     first_line_diff = new_first_line_started_from - @first_line_started_from
-    new_cursor, _, new_started_from, _ = calculate_nearest_cursor(@line, @cursor, @started_from, @byte_pointer, false)
+    new_cursor, new_cursor_max, new_started_from, new_byte_pointer = calculate_nearest_cursor(@buffer_of_lines[@line_index], @cursor, @started_from, @byte_pointer, false)
     new_started_from = calculate_height_by_width(prompt_width + new_cursor) - 1
     calculate_scroll_partial_screen(@highest_in_all, new_first_line_started_from + new_started_from)
     @previous_line_index = nil
@@ -485,6 +493,8 @@ class Reline::LineEditor
       @first_line_started_from = new_first_line_started_from
       @started_from = new_started_from
       @cursor = new_cursor
+      @cursor_max = new_cursor_max
+      @byte_pointer = new_byte_pointer
       move_cursor_down(first_line_diff + @started_from)
       Reline::IOGate.move_cursor_column((prompt_width + @cursor) % @screen_size.last)
       false
@@ -666,17 +676,13 @@ class Reline::LineEditor
       Reline::IOGate.move_cursor_column(0)
       if line.nil?
         if calculate_width(visual_lines[index - 1], true) == Reline::IOGate.get_screen_size.last
-          # reaches the end of line
-          if Reline::IOGate.win?
-            # A newline is automatically inserted if a character is rendered at
-            # eol on command prompt.
-          else
-            # When the cursor is at the end of the line and erases characters
-            # after the cursor, some terminals delete the character at the
-            # cursor position.
-            move_cursor_down(1)
-            Reline::IOGate.move_cursor_column(0)
-          end
+          # Reaches the end of line.
+          #
+          # When the cursor is at the end of the line and erases characters
+          # after the cursor, some terminals delete the character at the
+          # cursor position.
+          move_cursor_down(1)
+          Reline::IOGate.move_cursor_column(0)
         else
           Reline::IOGate.erase_after_cursor
           move_cursor_down(1)
@@ -685,10 +691,6 @@ class Reline::LineEditor
         next
       end
       @output.write line
-      if Reline::IOGate.win? and calculate_width(line, true) == Reline::IOGate.get_screen_size.last
-        # A newline is automatically inserted if a character is rendered at eol on command prompt.
-        @rest_height -= 1 if @rest_height > 0
-      end
       @output.flush
       if @first_prompt
         @first_prompt = false
@@ -1129,6 +1131,7 @@ class Reline::LineEditor
       new_lines = whole_lines
     end
     new_indent = @auto_indent_proc.(new_lines, @line_index, @byte_pointer, @check_new_auto_indent)
+    new_indent = @cursor_max if new_indent&.> @cursor_max
     if new_indent&.>= 0
       md = new_lines[@line_index].match(/\A */)
       prev_indent = md[0].count(' ')
@@ -1329,7 +1332,7 @@ class Reline::LineEditor
       cursor_line = @line.byteslice(0, @byte_pointer)
       insert_new_line(cursor_line, next_line)
       @cursor = 0
-      @check_new_auto_indent = true
+      @check_new_auto_indent = true unless Reline::IOGate.in_pasting?
     end
   end
 
@@ -1722,7 +1725,7 @@ class Reline::LineEditor
       @buffer_of_lines = Reline::HISTORY[@history_pointer].split("\n")
       @buffer_of_lines = [String.new(encoding: @encoding)] if @buffer_of_lines.empty?
       @line_index = line_no
-      @line = @buffer_of_lines.last
+      @line = @buffer_of_lines[@line_index]
       @rerender_all = true
     else
       @line = Reline::HISTORY[@history_pointer]
@@ -1770,7 +1773,7 @@ class Reline::LineEditor
         @line_index = line_no
       end
       @buffer_of_lines = [String.new(encoding: @encoding)] if @buffer_of_lines.empty?
-      @line = @buffer_of_lines.last
+      @line = @buffer_of_lines[@line_index]
       @rerender_all = true
     else
       if @history_pointer.nil? and substr.empty?
@@ -2385,6 +2388,9 @@ class Reline::LineEditor
       width = Reline::Unicode.get_mbchar_width(mbchar)
       @cursor_max -= width
       if @cursor > 0 and @cursor >= @cursor_max
+        byte_size = Reline::Unicode.get_prev_mbchar_size(@line, @byte_pointer)
+        mbchar = @line.byteslice(@byte_pointer - byte_size, byte_size)
+        width = Reline::Unicode.get_mbchar_width(mbchar)
         @byte_pointer -= byte_size
         @cursor -= width
       end
diff --git a/lib/reline/reline.gemspec b/lib/reline/reline.gemspec
index 0a0b129228..763c0f8bc3 100644
--- a/lib/reline/reline.gemspec
+++ b/lib/reline/reline.gemspec
@@ -14,7 +14,7 @@ Gem::Specification.new do |spec|
   spec.homepage      = 'https://github.com/ruby/reline'
   spec.license       = 'Ruby'
 
-  spec.files         = Dir['BSDL', 'COPYING', 'README.md', 'lib/**/*']
+  spec.files         = Dir['BSDL', 'COPYING', 'README.md', 'license_of_rb-readline', 'lib/**/*']
   spec.require_paths = ['lib']
 
   spec.required_ruby_version = Gem::Requirement.new('>= 2.5')
diff --git a/lib/reline/version.rb b/lib/reline/version.rb
index 8499989d07..9241aef5a9 100644
--- a/lib/reline/version.rb
+++ b/lib/reline/version.rb
@@ -1,3 +1,3 @@
 module Reline
-  VERSION = '0.2.0'
+  VERSION = '0.2.2'
 end
diff --git a/lib/reline/windows.rb b/lib/reline/windows.rb
index 937941b960..4f5fcb74bc 100644
--- a/lib/reline/windows.rb
+++ b/lib/reline/windows.rb
@@ -258,6 +258,7 @@ class Reline::Windows
     cursor = csbi[4, 4].unpack('L').first
     written = 0.chr * 4
     @@FillConsoleOutputCharacter.call(@@hConsoleHandle, 0x20, get_screen_size.last - cursor_pos.x, cursor, written)
+    @@FillConsoleOutputAttribute.call(@@hConsoleHandle, 0, get_screen_size.last - cursor_pos.x, cursor, written)
   end
 
   def self.scroll_down(val)