Backport lib/reline, ext/readline, and lib/irb for 3.0.1 (#4085)

* Get rid of inconsistent dll linkages against vcpkg readline

* [ruby/irb] Enhance colored inspect output

https://github.com/ruby/irb/commit/dffcdb5269

* [ruby/irb] Add color_printer.rb to gemspec

https://github.com/ruby/irb/commit/b4df0fd8b2

* [ruby/irb] Fix failing tests

https://github.com/ruby/irb/commit/7723ade899

* irb: add more syntax errors colorizing support (#3967)

* [ruby/irb] Do not colorize partially-correct inspect

This is to prevent a yellow-mixed output for ActiveSupport::TimeWithZone.

Follows up https://github.com/ruby/irb/pull/159 and https://github.com/ruby/ruby/pull/3967.

https://github.com/ruby/irb/commit/a5804c3560bb1de3ea8e40002635bff87f6a2825

* [ruby/irb] Remove unnecessary ignore_error in dispatch_seq

Just forgotten in https://github.com/ruby/irb/commit/a5804c3560bb1de3ea8e40002635bff87f6a2825

https://github.com/ruby/irb/commit/e42e548793

* Increase timeout for reline with --jit-wait

for failures like:
http://ci.rvm.jp/logfiles/brlog.trunk-mjit-wait.20201229-130509
http://ci.rvm.jp/logfiles/brlog.trunk-mjit-wait.20201229-165132
http://ci.rvm.jp/logfiles/brlog.trunk-mjit-wait.20201228-015519

* [ruby/irb] Stringify when a non-object is passed to PP#text

If a nested object is passed to #pp, it may be sometimes passed to the #text
method as an object without being stringified.

This is fixed on the Ruby main repository;
https://github.com/ruby/ruby/commit/433a3be86a811de0b4adbb92e054ee3a6fc6b4d8
but it was a bug of Ripper so still needs this workaround for using irb
as a gem on Ruby 3.0.0 or earlier.

Co-authored-by: k0kubun <takashikkbn@gmail.com>

https://github.com/ruby/irb/commit/8d13df22ee

* [ruby/irb] Newline in oneliner def doesn't reset indent

This closes ruby/irb#132.

https://github.com/ruby/irb/commit/43456dcf5e

* [ruby/irb] Escape invalid byte sequence in Exception

This fixes ruby/irb#141.

https://github.com/ruby/irb/commit/0815317d42

* [ruby/irb] Handle indentations related to keyword "do" correctly

This fixes ruby/irb#158.

https://github.com/ruby/irb/commit/964643400b

* [ruby/irb] Heredoc may contain multiple newlines in a single token

Use the start token as the indentation criteria so that it works properly in
heredoc.

ref. https://github.com/ruby/reline/pull/242

https://github.com/ruby/irb/commit/9704808dfd

* [ruby/irb] Use Ripper::Lexer#scan to take broken tokens

ref. https://github.com/ruby/reline/pull/242

https://github.com/ruby/irb/commit/54f90cb6c9

* [ruby/irb] Use error tokens if there are no correct tokens in the same place

For example, the broken code "%www" will result in only one error token.

https://github.com/ruby/irb/commit/9fa39a7cf3

* [ruby/irb] Ensure to restore $VERBOSE

https://github.com/ruby/irb/commit/cef474a76a

* 600x larger timeout for Reline

I didn't notice it's msec. 2.5s is too short.
http://ci.rvm.jp/results/trunk-mjit-wait@phosphorus-docker/3311385

* [ruby/irb] fix typo in `IRB::Irb#convert_invalid_byte_sequence`

https://github.com/ruby/irb/commit/d09d3c3d68

* [ruby/irb] do not escape a predicate method for doc namespace

* Fixes #88

https://github.com/ruby/irb/commit/d431a30af4

* [ruby/irb] refactoring an error handling in `IRB::Inspector`

* moved rescue clause to `#inspect_value` to catch all failures in inspectors
* test with all (currently five kind of) inspect modes
  - tweaked the input due to only `Marshal` can inspect(dump) a `BasicObject`

https://github.com/ruby/irb/commit/9d112fab8e

* [ruby/irb] Use Exception#full_message to show backtrace in the correct order

[Bug #17466]

https://github.com/ruby/irb/commit/1c76845cca

* [ruby/irb] Fix BACK_TRACE_LIMIT logic

https://github.com/ruby/irb/commit/30dc5d43fe

* irb: Drop lines from backtrace for tests in Ruby repository

* [ruby/reline] Update cursor correctly when just cursor moving

This fixes ruby/reline#236 and ruby/reline#239.

https://github.com/ruby/reline/commit/3e3c89d00b

* [ruby/reline] Correct var names in Reline were different from vi-*-mode-string

https://github.com/ruby/reline/commit/8255fc93b9

* [ruby/reline] Remove debug print

https://github.com/ruby/reline/commit/d7fbaedc6a

* [ruby/reline] Suppress crashing when auto_indent_proc returns broken indent info

Co-authored-by: Juanito Fatas <me@juanitofatas.com>

https://github.com/ruby/reline/commit/7c24276275

* [ruby/reline] Suppress crashing when dynamic_prompt_proc returns a broken prompt list

Co-authored-by: Juanito Fatas <me@juanitofatas.com>

https://github.com/ruby/reline/commit/558f7be168

* [ruby/reline] Suppress auto indent for adding newlines in pasting

Co-authored-by: Juanito Fatas <me@juanitofatas.com>

https://github.com/ruby/reline/commit/074bb017a7

* [ruby/reline] Add acknowledgments and license for rb-readline

https://github.com/ruby/reline/commit/19df59b916

* [ruby/irb] Fix comment, irb gem supports 2.5.0 or older

https://github.com/ruby/irb/commit/36118015ba

* should use `assert_include` here.

Random ordering test can introduce antoher candidate so it should be
`assert_include`.

* [ruby/irb] Add missing require

This is useful if you want to use IRB::ColorPrinter as a library like:

```
begin
  require 'irb/color_printer'
  IRB::ColorPrinter.pp(obj)
rescue LoadError
  pp(obj)
end
```

https://github.com/ruby/irb/commit/f8461691c7

* [ruby/irb] Make IRB::ColorPrinter.pp compatible with PP.pp

The incompatible interface is not helpful, again if you want to use it
as a standalone library, falling it back to PP.

Original PP.pp also ends with `out << "\n"`.

https://github.com/ruby/irb/commit/4c74c7d84c

* Suppress constant redefinition warnings

* Fix the failing test with XDG_CONFIG_HOME

* [ruby/irb] Version 1.3.1

https://github.com/ruby/irb/commit/c230d08911

* [ruby/reline] Handle ed_search_{prev,next}_history in multiline correctly

The current line was being handled incorrectly when displaying the hit
history, so it has been fixed to be correct.

https://github.com/ruby/reline/commit/a3df4343b3

* [ruby/reline] Move the cursor correctly when deleting at eol

This fixes ruby/reline#246.

https://github.com/ruby/reline/commit/07a73ba601

* [ruby/reline] Version 0.2.1

https://github.com/ruby/reline/commit/a3b3c6ee60

* [ruby/reline] Initialize a variable just in case

https://github.com/ruby/reline/commit/29b10f6e98

* [ruby/reline] Tests with yamatanooroti don't need chdir

Because of chdir, log files ware created in temporary directries on Windows.

https://github.com/ruby/reline/commit/200b469a68

* [ruby/reline] Windows needs more times to wait rendering

https://github.com/ruby/reline/commit/53ff2b09c7

* [ruby/reline] Support for change in Windows-specific behavior at eol

The behavior of automatically moving the cursor to the next line when
displaying a char at the eol on Windows suddenly disappeared.

https://github.com/ruby/reline/commit/cad4de6ee8

* [ruby/reline] Reline::Windows.erase_after_cursor erases attributes too

https://github.com/ruby/reline/commit/68b961dfc7

* [ruby/irb] [ruby/irb] [ruby/reline] Version 0.2.2

https://github.com/ruby/reline/commit/dfb710946f

https://github.com/ruby/irb/commit/1a1cdf9628

https://github.com/ruby/irb/commit/fe99faf8bd

* [ruby/irb] handle `__ENCODING__` as a keyword as well

https://github.com/ruby/irb/commit/a6a33d908f

* [ruby/irb] handle repeated exception separately

https://github.com/ruby/irb/commit/fcf6b34bc5

* [ruby/irb] skip a failling test on TruffleRuby

* due to the difference of backtrace pointed out by @aycabta

https://github.com/ruby/irb/commit/5e00a0ae61

* [ruby/irb] Version 1.3.2

https://github.com/ruby/irb/commit/a7699026cc

Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com>
Co-authored-by: Nobuhiro IMAI <nov@yo.rim.or.jp>
Co-authored-by: Koichi Sasada <ko1@atdot.net>
Co-authored-by: Hiroshi SHIBATA <hsbt@ruby-lang.org>

1 files changed, 16 insertions, 3 deletions

diff --git a/lib/irb/color.rb b/lib/irb/color.rb
index 0f49291d85..a054bb20f8 100644
--- a/lib/irb/color.rb
+++ b/lib/irb/color.rb
@@ -17,7 +17,7 @@ module IRB # :nodoc:
     CYAN      = 36
 
     TOKEN_KEYWORDS = {
-      on_kw: ['nil', 'self', 'true', 'false', '__FILE__', '__LINE__'],
+      on_kw: ['nil', 'self', 'true', 'false', '__FILE__', '__LINE__', '__ENCODING__'],
       on_const: ['ENV'],
     }
     private_constant :TOKEN_KEYWORDS
@@ -60,6 +60,10 @@ module IRB # :nodoc:
         on_words_beg:       [[RED, BOLD],             ALL],
         on_parse_error:     [[RED, REVERSE],          ALL],
         compile_error:      [[RED, REVERSE],          ALL],
+        on_assign_error:    [[RED, REVERSE],          ALL],
+        on_alias_error:     [[RED, REVERSE],          ALL],
+        on_class_name_error:[[RED, REVERSE],          ALL],
+        on_param_error:     [[RED, REVERSE],          ALL],
       }
     rescue NameError
       # Give up highlighting Ripper-incompatible older Ruby
@@ -67,6 +71,9 @@ module IRB # :nodoc:
     end
     private_constant :TOKEN_SEQ_EXPRS
 
+    ERROR_TOKENS = TOKEN_SEQ_EXPRS.keys.select { |k| k.to_s.end_with?('error') }
+    private_constant :ERROR_TOKENS
+
     class << self
       def colorable?
         $stdout.tty? && supported? && (/mswin|mingw/ =~ RUBY_PLATFORM || (ENV.key?('TERM') && ENV['TERM'] != 'dumb'))
@@ -107,7 +114,7 @@ module IRB # :nodoc:
       # If `complete` is false (code is incomplete), this does not warn compile_error.
       # This option is needed to avoid warning a user when the compile_error is happening
       # because the input is not wrong but just incomplete.
-      def colorize_code(code, complete: true)
+      def colorize_code(code, complete: true, ignore_error: false)
         return code unless colorable?
 
         symbol_state = SymbolState.new
@@ -115,6 +122,11 @@ module IRB # :nodoc:
         length = 0
 
         scan(code, allow_last_error: !complete) do |token, str, expr|
+          # IRB::ColorPrinter skips colorizing fragments with any invalid token
+          if ignore_error && ERROR_TOKENS.include?(token)
+            return Reline::Unicode.escape_for_print(code)
+          end
+
           in_symbol = symbol_state.scan_token(token)
           str.each_line do |line|
             line = Reline::Unicode.escape_for_print(line)
@@ -180,11 +192,12 @@ module IRB # :nodoc:
             end
           end
         end
+      ensure
         $VERBOSE = verbose
       end
 
       def dispatch_seq(token, expr, str, in_symbol:)
-        if token == :on_parse_error or token == :compile_error
+        if ERROR_TOKENS.include?(token)
           TOKEN_SEQ_EXPRS[token][0]
         elsif in_symbol
           [YELLOW]