diff options
| author | Koichi ITO <koic.ito@gmail.com> | 2024-03-13 00:00:02 +0900 |
|---|---|---|
| committer | git <svn-admin@ruby-lang.org> | 2024-03-12 15:17:20 +0000 |
| commit | c637f53edd0c3f285d29d8679903d3b8bfcfbe75 (patch) | |
| tree | b1edcad83829a787dfaa1ee3aec6692ad668478d /lib/prism/translation/parser | |
| parent | a41649bff4611a6004471e89cac31e6535e2cff8 (diff) | |
[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for `tBACK_REF2`:
## Parser gem (Expected)
Returns `tBACK_REF2` token:
```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[[:tCONSTANT, ["A", #<Parser::Source::Range example.rb 0...1>]], [:tCOLON2, ["::", #<Parser::Source::Range example.rb 1...3>]],
[:tBACK_REF2, ["`", #<Parser::Source::Range example.rb 3...4>]]]
```
## `Prism::Translation::Parser` (Actual)
Previously, the parser returned `tXSTRING_BEG` token when parsing the following:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[[:tCONSTANT, ["A", #<Parser::Source::Range example.rb 0...1>]], [:tCOLON2, ["::", #<Parser::Source::Range example.rb 1...3>]],
[:tXSTRING_BEG, ["`", #<Parser::Source::Range example.rb 3...4>]]]
```
After the update, the parser now returns `tBACK_REF2` token for the same input:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[[:tCONSTANT, ["A", #<Parser::Source::Range example.rb 0...1>]], [:tCOLON2, ["::", #<Parser::Source::Range example.rb 1...3>]],
[:tBACK_REF2, ["`", #<Parser::Source::Range example.rb 3...4>]]]
```
This correction enables the restoration of `constants.txt` as a test case.
https://github.com/ruby/prism/commit/7f63b28f98
Diffstat (limited to 'lib/prism/translation/parser')
| -rw-r--r-- | lib/prism/translation/parser/lexer.rb | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/lib/prism/translation/parser/lexer.rb b/lib/prism/translation/parser/lexer.rb index 590adef323..0f16902d40 100644 --- a/lib/prism/translation/parser/lexer.rb +++ b/lib/prism/translation/parser/lexer.rb @@ -24,7 +24,7 @@ module Prism AMPERSAND_DOT: :tANDDOT, AMPERSAND_EQUAL: :tOP_ASGN, BACK_REFERENCE: :tBACK_REF, - BACKTICK: :tXSTRING_BEG, + BACKTICK: :tBACK_REF2, BANG: :tBANG, BANG_EQUAL: :tNEQ, BANG_TILDE: :tNMATCH, |