diff options
| author | Koichi ITO <koic.ito@gmail.com> | 2024-03-14 00:37:16 +0900 |
|---|---|---|
| committer | git <svn-admin@ruby-lang.org> | 2024-03-13 16:02:10 +0000 |
| commit | 0f076fa520999a42d88a081d7674495db2f65b3f (patch) | |
| tree | b896b4782c580f5738fc9a8040a79ab3ede270ab /lib/prism/translation/parser | |
| parent | eb4082284aace391a16a389a70eeaf1e7db5c542 (diff) | |
[ruby/prism] Fix an AST and token incompatibility for `Prism::Translation::Parser`
This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
for empty xstring literal.
## Parser gem (Expected)
```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
[:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
```
## `Prism::Translation::Parser` (Actual)
Previously, the AST and tokens returned by the Parser gem were different:
```console
$ bunele exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr, s(:str, "")), [], [[:tBACK_REF2, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
[:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
```
After this correction, the AST and tokens returned by the Parser gem are the same:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
[:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
```
https://github.com/ruby/prism/commit/4ac89dcbb5
Diffstat (limited to 'lib/prism/translation/parser')
| -rw-r--r-- | lib/prism/translation/parser/compiler.rb | 2 | ||||
| -rw-r--r-- | lib/prism/translation/parser/lexer.rb | 2 |
2 files changed, 2 insertions, 2 deletions
diff --git a/lib/prism/translation/parser/compiler.rb b/lib/prism/translation/parser/compiler.rb index cf628cddd4..50c47c068e 100644 --- a/lib/prism/translation/parser/compiler.rb +++ b/lib/prism/translation/parser/compiler.rb @@ -1678,7 +1678,7 @@ module Prism children, closing = visit_heredoc(node.to_interpolated) builder.xstring_compose(token(node.opening_loc), children, closing) else - parts = if node.unescaped.lines.count <= 1 + parts = if node.unescaped.lines.one? [builder.string_internal([node.unescaped, srange(node.content_loc)])] else start_offset = node.content_loc.start_offset diff --git a/lib/prism/translation/parser/lexer.rb b/lib/prism/translation/parser/lexer.rb index 92495ab3d2..9cf86476ba 100644 --- a/lib/prism/translation/parser/lexer.rb +++ b/lib/prism/translation/parser/lexer.rb @@ -326,7 +326,7 @@ module Prism type = :tIDENTIFIER end when :tXSTRING_BEG - if (next_token = lexed[index][0]) && next_token.type != :STRING_CONTENT + if (next_token = lexed[index][0]) && next_token.type != :STRING_CONTENT && next_token.type != :STRING_END type = :tBACK_REF2 end end |