<feed xmlns='http://www.w3.org/2005/Atom'>
<title>ruby.git/lib/prism/translation/parser/lexer.rb, branch v3_4_9</title>
<subtitle>The Ruby Programming Language</subtitle>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/'/>
<entry>
<title>Bump Prism version to 1.5.0</title>
<updated>2025-09-12T21:30:16+00:00</updated>
<author>
<name>Takashi Kokubun</name>
<email>takashikkbn@gmail.com</email>
</author>
<published>2025-09-12T20:09:30+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=52b3f006cb31672c45b26f6a44a609fd1b8e2ee5'/>
<id>52b3f006cb31672c45b26f6a44a609fd1b8e2ee5</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Fix `kDO_LAMBDA` token incompatibility for `Prism::Translation::Parser::Lexer`</title>
<updated>2024-09-20T17:17:21+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-09-10T03:22:13+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=75ed086348da66e4cfe9488ae9ece5462dd2aef9'/>
<id>75ed086348da66e4cfe9488ae9ece5462dd2aef9</id>
<content type='text'>
## Summary

This PR fixes `kDO_LAMBDA` token incompatibility between Parser gem and `Prism::Translation::Parser` for lambda `do` block.

### Parser gem (Expected)

Returns `kDO_LAMBDA` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-&gt; do end"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 0...2&gt;]], [:kDO_LAMBDA, ["do", #&lt;Parser::Source::Range example.rb 3...5&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 6...9&gt;]]]
```

### `Prism::Translation::Parser` (Actual)

Previously, the parser returned `kDO` token when parsing the following:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-&gt; do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 0...2&gt;]], [:kDO, ["do", #&lt;Parser::Source::Range example.rb 3...5&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 6...9&gt;]]]
```

After the update, the parser now returns `kDO_LAMBDA` token for the same input:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-&gt; do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 0...2&gt;]], [:kDO_LAMBDA, ["do", #&lt;Parser::Source::Range example.rb 3...5&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 6...9&gt;]]]
```

## Additional Information

Unfortunately, this kind of edge case doesn't work as expected; `kDO` is returned instead of `kDO_LAMBDA`.
However, since `kDO` is already being returned in this case, there is no change in behavior.

### Parser gem

Returns `tLAMBDA` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-&gt; (foo = -&gt; (bar) {}) do end"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.3.5 (2024-09-03 revision https://github.com/ruby/prism/commit/ef084cc8f4) [x86_64-darwin23]
[[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 0...2&gt;]], [:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 3...4&gt;]],
[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 4...7&gt;]], [:tEQL, ["=", #&lt;Parser::Source::Range example.rb 8...9&gt;]],
[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 10...12&gt;]], [:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 13...14&gt;]],
[:tIDENTIFIER, ["bar", #&lt;Parser::Source::Range example.rb 14...17&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 17...18&gt;]],
[:tLAMBEG, ["{", #&lt;Parser::Source::Range example.rb 19...20&gt;]], [:tRCURLY, ["}", #&lt;Parser::Source::Range example.rb 20...21&gt;]],
[:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 21...22&gt;]], [:kDO_LAMBDA, ["do", #&lt;Parser::Source::Range example.rb 23...25&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 26...29&gt;]]]
```

### `Prism::Translation::Parser`

Returns `kDO` token:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-&gt; (foo = -&gt; (bar) {}) do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.5 (2024-09-03 revision https://github.com/ruby/prism/commit/ef084cc8f4) [x86_64-darwin23]
[[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 0...2&gt;]], [:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 3...4&gt;]],
[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 4...7&gt;]], [:tEQL, ["=", #&lt;Parser::Source::Range example.rb 8...9&gt;]],
[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 10...12&gt;]], [:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 13...14&gt;]],
[:tIDENTIFIER, ["bar", #&lt;Parser::Source::Range example.rb 14...17&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 17...18&gt;]],
[:tLAMBEG, ["{", #&lt;Parser::Source::Range example.rb 19...20&gt;]], [:tRCURLY, ["}", #&lt;Parser::Source::Range example.rb 20...21&gt;]],
[:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 21...22&gt;]], [:kDO, ["do", #&lt;Parser::Source::Range example.rb 23...25&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 26...29&gt;]]]
```

As the intention is not to address such special cases at this point, a comment has been left indicating that this case still returns `kDO`.
In other words, `kDO_LAMBDA` will now be returned except for edge cases after this PR.

https://github.com/ruby/prism/commit/2ee480654c
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
## Summary

This PR fixes `kDO_LAMBDA` token incompatibility between Parser gem and `Prism::Translation::Parser` for lambda `do` block.

### Parser gem (Expected)

Returns `kDO_LAMBDA` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-&gt; do end"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 0...2&gt;]], [:kDO_LAMBDA, ["do", #&lt;Parser::Source::Range example.rb 3...5&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 6...9&gt;]]]
```

### `Prism::Translation::Parser` (Actual)

Previously, the parser returned `kDO` token when parsing the following:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-&gt; do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 0...2&gt;]], [:kDO, ["do", #&lt;Parser::Source::Range example.rb 3...5&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 6...9&gt;]]]
```

After the update, the parser now returns `kDO_LAMBDA` token for the same input:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-&gt; do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 0...2&gt;]], [:kDO_LAMBDA, ["do", #&lt;Parser::Source::Range example.rb 3...5&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 6...9&gt;]]]
```

## Additional Information

Unfortunately, this kind of edge case doesn't work as expected; `kDO` is returned instead of `kDO_LAMBDA`.
However, since `kDO` is already being returned in this case, there is no change in behavior.

### Parser gem

Returns `tLAMBDA` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-&gt; (foo = -&gt; (bar) {}) do end"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.3.5 (2024-09-03 revision https://github.com/ruby/prism/commit/ef084cc8f4) [x86_64-darwin23]
[[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 0...2&gt;]], [:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 3...4&gt;]],
[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 4...7&gt;]], [:tEQL, ["=", #&lt;Parser::Source::Range example.rb 8...9&gt;]],
[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 10...12&gt;]], [:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 13...14&gt;]],
[:tIDENTIFIER, ["bar", #&lt;Parser::Source::Range example.rb 14...17&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 17...18&gt;]],
[:tLAMBEG, ["{", #&lt;Parser::Source::Range example.rb 19...20&gt;]], [:tRCURLY, ["}", #&lt;Parser::Source::Range example.rb 20...21&gt;]],
[:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 21...22&gt;]], [:kDO_LAMBDA, ["do", #&lt;Parser::Source::Range example.rb 23...25&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 26...29&gt;]]]
```

### `Prism::Translation::Parser`

Returns `kDO` token:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-&gt; (foo = -&gt; (bar) {}) do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.5 (2024-09-03 revision https://github.com/ruby/prism/commit/ef084cc8f4) [x86_64-darwin23]
[[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 0...2&gt;]], [:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 3...4&gt;]],
[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 4...7&gt;]], [:tEQL, ["=", #&lt;Parser::Source::Range example.rb 8...9&gt;]],
[:tLAMBDA, ["-&gt;", #&lt;Parser::Source::Range example.rb 10...12&gt;]], [:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 13...14&gt;]],
[:tIDENTIFIER, ["bar", #&lt;Parser::Source::Range example.rb 14...17&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 17...18&gt;]],
[:tLAMBEG, ["{", #&lt;Parser::Source::Range example.rb 19...20&gt;]], [:tRCURLY, ["}", #&lt;Parser::Source::Range example.rb 20...21&gt;]],
[:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 21...22&gt;]], [:kDO, ["do", #&lt;Parser::Source::Range example.rb 23...25&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 26...29&gt;]]]
```

As the intention is not to address such special cases at this point, a comment has been left indicating that this case still returns `kDO`.
In other words, `kDO_LAMBDA` will now be returned except for edge cases after this PR.

https://github.com/ruby/prism/commit/2ee480654c
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`</title>
<updated>2024-09-09T19:01:30+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-09-08T09:25:47+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=7a6533452807aa432f097db4e637e4c480645d6b'/>
<id>7a6533452807aa432f097db4e637e4c480645d6b</id>
<content type='text'>
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for double splat argument.

## Parser gem (Expected)

Returns `tDSTAR` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:kDEF, ["def", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:tIDENTIFIER, ["f", #&lt;Parser::Source::Range example.rb 4...5&gt;]],
[:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 5...6&gt;]], [:tDSTAR, ["**", #&lt;Parser::Source::Range example.rb 6...8&gt;]],
[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 8...11&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 11...12&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 13...16&gt;]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the parser returned `tPOW` token when parsing the following:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:kDEF, ["def", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:tIDENTIFIER, ["f", #&lt;Parser::Source::Range example.rb 4...5&gt;]],
[:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 5...6&gt;]], [:tPOW, ["**", #&lt;Parser::Source::Range example.rb 6...8&gt;]],
[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 8...11&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 11...12&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 13...16&gt;]]]
```

After the update, the parser now returns `tDSTAR` token for the same input:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:kDEF, ["def", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:tIDENTIFIER, ["f", #&lt;Parser::Source::Range example.rb 4...5&gt;]],
[:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 5...6&gt;]], [:tDSTAR, ["**", #&lt;Parser::Source::Range example.rb 6...8&gt;]],
[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 8...11&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 11...12&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 13...16&gt;]]]
```

With this change, the following code could be removed from test/prism/ruby/parser_test.rb:

```diff
-          when :tPOW
-            actual_token[0] = expected_token[0] if expected_token[0] == :tDSTAR
```

`tPOW` is the token type for the behavior of `a ** b`, and its behavior remains unchanged:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "a ** b"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["a", #&lt;Parser::Source::Range example.rb 0...1&gt;]], [:tPOW, ["**", #&lt;Parser::Source::Range example.rb 2...4&gt;]],
[:tIDENTIFIER, ["b", #&lt;Parser::Source::Range example.rb 5...6&gt;]]]
```

https://github.com/ruby/prism/commit/66bde35a44
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for double splat argument.

## Parser gem (Expected)

Returns `tDSTAR` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:kDEF, ["def", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:tIDENTIFIER, ["f", #&lt;Parser::Source::Range example.rb 4...5&gt;]],
[:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 5...6&gt;]], [:tDSTAR, ["**", #&lt;Parser::Source::Range example.rb 6...8&gt;]],
[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 8...11&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 11...12&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 13...16&gt;]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the parser returned `tPOW` token when parsing the following:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:kDEF, ["def", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:tIDENTIFIER, ["f", #&lt;Parser::Source::Range example.rb 4...5&gt;]],
[:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 5...6&gt;]], [:tPOW, ["**", #&lt;Parser::Source::Range example.rb 6...8&gt;]],
[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 8...11&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 11...12&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 13...16&gt;]]]
```

After the update, the parser now returns `tDSTAR` token for the same input:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:kDEF, ["def", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:tIDENTIFIER, ["f", #&lt;Parser::Source::Range example.rb 4...5&gt;]],
[:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 5...6&gt;]], [:tDSTAR, ["**", #&lt;Parser::Source::Range example.rb 6...8&gt;]],
[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 8...11&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 11...12&gt;]],
[:kEND, ["end", #&lt;Parser::Source::Range example.rb 13...16&gt;]]]
```

With this change, the following code could be removed from test/prism/ruby/parser_test.rb:

```diff
-          when :tPOW
-            actual_token[0] = expected_token[0] if expected_token[0] == :tDSTAR
```

`tPOW` is the token type for the behavior of `a ** b`, and its behavior remains unchanged:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "a ** b"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["a", #&lt;Parser::Source::Range example.rb 0...1&gt;]], [:tPOW, ["**", #&lt;Parser::Source::Range example.rb 2...4&gt;]],
[:tIDENTIFIER, ["b", #&lt;Parser::Source::Range example.rb 5...6&gt;]]]
```

https://github.com/ruby/prism/commit/66bde35a44
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`</title>
<updated>2024-09-07T22:36:38+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-09-07T14:21:29+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=47742841246c637053dc8dad447c7e5e28182676'/>
<id>47742841246c637053dc8dad447c7e5e28182676</id>
<content type='text'>
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for left parenthesis.

## Parser gem (Expected)

Returns `tLPAREN2` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 \
-ve 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 3...4&gt;]],
[:tSYMBOL, ["bar", #&lt;Parser::Source::Range example.rb 4...8&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 8...9&gt;]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the parser returned `tLPAREN` token when parsing the following:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:tLPAREN, ["(", #&lt;Parser::Source::Range example.rb 3...4&gt;]],
[:tSYMBOL, ["bar", #&lt;Parser::Source::Range example.rb 4...8&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 8...9&gt;]]]
```

After the update, the parser now returns `tLPAREN2` token for the same input:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 3...4&gt;]],
[:tSYMBOL, ["bar", #&lt;Parser::Source::Range example.rb 4...8&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 8...9&gt;]]]
```

The `PARENTHESIS_LEFT` token in Prism is classified as either `tLPAREN` or `tLPAREN2` in the Parser gem.
The tokens that were previously all classified as `tLPAREN` are now also classified to `tLPAREN2`.

With this change, the following code could be removed from `test/prism/ruby/parser_test.rb`:

```diff
-          when :tLPAREN
-            actual_token[0] = expected_token[0] if expected_token[0] == :tLPAREN2
```

https://github.com/ruby/prism/commit/04d6f3478d
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for left parenthesis.

## Parser gem (Expected)

Returns `tLPAREN2` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 \
-ve 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 3...4&gt;]],
[:tSYMBOL, ["bar", #&lt;Parser::Source::Range example.rb 4...8&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 8...9&gt;]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the parser returned `tLPAREN` token when parsing the following:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:tLPAREN, ["(", #&lt;Parser::Source::Range example.rb 3...4&gt;]],
[:tSYMBOL, ["bar", #&lt;Parser::Source::Range example.rb 4...8&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 8...9&gt;]]]
```

After the update, the parser now returns `tLPAREN2` token for the same input:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #&lt;Parser::Source::Range example.rb 0...3&gt;]], [:tLPAREN2, ["(", #&lt;Parser::Source::Range example.rb 3...4&gt;]],
[:tSYMBOL, ["bar", #&lt;Parser::Source::Range example.rb 4...8&gt;]], [:tRPAREN, [")", #&lt;Parser::Source::Range example.rb 8...9&gt;]]]
```

The `PARENTHESIS_LEFT` token in Prism is classified as either `tLPAREN` or `tLPAREN2` in the Parser gem.
The tokens that were previously all classified as `tLPAREN` are now also classified to `tLPAREN2`.

With this change, the following code could be removed from `test/prism/ruby/parser_test.rb`:

```diff
-          when :tLPAREN
-            actual_token[0] = expected_token[0] if expected_token[0] == :tLPAREN2
```

https://github.com/ruby/prism/commit/04d6f3478d
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] (parser) Fix up tokens for empty symbol</title>
<updated>2024-06-19T01:18:39+00:00</updated>
<author>
<name>Kevin Newton</name>
<email>kddnewton@gmail.com</email>
</author>
<published>2024-06-12T13:44:28+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=24f48382bc3ab62ca0979f1a6879f0d82134abfe'/>
<id>24f48382bc3ab62ca0979f1a6879f0d82134abfe</id>
<content type='text'>
https://github.com/ruby/prism/commit/5985ab7687
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
https://github.com/ruby/prism/commit/5985ab7687
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Fix token incompatibility for `Prism::Translation::Parser::Lexer`</title>
<updated>2024-03-16T17:55:38+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-03-16T06:21:36+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=3605d6076dab516c7b483a8be6038d5b6da1845a'/>
<id>3605d6076dab516c7b483a8be6038d5b6da1845a</id>
<content type='text'>
This PR fixes token incompatibility for `Prism::Translation::Parser::Lexer` when using backquoted heredoc indetiner:

```ruby
&lt;&lt;-`  FOO`
a
b
     FOO
```

## Parser gem (Expected)

Returns `tXSTRING_BEG` as the first token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr,
  s(:str, "a\n"),
  s(:str, "b\n")), [], [[:tXSTRING_BEG, ["&lt;&lt;`", #&lt;Parser::Source::Range example.rb 0...10&gt;]],
[:tSTRING_CONTENT, ["a\n", #&lt;Parser::Source::Range example.rb 11...13&gt;]],
[:tSTRING_CONTENT, ["b\n", #&lt;Parser::Source::Range example.rb 13...15&gt;]],
[:tSTRING_END, ["  FOO", #&lt;Parser::Source::Range example.rb 15...23&gt;]], [:tNL, [nil, #&lt;Parser::Source::Range example.rb 10...11&gt;]]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the tokens returned by the Parser gem were different. The escaped backslash does not match in the `tSTRING_BEG` token and
value of `tSTRING_END` token.

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr,
  s(:str, "a\n"),
  s(:str, "b\n")), [], [[:tSTRING_BEG, ["&lt;&lt;\"", #&lt;Parser::Source::Range example.rb 0...10&gt;]],
[:tSTRING_CONTENT, ["a\n", #&lt;Parser::Source::Range example.rb 11...13&gt;]],
[:tSTRING_CONTENT, ["b\n", #&lt;Parser::Source::Range example.rb 13...15&gt;]],
[:tSTRING_END, ["`  FOO`", #&lt;Parser::Source::Range example.rb 15...23&gt;]], [:tNL, [nil, #&lt;Parser::Source::Range example.rb 10...11&gt;]]]]
```

After this correction, the AST and tokens returned by the Parser gem are the same:

```console
$ bunlde exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr,
  s(:str, "a\n"),
  s(:str, "b\n")), [], [[:tXSTRING_BEG, ["&lt;&lt;`", #&lt;Parser::Source::Range example.rb 0...10&gt;]],
[:tSTRING_CONTENT, ["a\n", #&lt;Parser::Source::Range example.rb 11...13&gt;]],
[:tSTRING_CONTENT, ["b\n", #&lt;Parser::Source::Range example.rb 13...15&gt;]],
[:tSTRING_END, ["  FOO", #&lt;Parser::Source::Range example.rb 15...23&gt;]], [:tNL, [nil, #&lt;Parser::Source::Range example.rb 10...11&gt;]]]]
```

https://github.com/ruby/prism/commit/308f8d85a1
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This PR fixes token incompatibility for `Prism::Translation::Parser::Lexer` when using backquoted heredoc indetiner:

```ruby
&lt;&lt;-`  FOO`
a
b
     FOO
```

## Parser gem (Expected)

Returns `tXSTRING_BEG` as the first token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr,
  s(:str, "a\n"),
  s(:str, "b\n")), [], [[:tXSTRING_BEG, ["&lt;&lt;`", #&lt;Parser::Source::Range example.rb 0...10&gt;]],
[:tSTRING_CONTENT, ["a\n", #&lt;Parser::Source::Range example.rb 11...13&gt;]],
[:tSTRING_CONTENT, ["b\n", #&lt;Parser::Source::Range example.rb 13...15&gt;]],
[:tSTRING_END, ["  FOO", #&lt;Parser::Source::Range example.rb 15...23&gt;]], [:tNL, [nil, #&lt;Parser::Source::Range example.rb 10...11&gt;]]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the tokens returned by the Parser gem were different. The escaped backslash does not match in the `tSTRING_BEG` token and
value of `tSTRING_END` token.

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr,
  s(:str, "a\n"),
  s(:str, "b\n")), [], [[:tSTRING_BEG, ["&lt;&lt;\"", #&lt;Parser::Source::Range example.rb 0...10&gt;]],
[:tSTRING_CONTENT, ["a\n", #&lt;Parser::Source::Range example.rb 11...13&gt;]],
[:tSTRING_CONTENT, ["b\n", #&lt;Parser::Source::Range example.rb 13...15&gt;]],
[:tSTRING_END, ["`  FOO`", #&lt;Parser::Source::Range example.rb 15...23&gt;]], [:tNL, [nil, #&lt;Parser::Source::Range example.rb 10...11&gt;]]]]
```

After this correction, the AST and tokens returned by the Parser gem are the same:

```console
$ bunlde exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr,
  s(:str, "a\n"),
  s(:str, "b\n")), [], [[:tXSTRING_BEG, ["&lt;&lt;`", #&lt;Parser::Source::Range example.rb 0...10&gt;]],
[:tSTRING_CONTENT, ["a\n", #&lt;Parser::Source::Range example.rb 11...13&gt;]],
[:tSTRING_CONTENT, ["b\n", #&lt;Parser::Source::Range example.rb 13...15&gt;]],
[:tSTRING_END, ["  FOO", #&lt;Parser::Source::Range example.rb 15...23&gt;]], [:tNL, [nil, #&lt;Parser::Source::Range example.rb 10...11&gt;]]]]
```

https://github.com/ruby/prism/commit/308f8d85a1
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Fix token incompatibility for `Prism::Translation::Parser::Lexer`</title>
<updated>2024-03-15T18:08:39+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-03-15T16:20:58+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=e3a82d79fd727c90638ef697cb1b5ad73c7e62c0'/>
<id>e3a82d79fd727c90638ef697cb1b5ad73c7e62c0</id>
<content type='text'>
This PR fixes token incompatibility for `Prism::Translation::Parser::Lexer`
when using escaped backslash in string literal:

```ruby
"\\ foo \\ bar"
```

## Parser gem (Expected)

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #&lt;Parser::Source::Range example.rb 0...15&gt;]],
[:tNL, [nil, #&lt;Parser::Source::Range example.rb 15...16&gt;]]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the tokens returned by the Parser gem were different. The escaped backslash does not match in the `tSTRING` token:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\\\ foo \\\\ bar", #&lt;Parser::Source::Range example.rb 0...15&gt;]],
[:tNL, [nil, #&lt;Parser::Source::Range example.rb 15...16&gt;]]]]
```

After this correction, the AST and tokens returned by the Parser gem are the same:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #&lt;Parser::Source::Range example.rb 0...15&gt;]],
[:tNL, [nil, #&lt;Parser::Source::Range example.rb 15...16&gt;]]]]
```

The reproduction test is based on the following strings.txt and exists:
https://github.com/ruby/prism/blob/v0.24.0/test/prism/fixtures/strings.txt#L79

But, the restoration has not yet been performed due to remaining other issues in strings.txt.

https://github.com/ruby/prism/commit/2c44e7e307
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This PR fixes token incompatibility for `Prism::Translation::Parser::Lexer`
when using escaped backslash in string literal:

```ruby
"\\ foo \\ bar"
```

## Parser gem (Expected)

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #&lt;Parser::Source::Range example.rb 0...15&gt;]],
[:tNL, [nil, #&lt;Parser::Source::Range example.rb 15...16&gt;]]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the tokens returned by the Parser gem were different. The escaped backslash does not match in the `tSTRING` token:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\\\ foo \\\\ bar", #&lt;Parser::Source::Range example.rb 0...15&gt;]],
[:tNL, [nil, #&lt;Parser::Source::Range example.rb 15...16&gt;]]]]
```

After this correction, the AST and tokens returned by the Parser gem are the same:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #&lt;Parser::Source::Range example.rb 0...15&gt;]],
[:tNL, [nil, #&lt;Parser::Source::Range example.rb 15...16&gt;]]]]
```

The reproduction test is based on the following strings.txt and exists:
https://github.com/ruby/prism/blob/v0.24.0/test/prism/fixtures/strings.txt#L79

But, the restoration has not yet been performed due to remaining other issues in strings.txt.

https://github.com/ruby/prism/commit/2c44e7e307
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Fix a token incompatibility for `Prism::Translation::Parser::Lexer`</title>
<updated>2024-03-15T18:07:59+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-03-15T15:30:01+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=c9da8d67fdb9fab82f76d583239f5b9761f60350'/>
<id>c9da8d67fdb9fab82f76d583239f5b9761f60350</id>
<content type='text'>
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser`
for the heredocs_leading_whitespace.txt test.

https://github.com/ruby/prism/commit/7d45fb1eed
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser`
for the heredocs_leading_whitespace.txt test.

https://github.com/ruby/prism/commit/7d45fb1eed
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Fix an AST and token incompatibility for `Prism::Translation::Parser`</title>
<updated>2024-03-15T12:31:40+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-03-14T16:22:14+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=c0b8dee95a5412f395486a9bcb4959f93509cecb'/>
<id>c0b8dee95a5412f395486a9bcb4959f93509cecb</id>
<content type='text'>
This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
for dstring literal:

```ruby
"foo
  #{bar}"
```

## Parser gem (Expected)

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:dstr,
  s(:str, "foo\n"),
  s(:str, "  "),
  s(:begin,
    s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #&lt;Parser::Source::Range example.rb 0...1&gt;]],
[:tSTRING_CONTENT, ["foo\n", #&lt;Parser::Source::Range example.rb 1...5&gt;]],
[:tSTRING_CONTENT, ["  ", #&lt;Parser::Source::Range example.rb 5...7&gt;]],
[:tSTRING_DBEG, ["\#{", #&lt;Parser::Source::Range example.rb 7...9&gt;]],
[:tIDENTIFIER, ["bar", #&lt;Parser::Source::Range example.rb 9...12&gt;]],
[:tSTRING_DEND, ["}", #&lt;Parser::Source::Range example.rb 12...13&gt;]],
[:tSTRING_END, ["\"", #&lt;Parser::Source::Range example.rb 13...14&gt;]], [:tNL, [nil, #&lt;Parser::Source::Range example.rb 14...15&gt;]]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the AST and tokens returned by the Parser gem were different. In this case, `dstr` node should not be nested:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:dstr,
  s(:dstr,
    s(:str, "foo\n"),
    s(:str, "  ")),
  s(:begin,
    s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #&lt;Parser::Source::Range example.rb 0...1&gt;]],
[:tSTRING_CONTENT, ["foo\n", #&lt;Parser::Source::Range example.rb 1...5&gt;]],
[:tSTRING_CONTENT, ["  ", #&lt;Parser::Source::Range example.rb 5...7&gt;]],
[:tSTRING_DBEG, ["\#{", #&lt;Parser::Source::Range example.rb 7...9&gt;]],
[:tIDENTIFIER, ["bar", #&lt;Parser::Source::Range example.rb 9...12&gt;]],
[:tSTRING_DEND, ["}", #&lt;Parser::Source::Range example.rb 12...13&gt;]],
[:tSTRING_END, ["\"", #&lt;Parser::Source::Range example.rb 13...14&gt;]], [:tNL, [nil, #&lt;Parser::Source::Range example.rb 14...15&gt;]]]]
```

After this correction, the AST and tokens returned by the Parser gem are the same:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:dstr,
  s(:str, "foo\n"),
  s(:str, "  "),
  s(:begin,
    s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #&lt;Parser::Source::Range example.rb 0...1&gt;]],
[:tSTRING_CONTENT, ["foo\n", #&lt;Parser::Source::Range example.rb 1...5&gt;]],
[:tSTRING_CONTENT, ["  ", #&lt;Parser::Source::Range example.rb 5...7&gt;]],
[:tSTRING_DBEG, ["\#{", #&lt;Parser::Source::Range example.rb 7...9&gt;]],
[:tIDENTIFIER, ["bar", #&lt;Parser::Source::Range example.rb 9...12&gt;]],
[:tSTRING_DEND, ["}", #&lt;Parser::Source::Range example.rb 12...13&gt;]],
[:tSTRING_END, ["\"", #&lt;Parser::Source::Range example.rb 13...14&gt;]], [:tNL, [nil, #&lt;Parser::Source::Range example.rb 14...15&gt;]]]]
```

https://github.com/ruby/prism/commit/c1652a9ee7
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
for dstring literal:

```ruby
"foo
  #{bar}"
```

## Parser gem (Expected)

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:dstr,
  s(:str, "foo\n"),
  s(:str, "  "),
  s(:begin,
    s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #&lt;Parser::Source::Range example.rb 0...1&gt;]],
[:tSTRING_CONTENT, ["foo\n", #&lt;Parser::Source::Range example.rb 1...5&gt;]],
[:tSTRING_CONTENT, ["  ", #&lt;Parser::Source::Range example.rb 5...7&gt;]],
[:tSTRING_DBEG, ["\#{", #&lt;Parser::Source::Range example.rb 7...9&gt;]],
[:tIDENTIFIER, ["bar", #&lt;Parser::Source::Range example.rb 9...12&gt;]],
[:tSTRING_DEND, ["}", #&lt;Parser::Source::Range example.rb 12...13&gt;]],
[:tSTRING_END, ["\"", #&lt;Parser::Source::Range example.rb 13...14&gt;]], [:tNL, [nil, #&lt;Parser::Source::Range example.rb 14...15&gt;]]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the AST and tokens returned by the Parser gem were different. In this case, `dstr` node should not be nested:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:dstr,
  s(:dstr,
    s(:str, "foo\n"),
    s(:str, "  ")),
  s(:begin,
    s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #&lt;Parser::Source::Range example.rb 0...1&gt;]],
[:tSTRING_CONTENT, ["foo\n", #&lt;Parser::Source::Range example.rb 1...5&gt;]],
[:tSTRING_CONTENT, ["  ", #&lt;Parser::Source::Range example.rb 5...7&gt;]],
[:tSTRING_DBEG, ["\#{", #&lt;Parser::Source::Range example.rb 7...9&gt;]],
[:tIDENTIFIER, ["bar", #&lt;Parser::Source::Range example.rb 9...12&gt;]],
[:tSTRING_DEND, ["}", #&lt;Parser::Source::Range example.rb 12...13&gt;]],
[:tSTRING_END, ["\"", #&lt;Parser::Source::Range example.rb 13...14&gt;]], [:tNL, [nil, #&lt;Parser::Source::Range example.rb 14...15&gt;]]]]
```

After this correction, the AST and tokens returned by the Parser gem are the same:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:dstr,
  s(:str, "foo\n"),
  s(:str, "  "),
  s(:begin,
    s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #&lt;Parser::Source::Range example.rb 0...1&gt;]],
[:tSTRING_CONTENT, ["foo\n", #&lt;Parser::Source::Range example.rb 1...5&gt;]],
[:tSTRING_CONTENT, ["  ", #&lt;Parser::Source::Range example.rb 5...7&gt;]],
[:tSTRING_DBEG, ["\#{", #&lt;Parser::Source::Range example.rb 7...9&gt;]],
[:tIDENTIFIER, ["bar", #&lt;Parser::Source::Range example.rb 9...12&gt;]],
[:tSTRING_DEND, ["}", #&lt;Parser::Source::Range example.rb 12...13&gt;]],
[:tSTRING_END, ["\"", #&lt;Parser::Source::Range example.rb 13...14&gt;]], [:tNL, [nil, #&lt;Parser::Source::Range example.rb 14...15&gt;]]]]
```

https://github.com/ruby/prism/commit/c1652a9ee7
</pre>
</div>
</content>
</entry>
<entry>
<title>[ruby/prism] Fix an AST and token incompatibility for `Prism::Translation::Parser`</title>
<updated>2024-03-13T16:02:10+00:00</updated>
<author>
<name>Koichi ITO</name>
<email>koic.ito@gmail.com</email>
</author>
<published>2024-03-13T15:37:16+00:00</published>
<link rel='alternate' type='text/html' href='https://git.ruby-lang.org/ruby.git/commit/?id=0f076fa520999a42d88a081d7674495db2f65b3f'/>
<id>0f076fa520999a42d88a081d7674495db2f65b3f</id>
<content type='text'>
This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
for empty xstring literal.

## Parser gem (Expected)

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr), [], [[:tXSTRING_BEG, ["`", #&lt;Parser::Source::Range /tmp/s.rb 0...1&gt;]],
[:tSTRING_END, ["`", #&lt;Parser::Source::Range /tmp/s.rb 1...2&gt;]]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the AST and tokens returned by the Parser gem were different:

```console
$ bunele exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr, s(:str, "")), [], [[:tBACK_REF2, ["`", #&lt;Parser::Source::Range /tmp/s.rb 0...1&gt;]],
[:tSTRING_END, ["`", #&lt;Parser::Source::Range /tmp/s.rb 1...2&gt;]]]]
```

After this correction, the AST and tokens returned by the Parser gem are the same:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr), [], [[:tXSTRING_BEG, ["`", #&lt;Parser::Source::Range /tmp/s.rb 0...1&gt;]],
[:tSTRING_END, ["`", #&lt;Parser::Source::Range /tmp/s.rb 1...2&gt;]]]]
```

https://github.com/ruby/prism/commit/4ac89dcbb5
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
for empty xstring literal.

## Parser gem (Expected)

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr), [], [[:tXSTRING_BEG, ["`", #&lt;Parser::Source::Range /tmp/s.rb 0...1&gt;]],
[:tSTRING_END, ["`", #&lt;Parser::Source::Range /tmp/s.rb 1...2&gt;]]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the AST and tokens returned by the Parser gem were different:

```console
$ bunele exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr, s(:str, "")), [], [[:tBACK_REF2, ["`", #&lt;Parser::Source::Range /tmp/s.rb 0...1&gt;]],
[:tSTRING_END, ["`", #&lt;Parser::Source::Range /tmp/s.rb 1...2&gt;]]]]
```

After this correction, the AST and tokens returned by the Parser gem are the same:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
[s(:xstr), [], [[:tXSTRING_BEG, ["`", #&lt;Parser::Source::Range /tmp/s.rb 0...1&gt;]],
[:tSTRING_END, ["`", #&lt;Parser::Source::Range /tmp/s.rb 1...2&gt;]]]]
```

https://github.com/ruby/prism/commit/4ac89dcbb5
</pre>
</div>
</content>
</entry>
</feed>
