| Age | Commit message (Collapse) | Author |
|---|
|
|
|
The lexer did not jump to the `heredoc_end`, causing the heredoc end delimiter
to be parsed twice.
Normally the heredocs get flushed when a newline is encountered. But because
the newline is part of the string delimiter, that codepath is not taken.
Fixes [Bug #21758]
https://github.com/ruby/prism/commit/7440eb4b11
|
|
Mostly not having to list version-specific excludes when testing against ripper/parse.y
Also don't test new syntax additions against the parser gems. The version support
for them may (or may not) be expanded but we shouldn't bother while the ruby version
hasn't even released yet.
(ruby_parser translation is not versioned, so let as is for now)
I also removed excludes that have since been implemented by parse.y
https://github.com/ruby/prism/commit/e5a0221c37
|
|
https://github.com/ruby/prism/commit/b00d098f9a
|
|
See https://github.com/ruby/ruby/commit/6d81969b475262aba251e99b518181bdf7c5a523
It leaves the old variant around. RuboCop for examples accesses `Prism::Translation::Parser35`
to test against ruby-head. For now I left these simply as an alias
https://github.com/ruby/prism/commit/d0a823f045
|
|
Followup to https://github.com/ruby/prism/pull/2213
Before:
```sh
$ ruby -ve "puts 42.~@"
ruby 3.4.6 (2025-09-16 revision https://github.com/ruby/prism/commit/dbd83256b1) +PRISM [x86_64-linux]
-e:1:in '<main>': undefined method '~@' for an instance of Integer (NoMethodError)
Did you mean? ~
```
After (matches parse.y):
```sh
$ ./miniruby -ve "puts 42.~@"
ruby 3.5.0dev (2025-10-16T03:40:45Z master https://github.com/ruby/prism/commit/1d95d75c3f) +PRISM [x86_64-linux]
-43
```
https://github.com/ruby/prism/commit/a755bf228f
|
|
This one has been on my mind for a while now.
Currently, there are only tests against the latest syntax version.
This changes the snapshot structure as follows:
* Snapshots at their current location are tested against all syntax versions
* Snapshots inside a version folder like "3.3" are tested against all versions starting from that version
* Snapshots inside a version folder like "3.3-4.2" are tested against all versions in the given range.
This makes sure that as new syntax is added, older versions still work as expected.
I also added a few tests for now valid syntax that should be invalid in older versions (and the other way around as well)
These tests run really fast. So even though it does 3x the work for these, I am still able to run the whole test suite in just 11 seconds.
https://github.com/ruby/prism/commit/5191b1aa68
|
|
arguments
https://bugs.ruby-lang.org/issues/21168#note-5
The added code samples align with `parse.y`, except for `foo(bar baz do end)` which `parse.y` currently rejects but shouldn't.
https://github.com/ruby/prism/commit/3a4e102d80
|
|
This was a limitation of parse.y that prism intentionally replicated.
https://github.com/ruby/prism/commit/8fd12d594c
|
|
https://github.com/ruby/prism/commit/3f58fa7705
|
|
Generally I have been good about safely accessing the tokens but failed
to properly guard against no tokens in places
where it could theoretically happen through invalid syntax.
I added a test case for one occurance, other changes are theoretical only.
https://github.com/ruby/prism/commit/4a3866af19
|
|
https://github.com/ruby/prism/commit/a6b448b10f
|
|
Currently Prism returns `42` for code like this:
```ruby
42.tap { it = it; p it } # => 42
```
But parse.y returns `nil`:
```ruby
42.tap { it = it; p it } # => nil
```
In parse.y, it on the right-hand side is parsed as a local variable.
In Prism, it was parsed as the implicit block parameter it, which caused this inconsistent behavior.
This change makes the right-hand side it to be parsed as a local variable, aligning with parse.y's behavior.
Bug ticket: https://bugs.ruby-lang.org/issues/21139
https://github.com/ruby/prism/commit/cf3bbf9d2c
|
|
https://github.com/ruby/prism/commit/27d284bbb8
|
|
https://github.com/ruby/prism/commit/c2e372a8d8
|
|
It also updates to latest `parser`, which allows numbered
parameters in pattern matching pin,
passing `patterns.txt` and `case.txt`
https://github.com/ruby/prism/commit/bdcc8b3dc5
|
|
`StringNode` and `SymbolNode` don't have the same shape
(`content` vs `value`) and that wasn't handled.
I believe the logic for the common case can be reused.
I simply left the special handling for implicit nodes in pattern matching
and fall through otherwise.
NOTE: patterns.txt is not actually tested at the moment,
because it contains syntax that `parser` mistakenly rejects.
But I checked manually that this doesn't introduce other failures.
https://github.com/whitequark/parser/pull/1060
https://github.com/ruby/prism/commit/55adfaa895
|
|
It's not my favorite api but for users that currently use the same thing
from `parser`, moving over is more difficult
than it needs to be.
If you plan to support both old and new ruby versions, you definitly need to
branch somewhere on the ruby version
to either choose prism or parser.
But with prism you then need to enumerate all the versions again and choose the correct one.
Also, don't recommend to use `Prism::Translation::Parser` in docs. It's version-less
but actually always just uses Ruby 3.4 which is probably
not what the user intended.
Note: parser also warns when the patch version doesn't match what it expects. But I don't think prism has such a concept,
and anyways it would require releases anytime ruby releases, which I don't think is very desirable
https://github.com/ruby/prism/commit/77177f9e92
|
|
builder class
In https://github.com/ruby/prism/pull/3494 I added a bit of code
so that using the new builder doesn't break stuff.
This code can be dropped when it is enforced that builder
is _always_ the correct subclass (and makes future issues like that unlikely).
https://github.com/ruby/prism/commit/193d4b806d
|
|
This can get triggered even if the list of statements only contains
a single statement. This is necessary to properly support compiling
```ruby
defined? (;a)
defined? (a;)
```
as "expression". Previously these were parsed as statements lists
with single statements in them.
https://github.com/ruby/prism/commit/b63b5d67a9
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
https://github.com/ruby/prism/commit/422d5c4c64
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
Turns out, it was already almost correct. If you disregard \c and \M style escapes, only a single character is allowed to be escaped in a regex so most tests passed already.
There was also a mistake where the wrong value was constructed for the ast, this is now fixed.
One test fails because of this, but I'm fairly sure it is because of a parser bug. For `/\“/`, the backslash is supposed to be removed because it is a multibyte character. But tbh,
I don't entirely understand all the rules.
Fixes more than half of the remaining ast differences for rubocop tests
https://github.com/ruby/prism/commit/e1c75f304b
|
|
Also fixes a token incompatibility for the word separator. parser only considers whitespace until the first newline
https://github.com/ruby/prism/commit/bd3dd2b62a
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
## Summary
`itblock` node is added to support the `it` block parameter syntax introduced in Ruby 3.4.
```console
$ ruby -Ilib -rprism -rprism/translation/parser34 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { it }"; \
p Prism::Translation::Parser34.new.tokenize(buffer)[0]'
s(:itblock,
s(:send, nil, :proc), :it,
s(:lvar, :it))
```
This node design is similar to the `numblock` node, which was introduced for the numbered parameter syntax in Ruby 2.7.
```
$ ruby -Ilib -rprism -rprism/translation/parser34 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { _1 }"; \
p Prism::Translation::Parser34.new.tokenize(buffer)[0]'
s(:numblock,
s(:send, nil, :proc), 1,
s(:lvar, :_1))
```
The difference is that while numbered parameters can have multiple parameters, the `it` block parameter syntax allows only a single parameter.
In Ruby 3.3, the conventional node prior to the `it` block parameter syntax is returned.
```console
$ ruby -Ilib -rprism -rprism/translation/parser33 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { it }"; \
p Prism::Translation::Parser33.new.tokenize(buffer)[0]'
s(:block,
s(:send, nil, :proc),
s(:args),
s(:send, nil, :it))
```
## Development Note
The Parser gem does not yet support the `it` block parameter syntax. This is the first case where Prism's node design precedes that of the Parser gem.
When implementing https://github.com/whitequark/parser/issues/962, this node design will need to be taken into consideration.
https://github.com/ruby/prism/commit/c141e1420a
|
|
I want to add new node types to the parser translator, for example `itblock`. The bulk of the work is already done by prism itself. In the `parser`
builder, this would be a 5-line change at most but we don't control that here.
Instead, we can add our own builder and either overwrite the few methods we need,
or just inline the complete builder. I'm not sure yet which would be better.
`rubocop-ast` uses its own builder for `parser`. For this to correctly work, it must explicitly choose to extend the
prism builder and use it, same as it currently chooses to use a different parser when prism is used.
I'd like to enforce that the builder for prism extends its custom one since it will lead to
some pretty weird issues otherwise. But first, I'd like to change `rubocop-ast` to make use of this.
https://github.com/ruby/prism/commit/b080e608a8
|
|
Without a change from `parser`, this is impossible to correctly handle.
https://github.com/ruby/prism/commit/ca1d44e808
|
|
Turns out, it was already almost correct. If you disregard \c and \M style escapes, only a single character is allowed to be escaped in a regex so most tests passed already.
There was also a mistake where the wrong value was constructed for the ast, this is now fixed.
One test fails because of this, but I'm fairly sure it is because of a parser bug. For `/\“/`, the backslash is supposed to be removed because it is a multibyte character. But tbh,
I don't entirely understand all the rules.
Fixes more than half of the remaining ast differences for rubocop tests
|
|
When there were more actual tokens than expected tokens, the test would still pass.
Now it's possible to receive an assertion like this:
```
expected: []
actual: [:tNL, [nil, #<Parser::Source::Range comment_single.txt 8...9>]]
<[]> expected but was
<[:tNL, [nil, #<Parser::Source::Range comment_single.txt 8...9>]]>
```
https://github.com/ruby/prism/commit/e9e9a48e43
|
|
It falsely considered it to be a single backtick command
https://github.com/ruby/prism/commit/dd762be590
|
|
```rb
<<~F
foo
#{}
bar
F
```
has zero common whitespace.
https://github.com/ruby/prism/commit/1f3c222a06
|
|
Tests worked around this but the incompatibility is not hard to fix.
This fixes 17 token incompatibilies in tests here that were previously passing
https://github.com/ruby/prism/commit/101962526d
|
|
Skipping detecting the encoding is almost always right, just for binary it should actually happen.
A symbol containing escapes that are invalid
in utf-8 would fail to parse since symbols must be valid in the script encoding.
Additionally, the parser gem would raise an exception somewhere during string handling
https://github.com/ruby/prism/commit/fa0154d9e4
|
|
This leaves `\c` and `\M` escaping but I don't understand how these should even work yet. Maybe later.
https://github.com/ruby/prism/commit/13db3e8cb9
|
|
Slightly tweaking the import script becaues of backtrace format changes in Ruby 3.4
Most tests pass in all parsers, with only a handful of failures overall
https://github.com/ruby/prism/commit/9b5b785aa4
|
|
translator
Much of this logic should be shared between interpolated symbols and regexps.
It's also incorrect when the node contains a literal `\\n` (same as for plain string nodes at the moment)
https://github.com/ruby/prism/commit/561914f99b
|
|
Turns out, the vast majority of work was already done with handling the same for heredocs
I'm confident this should also apply to actual string nodes (there's even a todo for it) but
no tests change if I apply it there too, so I can't say for sure if the logic would be correct.
The individual test files are a bit too large, maybe something else would break that currently passes.
Leaving it for later to look more closely into that.
https://github.com/ruby/prism/commit/6bba1c54e1
|
|
That issue is exactly about what this test file contains:
A single-quoted heredocs with backslashes
https://github.com/ruby/prism/commit/4820a44c7b
|
|
Heredocs that contain "\\n" don't start a new string node.
https://github.com/ruby/prism/commit/61d9d3a15e
|
|
`Prism::Translation::Parser::Lexer`
## Summary
This PR fixes `kDO_LAMBDA` token incompatibility between Parser gem and `Prism::Translation::Parser` for lambda `do` block.
### Parser gem (Expected)
Returns `kDO_LAMBDA` token:
```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 3...5>]],
[:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]]
```
### `Prism::Translation::Parser` (Actual)
Previously, the parser returned `kDO` token when parsing the following:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO, ["do", #<Parser::Source::Range example.rb 3...5>]],
[:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]]
```
After the update, the parser now returns `kDO_LAMBDA` token for the same input:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 3...5>]],
[:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]]
```
## Additional Information
Unfortunately, this kind of edge case doesn't work as expected; `kDO` is returned instead of `kDO_LAMBDA`.
However, since `kDO` is already being returned in this case, there is no change in behavior.
### Parser gem
Returns `tLAMBDA` token:
```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> (foo = -> (bar) {}) do end"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.3.5 (2024-09-03 revision https://github.com/ruby/prism/commit/ef084cc8f4) [x86_64-darwin23]
[[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]],
[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 4...7>]], [:tEQL, ["=", #<Parser::Source::Range example.rb 8...9>]],
[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 10...12>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 13...14>]],
[:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 14...17>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 17...18>]],
[:tLAMBEG, ["{", #<Parser::Source::Range example.rb 19...20>]], [:tRCURLY, ["}", #<Parser::Source::Range example.rb 20...21>]],
[:tRPAREN, [")", #<Parser::Source::Range example.rb 21...22>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 23...25>]],
[:kEND, ["end", #<Parser::Source::Range example.rb 26...29>]]]
```
### `Prism::Translation::Parser`
Returns `kDO` token:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> (foo = -> (bar) {}) do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.5 (2024-09-03 revision https://github.com/ruby/prism/commit/ef084cc8f4) [x86_64-darwin23]
[[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]],
[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 4...7>]], [:tEQL, ["=", #<Parser::Source::Range example.rb 8...9>]],
[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 10...12>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 13...14>]],
[:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 14...17>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 17...18>]],
[:tLAMBEG, ["{", #<Parser::Source::Range example.rb 19...20>]], [:tRCURLY, ["}", #<Parser::Source::Range example.rb 20...21>]],
[:tRPAREN, [")", #<Parser::Source::Range example.rb 21...22>]], [:kDO, ["do", #<Parser::Source::Range example.rb 23...25>]],
[:kEND, ["end", #<Parser::Source::Range example.rb 26...29>]]]
```
As the intention is not to address such special cases at this point, a comment has been left indicating that this case still returns `kDO`.
In other words, `kDO_LAMBDA` will now be returned except for edge cases after this PR.
https://github.com/ruby/prism/commit/2ee480654c
|
|
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for double splat argument.
## Parser gem (Expected)
Returns `tDSTAR` token:
```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]],
[:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tDSTAR, ["**", #<Parser::Source::Range example.rb 6...8>]],
[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]],
[:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]]
```
## `Prism::Translation::Parser` (Actual)
Previously, the parser returned `tPOW` token when parsing the following:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]],
[:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tPOW, ["**", #<Parser::Source::Range example.rb 6...8>]],
[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]],
[:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]]
```
After the update, the parser now returns `tDSTAR` token for the same input:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def f(**foo) end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:tIDENTIFIER, ["f", #<Parser::Source::Range example.rb 4...5>]],
[:tLPAREN2, ["(", #<Parser::Source::Range example.rb 5...6>]], [:tDSTAR, ["**", #<Parser::Source::Range example.rb 6...8>]],
[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 8...11>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 11...12>]],
[:kEND, ["end", #<Parser::Source::Range example.rb 13...16>]]]
```
With this change, the following code could be removed from test/prism/ruby/parser_test.rb:
```diff
- when :tPOW
- actual_token[0] = expected_token[0] if expected_token[0] == :tDSTAR
```
`tPOW` is the token type for the behavior of `a ** b`, and its behavior remains unchanged:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "a ** b"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["a", #<Parser::Source::Range example.rb 0...1>]], [:tPOW, ["**", #<Parser::Source::Range example.rb 2...4>]],
[:tIDENTIFIER, ["b", #<Parser::Source::Range example.rb 5...6>]]]
```
https://github.com/ruby/prism/commit/66bde35a44
|
|
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for left parenthesis.
## Parser gem (Expected)
Returns `tLPAREN2` token:
```console
$ bundle exec ruby -Ilib -rparser/ruby33 \
-ve 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]],
[:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]]
```
## `Prism::Translation::Parser` (Actual)
Previously, the parser returned `tLPAREN` token when parsing the following:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN, ["(", #<Parser::Source::Range example.rb 3...4>]],
[:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]]
```
After the update, the parser now returns `tLPAREN2` token for the same input:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "foo(:bar)"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master https://github.com/ruby/prism/commit/eb144ef91e) [x86_64-darwin23]
[[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 0...3>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]],
[:tSYMBOL, ["bar", #<Parser::Source::Range example.rb 4...8>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 8...9>]]]
```
The `PARENTHESIS_LEFT` token in Prism is classified as either `tLPAREN` or `tLPAREN2` in the Parser gem.
The tokens that were previously all classified as `tLPAREN` are now also classified to `tLPAREN2`.
With this change, the following code could be removed from `test/prism/ruby/parser_test.rb`:
```diff
- when :tLPAREN
- actual_token[0] = expected_token[0] if expected_token[0] == :tLPAREN2
```
https://github.com/ruby/prism/commit/04d6f3478d
|
|
https://github.com/ruby/prism/commit/9e4fb665ee
|
|
https://github.com/ruby/prism/commit/5985ab7687
|
|
https://github.com/ruby/prism/commit/f7faedfb3f
|
|
https://github.com/ruby/prism/commit/1528d3c019
|
|
https://github.com/ruby/prism/commit/d218e65561
|
|
https://github.com/ruby/prism/commit/100340bc6b
|
