| Age | Commit message (Collapse) | Author |
|---|
|
Prism::Translation::Ripper"
This reverts commit 58f1127b51cf4fbb1f334f8701a041f40701dca2.
|
|
Prism::Translation::Ripper"
This reverts commit 35a7b5159f39de2cac848c072674e5350cc41aa4.
This broke syntax_suggest.
https://github.com/ruby/ruby/actions/runs/21167011751/job/60874111912
|
|
Prism::Translation::Ripper
* Handle line continuations.
* Handle space at the end of file in LexCompat.
https://github.com/ruby/prism/commit/32bd13eb7d
Co-authored-by: Earlopain <14981592+Earlopain@users.noreply.github.com>
|
|
accesses to offsets
https://github.com/ruby/prism/commit/ff81a29ba5
|
|
The filter class is a 1:1 copy of ruby.
rdoc has 32 test failures. It seems to expect `on_sp` in some cases to render code as written.
https://github.com/ruby/prism/commit/74bb12c825
|
|
It's public API and trivial to implement.
https://github.com/ruby/prism/commit/e77545f8b5
|
|
This is everything that `irb` uses. It works in their test-suite, but there are 20 failures when using the shim that I haven't looked into at all.
`parse` is not used by `irb`. `scan` is, and it's basically `parse` but also including errors. `irb` doesn't seem to care about the errors, so I didn't implement that.
https://github.com/ruby/prism/commit/2c5826b39f
|
|
It is for example used by `irb`, `rdoc`, `syntax_suggest`
https://github.com/ruby/prism/commit/255aeb2485
|
|
One per version seems excessive.
Do note that `rubocop-ast` used to require individual parser files. I wouldn't consider that to be part of the API since everything is autoloaded.
From a GitHub code search, I didn't find anyone else doing it like that.
https://github.com/ruby/prism/commit/458f622c34
|
|
Ripper exposes Ripper::Lexer:State in its output, which is a bit of a problem. To make this work, I basically copy-pasted the implementation.
I'm unsure if that is acceptable and added a test to make sure that these values never go out of sync.
I don't imagine them changing often, prism maps them 1:1 for its own usage.
This also fixed the shim by accident. `Ripper.lex` went to `Translation::Ripper.lex` when it should have been the original. Removing the need for the original resolves that issue.
https://github.com/ruby/prism/commit/2c0bea076d
|
|
Ripper is either not used or loaded where it is actually needed
https://github.com/ruby/prism/commit/a73a4fb00c
|
|
* assert_raise's 2nd argument is the failure message,
shown when the expected exception is not raised.
It's not the expected message.
See https://github.com/test-unit/test-unit/issues/347
https://github.com/ruby/prism/commit/e3df994d47
|
|
The lexer did not jump to the `heredoc_end`, causing the heredoc end delimiter
to be parsed twice.
Normally the heredocs get flushed when a newline is encountered. But because
the newline is part of the string delimiter, that codepath is not taken.
Fixes [Bug #21758]
https://github.com/ruby/prism/commit/7440eb4b11
|
|
Mostly not having to list version-specific excludes when testing against ripper/parse.y
Also don't test new syntax additions against the parser gems. The version support
for them may (or may not) be expanded but we shouldn't bother while the ruby version
hasn't even released yet.
(ruby_parser translation is not versioned, so let as is for now)
I also removed excludes that have since been implemented by parse.y
https://github.com/ruby/prism/commit/e5a0221c37
|
|
https://github.com/ruby/prism/commit/b00d098f9a
|
|
See https://github.com/ruby/ruby/commit/6d81969b475262aba251e99b518181bdf7c5a523
It leaves the old variant around. RuboCop for examples accesses `Prism::Translation::Parser35`
to test against ruby-head. For now I left these simply as an alias
https://github.com/ruby/prism/commit/d0a823f045
|
|
Followup to https://github.com/ruby/prism/pull/2213
Before:
```sh
$ ruby -ve "puts 42.~@"
ruby 3.4.6 (2025-09-16 revision https://github.com/ruby/prism/commit/dbd83256b1) +PRISM [x86_64-linux]
-e:1:in '<main>': undefined method '~@' for an instance of Integer (NoMethodError)
Did you mean? ~
```
After (matches parse.y):
```sh
$ ./miniruby -ve "puts 42.~@"
ruby 3.5.0dev (2025-10-16T03:40:45Z master https://github.com/ruby/prism/commit/1d95d75c3f) +PRISM [x86_64-linux]
-43
```
https://github.com/ruby/prism/commit/a755bf228f
|
|
This one has been on my mind for a while now.
Currently, there are only tests against the latest syntax version.
This changes the snapshot structure as follows:
* Snapshots at their current location are tested against all syntax versions
* Snapshots inside a version folder like "3.3" are tested against all versions starting from that version
* Snapshots inside a version folder like "3.3-4.2" are tested against all versions in the given range.
This makes sure that as new syntax is added, older versions still work as expected.
I also added a few tests for now valid syntax that should be invalid in older versions (and the other way around as well)
These tests run really fast. So even though it does 3x the work for these, I am still able to run the whole test suite in just 11 seconds.
https://github.com/ruby/prism/commit/5191b1aa68
|
|
arguments
https://bugs.ruby-lang.org/issues/21168#note-5
The added code samples align with `parse.y`, except for `foo(bar baz do end)` which `parse.y` currently rejects but shouldn't.
https://github.com/ruby/prism/commit/3a4e102d80
|
|
If a character literal was followed by a string concatenation, then
the forced encoding of the string concatenation could accidentally
overwrite the explicit encoding of the character literal. We now
handle this properly.
https://github.com/ruby/prism/commit/125c375d74
|
|
This was a limitation of parse.y that prism intentionally replicated.
https://github.com/ruby/prism/commit/8fd12d594c
|
|
https://github.com/ruby/prism/commit/3f58fa7705
|
|
Generally I have been good about safely accessing the tokens but failed
to properly guard against no tokens in places
where it could theoretically happen through invalid syntax.
I added a test case for one occurance, other changes are theoretical only.
https://github.com/ruby/prism/commit/4a3866af19
|
|
https://github.com/ruby/prism/commit/a6b448b10f
|
|
Currently Prism returns `42` for code like this:
```ruby
42.tap { it = it; p it } # => 42
```
But parse.y returns `nil`:
```ruby
42.tap { it = it; p it } # => nil
```
In parse.y, it on the right-hand side is parsed as a local variable.
In Prism, it was parsed as the implicit block parameter it, which caused this inconsistent behavior.
This change makes the right-hand side it to be parsed as a local variable, aligning with parse.y's behavior.
Bug ticket: https://bugs.ruby-lang.org/issues/21139
https://github.com/ruby/prism/commit/cf3bbf9d2c
|
|
https://github.com/ruby/prism/commit/27d284bbb8
|
|
https://github.com/ruby/prism/commit/7362b114a3
|
|
https://github.com/ruby/prism/commit/c2e372a8d8
|
|
It also updates to latest `parser`, which allows numbered
parameters in pattern matching pin,
passing `patterns.txt` and `case.txt`
https://github.com/ruby/prism/commit/bdcc8b3dc5
|
|
`StringNode` and `SymbolNode` don't have the same shape
(`content` vs `value`) and that wasn't handled.
I believe the logic for the common case can be reused.
I simply left the special handling for implicit nodes in pattern matching
and fall through otherwise.
NOTE: patterns.txt is not actually tested at the moment,
because it contains syntax that `parser` mistakenly rejects.
But I checked manually that this doesn't introduce other failures.
https://github.com/whitequark/parser/pull/1060
https://github.com/ruby/prism/commit/55adfaa895
|
|
It's not my favorite api but for users that currently use the same thing
from `parser`, moving over is more difficult
than it needs to be.
If you plan to support both old and new ruby versions, you definitly need to
branch somewhere on the ruby version
to either choose prism or parser.
But with prism you then need to enumerate all the versions again and choose the correct one.
Also, don't recommend to use `Prism::Translation::Parser` in docs. It's version-less
but actually always just uses Ruby 3.4 which is probably
not what the user intended.
Note: parser also warns when the patch version doesn't match what it expects. But I don't think prism has such a concept,
and anyways it would require releases anytime ruby releases, which I don't think is very desirable
https://github.com/ruby/prism/commit/77177f9e92
|
|
Instead of requiring the consumer to provide a list of all events which
they wish to handle, we can give them to option of dynamically detecting
them, by scanning the listener's public methods.
This approach is similar to that used by Minitest (scanning for `test_`
methods) and Rails generators (running all public methods in the order
they are defined).
While this is slower than specifying a hard coded list, the penalty is
only during registration. There is no change the the behaviour of
dispatching the events.
https://github.com/ruby/prism/commit/781ebed743
|
|
builder class
In https://github.com/ruby/prism/pull/3494 I added a bit of code
so that using the new builder doesn't break stuff.
This code can be dropped when it is enforced that builder
is _always_ the correct subclass (and makes future issues like that unlikely).
https://github.com/ruby/prism/commit/193d4b806d
|
|
https://github.com/ruby/prism/commit/2afe89f8ce
|
|
This can get triggered even if the list of statements only contains
a single statement. This is necessary to properly support compiling
```ruby
defined? (;a)
defined? (a;)
```
as "expression". Previously these were parsed as statements lists
with single statements in them.
https://github.com/ruby/prism/commit/b63b5d67a9
|
|
In https://github.com/ruby/prism/commit/26370079291a420c6b2b7be5cdbd5c609da62f21 I added tests but didn't modify them correctly
https://github.com/ruby/prism/commit/de021e74de
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
https://github.com/ruby/prism/commit/422d5c4c64
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
Turns out, it was already almost correct. If you disregard \c and \M style escapes, only a single character is allowed to be escaped in a regex so most tests passed already.
There was also a mistake where the wrong value was constructed for the ast, this is now fixed.
One test fails because of this, but I'm fairly sure it is because of a parser bug. For `/\“/`, the backslash is supposed to be removed because it is a multibyte character. But tbh,
I don't entirely understand all the rules.
Fixes more than half of the remaining ast differences for rubocop tests
https://github.com/ruby/prism/commit/e1c75f304b
|
|
Also fixes a token incompatibility for the word separator. parser only considers whitespace until the first newline
https://github.com/ruby/prism/commit/bd3dd2b62a
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
1. The string starts out as binary
2. `ち` is appended, forcing it back into utf-8
3. Some invalid byte sequences are tried to append
> incompatible character encodings: UTF-8 and BINARY (ASCII-8BIT)
This makes use of my wish to use `append_as_bytes`. Unfortunatly that method is rather new
so it needs a fallback
https://github.com/ruby/prism/commit/e31e94a775
|
|
## Summary
`itblock` node is added to support the `it` block parameter syntax introduced in Ruby 3.4.
```console
$ ruby -Ilib -rprism -rprism/translation/parser34 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { it }"; \
p Prism::Translation::Parser34.new.tokenize(buffer)[0]'
s(:itblock,
s(:send, nil, :proc), :it,
s(:lvar, :it))
```
This node design is similar to the `numblock` node, which was introduced for the numbered parameter syntax in Ruby 2.7.
```
$ ruby -Ilib -rprism -rprism/translation/parser34 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { _1 }"; \
p Prism::Translation::Parser34.new.tokenize(buffer)[0]'
s(:numblock,
s(:send, nil, :proc), 1,
s(:lvar, :_1))
```
The difference is that while numbered parameters can have multiple parameters, the `it` block parameter syntax allows only a single parameter.
In Ruby 3.3, the conventional node prior to the `it` block parameter syntax is returned.
```console
$ ruby -Ilib -rprism -rprism/translation/parser33 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { it }"; \
p Prism::Translation::Parser33.new.tokenize(buffer)[0]'
s(:block,
s(:send, nil, :proc),
s(:args),
s(:send, nil, :it))
```
## Development Note
The Parser gem does not yet support the `it` block parameter syntax. This is the first case where Prism's node design precedes that of the Parser gem.
When implementing https://github.com/whitequark/parser/issues/962, this node design will need to be taken into consideration.
https://github.com/ruby/prism/commit/c141e1420a
|
|
I want to add new node types to the parser translator, for example `itblock`. The bulk of the work is already done by prism itself. In the `parser`
builder, this would be a 5-line change at most but we don't control that here.
Instead, we can add our own builder and either overwrite the few methods we need,
or just inline the complete builder. I'm not sure yet which would be better.
`rubocop-ast` uses its own builder for `parser`. For this to correctly work, it must explicitly choose to extend the
prism builder and use it, same as it currently chooses to use a different parser when prism is used.
I'd like to enforce that the builder for prism extends its custom one since it will lead to
some pretty weird issues otherwise. But first, I'd like to change `rubocop-ast` to make use of this.
https://github.com/ruby/prism/commit/b080e608a8
|
|
Without a change from `parser`, this is impossible to correctly handle.
https://github.com/ruby/prism/commit/ca1d44e808
|
|
Turns out, it was already almost correct. If you disregard \c and \M style escapes, only a single character is allowed to be escaped in a regex so most tests passed already.
There was also a mistake where the wrong value was constructed for the ast, this is now fixed.
One test fails because of this, but I'm fairly sure it is because of a parser bug. For `/\“/`, the backslash is supposed to be removed because it is a multibyte character. But tbh,
I don't entirely understand all the rules.
Fixes more than half of the remaining ast differences for rubocop tests
|
|
When there were more actual tokens than expected tokens, the test would still pass.
Now it's possible to receive an assertion like this:
```
expected: []
actual: [:tNL, [nil, #<Parser::Source::Range comment_single.txt 8...9>]]
<[]> expected but was
<[:tNL, [nil, #<Parser::Source::Range comment_single.txt 8...9>]]>
```
https://github.com/ruby/prism/commit/e9e9a48e43
|
|
It falsely considered it to be a single backtick command
https://github.com/ruby/prism/commit/dd762be590
|
|
```rb
<<~F
foo
#{}
bar
F
```
has zero common whitespace.
https://github.com/ruby/prism/commit/1f3c222a06
|
