| Age | Commit message (Collapse) | Author |
|---|
|
There will be a bunch of other problems should 3.10 ever exists, but I guess why not fix this one now.
https://github.com/ruby/prism/commit/b385f47f8b
|
|
It's not my favorite api but for users that currently use the same thing
from `parser`, moving over is more difficult
than it needs to be.
If you plan to support both old and new ruby versions, you definitly need to
branch somewhere on the ruby version
to either choose prism or parser.
But with prism you then need to enumerate all the versions again and choose the correct one.
Also, don't recommend to use `Prism::Translation::Parser` in docs. It's version-less
but actually always just uses Ruby 3.4 which is probably
not what the user intended.
Note: parser also warns when the patch version doesn't match what it expects. But I don't think prism has such a concept,
and anyways it would require releases anytime ruby releases, which I don't think is very desirable
https://github.com/ruby/prism/commit/77177f9e92
|
|
https://github.com/ruby/prism/commit/c02429765b
|
|
https://github.com/ruby/prism/commit/d85c72a1b9
|
|
`category` is only supported from Ruby 3.0 onwards and prism can still run with Ruyb 2.7
https://github.com/ruby/prism/commit/335a193851
|
|
builder class
In https://github.com/ruby/prism/pull/3494 I added a bit of code
so that using the new builder doesn't break stuff.
This code can be dropped when it is enforced that builder
is _always_ the correct subclass (and makes future issues like that unlikely).
https://github.com/ruby/prism/commit/193d4b806d
|
|
https://github.com/ruby/prism/commit/71d31db496
|
|
|
|
Caused by https://github.com/ruby/prism/pull/3478 and https://github.com/ruby/prism/pull/3443
I also made the builder reference more explicit to clearly distinquish
between `::Parser` and `Prism::Translation::Parser`
https://github.com/ruby/prism/commit/d52aaa75b6
|
|
```
(a,), = []
PARSER====================
s(:masgn,
s(:mlhs,
s(:mlhs,
s(:lvasgn, :a))),
s(:array))
PRISM====================
s(:masgn,
s(:mlhs,
s(:lvasgn, :a)),
s(:array))
```
https://github.com/ruby/prism/commit/8aa1f4690e
|
|
In https://github.com/ruby/prism/commit/26370079291a420c6b2b7be5cdbd5c609da62f21 I added tests but didn't modify them correctly
https://github.com/ruby/prism/commit/de021e74de
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
https://github.com/ruby/prism/commit/422d5c4c64
|
|
I see `Array.include?` as 2.4% runtime. Probably because of `LPAREN_CONVERSION_TOKEN_TYPES` but
the others will be faster as well.
Also remove some inline array checks. They are specifically optimized in Ruby since 3.4, but for now prism is for >= 2.7
https://github.com/ruby/prism/commit/ca9500a3fc
|
|
`Integer#chr` performs some validation that we don't want/need. Octal escapes can go above 255, where it will then raise trying to convert.
`append_as_bytes` actually allows to pass a number, so we can just skip that call.
Although, on older rubies of course we still need to handle this in the polyfill.
I don't really like using `pack` but don't know of another way to do so.
For the utf-8 escapes, this is not an issue. Invalid utf-8 in these is simply a syntax error.
https://github.com/ruby/prism/commit/161c606b1f
|
|
https://github.com/ruby/prism/commit/09c59a3aa5
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
Turns out, it was already almost correct. If you disregard \c and \M style escapes, only a single character is allowed to be escaped in a regex so most tests passed already.
There was also a mistake where the wrong value was constructed for the ast, this is now fixed.
One test fails because of this, but I'm fairly sure it is because of a parser bug. For `/\“/`, the backslash is supposed to be removed because it is a multibyte character. But tbh,
I don't entirely understand all the rules.
Fixes more than half of the remaining ast differences for rubocop tests
https://github.com/ruby/prism/commit/e1c75f304b
|
|
Also fixes a token incompatibility for the word separator. parser only considers whitespace until the first newline
https://github.com/ruby/prism/commit/bd3dd2b62a
|
|
When the line contains no real newline but contains unescaped ones, then there will be one less entry
https://github.com/ruby/prism/commit/4ef093b600
|
|
translator
This is a followup to #3373, where the implementation
was extracted
https://github.com/ruby/prism/commit/2637007929
|
|
The offset cache contains an entry for each byte so it can't be accessed via the string length.
Adds tests for all variants except for this:
```
"fo
o" "ba
’"
```
For some reason, this still has the wrong offset.
https://github.com/ruby/prism/commit/a651126458
|
|
There are a few other locations that should be included in that check.
I think the end location must always be present but I left it in to be safe (maybe implicit begin somehow?)
https://github.com/ruby/prism/commit/545d07ddc3
|
|
`Integer#chr` performs some validation that we don't want/need. Octal escapes can go above 255, where it will then raise trying to convert.
`append_as_bytes` actually allows to pass a number, so we can just skip that call.
Although, on older rubies of course we still need to handle this in the polyfill.
I don't really like using `pack` but don't know of another way to do so.
For the utf-8 escapes, this is not an issue. Invalid utf-8 in these is simply a syntax error.
https://github.com/ruby/prism/commit/161c606b1f
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
I want to add new node types to the parser translator, for example `itblock`. The bulk of the work is already done by prism itself. In the `parser`
builder, this would be a 5-line change at most but we don't control that here.
Instead, we can add our own builder and either overwrite the few methods we need,
or just inline the complete builder. I'm not sure yet which would be better.
`rubocop-ast` uses its own builder for `parser`. For this to correctly work, it must explicitly choose to extend the
prism builder and use it, same as it currently chooses to use a different parser when prism is used.
I'd like to enforce that the builder for prism extends its custom one since it will lead to
some pretty weird issues otherwise. But first, I'd like to change `rubocop-ast` to make use of this.
https://github.com/ruby/prism/commit/b080e608a8
|
|
1. The string starts out as binary
2. `ち` is appended, forcing it back into utf-8
3. Some invalid byte sequences are tried to append
> incompatible character encodings: UTF-8 and BINARY (ASCII-8BIT)
This makes use of my wish to use `append_as_bytes`. Unfortunatly that method is rather new
so it needs a fallback
https://github.com/ruby/prism/commit/e31e94a775
|
|
https://github.com/ruby/prism/commit/422d5c4c64
|
|
Avoids an array allocation which matters more and more
the larger the file is.
I have it at 14% of runtime.
https://github.com/ruby/prism/commit/f65b90f27d
|
|
I see `Array.include?` as 2.4% runtime. Probably because of `LPAREN_CONVERSION_TOKEN_TYPES` but
the others will be faster as well.
Also remove some inline array checks. They are specifically optimized in Ruby since 3.4, but for now prism is for >= 2.7
https://github.com/ruby/prism/commit/ca9500a3fc
|
|
Temoprary backwards-compat code so that current users
don't break.
Eventually the Translation::Parser initializer should asser that the correct class is passed in.
https://github.com/ruby/prism/commit/66b0162b35
|
|
https://github.com/ruby/prism/commit/56eaf53732
|
|
https://github.com/ruby/prism/commit/10e5431b38
|
|
## Summary
`itblock` node is added to support the `it` block parameter syntax introduced in Ruby 3.4.
```console
$ ruby -Ilib -rprism -rprism/translation/parser34 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { it }"; \
p Prism::Translation::Parser34.new.tokenize(buffer)[0]'
s(:itblock,
s(:send, nil, :proc), :it,
s(:lvar, :it))
```
This node design is similar to the `numblock` node, which was introduced for the numbered parameter syntax in Ruby 2.7.
```
$ ruby -Ilib -rprism -rprism/translation/parser34 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { _1 }"; \
p Prism::Translation::Parser34.new.tokenize(buffer)[0]'
s(:numblock,
s(:send, nil, :proc), 1,
s(:lvar, :_1))
```
The difference is that while numbered parameters can have multiple parameters, the `it` block parameter syntax allows only a single parameter.
In Ruby 3.3, the conventional node prior to the `it` block parameter syntax is returned.
```console
$ ruby -Ilib -rprism -rprism/translation/parser33 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { it }"; \
p Prism::Translation::Parser33.new.tokenize(buffer)[0]'
s(:block,
s(:send, nil, :proc),
s(:args),
s(:send, nil, :it))
```
## Development Note
The Parser gem does not yet support the `it` block parameter syntax. This is the first case where Prism's node design precedes that of the Parser gem.
When implementing https://github.com/whitequark/parser/issues/962, this node design will need to be taken into consideration.
https://github.com/ruby/prism/commit/c141e1420a
|
|
This restores the missing method comments in https://github.com/ruby/prism/pull/3479.
https://github.com/ruby/prism/commit/78b8f67dee
|
|
Caused by https://github.com/ruby/prism/pull/3478 and https://github.com/ruby/prism/pull/3443
I also made the builder reference more explicit to clearly distinquish
between `::Parser` and `Prism::Translation::Parser`
https://github.com/ruby/prism/commit/d52aaa75b6
|
|
I want to add new node types to the parser translator, for example `itblock`. The bulk of the work is already done by prism itself. In the `parser`
builder, this would be a 5-line change at most but we don't control that here.
Instead, we can add our own builder and either overwrite the few methods we need,
or just inline the complete builder. I'm not sure yet which would be better.
`rubocop-ast` uses its own builder for `parser`. For this to correctly work, it must explicitly choose to extend the
prism builder and use it, same as it currently chooses to use a different parser when prism is used.
I'd like to enforce that the builder for prism extends its custom one since it will lead to
some pretty weird issues otherwise. But first, I'd like to change `rubocop-ast` to make use of this.
https://github.com/ruby/prism/commit/b080e608a8
|
|
Follow-up to https://github.com/Shopify/ruby-lsp/pull/1849.
This is an extension of `Prism::Translation::Parser` to implement https://github.com/Shopify/ruby-lsp/pull/1849.
It is based on the comments in https://github.com/Shopify/ruby-lsp/pull/1849#pullrequestreview-1966020868,
but also adds a default argument for delegation to `Parser::Base` super class.
Using this API, https://github.com/rubocop/rubocop-ast/pull/359 has been implemented in RuboCop AST.
As detailed in https://github.com/rubocop/rubocop-ast/pull/359, this change is expected to improve performance by 1.3x
for some source code.
Achieving a 1.3x speedup with such this simple modification is a significant improvement for Ruby LSP and its users.
https://github.com/ruby/prism/commit/925725291c
|
|
Fixes [Bug #21117]
https://github.com/ruby/prism/commit/19d4bab5a0
|
|
`builder.pair_label` is no good since it makes use of variables that the parser gem encountered.
Since the prism translator doesn't keep proper track of that information, the following code interprets
the implicit value as a local variable, even though it is not in scope:
```rb
def foo
bar = 123
end
{ bar: }
```
https://github.com/ruby/prism/commit/bbeb5b083a
|
|
Turns out, it was already almost correct. If you disregard \c and \M style escapes, only a single character is allowed to be escaped in a regex so most tests passed already.
There was also a mistake where the wrong value was constructed for the ast, this is now fixed.
One test fails because of this, but I'm fairly sure it is because of a parser bug. For `/\“/`, the backslash is supposed to be removed because it is a multibyte character. But tbh,
I don't entirely understand all the rules.
Fixes more than half of the remaining ast differences for rubocop tests
|
|
When parent scopes around an eval are forwarding parameters (like
*, **, &, or ...) we need to know that information when we are in
the parser. As such, we need to support passing that information
into the scopes option. In order to do this, unfortunately we need
a bunch of changes.
The scopes option was previously an array of array of strings.
These corresponded to the names of the locals in the parent scopes.
We still support this, but now additionally support passing in a
Prism::Scope instance at each index in the array. This Prism::Scope
class holds both the names of the locals as well as an array of
forwarding parameter names (symbols corresponding to the forwarding
parameters). There is convenience function on the Prism module that
creates a Prism::Scope object using Prism.scope.
In JavaScript, we now additionally support an object much the same
as the Ruby side. In Java, we now have a ParsingOptions.Scope class
that holds that information. In the dump APIs, these objects in all
3 languages will add an additional byte for the forwarding flags in
the middle of the scopes serialization.
All of this is in service of properly parsing the following code:
```ruby
def foo(*) = eval("bar(*)")
```
https://github.com/ruby/prism/commit/21abb6b7c4
|
|
https://github.com/ruby/prism/commit/7a93a307ac
|
|
https://github.com/ruby/prism/commit/8ab2532f09
|
|
To make it so that you can pass `freeze: true` to Prism parse
methods and get back a deeply-frozen AST that is Ractor-
shareable.
https://github.com/ruby/prism/commit/8e6a93b2d2
|
|
It falsely considered it to be a single backtick command
https://github.com/ruby/prism/commit/dd762be590
|
|
```rb
<<~F
foo
#{}
bar
F
```
has zero common whitespace.
https://github.com/ruby/prism/commit/1f3c222a06
|
|
Tests worked around this but the incompatibility is not hard to fix.
This fixes 17 token incompatibilies in tests here that were previously passing
https://github.com/ruby/prism/commit/101962526d
|
|
https://github.com/ruby/prism/commit/8d9d429155
|
|
multiline bytes
A rather silly issue with a rather simple fix.
The ranges already use the offset cache, this effectivly double-encoded them.
https://github.com/ruby/prism/commit/66b65634c0
|
