| Age | Commit message (Collapse) | Author |
|---|
|
Creating state classes is pretty expensive.
Since they are not modifiable, we can reuse them instead.
Benchmark script:
```rb
require "ripper"
require "prism"
require "benchmark/ips"
codes = Dir["**/*.rb"].map { File.read(it) }
Benchmark.ips do |x|
x.config(time: 10)
x.report("prism") { codes.each { Prism::Translation::Ripper.lex(it) } }
x.report("ripper") { codes.each { Ripper.lex(it) } }
x.compare!
end
```
Before:
```
ruby 4.0.0 (2025-12-25 revision https://github.com/ruby/prism/commit/553f1675f3) +PRISM [x86_64-linux]
Warming up --------------------------------------
prism 1.000 i/100ms
ripper 1.000 i/100ms
Calculating -------------------------------------
prism 0.293 (± 0.0%) i/s (3.42 s/i) - 3.000 in 10.248348s
ripper 0.633 (± 0.0%) i/s (1.58 s/i) - 7.000 in 11.055687s
Comparison:
ripper: 0.6 i/s
prism: 0.3 i/s - 2.16x slower
```
After
```
ruby 4.0.0 (2025-12-25 revision https://github.com/ruby/prism/commit/553f1675f3) +PRISM [x86_64-linux]
Warming up --------------------------------------
prism 1.000 i/100ms
ripper 1.000 i/100ms
Calculating -------------------------------------
prism 0.486 (± 0.0%) i/s (2.06 s/i) - 5.000 in 10.280413s
ripper 0.635 (± 0.0%) i/s (1.58 s/i) - 7.000 in 11.027169s
Comparison:
ripper: 0.6 i/s
prism: 0.5 i/s - 1.31x slower
```
https://github.com/ruby/prism/commit/bdde16554c
|
|
The filter class is a 1:1 copy of ruby.
rdoc has 32 test failures. It seems to expect `on_sp` in some cases to render code as written.
https://github.com/ruby/prism/commit/74bb12c825
|
|
It's public API and trivial to implement.
https://github.com/ruby/prism/commit/e77545f8b5
|
|
This is everything that `irb` uses. It works in their test-suite, but there are 20 failures when using the shim that I haven't looked into at all.
`parse` is not used by `irb`. `scan` is, and it's basically `parse` but also including errors. `irb` doesn't seem to care about the errors, so I didn't implement that.
https://github.com/ruby/prism/commit/2c5826b39f
|
|
It is for example used by `irb`, `rdoc`, `syntax_suggest`
https://github.com/ruby/prism/commit/255aeb2485
|
|
One per version seems excessive.
Do note that `rubocop-ast` used to require individual parser files. I wouldn't consider that to be part of the API since everything is autoloaded.
From a GitHub code search, I didn't find anyone else doing it like that.
https://github.com/ruby/prism/commit/458f622c34
|
|
Ripper is either not used or loaded where it is actually needed
https://github.com/ruby/prism/commit/a73a4fb00c
|
|
`compact_child_nodes` allocates an array. We can skip that step by simply yielding the nodes.
Benchmark for visiting the rails codebase:
```rb
require "prism"
require "benchmark/ips"
files = Dir.glob("../rails/**/*.rb")
results = files.map { Prism.parse_file(it) }
visitor = Prism::Visitor.new
Benchmark.ips do |x|
x.config(warmup: 3, time: 10)
x.report do
results.each do
visitor.visit(it.value)
end
end
end
RubyVM::YJIT.enable
Benchmark.ips do |x|
x.config(warmup: 3, time: 10)
x.report do
results.each do
visitor.visit(it.value)
end
end
end
```
Before:
```
ruby 3.4.8 (2025-12-17 revision https://github.com/ruby/prism/commit/995b59f666) +PRISM [x86_64-linux]
Warming up --------------------------------------
1.000 i/100ms
Calculating -------------------------------------
2.691 (± 0.0%) i/s (371.55 ms/i) - 27.000 in 10.089422s
ruby 3.4.8 (2025-12-17 revision https://github.com/ruby/prism/commit/995b59f666) +YJIT +PRISM [x86_64-linux]
Warming up --------------------------------------
1.000 i/100ms
Calculating -------------------------------------
7.278 (±13.7%) i/s (137.39 ms/i) - 70.000 in 10.071568s
```
After:
```
ruby 3.4.8 (2025-12-17 revision https://github.com/ruby/prism/commit/995b59f666) +PRISM [x86_64-linux]
Warming up --------------------------------------
1.000 i/100ms
Calculating -------------------------------------
3.429 (± 0.0%) i/s (291.65 ms/i) - 35.000 in 10.208580s
ruby 3.4.8 (2025-12-17 revision https://github.com/ruby/prism/commit/995b59f666) +YJIT +PRISM [x86_64-linux]
Warming up --------------------------------------
1.000 i/100ms
Calculating -------------------------------------
16.815 (± 0.0%) i/s (59.47 ms/i) - 169.000 in 10.054668s
```
~21% faster on the interpreter, ~56% with YJIT
https://github.com/ruby/prism/commit/bf631750cf
|
|
This PR updates the fallback version for `Prism::Translation::ParserCurrent` from 3.4 to 4.0.
Currently, the fallback resolves to `Parser34`, as shown below:
```console
$ ruby -v -rprism -rprism/translation/parser_current -e 'p Prism::Translation::ParserCurrent'
ruby 3.0.7p220 (2024-04-23 revision https://github.com/ruby/prism/commit/724a071175) [x86_64-darwin23]
warning: `Prism::Translation::Current` is loading Prism::Translation::Parser34, but you are running 3.0.
Prism::Translation::Parser34
```
Following the comment "Keep this in sync with released Ruby.",
it seems like the right time to set this to Ruby 4.0, which is scheduled for release this week.
https://github.com/ruby/prism/commit/115f0a118c
|
|
https://github.com/ruby/prism/commit/138db9ccc4
|
|
It's https://rubygems.org/gems/sexp_processor, not https://rubygems.org/gems/sexp
https://github.com/ruby/prism/commit/b8a00a5f15
|
|
ruby_parser.
Had to add a require of sexp since that came in indirectly via ruby_parser.
https://github.com/ruby/prism/commit/df677c324f
|
|
Tests were failing in Flay under Prism.
https://github.com/ruby/prism/commit/af9b3640a8
|
|
Otherwise, it uses the latest prism version
https://github.com/ruby/prism/commit/86406f63aa
|
|
See https://github.com/ruby/ruby/commit/6d81969b475262aba251e99b518181bdf7c5a523
It leaves the old variant around. RuboCop for examples accesses `Prism::Translation::Parser35`
to test against ruby-head. For now I left these simply as an alias
https://github.com/ruby/prism/commit/d0a823f045
|
|
In the case of attribute writes, there are use cases where you want
to know the location of the = sign. (Internally we actually need
this for translation to the writequark AST.)
https://github.com/ruby/prism/commit/bfc798a7ec
|
|
Symbol#name is only a thing since Ruby 3.0
https://github.com/ruby/prism/commit/2de82b15fc
|
|
Make it clear that it parses with the most recent version of Ruby
syntax.
https://github.com/ruby/prism/commit/7285d1fbab
|
|
The same also applies to `break`/`next`.
https://bugs.ruby-lang.org/issues/21540
https://github.com/ruby/prism/commit/3a38b192e3
|
|
Generally I have been good about safely accessing the tokens but failed
to properly guard against no tokens in places
where it could theoretically happen through invalid syntax.
I added a test case for one occurance, other changes are theoretical only.
https://github.com/ruby/prism/commit/4a3866af19
|
|
https://github.com/ruby/prism/commit/9f55551b09
|
|
https://github.com/ruby/prism/commit/c2e372a8d8
|
|
https://docs.ruby-lang.org/en/master/syntax/literals_rdoc.html#label-25w+and+-25W-3A+String-Array+Literals
> %W allow escape sequences described in Escape Sequences. However the continuation line <newline> is not usable because it is interpreted as the escaped newline described above.
https://github.com/ruby/prism/commit/f5c7460ad5
|
|
Instead, prefer `scan_byte` over `get_byte` since that already returns the byte as an integer, sidestepping conversion issues.
Fixes https://github.com/ruby/prism/issues/3582
https://github.com/ruby/prism/commit/7f3008b2b5
|
|
https://github.com/ruby/prism/commit/571ba378f5
|
|
https://github.com/ruby/prism/commit/641775e5fe
|
|
https://github.com/ruby/prism/commit/12af4e144e
|
|
Given this code
```ruby
begin
raise '42'
rescue => A[]
end
```
Prism fails with this backtrace
```
Error: test_unparser/corpus/literal/rescue.txt(Prism::ParserTest): NoMethodError: undefined method `arguments' for nil
prism/lib/prism/translation/parser/compiler.rb:1055:in `visit_index_target_node'
prism/lib/prism/node.rb:9636:in `accept'
prism/lib/prism/compiler.rb:30:in `visit'
prism/lib/prism/translation/parser/compiler.rb:218:in `visit_begin_node'
```
Seems like
```diff
- visit_all(node.arguments.arguments),
+ visit_all(node.arguments&.arguments || []),
```
fixes the problem.
https://github.com/ruby/prism/commit/76d01aeb6c
|
|
Because it ends up treating it as a local variable, and `a.x`
is not a valid local variable name.
I'm not big on pattern matching, but conceptually it makes sense to me
to treat anything inside ^() to not be
pattern matching syntax?
https://github.com/ruby/prism/commit/80dbd85c45
|
|
`StringNode` and `SymbolNode` don't have the same shape
(`content` vs `value`) and that wasn't handled.
I believe the logic for the common case can be reused.
I simply left the special handling for implicit nodes in pattern matching
and fall through otherwise.
NOTE: patterns.txt is not actually tested at the moment,
because it contains syntax that `parser` mistakenly rejects.
But I checked manually that this doesn't introduce other failures.
https://github.com/whitequark/parser/pull/1060
https://github.com/ruby/prism/commit/55adfaa895
|
|
[Bug #21197]
https://github.com/ruby/prism/commit/22be955ce9
Notes:
Merged: https://github.com/ruby/ruby/pull/12999
|
|
There hasn't been much that would actually affect parsers usage of it.
But, when adding new node types, these usually appear in the `Parser::Meta::NODE_TYPES`.
`itblock` was added, gets emitted by prism, and then `rubocop-ast` blindly delegates to `on_itblock`.
These methods are dynamically created through `NODE_TYPES`, which means that it will error if it
doesn't contain `itblock`.
This is unfortunate because in `rubocop-ast` these methods are eagerly defined but
the prism translator is lazily loaded on demand.
The simplest solution is to add them on the `parser` side (even if they are not emitted directly), and require that a version that contains those be used.
In summary when adding a new node type:
* Add it to `Parser::Meta::PRISM_TRANSLATION_PARSER_NODE_TYPES` (gets included in `NODE_TYPES`)
* Bump the minimum `parser` version used by `prism` to a version that contains the above change
* Actually emit that node type in `prism`
https://github.com/ruby/prism/commit/d73783d065
|
|
There will be a bunch of other problems should 3.10 ever exists, but I guess why not fix this one now.
https://github.com/ruby/prism/commit/b385f47f8b
|
|
It's not my favorite api but for users that currently use the same thing
from `parser`, moving over is more difficult
than it needs to be.
If you plan to support both old and new ruby versions, you definitly need to
branch somewhere on the ruby version
to either choose prism or parser.
But with prism you then need to enumerate all the versions again and choose the correct one.
Also, don't recommend to use `Prism::Translation::Parser` in docs. It's version-less
but actually always just uses Ruby 3.4 which is probably
not what the user intended.
Note: parser also warns when the patch version doesn't match what it expects. But I don't think prism has such a concept,
and anyways it would require releases anytime ruby releases, which I don't think is very desirable
https://github.com/ruby/prism/commit/77177f9e92
|
|
https://github.com/ruby/prism/commit/d85c72a1b9
|
|
builder class
In https://github.com/ruby/prism/pull/3494 I added a bit of code
so that using the new builder doesn't break stuff.
This code can be dropped when it is enforced that builder
is _always_ the correct subclass (and makes future issues like that unlikely).
https://github.com/ruby/prism/commit/193d4b806d
|
|
|
|
Caused by https://github.com/ruby/prism/pull/3478 and https://github.com/ruby/prism/pull/3443
I also made the builder reference more explicit to clearly distinquish
between `::Parser` and `Prism::Translation::Parser`
https://github.com/ruby/prism/commit/d52aaa75b6
|
|
```
(a,), = []
PARSER====================
s(:masgn,
s(:mlhs,
s(:mlhs,
s(:lvasgn, :a))),
s(:array))
PRISM====================
s(:masgn,
s(:mlhs,
s(:lvasgn, :a)),
s(:array))
```
https://github.com/ruby/prism/commit/8aa1f4690e
|
|
In https://github.com/ruby/prism/commit/26370079291a420c6b2b7be5cdbd5c609da62f21 I added tests but didn't modify them correctly
https://github.com/ruby/prism/commit/de021e74de
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
https://github.com/ruby/prism/commit/422d5c4c64
|
|
I see `Array.include?` as 2.4% runtime. Probably because of `LPAREN_CONVERSION_TOKEN_TYPES` but
the others will be faster as well.
Also remove some inline array checks. They are specifically optimized in Ruby since 3.4, but for now prism is for >= 2.7
https://github.com/ruby/prism/commit/ca9500a3fc
|
|
`Integer#chr` performs some validation that we don't want/need. Octal escapes can go above 255, where it will then raise trying to convert.
`append_as_bytes` actually allows to pass a number, so we can just skip that call.
Although, on older rubies of course we still need to handle this in the polyfill.
I don't really like using `pack` but don't know of another way to do so.
For the utf-8 escapes, this is not an issue. Invalid utf-8 in these is simply a syntax error.
https://github.com/ruby/prism/commit/161c606b1f
|
|
https://github.com/ruby/prism/commit/09c59a3aa5
|
|
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
|
|
Turns out, it was already almost correct. If you disregard \c and \M style escapes, only a single character is allowed to be escaped in a regex so most tests passed already.
There was also a mistake where the wrong value was constructed for the ast, this is now fixed.
One test fails because of this, but I'm fairly sure it is because of a parser bug. For `/\“/`, the backslash is supposed to be removed because it is a multibyte character. But tbh,
I don't entirely understand all the rules.
Fixes more than half of the remaining ast differences for rubocop tests
https://github.com/ruby/prism/commit/e1c75f304b
|
|
Also fixes a token incompatibility for the word separator. parser only considers whitespace until the first newline
https://github.com/ruby/prism/commit/bd3dd2b62a
|
|
When the line contains no real newline but contains unescaped ones, then there will be one less entry
https://github.com/ruby/prism/commit/4ef093b600
|
|
translator
This is a followup to #3373, where the implementation
was extracted
https://github.com/ruby/prism/commit/2637007929
|
