summaryrefslogtreecommitdiff
path: root/lib/prism/translation
AgeCommit message (Collapse)Author
2026-01-12Sync Prism to 1.8.0Takashi Kokubun
2025-12-18[ruby/prism] Add Ruby 4.1 as a version specifierEarlopain
https://github.com/ruby/prism/commit/138db9ccc4
2025-12-14[ruby/prism] Fix `sexp_processor` gem referenceEarlopain
It's https://rubygems.org/gems/sexp_processor, not https://rubygems.org/gems/sexp https://github.com/ruby/prism/commit/b8a00a5f15
2025-12-14[ruby/prism] Define RubyParser::SyntaxError directly and drop require for ↵Ryan Davis
ruby_parser. Had to add a require of sexp since that came in indirectly via ruby_parser. https://github.com/ruby/prism/commit/df677c324f
2025-12-12[ruby/prism] Fixed Prism::Translation::RubyParser's comment processingRyan Davis
Tests were failing in Flay under Prism. https://github.com/ruby/prism/commit/af9b3640a8
2025-12-02[ruby/prism] Fix the ripper translator to parse as the current rubyEarlopain
Otherwise, it uses the latest prism version https://github.com/ruby/prism/commit/86406f63aa
2025-11-08[ruby/prism] Rename Ruby 3.5 to Ruby 4.0Earlopain
See https://github.com/ruby/ruby/commit/6d81969b475262aba251e99b518181bdf7c5a523 It leaves the old variant around. RuboCop for examples accesses `Prism::Translation::Parser35` to test against ruby-head. For now I left these simply as an alias https://github.com/ruby/prism/commit/d0a823f045
2025-10-30[ruby/prism] Add equal_loc to call nodesKevin Newton
In the case of attribute writes, there are use cases where you want to know the location of the = sign. (Internally we actually need this for translation to the writequark AST.) https://github.com/ruby/prism/commit/bfc798a7ec
2025-09-15[ruby/prism] Fix back reference for ruby_parser on Ruby 2.7Earlopain
Symbol#name is only a thing since Ruby 3.0 https://github.com/ruby/prism/commit/2de82b15fc
2025-09-13[ruby/prism] Documentation for Prism::Translation::ParserKevin Newton
Make it clear that it parses with the most recent version of Ruby syntax. https://github.com/ruby/prism/commit/7285d1fbab
2025-09-11[ruby/prism] Reject some cases with `return` and command callsEarlopain
The same also applies to `break`/`next`. https://bugs.ruby-lang.org/issues/21540 https://github.com/ruby/prism/commit/3a38b192e3
2025-08-14[ruby/prism] Be more defensive in the parser translator lexerEarlopain
Generally I have been good about safely accessing the tokens but failed to properly guard against no tokens in places where it could theoretically happen through invalid syntax. I added a test case for one occurance, other changes are theoretical only. https://github.com/ruby/prism/commit/4a3866af19
2025-08-01[ruby/prism] Match RubyParser behavior for -> lambda argsJustin Collins
https://github.com/ruby/prism/commit/9f55551b09
2025-08-01[ruby/prism] RubyParser translation for stabby lambdas with `it`Justin Collins
https://github.com/ruby/prism/commit/c2e372a8d8
2025-06-30[ruby/prism] Fix parser translator with trailing backslash in `%W` /`%I` arrayEarlopain
https://docs.ruby-lang.org/en/master/syntax/literals_rdoc.html#label-25w+and+-25W-3A+String-Array+Literals > %W allow escape sequences described in Escape Sequences. However the continuation line <newline> is not usable because it is interpreted as the escaped newline described above. https://github.com/ruby/prism/commit/f5c7460ad5
2025-06-11[ruby/prism] Fix parser translator during string escaping with invalid utf-8Earlopain
Instead, prefer `scan_byte` over `get_byte` since that already returns the byte as an integer, sidestepping conversion issues. Fixes https://github.com/ruby/prism/issues/3582 https://github.com/ruby/prism/commit/7f3008b2b5
2025-05-29[ruby/prism] [DOC] Markup `__FILE__` as code, not emphasisNobuyoshi Nakada
https://github.com/ruby/prism/commit/571ba378f5
2025-05-29[ruby/prism] [DOC] Add code fencesNobuyoshi Nakada
https://github.com/ruby/prism/commit/641775e5fe
2025-05-29[ruby/prism] [DOC] Specify markdown mode to RDocNobuyoshi Nakada
https://github.com/ruby/prism/commit/12af4e144e
2025-04-12[ruby/prism] Fix parsing rescued exception via indexed assignmentviralpraxis
Given this code ```ruby begin raise '42' rescue => A[] end ``` Prism fails with this backtrace ``` Error: test_unparser/corpus/literal/rescue.txt(Prism::ParserTest): NoMethodError: undefined method `arguments' for nil prism/lib/prism/translation/parser/compiler.rb:1055:in `visit_index_target_node' prism/lib/prism/node.rb:9636:in `accept' prism/lib/prism/compiler.rb:30:in `visit' prism/lib/prism/translation/parser/compiler.rb:218:in `visit_begin_node' ``` Seems like ```diff - visit_all(node.arguments.arguments), + visit_all(node.arguments&.arguments || []), ``` fixes the problem. https://github.com/ruby/prism/commit/76d01aeb6c
2025-04-02[ruby/prism] Fix parser translator when splatting in pattern matching pinEarlopain
Because it ends up treating it as a local variable, and `a.x` is not a valid local variable name. I'm not big on pattern matching, but conceptually it makes sense to me to treat anything inside ^() to not be pattern matching syntax? https://github.com/ruby/prism/commit/80dbd85c45
2025-03-30[ruby/prism] Fix parser translator when pinning hash with string keysEarlopain
`StringNode` and `SymbolNode` don't have the same shape (`content` vs `value`) and that wasn't handled. I believe the logic for the common case can be reused. I simply left the special handling for implicit nodes in pattern matching and fall through otherwise. NOTE: patterns.txt is not actually tested at the moment, because it contains syntax that `parser` mistakenly rejects. But I checked manually that this doesn't introduce other failures. https://github.com/whitequark/parser/pull/1060 https://github.com/ruby/prism/commit/55adfaa895
2025-03-30[ruby/prism] Accept a newline after the defined? keywordKevin Newton
[Bug #21197] https://github.com/ruby/prism/commit/22be955ce9 Notes: Merged: https://github.com/ruby/ruby/pull/12999
2025-03-22[ruby/prism] Enforce a minimum `parser` version for the parser translatorEarlopain
There hasn't been much that would actually affect parsers usage of it. But, when adding new node types, these usually appear in the `Parser::Meta::NODE_TYPES`. `itblock` was added, gets emitted by prism, and then `rubocop-ast` blindly delegates to `on_itblock`. These methods are dynamically created through `NODE_TYPES`, which means that it will error if it doesn't contain `itblock`. This is unfortunate because in `rubocop-ast` these methods are eagerly defined but the prism translator is lazily loaded on demand. The simplest solution is to add them on the `parser` side (even if they are not emitted directly), and require that a version that contains those be used. In summary when adding a new node type: * Add it to `Parser::Meta::PRISM_TRANSLATION_PARSER_NODE_TYPES` (gets included in `NODE_TYPES`) * Bump the minimum `parser` version used by `prism` to a version that contains the above change * Actually emit that node type in `prism` https://github.com/ruby/prism/commit/d73783d065
2025-03-21[ruby/prism] Don't use `RUBY_VERSION.to_f`Earlopain
There will be a bunch of other problems should 3.10 ever exists, but I guess why not fix this one now. https://github.com/ruby/prism/commit/b385f47f8b
2025-03-20[ruby/prism] Add `Prism::Translation::ParserCurrent`Earlopain
It's not my favorite api but for users that currently use the same thing from `parser`, moving over is more difficult than it needs to be. If you plan to support both old and new ruby versions, you definitly need to branch somewhere on the ruby version to either choose prism or parser. But with prism you then need to enumerate all the versions again and choose the correct one. Also, don't recommend to use `Prism::Translation::Parser` in docs. It's version-less but actually always just uses Ruby 3.4 which is probably not what the user intended. Note: parser also warns when the patch version doesn't match what it expects. But I don't think prism has such a concept, and anyways it would require releases anytime ruby releases, which I don't think is very desirable https://github.com/ruby/prism/commit/77177f9e92
2025-03-19[ruby/prism] Polyfill Kernel#warn category parameterKevin Newton
https://github.com/ruby/prism/commit/d85c72a1b9
2025-03-19[ruby/prism] Warn when the parser translator receives an incompatible ↵Earlopain
builder class In https://github.com/ruby/prism/pull/3494 I added a bit of code so that using the new builder doesn't break stuff. This code can be dropped when it is enforced that builder is _always_ the correct subclass (and makes future issues like that unlikely). https://github.com/ruby/prism/commit/193d4b806d
2025-03-18Fix up merge conflicts for prism syncKevin Newton
2025-03-18[ruby/prism] Fix merge mishapEarlopain
Caused by https://github.com/ruby/prism/pull/3478 and https://github.com/ruby/prism/pull/3443 I also made the builder reference more explicit to clearly distinquish between `::Parser` and `Prism::Translation::Parser` https://github.com/ruby/prism/commit/d52aaa75b6
2025-03-18[ruby/prism] Only unnest parser mlhs nodes when no rest argument is providedEarlopain
``` (a,), = [] PARSER==================== s(:masgn, s(:mlhs, s(:mlhs, s(:lvasgn, :a))), s(:array)) PRISM==================== s(:masgn, s(:mlhs, s(:lvasgn, :a)), s(:array)) ``` https://github.com/ruby/prism/commit/8aa1f4690e
2025-03-18[ruby/prism] Fix parser translator multiline interpolated symbolsEarlopain
In https://github.com/ruby/prism/commit/26370079291a420c6b2b7be5cdbd5c609da62f21 I added tests but didn't modify them correctly https://github.com/ruby/prism/commit/de021e74de
2025-03-18[ruby/prism] Further refine string handling in the parser translatorEarlopain
Mostly around newlines and line continuation. * percent arrays need special backslash handling in the ast * Fix offset issue for heredocs with many line continuations (used wrong variable as index access) * More refined rules on when to simplify string tokens * Handle line continuations in squiggly heredocs * Correctly dedent squiggly heredocs with interpolation * Consider `':foo:` and `%s[foo]` to not be interpolation https://github.com/ruby/prism/commit/4edfe9d981
2025-03-18[ruby/prism] Use Set.new over to_setKevin Newton
https://github.com/ruby/prism/commit/422d5c4c64
2025-03-18[ruby/prism] Optimize array inclusion checks in the parser translatorEarlopain
I see `Array.include?` as 2.4% runtime. Probably because of `LPAREN_CONVERSION_TOKEN_TYPES` but the others will be faster as well. Also remove some inline array checks. They are specifically optimized in Ruby since 3.4, but for now prism is for >= 2.7 https://github.com/ruby/prism/commit/ca9500a3fc
2025-03-18[ruby/prism] Fix parser translator crash for certain octal escapesEarlopain
`Integer#chr` performs some validation that we don't want/need. Octal escapes can go above 255, where it will then raise trying to convert. `append_as_bytes` actually allows to pass a number, so we can just skip that call. Although, on older rubies of course we still need to handle this in the polyfill. I don't really like using `pack` but don't know of another way to do so. For the utf-8 escapes, this is not an issue. Invalid utf-8 in these is simply a syntax error. https://github.com/ruby/prism/commit/161c606b1f
2025-03-18[ruby/prism] Handle control and meta escapes in parser translationKevin Newton
https://github.com/ruby/prism/commit/09c59a3aa5
2025-03-18[ruby/prism] Further refine string handling in the parser translatorEarlopain
Mostly around newlines and line continuation. * percent arrays need special backslash handling in the ast * Fix offset issue for heredocs with many line continuations (used wrong variable as index access) * More refined rules on when to simplify string tokens * Handle line continuations in squiggly heredocs * Correctly dedent squiggly heredocs with interpolation * Consider `':foo:` and `%s[foo]` to not be interpolation https://github.com/ruby/prism/commit/4edfe9d981
2025-03-18[ruby/prism] Better handle regexp in the parser translatorEarlopain
Turns out, it was already almost correct. If you disregard \c and \M style escapes, only a single character is allowed to be escaped in a regex so most tests passed already. There was also a mistake where the wrong value was constructed for the ast, this is now fixed. One test fails because of this, but I'm fairly sure it is because of a parser bug. For `/\“/`, the backslash is supposed to be removed because it is a multibyte character. But tbh, I don't entirely understand all the rules. Fixes more than half of the remaining ast differences for rubocop tests https://github.com/ruby/prism/commit/e1c75f304b
2025-03-18[ruby/prism] Fix parser translator tokens for %-arrays with whitespace escapesEarlopain
Also fixes a token incompatibility for the word separator. parser only considers whitespace until the first newline https://github.com/ruby/prism/commit/bd3dd2b62a
2025-03-18[ruby/prism] Fix parser translator edge-case when multiline string ends with \nEarlopain
When the line contains no real newline but contains unescaped ones, then there will be one less entry https://github.com/ruby/prism/commit/4ef093b600
2025-03-18[ruby/prism] Better handle all kinds of multiline strings in the parser ↵Earlopain
translator This is a followup to #3373, where the implementation was extracted https://github.com/ruby/prism/commit/2637007929
2025-03-18[ruby/prism] Fix an incompatibility with the parser translatorEarlopain
The offset cache contains an entry for each byte so it can't be accessed via the string length. Adds tests for all variants except for this: ``` "fo o" "ba ’" ``` For some reason, this still has the wrong offset. https://github.com/ruby/prism/commit/a651126458
2025-03-18[ruby/prism] Fix parser translator rescue location with semicolon bodyEarlopain
There are a few other locations that should be included in that check. I think the end location must always be present but I left it in to be safe (maybe implicit begin somehow?) https://github.com/ruby/prism/commit/545d07ddc3
2025-03-18[ruby/prism] Further refine string handling in the parser translatorEarlopain
Mostly around newlines and line continuation. * percent arrays need special backslash handling in the ast * Fix offset issue for heredocs with many line continuations (used wrong variable as index access) * More refined rules on when to simplify string tokens * Handle line continuations in squiggly heredocs * Correctly dedent squiggly heredocs with interpolation * Consider `':foo:` and `%s[foo]` to not be interpolation https://github.com/ruby/prism/commit/4edfe9d981
2025-03-18[ruby/prism] Add a custom builder class for the parser translatorEarlopain
I want to add new node types to the parser translator, for example `itblock`. The bulk of the work is already done by prism itself. In the `parser` builder, this would be a 5-line change at most but we don't control that here. Instead, we can add our own builder and either overwrite the few methods we need, or just inline the complete builder. I'm not sure yet which would be better. `rubocop-ast` uses its own builder for `parser`. For this to correctly work, it must explicitly choose to extend the prism builder and use it, same as it currently chooses to use a different parser when prism is used. I'd like to enforce that the builder for prism extends its custom one since it will lead to some pretty weird issues otherwise. But first, I'd like to change `rubocop-ast` to make use of this. https://github.com/ruby/prism/commit/b080e608a8
2025-03-18[ruby/prism] Fix parser translator when unescaping invalid utf8Earlopain
1. The string starts out as binary 2. `ち` is appended, forcing it back into utf-8 3. Some invalid byte sequences are tried to append > incompatible character encodings: UTF-8 and BINARY (ASCII-8BIT) This makes use of my wish to use `append_as_bytes`. Unfortunatly that method is rather new so it needs a fallback https://github.com/ruby/prism/commit/e31e94a775
2025-03-13[ruby/prism] Use Set.new over to_setKevin Newton
https://github.com/ruby/prism/commit/422d5c4c64
2025-03-13[ruby/prism] Use `reverse_each` in the parser translatorEarlopain
Avoids an array allocation which matters more and more the larger the file is. I have it at 14% of runtime. https://github.com/ruby/prism/commit/f65b90f27d
2025-03-13[ruby/prism] Optimize array inclusion checks in the parser translatorEarlopain
I see `Array.include?` as 2.4% runtime. Probably because of `LPAREN_CONVERSION_TOKEN_TYPES` but the others will be faster as well. Also remove some inline array checks. They are specifically optimized in Ruby since 3.4, but for now prism is for >= 2.7 https://github.com/ruby/prism/commit/ca9500a3fc
generated by cgit v1.2.3 (git 2.25.1) at 2026-05-13 19:29:33 +0000