summaryrefslogtreecommitdiff
path: root/lib/prism/translation/parser/compiler.rb
AgeCommit message (Collapse)Author
2025-09-12Bump Prism version to 1.5.0Takashi Kokubun
2025-07-09[ruby/prism] Accept a newline after the defined? keyword [Backport #21197] ↵Takashi Kokubun
(#13327) * [ruby/prism] Accept a newline after the defined? keyword [Bug #21197] https://github.com/ruby/prism/commit/22be955ce9 * Fix a compilation error Co-authored-by: Stan Lo <stan001212@gmail.com> --------- Co-authored-by: Kevin Newton <kddnewton@gmail.com> Co-authored-by: Stan Lo <stan001212@gmail.com>
2025-04-14Revert "merge revision(s) 052794bfe1970e90f4f4f9e37fc362dd27903a8d: ↵Takashi Kokubun
[Backport #21197]" This reverts commit e630a0f7ae909dafe2e1dbc47baf90cd795b4a5f.
2025-04-14merge revision(s) 052794bfe1970e90f4f4f9e37fc362dd27903a8d: [Backport #21197]Takashi Kokubun
[ruby/prism] Accept a newline after the defined? keyword [Bug #21197] https://github.com/ruby/prism/commit/22be955ce9
2024-12-16[ruby/prism] Simplify srange_find in parser compilerKevin Newton
https://github.com/ruby/prism/commit/34efacc618 Notes: Merged: https://github.com/ruby/ruby/pull/12358
2024-12-16[ruby/prism] Add do keyword tracking for While/UntilKevin Newton
https://github.com/ruby/prism/commit/9686897290 Notes: Merged: https://github.com/ruby/ruby/pull/12358
2024-08-28[PRISM] Field renamingKevin Newton
Rename some fields that do not quite make sense. * CaseMatchNode#consequent -> CaseMatchNode#else_clause * CaseNode#consequent -> CaseNode#else_clause * IfNode#consequent -> IfNode#subsequent * RescueNode#consequent -> RescueNode#subsequent * UnlessNode#consequent -> UnlessNode#else_clause Notes: Merged: https://github.com/ruby/ruby/pull/11480
2024-06-07[ruby/prism] Handle chomped bytesize with lines without newlinesKevin Newton
https://github.com/ruby/prism/commit/1528d3c019
2024-06-07[ruby/prism] Ensure inner heredoc nodes have the correct locationKevin Newton
https://github.com/ruby/prism/commit/100340bc6b
2024-06-07[ruby/prism] Use correct newlines for heredoc inner linesKevin Newton
https://github.com/ruby/prism/commit/4a9a7a62af Co-authored-by: Jason Kim <jasonkim@github.com> Co-authored-by: Adam Hess <HParker@github.com>
2024-06-04[ruby/prism] (parser) split up regexp content by linesKevin Newton
https://github.com/ruby/prism/commit/85b4a5f804
2024-06-04[ruby/prism] (parser) handle quoted symbols in hash patternsKevin Newton
https://github.com/ruby/prism/commit/461aa5e658
2024-06-04[ruby/prism] (parser) fix up srange_find to anchor at the start of the sliceKevin Newton
https://github.com/ruby/prism/commit/aecce571d8
2024-06-04[ruby/prism] (parser) fix up nested multi writeKevin Newton
https://github.com/ruby/prism/commit/12e079c97e
2024-06-04[ruby/prism] Revert "Revert "Properly destructure procarg0 in parser ↵Kevin Newton
translation"" This reverts commit https://github.com/ruby/prism/commit/d8ae19d0334a. https://github.com/ruby/prism/commit/df1eda2811
2024-06-03[ruby/prism] Revert "Properly destructure procarg0 in parser translation"Kevin Newton
This reverts commit https://github.com/ruby/prism/commit/823e931ff230. https://github.com/ruby/prism/commit/d8ae19d033
2024-06-03[ruby/prism] Properly destructure procarg0 in parser translationKevin Newton
https://github.com/ruby/prism/commit/823e931ff2
2024-05-31[ruby/prism] Fix up heredoc location translation for parserKevin Newton
https://github.com/ruby/prism/commit/a4e164e22b
2024-05-31[ruby/prism] Match % strings in parserKevin Newton
https://github.com/ruby/prism/commit/840185110f
2024-05-31[ruby/prism] Use correct opening and closing parenthesis for array pattern ↵Kevin Newton
in parser https://github.com/ruby/prism/commit/beed43922c
2024-05-31[ruby/prism] Match match_hash_var when quotes are usedKevin Newton
https://github.com/ruby/prism/commit/f2a327449a
2024-05-31[ruby/prism] Match parser for match_rest in patternKevin Newton
https://github.com/ruby/prism/commit/785de2c39d
2024-05-22[ruby/prism] Fix support for 'it' implicit local variableKevin Newton
https://github.com/ruby/prism/commit/53bbcfe513
2024-05-21[ruby/prism] Reconfigure rationalsKevin Newton
This eliminates the subnode on RationalNode and replaces it with two integer fields, which represent the ratio for the rational. It also reduces those two integers if they both fit into 32 bits. Importantly, this PR does not implement bignum reduction. That's something I'd like to consider for the future, but it's simple enough for now to leave them unreduced, which makes it more useful than it used to be. https://github.com/ruby/prism/commit/86e06c7068
2024-05-10[ruby/prism] operator renameKevin Newton
https://github.com/ruby/prism/commit/b5e47f5c42
2024-05-03[ruby/prism] Change ConstantPathNode#child to ConstantPathNode#{name,name_loc}Kevin Newton
This has been requested for a long time, and I'm finally doing it now. Unfortunately this is a breaking change for all of the APIs. I've added in a Ruby method for `#child` that is deprecated so that existing usage doesn't break, but for everyone else this is going to be a bit of a pain. https://github.com/ruby/prism/commit/9cbe74464e
2024-04-23[ruby/prism] srange_find should only look on current lineKevin Newton
https://github.com/ruby/prism/commit/3604aa15e7
2024-04-19[ruby/prism] Fix it parameters for parser translationKevin Newton
https://github.com/ruby/prism/commit/2f3feb8d51
2024-04-18[ruby/prism] "Fix" transpose issue in parser compilerKevin Newton
https://github.com/ruby/prism/commit/593d637178
2024-04-12[ruby/prism] Fix parser translation's heredoc whitespace calculationStan Lo
Given this example: ```rb <<~HEREDOC #{x} HEREDOC ``` Both the parser gem and Prism's translation layer would generate the following AST: ``` s(:dstr, s(:begin, s(:int, 1)), s(:str, " a\n")) ``` However, the parser gem inserts a empty string node into this node's location, like: ``` <Parser::Source::Map::Heredoc:0x0000000104ce73b8 @expression=#<Parser::Source::Range (string) 0...10>, @heredoc_body=#<Parser::Source::Range (string) 11...20>, @heredoc_end=#<Parser::Source::Range (string) 20...27>, @node=s(:dstr, s(:str, ""), s(:begin, s(:int, 1)), s(:str, " a\n"))> ``` This is required to calculate the correct whitespace for the heredoc body. We need to adjust the translation layer to account for this. With this fix, we also won't need to ignore the tilde heredoc fixture anymore. https://github.com/ruby/prism/commit/e7372e3ba5
2024-04-04[ruby/prism] Fix up some comments in the parser compilerKevin Newton
https://github.com/ruby/prism/commit/e2147cddd8
2024-03-15[ruby/prism] Fix an AST and token incompatibility for ↵Koichi ITO
`Prism::Translation::Parser` This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser` for dstring literal: ```ruby "foo #{bar}" ``` ## Parser gem (Expected) ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:dstr, s(:str, "foo\n"), s(:str, " "), s(:begin, s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]], [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]], [:tSTRING_CONTENT, [" ", #<Parser::Source::Range example.rb 5...7>]], [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]], [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]], [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the AST and tokens returned by the Parser gem were different. In this case, `dstr` node should not be nested: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:dstr, s(:dstr, s(:str, "foo\n"), s(:str, " ")), s(:begin, s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]], [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]], [:tSTRING_CONTENT, [" ", #<Parser::Source::Range example.rb 5...7>]], [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]], [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]], [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]] ``` After this correction, the AST and tokens returned by the Parser gem are the same: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:dstr, s(:str, "foo\n"), s(:str, " "), s(:begin, s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]], [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]], [:tSTRING_CONTENT, [" ", #<Parser::Source::Range example.rb 5...7>]], [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]], [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]], [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]], [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]] ``` https://github.com/ruby/prism/commit/c1652a9ee7
2024-03-15[ruby/prism] Shareable constant nodesKevin Newton
https://github.com/ruby/prism/commit/473cfed6d0
2024-03-13[ruby/prism] Fix an AST and token incompatibility for ↵Koichi ITO
`Prism::Translation::Parser` This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser` for empty xstring literal. ## Parser gem (Expected) ```console $ bundle exec ruby -Ilib -rparser/ruby33 -ve \ 'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Parser::Ruby33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]], [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]] ``` ## `Prism::Translation::Parser` (Actual) Previously, the AST and tokens returned by the Parser gem were different: ```console $ bunele exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr, s(:str, "")), [], [[:tBACK_REF2, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]], [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]] ``` After this correction, the AST and tokens returned by the Parser gem are the same: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \ 'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] [s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]], [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]] ``` https://github.com/ruby/prism/commit/4ac89dcbb5
2024-03-13[ruby/prism] Fix an AST incompatibility for `Prism::Translation::Parser`Koichi ITO
Fixes ruby/prism#2480. This PR fixes an AST incompatibility between Parser gem and `Prism::Translation::Parser` for xstring literal with line breaks. The following case in ruby/prism#2480 has already been addressed in ruby/prism#2576: ```ruby "foo bar" ``` https://github.com/ruby/prism/commit/cf85e72c55
2024-03-12[ruby/prism] Fix an AST and token incompatibility for ↵Koichi ITO
`Prism::Translation::Parser` Fixes https://github.com/ruby/prism/pull/2515. This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser` for string literal with line breaks. https://github.com/ruby/prism/commit/c58466e5bf
2024-03-07[ruby/prism] Fix an AST and token incompatibility for ↵Koichi ITO
`Prism::Translation::Parser` Fixes https://github.com/ruby/prism/pull/2506. This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser` for symbols quoted with line breaks. https://github.com/ruby/prism/commit/06ab4df8cd
2024-03-06[ruby/prism] Small changes to make type-checking passUfuk Kayserilioglu
https://github.com/ruby/prism/commit/5b2970e75b
2024-03-06[ruby/prism] Fix implicit local variables in hashesKevin Newton
https://github.com/ruby/prism/commit/05e0c6792c
2024-03-06[ruby/prism] Add stubs for remaining ripper visit methodsKevin Newton
https://github.com/ruby/prism/commit/4ba9abf664
2024-03-04[ruby/prism] Add then keyword loc to when nodesKevin Newton
https://github.com/ruby/prism/commit/e1e613df16
2024-03-04[ruby/prism] Update lib/prism/translation/parser/compiler.rbKevin Newton
https://github.com/ruby/prism/commit/dccfd83bc4
2024-03-04[ruby/prism] Fix incompatibility AST for regexp match in ↵Koichi ITO
`Prism::Translation::Parser` This PR fixes the following incompatibility AST for regexp match between Parser gem and Prism: ## Parser gem Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "foo"), s(:regopt)), s(:send, nil, :bar)) ``` ## Prism (`Prism::Translation::Parser`) ### Before Returns an `send` node: ```console $ bundle exec ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] s(:send, s(:regexp, s(:str, "foo"), s(:regopt)), :=~, s(:send, nil, :bar)) ``` ### After Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "foo"), s(:regopt)), s(:send, nil, :bar)) ``` ## Background Found due to incompatibility with RuboCop's `Performance/EndWith`, `Performance/StringInclude, and `Performance/StartWith` cops. ## Note This is the incompatibility when the receiver is a regular expression literal and `=~` is used. Based on the node name `:match_with_lvasgn`, it appears that Prism's AST becomes more accurate in cases like `visit_match_write_node` only. However, as shown in the background, the current behavior of Parser gem is not like this. Considering compatibility with the published AST of Parser gem, the AST incompatibility will be addressed. This lvar-injecting feature appears to have not been supported by Parser gem for a long time: https://github.com/whitequark/parser/issues/69#issuecomment-19506391 There seems to be no indication that it will be supported. This PR prioritizes AST compatibility between the Parser gem and Prism. However, it is unclear whether this is the best approach. https://github.com/ruby/prism/commit/dff4abb170
2024-02-29[ruby/prism] Fix an incorrect parsing for `Prism::Translation::Parser`Koichi ITO
This PR fixes an incorrect parsing for `Prism::Translation::Parser` when one-line pattern mathing with Ruby 2.7 runtime. ## Expected Parsing should be done based on the specified Ruby parsing version, independent of the Ruby runtime version. When parsing for Ruby 3.3, it should return `:match_pattern_p` node: ```console $ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("foo in bar")' ruby 3.0.6p216 (2023-03-30 revision https://github.com/ruby/prism/commit/23a532679b) [x86_64-darwin19] s(:match_pattern_p, s(:send, nil, :foo), s(:match_var, :bar)) ``` ## Actual When parsing with Ruby 2.7 runtime, `match_pattern` node is returned, even though it is expected to parse for Ruby 3.3: ```console $ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("foo in bar")' ruby 2.7.8p225 (2023-03-30 revision https://github.com/ruby/prism/commit/1f4d455848) [x86_64-darwin19] s(:match_pattern, s(:send, nil, :foo), s(:match_var, :bar)) ``` The cause was the use of `RUBY_VERSION` for condition logic, which made it dependent on runtime Ruby version. `Prism::Translation::Parser` supports parsing for Ruby 3.3+. Therefore, the condition for parsing Ruby 2.7, which is not supported, is being removed. ## Background Found due to incompatibility with RuboCop's `Layout/SpaceAroundKeyword` and `Style/TernaryParentheses` cops. https://github.com/ruby/prism/commit/e752e251d2
2024-02-26[ruby/prism] Fix xstring heredoc parser translatorKevin Newton
https://github.com/ruby/prism/commit/4e0f703975
2024-02-26[ruby/prism] Fix parser translator for pinned expressionKevin Newton
https://github.com/ruby/prism/commit/eeae07193b
2024-02-26[ruby/prism] Handle negated numeric in parser translationKevin Newton
https://github.com/ruby/prism/commit/5877a95be4
2024-02-16[ruby/prism] Fix block_pass for []=Kevin Newton
https://github.com/ruby/prism/commit/bf79206220
2024-02-15[ruby/prism] Speed up creating Ruby ASTKevin Newton
When creating the Ruby AST, we were previously allocating Location objects for every node and every inner location. Instead, this commit changes it to pack both the start offset and length into a single u64 and pass that into the nodes. Then, when the locations are requested via a reader method, we lazily allocate the Location objects. https://github.com/ruby/prism/commit/de203dca83 Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
2024-02-07[ruby/prism] Support for Ruby 2.7Kevin Newton
https://github.com/ruby/prism/commit/1a15b70a8e
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-21 10:56:21 +0000