summaryrefslogtreecommitdiff
path: root/lib/prism
AgeCommit message (Collapse)Author
2024-03-06[ruby/prism] Add stubs for remaining ripper visit methodsKevin Newton
https://github.com/ruby/prism/commit/4ba9abf664
2024-03-06[ruby/prism] Reconsolidate ripper filesKevin Newton
https://github.com/ruby/prism/commit/7a277be5fb
2024-03-06[ruby/prism] Deprecate `TargetRubyVersion: 80_82_73_83_77.xx`Koichi ITO
Prism has been directly supported as a parser engine since RuboCop 1.62: https://github.com/rubocop/rubocop/releases/tag/v1.62.0 This makes specifying `TargetRubyVersion` with special values like `80_82_73_83_77.33` using the `prism/translation/parser/rubocop` file unnecessary. As a result, it would be possible to deprecate this approach. OTOH, early users might be surprised if `prism/translation/parser/rubocop` were to be suddenly removed. Therefore, this PR deprecates the parameters related to `prism/translation/parser/rubocop`. ```console $ bundle exec ruby -rrubocop -rprism/translation/parser/rubocop -e "RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.33).ast" WARN: Prism is directly supported since RuboCop 1.62. The `prism/translation/parser/rubocop` file is deprecated. WARN: Setting `TargetRubyVersion: 80_82_73_83_77.33` is deprecated. Set to `ParserEngine: parser_prism` and `TargetRubyVersion: 3.3` instead. $ bundle exec ruby -rrubocop -rprism/translation/parser/rubocop -e "RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.34).ast" WARN: Prism is directly supported since RuboCop 1.62. The `prism/translation/parser/rubocop` file is deprecated. WARN: Setting `TargetRubyVersion: 80_82_73_83_77.34` is deprecated. Set to `ParserEngine: parser_prism` and `TargetRubyVersion: 3.4` instead. ``` Eventually, it will be possible to remove it at some point. Regarding documentation, it has been updated to not show the old, discouraged usage but rather the new way of specifying it in RuboCop. https://github.com/ruby/prism/commit/0e4bc31463
2024-03-04[ruby/prism] Add then keyword loc to when nodesKevin Newton
https://github.com/ruby/prism/commit/e1e613df16
2024-03-04[ruby/prism] Update lib/prism/translation/parser/compiler.rbKevin Newton
https://github.com/ruby/prism/commit/dccfd83bc4
2024-03-04[ruby/prism] Fix incompatibility AST for regexp match in ↵Koichi ITO
`Prism::Translation::Parser` This PR fixes the following incompatibility AST for regexp match between Parser gem and Prism: ## Parser gem Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "foo"), s(:regopt)), s(:send, nil, :bar)) ``` ## Prism (`Prism::Translation::Parser`) ### Before Returns an `send` node: ```console $ bundle exec ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] s(:send, s(:regexp, s(:str, "foo"), s(:regopt)), :=~, s(:send, nil, :bar)) ``` ### After Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "foo"), s(:regopt)), s(:send, nil, :bar)) ``` ## Background Found due to incompatibility with RuboCop's `Performance/EndWith`, `Performance/StringInclude, and `Performance/StartWith` cops. ## Note This is the incompatibility when the receiver is a regular expression literal and `=~` is used. Based on the node name `:match_with_lvasgn`, it appears that Prism's AST becomes more accurate in cases like `visit_match_write_node` only. However, as shown in the background, the current behavior of Parser gem is not like this. Considering compatibility with the published AST of Parser gem, the AST incompatibility will be addressed. This lvar-injecting feature appears to have not been supported by Parser gem for a long time: https://github.com/whitequark/parser/issues/69#issuecomment-19506391 There seems to be no indication that it will be supported. This PR prioritizes AST compatibility between the Parser gem and Prism. However, it is unclear whether this is the best approach. https://github.com/ruby/prism/commit/dff4abb170
2024-03-04[ruby/prism] Include BSDmakefileKevin Newton
https://github.com/ruby/prism/commit/1fe507fe3d
2024-03-04[ruby/prism] Fix up some minor parser incompatibilitiesKevin Newton
https://github.com/ruby/prism/commit/c6c771d1fa
2024-02-29[ruby/prism] Command line options as a bitsetKevin Newton
https://github.com/ruby/prism/commit/369ffbd57e
2024-02-29[ruby/prism] Resync RBI and test it in CIKevin Newton
https://github.com/ruby/prism/commit/4ef4032774
2024-02-29[ruby/prism] Fix an incorrect parsing for `Prism::Translation::Parser`Koichi ITO
This PR fixes an incorrect parsing for `Prism::Translation::Parser` when one-line pattern mathing with Ruby 2.7 runtime. ## Expected Parsing should be done based on the specified Ruby parsing version, independent of the Ruby runtime version. When parsing for Ruby 3.3, it should return `:match_pattern_p` node: ```console $ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("foo in bar")' ruby 3.0.6p216 (2023-03-30 revision https://github.com/ruby/prism/commit/23a532679b) [x86_64-darwin19] s(:match_pattern_p, s(:send, nil, :foo), s(:match_var, :bar)) ``` ## Actual When parsing with Ruby 2.7 runtime, `match_pattern` node is returned, even though it is expected to parse for Ruby 3.3: ```console $ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("foo in bar")' ruby 2.7.8p225 (2023-03-30 revision https://github.com/ruby/prism/commit/1f4d455848) [x86_64-darwin19] s(:match_pattern, s(:send, nil, :foo), s(:match_var, :bar)) ``` The cause was the use of `RUBY_VERSION` for condition logic, which made it dependent on runtime Ruby version. `Prism::Translation::Parser` supports parsing for Ruby 3.3+. Therefore, the condition for parsing Ruby 2.7, which is not supported, is being removed. ## Background Found due to incompatibility with RuboCop's `Layout/SpaceAroundKeyword` and `Style/TernaryParentheses` cops. https://github.com/ruby/prism/commit/e752e251d2
2024-02-27[ruby/prism] Follow RuboCop's parser engine supportKoichi ITO
With the introduction of `Prism::Translation::Parser` to RuboCop (RuboCop AST), the number of arguments for `RuboCop::AST::ProcessedSource#parser_class` internal API will be changed: https://github.com/rubocop/rubocop-ast/pull/277 ## Before As a result, the following error will occur starting from the next release of RuboCop AST (< 1.30.0) : ```console $ bundle exec ruby -rrubocop/ast -rprism -rprism/translation/parser/rubocop -ve \ "p RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.33).ast" ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/rubocop.rb:25:in `parser_class': wrong number of arguments (given 2, expected 1) (ArgumentError) from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:309:in `create_parser' from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:219:in `parse' from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:47:in `initialize' from -e:1:in `new' from -e:1:in `<main>' ``` ## After This PR prevents the above error by updating the monkey patch to support the new argument: ```console $ bundle exec ruby -rrubocop/ast -rprism -rprism/translation/parser/rubocop -ve \ "p RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.33).ast" ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] s(:int, 42) ``` Moreover, to ensure compatibility with the existing RuboCop AST, conditional logic has been implemented to maintain backward compatibility. https://github.com/ruby/prism/commit/8a6909f4d5
2024-02-27[ruby/prism] Support -p, -n, -a, and -l command line optionsKevin Newton
https://github.com/ruby/prism/commit/959eb506ca
2024-02-26[ruby/prism] Fix xstring heredoc parser translatorKevin Newton
https://github.com/ruby/prism/commit/4e0f703975
2024-02-26[ruby/prism] Fix parser translator for pinned expressionKevin Newton
https://github.com/ruby/prism/commit/eeae07193b
2024-02-26[ruby/prism] Handle negated numeric in parser translationKevin Newton
https://github.com/ruby/prism/commit/5877a95be4
2024-02-24[ruby/prism] Rebase against mainKevin Newton
https://github.com/ruby/prism/commit/813e20d449
2024-02-24[ruby/prism] Less code modifications. More steep:ignore for nowGopal Patel
https://github.com/ruby/prism/commit/7905bdbf83
2024-02-24[ruby/prism] Replace awkward code changes with steep:ignoreGopal Patel
Also remove RBS for currently ignored files. Will follow-up when those check fully in later PRs. https://github.com/ruby/prism/commit/2cae58f86d
2024-02-24[ruby/prism] Add documentation for Location#source!Gopal Patel
https://github.com/ruby/prism/commit/467e1cc2c4
2024-02-24[ruby/prism] Relax Location#source to be optionalGopal Patel
https://github.com/ruby/prism/commit/9f00fe7510
2024-02-24[ruby/prism] Fix my RipperCompat logic mistake from expansion for steep flow ↵Gopal Patel
analysis https://github.com/ruby/prism/commit/f71a390c12
2024-02-24[ruby/prism] Remove Ripper from public RBS, type-assert remaining issuesGopal Patel
https://github.com/ruby/prism/commit/5fda7a0760
2024-02-24[ruby/prism] Split private typesGopal Patel
https://github.com/ruby/prism/commit/0209d093ec
2024-02-24[ruby/prism] Fix IgnoredNewlineToken comparison of Ripper::Lexer::StateGopal Patel
https://github.com/ruby/prism/commit/8c9502f61b
2024-02-24[ruby/prism] Use steep to type check RBS and Ruby filesGopal Patel
https://github.com/ruby/prism/commit/eabed9f4fd
2024-02-23[ruby/prism] Add some encoding debugging to make testing easierKevin Newton
https://github.com/ruby/prism/commit/0c042561c6
2024-02-23[ruby/prism] Duplicated when clausesKevin Newton
https://github.com/ruby/prism/commit/865b0d5fbe
2024-02-22[ruby/prism] Parse float valuesKevin Newton
https://github.com/ruby/prism/commit/9137226a52
2024-02-22[ruby/prism] Regenerate snapshots using integer valuesKevin Newton
2024-02-22[ruby/prism] Add an IntegerField for parsing integer valuesKevin Newton
https://github.com/ruby/prism/commit/120d8c0479
2024-02-22[ruby/prism] all constant nodes should have full_name and full_name_parts ↵Philip Mueller
methods https://github.com/ruby/prism/commit/792265ae0b
2024-02-21[ruby/prism] Allow skipping warnings as needed, and pass a reason through to ↵Noah Gibbs
Parser::Diagnostic https://github.com/ruby/prism/commit/6a84a3c9fb
2024-02-21[ruby/prism] Update lib/prism/translation/parser.rbNoah Gibbs
https://github.com/ruby/prism/commit/c3cc282343 Co-authored-by: Kevin Newton <kddnewton@gmail.com>
2024-02-21[ruby/prism] Translation::Parser should process warnings, not just errorsNoah Gibbs
https://github.com/ruby/prism/commit/ea7e400f85
2024-02-19[ruby/prism] Fix up comment stateKevin Newton
https://github.com/ruby/prism/commit/c6561becf0
2024-02-18[ruby/prism] Lazy methods for start and end offset on nodeKevin Newton
https://github.com/ruby/prism/commit/4510e2746d
2024-02-18[ruby/prism] Split up comments between leading and trailingKevin Newton
Also make them lazy to allocate the array, and also expose ParseResult#encoding. https://github.com/ruby/prism/commit/08ec7683ae
2024-02-16[ruby/prism] Fix block_pass for []=Kevin Newton
https://github.com/ruby/prism/commit/bf79206220
2024-02-16[ruby/prism] More rescue and ensure; constant handlingNoah Gibbs
https://github.com/ruby/prism/commit/207f579e70
2024-02-16[ruby/prism] Move visit methods into a Ripper CompilerNoah Gibbs
https://github.com/ruby/prism/commit/44c4dc5268
2024-02-16[ruby/prism] Fix lexing of `foo!` when it's a first thing to parseMax Prokopiev
https://github.com/ruby/prism/commit/7597aca76a
2024-02-15[ruby/prism] Make location methods thread-safeBenoit Daloze
* Before it could result in NoMethodError if multiple threads were calling location methods: https://gist.github.com/eregon/b78b7f266d7ee0a278a389cfd1782232 https://github.com/ruby/prism/commit/ff762dcccd
2024-02-15[PRISM] Sync prism version bumpKevin Newton
2024-02-15[ruby/prism] Speed up creating Ruby ASTKevin Newton
When creating the Ruby AST, we were previously allocating Location objects for every node and every inner location. Instead, this commit changes it to pack both the start offset and length into a single u64 and pass that into the nodes. Then, when the locations are requested via a reader method, we lazily allocate the Location objects. https://github.com/ruby/prism/commit/de203dca83 Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
2024-02-15[ruby/prism] Fix up invalid syntaxKevin Newton
https://github.com/ruby/prism/commit/8e3aad9e27
2024-02-15[ruby/prism] Handle more aliases. Better testing of prism ripper CLI and a ↵Noah Gibbs
test for it. https://github.com/ruby/prism/commit/cfd4f28cb3
2024-02-15[ruby/prism] Avoid extra String copies in the FFI backendBenoit Daloze
* For Prism.parse_file the file contents would be read as native, then converted to a Ruby String, then converted to a native String for pm_serialize_parse(). * Refactor the logic to always use a pm_string for the source code and pass that to other native functions. https://github.com/ruby/prism/commit/9002b3c47d
2024-02-15[ruby/prism] Support multi-versioning for `Prism::Translation::Parser`Koichi ITO
## Summary Fixes https://github.com/ruby/prism/pull/2356. I'm working on integrating Prism into RuboCop. This PR introduces `Prism::Translation::Parser33` and `Prism::Translation::Parser34`, named in accordance with the following comments. https://github.com/rubocop/rubocop/issues/12600#issuecomment-1932707748 Currently, `Prism::Translation::Parser` always operates in Ruby 3.4 mode. This means it will not parse as Ruby 3.3 even if `TargetRubyVersion: 80_82_73_83_77.33` is specified. Therefore, the `it` introduced in Ruby 3.4 is parsed incompatibly with Ruby 3.3. In Ruby 3.3, the expected name for an `lvar` is `:it`, not `:"0it"`. ### Expected AST The following is an expected AST when parsing Ruby 3.3 code: ```console $ bundle exec ruby -rprism -rprism/translation/parser33 -ve "p Prism::Translation::Parser33.parse('items.map { it.do_something }')" ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] s(:block, s(:send, s(:send, nil, :items), :map), s(:args), s(:send, s(:send, nil, :it), :do_something)) ``` ### Actual AST The following is an actual AST when parsing Ruby 3.3 code: ```console $ ruby -rprism -ve "p Prism::Translation::Parser.parse('items.map { it.do_something }')" ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22] s(:block, s(:send, s(:send, nil, :items), :map), s(:args), s(:send, s(:lvar, :"0it"), :do_something)) ``` `Prism::Translation::Parser33` and `Prism::Translation::Parser34` aim to correspond to Ruby 3.3 and Ruby 3.4, respectively. And, The hack of specifying `TargetRubyVersion: 80_82_73_83_77.33` is expected to become unnecessary in the future, but the behavior will be maintained until RuboCop's support is finalized: https://github.com/rubocop/rubocop/issues/12600#issuecomment-1933657732 ## Additional Information A private method named `convert_for_prism` is prepared to convert the `version` from Parser to the `version` expected by Prism. For example, a Parser-compatible value is `3.3`, whereas Prism expects `"3.3.0"`. `Parser#version` is not used in RuboCop, but it's unclear how it is utilized in other libraries that rely on the Parser gem. Therefore, logic to maintain compatibility between Parser and Prism is implemented. https://github.com/ruby/prism/commit/62d3991e22
2024-02-14Initialize the Prism::Source directly with all 3 fields for the C extensionBenoit Daloze
* Faster that way: $ ruby -Ilib -rprism -rbenchmark/ips -e 'Benchmark.ips { |x| x.report("parse") { Prism.parse("1 + 2") } }' 195.722k (± 0.5%) i/s rb_iv_set(): 179.609k (± 0.5%) i/s rb_funcall(): 190.030k (± 0.3%) i/s before this PR: 183.319k (± 0.4%) i/s
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-21 02:22:49 +0000