path: root/ext/ripper
AgeCommit message (Collapse)Author
2 days[Bug #19399] Parsing invalid heredoc inside block parameterNobuyoshi Nakada
Although this is of course invalid as Ruby code, allow to just parse and tokenize. Notes: Merged:
2022-12-02Introduce encoding check macroS-H-GAMELINKS
Notes: Merged:
2022-11-21Enhance keep_tokens option for RubyVM::AbstractSyntaxTree parsing methodsyui-knk
Implementation for Language Server Protocol (LSP) sometimes needs token information. For example both `m(1)` and `m(1, )` has same AST structure other than node locations then it's impossible to check the existence of `,` from AST. However in later case, it might be better to suggest variables list for the second argument. Token information is important for such case. This commit adds these methods. * Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of` * Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes. * Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node. [Feature #19070] Impacts on memory usage and performance are below: Memory usage: ``` $ cat test.rb root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true) $ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux] 11408kb # keep_tokens :false $ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb 17508kb # keep_tokens :true $ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb 30960kb ``` Performance: ``` $ cat ../ast_keep_tokens.yml prelude: | src = <<~SRC module M class C def m1(a, b) 1 + a + b end end end SRC benchmark: without_keep_tokens: | RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false) with_keep_tokens: | RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true) $ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml /home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \ --executables="compare-ruby::./ruby -I.ext/common --disable-gem" \ --executables="built-ruby::./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems --disable-gem" \ --output=markdown --output-compare -v ../ast_keep_tokens.yml compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux] built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux] warming up.. | |compare-ruby|built-ruby| |:--------------------|-----------:|---------:| |without_keep_tokens | 21.659k| 21.303k| | | 1.02x| -| |with_keep_tokens | 6.220k| 5.691k| | | 1.09x| -| ``` Notes: Merged:
2022-11-10Transition shape when object's capacity changesJemma Issroff
This commit adds a `capacity` field to shapes, and adds shape transitions whenever an object's capacity changes. Objects which are allocated out of a bigger size pool will also make a transition from the root shape to the shape with the correct capacity for their size pool when they are allocated. This commit will allow us to remove numiv from objects completely, and will also mean we can guarantee that if two objects share shapes, their IVs are in the same positions (an embedded and extended object cannot share shapes). This will enable us to implement ivar sets in YJIT using object shapes. Co-Authored-By: Aaron Patterson <> Notes: Merged:
2022-11-10Preprocess for older bison is no longer neededNobuyoshi Nakada
Notes: Merged:
2022-11-08Set default %printer for NODE ntermsyui-knk
Before: ``` Reducing stack by rule 639 (line 5062): $1 = token "integer literal" (1.0-1.1: 1) -> $$ = nterm simple_numeric (1.0-1.1: ) ``` After: ``` Reducing stack by rule 641 (line 5078): $1 = token "integer literal" (1.0-1.1: 1) -> $$ = nterm simple_numeric (1.0-1.1: NODE_LIT) ``` `"<*>"` is supported by Bison 2.3b (2008-05-27) or later. Therefore developers need to install Bison 2.3b+ to build ruby from source codes if their Bison is older. Minimum version requirement for Bison is changed to 3.0. See: [Feature #19068] Notes: Merged:
2022-09-08Process token IDs from id.def without id.hNobuyoshi Nakada
Fixes id.h error during updating ripper.c by `make after-update`. While it used to update id.h in the build directory, but was trying to update ripper.c in the source directory. In principle, files in the source directory can or should not depend on files in the build directory.
2022-02-22[Feature #18249] Update dependenciesPeter Zhu
Notes: Merged:
2021-12-09ext/ripper/lib/ripper/lexer.rb: Do not deprecate Ripper::Lexer::State#[]Yusuke Endoh
The old code of IRB still uses this method. The warning is noisy on rails console. In principle, Ruby 3.1 deprecates nothing, so let's avoid the deprecation for the while. I think It is not so hard to continue to maintain it as it is a trivial shim. Notes: Merged:
2021-12-02Define Ripper::Lexer::Elem#to_sNobuyoshi Nakada
Alias `#inspect` as `#to_s` also in the new `Ripper::Lexer::Elem` class, so that `puts` shows the attributes.
2021-12-02Deprecate `Lexer::Elem#[]` and `Lexer::State#[]`schneems
Discussed in > it would be enough to mimic only [] for almost all cases This adds back the `Lexer::Elem#[]` and `Lexer::State#[]` and adds deprecation warnings for them.
2021-12-02Only iterate Lexer heredoc arraysschneems
The last element in the `@buf` may be either an array or an `Elem`. In the case it is an `Elem` we iterate over every element, when we do not need to. This check guards that case by ensuring that we only iterate over an array of elements. Notes: Merged:
2021-12-02~1.10x faster Change Ripper.lex structs to classesschneems
## Concept I am proposing we replace the Struct implementation of data structures inside of ripper with real classes. This will improve performance and the implementation is not meaningfully more complicated. ## Example Struct versus class comparison: ```ruby Elem =, :event, :tok, :state, :message) do def initialize(pos, event, tok, state, message = nil) super(pos, event, tok,, message) end # ... def to_a a = super a.pop unless a.empty? a end end class ElemClass attr_accessor :pos, :event, :tok, :state, :message def initialize(pos, event, tok, state, message = nil) @pos = pos @event = event @tok = tok @state = @message = message end def to_a if @message [@pos, @event, @tok, @state, @message] else [@pos, @event, @tok, @state] end end end # stub state class creation for now class State; def initialize(val); end; end ``` ## MicroBenchmark creation ```ruby require 'benchmark/ips' require 'ripper' pos = [1, 2] event = :on_nl tok = "\n".freeze state = Ripper::EXPR_BEG Benchmark.ips do |x|"struct") {, event, tok, state) }"class ") {, event, tok, state) }! end; nil ``` Gives ~1.2x faster creation: ``` Warming up -------------------------------------- struct 263.983k i/100ms class 303.367k i/100ms Calculating ------------------------------------- struct 2.638M (± 5.9%) i/s - 13.199M in 5.023460s class 3.171M (± 4.6%) i/s - 16.078M in 5.082369s Comparison: class : 3170690.2 i/s struct: 2638493.5 i/s - 1.20x (± 0.00) slower ``` ## MicroBenchmark `to_a` (Called by Ripper.lex for every element) ```ruby require 'benchmark/ips' require 'ripper' pos = [1, 2] event = :on_nl tok = "\n".freeze state = Ripper::EXPR_BEG struct =, event, tok, state) from_class =, event, tok, state) Benchmark.ips do |x|"struct") { struct.to_a }"class ") { from_class.to_a }! end; nil ``` Gives 1.46x faster `to_a`: ``` Warming up -------------------------------------- struct 612.094k i/100ms class 893.233k i/100ms Calculating ------------------------------------- struct 6.121M (± 5.4%) i/s - 30.605M in 5.015851s class 8.931M (± 7.9%) i/s - 44.662M in 5.039733s Comparison: class : 8930619.0 i/s struct: 6121358.9 i/s - 1.46x (± 0.00) slower ``` ## MicroBenchmark data access ```ruby require 'benchmark/ips' require 'ripper' pos = [1, 2] event = :on_nl tok = "\n".freeze state = Ripper::EXPR_BEG struct =, event, tok, state) from_class =, event, tok, state) Benchmark.ips do |x|"struct") { struct.pos[1] }"class ") { from_class.pos[1] }! end; nil ``` Gives ~1.17x faster data access: ``` Warming up -------------------------------------- struct 1.694M i/100ms class 1.868M i/100ms Calculating ------------------------------------- struct 16.149M (± 6.8%) i/s - 81.318M in 5.060633s class 18.886M (± 2.9%) i/s - 95.262M in 5.048359s Comparison: class : 18885669.6 i/s struct: 16149255.8 i/s - 1.17x (± 0.00) slower ``` ## Full benchmark integration of this inside of Ripper.lex Inside of this repo with this commit ``` $ cd ext/ripper $ make $ cat test.rb file = File.join(__dir__, "../../array.rb") source = bench = Benchmark.measure do 10_000.times.each do Ripper.lex(source) end end puts bench ``` Then execute with and without this change 50 times: ``` rm new.txt rm old.txt for i in {0..50} do `ruby -Ilib -rripper -rbenchmark ./test.rb >> new.txt` `ruby -rripper -rbenchmark ./test.rb >> old.txt` done ``` I used derailed benchmarks internals to compare the results: ``` dir = Pathname(".") branch_info = {} branch_info["old"] = { desc: "Struct lex", time:, file: dir.join("old.txt"), name: "old" } branch_info["new"] = { desc: "Class lex", time:, file: dir.join("new.txt"), name: "new" } stats = ``` Which gave us: ``` ❤️ ❤️ ❤️ (Statistically Significant) ❤️ ❤️ ❤️ [new] (3.3139 seconds) "Class lex" ref: "new" FASTER 🚀🚀🚀 by: 1.1046x [older/newer] 9.4700% [(older - newer) / older * 100] [old] (3.6606 seconds) "Struct lex" ref: "old" Iterations per sample: Samples: 51 Test type: Kolmogorov Smirnov Confidence level: 99.0 % Is significant? (max > critical): true D critical: 0.30049534876137013 D max: 0.9607843137254902 Histograms (time ranges are in seconds): [new] description: [old] description: "Class lex" "Struct lex" ┌ ┐ ┌ ┐ [3.0, 3.3) ┤▇ 1 [3.0, 3.3) ┤ 0 [3.3, 3.6) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 47 [3.3, 3.6) ┤ 0 [3.5, 3.8) ┤▇▇ 2 [3.5, 3.8) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 46 [3.8, 4.1) ┤▇ 1 [3.8, 4.1) ┤▇▇▇ 4 [4.0, 4.3) ┤ 0 [4.0, 4.3) ┤ 0 [4.3, 4.6) ┤ 0 [4.3, 4.6) ┤▇ 1 └ ┘ └ ┘ # of runs in range # of runs in range ``` To sum this up, the "new" version of this code (using real classes instead of structs) is 10% faster across 50 runs with a statistical significance confidence level of 99%. Histograms are for visual checksum. Notes: Merged:
2021-11-25Keep the generated source files when clean [Bug #18363]Nobuyoshi Nakada
Notes: Merged:
2021-11-21Update dependenciesNobuyoshi Nakada
2021-10-05ruby tool/update-deps --fix卜部昌平
Notes: Merged:
2021-04-13dependency updates卜部昌平
Notes: Merged:
2021-02-19ripper: fix a bug of Ripper::Lexer with syntax error and heredoc [Bug #17644]Shugo Maeda
2021-01-17Fix Ripper with heredoc.manga_osyo
Notes: Merged:
2021-01-04ripper: call #pretty_print on also `state`Nobuyoshi Nakada
2020-12-19ripper: fix `#tok` on some error events [Bug 17345]Nobuhiro IMAI
sorting alias target by event arity, and setup suitable `Elem` for error. Notes: Merged:
2020-12-15Ripper: Refined error callbacks [Bug #17345]Nobuyoshi Nakada
Notes: Merged:
2020-12-15ripper: return pushed new token instead of the token listNobuyoshi Nakada
2020-11-26Store all kinds of syntax errors [Bug #17345]Nobuyoshi Nakada
2020-11-20[DOC] Ripper.{lex,tokenize} now always return full tokens. [ci skip]Nobuhiro IMAI
Notes: Merged:
2020-11-20[Feature #17276] Moved raise_errors support to Ripper::Lexer#parseNobuyoshi Nakada
2020-11-20Ripper.{lex,tokenize} return full tokens even if syntax errorNobuhiro IMAI
yet another implements [Feature #17276]
2020-11-17Update documentation for Ripper.{lex,tokenize,sexp,sexp_raw} [ci skip]Jeremy Evans
2020-11-17Support raise_errors keyword for Ripper.{lex,tokenize,sexp,sexp_raw}Jeremy Evans
Implements [Feature #17276] Notes: Merged: Merged-By: jeremyevans <>
2020-11-18fix public interfaceKoichi Sasada
To make some kind of Ractor related extensions, some functions should be exposed. * include/ruby/thread_native.h * rb_native_mutex_* * rb_native_cond_* * include/ruby/ractor.h * RB_OBJ_SHAREABLE_P(obj) * rb_ractor_shareable_p(obj) * rb_ractor_std*() * rb_cRactor and rm ractor_pub.h and rename srcdir/ractor.h to srcdir/ractor_core.h (to avoid conflict with include/ruby/ractor.h) Notes: Merged:
2020-09-03Introduce Ractor mechanism for parallel executionKoichi Sasada
This commit introduces Ractor mechanism to run Ruby program in parallel. See doc/ for more details about Ractor. See ticket [Feature #17100] to see the implementation details and discussions. [Feature #17100] This commit does not complete the implementation. You can find many bugs on using Ractor. Also the specification will be changed so that this feature is experimental. You will see a warning when you make the first Ractor with ``. I hope this feature can help programmers from thread-safety issues. Notes: Merged:
2020-08-27sed -i '/rmodule.h/d'卜部昌平
Notes: Merged:
2020-08-27sed -i '/r_cast.h/d'卜部昌平
Notes: Merged:
2020-08-27sed -i '\,2/extern.h,d'卜部昌平
Notes: Merged:
2020-05-29Allow references to $$ in Ripper DSLNobuyoshi Nakada
2020-05-11sed -i 's|ruby/impl|ruby/internal|'卜部昌平
To fix build failures. Notes: Merged:
2020-05-11sed -i s|ruby/3|ruby/impl|g卜部昌平
This shall fix compile errors. Notes: Merged:
2020-05-04Suppress warnings by gcc 10.1.0-RC-20200430Nobuyoshi Nakada
* Folding results should not be empty. If `OnigCodePointCount(to->n)` were 0, `for` loop using `fn` wouldn't execute and `ncs` elements are not initialized. ``` enc/unicode.c:557:21: warning: 'ncs[0]' may be used uninitialized in this function [-Wmaybe-uninitialized] 557 | for (i = 0; i < ncs[0]; i++) { | ~~~^~~ ``` * Cast to `enum yytokentype` Additional enums for scanner events by ripper are not included in `yytokentype`. ``` ripper.y:7274:28: warning: implicit conversion from 'enum <anonymous>' to 'enum yytokentype' [-Wenum-conversion] ```
2020-04-08Merge pull request #2991 from shyouhei/ruby.h卜部昌平
Split ruby.h Notes: Merged-By: shyouhei <>
2020-02-15Workaround for bison provided by scoop on mswin environmentHiroshi SHIBATA
Notes: Merged:
2020-01-20Get rid of use of special variablesJeremy Evans
Use `"\n"` and `IO#fileno` instead of `$/` and `$.` respectively. [Feature #14240]
2020-01-17Update dependencies in makefiles againKazuhiro NISHIYAMA
patch from
2019-12-31Updated dependencies on internal/warnings.hNobuyoshi Nakada
Needed for `UNALIGNED_MEMBER_ACCESS` using `COMPILER_WARNING_`* macros. Notes: Merged:
2019-12-26update dependencies卜部昌平
Notes: Merged:
2019-11-26Allow `$10` and more in the Ripper DSLNobuyoshi Nakada
2019-11-18Update dependenciesNobuyoshi Nakada
2019-11-13Update comment of Ripper.lexYuichiro Kaneko
This is follow up of 1f7cb4bee9.
2019-11-12Revert "Method reference operator"Nobuyoshi Nakada
This reverts commit 67c574736912003c377218153f9d3b9c0c96a17b. [Feature #16275]
2019-11-09Remove unneeded exec bits from some filesDavid Rodríguez
I noticed that some files in rubygems were executable, and I could think of no reason why they should be. In general, I think ruby files should never have the executable bit set unless they include a shebang, so I run the following command over the whole repo: ```bash find . -name '*.rb' -type f -executable -exec bash -c 'grep -L "^#!" $1 || chmod -x $1' _ {} \; ``` Notes: Merged:
2019-11-07Suppress unused variable warningNobuyoshi Nakada