| Age | Commit message (Collapse) | Author |
|---|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6700
|
|
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.
This commit adds these methods.
* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.
[Feature #19070]
Impacts on memory usage and performance are below:
Memory usage:
```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb
# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb
# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```
Performance:
```
$ cat ../ast_keep_tokens.yml
prelude: |
src = <<~SRC
module M
class C
def m1(a, b)
1 + a + b
end
end
end
SRC
benchmark:
without_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
with_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)
$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
--executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
--executables="built-ruby::./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems --disable-gem" \
--output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..
| |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens | 21.659k| 21.303k|
| | 1.02x| -|
|with_keep_tokens | 6.220k| 5.691k|
| | 1.09x| -|
```
Notes:
Merged: https://github.com/ruby/ruby/pull/6770
|
|
This commit adds a `capacity` field to shapes, and adds shape
transitions whenever an object's capacity changes. Objects which are
allocated out of a bigger size pool will also make a transition from the
root shape to the shape with the correct capacity for their size pool
when they are allocated.
This commit will allow us to remove numiv from objects completely, and
will also mean we can guarantee that if two objects share shapes, their
IVs are in the same positions (an embedded and extended object cannot
share shapes). This will enable us to implement ivar sets in YJIT using
object shapes.
Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
Notes:
Merged: https://github.com/ruby/ruby/pull/6699
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/6701
|
|
Before:
```
Reducing stack by rule 639 (line 5062):
$1 = token "integer literal" (1.0-1.1: 1)
-> $$ = nterm simple_numeric (1.0-1.1: )
```
After:
```
Reducing stack by rule 641 (line 5078):
$1 = token "integer literal" (1.0-1.1: 1)
-> $$ = nterm simple_numeric (1.0-1.1: NODE_LIT)
```
`"<*>"` is supported by Bison 2.3b (2008-05-27) or later.
https://git.savannah.gnu.org/cgit/bison.git/commit/?id=12e3584054c16ab255672c07af0ffc7bb220e8bc
Therefore developers need to install Bison 2.3b+ to build ruby from
source codes if their Bison is older.
Minimum version requirement for Bison is changed to 3.0.
See: https://bugs.ruby-lang.org/issues/19068 [Feature #19068]
Notes:
Merged: https://github.com/ruby/ruby/pull/6579
|
|
Fixes id.h error during updating ripper.c by `make after-update`.
While it used to update id.h in the build directory, but was trying to
update ripper.c in the source directory. In principle, files in the
source directory can or should not depend on files in the build
directory.
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/5474
|
|
The old code of IRB still uses this method. The warning is noisy on
rails console.
In principle, Ruby 3.1 deprecates nothing, so let's avoid the
deprecation for the while.
I think It is not so hard to continue to maintain it as it is a trivial
shim.
https://github.com/ruby/ruby/pull/5093
Notes:
Merged: https://github.com/ruby/ruby/pull/5219
|
|
Alias `#inspect` as `#to_s` also in the new `Ripper::Lexer::Elem`
class, so that `puts Ripper::Lexer.new(code).scan` shows the
attributes.
|
|
Discussed in https://github.com/ruby/ruby/pull/5093#issuecomment-964426481.
> it would be enough to mimic only [] for almost all cases
This adds back the `Lexer::Elem#[]` and `Lexer::State#[]` and adds deprecation warnings for them.
|
|
The last element in the `@buf` may be either an array or an `Elem`. In the case it is an `Elem` we iterate over every element, when we do not need to. This check guards that case by ensuring that we only iterate over an array of elements.
Notes:
Merged: https://github.com/ruby/ruby/pull/5093
|
|
## Concept
I am proposing we replace the Struct implementation of data structures inside of ripper with real classes.
This will improve performance and the implementation is not meaningfully more complicated.
## Example
Struct versus class comparison:
```ruby
Elem = Struct.new(:pos, :event, :tok, :state, :message) do
def initialize(pos, event, tok, state, message = nil)
super(pos, event, tok, State.new(state), message)
end
# ...
def to_a
a = super
a.pop unless a.empty?
a
end
end
class ElemClass
attr_accessor :pos, :event, :tok, :state, :message
def initialize(pos, event, tok, state, message = nil)
@pos = pos
@event = event
@tok = tok
@state = State.new(state)
@message = message
end
def to_a
if @message
[@pos, @event, @tok, @state, @message]
else
[@pos, @event, @tok, @state]
end
end
end
# stub state class creation for now
class State; def initialize(val); end; end
```
## MicroBenchmark creation
```ruby
require 'benchmark/ips'
require 'ripper'
pos = [1, 2]
event = :on_nl
tok = "\n".freeze
state = Ripper::EXPR_BEG
Benchmark.ips do |x|
x.report("struct") { Elem.new(pos, event, tok, state) }
x.report("class ") { ElemClass.new(pos, event, tok, state) }
x.compare!
end; nil
```
Gives ~1.2x faster creation:
```
Warming up --------------------------------------
struct 263.983k i/100ms
class 303.367k i/100ms
Calculating -------------------------------------
struct 2.638M (± 5.9%) i/s - 13.199M in 5.023460s
class 3.171M (± 4.6%) i/s - 16.078M in 5.082369s
Comparison:
class : 3170690.2 i/s
struct: 2638493.5 i/s - 1.20x (± 0.00) slower
```
## MicroBenchmark `to_a` (Called by Ripper.lex for every element)
```ruby
require 'benchmark/ips'
require 'ripper'
pos = [1, 2]
event = :on_nl
tok = "\n".freeze
state = Ripper::EXPR_BEG
struct = Elem.new(pos, event, tok, state)
from_class = ElemClass.new(pos, event, tok, state)
Benchmark.ips do |x|
x.report("struct") { struct.to_a }
x.report("class ") { from_class.to_a }
x.compare!
end; nil
```
Gives 1.46x faster `to_a`:
```
Warming up --------------------------------------
struct 612.094k i/100ms
class 893.233k i/100ms
Calculating -------------------------------------
struct 6.121M (± 5.4%) i/s - 30.605M in 5.015851s
class 8.931M (± 7.9%) i/s - 44.662M in 5.039733s
Comparison:
class : 8930619.0 i/s
struct: 6121358.9 i/s - 1.46x (± 0.00) slower
```
## MicroBenchmark data access
```ruby
require 'benchmark/ips'
require 'ripper'
pos = [1, 2]
event = :on_nl
tok = "\n".freeze
state = Ripper::EXPR_BEG
struct = Elem.new(pos, event, tok, state)
from_class = ElemClass.new(pos, event, tok, state)
Benchmark.ips do |x|
x.report("struct") { struct.pos[1] }
x.report("class ") { from_class.pos[1] }
x.compare!
end; nil
```
Gives ~1.17x faster data access:
```
Warming up --------------------------------------
struct 1.694M i/100ms
class 1.868M i/100ms
Calculating -------------------------------------
struct 16.149M (± 6.8%) i/s - 81.318M in 5.060633s
class 18.886M (± 2.9%) i/s - 95.262M in 5.048359s
Comparison:
class : 18885669.6 i/s
struct: 16149255.8 i/s - 1.17x (± 0.00) slower
```
## Full benchmark integration of this inside of Ripper.lex
Inside of this repo with this commit
```
$ cd ext/ripper
$ make
$ cat test.rb
file = File.join(__dir__, "../../array.rb")
source = File.read(file)
bench = Benchmark.measure do
10_000.times.each do
Ripper.lex(source)
end
end
puts bench
```
Then execute with and without this change 50 times:
```
rm new.txt
rm old.txt
for i in {0..50}
do
`ruby -Ilib -rripper -rbenchmark ./test.rb >> new.txt`
`ruby -rripper -rbenchmark ./test.rb >> old.txt`
done
```
I used derailed benchmarks internals to compare the results:
```
dir = Pathname(".")
branch_info = {}
branch_info["old"] = { desc: "Struct lex", time: Time.now, file: dir.join("old.txt"), name: "old" }
branch_info["new"] = { desc: "Class lex", time: Time.now, file: dir.join("new.txt"), name: "new" }
stats = DerailedBenchmarks::StatsFromDir.new(branch_info)
stats.call.banner
```
Which gave us:
```
❤️ ❤️ ❤️ (Statistically Significant) ❤️ ❤️ ❤️
[new] (3.3139 seconds) "Class lex" ref: "new"
FASTER 🚀🚀🚀 by:
1.1046x [older/newer]
9.4700% [(older - newer) / older * 100]
[old] (3.6606 seconds) "Struct lex" ref: "old"
Iterations per sample:
Samples: 51
Test type: Kolmogorov Smirnov
Confidence level: 99.0 %
Is significant? (max > critical): true
D critical: 0.30049534876137013
D max: 0.9607843137254902
Histograms (time ranges are in seconds):
[new] description: [old] description:
"Class lex" "Struct lex"
┌ ┐ ┌ ┐
[3.0, 3.3) ┤▇ 1 [3.0, 3.3) ┤ 0
[3.3, 3.6) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 47 [3.3, 3.6) ┤ 0
[3.5, 3.8) ┤▇▇ 2 [3.5, 3.8) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 46
[3.8, 4.1) ┤▇ 1 [3.8, 4.1) ┤▇▇▇ 4
[4.0, 4.3) ┤ 0 [4.0, 4.3) ┤ 0
[4.3, 4.6) ┤ 0 [4.3, 4.6) ┤▇ 1
└ ┘ └ ┘
# of runs in range # of runs in range
```
To sum this up, the "new" version of this code (using real classes instead of structs) is 10% faster across 50 runs with a statistical significance confidence level of 99%. Histograms are for visual checksum.
Notes:
Merged: https://github.com/ruby/ruby/pull/5093
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/5174
|
|
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/4909
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/4371
|
|
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/4083
|
|
|
|
sorting alias target by event arity, and setup suitable `Elem` for error.
Notes:
Merged: https://github.com/ruby/ruby/pull/3936
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/3909
|
|
|
|
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/3797
|
|
|
|
yet another implements [Feature #17276]
|
|
|
|
Implements [Feature #17276]
Notes:
Merged: https://github.com/ruby/ruby/pull/3774
Merged-By: jeremyevans <code@jeremyevans.net>
|
|
To make some kind of Ractor related extensions, some functions
should be exposed.
* include/ruby/thread_native.h
* rb_native_mutex_*
* rb_native_cond_*
* include/ruby/ractor.h
* RB_OBJ_SHAREABLE_P(obj)
* rb_ractor_shareable_p(obj)
* rb_ractor_std*()
* rb_cRactor
and rm ractor_pub.h
and rename srcdir/ractor.h to srcdir/ractor_core.h
(to avoid conflict with include/ruby/ractor.h)
Notes:
Merged: https://github.com/ruby/ruby/pull/3775
|
|
This commit introduces Ractor mechanism to run Ruby program in
parallel. See doc/ractor.md for more details about Ractor.
See ticket [Feature #17100] to see the implementation details
and discussions.
[Feature #17100]
This commit does not complete the implementation. You can find
many bugs on using Ractor. Also the specification will be changed
so that this feature is experimental. You will see a warning when
you make the first Ractor with `Ractor.new`.
I hope this feature can help programmers from thread-safety issues.
Notes:
Merged: https://github.com/ruby/ruby/pull/3365
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/3347
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/3346
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/3338
|
|
|
|
To fix build failures.
Notes:
Merged: https://github.com/ruby/ruby/pull/3079
|
|
This shall fix compile errors.
Notes:
Merged: https://github.com/ruby/ruby/pull/3079
|
|
* Folding results should not be empty.
If `OnigCodePointCount(to->n)` were 0, `for` loop using `fn`
wouldn't execute and `ncs` elements are not initialized.
```
enc/unicode.c:557:21: warning: 'ncs[0]' may be used uninitialized in this function [-Wmaybe-uninitialized]
557 | for (i = 0; i < ncs[0]; i++) {
| ~~~^~~
```
* Cast to `enum yytokentype`
Additional enums for scanner events by ripper are not included
in `yytokentype`.
```
ripper.y:7274:28: warning: implicit conversion from 'enum <anonymous>' to 'enum yytokentype' [-Wenum-conversion]
```
|
|
Split ruby.h
Notes:
Merged-By: shyouhei <shyouhei@ruby-lang.org>
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/2909
|
|
Use `"\n"` and `IO#fileno` instead of `$/` and `$.` respectively.
[Feature #14240]
|
|
patch from https://travis-ci.org/ruby/ruby/jobs/638231960
|
|
Needed for `UNALIGNED_MEMBER_ACCESS` using `COMPILER_WARNING_`*
macros.
Notes:
Merged: https://github.com/ruby/ruby/pull/2799
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/2711
|
|
|
|
|
|
This is follow up of 1f7cb4bee9.
|
|
This reverts commit 67c574736912003c377218153f9d3b9c0c96a17b.
[Feature #16275]
|
|
I noticed that some files in rubygems were executable, and I could think
of no reason why they should be.
In general, I think ruby files should never have the executable bit set
unless they include a shebang, so I run the following command over the
whole repo:
```bash
find . -name '*.rb' -type f -executable -exec bash -c 'grep -L "^#!" $1 || chmod -x $1' _ {} \;
```
Notes:
Merged: https://github.com/ruby/ruby/pull/2662
|
|
|
|
Tabs were expanded because previously the file did not have any tab indentation.
Please update your editor config, and use misc/expand_tabs.rb in the pre-commit hook.
|
