| Age | Commit message (Collapse) | Author |
|---|
|
|
|
Discussed in https://github.com/ruby/ruby/pull/5093#issuecomment-964426481.
> it would be enough to mimic only [] for almost all cases
This adds back the `Lexer::Elem#[]` and `Lexer::State#[]` and adds deprecation warnings for them.
|
|
Instead of accessing the struct as an array, access it via methods. There are other places inside of this file already using this API (for example https://github.com/ruby/ruby/blob/e0a5c3d2b71dfad038d7562fdd33f02ffd79232d/lib/irb/ruby-lex.rb#L829-L830).
This commit moves all struct array-ish calls to use their method calls instead. It is also ~1.23 faster accessing values via a method instead of as an array according to this microbenchmark:
```ruby
Elem = Struct.new(:pos, :event, :tok, :state, :message) do
def initialize(pos, event, tok, state, message = nil)
super(pos, event, tok, State.new(state), message)
end
# ...
def to_a
a = super
a.pop unless a.empty?
a
end
end
class ElemClass
attr_accessor :pos, :event, :tok, :state, :message
def initialize(pos, event, tok, state, message = nil)
@pos = pos
@event = event
@tok = tok
@state = State.new(state)
@message = message
end
def to_a
if @message
[@pos, @event, @tok, @state, @message]
else
[@pos, @event, @tok, @state]
end
end
end
# stub state class creation for now
class State; def initialize(val); end; end
```
```ruby
Benchmark.ips do |x|
x.report("struct") { struct[1] }
x.report("class ") { from_class.event }
x.compare!
end; nil
```
```
Warming up --------------------------------------
struct 1.624M i/100ms
class 1.958M i/100ms
Calculating -------------------------------------
struct 17.139M (± 2.6%) i/s - 86.077M in 5.025801s
class 21.104M (± 3.4%) i/s - 105.709M in 5.015193s
Comparison:
class : 21103826.3 i/s
struct: 17139201.5 i/s - 1.23x (± 0.00) slower
```
Notes:
Merged: https://github.com/ruby/ruby/pull/5093
|
|
The last element in the `@buf` may be either an array or an `Elem`. In the case it is an `Elem` we iterate over every element, when we do not need to. This check guards that case by ensuring that we only iterate over an array of elements.
Notes:
Merged: https://github.com/ruby/ruby/pull/5093
|
|
## Concept
I am proposing we replace the Struct implementation of data structures inside of ripper with real classes.
This will improve performance and the implementation is not meaningfully more complicated.
## Example
Struct versus class comparison:
```ruby
Elem = Struct.new(:pos, :event, :tok, :state, :message) do
def initialize(pos, event, tok, state, message = nil)
super(pos, event, tok, State.new(state), message)
end
# ...
def to_a
a = super
a.pop unless a.empty?
a
end
end
class ElemClass
attr_accessor :pos, :event, :tok, :state, :message
def initialize(pos, event, tok, state, message = nil)
@pos = pos
@event = event
@tok = tok
@state = State.new(state)
@message = message
end
def to_a
if @message
[@pos, @event, @tok, @state, @message]
else
[@pos, @event, @tok, @state]
end
end
end
# stub state class creation for now
class State; def initialize(val); end; end
```
## MicroBenchmark creation
```ruby
require 'benchmark/ips'
require 'ripper'
pos = [1, 2]
event = :on_nl
tok = "\n".freeze
state = Ripper::EXPR_BEG
Benchmark.ips do |x|
x.report("struct") { Elem.new(pos, event, tok, state) }
x.report("class ") { ElemClass.new(pos, event, tok, state) }
x.compare!
end; nil
```
Gives ~1.2x faster creation:
```
Warming up --------------------------------------
struct 263.983k i/100ms
class 303.367k i/100ms
Calculating -------------------------------------
struct 2.638M (± 5.9%) i/s - 13.199M in 5.023460s
class 3.171M (± 4.6%) i/s - 16.078M in 5.082369s
Comparison:
class : 3170690.2 i/s
struct: 2638493.5 i/s - 1.20x (± 0.00) slower
```
## MicroBenchmark `to_a` (Called by Ripper.lex for every element)
```ruby
require 'benchmark/ips'
require 'ripper'
pos = [1, 2]
event = :on_nl
tok = "\n".freeze
state = Ripper::EXPR_BEG
struct = Elem.new(pos, event, tok, state)
from_class = ElemClass.new(pos, event, tok, state)
Benchmark.ips do |x|
x.report("struct") { struct.to_a }
x.report("class ") { from_class.to_a }
x.compare!
end; nil
```
Gives 1.46x faster `to_a`:
```
Warming up --------------------------------------
struct 612.094k i/100ms
class 893.233k i/100ms
Calculating -------------------------------------
struct 6.121M (± 5.4%) i/s - 30.605M in 5.015851s
class 8.931M (± 7.9%) i/s - 44.662M in 5.039733s
Comparison:
class : 8930619.0 i/s
struct: 6121358.9 i/s - 1.46x (± 0.00) slower
```
## MicroBenchmark data access
```ruby
require 'benchmark/ips'
require 'ripper'
pos = [1, 2]
event = :on_nl
tok = "\n".freeze
state = Ripper::EXPR_BEG
struct = Elem.new(pos, event, tok, state)
from_class = ElemClass.new(pos, event, tok, state)
Benchmark.ips do |x|
x.report("struct") { struct.pos[1] }
x.report("class ") { from_class.pos[1] }
x.compare!
end; nil
```
Gives ~1.17x faster data access:
```
Warming up --------------------------------------
struct 1.694M i/100ms
class 1.868M i/100ms
Calculating -------------------------------------
struct 16.149M (± 6.8%) i/s - 81.318M in 5.060633s
class 18.886M (± 2.9%) i/s - 95.262M in 5.048359s
Comparison:
class : 18885669.6 i/s
struct: 16149255.8 i/s - 1.17x (± 0.00) slower
```
## Full benchmark integration of this inside of Ripper.lex
Inside of this repo with this commit
```
$ cd ext/ripper
$ make
$ cat test.rb
file = File.join(__dir__, "../../array.rb")
source = File.read(file)
bench = Benchmark.measure do
10_000.times.each do
Ripper.lex(source)
end
end
puts bench
```
Then execute with and without this change 50 times:
```
rm new.txt
rm old.txt
for i in {0..50}
do
`ruby -Ilib -rripper -rbenchmark ./test.rb >> new.txt`
`ruby -rripper -rbenchmark ./test.rb >> old.txt`
done
```
I used derailed benchmarks internals to compare the results:
```
dir = Pathname(".")
branch_info = {}
branch_info["old"] = { desc: "Struct lex", time: Time.now, file: dir.join("old.txt"), name: "old" }
branch_info["new"] = { desc: "Class lex", time: Time.now, file: dir.join("new.txt"), name: "new" }
stats = DerailedBenchmarks::StatsFromDir.new(branch_info)
stats.call.banner
```
Which gave us:
```
❤️ ❤️ ❤️ (Statistically Significant) ❤️ ❤️ ❤️
[new] (3.3139 seconds) "Class lex" ref: "new"
FASTER 🚀🚀🚀 by:
1.1046x [older/newer]
9.4700% [(older - newer) / older * 100]
[old] (3.6606 seconds) "Struct lex" ref: "old"
Iterations per sample:
Samples: 51
Test type: Kolmogorov Smirnov
Confidence level: 99.0 %
Is significant? (max > critical): true
D critical: 0.30049534876137013
D max: 0.9607843137254902
Histograms (time ranges are in seconds):
[new] description: [old] description:
"Class lex" "Struct lex"
┌ ┐ ┌ ┐
[3.0, 3.3) ┤▇ 1 [3.0, 3.3) ┤ 0
[3.3, 3.6) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 47 [3.3, 3.6) ┤ 0
[3.5, 3.8) ┤▇▇ 2 [3.5, 3.8) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 46
[3.8, 4.1) ┤▇ 1 [3.8, 4.1) ┤▇▇▇ 4
[4.0, 4.3) ┤ 0 [4.0, 4.3) ┤ 0
[4.3, 4.6) ┤ 0 [4.3, 4.6) ┤▇ 1
└ ┘ └ ┘
# of runs in range # of runs in range
```
To sum this up, the "new" version of this code (using real classes instead of structs) is 10% faster across 50 runs with a statistical significance confidence level of 99%. Histograms are for visual checksum.
Notes:
Merged: https://github.com/ruby/ruby/pull/5093
|
|
suseconds_t, which is the type of tv_usec, may be defined with a longer
size type than tv_nsec's type (long). So usec to nsec conversion needs
an explicit casting.
Notes:
Merged: https://github.com/ruby/ruby/pull/5200
|
|
|
|
The methods aren't called for FIXNUM, and it's best to have
consistent behavior.
Fixes [Bug #18377]
Notes:
Merged: https://github.com/ruby/ruby/pull/5199
|
|
|
|
The main impetus for this change is to fix [Bug #13392]. Previously, we
fired the "return" TracePoint event after popping the stack frame for
the block running as method (BMETHOD). This gave undesirable source
location outputs as the return event normally fires right before the
frame going away.
The iseq for each block can run both as a block and as a method. To
accommodate that, this commit makes vm_trace() fire call/return events for
instructions that have b_call/b_return events attached when the iseq is
running as a BMETHOD. The logic for rewriting to "trace_*" instruction
is tweaked so that when the user listens to call/return events,
instructions with b_call/b_return become trace variants.
To continue to provide the return value for non-local returns done using
the "return" or "break" keyword inside BMETHODs, the stack unwinding
code is tweaked. b_return events now provide the same return value as
return events for these non-local cases. A pre-existing test deemed not
providing a return value for these b_return events as a limitation.
This commit removes the checks for call/return TracePoint events that
happen when calling into BMETHODs when no TracePoints are active.
Technically, migrating just the return event is enough to fix the bug,
but migrating both call and return removes our reliance on
`VM_FRAME_FLAG_FINISH` and re-entering the interpreter when the caller
is already in the interpreter.
Notes:
Merged: https://github.com/ruby/ruby/pull/4637
|
|
This reverts commit 119626da947bf6492ef7a27abf3bf12de5d0d95a.
|
|
If YJIT isn't enabled, or hasn't finished booting, cb / ocb could be
null. This commit just checks to make sure they're available before
marking as executable
Co-Authored-By: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
Co-Authored-By: Kevin Newton <kddnewton@gmail.com>
Notes:
Merged: https://github.com/ruby/ruby/pull/5032
|
|
Some platforms don't want memory to be marked as writeable and
executable at the same time. When we write to the code block, we
calculate the OS page that the buffer position maps to. Then we call
`mprotect` to allow writes on that particular page. As an optimization,
we cache the "last written" aligned page which allows us to amortize the
cost of the `mprotect` call. In other words, sequential writes to the
same page will only call `mprotect` on the page once.
When we're done writing, we call `mprotect` on the entire JIT buffer.
This means we don't need to keep track of which pages were marked as
writeable, we let the OS take care of that.
Co-authored-by: John Hawthorn <john@hawthorn.email>
Notes:
Merged: https://github.com/ruby/ruby/pull/5032
|
|
It seems that since ruby openssl 2.1.0 [[1]], the distinguished name
submitted to `OpenSSL::X509::Name.parse` is not correctly parsed if it
does not contain the first slash:
~~~
$ ruby -v
ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-linux]
$ gem list | grep openssl
openssl (default: 2.2.0)
$ irb -r openssl
irb(main):001:0> OpenSSL::X509::Name.parse("CN=nobody/DC=example").to_s(OpenSSL::X509::Name::ONELINE)
=> "CN = nobody/DC=example"
irb(main):002:0> OpenSSL::X509::Name.parse("/CN=nobody/DC=example").to_s(OpenSSL::X509::Name::ONELINE)
=> "CN = nobody, DC = example"
~~~
Instead, use `OpenSSL::X509::Name.new` directly as suggested by upstream
maintainer.
[1]: https://github.com/ruby/openssl/commit/19c67cd10c57f3ab7b13966c36431ebc3fdd653b
https://github.com/rubygems/rubygems/commit/09ca0c2dae
Co-authored-by: Kazuki Yamaguchi <k@rhe.jp>
|
|
|
|
Previously, YJIT crashes with rb_bug() when asked to compile new methods
while out of executable memory.
To handle this situation gracefully, this change keeps track of all the
blocks compiled each invocation in case YJIT runs out of memory in the
middle of a compliation sequence. The list is used to free all blocks in
case compilation fails.
yjit_gen_block() is renamed to gen_single_block() to make it distinct from
gen_block_version(). Call to limit_block_version() and block_t
allocation is moved into the function to help tidy error checking in the
outer loop.
limit_block_version() now returns by value. I feel that an out parameter
with conditional mutation is unnecessarily hard to read in code that
does not need to go for last drop performance. There is a good chance
that the optimizer is able to output identical code anyways.
Notes:
Merged: https://github.com/ruby/ruby/pull/5191
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/5196
|
|
On Windows, as the input from console is encoded in the active
code page, convert the input to the internal encoding.
Notes:
Merged: https://github.com/ruby/ruby/pull/5196
|
|
|
|
|
|
|
|
bundler/lib/bundler/templates/newgem/github/workflows/main.yml.tt
https://github.com/rubygems/rubygems/commit/8836fe157b
Co-authored-by: David Rodríguez <deivid.rodriguez@riseup.net>
|
|
https://github.com/rubygems/rubygems/commit/3260173c59
|
|
https://github.com/rubygems/rubygems/commit/f5bead5634
|
|
This change avoids a YAML Float-to-String conversion, which turns a 3.0 into a "3". That can make names of builds less clear.
In order to use this new capability, I added a "name" descriptor to the matrix-created Job.
https://github.com/rubygems/rubygems/commit/6221241ad4
|
|
concurrently
When bundler parallel installer installs gems concurrently, one can get
confusing warnings like the following:
```
"[/home/runner/work/rubygems/rubygems/bundler/tmp/2/gems/system/specifications/zeitwerk-2.4.2.gemspec] isn't a Gem::Specification (NilClass instead).
```
I've got these warnings several times in the past, but I never managed
to reproduce them, and never look deeply into the root cause, but this
time a got a cause that reproduced quite frequently, so I looked into
it.
The problem is one thread reading a gemspec while another thread is
writing it. The write of the gemspec was not protected, so
`Gem::Specification.load` could end up seeing a truncated gemspec and
thus throw this warning.
The fix involve two changes:
* Change the methods that write gemspecs to use `Gem.binary_write` which
is protected by a lock.
* Fix `Gem.binary_write` to create the file lock at file creation time,
not when the file already exists after.
The realworld user problem caused by this issue happens in bundler, but
I'm fixing it in rubygems first, and then I'll backport to bundler
whatever needs backporting to fix the issue on the bundler side.
https://github.com/rubygems/rubygems/commit/a672e7555c
|
|
This reverts commit af604436d8141c34cb2e1e645b9b0d47bfd55a55.
The issue that led to introducing it was never reproduced. I tried to
repro with this patch and it still works just fine. Since this removal
is getting in the middle for some race conditions I'm facing, I'm
reverting the patch.
https://github.com/rubygems/rubygems/commit/2dd267f0e4
|
|
The current `setup_base_installer` ends up using the `quick_gem` helper,
which leaves the created specification installed. Instead, make sure to
use the `util_spec` helper, which does a similar thing but doesn't leave
the specification installed.
The idea is that tests do not rely on the installer removing existing
gemspecs, bacause I plan to stop doing that.
https://github.com/rubygems/rubygems/commit/843f1a0abc
|
|
The madvise() declaration should always be compiled on Solaris
to check whether the declaration is good on the environment.
For the purpose, the #if line is unnecessary.
(There was also a trivial typo that the #if was not closed
by #endif and the check always failed with preprocessor error.)
|
|
For older pp.rb which did not need io/console, and dealing with
`LoadError`.
|
|
At least, Error::ENOTSUP may be raised on some extreme environments
https://github.com/ruby/error_highlight/commit/2787983ff7
|
|
According to nobu, Errno::EBAD is raised on Windows.
|
|
The error is raised on Solaris
http://rubyci.s3.amazonaws.com/solaris10-gcc/ruby-master/log/20211130T030003Z.fail.html.gz
```
1) Failure:
TestRubyOptions#test_require [/export/home/users/chkbuild/cb-gcc/tmp/build/20211130T030003Z/ruby/test/ruby/test_rubyoptions.rb:265]:
pid 7386 exit 1
| /export/home/users/chkbuild/cb-gcc/tmp/build/20211130T030003Z/ruby/lib/pp.rb:67:in `winsize': Invalid argument - <STDOUT> (Errno::EINVAL)
```
|
|
Something goes wrong at loading libraries inside `mu_pp` in the
test overriding `Class.inherited`.
|
|
|
|
|
|
[Feature #12913]
|
|
|
|
SunC
```
"cont.c", line 24: identifier redeclared: madvise
current : function(pointer to char, unsigned int, int) returning int
previous: function(pointer to void, unsigned int, int) returning int : "/usr/include/sys/mman.h", line 232
```
GCC
```
cont.c:24:12: error: conflicting types for 'madvise'
24 | extern int madvise(caddr_t, size_t, int);
| ^~~~~~~
In file included from cont.c:16:
/usr/include/sys/mman.h:232:12: note: previous declaration of 'madvise' was here
232 | extern int madvise(void *, size_t, int);
| ^~~~~~~
```
|
|
The result may increase actually or not, since GC can finish
shorter than the timer granularity.
Notes:
Merged: https://github.com/ruby/ruby/pull/5193
Merged-By: nobu <nobu@ruby-lang.org>
|
|
https://github.com/rubygems/rubygems/commit/3ede1435ea
|
|
|
|
This reverts commit 27fb9d272daaae89089dfb61849ebe8e7aa6c833.
The test failure on Solaris 10 is due to incomplete IPv6 configuration
on the CI server, that have already been fixed.
Reference for the fix: https://centrify.force.com/support/Article/KB-1179-X11-Forwarding-fails-with-Centrify-OpenSSH-5-0-Solaris/
|
|
On Solaris, madvise(3C) is NOT defined for SUS (XPG4v2) or later,
but MADV_* macros are defined when __EXTENSIONS__ is defined.
This may cause compile error on Solaris 10 with GCC when
"-Werror=implicit-function-declaration" and "-D_XOPEN_SOURCE=600"
are added by configure.
|
|
|
|
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/5189
|
|
https://github.com/rubygems/rubygems/commit/62d54cbf08
|
|
RUBY_PLATFORM can be used since commit 576b2e64cdc5ea42ad345dd3c1c215e006c06fca .
Notes:
Merged: https://github.com/ruby/ruby/pull/5168
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/5188
|
