| Age | Commit message (Collapse) | Author |
|---|
|
dependencies
I am seeing the following error during bundle install:
```
Gem::MissingSpecError: Could not find 'ffi' (>= 1.15.5) among 48 total gem(s) (Gem::MissingSpecError)
```
This is reproducible with:
```ruby
source 'https://rubygems.org'
gem 'llhttp-ffi'
```
It seems only direct dependencies are checked when determining whether
a Gem with native extensions is ready to install. I believe this can
lead to a failure if a transitive dependency is not yet installed.
In the example above, llhttp-ffi depends on ffi-compiler, which depends
on ffi. Since ffi-compiler has no extensions, it is installed
immediately without waiting for ffi. When llhttp-ffi then checks its
direct dependencies, ffi-compiler is already installed, so llhttp-ffi
starts building its native extension. The build requires ffi, which may
not have been installed yet.
https://github.com/ruby/rubygems/commit/a1f5751177
|
|
- ### Problem
In https://github.com/ruby/rubygems/pull/9381, I explained the
issue about the "tail latency".
TL;DR When a gem with a native extensions is downloaded, the sooner
we starts compiling, the sooner it gets installed.
If a gem with a native extensions ends up at the end of the queue,
the longer `bundle install` becomes.
### Solution
I'd like to introduce a simple queue with priority. When a gem is
downloaded, we check whether that gem has a native extension and
if its dependencies are installed. If both conditions are met,
we add the gem in the priority queue to be picked up as quickly
as possible.
https://github.com/ruby/rubygems/commit/c271257444
|
|
- ### Problem
Bundler awaits for the dependencies of a gem to be download and installed
before it proceeds to downloading and installing the dependency itself.
This creates a bottleneck at the end of the installation process and
block a thread unnecessarily.
### Details
The installation strategy in Bundler is to await for "leaf gems" to
be download/installed before the "root gem" is processed.
For instance, in this scenario:
- Gem "foo" has a dependency on "bar" (We can call this the "root gem")
- Gem "bar" has no dependency (We call cal this the "leaf gems")
- A Gemfile adds "gem 'foo'"
In this case, only a single thread will have work to do, that is
because Bundler will queue the gem "bar" to be downloaded and
installed, and only when "bar" is finished, then Bundler will queue
work for "foo".
For **pure ruby gems**, this strategy is a waste of time because
during the installation, a gem's code is not evaluated and no
"require" statement is evaluated.
For gems with native extensions, this strategy make sense. When the
`extconf.rb` of a gem is evaluated, it's possible that the extconf
requires a dependency and therefore Bundler needs to wait for those
dependencies to be installed before it can execute the extconf.
A typical example is a native extension that require 'mini_portile2'.
### Solution
From the explanation above, I'd like to split the download from
the installation of a gem.
The tricky aspect is that there is no RubyGems API to know whether
a gem has a native extension. The only way to know is after we
download the gem and read the `metadata` from the tarball.
So the solution in this patch is as follow:
1. We download all gems without doing any checks.
2. Once the gems are downloaded, we now know whether a gem has a
native extensions.
3. If a gem is a pure ruby gems, we install it without waiting.
4. If a gem has a native extension, we check whether its dependencies
are installed. If a dependency is not yet installed, we put back
the gem in the queue. It will be dequeued later on and we'll redo
this logic.
### Performance gain
The speed gain highly depends on how deep the dependency tree is.
E.g. bar depends on foo which depends on baz which depends on jane ...
Previously we'd proceed to installing gems just one by one and only
a single thread would be used.
In a freshly generated Rails application, the speed gain is not that
important. And the reason is because we are having a "tail latency"
issue. Around the end of the installation process, the remaining
gems to be installed are the one with native extensions, since we
had to wait for for their dependencies to be installed. Compiling
them is the slowest part, and since we are doing it at the end then
the speed gain is not that noticeable.
### Misc
Another advantage of this change is to be able to more easily
implement this kind of feature https://github.com/ruby/rubygems/issues/9138
to let users solely download gems, without installing them .
https://github.com/ruby/rubygems/commit/e5e8c0a507
|
|
included
https://github.com/rubygems/rubygems/commit/b9a2d4d539
|
|
Also fix running when BUNDLE_NO_INSTALL happens to be set, same as with install/update commands
https://github.com/rubygems/rubygems/commit/a555fd6ccd
|
|
```
==> memprof.after.txt <==
Total allocated: 1.13 MB (2352 objects)
Total retained: 10.08 kB (78 objects)
==> memprof.before.txt <==
Total allocated: 46.27 MB (38439 objects)
Total retained: 9.94 kB (75 objects)
```
Yes, we were allocating 45MB of arrays in `dependencies_installed?`,
it was accidentally cubic.
https://github.com/rubygems/rubygems/commit/13ab874388
|
|
Move the check for unmet dependencies in lockfile just in time to be
able to re-resolve if unmet dependencies are found.
|
|
Following up on https://github.com/rubygems/rubygems/pull/6355, which
turned a crash into a nicer error message, this commit auto-heals the
corrupt lockfile instead.
In this particular case (a corrupt Gemfile.lock with missing
dependencies) the LazySpecification will not have accurate dependency
information, we have to materialize the SpecSet to determine there are
missing dependencies. We've already got a way to handle this, via
`SpecSet#incomplete_specs`, but it wasn't quite working for this case
because we'd get to `@incomplete_specs += lookup[name]` and
`lookup[name]` would be empty for the dependency.
With this commit we catch it a bit earlier, marking the parent spec
containing the missing dependency as incomplete.
https://github.com/rubygems/rubygems/commit/486ecb8f20
|
|
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.
Something similar came up in https://github.com/rubygems/rubygems/issues/6210,
and this commit would have helped with that case as well
(although we've already handled this a different way with #6219).
Details:
---
Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error:
```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```
Bundler 2.3.26 gave a helpful warning:
```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```
But then continued on and crashed while trying to report the unmet
dependency:
```
--- ERROR REPORT TEMPLATE -------------------------------------------------------
NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'
...
```
Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:
```
--- ERROR REPORT TEMPLATE -------------------------------------------------------
fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0
* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0
* #<Thread:0x0000000108dea4a0@Parallel Installer Worker #1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0
<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'
...
```
Changes
---
This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue. When
that happens we treat it as a failure so the install can finish.
That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.
https://github.com/rubygems/rubygems/commit/d73001a21d
|
|
`trying to manually editing` doesn't seem quite grammatically
correct. We could change it to `trying to manually edit` (is that a
split infinitive?), but I don't think `trying to` adds much here so
I've removed it instead so `editing` is the verb.
For the list of dependencies, the wording before this commit seemed to
reverse the dependency. "B, depended on A" sounds like B depends on A
(or did in the past but doesn't anymore?), but that's not correct. I
think there's a missing word: "B, depended on by A", but I find "B,
dependency of A" a bit nicer.
https://github.com/rubygems/rubygems/commit/49a31257e3
|
|
from https://github.com/rubygems/rubygems/commit/72fd3dd2096af16d797ad0cd8e0d2a8869e240b3
Notes:
Merged: https://github.com/ruby/ruby/pull/7025
|
|
We have a check for a corrupt lockfile right before installing. However,
the check accounted for locked specs not satisfying locked dependencies,
but not for locked specs missing for some locked dependencies.
Instead of fixing this check, I decided to remove it in favor of
automatically detecting the situation and re-resolve to automatically
fix the lockfile rather than printing a warning but leave the problem
there.
https://github.com/rubygems/rubygems/commit/4a7a584252
Notes:
Merged: https://github.com/ruby/ruby/pull/6966
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/4383
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/3659
|
|
The main thread may detect that installation has finished and thus
abort, while the thread responsible for installing the spec that failed
has not yet set the error message.
In this case, the install process will abort with a misterious "empty"
exception. I can be force the issue to reproduce by applying the
following patch:
```diff
diff --git a/bundler/lib/bundler/installer/parallel_installer.rb b/bundler/lib/bundler/installer/parallel_installer.rb
index 3dee9f4664..7827a11d11 100644
--- a/bundler/lib/bundler/installer/parallel_installer.rb
+++ b/bundler/lib/bundler/installer/parallel_installer.rb
@@ -166,6 +166,7 @@ module Bundler
spec_install.post_install_message = message unless message.nil?
else
spec_install.state = :failed
+ sleep 1
spec_install.error = "#{message}\n\n#{require_tree_for_spec(spec_install.spec)}"
end
Plugin.hook(Plugin::Events::GEM_AFTER_INSTALL, spec_install)
@@ -183,6 +184,7 @@ module Bundler
end
def finished_installing?
+ sleep 0.5
@specs.all? do |spec|
return true if spec.failed?
spec.installed?
diff --git a/bundler/lib/bundler/rubygems_gem_installer.rb b/bundler/lib/bundler/rubygems_gem_installer.rb
index 8ce33c3953..c585cd517b 100644
--- a/bundler/lib/bundler/rubygems_gem_installer.rb
+++ b/bundler/lib/bundler/rubygems_gem_installer.rb
@@ -42,6 +42,7 @@ module Bundler
return true unless source = @package.instance_variable_get(:@gem)
return true unless source.respond_to?(:with_read_io)
digest = source.with_read_io do |io|
+ raise BundlerError, "asdafss"
digest = SharedHelpers.digest(:SHA256).new
digest << io.read(16_384) until io.eof?
io.rewind
```
and running `bin/rspec spec/install/gems/compact_index_spec.rb:892` will
result in
```
Run options:
include {:locations=>{"./spec/install/gems/compact_index_spec.rb"=>[892]}}
exclude {:jruby=>true, :readline=>false, :permissions=>false, :no_color_tty=>false, :ruby_repo=>false, :bundler=>"!= 2", :git=>"!= 2.26.2", :rubygems=>"!= 3.2.0.pre1", :realworld=>true, :sudo=>true}
Randomized with seed 59277
F
Retried examples: 0
Failures:
1) compact index api checksum validation raises when the checksum is the wrong length
Failure/Error: expect(err).to include("The given checksum for rack-1.0.0 (\"checksum!\") is not a valid SHA256 hexdigest nor base64digest")
expected "" to include "The given checksum for rack-1.0.0 (\"checksum!\") is not a valid SHA256 hexdigest nor base64digest"
Commands:
$ /home/deivid/.rbenv/versions/2.7.1/bin/ruby -I/home/deivid/Code/rubygems/rubygems/bundler/spec -r/home/deivid/Code/rubygems/rubygems/bundler/spec/support/artifice/compact_index_wrong_gem_checksum.rb -r/home/deivid/Code/rubygems/rubygems/bundler/spec/support/hax.rb /home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/bin/bundle install --verbose --retry 0
Running `bundle install --retry 0 --verbose` with bundler 2.2.0.dev
Found changes from the lockfile, re-resolving dependencies because the list of sources changed, the dependencies in your gemfile changed, you added a new platform to your gemfile
HTTP GET http://localgemserver.test/versions
HTTP 200 OK http://localgemserver.test/versions
Fetching gem metadata from http://localgemserver.test/
Looking up gems ["rack"]
HTTP GET http://localgemserver.test/info/rack
HTTP 200 OK http://localgemserver.test/info/rack
Resolving dependencies...
Using bundler 2.2.0.dev
0: bundler (2.2.0.dev) from /home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/specifications/bundler-2.2.0.dev.gemspec
Fetching rack 1.0.0
Installing rack 1.0.0
Bundler::InstallError:
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer/parallel_installer.rb:199:in `handle_error'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer/parallel_installer.rb:102:in `call'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer/parallel_installer.rb:78:in `call'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer.rb:271:in `install_in_parallel'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer.rb:197:in `install'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer.rb:92:in `block in run'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/process_lock.rb:12:in `block in lock'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/process_lock.rb:9:in `open'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/process_lock.rb:9:in `lock'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer.rb:73:in `run'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer.rb:25:in `install'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/cli/install.rb:66:in `run'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/cli.rb:261:in `block in install'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/settings.rb:121:in `temporary'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/cli.rb:260:in `install'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/cli.rb:30:in `dispatch'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/cli.rb:24:in `start'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/exe/bundle:49:in `block in <top (required)>'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/exe/bundle:37:in `<top (required)>'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/bin/bundle:23:in `load'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/bin/bundle:23:in `<main>'
# $? => 5
# ./spec/install/gems/compact_index_spec.rb:892:in `block (3 levels) in <top (required)>'
# ./spec/spec_helper.rb:104:in `block (4 levels) in <top (required)>'
# ./spec/spec_helper.rb:104:in `block (3 levels) in <top (required)>'
# ./spec/support/helpers.rb:352:in `block in with_gem_path_as'
# ./spec/support/helpers.rb:366:in `without_env_side_effects'
# ./spec/support/helpers.rb:348:in `with_gem_path_as'
# ./spec/spec_helper.rb:101:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:73:in `block (2 levels) in <top (required)>'
# ./spec/support/rubygems_ext.rb:90:in `load'
# ./spec/support/rubygems_ext.rb:90:in `gem_load_and_activate'
# ./spec/support/rubygems_ext.rb:18:in `gem_load'
Finished in 3.01 seconds (files took 0.14209 seconds to load)
1 example, 1 failure
Failed examples:
rspec ./spec/install/gems/compact_index_spec.rb:886 # compact index api checksum validation raises when the checksum is the wrong length
Randomized with seed 59277
```
Without any mention to `BundlerError` and the original "asdafss" message.
Fix the issue by making sure the error message is set before the ":failed"
status.
https://github.com/rubygems/rubygems/commit/83c8feb2c4
Notes:
Merged: https://github.com/ruby/ruby/pull/3184
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/3086
|
|
Pick from 8dd59e3ba97eb80a599f8149f31bf40773b69dc0
|
|
https://github.com/bundler/bundler/commit/a53709556b95a914e874b22ed2116a46b0528852
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67539 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* bin/*, lib/bundler/*, lib/bundler.rb, spec/bundler, man/*:
Merge from latest stable branch of bundler/bundler repository and
added workaround patches. I will backport them into upstream.
* common.mk, defs/gmake.mk: Added `test-bundler` task for test suite
of bundler.
* tool/sync_default_gems.rb: Added sync task for bundler.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65509 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
I faced a big issue about Bundler with ruby core.
I have no time to resolve it issue before 2.5 final release.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61416 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* lib/bundler, spec/bundler: Merge bundler-1.16.0.
* common.mk: rspec examples of bundler-1.16.0 needs require option.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60603 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
rubygems 2.7.x depends bundler-1.15.x. This is preparation for
rubygems and bundler migration.
* lib/bundler.rb, lib/bundler/*: files of bundler-1.15.4
* spec/bundler/*: rspec examples of bundler-1.15.4. I applied patches.
* https://github.com/bundler/bundler/pull/6007
* Exclude not working examples on ruby repository.
* Fake ruby interpriter instead of installed ruby.
* Makefile.in: Added test task named `test-bundler`. This task is only
working macOS/linux yet. I'm going to support Windows environment later.
* tool/sync_default_gems.rb: Added sync task for bundler.
[Feature #12733][ruby-core:77172]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59779 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
