summaryrefslogtreecommitdiff
path: root/lib/bundler/fetcher
AgeCommit message (Collapse)Author
2025-12-04[ruby/rubygems] Increase connection pool to allow for up to 70% speed increase:Edouard CHIN
- ### TL;DR Bundler is heavily limited by the connection pool which manages a single connection. By increasing the number of connection, we can drastiscally speed up the installation process when many gems need to be downloaded and installed. ### Benchmark There are various factors that are hard to control such as compilation time and network speed but after dozens of tests I can consistently get aroud 70% speed increase when downloading and installing 472 gems, most having no native extensions (on purpose). ``` # Before bundle install 28.60s user 12.70s system 179% cpu 23.014 total # After bundle install 30.09s user 15.90s system 281% cpu 16.317 total ``` You can find on this gist how this was benchmarked and the Gemfile used https://gist.github.com/Edouard-chin/c8e39148c0cdf324dae827716fbe24a0 ### Context A while ago in #869, Aaron introduced a connection pool which greatly improved Bundler speed. It was noted in the PR description that managing one connection was already good enough and it wasn't clear whether we needed more connections. Aaron also had the intuition that we may need to increase the pool for downloading gems and he was right. > We need to study how RubyGems uses connections and make a decision > based on request usage (e.g. only use one connection for many small > requests like bundler API, and maybe many connections for > downloading gems) When bundler downloads and installs gem in parallel https://github.com/ruby/rubygems/blob/4f85e02fdd89ee28852722dfed42a13c9f5c9193/bundler/lib/bundler/installer/parallel_installer.rb#L128 most threads have to wait for the only connection in the pool to be available which is not efficient. ### Solution This commit modifies the pool size for the fetcher that Bundler uses. RubyGems fetcher will continue to use a single connection. The bundler fetcher is used in 2 places. 1. When downloading gems https://github.com/ruby/rubygems/blob/4f85e02fdd89ee28852722dfed42a13c9f5c9193/bundler/lib/bundler/source/rubygems.rb#L481-L484 2. When grabing the index (not the compact index) using the `bundle install --full-index` flag. https://github.com/ruby/rubygems/blob/4f85e02fdd89ee28852722dfed42a13c9f5c9193/bundler/lib/bundler/fetcher/index.rb#L9 Having more connections in 2) is not any useful but tweaking the size based on where the fetcher is used is a bit tricky so I opted to modify it at the class level. I fiddle with the pool size and found that 5 seems to be the sweet spot at least for my environment. https://github.com/ruby/rubygems/commit/6063fd9963
2025-09-16[rubygems/rubygems] Remove `allow_offline_install` settingDavid Rodríguez
And let the feature always be enabled, so I'm not sure why we'd need this configurable. https://github.com/rubygems/rubygems/commit/5a27f0c1e3
2025-07-02[rubygems/rubygems] Add Errno::ENETDOWN and Errno::EHOSTUNREACH to non ↵David Rodríguez
retryable errors Connection errors as well, so useless to retry. https://github.com/rubygems/rubygems/commit/d2d211651a
2025-07-02[rubygems/rubygems] Split HTTP_ERRORS into retryable and non retryableDavid Rodríguez
https://github.com/rubygems/rubygems/commit/c241a640fc
2025-07-02[rubygems/rubygems] Handle connection refused and Errno::EADDRNOTAVAIL as ↵David Rodríguez
non-retryable https://github.com/rubygems/rubygems/commit/cd529776d5
2025-07-02[rubygems/rubygems] Move Bundler::Fetcher::HTTP_ERRORS to ↵David Rodríguez
Bundler::Fetcher::DOWNLOADER And deprecate the old constant. It's only used in this class, and in Bundler::Fetcher there's already FAIL_ERRORS, very similar to it. So this makes things less confusing. https://github.com/rubygems/rubygems/commit/d32ed63d6f
2025-05-13Handle to look up CGI::EscapeExt instead of using LoadError. cgi/escape is ↵Hiroshi SHIBATA
provided snce Ruby 2.3 Notes: Merged: https://github.com/ruby/ruby/pull/13311
2025-05-09Use cgi/escape instead of cgi/utilHiroshi SHIBATA
Notes: Merged: https://github.com/ruby/ruby/pull/13275
2025-04-24Use cgi/util if that uses like CGI.escape methodsHiroshi SHIBATA
2024-10-04[rubygems/rubygems] Update required_ruby_version to 3.1Samuel Giddins
3.0 has been EOL since march, drop support for it before the 3.4 release is cut Signed-off-by: Samuel Giddins <segiddins@segiddins.me> https://github.com/rubygems/rubygems/commit/fc1f03b06a
2024-05-31[rubygems/rubygems] Move compact index concurrency to fetcherMartin Emde
https://github.com/rubygems/rubygems/commit/ffd3711d00
2024-05-25[rubygems/rubygems] Reorganize and refactor CompactIndexClientMartin Emde
https://github.com/rubygems/rubygems/commit/71bcf354f5
2024-01-29[rubygems/rubygems] Use rubygems vendored uri from Bundler when availableDavid Rodríguez
https://github.com/rubygems/rubygems/commit/5d6a8f2fb4
2023-12-13[rubygems/rubygems] Use vendored net-http in BundlerDavid Rodríguez
https://github.com/rubygems/rubygems/commit/0d758e8926
2023-11-27Opaque Etags for compact index requestsJosef Šimánek
This changes the CompactIndexClient to store etags received from the compact index in separate files rather than relying on the MD5 checksum of the file as the etag. Smoothes the upgrade from md5 etags to opaque by generating them when no etag file exists. This should reduce the initial impact of changing the caching behavior by reducing cache misses when the MD5 etag is the same. Eventually, the MD5 behavior should be retired and the etag should be considered completely opaque with no assumption that MD5 would match.
2023-11-15[rubygems/rubygems] User bundler UA when downloading gemsSamuel Giddins
Gem::RemoteFetcher uses Gem::Request, which adds the RubyGems UA. Gem::RemoteFetcher is used to download gems, as well as the full index. We would like the bundler UA to be used whenever bundler is making requests. This PR also avoids unsafely mutating the headers hash on the shared `Gem::RemoteFetcher.fetcher` instance, which could cause corruption or incorrect headers when making parallel requests. Instead, we create one remote fetcher per rubygems remote, which is similar to the connection segregation bundler is already doing https://github.com/rubygems/rubygems/commit/f0e8dacdec
2023-09-19[rubygems/rubygems] Lazily construct fetcher debug messagesSamuel Giddins
Avoids constructing several strings https://github.com/rubygems/rubygems/commit/8a322dbe11
2023-08-28[rubygems/rubygems] Refactor Fetcher#api_fetcher? and fetcher loading logicMartin Emde
https://github.com/rubygems/rubygems/commit/f664d60114
2023-08-16[rubygems/rubygems] Show better error when PAT can't authenticate to a ↵David Rodríguez
private server Before: ``` Fetching gem metadata from https://rubygems.org/........ Fetching source index from https://rubygems.pkg.github.com/my-org/ Bad username or password for https://x-access-token@rubygems.pkg.github.com/my-org/. Please double-check your credentials and correct them. ``` After: ``` Fetching gem metadata from https://rubygems.org/........ Fetching source index from https://rubygems.pkg.github.com/my-org/ Access token could not be authenticated for https://x-access-token@rubygems.pkg.github.com/my-org/. Make sure it's valid and has the necessary scopes configured. ``` https://github.com/rubygems/rubygems/commit/2ae69c964a
2023-08-16[rubygems/rubygems] Bad auth should not fallback eitherDavid Rodríguez
https://github.com/rubygems/rubygems/commit/ba3ea27869
2023-06-16[rubygems/rubygems] Auto-correct Performance/FlatMapHiroshi SHIBATA
https://github.com/rubygems/rubygems/commit/b696edcd73
2023-03-23Fix logging the fallback to the full index on GemspecError'sDavid Rodríguez
The debug message suggests retrying using `--full-index`, but the retry is happening automatically. Just log that we are falling back to the full index, like we do with other errors. Notes: Merged: https://github.com/ruby/ruby/pull/7582
2022-12-26Merge RubyGems/Bundler masterHiroshi SHIBATA
from https://github.com/rubygems/rubygems/commit/72fd3dd2096af16d797ad0cd8e0d2a8869e240b3 Notes: Merged: https://github.com/ruby/ruby/pull/7025
2022-12-20[rubygems/rubygems] Use safe Marshal deserialization for dependency API ↵Josef Šimánek
response. - adds Bundler#safe_load_marshal and Bundler::SAFE_MARSHAL_CLASSES listing safe classes to deserialize https://github.com/rubygems/rubygems/commit/e947c608cc Notes: Merged: https://github.com/ruby/ruby/pull/6966
2022-12-15Merge RubyGems/Bundler masterHiroshi SHIBATA
Pick from https://github.com/rubygems/rubygems/commit/084f7d1f21f6fc3e2bb685b7bda3653fb2891c6e Notes: Merged: https://github.com/ruby/ruby/pull/6936
2022-12-12Merge RubyGems/Bundler masterHiroshi SHIBATA
from https://github.com/rubygems/rubygems/commit/bfb0ae69776069155d2092702bfbb5a12617d85a Notes: Merged: https://github.com/ruby/ruby/pull/6906
2022-08-03Merge rubygems/bundler HEADHiroshi SHIBATA
Pick from https://github.com/rubygems/rubygems/commit/8331e63263081a6aa690d8025d2957f30c4e814a Notes: Merged: https://github.com/ruby/ruby/pull/6209
2022-06-24Sync RubyGems & Bundler with upstream repoDavid Rodríguez
Notes: Merged: https://github.com/ruby/ruby/pull/6054
2022-01-19Merge rubygems/rubygems HEAD.Hiroshi SHIBATA
Picked at 12aeef6ba9a3be0022be9934c1a3e4c46a03ed3a Notes: Merged: https://github.com/ruby/ruby/pull/5462
2021-07-07Sync latest bundler & rubygems development versionDavid Rodríguez
Notes: Merged: https://github.com/ruby/ruby/pull/4533
2021-05-11Sync bundler & rubygemsDavid Rodríguez
Notes: Merged: https://github.com/ruby/ruby/pull/4367
2020-12-08Merge prepare version of Bundler 2.2.0Hiroshi SHIBATA
Notes: Merged: https://github.com/ruby/ruby/pull/3864
2020-10-15Merge bundler-2.2.0.rc.2Hiroshi SHIBATA
Notes: Merged: https://github.com/ruby/ruby/pull/3659
2019-12-15Prepare to release bundler-2.1.0Hiroshi SHIBATA
Notes: Merged: https://github.com/ruby/ruby/pull/2753
2019-08-03[bundler/bundler] Use rubygems utility if availableDavid Rodríguez
https://github.com/bundler/bundler/commit/7eccba0e52
2019-08-03[bundler/bundler] Fix file:// handling under WindowsDavid Rodríguez
Windows paths do not start with a slash, so we add an extra slash to separate the host from the path in file:// urls. Otherwise "D:" is parsed as the host segment in the URI. The path for those URLs now starts with "/", so we ignore that leading character when using the URI's path. This reduces Windows CI spec failures from 429 to 355. https://github.com/bundler/bundler/commit/1b7e274cbc
2019-06-09Merge bundler master from upstream.Hiroshi SHIBATA
Pick from 8dd59e3ba97eb80a599f8149f31bf40773b69dc0
2019-04-14Merge Bundler 2.1.0.pre.1 as developed version from upstream.hsbt
https://github.com/bundler/bundler/commit/a53709556b95a914e874b22ed2116a46b0528852 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67539 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-02Added bundler as default gems. Revisit [Feature #12733]hsbt
* bin/*, lib/bundler/*, lib/bundler.rb, spec/bundler, man/*: Merge from latest stable branch of bundler/bundler repository and added workaround patches. I will backport them into upstream. * common.mk, defs/gmake.mk: Added `test-bundler` task for test suite of bundler. * tool/sync_default_gems.rb: Added sync task for bundler. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65509 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-22Postponing the Bundler merge.hsbt
I faced a big issue about Bundler with ruby core. I have no time to resolve it issue before 2.5 final release. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61416 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-11-01Update bundled bundler to 1.16.0.hsbt
* lib/bundler, spec/bundler: Merge bundler-1.16.0. * common.mk: rspec examples of bundler-1.16.0 needs require option. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60603 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-09-08Merge bundler to standard libraries.hsbt
rubygems 2.7.x depends bundler-1.15.x. This is preparation for rubygems and bundler migration. * lib/bundler.rb, lib/bundler/*: files of bundler-1.15.4 * spec/bundler/*: rspec examples of bundler-1.15.4. I applied patches. * https://github.com/bundler/bundler/pull/6007 * Exclude not working examples on ruby repository. * Fake ruby interpriter instead of installed ruby. * Makefile.in: Added test task named `test-bundler`. This task is only working macOS/linux yet. I'm going to support Windows environment later. * tool/sync_default_gems.rb: Added sync task for bundler. [Feature #12733][ruby-core:77172] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59779 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-19 11:17:32 +0000