| Age | Commit message (Collapse) | Author |
|---|
|
I would like to start making some of the methods in Gem::Package
private so that we can refactor them better. Right now we have many
methods that are public, and since they are public we can't refactor
them. Historically, I think "private" methods have just been tagged
with :nodoc:, but I would like to be more strict about our APIs
https://github.com/ruby/rubygems/commit/fb352e9176
|
|
`normalize_path` is a pretty hot path, it's called many times per file
in each gem. Since the platform isn't going to change from call to call,
we can conditionally define `normalize_path` based on the value of
`Gem.win_platform?`.
https://github.com/rubygems/rubygems/commit/d5e61411f2
|
|
When we're extracting a gem, it should be extracted to an empty
directory. Trying to remove every file before extracting the file
greatly slows the tar extraction process.
This change increases tar extraction speed by about 10%:
Master branch:
```
aaron@tc ~/g/tartest (main)> ruby -I../rubygems/lib:../vernier/lib test.rb
{MEAN: 0.https://github.com/rubygems/rubygems/commit/670195550005}
{stddev: 0.https://github.com/rubygems/rubygems/commit/029875687861}
{"mib/s": 20.https://github.com/rubygems/rubygems/commit/889425481677}
```
This commit:
```
aaron@tc ~/g/tartest (main)> ruby -I../rubygems/lib:../vernier/lib test.rb
{MEAN: 0.https://github.com/rubygems/rubygems/commit/600223749969}
{stddev: 0.https://github.com/rubygems/rubygems/commit/037224226667}
{"mib/s": 23.https://github.com/rubygems/rubygems/commit/324635189326}
```
https://github.com/rubygems/rubygems/commit/e40a153560
|
|
Before this patch we would use `IO.copy_stream` with the tar entry
object rather than just straight to the IO. That means every time
copy_stream wanted data, we would have to proxy the call.
The reason we did this is because every tar entry object _shares_ the
same IO object, and previous to https://github.com/rubygems/rubygems/commit/8927533b0a47
we would call `IO.copy_stream` _without_ a size. Without passing a
size, copy_stream will just read until there is nothing left to read, so
these proxy object emulate finding "the end of the file" (where "end of
file" means "end of tar chunk"). Without emulating this "end of file"
behavior, copy_stream would just keep reading past the end of the tar
chunk.
However, now that we're passing the size to copy_stream, we can bypass
the proxy object overhead and just use the IO object directly because
copy_stream knows exactly the number of bytes it needs to read and will
stop when it reaches the goal.
https://github.com/rubygems/rubygems/commit/857002c135
|
|
When extracting tar files, the tar header actually knows the exact size
of the file we need to extract. Before this commit, we would read the
file from the tar file until it returned `nil`. We can be a little more
efficient when copying by passing the size to copy_stream
https://github.com/rubygems/rubygems/commit/8927533b0a
|
|
We already have the open file descriptor, so we can avoid the overhead
of resolving the filepath (as well as the overhead inside `FileUtils`)
by just calling `chmod` on the file descriptor itself.
https://github.com/rubygems/rubygems/commit/60c14bbeee
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/13312
|
|
|
|
This gets in the middle if we ever start allowing to build as if using a
different RubyGems version than the one being run.
This could be useful to make `gem rebuild` a little more usable, and
it's already done by Bundler specs which already make this method a noop
when they need this.
I'm not sure forcefully setting this, even if user explicitly specified
something else is helpful.
Since this could potentially prevent gems explicitly setting a constant
RubyGems version from building, I changed the error of incorrect
RubyGems version from a hard error to a warning, since it will start
happening in those cases if we stop overwriting the version.
https://github.com/rubygems/rubygems/commit/45676af80d
|
|
https://github.com/rubygems/rubygems/commit/c4e75b9f74
|
|
files in a gem package.
This is to prevent a malicious gem from causing a denial of service by
including a very large metadata or checksums file,
which is then read into memory in its entirety just by opening the gem package.
This is guaranteed to limit the amount of memory needed, since
gzips (which use deflate streams for compression) have a maximum compression
ratio of 1032:1, so the uncompressed size of the metadata or checksums file
will be at most 1032 times the size of the (limited) amount of data read.
This prevents a gem from causing 500GB of memory to be allocated
to read a 500MB metadata file.
https://github.com/rubygems/rubygems/commit/a596e3c5ec
|
|
https://github.com/rubygems/rubygems/commit/241d0aafcd
|
|
Most of the calls to `FormatError.new` pass `@gem` for the second argument, which has a `path` method.
But in one case—on package.rb:691 in `verify_gz`, the `source` argument is a `String`.
So if there's ever a GZip decode error when attempting to read the contents of the `data.tar.gz` file, instead of reporting the underlying GZip error (which might be something like "unexpected end of file"), we would report instead a NoMethodError coming from package.rb
```
Exception while verifying sorbet-0.5.11301.gem
ERROR: While executing gem ... (NoMethodError)
undefined method `path' for "data.tar.gz":String
@path = source.path
^^^^^
```
There are two ways to fix this:
1. Make `FormatError#initialize` aware of the fact that `source` might sometimes be a `String`
2. Make the call to `FormatError.new` in `verify_gz` pass `@gem` instead of `entry.full_name`.
I've chosen 1 because I think it's more useful to see "unexpected end of file in data.tar.gz" instead of "unexpected end of file in sorbet-0.5.11301.gem." The end of file **is actually** in data.tar.gz, not in the gem file itself, which was decoded successfully.
|
|
https://github.com/rubygems/rubygems/commit/fd5cb7396f
|
|
This reverts commit 01f9766aa05182a7bbdc914a5dcd8a36ebade861.
|
|
|
|
|
|
For now, on a small rails app I have hanging around:
```
==> memprof.after.txt <==
Total allocated: 872.51 MB (465330 objects)
Total retained: 40.48 kB (326 objects)
==> memprof.before.txt <==
Total allocated: 890.79 MB (1494026 objects)
Total retained: 40.40 kB (328 objects)
```
Not a huge difference in memory usage, but it's a drastic improvement
in total number of allocations.
Additionally, this will pay huge dividends once
https://github.com/ruby/zlib/pull/61 is merged, as it will allow us to
completely avoid allocations in the repeated calls to readpartial,
which currently accounts for most of the memory usage shown above.
https://github.com/rubygems/rubygems/commit/f78d45d927
|
|
https://github.com/rubygems/rubygems/commit/bb66253f2c
|
|
https://github.com/rubygems/rubygems/commit/10c26a483d
|
|
https://github.com/rubygems/rubygems/commit/ac3b85bd5e
|
|
https://github.com/rubygems/rubygems/commit/58173ff2ea
|
|
corrupt gem
Gem::Package::TarReader::Entry now raises EOFError or returns nil
appropriately based on Ruby core IO.read and IO.readpartial behavior.
Zlib will respond accordingly by raising Zlib::GzipFile::Error on EOF.
When verifying a gem or extracting contents, raise FormatError similar
to other cases of corrupt gems.
Addresses a bug where Gem::Package would attempt to call size on nil
instead of raising a more descriptive and useful error, leading users
to assume the problem is internal to rubygems.
Remove unused error class TarReader::UnexpectedEOF that was never raised
since the NoMethodError on nil would happen first. Use EOFError instead.
https://github.com/rubygems/rubygems/commit/dc6129644b
|
|
https://github.com/rubygems/rubygems/commit/6d9e8025dc
|
|
|
|
https://github.com/rubygems/rubygems/commit/132a56569d
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/7582
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/7582
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/7582
|
|
https://github.com/rubygems/rubygems/commit/05cc97bdf8
Notes:
Merged: https://github.com/ruby/ruby/pull/7582
|
|
|
|
https://github.com/rubygems/rubygems/commit/80b57da926
|
|
https://github.com/rubygems/rubygems/commit/7c1168c623
|
|
https://github.com/rubygems/rubygems/commit/860669b08a
|
|
https://github.com/rubygems/rubygems/commit/cb554f6eb7
|
|
https://github.com/rubygems/rubygems/commit/0542726fd3
|
|
https://github.com/rubygems/rubygems/commit/d8efd919db
|
|
from https://github.com/rubygems/rubygems/commit/72fd3dd2096af16d797ad0cd8e0d2a8869e240b3
Notes:
Merged: https://github.com/ruby/ruby/pull/7025
|
|
When extracting files from the tarball, a mode is retrieved from
the header. Occasionally you'll encounter a gem that was packaged
on a system whose permission bits result in a value that is larger
than the value that File.chmod will allow (anything >= 2^16). In
that case the extraction fails with a RangeError, which is pretty
esoteric.
If you extract the tarball with the tar and gunzip utilities, the
file permissions end up being just the bottom 16 bits masked off
from the original value. I've mirrored that behavior here. Per the
tar spec:
> Modes which are not supported by the operating system restoring
> files from the archive will be ignored.
I think that basically means what I've done here.
---
This commit also changes the behavior very slightly with regard to
when the chmod is called. Previously it was called while the file
descriptor was still open, but after the write call.
When write flushes, the file permissions are changed to the mode
value from the File.open call, undoing the changes made by
FileUtils.chmod. CRuby appears to flush the buffer after the
chmod call, whereas TruffleRuby flushes before the chmod call.
So the file permissions can change depending on implementation.
Both implementations end up getting the correct file permissions
for the bottom 9 bits (user, group, world), but differ with
regard to the sticky bit in the next 3.
To get consistent behavior, this commit changes it to close the
file descriptor before attempting to chmod anything, which makes
it consistent because the write flushes in both cases.
https://github.com/rubygems/rubygems/commit/22ce076e99
|
|
Pick from https://github.com/rubygems/rubygems/commit/dfbb5a38114640e0d8d616861607f3de73ee0199
Notes:
Merged: https://github.com/ruby/ruby/pull/6224
|
|
https://github.com/rubygems/rubygems/commit/425b78637f
|
|
https://github.com/rubygems/rubygems/commit/3973773005
|
|
Signed-off-by: Takuya Noguchi <takninnovationresearch@gmail.com>
|
|
Other code must not be between the doc and class definition.
https://github.com/rubygems/rubygems/commit/366784aae5
|
|
Picked at 12aeef6ba9a3be0022be9934c1a3e4c46a03ed3a
Notes:
Merged: https://github.com/ruby/ruby/pull/5462
|
|
https://github.com/rubygems/rubygems/commit/f7f504b24c
|
|
Since symlinks and absolute paths are already checked.
https://github.com/rubygems/rubygems/commit/de19bc4c7e
|
|
If we explicitly disallow the creation of symlinks that point to files
outside of the destination directory, we can avoid any other safety
checks while creating directories, because we can be sure they will
always fall under the destination directory as well.
https://github.com/rubygems/rubygems/commit/555692b8de
|
|
https://github.com/rubygems/rubygems/commit/0a0ad34af3
|
|
`File.realpath` already expands paths.
https://github.com/rubygems/rubygems/commit/25524ebbeb
|
