| Age | Commit message (Collapse) | Author |
|---|
|
This commit removes usages of rb_gc_force_recycle since it is a burden
to maintain and makes changes to the GC difficult.
Notes:
Merged: https://github.com/ruby/ruby/pull/4363
|
|
https://github.com/ruby/zlib/commit/82e9a636a6
|
|
See also: https://github.com/ruby/csv/issues/117#issuecomment-933289373
How to reproduce with x.csv.gz in the issue comment:
Zlib::GzipReader.open("x.csv.gz") do |rio|
rio.gets(nil, 1024)
while line = rio.gets(nil, 8192)
raise line unless line.valid_encoding?
end
end
Reported by Dimitrij Denissenko. Thanks!!!
https://github.com/ruby/zlib/commit/b1f182e98f
|
|
https://github.com/ruby/zlib/commit/dd593acaee
|
|
https://github.com/ruby/zlib/commit/434eba55ae
|
|
https://github.com/ruby/zlib/commit/44a56d36e7
|
|
https://github.com/ruby/zlib/commit/931aa7a272
|
|
https://github.com/ruby/zlib/commit/098c50255d
|
|
multithreaded use
I'm not sure whether this handles all multithreaded use cases,
but this handles the example that crashes almost immediately
and does 10,000,000 total deflates using 100 separate threads.
To prevent the tests from taking forever, the committed test
for this uses only 10,000 deflates across 10 separate threads,
which still causes a segfault in the previous implementation
almost immediately.
Fixes [Bug #17803]
https://github.com/ruby/zlib/commit/4b1023b3f2
|
|
* See [Feature #17752]
Notes:
Merged: https://github.com/ruby/ruby/pull/4428
|
|
|
|
|
|
Notes:
Merged: https://github.com/ruby/ruby/pull/4248
|
|
|
|
If a buffer keyword argument is given, it is used as the buffer,
instead of creating new strings. This can result in significantly
lower memory usage during inflation.
Implements #19
https://github.com/ruby/zlib/commit/dac9a9b57d
|
|
This reads from the IO in 8192 byte chunks, so you don't need to
have the entire string in memory.
Fixes #16
https://github.com/ruby/zlib/commit/ba9793c550
|
|
This behavior differs from File#gets.
Fixes [Bug #13904]
https://github.com/ruby/zlib/commit/448e41efdd
|
|
This commit deletes
{IO,ARGF,StringIO,Zib::GZipReader}#{bytes,chars,lines,codepoints}, which
have been deprecated since c47c095b9740e7c19d6fdca29ab661c1089221d4.
Note that String also has those methods. They are neither depreacted
nor deleted because they are not aliases of counterpart each_something.
|
|
gz file
Most gzip tools support concatenated gz streams in a gz file. This
offers a way to handle such gz files in Ruby.
Fixes [Bug #9790]
Fixes [Bug #11180]
Fixes [Bug #14804]
https://github.com/ruby/zlib/commit/e2ce56de7d
|
|
https://github.com/ruby/zlib/commit/5af77c1ee8
|
|
|
|
https://github.com/ruby/zlib/commit/3e98e4cac3
|
|
Before this change, it was not possible to write out zero for the
timestamp part of a Gzip file's header, as calling GzipWriter#mtime with
zero was ignored.
Judging from the docs for `GzipWriter#mtime=`, it should be possible to
indicate that no timestamp is available by calling the method with zero.
https://github.com/ruby/zlib/commit/310be39cac
|
|
Ruby 2.7 deprecates taint and it no longer has an effect.
The lack of taint support should not cause a problem in
previous Ruby versions.
https://github.com/ruby/zlib/commit/21711ed0ce
|
|
https://github.com/ruby/zlib/commit/00ead8cb2c
|
|
Cfuncs that use rb_scan_args with the : entry suffer similar keyword
argument separation issues that Ruby methods suffer if the cfuncs
accept optional or variable arguments.
This makes the following changes to : handling.
* Treats as **kw, prompting keyword argument separation warnings
if called with a positional hash.
* Do not look for an option hash if empty keywords are provided.
For backwards compatibility, treat an empty keyword splat as a empty
mandatory positional hash argument, but emit a a warning, as this
behavior will be removed in Ruby 3. The argument number check
needs to be moved lower so it can correctly handle an empty
positional argument being added.
* If the last argument is nil and it is necessary to treat it as an option
hash in order to make sure all arguments are processed, continue to
treat the last argument as the option hash. Emit a warning in this case,
as this behavior will be removed in Ruby 3.
* If splitting the keyword hash into two hashes, issue a warning, as we
will not be splitting hashes in Ruby 3.
* If the keyword argument is required to fill a mandatory positional
argument, continue to do so, but emit a warning as this behavior will
be going away in Ruby 3.
* If keyword arguments are provided and the last argument is not a hash,
that indicates something wrong. This can happen if a cfunc is calling
rb_scan_args multiple times, and providing arguments that were not
passed to it from Ruby. Callers need to switch to the new
rb_scan_args_kw function, which allows passing of whether keywords
were provided.
This commit fixes all warnings caused by the changes above.
It switches some function calls to *_kw versions with appropriate
kw_splat flags. If delegating arguments, RB_PASS_CALLED_KEYWORDS
is used. If creating new arguments, RB_PASS_KEYWORDS is used if
the last argument is a hash to be treated as keywords.
In open_key_args in io.c, use rb_scan_args_kw.
In this case, the arguments provided come from another C
function, not Ruby. The last argument may or may not be a hash,
so we can't set keyword argument mode. However, if it is a
hash, we don't want to warn when treating it as keywords.
In Ruby files, make sure to appropriately use keyword splats
or literal keywords when calling Cfuncs that now issue keyword
argument separation warnings through rb_scan_args. Also, make
sure not to pass nil in place of an option hash.
Work around Kernel#warn warnings due to problems in the Rubygems
override of the method. There is an open pull request to fix
these issues in Rubygems, but part of the Rubygems tests for
their override fail on ruby-head due to rb_scan_args not
recognizing empty keyword splats, which this commit fixes.
Implementation wise, adding rb_scan_args_kw is kind of a pain,
because rb_scan_args takes a variable number of arguments.
In order to not duplicate all the code, the function internals need
to be split into two functions taking a va_list, and to avoid passing
in a ton of arguments, a single struct argument is used to handle
the variables previously local to the function.
Notes:
Merged-By: jeremyevans <code@jeremyevans.net>
|
|
After 5e86b005c0f2ef30df2f9906c7e2f3abefe286a2, I now think ANYARGS is
dangerous and should be extinct. This commit deletes ANYARGS from
rb_ensure, which also revealed many arity / type mismatches.
|
|
After 5e86b005c0f2ef30df2f9906c7e2f3abefe286a2, I now think ANYARGS is
dangerous and should be extinct. This commit deletes ANYARGS from
rb_rescue / rb_rescue2, which revealed many arity / type mismatches.
|
|
zlib and bignum both contain unblocking functions which are
async-signal-safe and do not require spawning additional
threads.
We can execute those functions directly in signal handlers
without incurring overhead of extra threads, so provide C-API
users the ability to deal with that. Other C-API users may
have similar need.
This flexible API can supercede existing uses of
rb_thread_call_without_gvl and rb_thread_call_without_gvl2 by
introducing a flags argument to control behavior.
Note: this API is NOT finalized. It needs approval from other
committers. I prefer shorter name than previous
rb_thread_call_without_gvl* functions because my eyes requires
big fonts.
[Bug #15499]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66712 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
Apparently, a component of Rails implements a buffering .write
method which keeps the String buffer around and makes it unsafe
for us to clear it after calling .write.
This caused Rack::Deflater to give empty results when enabled.
Fortunately, per r61631 / a55abcc0ca6f628fc05304f81e5a044d65ab4a68,
this misguided optimization was only worth a small (0.5MB) savings
and we still benefit from the majority of the memory savings in
that change.
Thanks to zunda for the bug report.
[ruby-core:90133] [Bug #15356]
Fixes: r61631 (commit a55abcc0ca6f628fc05304f81e5a044d65ab4a68)
("zlib: reduce garbage on gzip writes (deflate)")
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66268 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65141 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
No point in having a long-lived cbuf in "struct gzfile"
since GZFILE_CBUF_CAPA is smaller than RSTRING_EMBED_LEN_MAX
(even on 32-bit). We can also have rb_econv_convert write
directly to the return value instead of an intermediate buffer.
This brings "struct gzfile" from 264 to 256 bytes on 64-bit
systems to avoid taking an additional cache line.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63993 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
Init functions are called only once, cache is useless.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62429 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
For garbage-concious users who use the `outbuf' argument of
`readpartial' to supply a destination buffer, this provides
a drastic reduction in garbage when inflating large inputs
in a streaming fashion.
This results in a anonymous RSS reduction in the reader
similar to the reduction in the writer from r61631.
Results using the test script from r61631
<https://svn.ruby-lang.org/cgi-bin/viewvc.cgi?view=revision&revision=61631>
Before:
writer 7.359999 0.000000 7.359999 ( 7.360639)
writer RssAnon: 4040 kB
reader 6.346667 0.070000 6.416667 ( 7.387654)
reader RssAnon: 98272 kB
After:
writer 7.309999 0.000000 7.309999 ( 7.310651)
writer RssAnon: 4048 kB
reader 6.146666 0.003333 6.149999 ( 7.334868)
reader RssAnon: 4300 kB
* ext/zlib/zlib.c (struct read_raw_arg): new struct
(gzfile_read_raw_partial): use read_raw_arg
(gzfile_read_raw_rescue): ditto
(gzfile_read_raw): accept outbuf, use read_raw_arg
(gzfile_read_raw_ensure): accept outbuf
(gzfile_read_header): ditto
(gzfile_check_footer): ditto
(gzfile_read_more): ditto
(gzfile_read_raw_until_zero): adjust for changes
(gzfile_fill): ditto
(gzfile_readpartial): ditto
(gzfile_read_all): ditto
(gzfile_getc): ditto
(gzfile_reader_end_run): ditto
(gzfile_reader_get_unused): ditto
(rb_gzreader_initialize): ditto
(gzreader_skip_linebreaks): ditto
(gzreader_gets): ditto
(zlib_gunzip_run): ditto
[ruby-core:84660] [Feature #14319]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61665 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
Zlib::GzipWriter generated large amounts of garbage from
(struct zstream).input. Reuse the .input field when it is
hidden, and recycle it when its lifetime is over. This change
alone reduced memory usage of the writer from 90MB to 4.5MB.
For the detached buffer of compressed data used by
gzfile_write_raw, we can only clear the string (not recycle it)
since user code may hold references to it (but the data would be
clobbered, anyways). This reduced memory usage slightly by
around 0.5MB (because it's smaller compressed data).
Combined, these changes reduce the anonymous RSS memory of a
dedicated writer process from over 90MB to under 4MB.
before:
# user system total real
writer 7.823332 0.053333 7.876665 ( 7.881464)
writer RssAnon: 92944 kB
reader 6.969999 0.076666 7.046665 ( 7.906377)
reader RssAnon: 109820 kB
after:
writer 7.359999 0.000000 7.359999 ( 7.360639)
writer RssAnon: 4040 kB
reader 6.346667 0.070000 6.416667 ( 7.387654)
reader RssAnon: 98272 kB
Script used:
-------
require 'zlib'
require 'benchmark'
nr = 16384 * 2
def stats(pfx, bm)
str = "#{bm}#{File.readlines("/proc/#$$/status").grep(/^RssAnon:/)[0]}"
puts str.gsub!(/^/m, pfx)
end
rd, wr = IO.pipe
pid = fork do
buf = ((0..255).map(&:chr).join * 128).freeze
rd.close
gzip = Zlib::GzipWriter.new(wr)
bm = Benchmark.measure do
nr.times { gzip.write(buf) }
gzip.close
wr.close
end
stats('writer ', bm)
end
wr.close
buf = ''
gunzip = Zlib::GzipReader.new(rd)
n = 0
bm = Benchmark.measure do
begin
gunzip.readpartial(16384, buf)
n += buf.size
rescue EOFError
break
end while true
end
stats('reader ', bm)
Process.waitall
-------
* ext/zlib/zlib.c (zstream_discard_input): reuse or recycle hidden input
(zstream_reset_input): clear hidden input
(zstream_run): detach input and recycle after use
(gzfile_write_raw): clear buffer after write
[ruby-core:84638] [Feature #14315]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61631 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
No need to reveal strings freshly created with rb_str_new.
* ext/zlib/zlib.c (zstream_detach_input): remove redundant call
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61612 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* ext/zlib/zlib.c (rb_gzwriter_write): accepts multiple arguments.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60379 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* ext/zlib/zlib.c (zlib_gunzip): gz0 is a structure variable on
the stack, no longer valid after exit by an exception. ensure
to free instead. [Bug #13982]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60131 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* ext/zlib/zlib.c (zlib_gunzip): clear zstream to fix memory leak.
[ruby-core:83162] [Bug #13982]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60130 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60042 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* ext/zlib/zlib.c (rb_gzfile_total_out): cast to long not to
result in an unsigned long to normalized to Fixnum on LLP64
platforms. [ruby-core:81488]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59337 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
of file [Bug #13616]
patched by Andrew Haines <andrew@haines.org.nz> [ruby-core:81488]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59333 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* ext/zlib/zlib.c (zstream): manage capacity and size of `buf`
instead of size and separated member `buf_filled`. reported by
Christian Jalio (jalio) at https://hackerone.com/reports/211958
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58526 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* ext/zlib/zlib.c (zstream_buffer_ungetbyte): simplify by using
zstream_buffer_ungets().
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58525 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* ext/zlib/zlib.c (zstream_expand_buffer_non_stream): rename from
zstream_expand_buffer_without_gvl() and replace duplicate code
in zstream_expand_buffer().
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* ext/zlib/zlib.c: [DOC] update as keyword arguments.
[Feature #13020]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57129 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57126 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57049 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* ext/zlib/zlib.c (zstream_discard_input): replace with unread
portion substring, not modifying the input buffer directly.
[ruby-core:78567] [Bug #13021]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57042 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
- fix a typo (`GzipReadr` -> `GzipReader`)
- `Zlib::GzipReader.new` does not take block
- fix encoding
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57037 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
