WEBrick: prevent response splitting and header injection

This is a follow up to d9d4a28.
The commit prevented CRLR, but did not address an isolated CR or an
isolated LF.

Co-Authored-By: NARUSE, Yui <naruse@airemix.jp>


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_5@67813 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Loop with String#scan without creating substrings

Create the substrings necessary parts only, instead of cutting the
rest of the buffer.  Also removed a useless, probable typo, regexp.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_5@67811 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

It breaks test-all


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_5@63025 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

	win32/file.c: relative path with drive letter

	* win32/file.c (IS_ABSOLUTE_PATH_P): home directory should not be
	  a relative path regardless a drive letter.   PathIsRelativeW
	  returns FALSE on such path.  [ruby-core:86356] [Bug #14638]
	------------------------------------------------------------------------
	r62953 | normal | 2018-03-28 17:05:46 +0900 (Wed, 28 Mar 2018) | 15 lines

	webrick: favor .write over << method

	This will make the next change to use IO.copy_stream
	easier-to-read.  When we can drop Ruby 2.4 support in a few
	years, this will allow us to use writev(2) with multiple
	arguments for headers and chunked responses.

	* lib/webrick/cgi.rb (write): new wrapper method
	  lib/webrick/httpresponse.rb: (send_header): use socket.write
	  (send_body_io): ditto
	  (send_body_string): ditto
	  (send_body_proc): ditto
	  (_write_data): ditto
	  (ChunkedWrapper#write): ditto
	  (_send_file): ditto
	------------------------------------------------------------------------
	r62954 | normal | 2018-03-28 17:05:52 +0900 (Wed, 28 Mar 2018) | 14 lines

	webrick/httpresponse: IO.copy_stream for regular files

	Remove the redundant _send_file method since its functionality
	is unnecessary with IO.copy_stream.  IO.copy_stream also allows
	the use of sendfile under some OSes to speed up copies to
	non-TLS sockets.

	Testing with "curl >/dev/null" and "ruby -run -e httpd" to
	read a 1G file over Linux loopback reveals a reduction from
	around ~0.770 to ~0.490 seconds on the client side.

	* lib/webrick/httpresponse.rb (send_body_io): use IO.copy_stream
	  (_send_file): remove
	  [Feature #14237]
	------------------------------------------------------------------------
	r62955 | normal | 2018-03-28 17:05:57 +0900 (Wed, 28 Mar 2018) | 10 lines

	webrick: use IO.copy_stream for single range response

	This is also compatible with range responses generated
	by Rack::File (tested with rack 2.0.3).

	* lib/webrick/httpresponse.rb (send_body_io): use Content-Range
	* lib/webrick/httpservlet/filehandler.rb (make_partial_content):
	  use File object for the single range case
	* test/webrick/test_filehandler.rb (get_res_body): use send_body
	  to test result
	------------------------------------------------------------------------
	r62956 | normal | 2018-03-28 17:06:02 +0900 (Wed, 28 Mar 2018) | 7 lines

	test/webrick/test_filehandler.rb: stricter multipart range test

	We need to ensure we generate compatibile output in
	the face of future changes

	* test/webrick/test_filehandler.rb (test_make_partial_content):
	  check response body
	------------------------------------------------------------------------
	r62957 | normal | 2018-03-28 17:06:08 +0900 (Wed, 28 Mar 2018) | 8 lines

	webrick: quiet warning for multi-part ranges

	Content-Length is ignored by WEBrick::HTTPResponse even if we
	calculate it, so instead we chunk responses to HTTP/1.1 clients
	and terminate HTTP/1.0 connections.

	* lib/webrick/httpservlet/filehandler.rb (make_partial_content):
	  quiet warning
	------------------------------------------------------------------------
	r62958 | normal | 2018-03-28 17:06:13 +0900 (Wed, 28 Mar 2018) | 7 lines

	webrick/httpresponse: make ChunkedWrapper copy_stream-compatible

	The .write method needs to return the number of bytes written
	to avoid confusing IO.copy_stream.

	* lib/webrick/httpresponse.rb (ChunkedWrapper#write): return bytes written
	  (ChunkedWrapper#<<): return self
	------------------------------------------------------------------------
	r62959 | normal | 2018-03-28 17:06:18 +0900 (Wed, 28 Mar 2018) | 9 lines

	webrick: use IO.copy_stream for multipart response

	Use the new Proc response body feature to generate a multipart
	range response dynamically.  We use a flat array to minimize
	object overhead as much as possible; as many ranges may fit
	into an HTTP request header.

	* lib/webrick/httpservlet/filehandler.rb (multipart_body): new method
	  (make_partial_content): use multipart_body

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_5@63006 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Original patch by tenderlove (with minor style adjustments).

* lib/webrick/httpresponse.rb (send_header): call check_header
  (check_header): raise on embedded CRLF in header value
* test/webrick/test_httpresponse.rb
  (test_prevent_response_splitting_headers): new test
* (test_prevent_response_splitting_cookie_headers): ditto

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_5@62986 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Reading entire request or response bodies into memory can lead
to trivial denial-of-service attacks.  Introduce Fibers in both
cases to allow streaming.

WEBrick::HTTPRequest gains a new body_reader method to prepare
itself as a source for IO.copy_stream.  This allows the
WEBrick::HTTPRequest object to be used as the
Net::HTTPGenericRequest#body_stream= arg for Net::HTTP.

For HTTP proxy response bodies, we also use a Fiber to
to make the HTTP request and read the response body.

* lib/webrick/httprequest.rb (body_reader): new method
  (readpartial): ditto
* lib/webrick/httpproxy.rb (perform_proxy_request): use Fiber
  to stream response body
  (do_GET, do_HEAD): adjust call
  (do_POST): adjust call and supply body_reader
* test/webrick/test_httprequest.rb (test_chunked): test
  for IO.copy_stream compatibility
* test/webrick/test_httpproxy.rb (test_big_bodies): new test

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_5@62985 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

No changes to the actual code, this is a new test for
a feature for which no tests existed.  I don't understand
the Digest authentication code well at all, but this is
necessary for the subsequent change.

* test/webrick/test_httpauth.rb (test_digest_auth_int): new test
  (credentials_for_request): support bodies with POST

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_5@62983 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

While WEBrick::HTTPRequest#body provides a Proc interface
for streaming large request bodies, clients must not force
the server to use an excessively large chunk size.

* lib/webrick/httprequest.rb (read_chunk_size): limit each
  read and block.call to :InputBufferSize in config.
* test/webrick/test_httpserver.rb (test_big_chunks): new test

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_5@62982 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

"BadRequest" alone does not resolve correctly, it is in the
HTTPStatus namespace.

* lib/webrick/httprequest.rb (read_chunked): use correct exception
* test/webrick/test_httpserver.rb (test_eof_in_chunk): new test

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_5@62981 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

We use the same 112 KB limit started (AFAIK) by Mongrel, Thin,
and Puma to prevent malicious users from using up all the memory
with a single request.  This also limits the damage done by
excessive ranges in multipart Range: requests.

Due to the way we rely on IO#gets and the desire to keep
the code simple, the actual maximum header may be 4093 bytes
larger than 112 KB, but we're splitting hairs at that point.

* lib/webrick/httprequest.rb: define MAX_HEADER_LENGTH
  (read_header): raise when headers exceed max length

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_5@62979 b2dd03c8-39d4-4d8f-98ff-823fe69b080e