merge revision(s) 32578,33401,33403,33404,33531,33655,33679,33809,33900,33965,34067,34069,34087,34328,34330,34527,34772,34783,34839,34914,34953,34954,35153: [Backport #6212]

	* ext/psych/lib/psych.rb: updating version to match gem
	* ext/psych/psych.gemspec: ditto
	* ext/psych/lib/psych/visitors/to_ruby.rb: fixing deprecation warning

	* ext/psych/lib/psych.rb: define a new BadAlias error class.

	* ext/psych/lib/psych/visitors/to_ruby.rb: raise an exception when
	  deserializing an alias that does not exist.

	* test/psych/test_merge_keys.rb: corresponding test.

	* ext/psych/lib/psych.rb (load, parse): stop parsing or loading after
	  the first document has been parsed.

	* test/psych/test_stream.rb: pertinent tests.

	* ext/psych/lib/psych.rb (parse_stream, load_stream): if a block is
	  given, documents will be yielded to the block as they are parsed.
	  [ruby-core:42404] [Bug #5978]

	* ext/psych/lib/psych/handlers/document_stream.rb: add a handler that
	  yields documents as they are parsed

	* test/psych/test_stream.rb: corresponding tests.

	* ext/psych/lib/psych/core_ext.rb: only extend Kernel if IRB is loaded
	  in order to stop method pollution.

	* ext/psych/lib/psych.rb: default open YAML files with utf8 external
	  encoding. [ruby-core:42967]
	* test/psych/test_tainted.rb: ditto

	* ext/psych/parser.c: prevent a memory leak by protecting calls to
	  handler callbacks.
	* test/psych/test_parser.rb: test to demonstrate leak.

	* ext/psych/parser.c: set parser encoding based on the YAML input
	  rather than user configuration.
	* test/psych/test_encoding.rb: corresponding tests.
	* test/psych/test_parser.rb: ditto
	* test/psych/test_tainted.rb: ditto

	* ext/psych/parser.c: removed external encoding setter, allow parser
	  to be reused.
	* ext/psych/lib/psych/parser.rb: added external encoding setter.
	* test/psych/test_parser.rb: test parser reuse

	* ext/psych/lib/psych/visitors/to_ruby.rb: Added support for loading
	  subclasses of String with ivars
	* ext/psych/lib/psych/visitors/yaml_tree.rb: Added support for dumping
	  subclasses of String with ivars
	* test/psych/test_string.rb: corresponding tests

	* ext/psych/lib/psych/visitors/to_ruby.rb: Added ability to load array
	  subclasses with ivars.
	* ext/psych/lib/psych/visitors/yaml_tree.rb: Added ability to dump
	  array subclasses with ivars.
	* test/psych/test_array.rb: corresponding tests

	* ext/psych/emitter.c: fixing clang warnings. Thanks Joey!

	* ext/psych/lib/psych/visitors/to_ruby.rb: BigDecimals can be restored
	  from YAML.
	* ext/psych/lib/psych/visitors/yaml_tree.rb: BigDecimals can be dumped
	  to YAML.
	* test/psych/test_numeric.rb: tests for BigDecimal serialization

	* ext/psych/lib/psych/scalar_scanner.rb: Strings that look like dates
	  should be treated as strings and not dates.

	* test/psych/test_scalar_scanner.rb: corresponding tests.

	* ext/psych/lib/psych.rb (module Psych): parse and load methods take
	  an optional file name that is used when raising Psych::SyntaxError
	  exceptions
	* ext/psych/lib/psych/syntax_error.rb (module Psych): allow nil file
	  names and handle nil file names in the exception message
	* test/psych/test_exception.rb (module Psych): Tests for changes.

	* ext/psych/parser.c (parse): parse method can take an option file
	  name for use in exception messages.
	* test/psych/test_parser.rb: corresponding tests.

	* ext/psych/lib/psych.rb: remove autoload from psych
	* ext/psych/lib/psych/json.rb: ditto

	* ext/psych/lib/psych/tree_builder.rb: dump complex numbers,
	  rationals, etc with reference ids.
	* ext/psych/lib/psych/visitors/yaml_tree.rb: ditto
	* ext/psych/lib/psych/visitors/to_ruby.rb: loading complex numbers,
	  rationals, etc with reference ids.
	* test/psych/test_object_references.rb: corresponding tests

	* ext/psych/lib/psych/scalar_scanner.rb: make sure strings that look
	  like base 60 numbers are serialized as quoted strings.
	* test/psych/test_string.rb: test for change.

	* ext/psych/parser.c: remove unused variable.

	* ext/psych/lib/psych/syntax_error.rb: Add file, line, offset, and
	  message attributes during parse failure.
	* ext/psych/parser.c: Update parser to raise exception with correct
	  values.
	* test/psych/test_exception.rb: corresponding tests.

	* ext/psych/parser.c (parse): Use context_mark for indicating error
	  line and column.

	* ext/psych/lib/psych/scalar_scanner.rb: use normal begin / rescue
	  since postfix rescue cannot receive the exception class. Thanks
	  nagachika!

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_1_9_3@35165 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

11 files changed, 423 insertions, 10 deletions

diff --git a/test/psych/test_array.rb b/test/psych/test_array.rb
index ec6a1aa147..9eedbb4fda 100644
--- a/test/psych/test_array.rb
+++ b/test/psych/test_array.rb
@@ -2,11 +2,39 @@ require 'psych/helper'
 
 module Psych
   class TestArray < TestCase
+    class X < Array
+    end
+
+    class Y < Array
+      attr_accessor :val
+    end
+
     def setup
       super
       @list = [{ :a => 'b' }, 'foo']
     end
 
+    def test_subclass
+      yaml = Psych.dump X.new
+      assert_match X.name, yaml
+
+      list = X.new
+      list << 1
+      assert_equal X, list.class
+      assert_equal 1, list.first
+    end
+
+    def test_subclass_with_attributes
+      y = Psych.load Psych.dump Y.new.tap {|y| y.val = 1}
+      assert_equal Y, y.class
+      assert_equal 1, y.val
+    end
+
+    def test_backwards_with_syck
+      x = Psych.load "--- !seq:#{X.name} []\n\n"
+      assert_equal X, x.class
+    end
+
     def test_self_referential
       @list << @list
       assert_cycle(@list)
diff --git a/test/psych/test_encoding.rb b/test/psych/test_encoding.rb
index a341c47859..8efb676d9a 100644
--- a/test/psych/test_encoding.rb
+++ b/test/psych/test_encoding.rb
@@ -31,6 +31,79 @@ module Psych
       @emitter = Psych::Emitter.new @buffer
     end
 
+    def test_transcode_shiftjis
+      str = "こんにちは！"
+      loaded = Psych.load("--- こんにちは！".encode('SHIFT_JIS'))
+      assert_equal str, loaded
+    end
+
+    def test_transcode_utf16le
+      str = "こんにちは！"
+      loaded = Psych.load("--- こんにちは！".encode('UTF-16LE'))
+      assert_equal str, loaded
+    end
+
+    def test_transcode_utf16be
+      str = "こんにちは！"
+      loaded = Psych.load("--- こんにちは！".encode('UTF-16BE'))
+      assert_equal str, loaded
+    end
+
+    def test_io_shiftjis
+      t = Tempfile.new(['shiftjis', 'yml'], :encoding => 'SHIFT_JIS')
+      t.write '--- こんにちは！'
+      t.close
+
+      # If the external encoding isn't utf8, utf16le, or utf16be, we cannot
+      # process the file.
+      File.open(t.path, 'r', :encoding => 'SHIFT_JIS') do |f|
+        assert_raises ArgumentError do
+          Psych.load(f)
+        end
+      end
+
+      t.close(true)
+    end
+
+    def test_io_utf16le
+      t = Tempfile.new(['utf16le', 'yml'])
+      t.binmode
+      t.write '--- こんにちは！'.encode('UTF-16LE')
+      t.close
+
+      File.open(t.path, 'rb', :encoding => 'UTF-16LE') do |f|
+        assert_equal "こんにちは！", Psych.load(f)
+      end
+
+      t.close(true)
+    end
+
+    def test_io_utf16be
+      t = Tempfile.new(['utf16be', 'yml'])
+      t.binmode
+      t.write '--- こんにちは！'.encode('UTF-16BE')
+      t.close
+
+      File.open(t.path, 'rb', :encoding => 'UTF-16BE') do |f|
+        assert_equal "こんにちは！", Psych.load(f)
+      end
+
+      t.close(true)
+    end
+
+    def test_io_utf8
+      t = Tempfile.new(['utf8', 'yml'])
+      t.binmode
+      t.write '--- こんにちは！'.encode('UTF-8')
+      t.close
+
+      File.open(t.path, 'rb', :encoding => 'UTF-8') do |f|
+        assert_equal "こんにちは！", Psych.load(f)
+      end
+
+      t.close(true)
+    end
+
     def test_emit_alias
       @emitter.start_stream Psych::Parser::UTF8
       @emitter.start_document [], [], true
diff --git a/test/psych/test_exception.rb b/test/psych/test_exception.rb
index 806c5e2641..c6d98d7a99 100644
--- a/test/psych/test_exception.rb
+++ b/test/psych/test_exception.rb
@@ -16,6 +16,97 @@ module Psych
       @wups = Wups.new
     end
 
+    def test_load_takes_file
+      ex = assert_raises(Psych::SyntaxError) do
+        Psych.load '--- `'
+      end
+      assert_nil ex.file
+
+      ex = assert_raises(Psych::SyntaxError) do
+        Psych.load '--- `', 'meow'
+      end
+      assert_equal 'meow', ex.file
+    end
+
+    def test_psych_parse_stream_takes_file
+      ex = assert_raises(Psych::SyntaxError) do
+        Psych.parse_stream '--- `'
+      end
+      assert_nil ex.file
+      assert_match '(<unknown>)', ex.message
+
+      ex = assert_raises(Psych::SyntaxError) do
+        Psych.parse_stream '--- `', 'omg!'
+      end
+      assert_equal 'omg!', ex.file
+      assert_match 'omg!', ex.message
+    end
+
+    def test_load_stream_takes_file
+      ex = assert_raises(Psych::SyntaxError) do
+        Psych.load_stream '--- `'
+      end
+      assert_nil ex.file
+      assert_match '(<unknown>)', ex.message
+
+      ex = assert_raises(Psych::SyntaxError) do
+        Psych.load_stream '--- `', 'omg!'
+      end
+      assert_equal 'omg!', ex.file
+    end
+
+    def test_parse_file_exception
+      t = Tempfile.new(['parsefile', 'yml'])
+      t.binmode
+      t.write '--- `'
+      t.close
+      ex = assert_raises(Psych::SyntaxError) do
+        Psych.parse_file t.path
+      end
+      assert_equal t.path, ex.file
+      t.close(true)
+    end
+
+    def test_load_file_exception
+      t = Tempfile.new(['loadfile', 'yml'])
+      t.binmode
+      t.write '--- `'
+      t.close
+      ex = assert_raises(Psych::SyntaxError) do
+        Psych.load_file t.path
+      end
+      assert_equal t.path, ex.file
+      t.close(true)
+    end
+
+    def test_psych_parse_takes_file
+      ex = assert_raises(Psych::SyntaxError) do
+        Psych.parse '--- `'
+      end
+      assert_match '(<unknown>)', ex.message
+      assert_nil ex.file
+
+      ex = assert_raises(Psych::SyntaxError) do
+        Psych.parse '--- `', 'omg!'
+      end
+      assert_match 'omg!', ex.message
+    end
+
+    def test_attributes
+      e = assert_raises(Psych::SyntaxError) {
+        Psych.load '--- `foo'
+      }
+
+      assert_nil e.file
+      assert_equal 1, e.line
+      assert_equal 5, e.column
+      # FIXME: offset isn't being set correctly by libyaml
+      # assert_equal 5, e.offset
+
+      assert e.problem
+      assert e.context
+    end
+
     def test_convert
       w = Psych.load(Psych.dump(@wups))
       assert_equal @wups, w
diff --git a/test/psych/test_merge_keys.rb b/test/psych/test_merge_keys.rb
index b3ebe9b463..bf5968ff86 100644
--- a/test/psych/test_merge_keys.rb
+++ b/test/psych/test_merge_keys.rb
@@ -2,6 +2,15 @@ require 'psych/helper'
 
 module Psych
   class TestMergeKeys < TestCase
+    def test_missing_merge_key
+      yaml = <<-eoyml
+bar:
+  << : *foo
+      eoyml
+      exp = assert_raises(Psych::BadAlias) { Psych.load yaml }
+      assert_match 'foo', exp.message
+    end
+
     # [ruby-core:34679]
     def test_merge_key
       yaml = <<-eoyml
diff --git a/test/psych/test_numeric.rb b/test/psych/test_numeric.rb
index 9adb058a32..bae723aca9 100644
--- a/test/psych/test_numeric.rb
+++ b/test/psych/test_numeric.rb
@@ -1,4 +1,5 @@
 require 'psych/helper'
+require 'bigdecimal'
 
 module Psych
   ###
@@ -10,5 +11,15 @@ module Psych
       str = Psych.load('--- 090')
       assert_equal '090', str
     end
+
+    def test_big_decimal_tag
+      decimal = BigDecimal("12.34")
+      assert_match "!ruby/object:BigDecimal", Psych.dump(decimal)
+    end
+
+    def test_big_decimal_round_trip
+      decimal = BigDecimal("12.34")
+      assert_cycle decimal
+    end
   end
 end
diff --git a/test/psych/test_object_references.rb b/test/psych/test_object_references.rb
new file mode 100644
index 0000000000..77cc96e29d
--- /dev/null
+++ b/test/psych/test_object_references.rb
@@ -0,0 +1,67 @@
+require 'psych/helper'
+
+module Psych
+  class TestObjectReferences < TestCase
+    def test_range_has_references
+      assert_reference_trip 1..2
+    end
+
+    def test_module_has_references
+      assert_reference_trip Psych
+    end
+
+    def test_class_has_references
+      assert_reference_trip TestObjectReferences
+    end
+
+    def test_rational_has_references
+      assert_reference_trip Rational('1.2')
+    end
+
+    def test_complex_has_references
+      assert_reference_trip Complex(1, 2)
+    end
+
+    def test_datetime_has_references
+      assert_reference_trip DateTime.now
+    end
+
+    def assert_reference_trip obj
+      yml = Psych.dump([obj, obj])
+      assert_match(/\*\d+/, yml)
+      data = Psych.load yml
+      assert_equal data.first.object_id, data.last.object_id
+    end
+
+    def test_float_references
+      data = Psych.load <<-eoyml
+--- 
+- &name 1.2
+- *name
+      eoyml
+      assert_equal data.first, data.last
+      assert_equal data.first.object_id, data.last.object_id
+    end
+
+    def test_binary_references
+      data = Psych.load <<-eoyml
+---
+- &name !binary |-
+  aGVsbG8gd29ybGQh
+- *name
+      eoyml
+      assert_equal data.first, data.last
+      assert_equal data.first.object_id, data.last.object_id
+    end
+
+    def test_regexp_references
+      data = Psych.load <<-eoyml
+--- 
+- &name !ruby/regexp /pattern/i
+- *name
+      eoyml
+      assert_equal data.first, data.last
+      assert_equal data.first.object_id, data.last.object_id
+    end
+  end
+end
diff --git a/test/psych/test_parser.rb b/test/psych/test_parser.rb
index decb241d19..d8c53f2d0d 100644
--- a/test/psych/test_parser.rb
+++ b/test/psych/test_parser.rb
@@ -32,6 +32,49 @@ module Psych
       @handler.parser = @parser
     end
 
+    def test_exception_memory_leak
+      yaml = <<-eoyaml
+%YAML 1.1
+%TAG ! tag:tenderlovemaking.com,2009:
+--- &ponies
+- first element
+- *ponies
+- foo: bar
+...
+      eoyaml
+
+      [:start_stream, :start_document, :end_document, :alias, :scalar,
+       :start_sequence, :end_sequence, :start_mapping, :end_mapping,
+       :end_stream].each do |method|
+
+        klass = Class.new(Psych::Handler) do
+          define_method(method) do |*args|
+            raise
+          end
+        end
+
+        parser = Psych::Parser.new klass.new
+        2.times {
+          assert_raises(RuntimeError, method.to_s) do
+            parser.parse yaml
+          end
+        }
+      end
+    end
+
+    def test_multiparse
+      3.times do
+        @parser.parse '--- foo'
+      end
+    end
+
+    def test_filename
+      ex = assert_raises(Psych::SyntaxError) do
+        @parser.parse '--- `', 'omg!'
+      end
+      assert_match 'omg!', ex.message
+    end
+
     def test_line_numbers
       assert_equal 0, @parser.mark.line
       @parser.parse "---\n- hello\n- world"
@@ -80,15 +123,6 @@ module Psych
       assert_equal 19, @parser.mark.index
     end
 
-    def test_set_encoding_twice
-      @parser.external_encoding = Psych::Parser::UTF16LE
-
-      e = assert_raises(Psych::Exception) do
-        @parser.external_encoding = Psych::Parser::UTF16LE
-      end
-      assert_equal "don't set the encoding twice!", e.message
-    end
-
     def test_bom
       tadpole = 'おたまじゃくし'
 
@@ -108,6 +142,7 @@ module Psych
 
     def test_bogus_io
       o = Object.new
+      def o.external_encoding; nil end
       def o.read len; self end
 
       assert_raises(TypeError) do
diff --git a/test/psych/test_scalar_scanner.rb b/test/psych/test_scalar_scanner.rb
index 659909909e..cf0dfff6aa 100644
--- a/test/psych/test_scalar_scanner.rb
+++ b/test/psych/test_scalar_scanner.rb
@@ -1,4 +1,5 @@
 require 'psych/helper'
+require 'date'
 
 module Psych
   class TestScalarScanner < TestCase
@@ -20,6 +21,27 @@ module Psych
       end
     end
 
+    def test_scan_bad_dates
+      x = '2000-15-01'
+      assert_equal x, @ss.tokenize(x)
+
+      x = '2000-10-51'
+      assert_equal x, @ss.tokenize(x)
+
+      x = '2000-10-32'
+      assert_equal x, @ss.tokenize(x)
+    end
+
+    def test_scan_good_edge_date
+      x = '2000-1-31'
+      assert_equal Date.strptime(x, '%Y-%m-%d'), @ss.tokenize(x)
+    end
+
+    def test_scan_bad_edge_date
+      x = '2000-11-31'
+      assert_equal x, @ss.tokenize(x)
+    end
+
     def test_scan_date
       date = '1980-12-16'
       token = @ss.tokenize date
diff --git a/test/psych/test_stream.rb b/test/psych/test_stream.rb
index 4d8f137431..beca365608 100644
--- a/test/psych/test_stream.rb
+++ b/test/psych/test_stream.rb
@@ -2,6 +2,50 @@ require 'psych/helper'
 
 module Psych
   class TestStream < TestCase
+    def test_parse_partial
+      rb = Psych.parse("--- foo\n...\n--- `").to_ruby
+      assert_equal 'foo', rb
+    end
+
+    def test_load_partial
+      rb = Psych.load("--- foo\n...\n--- `")
+      assert_equal 'foo', rb
+    end
+
+    def test_parse_stream_yields_documents
+      list = []
+      Psych.parse_stream("--- foo\n...\n--- bar") do |doc|
+        list << doc.to_ruby
+      end
+      assert_equal %w{ foo bar }, list
+    end
+
+    def test_parse_stream_break
+      list = []
+      Psych.parse_stream("--- foo\n...\n--- `") do |doc|
+        list << doc.to_ruby
+        break
+      end
+      assert_equal %w{ foo }, list
+    end
+
+    def test_load_stream_yields_documents
+      list = []
+      Psych.load_stream("--- foo\n...\n--- bar") do |ruby|
+        list << ruby
+      end
+      assert_equal %w{ foo bar }, list
+    end
+
+    def test_load_stream_break
+      list = []
+      Psych.load_stream("--- foo\n...\n--- `") do |ruby|
+        list << ruby
+        break
+      end
+      assert_equal %w{ foo }, list
+    end
+
     def test_explicit_documents
       io     = StringIO.new
       stream = Psych::Stream.new(io)
diff --git a/test/psych/test_string.rb b/test/psych/test_string.rb
index 51f1280abf..c7d5c60623 100644
--- a/test/psych/test_string.rb
+++ b/test/psych/test_string.rb
@@ -2,6 +2,37 @@ require 'psych/helper'
 
 module Psych
   class TestString < TestCase
+    class X < String
+    end
+
+    class Y < String
+      attr_accessor :val
+    end
+
+    def test_backwards_with_syck
+      x = Psych.load "--- !str:#{X.name} foo\n\n"
+      assert_equal X, x.class
+      assert_equal 'foo', x
+    end
+
+    def test_empty_subclass
+      assert_match "!ruby/string:#{X}", Psych.dump(X.new)
+      x = Psych.load Psych.dump X.new
+      assert_equal X, x.class
+    end
+
+    def test_subclass_with_attributes
+      y = Psych.load Psych.dump Y.new.tap {|y| y.val = 1}
+      assert_equal Y, y.class
+      assert_equal 1, y.val
+    end
+
+    def test_string_with_base_60
+      yaml = Psych.dump '01:03:05'
+      assert_match "'01:03:05'", yaml
+      assert_equal '01:03:05', Psych.load(yaml)
+    end
+
     def test_tagged_binary_should_be_dumped_as_binary
       string = "hello world!"
       string.force_encoding 'ascii-8bit'
diff --git a/test/psych/test_tainted.rb b/test/psych/test_tainted.rb
index bf55d3b30e..fdcced4cf3 100644
--- a/test/psych/test_tainted.rb
+++ b/test/psych/test_tainted.rb
@@ -121,7 +121,9 @@ module Psych
       t.binmode
       t.write string
       t.close
-      File.open(t.path) { |f| @parser.parse f }
+      File.open(t.path, 'r:bom|utf-8') { |f|
+        @parser.parse f
+      }
       t.close(true)
     end
   end