Age | Commit message (Collapse) | Author |
|
rb_eConversionUndefinedError.
(rb_eConverterNotFoundError): renamed from rb_eNoConverterError.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19554 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
a patch from Tadashi Saito <shiba at mail2.accsnet.ne.jp> in
[ruby-dev:36346].
* encoding.c (Init_Encoding): rename EncodingCompatibilityError to
Encoding::CompatibilityError. [ruby-dev:36366]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19407 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
changed. [ruby-core:18578]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19349 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19301 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19252 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19151 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
it tend to cause security problem. If the behaviour is really
required, ECONV_INVALID_REPLACE with empty string can be used.
For example, CVE-2006-2313, CVE-2008-1036, [ruby-core:15645]
(ECONV_UNDEF_IGNORE): ditto.
* transcode.c (rb_econv_convert): follow the above change.
(econv_opts): ditto.
(Init_transcode): ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19123 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
writeconv_pre_ecopts.
(MakeOpenFile): initialize them.
* include/ruby/encoding.h (rb_str_transcode): take ecopts argument.
(rb_econv_flags): removed.
(rb_econv_prepare_opts): declared.
(rb_econv_open_opts): declared.
* io.c (make_writeconv): use rb_econv_open_opts.
(make_readconv): ditto.
(io_fwrite): follow rb_str_transcode change.
(rb_io_extract_modeenc): use rb_econv_prepare_opts.
(rb_file_open_generic): initialize encs.ecopts.
(rb_file_open_internal): ditto.
(rb_io_reopen): ditto.
(argf_ecopts): defined.
(argf_next_argv): set encs.ecopts.
(io_encoding_set): use rb_econv_prepare_opts.
(argf_set_encoding): set argf_ecopts.
* gc.c (gc_mark_children): mark encs.ecopts and writeconv_pre_ecopts
in T_FILE.
* transcode.c (transcode_loop): take ecopts argument. use
rb_econv_open_opts.
(rb_econv_flags): removed.
(rb_econv_prepare_opts): defined.
(rb_econv_open_opts): defined.
(str_transcode0): take ecopts.
(str_transcode): use rb_econv_prepare_opts.
(rb_str_transcode): take ecopts.
(econv_init): accept hash argument.
(econv_insert_output): follow rb_str_transcode change.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19109 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19086 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
eucJP-ms.
* enc/trans/japanese.trans (eucJP-ms): eucJP-ms is the correct
name of the encoding in Ruby.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19021 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19013 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* enc/trans/japanese.trans: use Citrus maps.
* enc/trans/CP: add maps from Citrus.
* enc/trans/JIS: ditto.
* test/ruby/test_transcode.rb: Shift_JIS and EUC-JP doesn't support
IBM extended characters.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19003 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
(contributed by Yoshihiro Kambayashi)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18862 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* transcode.c (rb_eNoConverter): new exception.
(rb_econv_open_exc): new function.
(transcode_loop): use rb_econv_open_exc.
* io.c (make_writeconv): use rb_econv_open_exc.
(make_readconv): ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18803 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
fixed comment strings (see r18291)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18772 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* tool/transcode-tblgen.rb: 8bit byte of ASCII-8BIT is a valid
(but unique to ASCII-8BIT) character.
* transcode.c (rb_eConversionUndefined): new error.
(rb_eInvalidByteSequence): ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18504 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
resetting a state of stateful encoding.
* enc/trans/iso2022.trans (rb_EUC_JP_to_ISO_2022_JP): specify
finish_eucjp_to_iso2022jp for resetstate_func.
* tool/transcode-tblgen.rb: specify NULL for resetstate_func.
* transcode.c (output_replacement_character): call resetstate_func
before appending the replacement character.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18503 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18500 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
(rb_trans_elem_t): new type.
(rb_trans_t): new type.
* transcode.c (transcode_dispatch_cb): removed.
(transcode_dispatch): removed.
(rb_transcoding_result_t): moved to rb_trans_result_t in
transcode_data.h.
(transcode_restartable0): goto follow_info when FUNsi.
(rb_transcoding_open): use get_transcoder_entry.
(rb_trans_open): new function.
(rb_trans_conv): ditto.
(rb_trans_close): ditto.
(trans_open_i): ditto.
(trans_sweep): ditto.
(more_output_buffer): take rb_trans_t instead of rb_transcoding as
an argument.
(transcode_loop): take from_encoding and to_encoding instead of tr
as arguments. use rb_trans_open/rb_trans_conv/rb_trans_close.
(str_transcode): don't use transcode_dispatch.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18498 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18448 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
from_utf8 field removed.
* tool/transcode-tblgen.rb: generate offsets range.
follow rb_transcoder change.
* transcode.c (transcode_loop): don't use from_utf8.
make invalid region from_unit_length wise.
* enc/trans/iso2022.erb.c: follow rb_transcoder and
transcode_generate_node change.
* enc/trans/utf_16_32.erb.c: follow rb_transcoder and
transcode_generate_node change.
explicit :invalid map removed.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18445 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
(rb_transcoder): preprocessor and postprocessor field removed.
change arguments of func_ii, func_si, func_io and func_so.
new field "finish_func".
* tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio
generatable.
* transcode.c (transcoder_lib_table): removed.
(transcoder_table): change structure.
(transcoder_key): removed because the above structure change.
(make_transcoder_entry): new function.
(get_transcoder_entry): ditto.
(rb_register_transcoder): follow the structure change.
(declare_transcoder): ditto.
(transcode_search_path): new function for breadth first search to
find a list of converters.
(transcode_search_path_i): new function.
(transcode_dispatch_cb): ditto.
(transcode_dispatch): use transcode_search_path.
(transcode_loop): follow the argument change.
(str_transcode): preprocessor and postprocessor stuff removed.
* enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion
re-implemented.
* enc/trans/japanese.erb.c: ISO-2022-JP stuff removed.
nute(23:52:53)% head -40 ChangeLog
Thu Aug 7 23:43:11 2008 Tanaka Akira <akr@fsij.org>
* transcode_data.h (rb_transcoding): new field "stateful".
(rb_transcoder): preprocessor and postprocessor field removed.
change arguments of func_ii, func_si, func_io and func_so.
new field "finish_func".
* tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio
generatable.
* transcode.c (transcoder_lib_table): removed.
(transcoder_table): change structure.
(transcoder_key): removed because the above structure change.
(make_transcoder_entry): new function.
(get_transcoder_entry): ditto.
(rb_register_transcoder): follow the structure change.
(declare_transcoder): ditto.
(transcode_search_path): new function for breadth first search to
find a list of converters.
(transcode_search_path_i): new function.
(transcode_dispatch_cb): ditto.
(transcode_dispatch): use transcode_search_path.
(transcode_loop): follow the argument change.
(str_transcode): preprocessor and postprocessor stuff removed.
* enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion
re-implemented.
* enc/trans/japanese.erb.c: ISO-2022-JP stuff removed.
* enc/trans/utf_16_32.erb.c: follow argument change of FUNso.
[ruby-dev:35798]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18419 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18388 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
rename from _get_replacement_character.
* transcode.c (output_replacement_character):
fix replacement on UTF-32{BE,LE}. [ruby-dev:35705]
* transcode.c (transcode_loop): ditto.
* test/ruby/test_transcode.rb (test_invalid_replace):
add for above.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18300 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
fix option1 and 3.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18295 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
character when convert to Unicode.
* test/ruby/test_transcode.rb (test_unicode_public_review_issue_121):
rename from test_public_review_issue_121.
* test/ruby/test_transcode.rb (test_unicode_public_review_issue_121):
enable option2.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18294 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
(contributed by Yoshihiro Kambayashi) and
test_public_review_issue_121
(see http://www.unicode.org/review/pr-121.html)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18291 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
test_iso_2022_jp(_1)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18203 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18160 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@16821 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* enc/trans/utf_16_32.c: bug fix (some invalid UTF-8 sequences
were legal)
* test/ruby/test_transcode.rb: test for above bug
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15786 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* transcode.c (transcode_loop): Adjusted detection of invalid
(ill-formed) UTF-8 sequences. Fixing potential security issue, see
http://www.unicode.org/versions/Unicode5.1.0/#Notable_Changes.
* test/ruby/test_transcode.rb: Added two tests for above fix.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15692 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* transcode.c: Added basic support for passing options to String#encode
via a hash. Currently only one option, with one value, is supported:
invalid: :ignore (dropping invalid byte sequences instead of
producing an error). Option naming is not yet stable!
* test/ruby/test_transcode.rb: Added a single test for invalid: :ignore
option. Not more tests because most data does not yet distinguish
between INVALID and UNKNOWN.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15565 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15393 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* transcode.c, enc/trans/utf_16_32.c, test/ruby/test_transcode.rb:
added UTF-32BE and UTF-32LE conversions.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15156 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* transcode.c, enc/trans/utf_16_32.c, test/ruby/test_transcode.rb:
added UTF-16LE conversions.
* fixed changelog for last commit
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15144 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* enc/trans/utf_16_32.c: new file, currently implementing
UTF-16BE conversions only.
* test/ruby/test_transcode.rb: Added tests for UTF-16BE;
made check_both_ways() use force_encoding differently.
* transcode_data.h, transcode.c: Support for more conversion
functions.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15142 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* transcode.c (transcode_dispatch): reverted some of the changes
in r14746.
* transcode.c, enc/trans/single_byte.c: Added conversions to/from
US-ASCII and ASCII-8BIT (using data tables).
* enc/trans/single_byte.c: Some spacing/ordering changes due to
automatic data file generation.
* transcode_data.h, transcode.c: Preliminary code for using
micro-conversion functions.
* test/ruby/test_transcode.rb: Added some tests for US-ASCII and
ASCII-8BIT conversions.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14766 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* test/ruby/test_transcode.rb: Added simple tests for
EUC-JP and Shift_JIS and tests for ASCII-only range
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14486 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* transcode_data_iso_8859.c: Changed from character constants
('\xC2') to integer contants (0xC2) for shorter files and
better readability; eliminated duplicated tables; changed
from -1 offset to actual UNDEF entry (not yet distinguishing
UNDEF and ILLEGAL correctly).
* test/ruby/test_transcode.rb: added a test for UNDEF conversion.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14251 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
* trancode.c: some minor formatting fixes
* transcode_data.h, transcode_data_iso_8859.c: Shortened
extremely frequently used macros to shorten file length.
* test/ruby/test_transcode.rb: Fixed name of test class;
added setup method to ensure all necessary encodings exist;
split tests into more test methods; added tests; fixed ordering
of arguments in assert_equal to have expected result first.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14236 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
AT it.aoyama.ac.jp>. [ruby-dev:32532]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14192 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14175 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|
|
code contributed by Martin Duerst.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14172 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
|