summaryrefslogtreecommitdiff
path: root/tool/transcode-tblgen.rb
AgeCommit message (Collapse)Author
2013-06-02* enc/trans/japanese_euc.trans, test/ruby/test_transcode.rb,ktsj
tool/transcode-tblgen.rb: change EUC-JP-2004 to EUC-JIS-2004. This is follow up to changes in r41024. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@41035 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-11-22Drop executable bits of files that are not executable.knu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@37795 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-04-24* enc/euc_jp.c: added EUC-JP-2004 and its alias EUC-JISX0213.usa
[ruby-dev:45571] [Feature #6349] Requested by Kyouhei Yanagita <yanagi@shakenbu.org>. * enc/trans/japanese_euc.trans: ditto. * enc/trans/JIS/JISX0213-[12]%UCS@{BMP,SIP}.src: JIS X 0213:2004 -> Unicode mapping table from NetBSD. * enc/trans/JIS/UCS@{BMP,SIP}%JISX0213-[12].src: Unicode -> JIX X 0213:2004 mapping table from NetBSD. * tool/transcode-tblgen.rb: added SIP support. * test/ruby/test_transcode.rb: tests of above changes. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35460 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-04-06* enc/unicode/name2ctype.h, tool/transcode-tblgen.rb: revertnobu
unlogged changes which committed by accident probably. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35247 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-04-06struct.c (documentation for rb_struct_members_m):duerst
fix 'array of strings' to 'array of symbols' [ruby-core:44152][Bug #6264] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35244 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-02-17* tool/transcode-tblgen.rb (import_ucm): don't use \h because thenaruse
script should work with ruby 1.8. * tool/enc-unicode.rb: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34650 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-05-15* remove trailing spaces.nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@31573 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-22* enc/big5.c: split CP950 from Big5.naruse
* enc/big5.c: split CP951 from Big5-HKSCS. * enc/trans/big5.trans: import conversion table of Big5, Big5-HKSCS, CP950, and CP951 from ICU. they need fallback conversions. ref [ruby-core:33256] http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/ * tool/transcode-tblgen.rb (import_ucm): add to import ucm files. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29869 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-21* tool/transcode-tblgen.rb: reuse hash object.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26996 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-21* tool/transcode-tblgen.rb (transcode_compile_tree): makeakr
valid_encoding mandatory unless from_encoding is registered in ValidEncoding. (transcode_tbl_only): ditto. (transcode_tblgen): ditto. (ValidEncoding): new function. * enc/trans/escape.trans: specify valid_encoding. * enc/trans/emoji_sjis_docomo.trans: ditto. * enc/trans/emoji.trans: ditto. * enc/trans/emoji_iso2022_kddi.trans: ditto. * enc/trans/big5.trans: ditto. * enc/trans/emoji_sjis_softbank.trans: ditto. * enc/trans/emoji_sjis_kddi.trans: ditto. * enc/trans/chinese.trans: use ValidEncoding() instead of ValidEncoding[]. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26995 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-20* tool/transcode-tblgen.rb (ActionMap.unambiguous_action): useakr
Array#delete. (ActionMap.merge2): defined. (ActionMap::PostMemo): removed. (transcode_compile_tree): use ActionMap.merge2. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26990 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-20* tool/transcode-tblgen.rb (each_firstbyte_range): tuned for lessakr
method calls. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26985 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-18* tool/transcode-tblgen.rb: scan singleton mappings sequentially.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26973 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-16* tool/transcode-tblgen.rb: specialize for singletom mappings.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26956 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-15* tool/transcode-tblgen.rb: redundant loop removed.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26946 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-15* tool/transcode-tblgen.rb: refactored to use tree as memo key.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26945 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-15* tool/transcode-tblgen.rb: more info in generating macro names.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26943 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-15* tool/transcode-tblgen.rb: ActionMap#each_firstbyte inlined.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26942 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-15* tool/transcode-tblgen.rb (transcode_tblgen): add valid_encodingakr
optional argument. * enc/trans/single_byte.trans use valid_encoding argument for transcode_tblgen. * enc/trans/chinese.trans: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26941 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-14* tool/transcode-tblgen.rb (Action#hash): defined.akr
(Action#eql?): ditto. (Action#==): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26924 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-14* tool/transcode-tblgen.rb: refactored.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26923 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-14* tool/transcode-tblgen.rb: consider valid_encoding for max_input.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26916 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-13* tool/transcode-tblgen.rb: reject ambiguous mapping.akr
* enc/trans/single_byte.trans: remove ambiguous maping such as \xD6 -> U+05F2 and \xD6\xC7 -> U+FB1F in Windows-1255 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26912 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-12* tool/transcode-tblgen.rb: show cosumed time at last.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26898 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-09* enc/trans/EMOJI/*.src, enc/trans/emoji*, enc/x-emoji.c, ↵muraken
test/ruby/enc/test_emoji.rb, tool/enc-emoji-citrus-gen.rb, tool/enc-emoji4unicode.rb, tool/jisx0208.rb, tool/test/test_jisx0208.rb: new encodings to support emoji charsets, which are used by Japanese mobile phones [ruby-dev:40528]. Thanks Yoji Shidara for a lot of contribution. * tool/transcode-tblgen.rb: modified for enc-emoji4unicode.rb. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26856 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-09* tool/transcode-tblgen.rb (ActionMap#generate_node):naruse
remove useless dup. * tool/transcode-tblgen.rb (transcode_tbl_only): remove duplicated method call for encode_utf8. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26855 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-09* tool/transcode-tblgen.rb (StrSet#parse, ActionMap#generate_node):mame
optimization (reusing object and improving memoize). [ruby-dev:40582] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26852 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-03* tool/transcode-tblgen.rb (HEX2): extracted from regexps.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26806 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-12-10* transcode_data.h, transcode.c, tool/transcode-tblgen.rb: Addedduerst
support for new transcoding instruction FUNsio (with Tatsuya Mizuno) * enc/trans/gb18030.trans: Significantly reduced GB18030 conversion table footprint using FUNsio and differences (with Tatsuya Mizuno) * test/ruby/test_transcode.rb: Minor name fix (from Tatsuya Mizuno) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26065 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-09-22* tool/transcode-tblgen.rb (transcode_generated_code): fixed type.nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@25033 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-07-24* enc/trans/big5.trans, big5-hkscs-tbl.rb:duerst
new Chinese BIG5-HKSCS transcoding (with Tatsuya Mizuno) * test/ruby/test_transcode.rb: added tests for the above (with Tatsuya Mizuno) * enc/big5.c: Added BIG5-HKSCS as a replicate encoding of BIG5 (short term solution, needs more work; with Tatsuya Mizuno) * tool/transcode-tblgen.rb: made 'pat' directly accessible in class StrSet git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24264 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-06-30* tool/*: executable.nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23909 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-05-05Fix: DON'T move in_p because before in_p is replaced by buffered data.naruse
* transcode.c: NOMAP is now multibyte direct map. * transcode.c: remove ASIS. * transcode_data.h: ditto. * tool/transcode-tb (ActionMap#generate_info): remove :asis. * tool/transcode-tb (ActionMap#generate_info): add :nomap0. * enc/trans/utf8_mac.trans: replace :asis by :nomap0. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23344 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-04-26* tool/transcode-tb (ActionMap#each_firstbyte):naruse
if :asis collides other mappings, use another. * tool/transcode-tb (ActionMap#generate_info): add :asis for ASIS. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23295 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-04-26* tool/transcode-tb (ActionMap#generate_node):naruse
Use ActionMap#gennode instead of generate_node because of initialization. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23293 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-14* enc/trans/gb18030.trans: get rid of a 1.9 feature for crossnobu
compile. [ruby-core:21345] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21512 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-14* enc/trans/gb18030.trans, gb18030-tbl.rb:duerst
new Chinese GB18030 transcoding (from Yoshihiro Kambayashi) * test/ruby/test_transcode.rb: added tests for the above (from Yoshihiro Kambayashi) * transcode_data.h, transcode.c, tool/transcode_tblgen.rb: added support for GB18030-specific 4-byte sequences (with Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21509 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-18* tool/transcode-tblgen.rb: added set_valid_byte_patternduerst
to reduce coupling between table generation script and specific encodings. * enc/trans/single_byte.trans: using set_valid_byte_pattern git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19831 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-14* enc/trans/single_byte.trans: added windows-1252duerst
* enc/trans/windows-1252-tbl.rb: new file (contributed by Yoshihiro Kambayashi) * tool/transcode-tblgen.rb: listed windows-1252 as '1byte' * test/ruby/test_transcode.rb: added test_windows_1252 (contributed by Yoshihiro Kambayashi) git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19778 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-15* transcode_data.h (STR1_LENGTH): defined.akr
(makeSTR1LEN): defined. * tool/transcode-tblgen.rb: use makeSTR1LEN. generate STR1 for 4 to 259 bytes. * transcode.c (rb_transcoding): new field: output_index. (transcode_restartable0): use STR1_LENGTH. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19366 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-09* tool/transcode-tblgen.rb (StrSet#hash): cache hash value.akr
(ActionMap#hash): ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19279 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-08* include/ruby/encoding.h (rb_econv_asciicompat_encoding): renamedakr
from rb_econv_stateless_encoding to apply stateless ASCII incompatible encodings such as UTF-16BE. * io.c (make_writeconv): use rb_econv_asciicompat_encoding. * transcode_data.h (rb_transcoder_asciicompat_type_t): renamed from rb_transcoder_stateful_type_t. (rb_transcoder): use rb_transcoder_asciicompat_type_t. * transcode.c: follow the type change. (asciicompat_encoding_i): renamed from stateless_encoding_i. (rb_econv_asciicompat_encoding): renamed from rb_econv_stateless_encoding. (econv_s_asciicompat_encoding): method renamed. * tool/transcode-tblgen.rb: follow the type change. * enc/trans/utf_16_32.trans: follow the type change. rb_from_UTF_16BE to UTF-8 is asciicompat_decoder. rb_from_UTF_16LE to UTF-8 is asciicompat_decoder. rb_from_UTF_32BE to UTF-8 is asciicompat_decoder. rb_from_UTF_32LE to UTF-8 is asciicompat_decoder. UTF-8 to rb_to_UTF_16BE is asciicompat_encoder. UTF-8 to rb_to_UTF_16LE is asciicompat_encoder. UTF-8 to rb_to_UTF_32BE is asciicompat_encoder. UTF-8 to rb_to_UTF_32LE is asciicompat_encoder. * enc/trans/newline.trans: follow the type change. universal newline decoder is asciicompat_converter. * enc/trans/escape.trans: follow the type change. * enc/trans/iso2022.trans: ditto. * enc/trans/japanese.trans: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19249 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-08* tool/transcode-tblgen.rb (ArrayCode): less string substitutions.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19242 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-07* tool/transcode-tblgen.rb (transcode_tblgen): log message refined.akr
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19225 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-07* enc/trans/escape.trans: use transcode_tblgen.akr
* tool/transcode-tblgen.rb: generate an empty line after str1. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19217 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-07* tool/transcode-tblgen.rb (ActionMap#str_name): new method toakr
generate a name base on string content. (ActionMap#gen_str): extracted from generate_info and use str_name. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19216 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-07* tool/transcode-tblgen.rb (ActionMap#generate_info): use a memo toakr
avoid duplication for STR1. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19215 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-07* transcode_data.h (STR1): defined for a string up to 255 bytes.akr
(STR1_BYTEINDEX): defined. (makeSTR1): defined. * tool/transcode-tblgen.rb: generate STR1. * transcode.c (transcode_restartable0): interpret STR1. * enc/trans/escape.trans (fun_so_escape_xml_chref): removed. STR1 is used instead. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19214 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-07* tool/transcode-tblgen.rb: o4 is usable only if the first byte isakr
f0-f7. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19212 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-05* tool/transcode-tblgen.rb (StrSet.parse): accept upper caseakr
hexadecimal digits. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19164 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit v1.2.3 (git 2.25.1) at 2024-04-26 03:51:01 +0000