summaryrefslogtreecommitdiff
path: root/lib/unicode_normalize
AgeCommit message (Collapse)Author
2019-04-05update to Unicode Version 12.1.0 (beta)duerst
Unicode Version 12.1.0 adds one single character, U+32FF SQUARE ERA NAME REIWA, for the new Japanese era starting on May 1st. 12.1.0 will be finalized only on May 7th, so we go with the beta version because further changes in the data we need are highly unlikely, and we want to make sure Ruby is ready for the new era. * common.mk: change UNICODE_VERSION to 12.1.0, UNICODE_BETA to YES * enc/unicode/12.1.0, enc/unicode/12.1.0/casefold.h, enc/unicode/12.1.0/name2ctype.h: add directory and generated data files for new version * lib/unicode_normalize/tables.rb: update for new character * test/ruby/test_regexp.rb: add test for character property age=12.1 * test/test_unicode_normalize.rb: add test for NFKC decomposition of new character This (mostly) completes issue #15195. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67441 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-04change lib/unicode_normalize/tables.rb to single item per line to make diffs ↵duerst
shorter * template/unicode_norm_gen.tmpl: Change formatting of output to produce only a single item (or range) for each line to make future diffs shorter and easier to understand and check. * lib/unicode_normalize/tables.rb: output of the above git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67439 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-03-06update Unicode version (and Emoji version) to 12.0.0duerst
- common.mk: set UNICODE_VERSION and UNICODE_EMOJI_VERSION to 12.0.0 - lib/unicode_normalize/tables.rb: update table data to Unicode version 12.0.0 - enc/unicode/12.0.0/casefold.h, enc/unicode/12.0.0/name2ctype.h: add generated files for Unicode version 12.0.0 This is the main commit for #15321. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67169 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-05update to Unicode 11.0.0 (main step, not complete yet)duerst
- common.mk: Change Unicode version to 11.0.0, and Emoji version to 11.0 - test/ruby/enc/test_emoji_breaks.rb: update hard-coded Emoji version - enc/unicode/11.0.0, enc/unicode/11.0.0/casefold.h, enc/unicode/name2ctype.h: Add generated files. Files for Unicode 10.0.0 will be removed once we are sure 11.0.0 works. - lib/unicode_normalize/tables.rb: Updated table. - regparse.c: Almost completely reimplement grapheme cluster detection in function node_extended_grapheme_cluster(). git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66213 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-02lib/*: Prefer require_relative over require, remove explicit extensionmarcandre
[#15206] [Fix GH-1976] git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65506 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-16revert r65091, r65090 because ci failsduerst
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65093 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-16update to Unicode 11.0.0 (basic step, not complete yet)duerst
- common.mk: Change Unicode version to 11.0.0 - enc/unicode/case-folding.rb, enc/unicode.c: Initial changes to deal with Gregorian Mtavruli. This should bring us up to the same level as e.g. Python 3.7, by following the Unicode tables exactly. But it will produce undesirable (mixed-case) results for String#capitalize. This will be addressed in a later commit. - enc/unicode/11.0.0, enc/unicode/11.0.0/casefold.h, enc/unicode/name2ctype.h: Add generated files. - lib/unicode_normalize/tables.rb: Updated table. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65091 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-28fix range check for Hangul jamo trailers in Unicode normalizationduerst
* lib/unicode_normalize/normalize.rb: Fix the range check for trailing Hangul jamo characters in Unicode normalization. Different from leading or vowel jamos, where LBASE and VBASE are actual characters, a value equal to TBASE expresses the absence of a trailing jamo. This fix is technically correct, but there was no bug because the regular expressions in lib/unicode_normalize/tables.rb eliminate jamos equal to TBASE from normalization processing. * test/test_unicode_normalize.rb: Add preventive test test_no_trailing_jamo based on https://github.com/python/cpython/commit/d134809cd3764c6a634eab7bb8995e3e2eff14d5 just for the case we ever get a regression. This closes issue #14934, thanks to MaLin (Lin Ma) for reporting. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-09-06update Ruby to Unicode 10.0.0duerst
- In common.mk, set UNICODE_VERSION to 10.0.0 - Generate and add enc/unicode/10.0.0/casefold.h and enc/unicode/10.0.0/name2ctype.h - Update lib/unicode_normalize/tables.rb git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59759 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-09add explanations about status of module UnicodeNormalizeduerst
In lib/unicode_normalize/normalize.rb, add explanations and clarifications about the status of the files and the module. [ci skip] This is in response to discussions at https://github.com/ruby/spec/pull/433 and https://bugs.ruby-lang.org/issues/5481#note-58. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58617 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-04move definition of String#unicode_normalized? to C to make sure it is documentedduerst
* lib/unicode_normalize.rb: Remove definition of String#unicode_normalized? (including documentation). Leave a comment explaining that the file is now empty. * string.c: Define String#unicode_normalized? in rb_str_unicode_normalized_p in C, (including documentation) * lib/unicode_normalize/normalize.rb: Remove (re)definition of String#unicode_normalized? to avoid warnings (when $VERBOSE==true) and problems when String is frozen git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58555 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-04move definition of String#unicode_normalize! to C to make sure it is documentedduerst
* lib/unicode_normalize.rb: Remove definition of String#unicode_normalize! (including documentation) * string.c: Define String#unicode_normalize! in rb_str_unicode_normalize_bang in C, (including documentation) * lib/unicode_normalize/normalize.rb: Remove (re)definition of String#unicode_normalize! to avoid warnings (when $VERBOSE==true) and problems when String is frozen git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58553 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-03move definition of String#unicode_normalize to C to make sure it is documentedduerst
* lib/unicode_normalize.rb: Remove definition of String#unicode_normalize (including documentation) * string.c: Define String#unicode_normalize in rb_str_unicode_normalize in C, (including documentation) * lib/unicode_normalize/normalize.rb: Remove (re)definition of String#unicode_normalize to avoid warnings (when $VERBOSE==true) and problems when String is frozen git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58550 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-02rework definition of String#unicode_normalize! and #unicode_normalized?duerst
simplify String#unicode_normalize! and #unicode_normalized? in lib/unicode_normalize.rb by redefining them in lib/unicode_normalize/normalize.rb git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58538 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-02rework definition of String#unicode_normalizeduerst
simplify String#unicode_normalize in lib/unicode_normalize.rb by redefining it in lib/unicode_normalize/normalize.rb git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58537 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-04-12nodoc UnicodeNormalize modulestomar
* lib/unicode_normalize/normalize.rb: [DOC] nodoc the internal UnicodeNormalize module. * lib/unicode_normalize/tables.rb: ditto. * template/unicode_norm_gen.tmpl: ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58329 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-09-07* common.mk: Updated Unicode version to 9.0.0 [Feature #12513]duerst
* unicode/9.0.0/casefold.h, name2ctype.h, unicode/data/9.0.0: new directories/files for Unicode version 9.0.0 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-07-20Fix commit misskazu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55713 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-07-20fix typoskazu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55711 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-07-19* lib/unicode_normalize/tables.rb: Removeduerst
UnicodeNormalize::UNICODE_VERSION (#12546). git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55706 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-16Add frozen_string_literal: false for all filesnaruse
When you change this to true, you may need to add more tests. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53141 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-11-10* lib/open-uri.rb: Remove indicator for "frozen_string_literal: true".akr
* lib/pp.rb: Ditto. * lib/prettyprint.rb: Ditto. * lib/resolv.rb: Ditto. * lib/securerandom.rb: Ditto. * lib/tmpdir.rb: Ditto. * lib/unicode_normalize/tables.rb: Ditto. * test/net/ftp/test_buffered_socket.rb: Ditto. * test/net/ftp/test_mlsx_entry.rb: Ditto. * test/open-uri/test_open-uri.rb: Ditto. * test/open-uri/test_ssl.rb: Ditto. * test/pathname/test_pathname.rb: Ditto. * test/test_pp.rb: Ditto. * test/test_prettyprint.rb: Ditto. * tool/transcode-tblgen.rb: Ditto. * ext/pathname/lib/pathname.rb: Ditto. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52526 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-10-02common.mk, lib/unicode_normalize/tables.rb: Change Unicodeduerst
Version for character normalization data from 7.0.0 to 8.0.0. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52000 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-09-29unicode_norm_gen.tmpl: end markernobu
* template/unicode_norm_gen.tmpl: pragma needs the end marker too, not only the beginning marker. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@51972 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-09-29tool/unicode_norm_gen.tmpl, lib/unicode_normalize/tables.rb:duerst
get rid of many .freeze commands by using frozen_string_literal pragma. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@51971 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-03-11normalize.rb: remove redundant hashnobu
* lib/unicode_normalize/normalize.rb (UnicodeNormalize): REGEXP_K matches only single chars which are keys of KOMPATIBLE_TABLE, so string in nfkd_one is always single char and one of the key of KOMPATIBLE_TABLE, that is that the default proc of NF_HASH_K only copies a pair in KOMPATIBLE_TABLE. therefore NF_HASH_K is a part of KOMPATIBLE_TABLE always, and just redundant. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49929 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-11tables.rb: addnobu
* lib/unicode_normalize/tables.rb: commit not to download and convert Unicode data files every time. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48386 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-09normalize.rb: fix syntax errornobu
* lib/unicode_normalize/normalize.rb (normalized): fix syntax error, `when` clase allows `*` but not `**`. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48340 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-09lib/unicode_normalize/normalize.rb: Replaced if-else by case in ↵duerst
self.normalized? in parallel to r48309. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48338 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-06unicode_norm_gen.tmpl: expand kompatible_tablenobu
* template/unicode_norm_gen.tmpl: expand kompatible_table so that recursive expansion is not needed at runtime. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48311 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-06normalize.rb: trivial optimizationsnobu
* lib/unicode_normalize/normalize.rb (nfc_one, normalize): trivial optimizations. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48309 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-06normalize.rb: explicit separatornobu
* lib/unicode_normalize/normalize.rb (canonical_ordering_one): use explicit separator, not to depend on $,. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48308 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-05lib/unicode_normalize/normalize.rb: Comment clarification. [ci skip]duerst
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48290 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-25lib/unicode_normalize/normalize.rb: added US_ASCIIduerst
as trivially supported encoding (is always normalized, and may appear mixed in with UTF-8 or other Unicode encodings). git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48134 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-21lib/unicode_normalize/tables.rb: remove auto generated file.nobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48074 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-21lib/unicode_normalize/tables.rb: Committing to make versionduerst
update easier and more predictable, and reducing compilation time. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48072 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-21lib/unicode_normalize/normalize.rb: Added comment to point toduerst
relevant portion of Unicode standard for Hangul (de)composition identifiers and algorithm. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48071 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-20unicode_normalize/normalize.rb: rename variablenobu
* lib/unicode_normalize/normalize.rb (hangul_decomp_one): use more descriptive name. leave [SLVT]BASE and [LVTNS]COUNT as they are vague names. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48055 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-20lib/unicode_normalize.rb: revert r48046. The s in sIndexduerst
is not hungarian notation. The variable name sIndex is directly taken from the relevant part of the Unicode Standard, where it is written SIndex and stands for 'syllable index'. See pp. 144/145 of http://www.unicode.org/versions/Unicode7.0.0/ch03.pdf. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48052 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-20unicode_normalize/normalize.rb: remove prefixnobu
* lib/unicode_normalize/normalize.rb (hangul_decomp_one): remove system hungarian prefix, nonsense in ruby. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48046 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-20unicode_normalize/normalize.rb: simplifynobu
* lib/unicode_normalize/normalize.rb (NF_HASH_{D,C,K}): remove first element by Hash#shift. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48045 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-20unicode_normalize/normalize.rb: remove unnecessary module namesnobu
* lib/unicode_normalize/normalize.rb (UnicodeNormalize): use self instead of module name and remove module name if unnecessary. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48044 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-19lib/unicode_normalize.rb: remove BOMsnobu
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48028 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-19lib/unicode_normalize/normalize.rb: Added a missing file extension in ↵duerst
require statement. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48022 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-19tool/unicode_norm_gen.rb, lib/unicode_normalize.rb:duerst
File name change from lib/unicode_normalize/normalize_tables.rb to lib/unicode_normalize/tables.rb. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48015 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-19* remove trailing spaces.svn
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48009 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-19lib/unicode_normalize/normalize.rb: Changed module name, adjusted copyright.duerst
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48008 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-19* properties.svn
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48007 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-19lib/unicode_normalize/normalize.rb: Importing fromduerst
https://github.com/duerst/eprun/blob/master/lib/normalize.rb. git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48005 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-13 04:34:57 +0000