1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
|
== \Encoding
=== The Basics
A {character encoding}[https://en.wikipedia.org/wiki/Character_encoding],
often shortened to _encoding_, is a mapping between:
- A sequence of 8-bit bytes (each byte in the range <tt>0..255</tt>).
- Characters in a specific character set.
Some character sets contain only 1-byte characters;
{US-ASCII}[https://en.wikipedia.org/wiki/ASCII], for example, has 256 1-byte characters.
This string, encoded in US-ASCII, has six characters that are stored as six bytes:
s = 'Hello!'.encode('US-ASCII') # => "Hello!"
s.encoding # => #<Encoding:US-ASCII>
s.bytes # => [72, 101, 108, 108, 111, 33]
Other encodings may involve multi-byte characters.
{UTF-8}[https://en.wikipedia.org/wiki/UTF-8], for example,
encodes more than one million characters, encoding each in one to four bytes.
The lowest-valued of these characters correspond to ASCII characters,
and so are 1-byte characters:
s = 'Hello!' # => "Hello!"
s.bytes # => [72, 101, 108, 108, 111, 33]
Other characters, such as the Euro symbol, are multi-byte:
s = "\u20ac" # => "€"
s.bytes # => [226, 130, 172]
=== The \Encoding \Class
==== \Encoding Objects
Ruby encodings are defined by constants in class \Encoding.
There can be only one instance of \Encoding for each of these constants.
\Method Encoding.list returns an array of \Encoding objects (one for each constant):
Encoding.list.size # => 103
Encoding.list.first.class # => Encoding
Encoding.list.take(3)
# => [#<Encoding:ASCII-8BIT>, #<Encoding:UTF-8>, #<Encoding:US-ASCII>]
==== Names and Aliases
\Method Encoding#name returns the name of an \Encoding:
Encoding::ASCII_8BIT.name # => "ASCII-8BIT"
Encoding::WINDOWS_31J.name # => "Windows-31J"
An \Encoding object has zero or more aliases;
method Encoding#names returns an array containing the name and all aliases:
Encoding::ASCII_8BIT.names
# => ["ASCII-8BIT", "BINARY"]
Encoding::WINDOWS_31J.names
#=> ["Windows-31J", "CP932", "csWindows31J", "SJIS", "PCK"]
\Method Encoding.aliases returns a hash of all alias/name pairs:
Encoding.aliases.size # => 71
Encoding.aliases.take(3)
# => [["BINARY", "ASCII-8BIT"], ["CP437", "IBM437"], ["CP720", "IBM720"]]
\Method Encoding.name_list returns an array of all the encoding names and aliases:
Encoding.name_list.size # => 175
Encoding.name_list.take(3)
# => ["ASCII-8BIT", "UTF-8", "US-ASCII"]
\Method +name_list+ returns more entries than method +list+
because it includes both the names and their aliases.
\Method Encoding.find returns the \Encoding for a given name or alias, if it exists:
Encoding.find("US-ASCII") # => #<Encoding:US-ASCII>
Encoding.find("US-ASCII").class # => Encoding
==== Default Encodings
\Method Encoding.find, above, also returns a default \Encoding
for each of these special names:
- +external+: the default external \Encoding:
Encoding.find("external") # => #<Encoding:UTF-8>
- +internal+: the default internal \Encoding (may be +nil+):
Encoding.find("internal") # => nil
- +locale+: the default \Encoding for a string from the environment:
Encoding.find("locale") # => #<Encoding:UTF-8> # Linux
Encoding.find("locale") # => #<Encoding:IBM437> # Windows
- +filesystem+: the default \Encoding for a string from the filesystem:
Encoding.find("filesystem") # => #<Encoding:UTF-8>
\Method Encoding.default_external returns the default external \Encoding:
Encoding.default_external # => #<Encoding:UTF-8>
\Method Encoding.default_external= sets that value:
Encoding.default_external = 'US-ASCII' # => "US-ASCII"
Encoding.default_external # => #<Encoding:US-ASCII>
\Method Encoding.default_internal returns the default internal \Encoding:
Encoding.default_internal # => nil
\Method Encoding.default_internal= sets the default internal \Encoding:
Encoding.default_internal = 'US-ASCII' # => "US-ASCII"
Encoding.default_internal # => #<Encoding:US-ASCII>
==== Compatible Encodings
\Method Encoding.compatible? returns whether two given objects are encoding-compatible
(that is, whether they can be concatenated);
returns the \Encoding of the concatenated string, or +nil+ if incompatible:
rus = "\u{442 435 441 442}"
eng = 'text'
Encoding.compatible?(rus, eng) # => #<Encoding:UTF-8>
s0 = "\xa1\xa1".force_encoding('iso-8859-1') # => "\xA1\xA1"
s1 = "\xa1\xa1".force_encoding('euc-jp') # => "\x{A1A1}"
Encoding.compatible?(s0, s1) # => nil
==== \Encoding Options
A number of methods in the Ruby core accept keyword arguments as encoding options.
Some of the options specify or utilize a _replacement_ _string_, to be used
in certain transcoding operations.
A replacement string may be in any encoding that can be converted
to the encoding of the destination string.
These keyword-value pairs specify encoding options:
- For an invalid byte sequence:
- <tt>:invalid: nil</tt> (default): Raise exception.
- <tt>:invalid: :replace</tt>: Replace each invalid byte sequence
with the replacement string.
Examples:
s = "\x80foo\x80"
s.encode('ISO-8859-3') # Raises Encoding::InvalidByteSequenceError.
s.encode('ISO-8859-3', invalid: :replace) # => "?foo?"
- For an undefined character:
- <tt>:undef: nil</tt> (default): Raise exception.
- <tt>:undef: :replace</tt>: Replace each undefined character
with the replacement string.
Examples:
s = "\x80foo\x80"
"\x80".encode('UTF-8', 'ASCII-8BIT') # Raises Encoding::UndefinedConversionError.
s.encode('UTF-8', 'ASCII-8BIT', undef: :replace) # => "�foo�"
- Replacement string:
- <tt>:replace: nil</tt> (default): Set replacement string to default value:
<tt>"\uFFFD"</tt> ("�") for a Unicode encoding, <tt>'?'</tt> otherwise.
- <tt>:replace: _some_string_</tt>: Set replacement string to the given +some_string+;
overrides +:fallback+.
Examples:
s = "\xA5foo\xA5"
options = {:undef => :replace, :replace => 'xyzzy'}
s.encode('UTF-8', 'ISO-8859-3', **options) # => "xyzzyfooxyzzy"
- Replacement fallback:
One of these may be specified:
- <tt>:fallback: nil</tt> (default): No replacement fallback.
- <tt>:fallback: _hash_like_object_</tt>: Set replacement fallback to the given
+hash_like_object+; the replacement string is <tt>_hash_like_object_[X]</tt>.
- <tt>:fallback: _method_</tt>: Set replacement fallback to the given
+method+; the replacement string is <tt>_method_(X)</tt>.
- <tt>:fallback: _proc_</tt>: Set replacement fallback to the given
+proc+; the replacement string is <tt>_proc_[X]</tt>.
Examples:
s = "\u3042foo\u3043"
hash = {"\u3042" => 'xyzzy'}
hash.default = 'XYZZY'
s.encode('ASCII', fallback: h) # => "xyzzyfooXYZZY"
def (fallback = "U+%.4X").escape(x)
self % x.unpack("U")
end
"\u{3042}".encode("US-ASCII", fallback: fallback.method(:escape)) # => "U+3042"
proc = Proc.new {|x| x == "\u3042" ? 'xyzzy' : 'XYZZY' }
s.encode('ASCII', fallback: proc) # => "XYZZYfooXYZZY"
- XML entities:
One of these may be specified:
- <tt>:xml: nil</tt> (default): No handling for XML entities.
- <tt>:xml: :text</tt>: Treat source text as XML;
replace each undefined character
with its upper-case hexdecimal numeric character reference,
except that:
- <tt>&</tt> is replaced with <tt>&</tt>.
- <tt><</tt> is replaced with <tt><</tt>.
- <tt>></tt> is replaced with <tt>></tt>.
- <tt>:xml: :attr</tt>: Treat source text as XML attribute value;
replace each undefined character
with its upper-case hexdecimal numeric character reference,
except that:
- The replacement string <tt>r</tt> is double-quoted (<tt>"r"</tt>).
- Each embedded double-quote is replaced with <tt>"</tt>.
- <tt>&</tt> is replaced with <tt>&</tt>.
- <tt><</tt> is replaced with <tt><</tt>.
- <tt>></tt> is replaced with <tt>></tt>.
Examples:
s = 'foo"<&>"bar' + "\u3042"
s.encode('ASCII', xml: :text) # => "foo\"<&>\"barあ"
s.encode('ASCII', xml: :attr) # => "\"foo"<&>"barあ\""
- Newlines:
One of these may be specified:
- <tt>:cr_newline: true</tt>: Replace each line-feed character (<tt>"\n"</tt>)
with a carriage-return character (<tt>"\r"</tt>).
- <tt>:crlf_newline: true</tt>: Replace each line-feed character (<tt>"\n"</tt>)
with a carriage-return/line-feed string (<tt>"\r\n"</tt>).
- <tt>:universal_newline: true</tt>: Replace each carriage-return/line-feed string
(<tt>"\r\n"</tt>) with a line-feed character (<tt>"\n"</tt>).
Examples:
s = "\n \r\n" # => "\n \r\n"
s.encode('ASCII', cr_newline: true) # => "\r \r\r"
s.encode('ASCII', crlf_newline: true) # => "\r\n \r\r\n"
s.encode('ASCII', universal_newline: true) # => "\n \n"
|
