Difference between revisions of "Perl"

From Organic Design wiki
(Perl and UTF-8)
(5 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{stub}}
+
== Perl and UTF-8 ==
 +
[https://blog.summercat.com/perl-and-character-encoding.html This excellent blog item on Perl's UTF-8 handling] clears up the confusion that can arise when dealing with UTF-8 in Perl. The main complexity is that Perl has an internal character encoding. This encoding allows Perl to hold Unicode characters in strings and treat them as single characters. This is useful as Unicode characters may be represented by multiple bytes in a Unicode encoding such as UTF-8. This means there are two types of strings: Those with characters in its internal encoding, and those not. The latter typically can be treated as binary data ("octets"), though strictly speaking they are octets only if they come from outside a program. i.e., not strings defined in source code.
 +
 
 +
One approach that can work is to have ''binmode'' set on all filehandles and ''stdout/stdin''. Then ''Encode::decode()'' at the point data enters the program, and ''Encode::encode()'' on its way out. This lets you avoid the complexity of I/O layers, and means you have a defined boundary where translation occurs.
 +
 
 +
Remember also to always use "UTF-8" rather than "utf8" as the encoding in the ''Encode module'' as only the former produces correct UTF-8, while the latter is something Perl specific and is not UTF-8.
 +
 
 +
See also [https://perl-begin.org/topics/files-and-directories/ Reading and writing UTF-8 examples with IO::All].
 +
 
 +
 
 +
== See also ==
 
*[[:Category:PERL|Our Perl scripts]]
 
*[[:Category:PERL|Our Perl scripts]]
 
*[http://man7.org/linux/man-pages/man3/strftime.3.html Date formatting symbols]
 
*[http://man7.org/linux/man-pages/man3/strftime.3.html Date formatting symbols]
 +
*[http://perl.com/pub/1999/10/DBI.html Basic DBI tutorial]
 +
*[[Regular expressions]] and [http://perldoc.perl.org/functions/quotemeta.html Quotemeta]
 
[[Category:Programming languages]]
 
[[Category:Programming languages]]

Revision as of 20:55, 9 January 2019

Perl and UTF-8

This excellent blog item on Perl's UTF-8 handling clears up the confusion that can arise when dealing with UTF-8 in Perl. The main complexity is that Perl has an internal character encoding. This encoding allows Perl to hold Unicode characters in strings and treat them as single characters. This is useful as Unicode characters may be represented by multiple bytes in a Unicode encoding such as UTF-8. This means there are two types of strings: Those with characters in its internal encoding, and those not. The latter typically can be treated as binary data ("octets"), though strictly speaking they are octets only if they come from outside a program. i.e., not strings defined in source code.

One approach that can work is to have binmode set on all filehandles and stdout/stdin. Then Encode::decode() at the point data enters the program, and Encode::encode() on its way out. This lets you avoid the complexity of I/O layers, and means you have a defined boundary where translation occurs.

Remember also to always use "UTF-8" rather than "utf8" as the encoding in the Encode module as only the former produces correct UTF-8, while the latter is something Perl specific and is not UTF-8.

See also Reading and writing UTF-8 examples with IO::All.


See also