?

3.3?Text Processing with Unicode?使用Unicode進行文字處理

?

Our programs will often need to deal with different languages, and different character sets. The concept of “plain text” is a fiction（虛構）. If you live in the English-speaking world you probably use ASCII, possibly without realizing it. If you live in Europe you might use one of the extended Latin character sets, containing such characters as “?” for Danish（丹麥語） and Norwegian（挪威語）, “?” for Hungarian（匈牙利語）, “?” for Spanish and Breton（法國的布列塔尼語）, and “ň” for Czech（捷克語） and Slovak（斯洛伐克語）. In this section, we will give an overview of how to use Unicode for processing texts that use non-ASCII character sets.

?

What Is Unicode??? 神馬是Unicode?

?

Unicode supports over a million characters. Each character is assigned a number, called a code point（編碼點）. In Python, code points are written in the form \uXXXX, where XXXX is the number in four-digit hexadecimal form（十六進制）.

?

Within a program, we can manipulate Unicode strings just like normal strings. However, when Unicode characters are stored in files or displayed on a terminal, they must be encoded as a stream of bytes. Some encodings (such as ASCII and Latin-2) use a single byte per code point, so they can support only a small subset of Unicode, enough for a single language. Other encodings (such as UTF-8) use multiple bytes and can represent the full range of Unicode characters.

?

Text in files will be in a particular encoding, so we need some mechanism for translating it into Unicode—translation into Unicode is called decoding（解碼）. Conversely, to write out Unicode to a file or a terminal, we first need to translate it into a suitable encoding— this translation out of Unicode is called encoding（編碼）, and is illustrated in Figure 3-3.

?

?? ? ???Figure 3-3. Unicode decoding and encoding

?

From a Unicode perspective, characters are abstract entities that can be realized as one or more glyphs（圖像字符）. Only glyphs can appear on a screen or be printed on paper. A font（字體） is a mapping from characters to glyphs.

?

Extracting Encoded Text from Files?從文件提取編碼文本

?

Let’s assume that we have a small text file, and that we know how it is encoded. For example, polish-lat2.txt, as the name suggests, is a snippet（片段） of Polish text (from the Polish Wikipedia; see http://pl.wikipedia.org/wiki/Biblioteka_Pruska ). This file is encoded as Latin-2, also known as ISO-8859-2. The function nltk.data.find() locates the file for us.

?

??>>>?path?=?nltk.data.find('corpora/unicode_samples/polish-lat2.txt')

?

The?Python?codecs（編碼解碼器） module provides functions to read encoded data into Unicode strings, and to write out Unicode strings in encoded form. The codecs.open() function takes an encoding parameter to specify the encoding of the file being read or written. So let’s import the codecs module, and call it with the encoding 'latin2' to open our Polish file as Unicode:

?

??> >>>?import?codecs

>>>?f?=?codecs.open(path,?encoding='latin2')

?

?

For a list of encoding parameters allowed by?codecs, see?http://docs.python.org/lib/standard-encodings.html . Note that we can write Unicode-encoded data to a file using f = codecs.open(path, 'w', encoding='utf-8'). Text read from the file object f will be returned in Unicode. As we pointed out earlier, in order to view this text on a terminal, we need to encode it, using a suitable encoding.

?

The Python-specific encoding unicode_escape is a dummy（傻瓜） encoding that converts all non-ASCII characters into their \uXXXX representations. Code points above the ASCII 0–127 range but below 256 are represented in the two-digit form \xXX.

?

??>>>?for?line?in?f:
??...?????line?=?line.strip()
??...?????print?line.encode('unicode_escape')
??Pruska?Biblioteka?Pa\u0144stwowa.?Jej?dawne?zbiory?znane?pod?nazw\u0105
??"Berlinka"?to?skarb?kultury?i?sztuki?niemieckiej.?Przewiezione?przez
??Niemc\xf3w?pod?koniec?II?wojny?\u015bwiatowej?na?Dolny?\u015al\u0105sk,?zosta\u0142y
??odnalezione?po?1945?r.?na?terytorium?Polski.?Trafi\u0142y?do?Biblioteki
??Jagiello\u0144skiej?w?Krakowie,?obejmuj\u0105?ponad?500?tys.?zabytkowych
??archiwali\xf3w,?m.in.?manuskrypty?Goethego,?Mozarta,?Beethovena,?Bacha.

?

?

The first line in this output illustrates a Unicode escape string preceded by the \u escape string, namely \u0144. The relevant Unicode character will be displayed on the screen as the glyph ń. In the third line of the preceding example, we see \xf3, which corresponds to the glyph ó, and is within the 128–255 range.

?

In Python, a Unicode string literal can be specified by preceding an ordinary string literal with a u, as in u'hello'. Arbitrary Unicode characters are defined using the \uXXXX escape sequence（轉義序列） inside a Unicode string literal. We find the integer ordinal（序數） of a character using ord(). For example:

?

??>>>?ord('a')
??97

?

?

The hexadecimal four-digit notation for 97 is 0061, so we can define a Unicode string literal with the appropriate escape sequence:

??>>>?a?=?u'\u0061'
??>>>?a
??u'a'
??>>>?print?a

??a?

?

Notice that the Python print statement is assuming a default encoding of the Unicode character, namely ASCII. However, ń is outside the ASCII range, so cannot be printed unless we specify an encoding. In the following example, we have specified that print should use the repr() of the string, which outputs the UTF-8 escape sequences (of the form \xXX) rather than trying to render（顯示） the glyphs.

?

??>>>?nacute?=?u'\u0144'
??>>>?nacute
??u'\u0144'
??>>>?nacute_utf?=?nacute.encode('utf8')
??>>>?print?repr(nacute_utf)
??'\xc5\x84'

?

?

If your operating system and locale are set up to render UTF-8 encoded characters, you ought to be able to give the Python command print nacute_utf and see ń on your screen.

Window的命令行不支持UTF-8的，使用print nacute_utf.decode('utf-8')可以顯示

?

There are many factors determining what glyphs are rendered on your screen. If you are sure that you have the correct encoding, but your Python code is still failing to produce the glyphs you expected, you should also check that you have the necessary fonts installed on your system.

?

The module unicodedata lets us inspect the properties of Unicode characters. In the following example, we select all characters in the third line of our Polish text outside the ASCII range and print their UTF-8 escaped value, followed by their code point integer using the standard Unicode convention (i.e., prefixing the hex digits with U+), followed by their Unicode name.

?

??>>>?import?unicodedata
??>>>?lines?=?codecs.open(path,?encoding='latin2').readlines()
??>>>?line?=?lines[2]
??>>>?print?line.encode('unicode_escape')
??Niemc\xf3w?pod?koniec?II?wojny?\u015bwiatowej?na?Dolny?\u015al\u0105sk,?zosta\u0142y\n
??>>>?for?c?in?line:
??...?????if?ord(c)?>?127:
??...?????????print?'%r?U+%04x?%s'?%?(c.encode('utf8'),?ord(c),?unicodedata.name(c))
??'\xc3\xb3'?U+00f3?LATIN?SMALL?LETTER?O?WITH?ACUTE
??'\xc5\x9b'?U+015b?LATIN?SMALL?LETTER?S?WITH?ACUTE
??'\xc5\x9a'?U+015a?LATIN?CAPITAL?LETTER?S?WITH?ACUTE
??'\xc4\x85'?U+0105?LATIN?SMALL?LETTER?A?WITH?OGONEK
??'\xc5\x82'?U+0142?LATIN?SMALL?LETTER?L?WITH?STROKE

If you replace the %r (which yields the repr() value) by %s in the format string of the?preceding code sample, and if your system supports UTF-8, you should see an output?like the following:

?

??ó U+00f3 LATIN SMALL LETTER O WITH ACUTE

??? U+015b LATIN SMALL LETTER S WITH ACUTE

??? U+015a LATIN CAPITAL LETTER S WITH ACUTE

??? U+0105 LATIN SMALL LETTER A WITH OGONEK

??? U+0142 LATIN SMALL LETTER L WITH STROKE

?

Alternatively, you may need to replace the encoding 'utf8' in the example by 'latin2', again depending on the details of your system.

The next examples illustrate how Python string methods and the re module accept Unicode strings.

?
??>>>?line.find(u'zosta\u0142y')
??54
??>>>?line?=?line.lower()
??>>>?print?line.encode('unicode_escape')
??niemc\xf3w?pod?koniec?ii?wojny?\u015bwiatowej?na?dolny?\u015bl\u0105sk,?zosta\u0142y\n
??>>>?import?re
??>>>?m?=?re.search(u'\u015b\w*',?line)
??>>>?m.group()?

??u?'\u015bwiatowej'

??

NLTK tokenizers allow Unicode strings as input, and correspondingly yield Unicode strings as output.

?

??>>>>?nltk.word_tokenize(line)?
??[u'niemc\xf3w',?u'pod',?u'koniec',?u'ii',?u'wojny',?u'\u015bwiatowej',
??u'na',?u'dolny',?u'\u015bl\u0105sk',?u'zosta\u0142y']

?

?

Using Your Local Encoding in Python?在Python使用你的本地編碼

?

If you are used to working with characters in a particular local encoding, you probably want to be able to use your standard methods for inputting and editing strings in a Python file. In order to do this, you need to include the string '# -*- coding: <coding> -*-' as the first or second line of your file. Note that <coding> has to be a string like 'latin-1', 'big5', or 'utf-8' (see Figure 3-4). Figure 3-4 also illustrates how regular expressions can use encoded strings.

?

轉載于:https://www.cnblogs.com/yuxc/archive/2011/08/06/2129447.html

總結

以上是生活随笔為你收集整理的Python自然语言处理学习笔记(19):3.3 使用Unicode进行文字处理的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。