繁简体 / 单字 / 词性:3 个进阶玩法

Advanced: Traditional/Simplified, Single Char, POS Tagging

老板提的词频经常是变体 — 我要看单字的频率(汉字研究)、我要繁体数据(港澳台用户)、我要带词性的统计(学术论文)。这一节搞定 3 个高频变体。

1. 单字频率(汉字研究)

from collections import Counter





text = '天地玄黄宇宙洪荒日月盈昃辰宿列张'


char_counter = Counter(text)





for char, count in char_counter.most_common(10):


    print(f'{char}  {count}')


# 洪  1


# 宇  1

2. 繁简体通吃(OpenCC)

import opencc





converter = opencc.OpenCC('s2t')


text_tw = converter.convert('软件工程师')





converter2 = opencc.OpenCC('t2s')


text_cn = converter2.convert('軟件工程師')





def normalize(text):


    t2s = opencc.OpenCC('t2s').convert(text)


    s2t = opencc.OpenCC('s2t').convert(text)


    return {t2s.lower(), s2t.lower()}

3. 带词性的词频(jieba.posseg)

import jieba.posseg as pseg





text = '小米手机拍照效果真不错,屏幕也很清晰'


words = pseg.lcut(text)





meaningful = [w.word for w in words if w.flag.startswith(('n', 'v'))]


print(meaningful)


# ['手机', '拍照', '效果', '屏幕']





adjs = [w.word for w in words if w.flag == 'a']


print(Counter(adjs).most_common(5))


# [('不错', 1), ('清晰', 1)]

这 3 个进阶玩法(单字/繁简/词性)是真实项目中绕不开的 80% 场景。剩下的 20%(异体字/方言/表情符号)看老板的具体需求再补。