Skip to content

Latest commit

 

History

History
105 lines (68 loc) · 3.9 KB

README.md

File metadata and controls

105 lines (68 loc) · 3.9 KB
Raw

langdetect

Build Status

Port of Nakatani Shuyo's language-detection library (version from 03/03/2014) to Python.

Installation

$ pip install git+https://github.com/FojleRabbiRabib/langdetect

Supported Python versions 2.7, 3.4+.

Languages

langdetect supports 56 languages including Banglish (be) out of the box (ISO 639-1 codes):

af, ar, be, bg, bn, ca, cs, cy, da, de, el, en, es, et, fa, fi, fr, gu, he,
hi, hr, hu, id, it, ja, kn, ko, lt, lv, mk, ml, mr, ne, nl, no, pa, pl,
pt, ro, ru, sk, sl, so, sq, sv, sw, ta, te, th, tl, tr, uk, ur, vi, zh-cn, zh-tw

Basic usage

To detect the language of the text:

>>> from langdetect import detect
>>> detect("War doesn't show who's right, just who's left.")
'en'
>>> detect("আমি বাংলায় কথা কই")
'bn'
>>> detect("ami banglai kotha koi") # Banglish for Bengali
'be'

To find out the probabilities for the top languages:

>>> from langdetect import detect_langs
>>> detect_langs("Otec matka syn.")
[sk:0.572770823327, pl:0.292872522702, cs:0.134356653968]

To detect the language of the text from speficific profiles:

from langdetect import DetectorFactory
from langdetect import detect
DetectorFactory.load_profiles = ['en', 'bn', 'be', 'hi']
detect("War doesn't show who's right, just who's left.")

To find out the probabilities for the top languages from speficific profiles:

from langdetect import DetectorFactory
from langdetect import detect_langs
DetectorFactory.load_profiles = ['en', 'bn', 'be', 'hi']
detect_langs("Hello World.")

NOTE

Language detection algorithm is non-deterministic, which means that if you try to run it on a text which is either too short or too ambiguous, you might get different results everytime you run it.

To enforce consistent results, call following code before the first language detection:

from langdetect import DetectorFactory
DetectorFactory.seed = 0

How to add new language?

You need to create a new language profile. The easiest way to do it is to use the langdetect.jar tool, which can generate language profiles from Wikipedia abstract database files or plain text.

Wikipedia abstract database files can be retrieved from "Wikipedia Downloads" (http://download.wikimedia.org/). They form '(language code)wiki-(version)-abstract.xml' (e.g. 'enwiki-20101004-abstract.xml' ).

usage: java -jar langdetect.jar --genprofile -d [directory path] [language codes]

  • Specify the directory which has abstract databases by -d option.
  • This tool can handle gzip compressed file.

Remark: The database filename in Chinese is like 'zhwiki-(version)-abstract-zh-cn.xml' or zhwiki-(version)-abstract-zh-tw.xml', so that it must be modified 'zh-cnwiki-(version)-abstract.xml' or 'zh-twwiki-(version)-abstract.xml'.

To generate language profile from a plain text, use the genprofile-text command.

usage: java -jar langdetect.jar --genprofile-text -l [language code] [text file path]

For more details see language-detection Wiki.

Original project

This library is a direct port of Google's language-detection library from Java to Python. All the classes and methods are unchanged, so for more information see the project's website or wiki.

Presentation of the language detection algorithm: http://www.slideshare.net/shuyo/language-detection-library-for-java.