Misplaced Pages

Revision as of 17:54, 24 February 2007

Text mining, sometimes alternately referred to as text data mining, refers generally to the process of deriving high quality information from text. High quality information is typically derived through the divining of patterns and trends through means such as statistical pattern learning. Text mining usually involves the process of structuring the input text (usually parsing, along with the addition of some derived linguistic features and the removal of others, and subsequent insertion into a database), deriving patterns within the structured data, and finally evaluation and interpretation of the output. 'High quality' in text mining usually refers to some combination of relevance, novelty, and interestingness. Typical text mining tasks include text categorization, text clustering, concept/entity extraction, sentiment analysis, document summarization, and entity relation modeling (i.e., learning relations between named entities).

History

Labour-intensive manual text-mining approaches first surfaced in the mid-1980s, but technological advances have enabled the field to advance swiftly during the past decade. Text mining is an interdisciplinary field which draws on information retrieval, data mining, machine learning, statistics, and computational linguistics. As most information (over 80%) is currently stored as text, text mining is believed to have a high commercial potential value.

Applications

Recently, text mining has been receiving attention in many areas, most notably in the security, commercial, and academic fields.

Security applications

One of the largest text mining applications that exists is probably the classified ECHELON surveillance system.

Software and Applications

Research and development departments of major companies, including IBM and Microsoft, are researching text mining techniques and developing programs to further automate the mining and analysis processes.

Academic applications

The issue of text mining is of importance to publishers who hold large databases of information requiring indexing for retrieval. This is particularly true in scientific disciplines, in which highly specific information is often contained within written text. Therefore, initiatives have been begun such as Nature's proposal for an open text mining interface (OTMI) and NIH's common Journal Publishing Document Type Definition (DTD) that would provide semantic cues to machines to answer specific queries contained within text without removing publisher barriers to public access.

Academic institutions have also become involved in the text mining initiative: The National Centre for Text Mining (NaCTeM), a collaborative effort between the Universities of Manchester, Liverpool and Salford, funded by the Joint Information Systems Committee (JISC) and two of the UK Research Councils aim to provide tools, carry out research and offer advice to the academic community, with an initial focus on text mining in the biological and biomedical sciences. In the United States, the School of Information at University of California, Berkeley is developing a program called BioText to assist bioscience researchers in text mining and analysis.

Commercial Software and Applications

• Attensity - suite of text mining solutions for a variety of industries.
• Clarabridge - text mining and categorization applications for customer, healthcare, and investigative analytics.
• Clearforest - text mining software to extract meaning from various forms of textual information.
• IBM Intelligent Miner for Text - commercial text mining software
• Inxight - provider of text analytics, search, and unstructured visualization technologies.
• Nstein Technologies - provider of text analytics, and asset/web content management technologies. (See Nstein)
• PolyAnalyst - commercial text mining software
• SAS Enterprise Miner - commercial text mining software
• SPSS - provider of TextSmart, SPSS Text Analysis for Surveys, and Clementine, commercial text analysis software
• TEMIS - TEMIS is a software editor providing innovative Information Discovery solutions to serve the Information Intelligence needs of business corporations.
• TextAnalyst - commercial text mining software
• Textalyser - an online text analysis tool for generating text analysis statistics of web pages and other texts.
• Topicalizer - an online text analysis tool for generating text analysis statistics of web pages and other texts.
• The "Ultimate Research Assistant" - a knowledge management tool that uses a combination of traditional search engine technology and text mining techniques to facilitate online research of complex topics.
• Anderson Analytics - provider of text analytics and content analysis especially as it relates to consumer behavior.

Implications

Until recently websites most often used text-based lexical searches. Text mining may allow searches to be directly answered by the semantic web. Text mining is also used in some email spam filters.

References

• Ronen Feldman and James Sanger, The Text Mining Handbook, Cambridge University Press, ISBN 9780521836579
• Vitorino Ramos, Juan J. Merelo, Self-Organized Stigmergic Document Maps: Environment as a Mechanism for Context Learning, in AEB´2002 – 1st Spanish Conference on Evolutionary and Bio-Inspired Algorithms, E. Alba, F. Herrera, J.J. Merelo et al. (Eds.), pp. 284-293, Centro Univ. de Mérida, Mérida, Spain, 6-8 Feb. 2002.

See also

• Text classification sometimes is considered a (sub)task of text mining.
• Web mining, a task that may involve text mining (e.g. first find appropriate web pages by classifying crawled web pages, then extract the desired information from the text content of these pages considered relevant).
• Data mining
• Information retrieval
• Natural language processing
• Computational linguistics
• Business intelligence
• BioCreative text mining evaluation in biomedical literature
• w-shingling
• Concept Mining
• Stop words

• Artificial intelligence applications
• Artificial intelligence
• Data mining