大数据分析与呆板学习范畴Python武器谱

老陈生

    曾经由于NLTK的缘故开始学习Python，之后徐徐成为我工作中的第一辅助脚本语言，固然开辟语言是C/C++，但平常的很 多文本数据处置惩罚使命都交给了Python。脱离腾讯创业后，第一个作品课程图谱也是选择了Python系的Flask框架，徐徐的将本身的绝大部门工作交 给了Python。这些年来，打仗和利用了许多Python工具包，特殊是在文本处置惩罚，科学盘算，呆板学习和数据发掘范畴，有许多许多良好的Python 工具包可供利用，以是作为Pythoner，也是相称幸福的。实在假如细致注意微博，你会发现许多这方面的分享，本身也Google了一下，发现也有同砚 总结了“Python呆板学习库”，不外总感觉缺少点什么。近来盛行一个词，全栈工程师(full stack engineer)，作为一个苦逼的创业者，自然的要把本身打造成一个full stack engineer，而这个过程中，这些Python工具包给本身提供了充足的火力，以是想起了这个系列。固然，这也仅仅是抛砖引玉，盼望各人能提供更多的 线索，来汇总整理一套Python网页爬虫，文本处置惩罚，科学盘算，呆板学习和数据发掘的武器谱。
一、Python网页爬虫工具集
一 个真实的项目，肯定是从获取数据开始的。无论文本处置惩罚，呆板学习和数据发掘，都必要数据，除了通过一些渠道购买大概下载的专业数据外，经常必要各人本身动 手爬数据，这个时间，爬虫就显得格外紧张了，幸好，Python提供了一批很不错的网页爬虫工具框架，既能爬取数据，也能获取和洗濯数据，我们也就从这里 开始了:
1. Scrapy
Scrapy, a fast high-level screen scraping and web crawling framework for Python.
鼎鼎台甫的Scrapy，信赖不少同砚都有耳闻，课程图谱中的许多课程都是依赖Scrapy抓去的，这方面的先容文章有许多，保举大牛pluskid从前的一篇文章:《Scrapy 轻松定制网络爬虫》，长期弥新。
官方主页:http://scrapy.org/
Github代码页: http://github.com/scrapy/scrapy
2. Beautiful Soup
You didn’t write that awful page. You’re just trying to get some data out of it. Beautiful Soup is here to help. Since 2004, it’s been saving programmers hours or days of work on quick-turnaround screen scraping projects.
读书的时间通过《团体聪明编程》这本书知道Beautiful Soup的，厥后也偶然会用用，非常棒的一套工具。客观的说，Beautifu Soup不完满是一套爬虫工具，必要共同urllib利用，而是一套HTML/XML数据分析，洗濯和获取工具。
官方主页:http://www.crummy.com/software/BeautifulSoup/
3. Python-Goose
Html Content / Article Extractor, web scrapping lib in Python
Goose最早是用Java写得，厥后用Scala重写，是一个Scala项目。Python-Goose用Python重写，依靠了Beautiful Soup。前段时间用过，感觉很不错，给定一个文章的URL, 获取文章的标题和内容很方便。
Github主页:http://github.com/grangier/python-goose
二、Python文本处置惩罚工具集
从 网页上获取文本数据之后，依据使命的差别，就必要举行根本的文本处置惩罚了，譬如对于英文来说，必要根本的tokenize，对于中文，则必要常见的中文分 词，进一步的话，无论英文中文，还可以词性标注，句法分析，关键词提取，文天职类，情绪分析等等。这个方面，特殊是面向英文范畴，有许多良好的工具包，我 们逐一道来。
1. NLTK — Natural Language Toolkit
NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, and an active discussion forum.
搞天然语言处置惩罚的同砚应该没有人不知道NLTK吧，这里也就不多说了。不外保举 两本册本给刚刚打仗NLTK大概必要具体相识NLTK的同砚: 一个是官方的《Natural Language Processing with Python》，以先容NLTK里的功能用法为主，同时附带一些Python知识，同时国内陈涛同砚友谊翻译了一个中文版，这里可以看到:保举《用 Python举行天然语言处置惩罚》中文翻译-NLTK配套书;别的一本是《Python Text Processing with NLTK 2.0 Cookbook》，这本书要深入一些，会涉及到NLTK的代码布局，同时会先容怎样定制本身的语料和模子等，相称不错。
官方主页:http://www.nltk.org/
Github代码页:http://github.com/nltk/nltk
2. Pattern
Pattern is a web mining module for the Python programming language.
It has tools for data mining (Google, Twitter and Wikipedia API, a web crawler, a HTML DOM parser), natural language processing (part-of-speech taggers, n-gram search, sentiment analysis, WordNet), machine learning (vector space model, clustering, SVM), network analysis and canvas visualization.
Pattern由比利时安特卫普大学CLiPS实行室出品，客 观的说，Pattern不但仅是一套文本处置惩罚工具，它更是一套web数据发掘工具，席卷了数据抓取模块(包罗Google, Twitter, 维基百科的API，以及爬虫和HTML分析器)，文本处置惩罚模块(词性标注，情绪分析等)，呆板学习模块(VSM, 聚类，SVM)以及可视化模块等，可以说，Pattern的这一整套逻辑也是这篇文章的构造逻辑，不外这里我们临时把Pattern放到文本处置惩罚部门。我 个人重要利用的是它的英文处置惩罚模块Pattern.en, 有许多很不错的文本处置惩罚功能，包罗底子的tokenize, 词性标注，句子切分，语法查抄，拼写纠错，情绪分析，句法分析等，相称不错。
官方主页:http://www.clips.ua.ac.be/pattern
3. TextBlob: Simplified Text Processing
TextBlob is a Python (2 and 3) library for processing textual data. It provides a simple API for diving into common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, classification, translation, and more.
TextBlob 是一个很故意思的Python文本处置惩罚工具包，它实在是基于上面两个Python工具包NLKT和Pattern做了封装(TextBlob stands on the giant shoulders of NLTK and pattern, and plays nicely with both)，同时提供了许多文本处置惩罚功能的接口，包罗词性标注，名词短语提取，情绪分析，文天职类，拼写查抄等，乃至包罗翻译和语言检测，不外这个是基于 Google的API的，有调用次数限定。TextBlob相对比力年轻，有爱好的同砚可以关注。
官方主页:http://textblob.readthedocs.org/en/dev/
Github代码页:http://github.com/sloria/textblob
4. MBSP for Python
MBSP is a text analysis system based on the TiMBL and MBT memory based learning applications developed at CLiPS and ILK. It provides tools for Tokenization and Sentence Splitting, Part of Speech Tagging, Chunking, Lemmatization, Relation Finding and Prepositional Phrase Attachment.
MBSP与Pattern同源，同出自比利时安特卫普大学CLiPS实行室，提供了Word Tokenization, 句子切分，词性标注，Chunking, Lemmatization，句法分析等根本的文本处置惩罚功能，感爱好的同砚可以关注。
官方主页:http://www.clips.ua.ac.be/pages/MBSP
5. Gensim: Topic modeling for humans
Gensim是一个相称专业的主题模子Python工具包，无论是代码照旧文档，我们曾经用《怎样盘算两个文档的相似度》先容过Gensim的安装和利用过程，这里就不多说了。
官方主页:http://radimrehurek.com/gensim/index.html
github代码页:http://github.com/piskvorky/gensim
6. langid.py: Stand-alone language identification system
语 言检测是一个很故意思的话题，不外相对比力成熟，这方面的办理方案许多，也有许多不错的开源工具包，不外对于Python来说，我利用过langid这个 工具包，也非常乐意保举它。langid现在支持97种语言的检测，提供了许多易用的功能，包罗可以启动一个发起的server，通过json调用其 API，可定制练习本身的语言检测模子等，可以说是“麻雀虽小，五脏俱全”。
Github主页:http://github.com/saffsd/langid.py
7. Jieba: 结巴中文分词
“结 巴”中文分词:做最好的Python中文分词组件 “Jieba” (Chinese for “to stutter”) Chinese text segmentation: built to be the best Python Chinese word segmentation module.
好了，终于可以说一个国内的Python文本处置惩罚工具包了:结巴分词，其功能包罗支持三种分词模式(准确模式、全模式、搜刮引擎模式)，支持繁体分词，支持自界说辞书等，是现在一个非常不错的Python中文分词办理方案。
Github主页:http://github.com/fxsjy/jieba
8. xTAS
xtas, the eXtensible Text Analysis Suite, a distributed text analysis package based on Celery and Elasticsearch.
感谢微博朋侪 @大山坡的春 提供的线索:我们组同事之前发布了xTAS，也是基于python的text mining工具包，接待利用，链接:http://t.cn/RPbEZOW。看起来很不错的样子，转头试用一下。
Github代码页:http://github.com/NLeSC/xtas
三、Python科学盘算工具包
说 起科学盘算，各人起首想起的是Matlab，集数值盘算，可视化工具及交互于一身，不外惋惜是一个贸易产物。开源方面除了GNU Octave在实验做一个雷同Matlab的工具包外，Python的这几个工具包聚集到一起也可以替换Matlab的相应功 能:NumPy+SciPy+Matplotlib+iPython。同时，这几个工具包，特殊是NumPy和SciPy，也是许多Python文本处置惩罚 & 呆板学习 & 数据发掘工具包的底子，非常紧张。末了再保举一个系列《用Python做科学盘算》，将会涉及到NumPy, SciPy, Matplotlib，可以做参考。
1. NumPy
NumPy is the fundamental package for scientific computing with Python. It contains among other things:
1)a powerful N-dimensional array object
2)sophisticated (broadcasting) functions
3)tools for integrating C/C++ and Fortran code
4) useful linear algebra, Fourier transform, and random number capabilities
Besides its obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide variety of databases.
NumPy险些是一个无法回避的科学 盘算工具包，最常用的大概是它的N维数组对象，其他还包罗一些成熟的函数库，用于整合C/C++和Fortran代码的工具包，线性代数、傅里叶变更和随 机数天生函数等。NumPy提供了两种根本的对象:ndarray(N-dimensional array object)和 ufunc(universal function object)。ndarray是存储单一数据范例的多维数组，而ufunc则是可以或许对数组举行处置惩罚的函数。
官方主页:http://www.numpy.org/
2. SciPy:Scientific Computing Tools for Python
SciPy refers to several related but distinct entities:
1)The SciPy Stack, a collection of open source software for scientific computing in Python, and particularly a specified set of core packages.
2)The community of people who use and develop this stack.
3)Several conferences dedicated to scientific computing in Python – SciPy, EuroSciPy and SciPy.in.
4)The SciPy library, one component of the SciPy stack, providing many numerical routines.
“SciPy 是一个开源的Python算法库和数学工具包，SciPy包罗的模块有最优化、线性代数、积分、插值、特别函数、快速傅里叶变更、信号处置惩罚和图像处置惩罚、常 微分方程求解和其他科学与工程中常用的盘算。其功能与软件MATLAB、Scilab和GNU Octave雷同。 Numpy和Scipy经常联合着利用，Python大多数呆板学习库都依靠于这两个模块。”—-引用自“Python呆板学习库”
官方主页:http://www.scipy.org/
3. Matplotlib
matplotlib is a python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. matplotlib can be used in python scripts, the python and ipython shell (ala MATLAB®* or Mathematica®†), web application servers, and six graphical user interface toolkits.
matplotlib 是python最闻名的画图库，它提供了一整套和matlab相似的下令API，非常得当交互式地举行制图。而且也可以方便地将它作为画图控件，嵌入 GUI应用步伐中。Matplotlib可以共同ipython shell利用，提供不亚于Matlab的画图体验，总之用过了都说好。
官方主页:http://matplotlib.org/
4. iPython
IPython provides a rich architecture for interactive computing with:
1)Powerful interactive shells (terminal and Qt-based).
2)A browser-based notebook with support for code, text, mathematical expressions, inline plots and other rich media.
3)Support for interactive data visualization and use of GUI toolkits.
4)Flexible, embeddable interpreters to load into your own projects.
5)Easy to use, high performance tools for parallel computing.
“iPython 是一个Python 的交互式Shell，比默认的Python Shell 好用得多，功能也更强盛。 她支持语法高亮、主动完成、代码调试、对象自省，支持 Bash Shell 下令，内置了很多很有效的功能和函式等，非常轻易利用。 ” 启动iPython的时间用这个下令“ipython –pylab”，默认开启了matploblib的画图交互，用起来很方便。
官方主页:http://ipython.org/
四、Python 呆板学习 & 数据发掘 工具包
呆板学习和数据发掘这两个概念不太好区分，这里就放到一起了。这方面的开源Python工具包有许多，这里先从认识的讲起，再增补其他泉源的资料，也接待各人增补。
1. scikit-learn: Machine Learning in Python
scikit-learn (formerly scikits.learn) is an open source machine learning library for the Python programming language. It features various classification, regression and clustering algorithms including support vector machines, logistic regression, naive Bayes, random forests, gradient boosting, k-means and DBSCAN, and is designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy.
首 先保举台甫鼎鼎的scikit-learn，scikit-learn是一个基于NumPy, SciPy, Matplotlib的开源呆板学习工具包，重要涵盖分类，回归和聚类算法，比方SVM， 逻辑回归，质朴贝叶斯，随机丛林，k-means等算法，代码和文档都非常不错，在很多Python项目中都有应用。比方在我们认识的NLTK中，分类器 方面就有专门针对scikit-learn的接口，可以调用scikit-learn的分类算法以及练习数据来练习分类器模子。这里保举一个视频，也是我 早期碰到scikit-learn的时间保举过的:保举一个Python呆板学习工具包Scikit-learn以及相干视频–Tutorial: scikit-learn – Machine Learning in Python
官方主页:http://scikit-learn.org/
2. Pandas: Python Data Analysis Library
Pandas is a software library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series.
第 一次打仗Pandas是由于Udacity上的一门数据分析课程“Introduction to Data Science” 的Project必要用Pandas库，以是学习了一下Pandas。Pandas也是基于NumPy和Matplotlib开辟的，重要用于数据分析和 数据可视化，它的数据布局DataFrame和R语言里的data.frame很像，特殊是对于时间序列数据有本身的一套分析机制，非常不错。这里保举一 本书《Python for Data Analysis》，作者是Pandas的主力开辟，依次先容了iPython, NumPy, Pandas里的相干功能，数据可视化，数据洗濯和加工，时间数据处置惩罚等，案例包罗金融股票数据发掘等，相称不错。
官方主页:http://pandas.pydata.org/
=======================================================
分割线，以上工具包根本上都是本身用过的，以下泉源于其他同砚的线索，特殊是《Python呆板学习库》，《23个python的呆板学习包》，做了一点增删修改，接待各人增补
========================================================
mlpy is a Python module for Machine Learning built on top of NumPy/SciPy and the GNU Scientific Libraries.
mlpy provides a wide range of state-of-the-art machine learning methods for supervised and unsupervised problems and it is aimed at finding a reasonable compromise among modularity, maintainability, reproducibility, usability and efficiency. mlpy is multiplatform, it works with Python 2 and 3 and it is Open Source, distributed under the GNU General Public License version 3.
官方主页:http://mlpy.sourceforge.net/
4. MDP:The Modular toolkit for Data Processing
Modular toolkit for Data Processing (MDP) is a Python data processing framework.
From the user’s perspective, MDP is a collection of supervised and unsupervised learning algorithms and other data processing units that can be combined into data processing sequences and more complex feed-forward network architectures.
From the scientific developer’s perspective, MDP is a modular framework, which can easily be expanded. The implementation of new algorithms is easy and intuitive. The new implemented units are then automatically integrated with the rest of the library.
The base of available algorithms is steadily increasing and includes signal processing methods (Principal Component Analysis, Independent Component Analysis, Slow Feature Analysis), manifold learning methods ([Hessian] Locally Linear Embedding), several classifiers, probabilistic methods (Factor Analysis, RBM), data pre-processing methods, and many others.
“MDP 用于数据处置惩罚的模块化工具包，一个Python数据处置惩罚框架。 从用户的观点，MDP是可以或许被整合到数据处置惩罚序列和更复杂的前馈网络布局的一批监视学习和非监视学习算法和其他数据处置惩罚单位。盘算依照速率和内存需求而高 效的实行。从科学开辟者的观点，MDP是一个模块框架，它可以或许被轻易地扩展。新算法的实现是轻易且直观的。新实现的单位然后被主动地与步伐库的别的部件进 行整合。MDP在神经科学的理论研究配景下被编写，但是它已经被计划为在利用可练习数据处置惩罚算法的任何环境中都是有效的。其站在用户一边的简朴性，各种不 同的随时可用的算法，及应用单位的可重用性，使得它也是一个有效的讲授工具。”
官方主页:http://mdp-toolkit.sourceforge.net/
5. PyBrain
PyBrain is a modular Machine Learning Library for Python. Its goal is to offer flexible, easy-to-use yet still powerful algorithms for Machine Learning Tasks and a variety of predefined environments to test and compare your algorithms.
PyBrain is short for Python-Based Reinforcement Learning, Artificial Intelligence and Neural Network Library. In fact, we came up with the name first and later reverse-engineered this quite descriptive “Backronym”.
“PyBrain(Python-Based Reinforcement Learning, Artificial Intelligence and Neural Network)是Python的一个呆板学习模块，它的目的是为呆板学习使命提供机动、易应、强盛的呆板学习算法。(这名字很霸气)
PyBrain正如其名，包罗神经网络、强化学习(及二者联合)、无监视学习、进化算法。由于现在的很多题目必要处置惩罚一连态和举动空间，必须利用函数迫近(如神经网络)以应对高维数据。PyBrain以神经网络为焦点，全部的练习方法都以神经网络为一个实例。”
官方主页:http://www.pybrain.org/
6. PyML – machine learning in Python
PyML is an interactive object oriented framework for machine learning written in Python. PyML focuses on SVMs and other kernel methods. It is supported on Linux and Mac OS X.
“PyML是一个Python呆板学习工具包，为各分类和回归方法提供机动的架构。它重要提供特性选择、模子选择、组合分类器、分类评估等功能。”
项目主页:http://pyml.sourceforge.net/
7. Milk:Machine learning toolkit in Python.
Its focus is on supervised classification with several classifiers available:
SVMs (based on libsvm), k-NN, random forests, decision trees. It also performs
feature selection. These classifiers can be combined in many ways to form
different classification systems.
“Milk 是Python的一个呆板学习工具箱，其重点是提供监视分类法与几种有用的分类分析:SVMs(基于libsvm)，K-NN，随机丛林经济和决议树。它 还可以举行特性选择。这些分类可以在很多方面相联合，形成差别的分类体系。对于无监视学习，它提供K-means和affinity propagation聚类算法。”
官方主页:http://luispedro.org/software/milk
http://luispedro.org/software/milk
8. PyMVPA: MultiVariate Pattern Analysis (MVPA) in Python
PyMVPA is a Python package intended to ease statistical learning analyses of large datasets. It offers an extensible framework with a high-level interface to a broad range of algorithms for classification, regression, feature selection, data import and export. It is designed to integrate well with related software packages, such as scikit-learn, and MDP. While it is not limited to the neuroimaging domain, it is eminently suited for such datasets. PyMVPA is free software and requires nothing but free-software to run.
“PyMVPA(Multivariate Pattern Analysis in Python)是为大数据集提供统计学习分析的Python工具包，它提供了一个机动可扩展的框架。它提供的功能有分类、回归、特性选择、数据导入导出、可视化等”
官方主页:http://www.pymvpa.org/
9. Pyrallel – Parallel Data Analytics in Python
Experimental project to investigate distributed computation patterns for machine learning and other semi-interactive data analytics tasks.
“Pyrallel(Parallel Data Analytics in Python)基于分布式盘算模式的呆板学习和半交互式的试验项目，可在小型集群上运行”
Github代码页:http://github.com/pydata/pyrallel
10. Monte – gradient based learning in Python
Monte (python) is a Python framework for building gradient based learning machines, like neural networks, conditional random fields, logistic regression, etc. Monte contains modules (that hold parameters, a cost-function and a gradient-function) and trainers (that can adapt a module’s parameters by minimizing its cost-function on training data).
Modules are usually composed of other modules, which can in turn contain other modules, etc. Gradients of decomposable systems like these can be computed with back-propagation.
“Monte (machine learning in pure Python)是一个纯Python呆板学习库。它可以敏捷构建神经网络、条件随机场、逻辑回归等模子，利用inline-C优化，极易利用和扩展。”
官方主页:http://montepython.sourceforge.net
11. Theano
Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. Theano features:
1)tight integration with NumPy – Use numpy.ndarray in Theano-compiled functions.
2)transparent use of a GPU – Perform data-intensive calculations up to 140x faster than with CPU.(float32 only)
3)efficient symbolic differentiation – Theano does your derivatives for function with one or many inputs.
4)speed and stability optimizations – Get the right answer for log(1+x) even when x is really tiny.
5)dynamic C code generation – Evaluate expressions faster.
6) extensive unit-testing and self-verification – Detect and diagnose many types of mistake.
Theano has been powering large-scale computationally intensive scientific investigations since 2007. But it is also approachable enough to be used in the classroom (IFT6266 at the University of Montreal).
“Theano 是一个 Python 库，用来界说、优化和模仿数学表达式盘算，用于高效的办理多维数组的盘算题目。Theano的特点:精密集成Numpy;高效的数据麋集型GPU盘算;高 效的符号微分运算;高速和稳固的优化;动态天生c代码;广泛的单位测试和自我验证。自2007年以来，Theano已被广泛应用于科学运算。theano 使得构建深度学习模子更加轻易，可以快速实现多种模子。PS:Theano，一位希腊玉人，Croton最有权势的Milo的女儿，厥后成为了毕达哥拉斯 的妻子。”
12. Pylearn2
Pylearn2 is a machine learning library. Most of its functionality is built on top of Theano. This means you can write Pylearn2 plugins (new models, algorithms, etc) using mathematical expressions, and theano will optimize and stabilize those expressions for you, and compile them to a backend of your choice (CPU or GPU).
“Pylearn2创建在theano上，部门依靠scikit-learn上，现在Pylearn2正处于开辟中，将可以处置惩罚向量、图像、视频等数据，提供MLP、RBM、SDA等深度学习模子。”
官方主页:http://deeplearning.net/software/pylearn2/
其他的，接待各人增补，这里也会连续更新这篇文章。

ssegg

楼主您的技术水准，我最服你，其他都是浮云

程德峰

加油，加油，不要沉下去，我是最热贴

tthu007

无私奉献，好工控人，32个赞送给你！！

Abercormfus

强烈支持楼主ing……

刘1990

真是被感动的痛哭流涕……

miao

原来还有这种操作，长见识了！

vov_B8c4M

已转发给朋友，一起感受这份快乐～

Katwin

这逻辑绝了，分析得太到位了吧

wll930121

浅评一下：内容优质，值得推荐～