Yellow Bee's Blog - 大数据
type: channel
link: http://www.yelbee.top/index.php/category/big_data
items: MSBD6000H：自然语言处理课程笔记
items: Use Gensim to train Word2Vec Model
items: 机器翻译史 - 从冷战到深度学习
items: Machine Learning HW3 Solution
items: Mathematical Learning HW1
items: Normalization in Machine Learning
items: Machine Learning HW1 Solution
items: Machine Learning HW2 Solution
items: 机器学习知识整理
items: An appreciation about Picasso's works visualization

MSBD6000H：自然语言处理课程笔记
type: item
date: 2020-09-27T14:07:00+00:00
description: 1 概述什么是自然语言处理？图灵测试图灵测试是 Alan Turing 在1950年提出的一种判断机器是否能像人类一样思考的测试。标准的图灵测试如下图所示，测试者C同时与一台机器A和一个人B进行对话，三者相互隔离并且只能通过文本的方式交流，比如键盘或者屏幕，在经过若干轮问答后，倘若测试者C不能准确地区分出A、B哪个是人哪个是机器，则机器通过测试，反之失败。NLP的定义Natural language processing (NLP) is a study of the computational treatment of human natural language.自然语言处理是语言学、计算机科学、信息工程和人工智能的一个分支，研究计算机和人类语言的交互，特别是如何通过计算机编程来处理和分析大量的语言数据。这是一轮机器与人对话的示意图，语言可以拆分为三个层次：声音信号、符号表示、语义，机器以语义作为输入输出，而人则以声音信号作为输入输出。从机器到人：$Intention \rightarrow Generation \rightarrow Synthesis$从人到机器：$Perception \rightarrow Interpretation \rightarrow Incorporation$为什么需要自然语言处理？本质上来说，自然语言是非常丰富的M. Hamel spoke of the French tongue, saying it was the most beautiful language in the world, the most clear, the most sensible. We must keep it ourselves and never forget it, because when a people if they hold onto their language it is like holding the prison key... --- The Last Lesson by Alphonse DaudetIn the beginning was the Word, and the Word was with God, and the Word was God.--- John 1:1 in the Bible从功能上来说，语言是高效而普遍的如何完成NLP任务？NLP的难点在哪里？语言是模棱两可的（ambiguous）Children make delicious snacksFried Chicken makes delicious snacksStolen wallet is found by treeStolen wallet is found by MacTrump wins on budget, more lies ahead中国足球大败德国中国乒乓球大败德国我们知道中国的乒乓球很强，所以“大败”指的是中国战胜了德国，而由于足球实在弱不禁风，肯定是输给德国了。这里有个basic concept在里面，就是人们对一个国家在不同运动领域擅长程度的判断，人类很容易做到，而机器想做到却很难。He is an unbelievable worker. (Extremely good vs. Not trustful)Vancouver Police shoot man Holding box cutter. (温哥华警察射杀持刀男子 or 温哥华警察持刀射杀男子)语境解释，比如说根据上下文确定代词的指代，这对于人类来说很简单，但对机器来说很难The book does not fit the bag, because it is too bigThe book does not fit the bag, because it is too smallEvery American has a motherEvery American has a president有时候语法正确不代表语义正确Beautiful a has Monica Flower. (syntax and semantics right)Colorless green ideas sleep furiously. (syntax right but semantics wrong)第一个句子非常正常，而第二个句子虽然语法上没有错误，但是意思上完全说不通，一个idea怎么能sleep呢？2 其他机器翻译的评价指标《机器翻译评价指标 — BLEU算法》：https://www.cnblogs.com/jiangxinyang/p/10523585.html《机器翻译评价指标》：https://www.cnblogs.…

Use Gensim to train Word2Vec Model
type: item
date: 2020-09-27T14:07:00+00:00
description: View source code on Github pageUse Gensim to train a word2vec modelUse Gensim tool to tain a word embedding model on text8 dataset and the training process is based on skip-gram architecture.Reference: [1] https://rare-technologies.com/deep-learning-with-word2vec-and-gensim/[2] https://radimrehurek.com/gensim/models/keyedvectors.html#gensim.models.keyedvectors.WordEmbeddingsKeyedVectors# import modules and set up logging from gensim.models import word2vec import logging logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO) # load up unzipped corpus from http://mattmahoney.net/dc/text8.zip sentences = word2vec.Text8Corpus('data/text8')# train the skip-gram model; default window=5 model = word2vec.Word2Vec(sentences, size=200)Task 1 Analogy PredictionInput: a pair of words <a,b> and a third word cOutput: the forth word d which holds "a is to be as c is to d"For example"man -> woman" => "king -> queen""Japan -> Japanese" => "Australia -> Australian""Paris -> France" => "Beijing -> China"The above task can describe as Math form:Given word vector $w_a,w_b,w_c$ to find a word vector $w_d$ satisfying:$$ w_d-w_c\approx w_b-w_a\Leftrightarrow w_d\approx w_b-w_a+w_c $$Then we solve$$ d=\mathop {\text{arg}\max}_{d\in V}\cos \left( w_b-w_a+w_c,w_d \right) =\mathop {\text{arg}\max}_{d\in V}\frac{\left( w_b-w_a+w_c,w_d \right) \cdot w_d}{|w_b-w_a+w_c,w_d|\cdot |w_d|} $$# Use gensim api test_groups = [ ('man', 'woman', 'king'), ('small', 'smaller', 'big'), ('italy', 'italian', 'china'), ('japan', 'tokyo', 'china'), ('cool', 'coolest', 'cold'), ('dark', 'darkest', 'easy'), ('listening', 'listened', 'moving'), ('looking', 'looked', 'swimming'), ('playing', 'played', 'taking'), ('increase', 'increases', 'decrease'), ('predict', 'predicts', 'shuffle'), ('provide', 'provides', 'search'), ('say', 'says', 'speak'), ('Austria', 'Austrian', 'Sweden'), ('Cambodia', 'Cambodian', 'Australia'), ('paying', 'paid', 'striking'), ('running', 'ran', 'taking'), …

机器翻译史 - 从冷战到深度学习
type: item
date: 2020-09-27T14:06:00+00:00
description: 原文作者：Ilya Pestov，英文版译者Vasily Zubarev，中文翻译来自机器之心的Panda。英文版《Machine Translation - From the Cold War to Deep Learning》，俄文版请访问http://vas3k.com/blog/machine_translation/我打开谷歌翻译的频率是打开 Facebook 的两倍，价格标签的即时翻译对我而言再也不是赛博朋克了。这已经成为了现实。很难想象这是机器翻译算法百年研发之战的结果，而且在那段时间的一半时间里其实都没什么明显的成功。我在本文中讨论的确切发展将立足于所有的现代语言处理系统——从搜索引擎到声控微波。我将探讨的是当今的在线翻译技术的演化和结构。1 历史轨迹1.1 起初故事开始于 1933 年。苏联科学家 Peter Troyanskii 向苏联科学院提交了《用于在将一种语言翻译成另一种语言时选择和打印词的机器》。这项发明非常简单——它有四种语言的卡片、一台打字机和一台旧式胶片相机。操作员先取文本的第一个词，然后找到对应的卡片，拍一张照片，再在打字机上键入其形态特征（名词、复数、性别等）。这台打字机的按键编码了其中一项特征。打字带和相机胶片是同时使用的，从而得到一组带有词及它们的形态的帧。尽管看起来很不错，但和苏联的很多事情都一样，人们认为这项发明是「没用的」。Troyanskii 用了 20 年时间试图完成他的发明，之后因心绞痛逝世。在 1956 年两位苏联科学家找到他的父母之前，这世上没人知道这种机器。那是冷战的铁幕刚刚降下的时候。在 1954 年 1 月 7 日，IBM 在纽约的总部启动了 Georgetown-IBM 实验。IBM 701 计算机有史以来第一次自动将 60 个俄语句子翻译成了英语。「一位不认识任何一个苏联语言词汇的女孩在 IBM 卡片上敲出了这些俄语消息。这个「大脑」以每秒两行半的惊人速度在一台自动打印机上赶制出了它的英语翻译。」——IBM 的新闻稿但是，宣告胜利的头条新闻里却隐藏了一个小小的细节。没人提到这些翻译得到的样本是经过精心挑选和测试过的，从而排除了歧义性。对于日常使用而言，该系统并不比口袋里的常用语手册更好。尽管如此，军备竞赛还是开始了：加拿大、德国、法国以及（特别是）日本全都加入到了机器翻译竞赛中。1.2 机器翻译竞赛改进机器翻译的徒劳工作持续了四十年之久。1966 年，US ALPAC 在其著名的报告中称机器翻译是昂贵的、不准确的和毫无希望的。他们转而建议将重点放在词典开发上，这将美国研究者排除在了竞赛之外近十年时间。即便如此，仅凭科学家和他们的尝试、研究和开发，现代自然语言处理的基础还是建立了起来。多亏了这些彼此监视的国家，当今所有的搜索引擎、垃圾信息过滤器和个人助理都出现了。2 机器翻译的进化史2.1 基于规则的机器翻译（RBMT）最早的基于规则的机器翻译思想出现于 70 年代。科学家研究了翻译员的工作，试图让当时还极其缓慢的计算机也能重复这些行为。这些系统包含：双语词典（比如，俄语->英语）每种语言一套语言学规则（比如，以 -heit、-keit、-ung 等特定后缀结尾的名词都是阴性词）这就是这种系统的全部。如有需要，该系统还能得到一些补充，比如增加姓名列表、拼写纠错器和音译功能。PROMPT 和 Systran 是 RBMT 系统中最有名的案例。如果你想感受下那个黄金时代的柔和气息，去试试 Aliexpress 吧。但即使它们也有一些细微差别和亚种。直接机器翻译这是机器翻译中最直接的类型。它会将文本分成词，然后翻译这些词，再稍微校正一下形态，最后协调句法得到结果；或多或少听起来还行。当太阳落山后，训练有素的语言学家还在为每个词编写规则。其输出会返回某种类型的翻译结果。通常情况下，结果很糟糕。就好像是这些语言学家白白浪费了自己的时间。现代系统完全不会使用这种方法，现代语言学家对此感激不尽。基于迁移的机器翻译与直接翻译相比，我们翻译时要做准备——首先确定句子的语法结构，就像上学时老师教的那样。然后我们再操作整个结构，而不是一个个的词。这有助于在翻译中得到相当好的词序转换。理论上是这样。但在实践中，这仍然会得到逐词翻译的结果并会让语言学家身疲力竭。一方面，它带来的是简化过的一般性语法规则。但另一方面，由于词结构的数量比单个的词要多得多，所以这又会变得更加复杂。语际机器翻译在这种方法中，源文本会被转换成中间表征，并且会被统一用于全世界的所有语言（中间语言）。这正是笛卡尔所梦想的那种中间语言：一种元语言，遵循普适的规则并且可以将翻译变成一种简单的「来回切换」任务。接下来，中间语言可以转换成任何目标语言，而这就是奇点！正是由于存在这种转换，所以语际机器翻译常常会和基于迁移的系统混…

Machine Learning HW3 Solution
type: item
date: 2020-09-27T14:05:00+00:00
description: ML-HW3 PDFUpdate on November 11, 2019: ML-HW3 Solution PDFQuestion 1(a) According to the initail values that the question is given. When feeding $(x_1,x_2)=(1,2)$ to the network. The outputs of the hidden units$$ \begin{aligned} h_{11} &= x_1w_{11}^{(1)}+x_2w_{21}^{(1)} = 1 \times 1 + 2 \times (-1) = -1\\ \mu_{11} &= ReLU(h_{11}) = 0\\ h_{12} &= x_1w_{12}^{(1)}+x_2w_{22}^{(1)} = 1 \times (-1) + 2 \times 1 = 1\\ \mu_{12} &= ReLU(h_{12}) = 1\\ h_{21} &= \mu_{11}w_{11}^{(2)}+\mu_{21}w_{21}^{(2)} = 0 \times (-1) + 1 \times 1 = 1\\ \mu_{21} &= ReLU(h_{21}) = 1\\ h_{22} &= \mu_{11}w_{12}^{(2)}+\mu_{12}w_{22}^{(2)} = 0 \times (-1) + 1 \times 1 = 1\\ \mu_{22} &= ReLU(h_{22}) = 1 \end{aligned} $$Therefore, the logit is$$ \sigma(z) = \sigma(\mu_{21}w_1^{(3)} + \mu_{22}w_2^{(3)}) = \sigma(1 \times 1 + 1 \times 1) = \sigma(2) = \frac{e^2}{e^2+1} = 0.88 $$The output distribution of the output units$$ P(y\mid x_1=1,x_2=2,\theta) = P(y\mid z=2) = \left\{ \begin{aligned} &\sigma(2),y=1\\ &\sigma(-2),y=0 \end{aligned} \right. = \sigma((2y-1)\cdot 2) = \sigma(4y-2) $$(b) The given loss function can be written to$$ L = -logP(y=0|z)= -log\sigma(-z) = -log(\frac{1}{1+e^z}) = log(1+e^z) $$So the derivative of error of the output unit$$ \frac{\partial L}{\partial z} = \frac{\partial}{\partial z}log(1+e^z) = \frac{e^z}{1+e^z} = \sigma(z) $$Then the derivative error of the hidden units$$ \delta_{21} = \frac{\partial L}{\partial h_{21}} = \frac{\partial L}{\partial z} \frac{\partial z}{\partial \mu_{21}} \frac{\partial \mu_{21}}{\partial h_{21}} = w_1^{(3)} = \sigma(2) = 0.88 $$$$ \delta_{22} = \frac{\partial L}{\partial h_{22}} = \frac{\partial L}{\partial \mu_{22}} \frac{\partial \mu_{22}}{\partial h_{22}} = w_2^{(3)} = \sigma(2) = 0.88 $$$$ \delta_{11} = \frac{\partial L}{\partial h_{11}} = \delta_{21} \frac{\partial h_{21}}{\partial \mu_{11}} \frac{\partial \mu_{11}}{\partial h_{11}} + \delta_{22} \frac{\partial h_{22}}{\partial \mu_{11}} \frac{\partial \mu_{11}}{\partial h_{11}} $$Becau…

Mathematical Learning HW1
type: item
date: 2020-09-27T14:05:00+00:00
description: Homework for Course MSBD5004 Mathematical Methods for Data Analysis, BDT, HKUSTQuestion 1Consider the vector space $R^{n}$(a) Check that $\|x\|_{\infty}=\max_{1 \leq i \leq n} |x_{1}|$ is indeed a norm on $\mathbb{R}^n$ According to the definition of norm , we check(1)$\begin{aligned} \| x\|_{\infty} =\max _{1 \leq i \leq n}\left|x_{i}\right| \geq 0, \forall x \in \mathbb{R}^{n}\end{aligned} $ and $\| x\|_{\infty} = 0 \Leftrightarrow |x_i|=0 \Leftrightarrow x=\overrightarrow{0}$(2)$\begin{aligned} \|\alpha x\|_{\infty}=\max _{1 \leq i \leq n}|a x_{i}|=|\alpha| \cdot \max _{1 \leq i \leq n}\left|x_{i}\right|=|\alpha| \cdot \| x\|_{\infty}, \forall x \in \mathbb{R}, \alpha \in \mathbb{R}\end{aligned}$(3)$\begin{aligned}\|x+y\|_{\infty}=\max _{1 \leq i \leq n}\left|x_{i}+y_{i}\right| =\max_{1 \leq i \leq n} \left| x_i \right| +\max_{1 \leq i \leq n}\left| y_{i}\right|=\|x\|_{\infty}+\|y\|_{\infty}, \quad \forall x, y \in \mathbb{R}^{n}\end{aligned}$Therefore, $\|x\|_{\infty}=\max _{1 \leq i \leq n}\left|x_{i}\right|$ is a norm on $\mathbb{R}^{n}$(b) Prove that: for any $x \in \mathbb{R}^{n}, \quad\|x\|_{\infty}= lim_{p \rightarrow \infty} \|x\|_p$Prove:$$ \begin{aligned} &\|x\|_{p}=\left(\sum_{i=1}^{n}\left|x_{i}\right|^{p}\right)^{1 / p} \\ &=\left(\left|x_{1}\right|^{p}+\left|x_{x}\right|^{p}+\cdots+\left|x_{n}\right|^{p}\right)^{1 / p} \end{aligned} $$Let$$ \left|x_{m}\right|=\max \left\{\left|x_{1}\right|,\left|x_{2}\right|, \cdots,\left|x_{n}\right|\right\} $$Because$$ \frac{\left|x_{1} \right|}{\left|x_{m} \right|}, \frac{\left|x_{2} \right|}{\left|x_{m}\right|}, \ldots, \frac{\left|x_{n}\right|}{\left|x_{m}\right|}<1 $$Then we can get this inequality$$ 1 \cdot |x_m| \leq \|x\|_{p}=\left[\left(\frac{x_1}{x_{m}}\right)^{p}+\left(\frac{x_2}{x_{m}}\right)^{p}+ \cdots + \left(\frac{x_n}{x_{m}}\right)^{p} \right]^{\frac{1}{p}} \cdot |x_m| \leq (n)^{\frac{1}{p}} \cdot |x_m| $$when $p \rightarrow \infty, (n)^{\frac{1}{p}} \rightarrow 1$ so $\| x \|_p \rightarrow |x_m|$F…

Normalization in Machine Learning
type: item
date: 2020-09-27T14:05:00+00:00
description: Normalization is a big and important topic in machine learning so in this article I collect three important normalization methods which are batch norm, layer norm and group norm. I would also display how the normalizaiton work using CS231n codes as examples.The following figure virtualizes the difference among batch norm, layer norm and group norm.Normalization methods . N as the batch axix, C as the channel axis (Dimension), (H,W) as spatial axes - image height and width. The pixels in blue are normalized by the same mean and variance, computed by aggregating the values of these pixels.1 The Comparison of BN, LN and GNBN - batch normalizationLN - layer normalizationGN - group normalizationNow we have$X$ : Matrix, mini-batch data of shape (N, D)$\gamma$ : Scale parameter$\beta$ : Shift paremeterN is the number of samples in a batch, and D is the number of features. And $x_{i,j}$ to represent the element in matrix X.2 ForwardBatch Normalization ( BN )$$ \begin{aligned} &\mu_j \leftarrow \frac{1}{N}\sum_{i=1}^N x_{i,j} \\ &\sigma_j^2 \leftarrow \frac{1}{N} \sum_{i=1}^N (x_{i,j} - \mu_j)^2 \\ &\hat{x_{i,j}} \leftarrow \frac{x_{i,j} - \mu_j}{\sqrt{\sigma_j^2+\epsilon}} \\ &y_{i,j} \leftarrow \gamma \hat{x_{i,j}} + \beta \equiv BN_{\gamma,\beta}(x_{i,j}) \end{aligned} $$The python code# x: Data of shape (N, D) # gamma: Scale parameter of shape (D,) # beta: Shift paremeter of shape (D,) N, D = x.shape mean = np.mean(x, axis=0, keepdims=True) # (1, D) var = np.var(x, axis=0, keepdims=True) # (1, D) x_norm = (x - mean) / np.sqrt(var + eps) # (N, D) out = x_norm * gamma + beta Layer Normalization ( LN )Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization. Instead of normalizin…

Machine Learning HW1 Solution
type: item
date: 2020-09-27T14:04:00+00:00
description: ML-HW1 PDFUpdate on October 4, 2019: ML-HW1 Solution PDFQuestion 1According to the known condition, make $x_0=1$ i.e. add a column with 1 in matrix $X$:$$ X = \begin{bmatrix} 1 & 0 & 1\\ 1 & 0 & 1\\ 1 & 1 & 1\\ 1 & 1 & 0\\ 1 & 1 & 0 \end{bmatrix} \quad y = \begin{bmatrix} 0 \\ 1 \\ 2 \\ 3 \\ 4 \end{bmatrix} $$The OLS solution:$$ \begin{aligned} \hat{W} &= (X^TX)^{-1} \cdot X^T \cdot y \\ &= \left(\begin{bmatrix} 1 & 1 & 1 & 1 & 1\\ 0 & 0 & 1 & 1 & 1\\ 1 & 1 & 1 & 0 & 0\\ \end{bmatrix} \cdot \begin{bmatrix} 1 & 0 & 1\\ 1 & 0 & 1\\ 1 & 1 & 1\\ 1 & 1 & 0\\ 1 & 1 & 0 \end{bmatrix}\right)^{-1} \cdot \begin{bmatrix} 1 & 1 & 1 & 1 & 1\\ 0 & 0 & 1 & 1 & 1\\ 1 & 1 & 1 & 0 & 0 \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 1 \\ 2 \\ 3 \\ 4 \end{bmatrix}\\ &= \begin{bmatrix} 5 & 3 & 3\\ 3 & 3 & 1\\ 3 & 1 & 3 \end{bmatrix}^{-1} \cdot \begin{bmatrix} 1 & 1 & 1 & 1 & 1\\ 0 & 0 & 1 & 1 & 1\\ 1 & 1 & 1 & 0 & 0 \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 1 \\ 2 \\ 3 \\ 4 \end{bmatrix}\\ \end{aligned} $$$$ \begin{aligned} \hat{W} &= \begin{bmatrix} 2 & -3/2 & -3/2\\ -3/2 & 3/2 & 1\\ -3/2 & 1 & 3/2 \end{bmatrix} \cdot \begin{bmatrix} 1 & 1 & 1 & 1 & 1\\ 0 & 0 & 1 & 1 & 1\\ 1 & 1 & 1 & 0 & 0 \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 1 \\ 2 \\ 3 \\ 4 \end{bmatrix}\\ &= \begin{bmatrix} 1/2 & 1/2 & -1 & 1/2 & 1/2\\ -1/2 & -1/2 & 1 & 0 & 0\\ 0 & 0 & 1 & -1/2 & -1/2 \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 1 \\ 2 \\ 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 2\\ 3/2\\ -3/2 \end{bmatrix} \end{aligned} $$Question 2$\frac{1}{3} \sum_{i=1}^3 (y_i-\omega_0-\omega_1x_i)^2 + \lambda \omega_1^2$ where $\lambda=1$ ---> (c)$\frac{1}{3} \sum_{i=1}^3 (y_i-\omega_0-\omega_1x_i)^2 + \lambda \omega_1^2$ where $\lambda=10$ ---> (b)$\frac{1}{3} \sum_{i=1}^3 (y_i-\omega_0-w_1x_i)^2 + \lambda (\omega_0^2+\omega_1^2)$ where $\lambda=1$ ---> (a)$\frac{1}{3} \sum_{i=1}^3 (y_i-\omega_0-\omega_1x_i)^2 + \lambda (\omega_0^2+\omega_1^2)$ where $\lambda=10$ ---> (d)Explanation If we give the linear regression line an equation $y=k…

Machine Learning HW2 Solution
type: item
date: 2020-09-27T14:04:00+00:00
description: ML-HW2 PDFUpdate on October 19, 2019: ML-HW2 Solution PDFQuestion 1(a) Considering Naive Bayes model, the features are conditionally independent. And because of the binary features, we can use Bernoulli distribution for the class-conditional probability$$ p(\vec{x}|y=c) = \prod_{j=1}^2Ber(x_j|\mu_{jc}) $$Use Bayes rules, the posterior probability$$ \begin{aligned} p(y=c|\vec{x}) &= \frac{P(\vec{x},y=c)}{P(\vec{x})} \\ &= \frac{P(y=c)}{P(\vec{x})} \cdot p(\vec{x}|y=c) \\ &= \frac{P(y=c)}{P(\vec{x})} \cdot \prod_{j=1}^2Ber(x_j|\mu_{jc}) \end{aligned} $$(b) For instance 1, $\vec{x}=\{x_1,x_2\}=\{0,0\}$$$ P(\vec{x},y=0) = P(y=0) \cdot Ber(x_1|\mu_{10}) \cdot Ber(x_2|\mu_{20}) = \frac{3}{8} \times 0 \times \frac{2}{3} = 0 $$$$ P(\vec{x},y=1) = P(y=1) \cdot Ber(x_1|\mu_{11}) \cdot Ber(x_2|\mu_{21}) = \frac{5}{8} \times \frac{4}{5} \times \frac{2}{5} = \frac{1}{5} $$$$ \therefore P(\vec{x}) = 0 + \frac{1}{5}=\frac{1}{5} $$$$ p(y=0|\vec{x}) = \frac{P(\vec{x},y=0)}{P(\vec{x})} = \frac{0}{1/5} = 0 $$$$ p(y=1|\vec{x}) = \frac{P(\vec{x},y=1)}{P(\vec{x})} = \frac{1/5}{1/5} = 1 $$For instance 7, $\vec{x}=\{x_1,x_2\}=\{1,1\}$$$ P(\vec{x},y=0) = P(y=0) \cdot P(\vec{x}|y=0) = P(y=0) \cdot Ber(x_1|\mu_{10}) \cdot Ber(x_2|\mu_{20}) = \frac{3}{8} \times 1 \times \frac{1}{3} = \frac{1}{8} $$$$ P(\vec{x},y=1) = P(y=1) \cdot Ber(x_1|\mu_{11}) \cdot Ber(x_2|\mu_{21}) = \frac{5}{8} \times \frac{1}{5} \times \frac{3}{5} = \frac{3}{40} $$$$ \therefore P(\vec{x}) = \frac{1}{8} + \frac{3}{40} = \frac{1}{5} $$$$ p(y=0|\vec{x}) = \frac{P(\vec{x},y=0)}{P(\vec{x})} = \frac{1/8}{1/5} = \frac{5}{8} $$$$ p(y=1|\vec{x}) = \frac{P(\vec{x},y=1)}{P(\vec{x})} = \frac{3/40}{1/5} = \frac{3}{8} $$Question 2If all classes shares the same covariance matrix$$ \begin{aligned} P(y=c|\vec{x}) & \propto P(y=c) \cdot P(\vec{x}|y) \\ & \propto \pi_c \cdot e^{-\frac{(\vec{x}-\vec{\mu_c})^T \cdot \Sigma^{-1} \cdot (\vec{x}-\vec{\mu_c)}}{2}} \\ &= \pi_c \cdot e^{-\frac{1}{2}\vec{x}^T\Sigma^{-1}\vec{x}} \cdot e^{\frac{1}{…

机器学习知识整理
type: item
date: 2020-09-27T14:02:00+00:00
description: 一直想写一篇文章总结一下机器学习的脉络，网络上零散的知识很多，但我觉得归纳最清晰的还是这张思维导图，来自Vas3k的一篇介绍机器学习（Machine Learning）的文章。从一张思维导图开始机器学习可以分为四大类：经典的机器学习（classical learning）强化学习（reinforcement learning）集成学习（ensemble methods）神经网络和深度学习（neural network and deep learning）当我们数据简单、特征清晰的时候，我们往往采用经典的机器学习方法，你可能会说，我们有那么高级的机器学习方法，譬如强大的神经网络和集成学习，为什么不用呢？当然，不是不能用，比如处理垃圾邮件分类时，我们使用贝叶斯分类器，就能够快速、准确地完成筛选任务，错误率低于1%，我们何必使用复杂数倍的神经网络呢？杀鸡焉用牛刀。而在面对复杂的数据和高维的特征时，我们往往会采用神经网络，甚至深度学习的方法来处理。而三个臭皮匠顶个诸葛亮讲的就是集成学习，当我们对结果的精确度要求较高的时候，我们会训练出多个模型，让它们互相矫正对方的错误，共同参与决策，这样可以提高准确性，降低错误率。但是有时候，我们没有数据，但是我们有能够交互的环境，通过环境的反馈来不断调整自身，最终适应环境，这就是强化学习的思路。这张知识图谱基本覆盖了实际的机器学习应用所用的各种方法，分类、回归、模式搜索、聚类、降维等等，后面我会具体说明，但为了保证知识脉络的完整性，我还会再讲讲我们为什么需要机器学习，而机器学习的三要素又是什么。我们为什么需要机器学习？首先我们需要弄清楚，我们为什么需要机器会学习？这里我引用一下Vas3k的例子。隔壁老王想买一辆车，他首先在互联网上浏览了几十个广告，发现新车大约是20,000美元，一年车龄的二手车是19,000美元，而用过两年的车是18,000美元，依此类推。结果我们聪明的老王同学发现汽车价格取决于车龄，每年下降1,000美元，但不会低于10,000美元。用机器学习的术语，比利发明了回归，他根据已知的历史数据预测了一个价值（价格）。除了估计汽车的价格，人们也尝试过估算iPhone的价格，或者弄清楚烧烤的时候，要根据人数决定带多少串鸡肉串、羊肉串，多少斤金针菇，人们总是这样做。那是不是世界上所有的问题都可以归结为一条简单的公式呢？不幸的是，这是不可能的。让我们回到汽车上。问题是，它们有不同的生产日期，数十种选择，技术条件，季节性需求激增，而估计只有老天爷才知道还有多少其他隐藏的因素。一个普通人在计算价格时，肯定无法将所有这些数据保留在脑海中。人们愚蠢而懒惰，我们需要机器为他们做数学。因此，我们为机器提供一些数据，并要求其查找与价格有关的所有隐藏模式。结果人们发现机器做预测的时候竟然做的比人还好，这让研究人员们异常兴奋。于是机器学习就诞生了。机器学习的三要素数据、特征和算法是机器学习的三要素，理论上而言，所有的机器学习任务都抽象为：根据已知数据对未知结果的预测。我们可以认为这是一个基本命题、基本任务，听起来天方夜谭的人工智能（Artificial Intelligence）也逃脱不了。什么是数据？我们想检测垃圾邮件，那么我们需要分析邮件内容，我们想预测股票，那么我们需要分析过去的股票市场、公司状况、国际市场、政策等等这些信息，这些都是数据。而如何去描述数据，就要用到数据的特征，比如邮件内容中单词出现的频率；而历史股价、成交量、K线、MACD等等这些特征参数则用于描述股票市场，同样，描述公司财务状况，描述国际市场，描述政策，都有不同的特征。最后，算法是我们解决问题的策略、方式和步骤，面对不同的机器学习任务、不同的数据集、不同的特征分布，我们会采取不同的方法，期望以最优的方式完成任务。伴随着机器学习，我们常常听到的数据科学、数据发掘这些名词，它们和机器学习又有什么关系呢？我们可以认为数据科学（Data Science）侧重处理特征，有时候数据的特征不会像文本中的词频这样明显，比如我们需要设计一个人机对话的人工智能，绝对不会仅仅是统计词频这么简单吧？这时候会需要发现或者构造合适的特征参数。而数据发掘（Data Mining）更注重算法这一块，比较著名的一个例子，要从一大段新闻稿中提取出主题（topics），我们需要找到一些频繁一起出现的单词，比如出现data往往后面会跟一个mining，我们会使用Apriori、FP-Growth算法来实现主题的提取。当然这些都是狭义的定义，如今机器学习这个领域里，已经不会太区分数据科学和数据发掘，说什么你是是一个数据科学家，而我是一个数据挖掘家，这样的说法是低级的，因为现在的机器学习任务都是相当复杂和具有挑战性的，我们已经不再严格区分数据科学、数据挖掘、机器学习这些名词的区别…

An appreciation about Picasso's works visualization
type: item
date: 2020-09-27T13:27:00+00:00
description: https://www.nationalgeographic.com/magazine/2018/05/genius-picasso-art-categories-infographic/This visualization presents the works of Picasso in Spanish. It was created after analyzing 8,000 Picasso’s works using Picasso techniques and it's a kind of Convex Treemap. Painted on a canvas, it encodes the number of works that Picasso worked on a given theme by the area of partition. The larger partition is, the more works that he created. The 12 major themes and subcategories are encoded with different color hue, grain, shapes and positions.Why this visualization impresses me? In one word, the visualization builds a bridge between Picasso's art styles and us. For instance, we can see there is a woman face in the middle of the picture. She has eyes, nose and even mouth. Zooming this part, we can see more details. Picasso prefers using orange-yellow and pieces objects to paint human faces. And he’s good at using the combination of pieces to reflect human’s emotion. This is his style and the design of this visualization imitates it. The shape, color and grain of each visualization category are related to the arts styles that it belongs to.Yes, we can have an overview about Picasso's works distribution through watching the partitions size and distribution. The encoding scheme is efficient. However, more importantly, the visualization itself is a work, a reproduction of Picasso's arts, a kind of respect to master Picasso. The color, size, shape and grain --- all these elements not only display the data, but also show the Picasso’s arts styles and his individual spirit on arts. What this visualization conveys is meaningful. We can see all the Picasso's art styles in one picture. It seems that it brings Picasso back to life and make all of us feel the glory of Picasso in the past.