uml.org.cn - TensorFlow??????????NLP
运用TensorFlow处理简单的NLP问题-人工智能-火龙果软件









Search Preview

运用TensorFlow处理简单的NLP问题-人工智能-火龙果软件

uml.org.cn
body { margin: 0px;padding:0 } #div1 { display: none; position: absolute; z-index: 1000; height: 100%; width: 100%; backgrou
.cn > uml.org.cn

SEO audit: Content analysis

Language Error! No language localisation is found.
Title 运用TensorFlow处理简单的NLP问题-人工智能-火龙果软件
Text / HTML ratio 55 %
Frame Excellent! The website does not use iFrame solutions.
Flash Excellent! The website does not have any flash contents.
Keywords cloud
Keywords consistency
Keyword Content Title Description Headings
Headings Error! The website does not use (H) tags.
Images We found 102 images on this web page.

SEO Keywords (Single)

Keyword Occurrence Density

SEO Keywords (Two Word)

Keyword Occurrence Density

SEO Keywords (Three Word)

Keyword Occurrence Density Possible Spam

SEO Keywords (Four Word)

Keyword Occurrence Density Possible Spam

Internal links in - uml.org.cn

????
文章-火龙果软件工程
iProcess
iProcess--火龙果软件工程技术中心
?ֲ?ʽϵͳ?еĻ????ܹ?
一文深入了解：分布式系统中的缓存架构-数据库
BitbucketPipeline+AmazonS3
基于 Bitbucket Pipeline + Amazon S3 的自动化运维体系-IT运维管理
΢??΢???????ؿ???ϵͳ
DAGOR：微信微服务过载控制系统-移动端开发
???ݿ?
数据库-火龙果软件工程
IT??ά????
IT运维管理-火龙果软件工程
΢????
微服务-火龙果软件工程
????vuex??״̬????ģʽ?ܹ?
理解vuex的状态管理模式架构
TensorFlow??????????NLP
运用TensorFlow处理简单的NLP问题-人工智能-火龙果软件
web????
JavaScript-火龙果软件工程
?˹?????
人工智能-火龙果软件工程
????TensorFlow?ڲ?ʵ??ԭ??
从系统和代码实现角度解析TensorFlow的内部实现原理-人工智能-火龙果软件
ʹ??IPython??ǿ????ʽ????
使用IPython增强交互式体验-Python
?????Ͳ?????????????Ӧ??
在 IBM Cloud 上创建和部署区块链聊天应用
python
python-火龙果软件工程
?Ƽ???
云计算-火龙果软件工程
VUE.JS??????????ʵ??
VUE.JS组件化开发实践
???????ɵ?·????C++???߳̿???
瑞萨C++多线程开发培训-火龙果软件
?й????Ų?Ʒ????
产品经理产品管理培训| 产品经理培训
ծȯ???ۻ??? ʹ??Selenium?????Զ???
使用Selenium进行有效的自动化测试-火龙果软件-UML软件工程组织
????֤ȯ?߿????ƶ???̨?ܹ?????
高可用架构培训|软件架构培训-火龙果软件
E2Biz ????UML+EA ҵ????ϵͳ??ģ
基于UML+EA进行嵌入式系统分析设计培训-火龙果软件-UML软件工程组织
???Ź???????????ѵ
工作量估算管理培训-火龙果软件
????
培训案例-火龙果软件
??ΰ?? ͨ????????֤???????ܹ?ʦ
软件架构师认证|软件架构师培训-火龙果软件
???? ͨ????????֤???????ܹ?ʦ
软件架构师认证|软件架构师培训-火龙果软件
??ѧ?? ͨ????????֤???????ܹ?ʦ
软件架构师认证|软件架构培训-火龙果软件
???? ͨ????????֤???????ܹ?ʦ
软件架构师认证|软件架构师培训-火龙果软件
Githubʹ?ÿ???????
Github使用快速上手-配置管理
?? Vuex ????һ???ʼ?Ӧ??
用 Vuex 构建一个笔记应用 -web开发
???ù???
配置管理-火龙果软件工程
????matplotlib?????ݿ??ӻ?
基于matplotlib的数据可视化 -Python
OnlineLearning?㷨??????ʵ??
Online Learning算法理论与实践-火龙果软件
????
每月技术月刊--UML软件工程组织
????????
火龙果软件工程技术中心
??ϵ????
联系我们-UML软件工程组织

Uml.org.cn Spined HTML

TensorFlow򵥵NLP-˹- Ծ֧ǵĹҵ 1Ԫ 10Ԫ 50Ԫ ֤룺     ֪ Ŀ Lib Ƶ Code iProcess γ ֤ ѯ   ɳ֮·   Ա          Ҫ      ÿ15ƪ ı ׷                 TensorFlow򵥵NLP   43     ۣ    2018-11-30   ༭Ƽ: cnblogs½ģ͡ѭ缰TensorFlow֪ʶ ǰ˹ܡǼ̡ݡһٵĴʣÿҹ˾Ҫ˹ܣ͸4-5ǰһҵеĶdz뵽֮ǰһ˵Ļ Big Data is like teenage sex: Everyone talks well-nigh it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims. ڿġBig DataԻɡAIˣڴһû״ݵʱ˹ܾͿʼһˡŸ̬ȣҲȥ̽ ǰѧ绹ǹҵ磬ѧϰܵ׷GoogleԴѧϰƽ̨TensorFlow֮󣬸ǸѧϰϽ͡ĿǰڿԴGithubпԴĿУTensorFlowΪԾƳڣ˼汾ݽ˵ܹЧؽʵ⡣ҪԲTensorFlowȻԴ(NLP)ļӦãôһӸԵʶTensorFlow ˵NLPʵҶǺϤ֮ǰҲδйNLPؾ飬ѧϰTensorFlowһЩۣ͵שˡǰÿ춼ڲıƵݣͨھЩݣǿһЩӱݵӦã롢ʶ𡢴ԱעԼϢȣЩNLP롣NLPУģһڣҪΧģչȽԭ(word2vec)ѭ(RNN)ʱ(LSTM)ѧϰģͣϸ TensorFlow ʵģ͡ ģ ģһָģͣǻһϿⴴõÿӳֵĸʣͨһ㽲ǿһ仰Dz˵ģѧϱʾΪ: ʽǣһӳֵĸʵڸǰĴ£źĴʳֵĸʡͨʹʽչõ ǴģҪIJÿʵΪ:ǰĴԤһʵĸʡЩʲһӣͿͨϹʽõһӳֵĸʡһ仰phpõԡҲȷDzȻԣѾִΪphpǡõġԡôֵĸΪP(phpǡõġԡ)=P(php)P(ǡ|php)P(õġ|phpǡ)P(ԡ|phpǡõġ)ʽϴôжΪһ仰Щͨ±Ҷ˹ʽõ ݴʽֿԽΪ Ͽ N ʣһӳΪ T ô NT ֿܣÿһֿܶҪ T ʲҪ TNT 棬󣬶ڴҪҲǾˡôαأ֮ǰٵķвͨô˼·һ͵Ĵǽʽ½ƣ ˼˵һʳֵĸֻǰ n-1 йأǰеĴйأļͳƵĿԣ˼Чʣִ֮Ϊ n-gram ģͣͨ n ȡ2~3ܵõЧܽn-gram ģ;ͳϿдʴֵĴһԼõʴĸʲ䱣Ԥһʱֱͨǰʹʽõӳֵĸʡ ҲģͣӻѧϰĽǶһʼȫЩʴĸֵͨһģͶԴʴĸʽнģȻһĿ꺯ŻĿ꣬õһŻIJҪĸʴʱŻIJֱӼõӦĴʴʡʴ P(w|context(w)) w context(w) ĺ context(w) ʾ w ģ൱ǰ n-gram ģ͵ǰ n-1 ʣôѧʾ Ŀ꺯öȻʾ( NN ϿдʵĴС) ͨŻ㷨СĿ꺯õһŻIJ в ΪȨֵƫáôѧϰģ[1]ʱαʾһأͨڻѧϰǽһΪһƵأģǽ()ʾΪͨword2vecôģ;ͿԱʾʾͼ 㡢ͶӰ㡢زԼ㣬ͶӰֻǶһԤдʽһӲһʱʾΪ m άô n-1 ӺΪ (n-1)m άӺΪ룬زٵ㣬 W U ֱΪͶӰ㵽ز㡢ز㵽Ȩֵp q ֱΪͶӰ㵽ز㡢ز㵽ƫòѧ£ ΪsigmoidΪزļΪ NN άӦϿдʵĴСһҪپsoftmaxһΪʽõԤϿÿʵĸʡģͿƺܼ򵥣ǴôأνһתΪʽأϸ (word2vec) Ҫ¾ǽѧʾDz One-hot Representation ʾһʣϿʵ NN ʣôάΪ NN ÿʱţڵ i ʣʾ˵ i ԪΪ1ԪΪ0 NN άִȱԶ׼һ˵ϿĴʵģرôάͷdz󣬲Ҵ֮ûйԣʵؿ̻Աʣ硰ѶС硱ͨOne-hotʾûκιΪ˿˷One-hot Representation ȱ㣬Mikolovһ Distributed Representation[2]˵⻰ڴҶݱCNNͼʶʱȴо紦NLP⣬˴NLPĸˮƽģΪһ֮һ˼壬Distributed Representation ǰѴʵϢֲͬķϣ One-hot Representation Ϣһϣǽӳ䵽 m άռ䣬ʾΪ m άҲ֮Ϊ Word EmbeddingһԼСάȣһ棬ԽйĴӳΪռڵĵ㣬֮Ĺͨռ̻ͼʾ ʱӳ䵽3άռ䣬ÿʱʾΪһ3άĴĽϽԿϵĴʣ֮ĴҲࡣ ҪõҪģѵõ˵ѵģ͹еõĸƷword2vecģͣ CBOW Skip-Gram ģ[3]ͼʾ CBOW ģǸݴʵԤ⵱ǰʣɴԤʵǰ c ɡ Skip-Gram ģ෴ͨǰȥԤġһϿΪѵͿͨģѵÿʵʾʵCBOW ģͻƽһЩֲϢΪʵΪ Skip-Gram ģͽIJΪѵõĽΪȷΪˣTensorFlow word2vec õ Skip-Gram ģͣӦ[2]һָΪŻ Skip-Gram ģͣؽԭ CBOW Skip-Gram ģϸڿԲ[3] Skip-Gram ģ ǰҲᵽ Skip-Gram ģǸݵǰȥԤģ䣺 php õ ԡ ٶɴԤʵǰ2ɣôϾӿԵõ (, ), (, php), (, õ), (, ), (õ, ), ѵĿΪ¶Ȼ c Ϊĵľ޶ȡ wt ǰc ʽԤ⡣c ԽѵȷǼ㸴ӶȼӴѵɱӦҲһȡ c Ϊ23ѵĽ Skip-Gram ģͲsoftmaxĿ꺯и p(wi+j|wi) Ϊ vw ʾ w w ʾԤΪ w Ȩֵ߶ǴѵIJѷ֣ͨϹʽÿʵʧҪõʵедʣһʵdzַʽDzʵʵġһ(ϡ, php)޷ǸݴʡϡȥԤʡphpôͿԿһ⣬ʡϡԤ⡰phpΪԤΪʿdzphpдʣΪ˼򻯼㣬ͨķʽÿгphpĴȡ k ΪôѵĿתΪ߼عĿ꺯 ϱʽ֮Ϊ NCE(Noise-contrastive estimation)[4]Ŀ꺯еȺұߵڶʾͨһ Pn(w)ֲIJ㷨ȡ kk ʧ[2]вһ򵥵һԪֲ˼㣬֮Ϊ(Negative Sampling)ϸܸ㷨 㷨 ʵеÿϿгֵƵиеͣ˵ЩƵʣѡΪĸʽϴ󣬶ЩƵʣѡΪĸʽСʵͨȨʵ֣ÿʵĴƵʾΪλ߶ϵһСֶΣڴʵСΪ NN Ͽ⣬ԽʵеĴʱʾΪλ߶ϵһ㣬ڵλ߶ϵȾ뻮 MM ȷ֣ M>>NM>>N ̾õһ i<Mi<MͨӳҵӦĴʣʾ [2]ʵʸƵʱһǼ򵥵ͳƴʵijִǶԴʵijִ ݴƵʽΪ Ƶʶβ һϿУܶೣĴʴ֣硰ġǡȡЩȻƵϸߣṩϢȴ١һ˵ЩƵʵĴѵ̫ı仯ΪѵٶȣƽƵʺ͸Ƶʣ[2]һԸƵʶβļɣÿʣ¸ʶѵ f(wi)ʾƵʽвѷ֣βЩ f(wi)>t νĸƵЧ t ϿĴСãһΪ 10-5 ҡ TensorFlowʵ ʵԭϴTensorFlowʵһ׵word2vecģ[5]TensorFlowḻapiԼǿļ棬ǿԷdzرģ͡ϿΪѵݣɨϿ⽨ֵ䣬ΪÿʱţͬʱЩƵmin_countĴʹ˵Щİɴһ(ϡ, php)øõɸʵֱΪϡphpԼɸlogitֵͨعʽõĿ꺯(3-3) ͼ ȶҲΪ embedding matrixҪͨѵõĴvocabulary_sizeʾʵСembedding_sizeʾάȣôΪ vocabulary_size embedding_sizeþȷֲʼ embeddings = tf.Variable( tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0)) ȨֵƫӦ3-3ʽе ȣʼΪ0 weights = tf.Variable( tf.truncated_normal([vocabulary_size, embedding_size], stddev=1.0 / math.sqrt(embedding_size))) biases = tf.Variable(tf.zeros([vocabulary_size])) һbatch룬ӴҵӦʾԼȨֵƫҵӦȷIJexamplesʣlabelsΪӦȷһάʾÿԪΪֵбţ # Embeddings for examples: [batch_size, embedding_size] example_emb = tf.nn.embedding_lookup(embeddings, examples) # Weights for labels: [batch_size, embedding_size] true_w = tf.nn.embedding_lookup(weights, labels) # Biases for labels: [batch_size, 1] true_b = tf.nn.embedding_lookup(biases, labels) õɷȷlabels_matrixΪȷʣʱЩʣnum_sampledΪdistortionΪʽ(3-4)еָ labels_matrix = tf.reshape( tf.cast(labels, dtype=tf.int64), [batch_size, 1]) # Negative sampling. sampled_ids, _, _ = tf.nn.fixed_unigram_candidate_sampler( true_classes=labels_matrix, num_true=1, num_sampled=num_samples, unique=True, range_max=vocab_size, distortion=0.75, unigrams=vocab_counts.tolist()) ҵӦȨֵƫò # Weights for sampled ids: [num_sampled, embedding_size] sampled_w = tf.nn.embedding_lookup(weights, sampled_ids) # Biases for sampled ids: [num_sampled, 1] sampled_b = tf.nn.embedding_lookup(biases, sampled_ids) ֱȷͷȷlogitֵ WX+bͨصõĿ꺯(3-3) # True logits: [batch_size, 1] true_logits = tf.reduce_sum(tf.mul(example_emb, true_w), 1) + true_b # Sampled logits: [batch_size, num_sampled] # We replicate sampled noise lables for all examples in the batch # using the matmul. sampled_b_vec = tf.reshape(sampled_b, [num_samples]) sampled_logits = tf.matmul(example_emb, sampled_w, transpose_b=True) + sampled_b_vec # cross-entropy(logits, labels) true_xent = tf.nn.sigmoid_cross_entropy_with_logits( true_logits, tf.ones_like(true_logits)) sampled_xent = tf.nn.sigmoid_cross_entropy_with_logits( sampled_logits, tf.zeros_like(sampled_logits)) # NCE-loss is the sum of the true and noise (sampled words) # contributions, averaged over the batch. loss = (tf.reduce_sum(true_xent) + tf.reduce_sum(sampled_xent)) / batch_size ѵģ ͼϺҪȥŻĿ꺯ݶ½𲽸²ҪȷѧϰŵУ𲽼ѧϰtrained_wordsΪѵĴwords_to_trainΪдѵĴ lr = init_learning_rate * tf.maximum( 0.0001, 1.0 - tf.cast(trained_words, tf.float32) / words_to_train) Żӣʹݶ½ѵģͣ optimizer = tf.train.GradientDescentOptimizer(lr) train = optimizer.minimize(loss, global_step=global_step, gate_gradients=optimizer.GATE_NONE) session.run(train) ֤ ϲ󣬼ɵõ󣬼еıembeddingsô֤õĴĺûأMikolov˷[2]һԹϵĴʣڿռе߽ƽУͼʾ Ϊˣ׼Լÿа4һԪ (w1,w2,w3,w4) һϺõĴÿԪ»¹ϵ ѭ(RNN) ಻ǴһƬհ׿ʼ˼һƪµʱǰȥģÿοһʺһʵʱִͷʼͳģǴ㵽زٵ㣬ÿ֮Ľڵӵģͨ粻߱书ܣѭ(Recurrent Neural NetworkRNN)⣬߱ԣͨڴʱ⣬ڶNLPУRNNȡ˾޴ɹԼ㷺Ӧá RNNУһеǰ뵱ǰй⣬ǰҲйأͼΪRNNһԪĽṹʾͼͼƬԴ[7] ͼܻǺ󣬸ʱнͼƽչõͼʽʾ RNN صģ RNN ˵Ȼܹ Ȼ RNN һȱ㣬ȻԽ֮ǰϢӵǰϣǰ֮ǰϢʱȺܴݶ˥ԭRNN ѧϰԶϢ½֮ΪʱLong-Term Dependencies⡣Ԥһ仰ɻϡһʣܲҪ̫ľͿԤ⵽һΪɡ£ϢҪԤĴ֮ʱȺСRNN Ժѧ֮ǰϢٱԤ⡰ԷὲһʣӵǰϢһʿһԣҪ׼ȷԤԣҪȥǰϢˣǰĵġ뵱ǰλõʱȽϴRNNѧԶϢ೤ʱϸڲο[8]˵ǣһ RNN ֣ʱ(Long Short Term Memory networks, LSTM)Խ⡣ ʱ(LSTM) LSTM һִѡԼ书ܵ RNNЧĽʱ⣬ѧϰ֮ǰĹؼϢͼʾΪ LSTM չʾͼ RNN , LSTM ֻÿԪṹ˸Ľ RNN УÿԪṹֻе LSTM ÿԪṹΪӣһ״̬ߣͼˮƽߣԼס֮ǰѧϢ(gate)״ֱ̬ΪšźšŵѡԵؽ֮ǰҪϢԱ洢ϢǸݵǰѧϰϢȻµǰ״̬ǽϵǰ͵ǰ״̬õһΪ⣬Ϊһʱ̵롣ѧķʽÿŵ˼ ţҪϢ£ ţҪӵϢ£ ôźţ״̬£ ţõϢ£ LSTM Ԫ붼һʱ̵뵱ǰʱ̵ͨconcatӶõ룬sigmoidΪŵɸѡֱõ ft it otɸѡֱѡ񲿷ַ״̬ѡǡѡ롢ѡ LSTM ṹԭϣݲͬʵӦóݱܶ LSTM ı壬 LSTM ϸο[7]һ LSTM [9]ýṹҲ TensorFlow LSTM ʵֵĸ[10] LSTM ѧϰصǰѾ LSTM ṹ LSTM ʵǽ LSTM ӣγɶز㣬ͼʾ ͼÿ LSTM Ԫڲṹͼʾ l t ʱ˵hlt-1 Ϊ l t-1 ʱ̣һʱ̣hl-1t Ϊ l-1-㣨һ㣩 t ʱ̵Ϊ l t ʱ̵롣 ĽṹԵõ l LSTM ѧ, clt-1 ʾһʱ̵״̬clt ʾɵǰº״̬ Ȼʵ֤ģ LSTM ׹ϣʵӦУҪȡ򻯷ϣг򻯷Dropout[11][12]һּ򵥸ЧDropout RNN/LTSM 硣ͼʾDropoutӦ߷룬һDropout ͼDropoutԣʽ(5-5)Ըдʽ DD ʾDropoutؽ hl-1t еķΪ㡣ͼʾɫʵ߱ʾ t-2 ʱ̵Ϣ t+2t+2 ʱΪԤIJο L+1 εDropout LL ʾIJ TensorFlowʵ ǰ LSTM ģԭʵ֮ǰᵽģͣǰԤһʣ롰ɻϡԤһʡɡʹ TensorFlow ʵ LSTM dzķ㣬Ϊ TensorFlow Ѿṩ˻ LSTM ԪṹOperationʵԭǻ[12]ĴDropout LSTM ģ͡οptb_word_lm.py LSTMģ TensorFlowṩOperationʵ LSTM ܼ򵥣ȶһ LSTM ԪsizeΪ LSTM ԪάȣٶDropout LSTM IJnum_layersõ RNN ṹԪ lstm_cell = tf.nn.rnn_cell.BasicLSTMCell(size, forget_bias=0.0) lstm_cell = tf.nn.rnn_cell.DropoutWrapper( lstm_cell, output_keep_prob=keep_prob) lamina = tf.nn.rnn_cell.MultiRNNCell([lstm_cell] * num_layers)   ÿθһbatch룬 LSTM ״̬ʼΪ0ʵɴʾȶһembeddingԲҪһʼûУѵеõģΪѵĸƷLSTMչnum_stepsÿһһbatchĴΪ룬 LSTM Ԫ״̬²õͨsoftmaxһʧ initial_state = cell.zero_state(batch_size, tf.float32) embedding = tf.get_variable("embedding", [vocab_size, size]) # input_data: [batch_size, num_steps] # targets [batch_size, num_steps] input_data = tf.placeholder(tf.int32, [batch_size, num_steps]) targets = tf.placeholder(tf.int32, [batch_size, num_steps]) inputs = tf.nn.embedding_lookup(embedding, input_data) outputs = [] for time_step in range(num_steps): (cell_output, state) = cell(inputs[:, time_step, :], state) outputs.append(cell_output) output = tf.reshape(tf.concat(1, outputs), [-1, size]) softmax_w = tf.get_variable("softmax_w", [size, vocab_size]) softmax_b = tf.get_variable("softmax_b", [vocab_size]) logits = tf.matmul(output, softmax_w) + softmax_b loss = tf.nn.seq2seq.sequence_loss_by_example( [logits], [tf.reshape(targets, [-1])], [tf.ones([batch_size * num_steps])]) ѵģ 򵥲ݶ½Żʧ𲽵ֱõfinal_stateΪLSTMҪѧϰIJ optimizer = tf.train.GradientDescentOptimizer(lr) train_op = optimizer.minimize(loss) for i in range(max_epoch): _, final_state = session.run([train_op, state], {input_data: x, targets: y})   ֤ģ ģѵϺѾõLSTM״̬룬LSTM󼴿ɵõˡ (cell_output, _) = cell(inputs, state) session.run(cell_output) С ʹTensorFlowѧϰʱDzҪ̫עڲʵϸڣֻѾŵģ͵ĹϣTensorFlowѾṩijԪṹͿԹģ͡ҲǡǡΪTensorFlowĸ߶ȳ󻯣ʱķѾʹTensorFlowĹУҪϸ̫һݿTensorɣTensorIJ㣬ҪԺεϸڵȵȣȻͻ       43  :   Ǹݿ ݿƾ̸ ݿƹ ݿܽ   ĵ ݿܵż ݿܵ ݿŻ ݿϵͳܵϵ ؿγ ݿŻ ߼ݿܹʦ ݲֿھ Hadoopԭܵ   APPƹ֮ù߽ݷ Hadoop Hivesql﷨ Ӧö༶ģʽ֧ź HBase ϸ HBaseϸ Spark̬Դ ...    ѵγ HadoopSparkݼܹ Hadoopԭ߼ʵ HadoopԭӦŻ ϵӦ ݵļʵ Sparkݴ γ...    ɹ GE ʵѵ ƹijӹ˾ Nodejs߼Ӧÿ ʢ滪 ׿Խ߱߱ ijϢ˾ Pythonѵ ijITϵͳ Բ йʴ Գģͼ(TMMI) Ժ ƷƷ ...          ÿ2ĵ/Ƶ ɨ΢Ŷά붩 ļ¿ ÿ300Դ     | ϵ | ICP10020922 110108001071