我正在學(xué)習(xí)一個大數(shù)據(jù)類，我的一個項目是在本地建立的Hadoop集群上運行Mapper/Reducer。在

我一直在為類使用Python和MRJob庫。在

下面是我當前用于Mapper/Reducer的Python代碼。在from mrjob.job import MRJob

from mrjob.step import MRStep

import re

import os

WORD_RE = re.compile(r"[\w']+")

choice = ""

class MRPrepositionsFinder(MRJob):

def steps(self):

return [

MRStep(mapper=self.mapper_get_words),

MRStep(reducer=self.reducer_find_prep_word)

]

def mapper_get_words(self, _, line):

# set word_list to indicators, convert to lowercase, and strip whitespace

word_list = set(line.lower().strip() for line in open("/hdfs/user/user/indicators.txt"))

# set filename to map_input_file

fileName = os.environ['map_input_file']

# itterate through each word in line

for word in WORD_RE.findall(line):

# if word is in indicators, yield chocie as filename

if word.lower() in word_list:

choice = fileName.split('/')[5]

yield (choice, 1)

def reducer_find_prep_word(self, choice, counts):

# each item of choice is (choice, count),

# so yielding results in value=choice, key=count

yield (choice, sum(counts))

if __name__ == '__main__':

MRPrepositionsFinder.run()

當我試圖在Hadoop集群上運行代碼時，我使用了以下命令：

^{pr2}$

不幸的是，每次運行該命令時都會出現(xiàn)以下錯誤：No configs found; falling back on auto-configuration

STDERR: Error: JAVA_HOME is not set and could not be found.

Traceback (most recent call last):

File "hrc_discover.py", line 37, in

MRPrepositionsFinder.run()

File "/usr/lib/python3.5/site-packages/mrjob-0.6.0.dev0-py3.5.egg/mrjob/job.py", line 432, in run

mr_job.execute()

File "/usr/lib/python3.5/site-packages/mrjob-0.6.0.dev0-py3.5.egg/mrjob/job.py", line 453, in execute

super(MRJob, self).execute()

File "/usr/lib/python3.5/site-packages/mrjob-0.6.0.dev0-py3.5.egg/mrjob/launch.py", line 161, in execute

self.run_job()

File "/usr/lib/python3.5/site-packages/mrjob-0.6.0.dev0-py3.5.egg/mrjob/launch.py", line 231, in run_job

runner.run()

File "/usr/lib/python3.5/site-packages/mrjob-0.6.0.dev0-py3.5.egg/mrjob/runner.py", line 437, in run

self._run()

File "/usr/lib/python3.5/site-packages/mrjob-0.6.0.dev0-py3.5.egg/mrjob/hadoop.py", line 346, in _run

self._find_binaries_and_jars()

File "/usr/lib/python3.5/site-packages/mrjob-0.6.0.dev0-py3.5.egg/mrjob/hadoop.py", line 361, in _find_binaries_and_jars

self.get_hadoop_version()

File "/usr/lib/python3.5/site-packages/mrjob-0.6.0.dev0-py3.5.egg/mrjob/hadoop.py", line 198, in get_hadoop_version

return self.fs.get_hadoop_version()

File "/usr/lib/python3.5/site-packages/mrjob-0.6.0.dev0-py3.5.egg/mrjob/fs/hadoop.py", line 117, in get_hadoop_version

stdout = self.invoke_hadoop(['version'], return_stdout=True)

File "/usr/lib/python3.5/site-packages/mrjob-0.6.0.dev0-py3.5.egg/mrjob/fs/hadoop.py", line 172, in invoke_hadoop

raise CalledProcessError(proc.returncode, args)

subprocess.CalledProcessError: Command '['/usr/bin/hadoop', 'version']' returned non-zero exit status 1

我瀏覽了一下互聯(lián)網(wǎng)，發(fā)現(xiàn)我需要導(dǎo)出我的JAVA_HOME變量-但我不想設(shè)置任何可能破壞我的設(shè)置的東西。在

如有任何幫助，將不勝感激，謝謝！在

總結(jié)

以上是生活随笔為你收集整理的python集群到hadoop_如何使用Hadoop流在本地Hadoop集群中运行MRJob？的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。