Gauss.Deng
- 浏览: 28833 次
- 性别:
- 来自: 深圳
-
文章分类
最新评论
-
jnh：
写的不错啊。标题多了一个“的”。官方文档：http://wik ...
使用Git下载Hadoop的到本地Eclipse开发环境 -
camlelxy：
按照lz的配置没有成功 按照文档Setup-----Inst ...
使用Git下载Hadoop的到本地Eclipse开发环境
问题场景
按照官网http://wiki.apache.org/hadoop/EclipseEnvironment指导，要把Hadoop下载到本地，并构建Eclipse开发环境，只需要三条指令：
即可，但当我在本地执行完第二条后，报出如下错误日志信息：
此问题我暂时还未分析具体原因和解决方案，暂时记录下。
展开分析
通过再此使用命令打出错误信息：
得到详细错误信息为：
通过上面错误信息，真方便找到解决方案。
根据上面的提示，访问https://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
得到如下提示信息：
Unlike many other errors, this exception is not generated by the Maven core itself but by a plugin. As a rule of thumb, plugins use this error to signal a problem in their configuration or the information they retrieved from the POM.
这里说的意思是：
这个错误不是Maven本身的错误，根据经验，可能是Maven使用的插件通过这个异常来标识它们没有从POM中获得相关的配置信息。
接下来进一步分析，通过Maven构建Hadoop过程中是否使用了插件。
从错误日志分析，编译过程使用了插件：maven-antrun-plugin。由于编译Hadoop-common过程中出错，所以进一步定位到hadoop-common工程下的POM.xml，可到看到下面信息：
上面是在Pom.xml文件中使用Maven下使用Ant插件，里面有一行：
看着让人不解，联系到HowToContribueToHadoop的文章http://wiki.apache.org/hadoop/HowToContribute可以推知，有可能由于本地没有安装ProtocolBuffers引起的，因为文章内部特别说明了：
如预期，在本地安装好了Protoc Buffer后，后面两条指令顺利执行完整，剩下的就依据官网把目录下的工程导入Eclipse后，就可以在Eclipse下学习调试源码。
按照官网http://wiki.apache.org/hadoop/EclipseEnvironment指导，要把Hadoop下载到本地，并构建Eclipse开发环境，只需要三条指令：
$ git clone git://git.apache.org/hadoop-common.git $ mvn install -DskipTests $ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
即可，但当我在本地执行完第二条后，报出如下错误日志信息：
[INFO]
[INFO] --- maven-antrun-plugin:1.6:run (compile-proto) @ hadoop-common ---
[INFO] Executing tasks
main:
[exec] target/compile-proto.sh: line 17: protoc: command not found
[exec] target/compile-proto.sh: line 17: protoc: command not found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [2.389s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [0.698s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [1.761s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [0.729s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.353s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [1.998s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [1.227s]
[INFO] Apache Hadoop Common .............................. FAILURE [1.132s]
[INFO] Apache Hadoop Common Project ...................... SKIPPED
[INFO] Apache Hadoop HDFS ................................ SKIPPED
[INFO] Apache Hadoop HttpFS .............................. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
[INFO] Apache Hadoop HDFS Project ........................ SKIPPED
[INFO] hadoop-yarn ....................................... SKIPPED
[INFO] hadoop-yarn-api ................................... SKIPPED
[INFO] hadoop-yarn-common ................................ SKIPPED
[INFO] hadoop-yarn-server ................................ SKIPPED
[INFO] hadoop-yarn-server-common ......................... SKIPPED
[INFO] hadoop-yarn-server-nodemanager .................... SKIPPED
[INFO] hadoop-yarn-server-web-proxy ...................... SKIPPED
[INFO] hadoop-yarn-server-resourcemanager ................ SKIPPED
[INFO] hadoop-yarn-server-tests .......................... SKIPPED
[INFO] hadoop-mapreduce-client ........................... SKIPPED
[INFO] hadoop-mapreduce-client-core ...................... SKIPPED
[INFO] hadoop-yarn-applications .......................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell ......... SKIPPED
[INFO] hadoop-yarn-site .................................. SKIPPED
[INFO] hadoop-mapreduce-client-common .................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED
[INFO] hadoop-mapreduce-client-app ....................... SKIPPED
[INFO] hadoop-mapreduce-client-hs ........................ SKIPPED
[INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED
[INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
[INFO] hadoop-mapreduce .................................. SKIPPED
[INFO] Apache Hadoop MapReduce Streaming ................. SKIPPED
[INFO] Apache Hadoop Distributed Copy .................... SKIPPED
[INFO] Apache Hadoop Archives ............................ SKIPPED
[INFO] Apache Hadoop Rumen ............................... SKIPPED
[INFO] Apache Hadoop Extras .............................. SKIPPED
[INFO] Apache Hadoop Tools Dist .......................... SKIPPED
[INFO] Apache Hadoop Tools ............................... SKIPPED
[INFO] Apache Hadoop Distribution ........................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.483s
[INFO] Finished at: Mon Jan 30 22:57:23 GMT+08:00 2012
[INFO] Final Memory: 24M/81M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (compile-proto) on project hadoop-common: An Ant BuildException has occured: exec returned: 127 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-common
此问题我暂时还未分析具体原因和解决方案，暂时记录下。
展开分析
通过再此使用命令打出错误信息：
$ mvn install -DskipTests -e
得到详细错误信息为：
[INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 9.387s [INFO] Finished at: Mon Jan 30 23:11:07 GMT+08:00 2012 [INFO] Final Memory: 19M/81M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (compile-proto) on project hadoop-common: An Ant BuildException has occured: exec returned: 127 -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (compile-proto) on project hadoop-common: An Ant BuildException has occured: exec returned: 127 at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: exec returned: 127 at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:283) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 19 more Caused by: /Users/apple/Documents/Hadoop-common-dev/hadoop-common/hadoop-common-project/hadoop-common/target/antrun/build-main.xml:23: exec returned: 127 at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:650) at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:676) at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:502) at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) at org.apache.tools.ant.Task.perform(Task.java:348) at org.apache.tools.ant.Target.execute(Target.java:390) at org.apache.tools.ant.Target.performTasks(Target.java:411) at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397) at org.apache.tools.ant.Project.executeTarget(Project.java:1366) at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:270) ... 21 more [ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR]
通过上面错误信息，真方便找到解决方案。
根据上面的提示，访问https://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
得到如下提示信息：
Unlike many other errors, this exception is not generated by the Maven core itself but by a plugin. As a rule of thumb, plugins use this error to signal a problem in their configuration or the information they retrieved from the POM.
这里说的意思是：
这个错误不是Maven本身的错误，根据经验，可能是Maven使用的插件通过这个异常来标识它们没有从POM中获得相关的配置信息。
接下来进一步分析，通过Maven构建Hadoop过程中是否使用了插件。
从错误日志分析，编译过程使用了插件：maven-antrun-plugin。由于编译Hadoop-common过程中出错，所以进一步定位到hadoop-common工程下的POM.xml，可到看到下面信息：
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<id>compile-proto</id>
<phase>generate-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<echo file="target/compile-proto.sh">
PROTO_DIR=src/main/proto
JAVA_DIR=target/generated-sources/java
which cygpath 2> /dev/null
if [ $? = 1 ]; then
IS_WIN=false
else
IS_WIN=true
WIN_PROTO_DIR=`cygpath --windows $PROTO_DIR`
WIN_JAVA_DIR=`cygpath --windows $JAVA_DIR`
fi
mkdir -p $JAVA_DIR 2> /dev/null
for PROTO_FILE in `ls $PROTO_DIR/*.proto 2> /dev/null`
do
if [ "$IS_WIN" = "true" ]; then
protoc -I$WIN_PROTO_DIR --java_out=$WIN_JAVA_DIR $PROTO_FILE
else
protoc -I$PROTO_DIR --java_out=$JAVA_DIR $PROTO_FILE
fi
done
</echo>
<exec executable="sh" dir="${basedir}" failonerror="true">
<arg line="target/compile-proto.sh"/>
</exec>
</target>
</configuration>
</execution>
<execution>
<id>compile-test-proto</id>
<phase>generate-test-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<echo file="target/compile-test-proto.sh">
PROTO_DIR=src/test/proto
JAVA_DIR=target/generated-test-sources/java
which cygpath 2> /dev/null
if [ $? = 1 ]; then
IS_WIN=false
else
IS_WIN=true
WIN_PROTO_DIR=`cygpath --windows $PROTO_DIR`
WIN_JAVA_DIR=`cygpath --windows $JAVA_DIR`
fi
mkdir -p $JAVA_DIR 2> /dev/null
for PROTO_FILE in `ls $PROTO_DIR/*.proto 2> /dev/null`
do
if [ "$IS_WIN" = "true" ]; then
protoc -I$WIN_PROTO_DIR --java_out=$WIN_JAVA_DIR $PROTO_FILE
else
protoc -I$PROTO_DIR --java_out=$JAVA_DIR $PROTO_FILE
fi
done
</echo>
<exec executable="sh" dir="${basedir}" failonerror="true">
<arg line="target/compile-test-proto.sh"/>
</exec>
</target>
</configuration>
</execution>
<execution>
<id>save-version</id>
<phase>generate-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<mkdir dir="${project.build.directory}/generated-sources/java"/>
<exec executable="sh">
<arg
line="${basedir}/dev-support/saveVersion.sh ${project.version} ${project.build.directory}/generated-sources/java"/>
</exec>
</target>
</configuration>
</execution>
<execution>
<id>generate-test-sources</id>
<phase>generate-test-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<mkdir dir="${project.build.directory}/generated-test-sources/java"/>
<taskdef name="recordcc" classname="org.apache.hadoop.record.compiler.ant.RccTask">
<classpath refid="maven.compile.classpath"/>
</taskdef>
<recordcc destdir="${project.build.directory}/generated-test-sources/java">
<fileset dir="${basedir}/src/test/ddl" includes="**/*.jr"/>
</recordcc>
</target>
</configuration>
</execution>
<execution>
<id>create-log-dir</id>
<phase>process-test-resources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<!--
TODO: there are tests (TestLocalFileSystem#testCopy) that fail if data
TODO: from a previous run is present
-->
<delete dir="${test.build.data}"/>
<mkdir dir="${test.build.data}"/>
<mkdir dir="${hadoop.log.dir}"/>
<copy toDir="${project.build.directory}/test-classes">
<fileset dir="${basedir}/src/main/conf"/>
</copy>
</target>
</configuration>
</execution>
<execution>
<phase>pre-site</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<tasks>
<copy file="src/main/resources/core-default.xml" todir="src/site/resources"/>
<copy file="src/main/xsl/configuration.xsl" todir="src/site/resources"/>
</tasks>
</configuration>
</execution>
</executions>
</plugin>
上面是在Pom.xml文件中使用Maven下使用Ant插件，里面有一行：
<echo file="target/compile-proto.sh">
看着让人不解，联系到HowToContribueToHadoop的文章http://wiki.apache.org/hadoop/HowToContribute可以推知，有可能由于本地没有安装ProtocolBuffers引起的，因为文章内部特别说明了：
引用
Hadoop 0.23+ must have Google's ProtocolBuffers for compilation to work.
接下来打算在本地重新安装ProtocolBuffers后再编译部署。
如预期，在本地安装好了Protoc Buffer后，后面两条指令顺利执行完整，剩下的就依据官网把目录下的工程导入Eclipse后，就可以在Eclipse下学习调试源码。
分享到：
评论
2 楼
jnh
2014-03-27
写的不错啊。
标题多了一个“的”。
官方文档：http://wiki.apache.org/hadoop/EclipseEnvironment
标题多了一个“的”。
官方文档：http://wiki.apache.org/hadoop/EclipseEnvironment
1 楼
camlelxy
2013-12-17
按照lz的配置没有成功 按照文档
Setup
-----
Install protobuf 2.5.0 (Download from http://code.google.com/p/protobuf/downloads/list)
- install the protoc executable (configure, make, make install)
- install the maven artifact (cd java; mvn install)
然后
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib加上这个以后才编译成功
我在ubuntu下编译的开2个窗口就会有问题。安装proto以后到hadoop-common目录下编译必须在同一个session中
Setup
-----
Install protobuf 2.5.0 (Download from http://code.google.com/p/protobuf/downloads/list)
- install the protoc executable (configure, make, make install)
- install the maven artifact (cd java; mvn install)
然后
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib加上这个以后才编译成功
我在ubuntu下编译的开2个窗口就会有问题。安装proto以后到hadoop-common目录下编译必须在同一个session中
发表评论
-
（转）Hadoop即将过时了吗？
2012-08-03 15:55 1440原文地址：http://www.kuqin.com/datab ... -
UNIX高手的10个习惯
2012-01-30 05:19 991直接阅读原连接： http://www.ibm.com/dev ... -
Hadoop的Eclipse插件报错
2012-01-23 21:26 964相关错误已在hadoop的bug库中，见连接： https:/ ... -
MacBook Air如何散热？
2012-01-21 21:00 2505使用MacBook Air已经有十几天了，一个感觉就是太爽了， ... -
Unable to load realm info from SCDynamicStore
2012-01-21 20:19 7865操作环境： Mac OS X 10.7.2 hadoop-0. ...
相关推荐
eclipse git插件下载。 用于eclipse集成git。多款插件。
org.eclipse.egit-updatesite-1.3.0.201202151440-r-site eclipse git插件
git代码管理工具 eclipse离线安装包，无需在线安装，直接复制至eclipse安装目录即可
eclipse GIT_4.6.1离线插件包 下载后解压到eclipse安装目录，重启Eclipse即可。
资源包括git简介，码云的注册使用，eclipse中的git使用，以及整个eclipse加码云的整个git协作开发环境的搭建过程说明，需要的请下载使用
eclipse配置git详细说明
eclipse中使用git，实现项目组代码的实时更新，便于开发，加快项目的开发速度
Eclipse的git插件
git与Eclipse的整合git与Eclipse的整合git与Eclipse的整合git与Eclipse的整合git与Eclipse的整合git与Eclipse的整合git与Eclipse的整合git与Eclipse的整合
一个很全的Git使用包括冲突的解决，命令使用Git。IDEA中使用Git。在Eclipse中使用Git。
使用Eclipse加载git启动项，使用Eclipse加载git启动项介绍说明
在eclipse或myeclipse中配置和使用git,以及git的常用页面截图，常用操作方式等
eclipse本地开发代码通过git库管理；导入java项目如何修改为web项目
eclipse中git插件eclipse中git插件eclipse中git插件eclipse中git插件eclipse中git插件
对于如何创建一个Laravel项目，相信对新接触Laravel的朋友并不存在太多的问题，但是今天我们要来看一下如何将已有的Laravel项目迁移（复制）到新的开发环境。 我们需要用到的工具是git，如果你不知道git是什么，这里...
2 下载远程项目的GIT库到本地[code]$ git clone git://远程Git库地址 filename[code] filename 是你本地的文件夹名字将远程库克隆到这个文件夹，此文件是自己建立的 3 常用命令 （1）git branch 查看本地分支
git版本管理已经很流行了，相比较与svn有很多优点，因此，大家有必要了解一下在eclipse中如何使用git管理版本
win7+ GitBash安装、GitLab使用、Eclipse配置，及3者配合进行版本控制的管理，小白级攻略哦
Eclipse git插件包下载；eclipse git插件使用后能够帮助用户们在编程过程中获得更加便捷的编程操作体验，拥有十分不错的操作效果，让你轻松的完成各项所需的编程工作，实现编程任务的完成！
eclipse git;Myeclipse10 git;eclipse git;Myeclipse10 egit;eclipse git;安装包文件