Monday, May 22, 2017

What is the proper value of HADOOP_HOME and PIG_CLASSPATH for APACHE HADOOP version 2.8.0?

By Hường Hana 4:00 PM , , Leave a Comment

I have the problem executing the Hadoop command from the PIG command line. The command and the error stack is below

My instructor suspects that it is because HADDOP_HOME and PIG_CLASSPATH are incorrect. I am on the HADOOP version 2.8.0.

So, originally I had HADOOP_HOME as 

HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/

Then I switched the following setup:

HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/libexec/etc/hadoop

PIG_CLASSPATH is defined as $HADOOP_HOME

Commands I used in pig:

A = LOAD '/Users/anarinsky/Downloads/loaddata1.txt';  B = MAPREDUCE '/Users/anarinsky/workspace/wordcount/target/wordcount-1.jar' STORE A INTO '/Users/anarinsky/Downloads/tempwrite2' LOAD  '/Users/anarinsky/Downloads/tempwrite2' AS (word:chararray, count:int) `com.systemskills.hadoop.wordcount.WordCountDriver /wordcountdata /Users/anarinsky/Downloads/pigoptdir`;

Pig Stack Trace

ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative  org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias B     at org.apache.pig.PigServer.openIterator(PigServer.java:1019)     at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:747)     at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376)     at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231)     at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206)     at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)     at org.apache.pig.Main.run(Main.java:564)     at org.apache.pig.Main.main(Main.java:176)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:498)     at org.apache.hadoop.util.RunJar.run(RunJar.java:234)     at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias B     at org.apache.pig.PigServer.storeEx(PigServer.java:1122)     at org.apache.pig.PigServer.store(PigServer.java:1081)     at org.apache.pig.PigServer.openIterator(PigServer.java:994)     ... 13 more Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:321)     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:629)     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)     at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:308)     at org.apache.pig.PigServer.launchPlan(PigServer.java:1474)     at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1459)     at org.apache.pig.PigServer.storeEx(PigServer.java:1118)     ... 15 more

1 Answers

Answers 1

Alex! Unfortunately, it's not related to Pig paths (tried it on my configured hadoop cluster) with same result. The error you get refers to the fact that Physical plan compiler has a bug in compile method. So in order to make your attempt work you have two possibilities

  1. Run native MR job using hadoop and after it finishes process it's results in pig

  2. Edit pig source code and compile your own version. You'll need to edit org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler#compile method and replace

    for (PhysicalOperator op : leaves) {     if (!(op instanceof POStore)) {         int errCode = 2025;         String msg = "Expected leaf of reduce plan to " +             "always be POStore. Found " + op.getClass().getSimpleName();         throw new MRCompilerException(msg, errCode, PigException.BUG);     } }

with 

    for (PhysicalOperator op : leaves) {         if (!(op instanceof POStore) && !(op instanceof PONative)) {             int errCode = 2025;             String msg = "Expected leaf of reduce plan to " +                 "always be POStore. Found " + op.getClass().getSimpleName();             throw new MRCompilerException(msg, errCode, PigException.BUG);         }     }
If You Enjoyed This, Take 5 Seconds To Share It

0 comments:

Post a Comment