Wednesday, March 29, 2017

Yarn slave nodes are not communicating with master node?

By Hường Hana 2:30 PM , , Leave a Comment

I am not able to see my nodes when I do yarn node -list, even though I have configured /etc/hadoop/conf/yarn-site.xml with the correct properties (it seems to me, at least according to this question Slave nodes not in Yarn ResourceManager).

Here's what I've done so far:

  • installed resourcemanager on the master
  • installed nodemanager on the slaves

  • checked yarn-site.xml for this on ALL the nodes:

    <property> <name>yarn.resourcemanager.hostname</name> <value>master-node</value> </property>

  • after modifying the config file, restarted resourcemanager and nodemanager on the master and slaves, respectively.

But yet when I do yarn node -list I only see 

Total Nodes: 0  Node-Id       Node-state    Node-Http-Address      Number-of-Running-Containers

At my nodes, I looked at the .out files in /var/log/hadoop-yarn/ and I see this in them: 

ulimit -a core file size          (blocks, -c) 0 data seg size           (kbytes, -d) unlimited scheduling priority             (-e) 0 file size               (blocks, -f) unlimited pending signals                 (-i) 244592 max locked memory       (kbytes, -l) 64 max memory size         (kbytes, -m) unlimited open files                      (-n) 32768 pipe size            (512 bytes, -p) 8 POSIX message queues     (bytes, -q) 819200 real-time priority              (-r) 0 stack size              (kbytes, -s) 10240 cpu time               (seconds, -t) unlimited max user processes              (-u) 65536 virtual memory          (kbytes, -v) unlimited file locks                      (-x) unlimited

EDIT: when I look at the .log files I see the following, but I'm not sure how to fix it:

    INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state STARTED; cause:  org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: <master node ip>:8020:8031 (configuration property 'yarn.resourcemanager.resource-tracker.address')  Caused by: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: <master node ip>:8020:8031 (configuration property 'yarn.resourcemanager.resource-tracker.address')

How do I connect my slave nodes to my master node?

3 Answers

Answers 1

The value set for yarn.resourcemanager.hostname acts as the base value for all the ResourceManager properties. The property yarn.resourcemanager.resource-tracker.address defaults to the value of ${yarn.resourcemanager.hostname}:8031. Refer yarn-default.xml for the complete list of default YARN configurations.

And from the nodemanager ERROR log,

Caused by: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: <master node ip>:8020:8031 (configuration property 'yarn.resourcemanager.resource-tracker.address')

It looks the yarn.resourcemanager.hostname property is configured incorrectly as <master node ip>:8020 instead of <master node ip> on the slave nodes.

Edit the yarn-site.xml on all the nodes to have

<property>    <name>yarn.resourcemanager.hostname</name>    <value>master_node</value> <!-- IP address or Hostname of the node where Resource Manager is started, Omit the port number --> </property>

Finally, restart the YARN services.

Answers 2

please set all this properties and try     <property>         <name>yarn.resourcemanager.address</name>         <value>master_node:8032</value>       </property>       <property>         <name>yarn.resourcemanager.admin.address</name>         <value>master_node:8033</value>       </property>       <property>         <name>yarn.resourcemanager.scheduler.address</name>         <value>master_node:8030</value>       </property>       <property>         <name>yarn.resourcemanager.resource-tracker.address</name>         <value>master_node:8031</value>       </property>       <property>         <name>yarn.resourcemanager.webapp.address</name>         <value>master_node:8088</value>       </property>       <property>         <name>yarn.resourcemanager.webapp.https.address</name>         <value>master_node:8090</value>       </property>

Answers 3

You need to set ip for yarn.resourcemanager.hostname property. if you want to use the hostname, your machine needs to know to which ip that hostname is pointing to. So you need to add host entry in /etc/hosts file.

To do that,

  1. Open terminal

  2. Type vim /etc/hosts and hit enter

  3. Add this line at end of the file (use key i to enable insertion)

    <your resourcemanager ip><space><your hostname>

    example: `192.168.1.23 master-node`

  4. Save the file by typing <Esc> + :wq

  5. Restart the nodemanager

I recommend using ambari kind of managing tool to do these kind of stuffs. This allows easy modification of configuration at any time for hadoop environment. Because manual work always have more chance for error.

If You Enjoyed This, Take 5 Seconds To Share It

0 comments:

Post a Comment