For now I have hadoop job which creates counters with pretty big name. For example, the following one: stats.counters.server-name.job.job-name.mapper.site.site-name.qualifier.qualifier-name.super-long-string-which-is-not-within-standard-limits. This counter is truncated on web interface and on getName() method call. I've found out that hadoop have limitations on the counter max name and this settings id mapreduce.job.counters.counter.name.max is for configuring this limit. So I incremented this to 500 and web interface now shows full counter name. But getName() of the counter still returns truncated name.
Cound somebody, please, explain this or point me on my mistakes? Thank you.
EDIT 1
My hadoop server configuration is consist of single server with hdfs, yarn and map-reduce itself on it. During map-reduce there are some counter increments and after job is completed, in ToolRunner I fetch counters with use of org.apache.hadoop.mapreduce.Job#getCounters.
EDIT 2
Hadoop version is the following:
Hadoop 2.6.0-cdh5.8.0 Subversion http://github.com/cloudera/hadoop -r 042da8b868a212c843bcbf3594519dd26e816e79 Compiled by jenkins on 2016-07-12T22:55Z Compiled with protoc 2.5.0 From source with checksum 2b6c319ecc19f118d6e1c823175717b5 This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.8.0.jar I made some additional investigation and it seems that this issue describes situation similar to mine. But it's pretty confusing cause I'm able to increase number of counters but not the length of counter's name...
EDIT 3
Today I spent pretty much time debugging internals of the hadoop. Some interesting stuff:
org.apache.hadoop.mapred.ClientServiceDelegate#getJobCountersmethod returns bunch of counters from yarn with TRUNCATED names and FULL display names.- Was unable to debug maps and reducers itself but with help of logging it seems that
org.apache.hadoop.mapreduce.Counter#getNamemethod works correctly during reducer execution.
0 comments:
Post a Comment