Saturday, 24 August 2013

hadoop partitioner getting incorrect reduce count

hadoop partitioner getting incorrect reduce count

I'm working on partitioner today. Its the basic program in hadoop custom
partitioners. Below is my partitioner code snippet.
public class VowelConsPartitioner extends Partitioner {
@Override
public int getPartition(Text letterType, IntWritable count, int redCnt) {
// TODO Auto-generated method stub
//System.out.println("reduce cnt in partitioner: "+redCnt);
if(letterType.toString().equalsIgnoreCase("vowel")){
//System.out.println("vowel sound: "+1%redCnt);
return letterType.toString().hasCode()%redCnt;
}
if(letterType.toString().equalsIgnoreCase("consonent")){
//System.out.println("Cons sound: "+2%redCnt);
return letterType.toString().hasCode()%redCnt;
}
else
return 0;
}
}
And I set my reducers in my driver class like this....
job.setNumReduceTasks(3);
job.setPartitionerClass(VowelConsPartitioner.class);
I want to keep more than 1 reducer. But I'm getting the o/p in one reducer
only. Moreover if you see the partitioner code, the first sysout(I've
commented) was giving me redCnt as 1. I'm not sure how thats happening
when I set its count to 3 from my driver class. Could someone help me out
on this?
FYI... I'm making jar & running this on HDFS.

No comments:

Post a Comment