get unique columns

parilango · Post by **parilango** » Tue Oct 18, 2016 8:52 am

I have two files and I am getting few columns based on one common column.. but I want the unique records not based on the key column.

right now I am using sort stage and passing all the column name as key column in the hash partition. is this the right approach or do we have any better approach to get unique records from all the column

chulett · Post by **chulett** » Tue Oct 18, 2016 9:52 am

Is that approach working for you?

parilango · Post by **parilango** » Tue Oct 18, 2016 10:54 am

yeah it Is working. is there any other approach we can follow?

abc123 · Post by **abc123** » Fri Nov 11, 2016 10:10 am

You can also use Sort stage and then Remove Duplicate stage. Leave the partitioning as Auto.