Problem solution for Spark use case

1) SO I have 60 Millions usa and canada postals Created dataframe customer_df => good and bad => 60M 2) downloaded some Good postals from internet to filter the customer_df data Created one dataframe good_df => which is good postals => 1M 3) Perfomed Join between customer_df and good_df wiht zipcode to seperate the good values filter_df = good zip [login to view URL](cus_df,zipcode) 4) Then seperated bad data with the below logic bad_df = [login to view URL](filter_df) Now still we can filter bad_df with city names city_df = [login to view URL](bad_df,city) Then did unioin between both df's total_filter = [login to view URL](city_df) it taking 1.30 mints (used spark with 8 node cluster each node 32 gb => spark-submit driver memory -8g and num-executors - 8 and executor-memory- 8g) any other technology or any other tool to clean-up the data within 15 to 20 mints(again customer data is 60M