HFSP: Bringing Size-Based Scheduling To Hadoop

Abstract

Size-based scheduling is becoming older day by day, it has been recognized as an powerfull approach to
assure fairness and near optimal system response time.We introduce HFSP, a scheduler acquainting this technique to
Hadoop which real, multi-server, complex and widely used system.
Initial job information is needed in sized based scheduling, which is not available in hadoop. HFSP develops
such information by evaluating it on-line during job execution.
Our experiments, which are based on realistic workloads generated via standard benchmarking suite,
recognizes a significant decrease in a system response time by using Hadoop Fair Scheduler, and reflects that HFSP is
largely tolerant to job size estimation.