Scalability Versus Accuracy Trade-offs in Distributed Big Data Processing Frameworks: A Comparative Evaluation of Apache Spark, Flink, and Dask Using Benchmark Datasets

• Isiaq O. ALABI

  Author

• Hassan T. ABDULAZEEZ

  Author

• Sulaiman AHMAD

  Author

• Yahaya M. SANI

  Author

The exponential growth in data volume, velocity, and variety has intensified demand for distributed processing frameworks that balance computational scalability with analytical accuracy. Apache Spark, Apache Flink, and Dask represent three dominant open-source ecosystems, yet selecting an appropriate framework requires nuanced understanding of their performance characteristics under diverse workloads. This study presents a systematic comparative evaluation of these frameworks using standardized benchmark datasets (Transactions Processing Performance Council-Decision Support (TPC-DS) at 100 GB scale factor and HiBench version 7.1) across four dimensions: execution time, memory consumption, fault tolerance, and result consistency. Experiments were conducted on Amazon Web Services EC2 infrastructure using identical c5.4xlarge instances (16 vCPUs, 32 GB RAM) configured in standalone cluster mode. Results demonstrate that Spark achieved optimal performance for batch-oriented SQL workloads, completing 92 of 99 TPC-DS queries with the lowest average runtime (18% faster than Flink, 32% faster than Dask). Flink exhibited superior latency characteristics and exactly-once processing semantics, recovering from simulated node failures within 12 seconds compared to Spark's 45 seconds. Dask demonstrated competitive performance for iterative machine learning tasks but exhibited higher memory volatility and occasional floating-point inconsistencies during fault recovery. These findings provide empirical guidance for practitioners designing analytics pipelines in domains requiring both timeliness and computational precision, including cybersecurity threat detection and financial analytics.

• FJET_21_35_35

Copyright (c) 2026 FUDMA Journal of Engineering and Technology

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.