Distribuera mera - Spark och Hadoop utan Big Data

Distribution as a concept means that a task (for example, data storage or code execution) is parallelized on multiple computers. It goes hand in hand with the concept of big data – extreme amounts of data that can’t be processed by a single computer. Because of this, the most established tools for distributed parallelization is tools that are designed to handle big data. This thesis explores whether two such tools, Spark (distributed code execution) and Hadoop Distributed File System (distributed data storage), are also suited for handling smaller amounts of data. Distribution is a potentially cheap and scalable way of working even for small amounts of data. The primary method of the report is performance tests. As a side track, an abstraction layer that allows for code to be executed either distributed or locally is implemented by using Java streams as a local equivalent of Spark. With this abstraction layer small tasks that are only sometimes suited for distribution can choose the best alternative at run time. It is concluded that these tools can be useful even for small amounts of data, and even when the execution time for a non-distributed solution is very short (under a minute)