The implementation of web scraping in shopping management application

In general, Indonesian consumers shopping is not based on need instead of desire. This caused a new habit to shopped spontaneously without planned ( impulse buying ) that made consumers are consumptive and extravagant. At this moment, the supermarket always compete in providing prices on any of its products to make the consumer is difficult to determine the supermarket which gives the cheapest price. Shopping management application that implements a web scraping method will change the mindset of consumers to shop for the needs of his household become frugal, clever, and wise. Implementation of web scraping aims to provide relevant information from the supermarket price comparison, price recommendations to consumers, as well as changes in prices of products sold by both supermarkets are based on an analysis of data that have been collected through web scraping. In this study the authors apply some methods i.e. web crawling, web scraping, and data cleansing to build the withdrawal system and designing a data supermarket shopping management application. These methods are processes that will be applied to reached the purpose of research that has been set. In the process of research the author using Pentaho Data Integration application that provides solutions to problems in the field of ETL (Extract, Transform, and Load) to obtain data supermarkets like product category, product name, normal price, promotions price, URL category, and the URL of the product. The data that has been obtained will be cleaned by applying methods of data cleansing and stored into the database with MySQL. The user interface of shopping management application will be designed using the PHP programming language to display information for the user. The results of this research is an shopping management application that displays information related to products , price comparisons are issued by both supermarkets, price recommendations set by the automation system as well as the price changes of products in each supermarket