Automation is my hobby. Admittedly I never thought that I would be interested in it when I studied computer science and engineering back in my university days (my father said my personality befits a software engineer, and I blindly followed his advice). However, during my journey to become a competent software developer I invested much time in simple scripts to do repetitive work for me. While most of the above examples involve automating games (via something like Sikuli which can control your screen), lately I have been turning my attention to scraping websites and integrating APIs for data aggregation. This post contains some of the tips and tricks I have used in order to make your web crawler quicker and more robust.