From Python Basics to Data-Scraping Success: Michael’s Journey to Independence

Hi everyone. My name is Michael, as banal as it may sound. I’m a 30-year-old freelancer from Illinois, USA.

I first heard about data parsing at Illinois State University in 2012-2013, when I was studying to become a programmer. It seemed interesting and fun, but I had no idea how much it would change my life in the future.

Everything began with a small project during my internship at an IT company. I was tasked to gather and analyze data for our product. Most of the data was scattered across various websites, and that’s when I recalled parsing. I learned Python and web scraping libraries like BeautifulSoup and Scrapy. The project was a success, I received a bonus (and spent it 🙂), and I realized that I enjoyed the process.

A couple of years after graduating, I was working as a programmer, but I was often thinking about starting my own business. That’s when the idea to use web scraping for making money hit me. I started to look for clients who needed structured data. Surprisingly, there were many of them.

In my work, I use several tools and programs:

1. Python: The main programming language I use for writing web scraping scripts. Python has powerful libraries for web scraping such as BeautifulSoup, Scrapy, and Selenium.

2. BeautifulSoup: A Python library used for parsing HTML and XML documents. It’s perfect for extracting data from web pages.

3. Scrapy: Another powerful Python library for web scraping. Scrapy has extensive functionality and is designed for large-scale scraping.

4. Selenium: Selenium is typically used for automated web application testing, but it can also be used for web scraping, especially in cases when data is dynamically loaded using JavaScript.

5. Jupyter Notebook: An interactive environment for writing and testing Python code. It’s great for exploratory data analysis and for prototyping web scraping scripts.

6. SQL/NoSQL databases: I use SQL and NoSQL databases for storing and processing large volumes of collected data. PostgreSQL, MongoDB, and MySQL are some of my preferred databases.

7. Proxy: To circumvent IP restrictions and increase scraping speed, I use paid proxy services.

8. Cron or other task schedulers: I use them for automatically running my web scraping scripts at a specific time.

Now that I have a set of tools, and I know when and how to use them properly, my work takes very little time. If before I could sit on a project for several days, now setup takes from 1 to 4 hours, then everything works automatically.

I have several channels to find clients:

1. Freelance platforms: Websites like Upwork, Freelancer, and Fiverr provide plenty of opportunities to find clients in need of web scraping services. I actively use these platforms to find projects that match my skills.

2. Social networks: LinkedIn has become one of the best platforms for finding B2B clients. I’m active on LinkedIn, posting articles about web scraping and reaching out to companies that I think might be interested in my services.

3. Forums and communities: I’m also active on programming and web scraping forums and communities like StackOverflow and Reddit. This not only helps me stay updated with the latest trends in web scraping but also helps me find clients.

4. Networking events and conferences: I try to attend data and IT-related events and conferences as they provide an excellent opportunity to meet potential clients and partners.

5. I tried running a blog, and clients did come from there, but it takes a lot of time, and I had to close it.

Why am I writing all of this? Because many people, especially young ones, don’t know what to do and where to make money for a living.

With my example, I want to show that a little knowledge (the basics of Python can be mastered in a few weeks), desire, and hard work can help you achieve goals and become independent in life.