Selenium Python (3). How to Bypass Selenium Detection Using Python: User Agents and Background Execution

Web scraping and browser automation have become integral for many businesses and developers. However, many websites now detect and block automated browsing. This article will explore how to bypass Selenium detection using Python by manipulating user agents and running Selenium in the background. We’ll dive into detailed steps, tools, and best practices to ensure successful web scraping.

Understanding Selenium Detection

Before we bypass detection, let’s understand how it works. Websites can detect Selenium by checking for the presence of certain web driver flags and properties. When a site identifies these flags, it can block access or present misleading data. For example, when you open a site using a standard Chrome browser, it responds as expected. However, when you open the same site using Selenium, the website can detect the automation and block it. This detection happens because Selenium sets specific flags that websites can look for.

Changing WebDriver Flags

To bypass Selenium detection, one effective method is to modify the WebDriver flags.

1. Firefox Configuration: Open the Firefox configuration page by typing about:config in the address bar. Locate the flag related to WebDriver and set it to false.
2. Code Implementation:

from selenium import webdriver

# Set Firefox preferences
options = webdriver.FirefoxOptions()
options.set_preference("dom.webdriver.enabled", False)
options.set_preference('useAutomationExtension', False)

driver = webdriver.Firefox(options=options)

This script disables the WebDriver detection flag, making the browser appear as a regular user-driven instance.

User Agents

A user agent is a string that a browser sends to a web server to identify itself. Changing the user agent string can make Selenium requests indistinguishable from regular browser requests.

Steps to Change User Agent:

1. Identify a common user agent string: Example: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
2. Implement the Change in Selenium:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")

driver = webdriver.Chrome(options=options)

By setting a custom user agent, we can bypass many basic detections.

Running Selenium in the Background

Running the browser in the background is another crucial aspect of evading detection. This can be achieved by running the browser in headless mode.

Implementation:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument("--headless")

driver = webdriver.Chrome(options=options)

Running in headless mode means no graphical interface is displayed, which is essential for running automated tasks on servers.

Disabling Browser Notifications and Sounds

Automated browsing often involves handling unexpected pop-ups and notifications. Disabling these can streamline the process.

Code Example:

from selenium import webdriver

options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications": 2}
options.add_experimental_option("prefs", prefs)
options.add_argument("--mute-audio")

driver = webdriver.Chrome(options=options)

This script disables notifications and mutes audio, ensuring uninterrupted automation.

Parsing Data Example

Let’s consider a practical example of parsing nicknames from a site that generates random usernames.

Steps:

1. Load the site and interact with elements:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")

driver = webdriver.Chrome(options=options)
driver.get("https://example.com")

# Locate the username field and extract nicknames
usernames = []
for _ in range(10):
    nickname = driver.find_element(By.ID, "nickname").text
    usernames.append(nickname)
    driver.find_element(By.ID, "generate").click()
print(usernames)

Conclusion

By manipulating WebDriver flags, changing user agents, running Selenium in the background, and disabling browser notifications, you can effectively bypass Selenium detection. These techniques are essential for seamless and undetected web scraping and automation. Implementing these methods ensures that your automated tasks remain uninterrupted and efficient. Remember to always use web scraping and automation ethically, respecting website terms of service and data privacy laws. For more advanced techniques and regular updates, stay tuned to our blog on FineProxy.org. Feel free to share your ideas and feedback in the comments below. If you liked this article, don’t forget to subscribe to our channel and leave a like. Happy scraping!

By implementing these steps and adjusting settings as necessary, you can ensure your automation projects run smoothly and undetected.