-
|
Hi I have 100 proxy servers (ipv6) at my disposal and for a long time I have been trying to write a program based on your class that does something like this. In a loop, it sets up a proxy server, connects to the selected page with the Cloudflate preloader (security check), when the page loads (there is immediately json on it), cookies are downloaded, saved in the database, the same is done for the next proxy, etc. target_url = 'https://www.xxxx.com/api/1.0.0/products/?file.json'
async with (webdriver.Chrome(options=options, max_ws_size=2** 60, debug=True) as driver):
driver.base_target.socket.on_closed.append(lambda code, reason: driver.quit())
user_agent = await driver.execute_async_script("return navigator.userAgent;")
await asyncio.sleep(2)
while True:
proxies = proxy_db.get_proxy_list_to_recive_cf_cookie()
# if there is no cookie to refresh, wait
if not proxies:
time.sleep(60 * 4)
continue
for proxy in proxies:
try:
ip = proxy["ip"]
port = proxy["port"]
step = step + 1
await driver.set_single_proxy(
'http://{username}:{password}@{ip}:{port}'.format(
ip=ip,
port=port,
username=proxy["username"],
password=proxy["password"].rstrip(),
)
)
await driver.get(target_url, wait_load=True, timeout=loading_timeout)
await asyncio.sleep(5)
wait_step = 1
found = False
while wait_step <= 5 and not found:
found = await driver.find_element(By.TAG_NAME, "pre")
if found:
cookies = await driver.get_cookies()
for cookies in cookies:
if cookie["name"] == "cf_clearance":
cookie_value = cookie["value"]
# save the downloaded copokie in the database
proxy_db.set_cookie_for_proxy(ip, port, cookie_value, user_agent)
break
await asyncio.sleep(0.5)
wait_step = wait_step + 1
await driver.delete_all_cookies()
await driver.clear_proxy()
except NoSuchElementException:
errors = errors + 1
continue
finally:
if timeouts_errors == 5:
driver.quit()
exit()One more question, in the case as described, where is the Cloudflate preloader with security check, what is the easiest way to wait for me to be redirected to the right page and load it? The only thing that worked for me was this: await driver.get(target_url, wait_load=True, timeout=loading_timeout)
await asyncio.sleep(5)
wait_step = 1
found = False
while wait_step <= 5 and not found:
found = await driver.find_element(By.TAG_NAME, "pre")
if found:
cookies = await driver.get_cookies()
for cookie in cookies:
if cookie["name"] == "cf_clearance":
cookie_value = cookie["value"]
# save cookiue to DB
proxy_db.set_cookie_for_proxy(ip, port, cookie_value, user_agent)
break
await asyncio.sleep(0.5)
wait_step = wait_step + 1In plain selenium I used: Regarding the errors I receive when the script is running, for example: Thank you, for your help. |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 13 replies
-
|
@juhacz Does found = await driver.find_element(By.TAG_NAME, "pre", timeout=5)work for you? |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately, just using doesn't do anything. After Cloudflare performs a security check, the Chrome window closes and the program (loop in the program) stops, no exception occurs, the program simply stops. I have to stop the program using ctrl+c or clicking Stop in PHPStorm When will I use: The loop will do 5-7 iterations and the same thing happens, the Chrome window closes, the loop stops, there is no exception. |
Beta Was this translation helpful? Give feedback.
-
|
@kaliiiiiiiiii In my opinion, the key is this exception:
Isn't it the case that my program tries to look for "pre" in the cloudflare security check page instead of the json page that is supposed to appear after the security check? When Cloudflare redirects me to the correct json page after checking security, I get an error. |
Beta Was this translation helpful? Give feedback.
-
Indeed. And |
Beta Was this translation helpful? Give feedback.
-
|
@kaliiiiiiiiii : Were you able to fix the problem? In commit b90265b I see the annotation fix memory leak & improve waiting for webelement, but after checking this version nothing has changed. |
Beta Was this translation helpful? Give feedback.
-
|
still have to do some testing: ) |
Beta Was this translation helpful? Give feedback.
-
@juhacz Are you sure the dev-version has been installed correctly with those changes? If yes, then I'm pretty sure it either works or there's a different error. |
Beta Was this translation helpful? Give feedback.
-
|
I tested the version bdf9a7f and it works as it should. Thank You |
Beta Was this translation helpful? Give feedback.
-
|
@kaliiiiiiiiii The problem only occurs when using a proxy, another program does not cause problems when iterating several thousand times. |
Beta Was this translation helpful? Give feedback.
-
|
huh that's bad - maybe a memory leak. or at least smth piling up. |
Beta Was this translation helpful? Give feedback.
-
|
@juhacz Can't reproduce that (don't have any proxies I can test this with anymore) |
Beta Was this translation helpful? Give feedback.
-
|
@kaliiiiiiiiii https://webshare.io provides free plan 10 proxies for every user after registration. If something goes wrong, let me know and I'll try to help with my proxy server. |
Beta Was this translation helpful? Give feedback.
-
|
It turned out that the problem was most likely a bad proxy configuration and not a class problem. |
Beta Was this translation helpful? Give feedback.
-
|
good to know! |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, but the problem is not with the proxy server. It's on the selenium-driverless side, I hope it can be fixed. |
Beta Was this translation helpful? Give feedback.
-
|
I checked version 1.9.1 and the problem does not occur there. I went back to using selenium-driverless. Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
huh I wonder why that would be the case - but that's good then I suppose haha |
Beta Was this translation helpful? Give feedback.
I checked version 1.9.1 and the problem does not occur there. I went back to using selenium-driverless. Thank you.