python - Playwright commands are randomly not working leading to timeout errors - Stack Overflow

I'm making a bot using Python playwright which works but at random times random commands will just be skipped or not work leading to a timeout on the next command.

from multiprocessing.connection import Client
from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup
from pymongo.mongo_client import MongoClient
from pymongo.server_api import ServerApi
from bson import ObjectId
Import Time
price1 = 0
price12 = 0
price_state1 = 0
price_state2 = 0
num = 0
While True:
    def run_playwright_test():
        with sync_playwright() as p:
# Launch the browser (headless=False to see the browser)
            browser = p.firefox.launch(headless = False)
            Context = browser.new_context()
           Context.set_default_timeout (0)
            Page = Context.new_page()
            Global price1
            Global price2
            Global price_state1
            Global price_state2
           page.goto('')
            page.fill('input[name="mail"]', 'password)
            page.fill('input[name="p"]', 'password)
            page.click('input[name="s"]')
            # oil market
           page.locator('div[action="storage"].item_menu.storage_menu.ajax_action.header_menu_item.tc').click();
            page.click('div.tip.float_left.white.imp.storage_item.pointer.hov.ib_border');
            # oil
            page.wait_for_selector('div.storage_price')
            html_content = Page.Content()
            soup = BeautifulSoup(html_content, 'html.parser')
            price_span1 = soup.find('div', class_='storage_price').find('span', class_='dot')
            if price_span1:
                price_1 = price_span1.get_text(strip = True)
                price_1 = price_1.replace('$', '').replace(' ', '')  # Remove '$' and spaces
                price_1 = price_1.replace('.', '')  # Remove '$' and spaces
                price_1 = float(price_1)  # Convert to a numeric value (float)
                print(f"Extracted price: {price_1}")
            Else:
                Print ("Price not found")
            # ore market
           page.locator('div[action="storage"].item_menu.storage_menu.ajax_action.header_menu_item.tc').click();
            page.click('div.storage_number span[url="4"]')
            # ore
           page.wait_for_selector('div.storage_price')
            Time.sleep (10)
            html_content = Page.Content()
            soup = BeautifulSoup(html_content, 'html.parser')
            price_span2 = soup.find('div', class_='storage_price').find('span', class_='dot')
            if price_span2:
                price_2 = price_span2.get_text(strip = True)
                price_2 = price_2.replace('$', '').replace(' ', '')  # Remove '$' and spaces
                price_2 = price_2.replace('.', '')  # Remove '$' and spaces
                price_2 = float(price_2)
                print(f"Extracted price: {price_2}")
            Else:
                Print ("Price not found")

I'm using the raspberry pi5 and the Python idle in a venv

The code above is just a small part of it runs a lot more the one that seems to fail or skip the most is the line before the:

page.wait_for_selector('div.storage_price')

Note that I changed the timeout to 0 which stops the error but stills pauses and does not move on.

If any one can tell me why this is happening and how to fix it that will be great.

I'm making a bot using Python playwright which works but at random times random commands will just be skipped or not work leading to a timeout on the next command.

from multiprocessing.connection import Client
from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup
from pymongo.mongo_client import MongoClient
from pymongo.server_api import ServerApi
from bson import ObjectId
Import Time
price1 = 0
price12 = 0
price_state1 = 0
price_state2 = 0
num = 0
While True:
    def run_playwright_test():
        with sync_playwright() as p:
# Launch the browser (headless=False to see the browser)
            browser = p.firefox.launch(headless = False)
            Context = browser.new_context()
           Context.set_default_timeout (0)
            Page = Context.new_page()
            Global price1
            Global price2
            Global price_state1
            Global price_state2
           page.goto('https://rivalregions.com/#overview')
            page.fill('input[name="mail"]', 'password)
            page.fill('input[name="p"]', 'password)
            page.click('input[name="s"]')
            # oil market
           page.locator('div[action="storage"].item_menu.storage_menu.ajax_action.header_menu_item.tc').click();
            page.click('div.tip.float_left.white.imp.storage_item.pointer.hov.ib_border');
            # oil
            page.wait_for_selector('div.storage_price')
            html_content = Page.Content()
            soup = BeautifulSoup(html_content, 'html.parser')
            price_span1 = soup.find('div', class_='storage_price').find('span', class_='dot')
            if price_span1:
                price_1 = price_span1.get_text(strip = True)
                price_1 = price_1.replace('$', '').replace(' ', '')  # Remove '$' and spaces
                price_1 = price_1.replace('.', '')  # Remove '$' and spaces
                price_1 = float(price_1)  # Convert to a numeric value (float)
                print(f"Extracted price: {price_1}")
            Else:
                Print ("Price not found")
            # ore market
           page.locator('div[action="storage"].item_menu.storage_menu.ajax_action.header_menu_item.tc').click();
            page.click('div.storage_number span[url="4"]')
            # ore
           page.wait_for_selector('div.storage_price')
            Time.sleep (10)
            html_content = Page.Content()
            soup = BeautifulSoup(html_content, 'html.parser')
            price_span2 = soup.find('div', class_='storage_price').find('span', class_='dot')
            if price_span2:
                price_2 = price_span2.get_text(strip = True)
                price_2 = price_2.replace('$', '').replace(' ', '')  # Remove '$' and spaces
                price_2 = price_2.replace('.', '')  # Remove '$' and spaces
                price_2 = float(price_2)
                print(f"Extracted price: {price_2}")
            Else:
                Print ("Price not found")

I'm using the raspberry pi5 and the Python idle in a venv

The code above is just a small part of it runs a lot more the one that seems to fail or skip the most is the line before the:

page.wait_for_selector('div.storage_price')

Note that I changed the timeout to 0 which stops the error but stills pauses and does not move on.

If any one can tell me why this is happening and how to fix it that will be great.

• python
• timeout
• playwright
• playwright-python

Share Improve this question Follow edited Jan 18 at 16:43 ggorlen 58.1k88 gold badges115115 silver badges157157 bronze badges asked Jan 15 at 7:24 Jackson TaylorJackson Taylor 3111 silver badge11 bronze badge 2

• Tag python-idle is about the IDE, not timeouts. – Terry Jan Reedy Commented Jan 15 at 22:19
• Could you share your expected output? Why do you want to randomly skip commands? Curious about your use case. Changing the timeout to 0 never helps fix problems, it only suppresses them. I also suggest avoiding BeautifulSoup--Playwright already lets you select on the live page, so you're adding a library and snapshotting potentially stale state, and generally adding complexity with BS. I also suggest fixing the syntax errors in this script so it's easy to run. Thanks. – ggorlen Commented Jan 18 at 16:42

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 2

time.sleep(x) leads to some issues
see: https://playwright.dev/python/docs/library#timesleep-leads-to-outdated-state
better use page.wait_for_timeout(x * 1000)