Pytest --parallel multiprocessing error

[emirinanc]

Hi Kevlened,

I've been trying to run parallel selenium tests on Browserstack using pytest parallel but I've been failing to do so due to an dependency error because of the multiprocessing package. This issue is likely due to a misunderstanding on my part, but I am unable find the solution for it on Google or Stack Overflow. I would dearly appreciate your assistance.

I will briefly describe what I am attempting to do and share the error log with you.

I was inspired by this video to run 3 tests in parallel on Browserstack and I wrote a simple login test for three different platforms (namely Windows, iOS and Anroid) in PyCharm to run with pytest.

My project structure is as follows:

The test_loginTest.py file looks like this and I am using an interpreter with version Python 3.7.3 through Anaconda:

from ....Parallel import login_test_android,login_test_ios,login_test_windows

def test_login_test_windows():
    login_test_windows()
    assert True

def test_login_test_ios():
    login_test_ios()
    assert True

def test_login_test_android():
    login_test_android()
    assert True

I use the following command on the Command Line tool to successfully run my tests:

python -m pytest test_loginTest.py
(note: somehow running the command pytest test_loginTest.py fails to run as it fails to import the selenium package)

I then later use the following command on the Command Line tool to run tests in parallel on Browserstack:
python -m pytest --workers 3 test_loginTest.py

This command causes the following errors:

===================================================================================test session starts ===================================================================================
platform win32 -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\User\PycharmProjects\testin_PoC
plugins: Faker-4.1.2, arraydiff-0.3, doctestplus-0.3.0, openfiles-0.3.2, parallel-0.1.0, remotedata-0.3.1
collected 3 items                                                                                                                                                                                                   
pytest-parallel: 1 worker (process), 1 test per worker (thread)
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\site-packages\_pytest\main.py", line 213, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\site-packages\_pytest\main.py", line 257, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\site-packages\pluggy\hooks.py", line 289, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\site-packages\pluggy\manager.py", line 87, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\site-packages\pluggy\manager.py", line 81, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\site-packages\pluggy\callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\site-packages\pluggy\callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\site-packages\pytest_parallel\__init__.py", line 302, in pytest_runtestloop
INTERNALERROR>     process.start()
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\multiprocessing\process.py", line 112, in start
INTERNALERROR>     self._popen = self._Popen(self)
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
INTERNALERROR>     return _default_context.get_context().Process._Popen(process_obj)
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
INTERNALERROR>     return Popen(process_obj)
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
INTERNALERROR>     reduction.dump(process_obj, to_child)
INTERNALERROR>   File "C:\Users\User\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
INTERNALERROR>     ForkingPickler(file, protocol).dump(obj)
INTERNALERROR> AttributeError: Can't pickle local object 'ArgumentParser.__init__.<locals>.identity'

===================================================================================no tests ran in 0.18 seconds ===================================================================================
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\User\Anaconda3\lib\multiprocessing\connection.py", line 312, in _recv_bytes
    nread, err = ov.GetOverlappedResult(True)
BrokenPipeError: [WinError 109] The pipe has been ended

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\User\Anaconda3\lib\threading.py", line 917, in _bootstrap_inner
    self.run()
  File "C:\Users\User\Anaconda3\lib\threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\User\Anaconda3\lib\site-packages\pytest_parallel\__init__.py", line 348, in process_responses
    event_name, kwargs = queue.get()
  File "<string>", line 2, in get
  File "C:\Users\User\Anaconda3\lib\multiprocessing\managers.py", line 796, in _callmethod
    kind, result = conn.recv()
  File "C:\Users\User\Anaconda3\lib\multiprocessing\connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "C:\Users\User\Anaconda3\lib\multiprocessing\connection.py", line 321, in _recv_bytes
    raise EOFError
EOFError

[ravimamillapalli]

Yes; this is still error as of 4th Jan 2021;
It appears the same was fixed previously : #9
Python version :Python 3.8.7rc1
Pytest--Parallel 0.1.0
On windows OS
Pytest--Parallel is not usable if this is not fixed;