Selenium rework

[tomas-edwardsson]

Make each app within adagios run it's own browser session.
Initial work to allow for testing multiple browsers in the same run.
Remove class variables and other changes suggested in #441

[palli]

Please fix travis unittest before we can merge this pull request.

[palli]

Thanks. Looking cleaner now.

The unit tests are not running on my machine. They fail with the error below and after the tests run port 8000 is occupied by nagios3 for some odd reason.

Seems like somewhere instead of selenium starting django on port 8000, it starts a livestatus socket on that port.

The tests are failing because firefox tries to open port 8000 but the page is hanging.

$ python manage.py test
Creating test database for alias 'default'...
........./opt/pynag/pynag/Control/init.py:75: FutureWarning: nagios_init is deprecated, use service_name instead
FutureWarning)

............/usr/lib/python2.7/dist-packages/django/http/response.py:330: DeprecationWarning: Using mimetype keyword argument is deprecated, use content_type instead
super(HttpResponse, self).init(_args, *_kwargs)

...E......E................................

ERROR: setUpClass (adagios.status.tests.SeleniumStatusTestCase)

Traceback (most recent call last):
File "/home/palli/workspace/adagios/adagios/../adagios/seleniumtests.py", line 61, in setUpClass
firefox = webdriver.Firefox()
File "/usr/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 51, in init
self.binary, timeout),
File "/usr/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in init
self.binary.launch_browser(self.profile)
File "/usr/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 44, in launch_browser
self.wait_until_connectable()
File "/usr/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 86, in wait_until_connectable
self.profile.path, self.get_firefox_output()))
WebDriverException: Message: 'Can't load the profile. Profile Dir: /tmp/tmprH7OA8 Firefox output: \n(process:343): GLib-CRITICAL *: g_slice_set_config: assertion 'sys_page_size == 0' failed\n\n(firefox:343): GLib-GObject-WARNING *: Attempt to add property GnomeProgram::sm-connect after class was initialised\n\n(firefox:343): GLib-GObject-WARNING *: Attempt to add property GnomeProgram::show-crash-dialog after class was initialised\n\n(firefox:343): GLib-GObject-WARNING *_: Attempt to add property GnomeProgram::display after class was initialised\n\n(firefox:343): GLib-GObject-WARNING **: Attempt to add property GnomeProgram::default-icon after class was initialised\n1415543152077\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/addons/XPIProvider.jsm: ["XPIProvider"]\n1415543152078\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/LightweightThemeManager.jsm: ["LightweightThemeManager"]\n1415543152081\taddons.xpi\tDEBUG\tstartup\n1415543152082\taddons.xpi\tINFO\tMapping online-accounts@lists.launchpad.net to /usr/lib/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/online-accounts@lists.launchpad.net\n1415543152082\taddons.xpi\tINFO\tMapping ubufox@ubuntu.com to /usr/share/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/ubufox@ubuntu.com\n1415543152082\taddons.xpi\tINFO\tMapping webapps-team@lists.launchpad.net to /usr/share/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/webapps-team@lists.launchpad.net\n1415543152083\taddons.xpi\tINFO\tMapping {2e1445b0-2682-11e1-bfc2-0800200c9a66} to /usr/share/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/{2e1445b0-2682-11e1-bfc2-0800200c9a66}\n1415543152083\taddons.xpi\tINFO\tMapping langpack-en-ZA@firefox.mozilla.org to /usr/lib/firefox/browser/extensions/langpack-en-ZA@firefox.mozilla.org.xpi\n1415543152083\taddons.xpi\tINFO\tMapping langpack-en-GB@firefox.mozilla.org to /usr/lib/firefox/browser/extensions/langpack-en-GB@firefox.mozilla.org.xpi\n1415543152084\taddons.xpi\tINFO\tMapping {972ce4c6-7e08-4474-a285-3208198ce6fd} to /usr/lib/firefox/browser/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}\n1415543152084\taddons.xpi\tINFO\tMapping fxdriver@googlecode.com to /tmp/tmprH7OA8/extensions/fxdriver@googlecode.com\n1415543152085\taddons.xpi\tDEBUG\tcheckForChanges\n1415543152099\taddons.xpi\tDEBUG\tNo changes found\n1415543152102\taddons.xpi\tDEBUG\tRegistering manifest for /usr/lib/firefox/browser/extensions/langpack-en-ZA@firefox.mozilla.org.xpi\n1415543152103\taddons.xpi\tDEBUG\tRegistering manifest for /usr/lib/firefox/browser/extensions/langpack-en-GB@firefox.mozilla.org.xpi\n'

ERROR: setUpClass (adagios.objectbrowser.tests.SeleniumObjectBrowserTestCase)

Traceback (most recent call last):
File "/home/palli/workspace/adagios/adagios/../adagios/seleniumtests.py", line 61, in setUpClass
firefox = webdriver.Firefox()
File "/usr/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 51, in init
self.binary, timeout),
File "/usr/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in init
self.binary.launch_browser(self.profile)
File "/usr/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 44, in launch_browser
self.wait_until_connectable()
File "/usr/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 86, in wait_until_connectable
self.profile.path, self.get_firefox_output()))
WebDriverException: Message: 'Can't load the profile. Profile Dir: /tmp/tmpNn2tn3 Firefox output: \n(process:6875): GLib-CRITICAL *: g_slice_set_config: assertion 'sys_page_size == 0' failed\n\n(firefox:6875): GLib-GObject-WARNING *: Attempt to add property GnomeProgram::sm-connect after class was initialised\n\n(firefox:6875): GLib-GObject-WARNING *: Attempt to add property GnomeProgram::show-crash-dialog after class was initialised\n\n(firefox:6875): GLib-GObject-WARNING *_: Attempt to add property GnomeProgram::display after class was initialised\n\n(firefox:6875): GLib-GObject-WARNING **: Attempt to add property GnomeProgram::default-icon after class was initialised\n1415543185359\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/addons/XPIProvider.jsm: ["XPIProvider"]\n1415543185361\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/LightweightThemeManager.jsm: ["LightweightThemeManager"]\n1415543185363\taddons.xpi\tDEBUG\tstartup\n1415543185364\taddons.xpi\tINFO\tMapping online-accounts@lists.launchpad.net to /usr/lib/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/online-accounts@lists.launchpad.net\n1415543185365\taddons.xpi\tINFO\tMapping ubufox@ubuntu.com to /usr/share/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/ubufox@ubuntu.com\n1415543185365\taddons.xpi\tINFO\tMapping webapps-team@lists.launchpad.net to /usr/share/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/webapps-team@lists.launchpad.net\n1415543185365\taddons.xpi\tINFO\tMapping {2e1445b0-2682-11e1-bfc2-0800200c9a66} to /usr/share/mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/{2e1445b0-2682-11e1-bfc2-0800200c9a66}\n1415543185366\taddons.xpi\tINFO\tMapping langpack-en-ZA@firefox.mozilla.org to /usr/lib/firefox/browser/extensions/langpack-en-ZA@firefox.mozilla.org.xpi\n1415543185366\taddons.xpi\tINFO\tMapping langpack-en-GB@firefox.mozilla.org to /usr/lib/firefox/browser/extensions/langpack-en-GB@firefox.mozilla.org.xpi\n1415543185366\taddons.xpi\tINFO\tMapping {972ce4c6-7e08-4474-a285-3208198ce6fd} to /usr/lib/firefox/browser/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}\n1415543185367\taddons.xpi\tINFO\tMapping fxdriver@googlecode.com to /tmp/tmpNn2tn3/extensions/fxdriver@googlecode.com\n1415543185367\taddons.xpi\tDEBUG\tcheckForChanges\n1415543185381\taddons.xpi\tDEBUG\tNo changes found\n1415543185384\taddons.xpi\tDEBUG\tRegistering manifest for /usr/lib/firefox/browser/extensions/langpack-en-ZA@firefox.mozilla.org.xpi\n1415543185385\taddons.xpi\tDEBUG\tRegistering manifest for /usr/lib/firefox/browser/extensions/langpack-en-GB@firefox.mozilla.org.xpi\n'

---

Ran 62 tests in 107.764s

FAILED (errors=2)
Destroying test database for alias 'default'...

[palli]

Please also remove unneeded imports from this file.

[tomas-edwardsson]

I believe the issue you are having is a selenium/firefox communcations error, you can check http://stackoverflow.com/questions/6682009/selenium-firefoxprofile-exception-cant-load-the-profile for pointers.

Regarding the port 8000 issue, here is whats really happening:

• LiveServerTestCase starts adagios listening on port 8000
• FakeAdagiosEnvironment starts nagios -d in background
• Loading the selenium firefox webdriver fails in setUpClass which means that tearDownClass is never run, that is, cleanup and killing the nagios process never happens
• The python process which is running the tests exits
• The port is assigned to the orphaned child

I managed to prove this with the following code:

#!/usr/bin/python

import time
import socket
from subprocess import Popen, PIPE
s = socket.socket()
s.bind(("", 9999))
s.listen(1)

Popen(['nagios', '-d', '/tmp/tmp5DPVb0nagios-environment/nagios.cfg'],
      stdout=PIPE).communicate()[0]

time.sleep(10)

Running netstat before the python script exits

[tommi@fedora-dev ~]$ netstat -anp|grep 9999
tcp        0      0 0.0.0.0:9999            0.0.0.0:*               LISTEN      8331/python         

After the 10 second sleep is done, netstat looks like this

[tommi@fedora-dev ~]$ netstat -anp|grep 9999
tcp        0      0 0.0.0.0:9999            0.0.0.0:*               LISTEN      8333/nagios         

I'm pushing code where the FakeAdagiosEnvironment is not setup until the selenium webdriver has been initialized.

[tomas-edwardsson]

@palli and I've removed all unused imports from utils.py.

[palli]

If unhandled exception in setUpClass is causing tearDown never to be called, then i think it makes since to wrap try/except around webdriver calls with its own cleanup.

[tomas-edwardsson]

I don't think there is a need since I moved webdriver above the fake environment, therefore there is not nagios environment to tear down on selenium init failure.

[palli]

New light on this pull request, it seems like working versions of the firefox webdriver seems hard to come by without resulting to using pip, can you modify this pull request so that the tests are simply skipped if it fails to load the web driver ?

If need be, you can create seperate unit tests for loading of the web driver, if you don't want to hide breakage in the loading code.

[palli]

Looks like skipping on webdriver error is already there (but overly broad imho, i.e. no need to skip inside travis). Good enough as is. Thanks for fixing the build!