Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

3PAR gives different error on 3.2.2 and 3.3.1 when wsapi session exhausts #584

Open
sneharai4 opened this issue Apr 17, 2019 · 3 comments
Open

3PAR gives different error on 3.2.2 and 3.3.1 when wsapi session exhausts #584

sneharai4 opened this issue Apr 17, 2019 · 3 comments
Labels
bug low
Milestone
3.2

Comments

@sneharai4
Copy link
Collaborator

sneharai4 commented Apr 17, 2019 •
edited
Loading

wsapisession exhausts on 3PAR for 3.2.2 we get below error in plugin container log:

Note: Below error trace is not logged in /etc/hpedockerplugin/3pardcv.log which is incorrect.

2019-04-16 09:39:37.159 18 INFO hpedockerplugin.synchronization [-] Lock released: [caller=create_volume, lock-name=sneha_vol_1]�[00m
2019-04-16T09:39:37+0530 [twisted.web.server.Request#critical] 
	Traceback (most recent call last):
	  File "/usr/lib/python3.6/site-packages/Twisted-18.7.0rc1-py3.6-linux-x86_64.egg/twisted/web/server.py", line 258, in render
	    body = resrc.render(self)
	  File "/usr/lib/python3.6/site-packages/klein-17.10.0-py3.6.egg/klein/resource.py", line 210, in render
	    d = defer.maybeDeferred(_execute)
	  File "/usr/lib/python3.6/site-packages/Twisted-18.7.0rc1-py3.6-linux-x86_64.egg/twisted/internet/defer.py", line 151, in maybeDeferred
	    result = f(*args, **kw)
	  File "/usr/lib/python3.6/site-packages/klein-17.10.0-py3.6.egg/klein/resource.py", line 204, in _execute
	    **kwargs)
	--- <exception caught here> ---
	  File "/usr/lib/python3.6/site-packages/Twisted-18.7.0rc1-py3.6-linux-x86_64.egg/twisted/internet/defer.py", line 151, in maybeDeferred
	    result = f(*args, **kw)
	  File "/usr/lib/python3.6/site-packages/klein-17.10.0-py3.6.egg/klein/app.py", line 128, in execute_endpoint
	    return endpoint_f(self._instance, *args, **kwargs)
	  File "/usr/lib/python3.6/site-packages/klein-17.10.0-py3.6.egg/klein/app.py", line 227, in _f
	    return _call(instance, f, request, *a, **kw)
	  File "/usr/lib/python3.6/site-packages/klein-17.10.0-py3.6.egg/klein/app.py", line 50, in _call
	    result = f(*args, **kwargs)
	  File "/python-hpedockerplugin/hpedockerplugin/hpe_storage_api.py", line 393, in volumedriver_create
	    rcg_name)
	  File "/python-hpedockerplugin/hpedockerplugin/backend_orchestrator.py", line 203, in volumedriver_create
	    rcg_name)
	  File "/python-hpedockerplugin/hpedockerplugin/backend_orchestrator.py", line 154, in __execute_request
	    return getattr(volume_mgr, request)(volname, *args, **kwargs)
	  File "/python-hpedockerplugin/hpedockerplugin/synchronization.py", line 47, in _wrapped
	    return __synchronized('VOL', lock_name, f, *a, **k)
	  File "/python-hpedockerplugin/hpedockerplugin/synchronization.py", line 23, in __synchronized
	    return f(*a, **k)
	  File "/python-hpedockerplugin/hpedockerplugin/volume_manager.py", line 259, in create_volume
	    msg = (_('Create volume failed with error: %s'), six.text_type(ex))
	  File "/usr/lib/python3.6/site-packages/python_3parclient-4.2.9-py3.6.egg/hpe3parclient/exceptions.py", line 104, in __str__
	    
	builtins.AttributeError: 'RequestException' object has no attribute 'message'

wsapisession exhausts on 3PAR for 3.3.1 we get below error in log(container+3pardcv):

2019-04-16 09:56:26.659 19 INFO hpedockerplugin.synchronization [-] Lock acquired: [caller=create_volume, lock-name=sneha_vol_1]�[00m
2019-04-16 09:56:26.659 19 INFO hpedockerplugin.volume_manager [-] In _volumedriver_create�[00m
2019-04-16 09:56:26.659 19 DEBUG etcd.client [-] Issuing read for key /volumes/ with args {'recursive': True} read /usr/lib/python3.6/site-packages/python_etcd-0.4.5-py3.6.egg/etcd/client.py:582�[00m
2019-04-16 09:56:26.660 19 INFO hpedockerplugin.etcdutil [-] Get volbyname: volname is sneha_vol_1�[00m
2019-04-16T09:56:26+0530 [twisted.logger._global#warn] /usr/lib/python3.6/site-packages/urllib3-1.23-py3.6.egg/urllib3/connectionpool.py:857: urllib3.exceptions.InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
2019-04-16T09:56:26+0530 [twisted.logger._global#warn] /usr/lib/python3.6/site-packages/urllib3-1.23-py3.6.egg/urllib3/connectionpool.py:857: urllib3.exceptions.InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
2019-04-16 09:56:26.746 19 DEBUG hpedockerplugin.hpe.hpe_3par_common [-] Connecting to 3PAR client_login /python-hpedockerplugin/hpedockerplugin/hpe/hpe_3par_common.py:159�[00m
2019-04-16T09:56:26+0530 [twisted.logger._global#warn] /usr/lib/python3.6/site-packages/urllib3-1.23-py3.6.egg/urllib3/connectionpool.py:857: urllib3.exceptions.InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
2019-04-16T09:56:27+0530 [twisted.logger._global#warn] /usr/lib/python3.6/site-packages/urllib3-1.23-py3.6.egg/urllib3/connectionpool.py:857: urllib3.exceptions.InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
2019-04-16T09:56:30+0530 [twisted.logger._global#warn] /usr/lib/python3.6/site-packages/urllib3-1.23-py3.6.egg/urllib3/connectionpool.py:857: urllib3.exceptions.InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
2019-04-16T09:56:37+0530 [twisted.logger._global#warn] /usr/lib/python3.6/site-packages/urllib3-1.23-py3.6.egg/urllib3/connectionpool.py:857: urllib3.exceptions.InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
2019-04-16T09:56:52+0530 [twisted.logger._global#warn] /usr/lib/python3.6/site-packages/urllib3-1.23-py3.6.egg/urllib3/connectionpool.py:857: urllib3.exceptions.InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager [-] ('Create volume failed with error: %s', 'Service Unavailable (HTTP 503) 31 - maximum number of server connections has been reached'): hpe3parclient.exceptions.HTTPServiceUnavailable: Service Unavailable (HTTP 503) 31 - maximum number of server connections has been reached
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager Traceback (most recent call last):
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager   File "/python-hpedockerplugin/hpedockerplugin/volume_manager.py", line 237, in create_volume
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager     bkend_vol_name = self._create_volume(vol, undo_steps)
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager   File "/python-hpedockerplugin/hpedockerplugin/volume_manager.py", line 1774, in _create_volume
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager     bkend_vol_name = self._hpeplugin_driver.create_volume(vol_specs)
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager   File "/python-hpedockerplugin/hpedockerplugin/hpe/hpe_3par_iscsi.py", line 172, in create_volume
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager     common = self._login()
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager   File "/python-hpedockerplugin/hpedockerplugin/hpe/hpe_3par_iscsi.py", line 82, in _login
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager     common.client_login()
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager   File "/python-hpedockerplugin/hpedockerplugin/hpe/hpe_3par_common.py", line 161, in client_login
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager     self.src_bkend_config.hpe3par_password)
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager   File "/usr/lib/python3.6/site-packages/python_3parclient-4.2.9-py3.6.egg/hpe3parclient/client.py", line 294, in login
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager     self.http.authenticate(username, password, optional)
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager   File "/usr/lib/python3.6/site-packages/python_3parclient-4.2.9-py3.6.egg/hpe3parclient/http.py", line 124, in authenticate
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager     resp, body = self.post('/credentials', body=info)
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager   File "/usr/lib/python3.6/site-packages/python_3parclient-4.2.9-py3.6.egg/hpe3parclient/http.py", line 376, in post
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager     return self._cs_request(url, 'POST', **kwargs)
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager   File "/usr/lib/python3.6/site-packages/python_3parclient-4.2.9-py3.6.egg/hpe3parclient/http.py", line 321, in _cs_request
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager     **kwargs)
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager   File "/usr/lib/python3.6/site-packages/python_3parclient-4.2.9-py3.6.egg/hpe3parclient/http.py", line 297, in _time_request
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager     resp, body = self.request(url, method, **kwargs)
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager   File "/usr/lib/python3.6/site-packages/python_3parclient-4.2.9-py3.6.egg/hpe3parclient/http.py", line 279, in request
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager     raise ex
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager   File "/usr/lib/python3.6/site-packages/python_3parclient-4.2.9-py3.6.egg/hpe3parclient/http.py", line 262, in request
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager     raise exceptions.from_response(resp, body)
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager hpe3parclient.exceptions.HTTPServiceUnavailable: Service Unavailable (HTTP 503) 31 - maximum number of server connections has been reached
2019-04-16 09:56:52.992 19 ERROR hpedockerplugin.volume_manager �[00m
@prablr79 prablr79 added this to the 3.2 milestone Apr 17, 2019
@sneharai4
Copy link
Collaborator Author

Adding complete logs for reference.
3pardcv.log
plugin_container.log

@prablr79 prablr79 added the Fixed Code fixed label Jun 3, 2019
@wdurairaj
Copy link
Collaborator

@prablr79, please remove the Fixed label. This is not fixed yet.

@prablr79
Copy link
Contributor

@wdurairaj this was communicated in Bug scrub, no problem removing Fixed tag since this needs a fix will again look into the bug scrub meeting during current sprint cycle.

@prablr79 prablr79 removed the Fixed Code fixed label Jun 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug low
Projects
None yet
Milestone
3.2
Development

No branches or pull requests
3 participants
@wdurairaj @sneharai4 @prablr79