Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Celery integration #808

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

dsborets
Copy link

@dsborets dsborets commented Nov 2, 2019

Celery integration to be able to trace asynchronous tasks

@dsborets dsborets requested review from c24t, reyang, songy23 and a team as code owners November 2, 2019 20:15
@googlebot
Copy link

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@dsborets
Copy link
Author

dsborets commented Nov 5, 2019

Unfortunately travis build keeps failing because of two things:

  1. pkg_resources.ContextualVersionConflict: (setuptools 39.1.0 (/home/travis/build/census-instrumentation/opencensus-python/.tox/py36-unit/lib/python3.6/site-packages), Requirement.parse('setuptools>=40.3.0'), {'google-auth'})
    It's only for python 3.4, 3.6
    I cannot reproduce it locally. When I'm trying to run tox it's failing but for different reason that I don't see on travis, weird...
  2. For python 3.7 tests keeps failing because of sort: it suggesting to remove an empty line, but if I do it python 2.7 will fail because of the same sort is suggesting to add an empty line in the same position
    Any ideas how to fix it?

@lzchen
Copy link
Contributor

lzchen commented Nov 5, 2019

@dsborets
Hi, thanks so much for contributing!
For (1), when you run tox locally, what error are you getting?
For (2), try running isort -rc . on the root folder of opencensus. If it is still failing for one of the versions, take a look at isort configurations to see if you can ignore the error.

@dsborets
Copy link
Author

dsborets commented Nov 5, 2019

@lzchen thanks for helping with troubleshooting this

For (1) I'm getting multiple of:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 101, in worker
    inqueue._writer.close()
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 177, in close
    self._close()
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 361, in _close
    _close(self._handle)
OSError: [Errno 9] Bad file descriptor```

which, seems like could be ignored. And than an exception:

================================================================================================================== FAILURES ==================================================================================================================
____________________________________________________________________________________________________ Test_threading_trace.test_wrap_pool _____________________________________________________________________________________________________

self = <test_threading_trace.Test_threading_trace testMethod=test_wrap_pool>

    def test_wrap_pool(self):
        _tracer = tracer.Tracer()
        execution_context.set_opencensus_tracer(tracer)
    
        trace.trace_integration()
        context = tracer.Tracer().span_context
        print(context.trace_id)
    
        pool = Pool(processes=1)
        with _tracer.span(name='span1'):
>           result = pool.apply_async(fake_pooled_func, ()).get(timeout=1)

contrib/opencensus-ext-threading/tests/test_threading_trace.py:105: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <multiprocessing.pool.ApplyResult object at 0x117305810>, timeout = 1

    def get(self, timeout=None):
        self.wait(timeout)
        if not self.ready():
>           raise TimeoutError
E           multiprocessing.context.TimeoutError

/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py:653: TimeoutError
------------------------------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------------------------------

For (2): When I ran isort -rc . I've got nothing as an output so the code has no problem with import sorting. What is really weird why with the same .isort.cong file the results are different for different python versions?

Exclude `trace.py` file from 'isort' checking. For some reason 'isort' complain about the same line with ambivalent suggestions (remove/add an empty line) when running in different python environments (py27 and py37)
@dsborets
Copy link
Author

dsborets commented Nov 7, 2019

@googlebot I fixed it.

@googlebot
Copy link

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

@googlebot googlebot added cla: yes and removed cla: no labels Nov 7, 2019
@dsborets
Copy link
Author

Hi @songy23, @reyang, @c24t. Could you please review this PR? We are going to use this instrumentation for our services soon. Will be great to have it as part of the standard instrumentation and not as our local package. I'm going to port it to the Opentelemetry project soon

@tmc
Copy link

tmc commented Jan 16, 2020

@dsborets can you look into the circle failure?

@dsborets
Copy link
Author

Hi @tmc That failure happened after I merged the code from the 'master' to my branch. I didn't have this problem before in my branch. Also, it seems like something wrong with sqlite module importing:

Traceback (most recent call last):
  File "/root/repo/.nox/unit-2-7/bin/py.test", line 8, in <module>
    sys.exit(main())
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/config/__init__.py", line 63, in main
    config = _prepareconfig(args, plugins)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/config/__init__.py", line 207, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/helpconfig.py", line 94, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/callers.py", line 81, in get_result
    _reraise(*ex)  # noqa
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/config/__init__.py", line 687, in pytest_cmdline_parse
    self.parse(args)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/config/__init__.py", line 895, in parse
    self._preparse(args, addopts=addopts)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/config/__init__.py", line 841, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/manager.py", line 299, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/importlib_metadata/__init__.py", line 94, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/assertion/rewrite.py", line 304, in load_module
    exec(co, mod.__dict__)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pytest_cov/plugin.py", line 6, in <module>
    import coverage
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/coverage/__init__.py", line 15, in <module>
    from coverage.control import Coverage, process_startup
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/coverage/control.py", line 20, in <module>
    from coverage.data import CoverageData, combine_parallel_data
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/coverage/data.py", line 17, in <module>
    from coverage.sqldata import CoverageData
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/coverage/sqldata.py", line 15, in <module>
    import sqlite3
  File "/usr/local/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/usr/local/lib/python2.7/sqlite3/dbapi2.py", line 28, in <module>
    from _sqlite3 import *
ImportError: No module named _sqlite3

I cannot reproduce it locally, all tests are passing.
It could be some temporary glitch so the next build will be successful, unfortunately I cannot re-run only that step.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants