diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 260ce61..55f7f79 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,5 +1,7 @@ # Contributing +## Installation + After cloning this repo, create a [virtualenv](https://virtualenv.pypa.io/en/stable/) and ensure dependencies are installed by running: ```sh @@ -30,4 +32,66 @@ If you wish to run against a specific version defined in the `tox.ini` file: tox -e py36 ``` -Tox can only use whatever versions of Python are installed on your system. When you create a pull request, Travis will also be running the same tests and report the results, so there is no need for potential contributors to try to install every single version of Python on their own system ahead of time. We appreciate opening issues and pull requests to make PyMS even more stable & useful! +Tox can only use whatever versions of Python are installed on your system. When you create a pull request, Travis will also be running the same tests and report the results, so there is no need for potential contributors to try to install every single version of Python on their own system ahead of time. + +## Pipenv + +### Advantages over plain pip and requirements.txt +[Pipenv](https://pipenv.readthedocs.io/en/latest/) generates two files: a `Pipfile`and a `Pipfile.lock`. +* `Pipfile`: Is a high level declaration of the dependencies of your project. It can contain "dev" dependencies (usually test related stuff) and "standard" dependencies which are the ones you'll need for your project to function +* `Pipfile.lock`: Is the "list" of all the dependencies your Pipfile has installed, along with their version and their hashes. This prevents two things: Conflicts between dependencies and installing a malicious module. + +### How to... + +Here the most 'common' `pipenv` commands, for a more in-depth explanation please refer to the [official documentation](https://pipenv.readthedocs.io/en/latest/). + +#### Install pipenv +```bash +pip install pipenv +``` + +#### Install dependencies defined in a Pipfile +```bash +pipenv install +``` + +#### Install both dev and "standard" dependencies defined in a Pipfile +```bash +pipenv install --dev +``` + +#### Install a new module +```bash +pipenv install django +``` + +#### Install a new dev module (usually test related stuff) +```bash +pipenv install nose --dev +``` + +#### Install dependencies in production +```bash +pipenv install --deploy +``` + +#### Start a shell +```bash +pipenv shell +``` + +## Documentation + +This project use MkDocs + +* `mkdocs new [dir-name]` - Create a new project. +* `mkdocs serve` - Start the live-reloading docs server. +* `mkdocs build` - Build the documentation site. +* `mkdocs help` - Print this help message. + +### Project layout + + mkdocs.yml # The configuration file. + docs/ + index.md # The documentation homepage. + ... # Other markdown pages, images and other files. \ No newline at end of file diff --git a/README.md b/README.md index fe5e7ba..e0b9753 100644 --- a/README.md +++ b/README.md @@ -10,8 +10,30 @@ PyMS, Python MicroService, is a collections of libraries, best practices and recommended ways to build microservices with Python. +## Documentation + To know how use, install or build a project see the docs: https://py-ms.readthedocs.io/en/latest/ +## Motivation + +When we started to create microservice with no idea, we were looking for tutorials, guides, best practices, but we found +nothing to create professional projects. Most articles say: +- "Install flask" +- "Create routes" +- (Sometimes) "Create a swagger specs" +- "TA-DA! you have a microservice" + +But... what happens with our configuration out of code like Kubernetes configmap? what happens with transactionality? +If we have many microservices, what happens with traces?. + +There are many problems around Python and microservices and we can`t find anyone to give a solution. + +We start creating these projects to try to solve all the problems we have found in our professional lives about +microservices architecture. + +Nowadays, is not perfect and we have a looong roadmap, but we hope this library could help other felas and friends ;) + + ## Installation ```bash pip install py-ms @@ -23,80 +45,21 @@ pip install py-ms Module to read yaml or json configuration from a dictionary or a path. ### pyms/flask/app -With the funcion `create_app` initialize the Flask app, register [blueprints](http://flask.pocoo.org/docs/0.12/blueprints/) -and intialize all libraries like Swagger, database, trace system, custom logger format, etc. +With the function `create_app` initialize the Flask app, register [blueprints](http://flask.pocoo.org/docs/0.12/blueprints/) +and initialize all libraries such as Swagger, database, trace system, custom logger format, etc. + +### pyms/flask/services +Integrations and wrappers over common libs like request, swagger, connexion ### pyms/flask/healthcheck -This views is usually used by Kubernetes, Eureka and other systems to check if our application is up and running. +This view is usually used by Kubernetes, Eureka and other systems to check if our application is running. ### pyms/logger Print logger in JSON format to send to server like Elasticsearch. Inject span traces in logger. -### pyms/rest_template -Encapsulate common rest operations between business services propagating trace headers if configured. - ### pyms/tracer -Create an injector `flask_opentracing.FlaskTracer` to use in our projects - -## Pipenv - -### Advantages over plain pip and requirements.txt -[Pipenv](https://pipenv.readthedocs.io/en/latest/) generates two files: a `Pipfile`and a `Pipfile.lock`. -* `Pipfile`: Is a high level declaration of the dependencies of your project. It can contain "dev" dependencies (usually test related stuff) and "standard" dependencies which are the ones you'll need for your project to function -* `Pipfile.lock`: Is the "list" of all the dependencies your Pipfile has installed, along with their version and their hashes. This prevents two things: Conflicts between dependencies and installing a malicious module. - -### How to... - -Here the most 'common' `pipenv` commands, for a more in-depth explanation please refer to the [official documentation](https://pipenv.readthedocs.io/en/latest/). - -#### Install pipenv -```bash -pip install pipenv -``` - -#### Install dependencies defined in a Pipfile -```bash -pipenv install -``` - -#### Install both dev and "standard" dependencies defined in a Pipfile -```bash -pipenv install --dev -``` - -#### Install a new module -```bash -pipenv install django -``` - -#### Install a new dev module (usually test related stuff) -```bash -pipenv install nose --dev -``` - -#### Install dependencies in production -```bash -pipenv install --deploy -``` - -#### Start a shell -```bash -pipenv shell -``` - -## Documentation - -This project use MkDocs - -* `mkdocs new [dir-name]` - Create a new project. -* `mkdocs serve` - Start the live-reloading docs server. -* `mkdocs build` - Build the documentation site. -* `mkdocs help` - Print this help message. - -### Project layout - - mkdocs.yml # The configuration file. - docs/ - index.md # The documentation homepage. - ... # Other markdown pages, images and other files. +Create an injector `flask_opentracing.FlaskTracer` to use in our projects. +## How To Contrib +We appreciate opening issues and pull requests to make PyMS even more stable & useful! See [This doc](COONTRIBUTING.md) +for more details \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index b1b5ab7..822f694 100644 --- a/docs/index.md +++ b/docs/index.md @@ -3,6 +3,26 @@ PyMS, Python MicroService, is a collection of libraries, best practices and recommended ways to build microservices with Python. +## Motivation + +When we started to create microservice with no idea, we were looking for tutorials, guides, best practices, but we found +nothing to create professional projects. Most articles say: +- "Install flask" +- "Create routes" +- (Sometimes) "Create a swagger specs" +- "TA-DA! you have a microservice" + +But... what happens with our configuration out of code like Kubernetes configmap? what happens with transactionality? +If we have many microservices, what happens with traces?. + +There are many problems around Python and microservices and we can`t find anyone to give a solution. + +We start creating these projects to try to solve all the problems we have found in our professional lives about +microservices architecture. + +Nowadays, is not perfect and we have a looong roadmap, but we hope this library could help other felas and friends ;) + + ## Index: * [PyMS structure](structure.md) * [Configuration](configuration.md) diff --git a/docs/structure.md b/docs/structure.md index ff035b9..855a51a 100644 --- a/docs/structure.md +++ b/docs/structure.md @@ -7,6 +7,9 @@ Module to read yaml or json configuration from a dictionary or a path. With the function `create_app` initialize the Flask app, register [blueprints](http://flask.pocoo.org/docs/0.12/blueprints/) and initialize all libraries such as Swagger, database, trace system, custom logger format, etc. +### pyms/flask/services +Integrations and wrappers over common libs like request, swagger, connexion + ### pyms/flask/healthcheck This view is usually used by Kubernetes, Eureka and other systems to check if our application is running. diff --git a/pylintrc b/pylintrc index 325f0e4..b43dc8e 100644 --- a/pylintrc +++ b/pylintrc @@ -54,7 +54,7 @@ confidence= # --enable=similarities". If you want to run only the classes checker, but have # no Warning level messages displayed, use"--disable=all --enable=classes # --disable=W" -disable=C0301 +disable=C0301,C0111,C0103,logging-format-interpolation # Enable the message, report, category or checker with the given id(s). You can # either give multiple identifier separated by comma (,) or put this option diff --git a/pyms/__init__.py b/pyms/__init__.py index a2efb9b..b04ec21 100644 --- a/pyms/__init__.py +++ b/pyms/__init__.py @@ -2,4 +2,4 @@ __email__ = "a.vara.1986@gmail.com" -__version__ = "1.1.0" \ No newline at end of file +__version__ = "1.2.0" diff --git a/pyms/flask/app/create_app.py b/pyms/flask/app/create_app.py index 6275f6a..7b50d0b 100644 --- a/pyms/flask/app/create_app.py +++ b/pyms/flask/app/create_app.py @@ -8,7 +8,7 @@ from pyms.config.conf import get_conf from pyms.constants import LOGGER_NAME, SERVICE_ENVIRONMENT from pyms.flask.healthcheck import healthcheck_blueprint -from pyms.flask.services.driver import ServicesManager +from pyms.flask.services.driver import ServicesManager, DriverService from pyms.logger import CustomJsonFormatter from pyms.tracer.main import init_lightstep_tracer @@ -35,6 +35,8 @@ def __call__(cls, *args, **kwargs): class Microservice(metaclass=SingletonMeta): service = None application = None + swagger = DriverService + requests = DriverService def __init__(self, *args, **kwargs): self.service = kwargs.get("service", os.environ.get(SERVICE_ENVIRONMENT, "ms")) @@ -70,14 +72,15 @@ def init_logger(self): def init_app(self) -> Flask: if getattr(self, "swagger", False): app = connexion.App(__name__, specification_dir=os.path.join(self.path, self.swagger.path)) - app.add_api(self.swagger.file, - arguments={'title': self.config.APP_NAME}, - base_path=self.config.APPLICATION_ROOT - ) + app.add_api( + self.swagger.file, + arguments={'title': self.config.APP_NAME}, + base_path=self.config.APPLICATION_ROOT + ) # Invert the objects, instead connexion with a Flask object, a Flask object with application = app.app - application._connexion_app = app + application.connexion_app = app else: application = Flask(__name__, static_folder=os.path.join(self.path, 'static'), template_folder=os.path.join(self.path, 'templates')) @@ -121,4 +124,4 @@ def add_error_handler(self, code_or_exception, handler): :param code_or_exception: HTTP error code or exception :param handler: callback for error handler """ - self.application._connexion_app.add_error_handler(code_or_exception, handler) + self.application.connexion_app.add_error_handler(code_or_exception, handler) diff --git a/pyms/flask/app/create_config.py b/pyms/flask/app/create_config.py index f21fcb0..e93c3fc 100644 --- a/pyms/flask/app/create_config.py +++ b/pyms/flask/app/create_config.py @@ -3,4 +3,4 @@ def config(): ms = Microservice() - return ms.config \ No newline at end of file + return ms.config diff --git a/pyms/flask/healthcheck/__init__.py b/pyms/flask/healthcheck/__init__.py index bc7cac8..3c411f6 100644 --- a/pyms/flask/healthcheck/__init__.py +++ b/pyms/flask/healthcheck/__init__.py @@ -1,9 +1,3 @@ -"""Init file -""" -from __future__ import unicode_literals, print_function, absolute_import, division +from pyms.flask.healthcheck.healthcheck import healthcheck_blueprint -from flask import Blueprint - -healthcheck_blueprint = Blueprint('healthcheck', __name__, static_url_path='/static') - -from pyms.flask.healthcheck import healthcheck +__all__ = ['healthcheck_blueprint'] diff --git a/pyms/flask/healthcheck/healthcheck.py b/pyms/flask/healthcheck/healthcheck.py index 6ded4dc..b0429d6 100644 --- a/pyms/flask/healthcheck/healthcheck.py +++ b/pyms/flask/healthcheck/healthcheck.py @@ -1,6 +1,8 @@ -"""Healthcheck -""" -from pyms.flask.healthcheck import healthcheck_blueprint +from __future__ import unicode_literals, print_function, absolute_import, division + +from flask import Blueprint + +healthcheck_blueprint = Blueprint('healthcheck', __name__, static_url_path='/static') @healthcheck_blueprint.route('/healthcheck', methods=['GET']) diff --git a/pyms/flask/services/requests.py b/pyms/flask/services/requests.py index 218f927..bb222c0 100644 --- a/pyms/flask/services/requests.py +++ b/pyms/flask/services/requests.py @@ -5,12 +5,37 @@ import opentracing import requests from flask import current_app, request +from requests.adapters import HTTPAdapter +from requests.packages.urllib3.util.retry import Retry from pyms.constants import LOGGER_NAME from pyms.flask.services.driver import DriverService logger = logging.getLogger(LOGGER_NAME) +DEFAULT_RETRIES = 3 + +DEFAULT_STATUS_RETRIES = (500, 502, 504) + + +def retry(f): + def wrapper(*args, **kwargs): + response = False + i = 0 + response_ok = False + retries = args[0].retries + status_retries = args[0].status_retries + while i < retries and response_ok is False: + response = f(*args, **kwargs) + i += 1 + if response.status_code not in status_retries: + response_ok = True + logger.debug("Response {}".format(response)) + if not response_ok: + logger.warning("Response ERROR: {}".format(response)) + return response + return wrapper + class Service(DriverService): service = "requests" @@ -21,8 +46,32 @@ class Service(DriverService): def __init__(self, service, *args, **kwargs): """Initialization for trace headers propagation""" super().__init__(service, *args, **kwargs) + self.retries = self.config.retries or DEFAULT_RETRIES + self.status_retries = self.config.status_retries or DEFAULT_STATUS_RETRIES - def insert_trace_headers(self, headers): + def requests(self, session: requests.Session): + """ + A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a + second try without a delay). urllib3 will sleep for: {backoff factor} * (2 ^ ({number of total retries} - 1)) + seconds. If the backoff_factor is 0.1, then sleep() will sleep for [0.0s, 0.2s, 0.4s, ...] between retries. + It will never be longer than Retry.BACKOFF_MAX. By default, backoff is disabled (set to 0). + :param session: + :return: + """ + session_r = session or requests.Session() + retry = Retry( + total=self.retries, + read=self.retries, + connect=self.retries, + backoff_factor=0.3, + status_forcelist=self.status_retries, + ) + adapter = HTTPAdapter(max_retries=retry) + session_r.mount('http://', adapter) + session_r.mount('https://', adapter) + return session_r + + def insert_trace_headers(self, headers: dict) -> dict: """Inject trace headers if enabled. :param headers: dictionary of HTTP Headers to send. @@ -38,7 +87,13 @@ def insert_trace_headers(self, headers): logger.debug("Tracer error {}".format(ex)) return headers - def _get_headers(self, headers): + def propagate_headers(self, headers: dict) -> dict: + for k, v in request.headers: + if not headers.get(k): + headers.update({k: v}) + return headers + + def _get_headers(self, headers, propagate_headers=False): """If enabled appends trace headers to received ones. :param headers: dictionary of HTTP Headers to send. @@ -52,7 +107,8 @@ def _get_headers(self, headers): self._tracer = current_app.tracer if self._tracer: headers = self.insert_trace_headers(headers) - + if self.config.propagate_headers or propagate_headers: + headers = self.propagate_headers(headers) return headers @staticmethod @@ -81,10 +137,11 @@ def parse_response(self, response): data = data.get(self.config.data, {}) return data except ValueError: - current_app.logger.warning("Response.content is not a valid json {}".format(response.content)) + logger.warning("Response.content is not a valid json {}".format(response.content)) return {} - def get(self, url, path_params=None, params=None, headers=None, **kwargs): + @retry + def get(self, url, path_params=None, params=None, headers=None, propagate_headers=False, **kwargs): """Sends a GET request. :param url: URL for the new :class:`Request` object. Could contain path parameters @@ -98,11 +155,12 @@ def get(self, url, path_params=None, params=None, headers=None, **kwargs): """ full_url = self._build_url(url, path_params) - headers = self._get_headers(headers) - current_app.logger.info("Get with url {}, params {}, headers {}, kwargs {}". - format(full_url, params, headers, kwargs)) - response = requests.get(full_url, params=params, headers=headers, **kwargs) - current_app.logger.info("Response {}".format(response)) + headers = self._get_headers(headers=headers, propagate_headers=propagate_headers) + logger.info("Get with url {}, params {}, headers {}, kwargs {}". + format(full_url, params, headers, kwargs)) + + session = requests.Session() + response = self.requests(session=session).get(full_url, params=params, headers=headers, **kwargs) return response @@ -122,6 +180,7 @@ def get_for_object(self, url, path_params=None, params=None, headers=None, **kwa response = self.get(url, path_params=path_params, params=params, headers=headers, **kwargs) return self.parse_response(response) + @retry def post(self, url, path_params=None, data=None, json=None, headers=None, **kwargs): """Sends a POST request. @@ -138,10 +197,12 @@ def post(self, url, path_params=None, data=None, json=None, headers=None, **kwar full_url = self._build_url(url, path_params) headers = self._get_headers(headers) - current_app.logger.info("Post with url {}, data {}, json {}, headers {}, kwargs {}".format(full_url, data, json, - headers, kwargs)) - response = requests.post(full_url, data=data, json=json, headers=headers, **kwargs) - current_app.logger.info("Response {}".format(response)) + logger.info("Post with url {}, data {}, json {}, headers {}, kwargs {}".format(full_url, data, json, + headers, kwargs)) + + session = requests.Session() + response = self.requests(session=session).post(full_url, data=data, json=json, headers=headers, **kwargs) + logger.info("Response {}".format(response)) return response @@ -162,6 +223,7 @@ def post_for_object(self, url, path_params=None, data=None, json=None, headers=N response = self.post(url, path_params=path_params, data=data, json=json, headers=headers, **kwargs) return self.parse_response(response) + @retry def put(self, url, path_params=None, data=None, headers=None, **kwargs): """Sends a PUT request. @@ -178,10 +240,12 @@ def put(self, url, path_params=None, data=None, headers=None, **kwargs): full_url = self._build_url(url, path_params) headers = self._get_headers(headers) - current_app.logger.info("Put with url {}, data {}, headers {}, kwargs {}".format(full_url, data, headers, - kwargs)) - response = requests.put(full_url, data, headers=headers, **kwargs) - current_app.logger.info("Response {}".format(response)) + logger.info("Put with url {}, data {}, headers {}, kwargs {}".format(full_url, data, headers, + kwargs)) + + session = requests.Session() + response = self.requests(session=session).put(full_url, data, headers=headers, **kwargs) + logger.info("Response {}".format(response)) return response @@ -202,6 +266,7 @@ def put_for_object(self, url, path_params=None, data=None, headers=None, **kwarg response = self.put(url, path_params=path_params, data=data, headers=headers, **kwargs) return self.parse_response(response) + @retry def delete(self, url, path_params=None, headers=None, **kwargs): """Sends a DELETE request. @@ -215,8 +280,10 @@ def delete(self, url, path_params=None, headers=None, **kwargs): full_url = self._build_url(url, path_params) headers = self._get_headers(headers) - current_app.logger.info("Delete with url {}, headers {}, kwargs {}".format(full_url, headers, kwargs)) - response = requests.delete(full_url, headers=headers, **kwargs) - current_app.logger.info("Response {}".format(response)) + logger.info("Delete with url {}, headers {}, kwargs {}".format(full_url, headers, kwargs)) + + session = requests.Session() + response = self.requests(session=session).delete(full_url, headers=headers, **kwargs) + logger.info("Response {}".format(response)) return response diff --git a/pyms/logger/__init__.py b/pyms/logger/__init__.py index 8622f8c..50223f3 100644 --- a/pyms/logger/__init__.py +++ b/pyms/logger/__init__.py @@ -1,3 +1,5 @@ """Init file """ -from pyms.logger.logger import CustomJsonFormatter \ No newline at end of file +from pyms.logger.logger import CustomJsonFormatter + +__all__ = ['CustomJsonFormatter', ] diff --git a/pyms/tracer/__init__.py b/pyms/tracer/__init__.py index 38c2a96..5732cc7 100644 --- a/pyms/tracer/__init__.py +++ b/pyms/tracer/__init__.py @@ -1,2 +1,2 @@ """Init file -""" \ No newline at end of file +""" diff --git a/tests/test_requests.py b/tests/test_requests.py index 4993a28..d8529e1 100644 --- a/tests/test_requests.py +++ b/tests/test_requests.py @@ -2,12 +2,14 @@ """ import json import os +import time import unittest import requests_mock from pyms.constants import CONFIGMAP_FILE_ENVIRONMENT from pyms.flask.app import Microservice +from pyms.flask.services.requests import DEFAULT_RETRIES class RequestServiceTests(unittest.TestCase): @@ -20,8 +22,7 @@ def setUp(self): os.environ[CONFIGMAP_FILE_ENVIRONMENT] = os.path.join(self.BASE_DIR, "config-tests.yml") ms = Microservice(service="my-ms", path=__file__) self.app = ms.create_app() - with self.app.app_context(): - self.request = ms.requests + self.request = ms.requests @requests_mock.Mocker() def test_get(self, mock_request): @@ -207,3 +208,82 @@ def test_delete(self, mock_request): self.assertEqual(204, response.status_code) self.assertEqual('', response.text) + + def test_propagate_headers_empty(self, ): + input_headers = { + + } + expected_headers = { + 'Content-Length': '12', + 'Content-Type': 'application/x-www-form-urlencoded', + 'Host': 'localhost' + } + with self.app.test_request_context( + '/tests/', data={'format': 'short'}): + headers = self.request.propagate_headers(input_headers) + + self.assertEqual(expected_headers, headers) + + def test_propagate_headers_no_override(self): + input_headers = { + 'Host': 'my-server' + } + expected_headers = { + 'Host': 'my-server' + } + with self.app.test_request_context( + '/tests/'): + headers = self.request.propagate_headers(input_headers) + + self.assertEqual(expected_headers, headers) + + def test_propagate_headers_propagate(self): + input_headers = { + } + expected_headers = { + 'Content-Length': '12', + 'Content-Type': 'application/x-www-form-urlencoded', + 'Host': 'localhost', + 'A': 'b', + } + with self.app.test_request_context( + '/tests/', data={'format': 'short'}, headers={'a': 'b'}): + headers = self.request.propagate_headers(input_headers) + + self.assertEqual(expected_headers, headers) + + def test_propagate_headers_propagate_no_override(self): + input_headers = { + 'Host': 'my-server', + 'Span': '1234', + } + expected_headers = { + 'Host': 'my-server', + 'A': 'b', + 'Span': '1234', + } + with self.app.test_request_context( + '/tests/', headers={'a': 'b', 'span': '5678'}): + headers = self.request.propagate_headers(input_headers) + + self.assertEqual(expected_headers, headers) + + @requests_mock.Mocker() + def test_retries_with_500(self, mock_request): + url = 'http://localhost:9999' + with self.app.app_context(): + mock_request.get(url, text="", status_code=500) + response = self.request.get(url) + + self.assertEqual(DEFAULT_RETRIES, mock_request.call_count) + self.assertEqual(500, response.status_code) + + @requests_mock.Mocker() + def test_retries_with_200(self, mock_request): + url = 'http://localhost:9999' + with self.app.app_context(): + mock_request.get(url, text="", status_code=200) + response = self.request.get(url) + + self.assertEqual(1, mock_request.call_count) + self.assertEqual(200, response.status_code)