Improvement Suggestions - from Community Call - 03/07/2021

[lalitkale]

SigNoz WishList

After our first community call, @ankitnayan @pranay01 asked for suggestions for the overall project. Here is the list of suggestions we briefly touched upon in the call. All these are improvement opportunities for the Signoz project and the community. I have divided my suggestions based on different stakeholders perspective which are approaching Signoz.

Introduction

End-User Experience

• Story about Name
• why this project Exists
• What pain points it is solving
• Better Documentation for Beginner of Project
  • Built packages
  • Installers/Packages for Different OS/Virtualizations/Public Cloud Providers
• FAQs

End-User Developer Experience

• Clear instructions for Getting started
• Clear instructions for setting up the Project
• Add Unit Tests For Query Service and Flattener Processor

Application Performance Management (APM) Team Experience

• SigNoz's performance Benchmarks
• Comparison with existing marketing solutions

Contributor Experience

• Making things transparent for contributors eg. Test Env. URLs, Docs Wiki etc.
• Improvements in CICD Pipeline

Project Maintainer Experience

• PR Governance and Basic checklist
• Bot for verifying primary checks on PRs (if there is issue associated with the PR, Is it labelled correctly etc, LGTM)

Features:

• Windows/Different *nix distributions Support
• Different programming languages Support
• Support for other OLAP Stores - Druid and Apache Pinot, Apache Kylin

Architecture and System Design

• Query Service have multiple responsibilities and difficult to scale itself.
• Can Flattening Processor stream be shared? This way, we can ingest more data
• Clarity on Backup, Data Retention policies

[pranay01]

Thanks @lalitkale for sharing this note. This is a great framework to think from perspective of different persona's experience with SigNoz.

Can you elaborate a but more on what do you mean by Query Service have multiple responsibilities and difficult to scale itself.?

[lalitkale]

"Query Service have multiple responsibilities and difficult to scale itself" is coming from my half understanding of codebase and maybe an overall project. Let me know, if I have the wrong assumptions and interpretations below. I am no expert of Go-Lang but surely improving my language game.

1. If I refer to responsibilities of query service: The readme states
   • parse the request from Frontend
   • create relevant Druid queries (and all other supported database queries)
   • parse responses from databases and handle errors if any
   • build response in the format accepted by Frontend

From my perspective Parsing the Request from frontend then composing requests DataStore queries needs two different modules or subsystems. Here query service is more of treated as the orchestrator of everything above.

2. I had tried to get some understanding of code and noticed the following points which may affect performance and/or scaling. The solutions may be as simple as exposing it as configurations to as complex as replacing the structure/algo of components. I am open to discussion to improve my understanding and in the process help the betterment of the codebase also. If nothing, we can discuss and discard these. e.g.

• https://github.com/SigNoz/signoz/blob/main/pkg/query-service/druidQuery/query.go#L189 BatchSize is hardcoded
• https://github.com/SigNoz/signoz/blob/main/pkg/query-service/druidQuery/query.go#L235 Metrics definitions should be dynamically picked up to support multiple out of the box and custom definitions.

I am in process of setting up the project and going through the backend codebase and pardon me if any of the statements or concerns are misplaced. I am more willing to learn and contribute further to the codebase. Thank you.