Improving Code Structure and Encapsulation in the API

[varshith257]

Description:
In recent observations regarding the interdependency of features and the difficulty in maintaining and modifying the codebase, this discussion has initiated to be focused on improving the overall structure and encapsulation of our API code. By addressing these concerns, we aim to enhance the maintainability, scalability, and ease of development of our project.

Reference #1997 (comment)

Further, a lot of the features currently in our API are heavily interlaced and dependent on each other, with redundant code at the level of individual resolvers, which I think is not optimal, as the size the organization grows. Almost everything is handled by GraphQL, which is inherently not optimal to do a lot of things.
I believe that any new feature that we may integrate in the API, must be abstracted and encapsulated within a dedicated section of the API, which should just expose an interface to interact with, and should be a black box to the rest of the API. This would particularly be beneficial in an opensource setup, as any error that might occur in that particular feature, would be known to be isolated within that section. And debugging it for new contributors would be easier due to the known location.
Currently, even a simple change of name of a function leads to an unnecessary cascading effect of changes in dozens of files, further tracking down every file where a particular function might have been used is also a method of hit and trial, as everything is interdependent. We may not implement these right away, but we can start thinking in that direction.

@palisadoes Here's the discussion, @xoldyckk @Anubhav-2003

Note

Please make sure to not to create many threads in this discussion it's very hard to track

[varshith257]

Before moving to the way for solutions, I would like to take part with all our API contributors in this discussion to share their experiences and observations to understand the issues they faced or facing. Then we can come up with the perfect solution for it.

@palisadoes Can you please take a look at this? We have to make our API Slack members to participate here to get a deeper understanding of their experiences

[Anubhav-2003]

@varshith257 Thank you for creating this discussion. I feel that implementing new features adhering to the principle of “separation of concerns”, by encapsulating and abstracting them behind an interface, would greatly enhance the structure of our code base, improving maintainability as well as make it less overwhelming for new contributors. Let’s see the insight from others.

[xoldd]

Depending on a single endpoint is what graphql is meant for.

[varshith257]

Okay, let's evaluate this approach in the graphql yoga migration work

[Anubhav-2003]

@xoldyckk Respected sir,

Actually I was trying to suggest that we can take certain parts of the API, outside of the GraphQL section, and implement it in a more comprehensive manner like user authentication. But if you don't find the idea to be useful, then we will shift our approach.

Thank You.

[xoldd]

Object Oriented Programming is bad in most cases. There has been a fair bit of discussion in the past regarding pitfalls of OOP, you can look that up on the internet. It is why modern programming languages like rust and golang have refrained from allowing it in their language specification.

@Anubhav-2003 I had already mentioned about seperating authentication from graphql and implementing it using REST apis in the other discussion tab because it is a requirement for it. Many authentication flows require calling strict RESTful endpoints that is hard to do with graphql because it uses a single endpoint for everything and requires the graph query language for communication. Your suggestion for OOP lacks any substance and doesn't provide real arguments for why it should be implemented. I don't see any complexity in the current codebase just more verbosity and that verbosity cannot be prevented no matter what form of code design you follow. Most of the codebase lives in resolvers which are simple functions and are clearly seperated. From my experience OOP is always more concerned about stupid code aesthetics instead of writing actual code that solves problems.

[Anubhav-2003]

@xoldyckk Respected sir, I completely agree with you on the user authentication, actually when I said about implementing object oriented paradigm, I was not meaning about the entire code base, but rather taking out certain functionalities out of the graphQL section, like user authentication and static file transactions, and implementing them using a restful approach for more comprehensive implementations. The reason I am saying so is not because of complexity but rather because of redundancy of logic in certain parts.

Everyone has their POV and I respect yours, but please don’t say object oriented programming is useless, as for the substance as to why oops should be implemented” I do believe that breaking down the CodeBase into smaller sections and keeping each section completely isolated from the other to the maximum extent possible, has great benefits, especially in an open source environment where coordination among different contributors is less compared to a dedicated firm. Sometimes it’s not just about implementing something that solves the problem, but also implementing it in a structured fashion, so that other people find it easier to understand. I have experience working on the codebase of multinational organisations like “Groww”, which actually needs low latency implementations due to being a trading firm, has their entire code base based on the principle of modularity. Currently the size of the Talawa API repository is small, so the current code structure is working fine but having the entire core implementation of the API within a single folder called resolvers is definitely not optimal in the long run.

Suggestions that I am making are not shortsighted, but keeping our future in mind, where having a production worthy system design should be a top priority. Having an entire API completely dependent upon a single graphQL end point even for tasks for which it is inherently not optimal, is definitely not one. I completely respect your decisions as you are senior to me in this organisation, and whatever you say we will be looking to collaborate on that, but please don’t outrightly call an idea “useless”, because I always respect everyone but also expect it in reciprocation, and especially an idea that is used by some of the best trading firms, where I have seen some of the best implementation of code in my life, simply goes against the spirit of “palisadoes”.

Thank you.

[varshith257]

Important

In this thread, you can share your experiences and observations about the difficulties faced in the API codebase

[pranshugupta54]

Thread on the point:

I believe that we should focus more on the 'multimedia aspect of it with Real-time Communication' to increase our user engagement. And rather than limiting ourselves to small non-profit organizations, focus on being an open source alternative to regular social media in general, which can be used by just about any organization. I believe that improving the current mechanism of 'Posts', as well as adding new features for interactivity like short reels, and support for conference calls among members of the same organizations can help us go a long way to achieve the goal of being a social media of organizations.

Our priority should be to clarify that our main objective is not to create alternatives to mainstream social media platforms like Instagram, Facebook, or Snapchat. Instead, we aim to develop a superior application tailored for small, non-profit organizations in countries such as Jamaica. In these regions, we cannot assume that individuals are proficient in technology or have access to basic tech resources. Our goal is to empower small community groups to enhance their technological literacy and undertake fundamental tasks such as managing memberships. We must refrain from creating elaborate products with features like real-time media that would incur significant hosting costs. Many communities lack the resources and technical expertise necessary to host such technologies. Therefore, our focus should be on crafting a user-friendly platform that minimizes costs for organizers and is accessible to a wide range of users.

[varshith257]

@pranshugupta54 Yeah, it's just a suggestion made by @Anubhav-2003 in his mind about improvements. Even though our community goal isn't to provide social media platform but to be clear to support religious organizations, charities and other non-profits who lack funding to efficiently manage their Community Organization

We can make use of discussion #1997 for this. I think it's irrelevant to this discussion

[Anubhav-2003]

You are absolutely correct, and I’m not suggesting to deviate from our original aim. But what I am suggesting is the fact that there are many other platforms which are providing kind of a similar experience as we do to our communities. If we want our project to grow in the open source community, we need a differentiating factor, and implementing new features does not necessarily mean scraping the old interface.

Also, we cannot generalise the size of a community. Certain organisations are larger than others with more resources, we can always have multiple versions of our offerings depending on the size of our client, also there are multiple ways by which you can create your own multimedia content gateway without relying on cloud based vendor. this would definitely not incur extra costs, and Can be hosted in the same compute instance alongside our API. Palisades now is a global community, so I believe that it is our duty to broaden the horizon of its offerings, not restricting itself to be a niche.

These are purely my POV, which can be implemented for improvisation, but your point is also completely valid.

Thank you.

[varshith257]

I think @palisadoes and @DMills27 are right persons to answer this

[Anubhav-2003]

#1997 (comment) I am also of the opinion that GridFS is not optimal for storing multimedia content in the long run as it has numerous security related vulnerabilities, we can discuss about the idea over here in this discussion thread.

[AVtheking]

@varshith257 @Anubhav-2003 what other options you have other than gridFS ?

[varshith257]

@AVtheking I think it's part of the GSOC project and couldn't be a good option to suggest. My opinion here is not to limit to gridFS as it is a discussion of a long back and think of alternative solutions that relate to Talawa goals

[palisadoes]

The goal is to have a solution that allows API installation on a single server that can support all its features without using external cloud providers using configuration defaults. This must be the priority as it makes installation simple.

If cloud providers are used, they must be optional and only require the credentials of the parent organization running the app, and not The Palisadoes Foundation. The Talawa community must not be expected to fund the cloud services used by others.

Having a fully non cloud solution that works now with an easy migration path to cloud support in future years is preferred to having cloud support now.

Our priority must be simple features and configuration that work

[Shubh152]

Making any external cloud support optional will not be feasible as we will at least need some cloud database to store the organisation's data. If we do it on a single API instance and only use the disk space of the cloud VPS, we will have to implement ways to replicate it to multiple servers like done by cloud providers and may need to work on additional security features. Also, the only upsides of cloud VPS are affordability and more control but it is a lot less reliable.

[Anubhav-2003]

@palisadoes
Respected sir,

My point is also exactly the same, that we can implement new features without relying on cloud vendors, by setting up in-house systems, than can run in the same instance powering our API, this would incur no extra cost but would definitely improve the overall architecture of our API. But people often confuse new features with cloud based services, often forgetting that anything that is in cloud can also be built in-house to eliminate costs.

Thank You.

[Anubhav-2003]

@Shubh152 Actually cost is a major factor, therefore relying on cloud vendors is not an option right now due to its expense. But you are also right that its not as reliable as dedicated cloud services. But will suit our purpose, and would be an upgrade over the existing implementations.

[Shubh152]

@Anubhav-2003 I understand that cost efficiency is a primary concern, but If we use an on-premises database like PostgreSQL with the available disk space, we will have to think about horizontal read or read-write scaling on multiple servers, which adds a layer of complexity to setting up the application for our users compared to a cloud database.

[Anubhav-2003]

@Shubh152 What you are saying is valid. But if and when we reach the scale of traffic which would necessitate having multiple instances, I think we would not have the financial constraint to use on-prem implementations, and would shift to cloud based vendors. But that is not the main topic of discussion in this thread. The main aim of this discussion is to actually talk on the architecture of the code base, and implement new features at their dedicated sections, adhering to the principal of 'Separation of concerns' through abstraction, and encapsulation, to reduce inter-dependencies between separate functionalities.

You can share your experience regarding this topic, via your previous experience while solving issues in this organization. It would help us get an insight into everyone's perspective, and reach a conclusion.

Thank You.