Add Elasticsearch as read only database post #546

solita-villep-hamalainen · 2025-01-22T10:10:49Z

I couldn't get the server to work locally but it looked ok in vs code. Do you think that's enough?

solita-tommira

Excellent work. Left some, hopefully constructive 😄comments

solita-tommira · 2025-01-30T07:05:20Z

_posts/2025-01-22-elasticsearch-as-read-only-database.md

@@ -0,0 +1,139 @@
+---
+layout: post
+title: Elasticsearch as read only database


-> "Elasticsearch as a read database"?

Should "only" be dropped from the title and body? As it can't be wholly read only as you do index stuff into the Elastic index? 😊

solita-tommira · 2025-01-30T07:07:23Z

_posts/2025-01-22-elasticsearch-as-read-only-database.md

+title: Elasticsearch as read only database
+author: villephamalainen
+excerpt: >
+  Elasticsearch is a powerful, open-source search and analytics engine. It was initially released in 2010 and has since been adopted widely. Solita has a long history of using Elasticsearch and has found good use cases for it. One of them is using Elasticsearch as a read only database on top of traditional SQL database. This architecture has the benefit of modeling full domain relations to the SQL database and getting the lightning-fast searches from Elasticsearch where needed.


Would it be reasonable to change "read only" to just "read" in this excerpt as well? 😊

solita-tommira · 2025-01-30T07:12:46Z

_posts/2025-01-22-elasticsearch-as-read-only-database.md

+-	Full-Text Search: Optimized for fast and relevant full-text searches.
+-	Scalability: Horizontally scalable with support for data sharding and replication.
+-	Real-Time Analytics: Capable of performing complex queries and aggregations in near real-time.
+-	Versatility: Supports a wide range of use cases, including log and event data analysis, application monitoring, and security analytics


I would perhaps mention as a usage as the "primary search engine" as I would consider that to be the primary use case for ES

I don't really understand how is that different from the first bullet point. Could you elaborate?

Discussed via Slack 😊

solita-tommira · 2025-01-30T07:14:02Z

_posts/2025-01-22-elasticsearch-as-read-only-database.md

+SQL Server: Optimized for transaction processing and complex queries on structured data, providing robust performance for OLTP workloads.
+
+**Use Cases**<br>
+Elasticsearch: Best suited for log and event data analysis, real-time application monitoring, and security analytics.<br>


"...suited for full text search,...? 😊

solita-tommira · 2025-01-30T07:16:03Z

_posts/2025-01-22-elasticsearch-as-read-only-database.md

+## Architecture
+
+Now that we have covered the basics let’s get back to my use case.
+In many applications it is typical to use just one SQL database for everything. This is a very simple setup, and it works well in many cases, especially with small applications. When the application userbase gets a little bigger or the application uses search heavily this setup might start to slow down. In these cases, it might be a good time to start thinking about different approaches.


Would it be good just to say "When the application gets bigger..." as applications can get bigger in a multitude of ways just as code base getting bigger, users getting bigger, getting more data, etc.

solita-tommira · 2025-01-30T11:23:41Z

_posts/2025-01-22-elasticsearch-as-read-only-database.md

+
+## Test setup
+
+I created a new ASP .NET Core 8 project with connections to local SQL Server and Elasticsearch (using docker). The code can be found [here](https://github.com/solita/sqlserver-vs-elasticsearch). The project contains a very simple test entity with just 3 properties which are written and read from the database. All the read tests were done with 100k entities in the database. Write tests were done to empty database.


Was SQL Server running also in Docker? If not, then the comparison is not really fair because Docker adds some performance overhead. It might be worth mentioning here at least 🤔

solita-tommira · 2025-01-30T11:25:21Z

_posts/2025-01-22-elasticsearch-as-read-only-database.md

+[![Figure 4: Write CPU comparison.](/img/2025-elasticsearch-as-read-only-database/elasticsearch-vs-sqlserver-cpu.png)](/img/2025-elasticsearch-as-read-only-database/elasticsearch-vs-sqlserver-cpu.png)<br>
+*Figure 4: Write CPU comparison.*
+
+As seen from the CPU comparison Elasticsearch uses multiple cores when writing and SQL server does not. Writing speed might be more even with a single core instance.


Interesting 👍

solita-tommira · 2025-01-30T11:34:12Z

_posts/2025-01-22-elasticsearch-as-read-only-database.md

+
+Applying architecture with Elasticsearch as read only database does wonders for performance in some areas. But it does have its drawbacks as well. The databases need to be kept in synchronization with each other. There is also the cost of development time and complexity of this type of architecture.
+
+Last, I would like to offer some use cases where we have found Elasticsearch to be worth it. Obviously if you need to search from large amounts of data, Elasticsearch is really good. One use case we stumbled into is real time dashboards. It might be quite taxing for SQL database to offer this, and caching is a bit problematic if the data changes frequently. But Elasticsearch seems to be a very good solution for that.


Is stumbled the correct wording here? 😄 It's sort of natural that ES is good on real time dashboards as it's an analytics engine as well. 👍

We did kinda stumble on it in the project. 😄

But you are correct that it's quite clear from the documentation. I'll rephrase it.

solita-tommira · 2025-01-30T11:39:10Z

_posts/2025-01-22-elasticsearch-as-read-only-database.md

+
+## Test setup
+
+I created a new ASP .NET Core 8 project with connections to local SQL Server and Elasticsearch (using docker). The code can be found [here](https://github.com/solita/sqlserver-vs-elasticsearch). The project contains a very simple test entity with just 3 properties which are written and read from the database. All the read tests were done with 100k entities in the database. Write tests were done to empty database.


What were the ES and SQL Server versions?

solita-tommira · 2025-01-30T11:45:37Z

_posts/2025-01-22-elasticsearch-as-read-only-database.md

+
+Tests included:
+-	Read one entity
+-	Search with string


Did you configure SQL server for full text search? https://learn.microsoft.com/en-us/sql/relational-databases/search/full-text-search? It might be worth mentioning whether or not 🤔

- Change title from read-only database to read database - Other minor tweaks to text

solita-villep-hamalainen added 3 commits January 22, 2025 12:08

Add Elasticsearch as read only database post

ecdb4ad

Update post date

3908428

Add links to Elastic stack

aa651a2

solita-tommira reviewed Jan 31, 2025

View reviewed changes

tatut approved these changes Jan 31, 2025

View reviewed changes

solita-villep-hamalainen added 4 commits January 31, 2025 15:51

Multiple small changes

f0aa109

- Change title from read-only database to read database - Other minor tweaks to text

Update date and add new key feature to Elastic

c3b853c

Add chapter about tests no being very extensive

ebf849b

Correct spelling mistakes

fdf1a6a

tatut merged commit f1bfd6a into solita:master Feb 3, 2025
1 check passed


		## Test setup

		I created a new ASP .NET Core 8 project with connections to local SQL Server and Elasticsearch (using docker). The code can be found [here](https://github.com/solita/sqlserver-vs-elasticsearch). The project contains a very simple test entity with just 3 properties which are written and read from the database. All the read tests were done with 100k entities in the database. Write tests were done to empty database.


		Applying architecture with Elasticsearch as read only database does wonders for performance in some areas. But it does have its drawbacks as well. The databases need to be kept in synchronization with each other. There is also the cost of development time and complexity of this type of architecture.

		Last, I would like to offer some use cases where we have found Elasticsearch to be worth it. Obviously if you need to search from large amounts of data, Elasticsearch is really good. One use case we stumbled into is real time dashboards. It might be quite taxing for SQL database to offer this, and caching is a bit problematic if the data changes frequently. But Elasticsearch seems to be a very good solution for that.

Add Elasticsearch as read only database post #546

Add Elasticsearch as read only database post #546

Uh oh!

Conversation

solita-villep-hamalainen commented Jan 22, 2025

Uh oh!

solita-tommira left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

solita-tommira Feb 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

solita-tommira Feb 3, 2025 •

edited

Loading