ElasticSearch for .NET Developers

For the past year I have been evaluating and working and even presented ElasticSearch, and I thought it would be good to showcase a series of article on ElasticSearch for .NET Developers. What it brings to the table when developing a software solution. I also did a talk on ElasticSearch at Montreal DevTeach, if you are interested in my slides feel free to view them on slideshare or my blog.

Without further adieu, lets get started and lets look at what ElasticSearch really is.

First off, ElasticSearch some consider it as ELK Stack but for new branding they have been trying to call themselves Elastic Stack rather, although the ELK has been stuck with many people and google searches, but we from here on we will call it just Elastic Stack.

So what does the Elastic Stack consist of you may wonder?
Basically the Elastic Stack consist of ElasticSearch, Logstash and Kibana. Lets go through them individually so that we can understand what each component does and brings to a software solution.

ElasticSearch

This is the core main search engine or store that you use for storing your data, it is build in Java. It stores documents in json format and uses Lucene to index it, elastic search provides and builds metadata upon the index that was created by Lucene (Note: Lucene is build in Java, there is also a port of Lucene to .NET called NLucene)

Some people may think that ElasticSearch is a database that we store data into like mysql, postgres or mssql, but I would say Elastic is not really a database since there is no db file and does not have relationships like SQL. Its more like a NOSQL solution but not quite like mongodb either. The best thing to describe it, I would say is think of it as a Search Engine where you store documents in. I know its confusing at first but don’t worry it will come clear later or once you start playing around with it.

Logstash

Logstash is another module/component/service. You can use logstash without using ElasticSearch, the main functionality of Logstash is to get some input, filter it and output it somewhere, again the output does not need to be ElasticSearch but usually it is. An example of logstash could be I have IIS logs or Apache Logs I need to input them into logstash, and I would like to geo tag each of the IP address and store them into ElasticSearch or some database. Main idea of Logstash is (INPUT -> FILTER -> OUTPUT) simple. One more thing to note is Logstash is build with JRuby on the JVM and there are tones of open source plugins for Logstash that one can download, even to anonymized the data or encrypt etc before outputting the data.

Kibana

Kibana is the graphical user interface for ElasticSearch, it is used analyzing your data and for creating charts from ElasticSearch data. It is quite powerful, one can slice and dice many kinds of charts using Kibana.
Kibana is build with node.js and its a single page app (SPA) application.

Beats

Beats are basically light weight shippers of data. There are many types of beats, eg. filebeat is used for shipping file data (e.g apache.log) to ElasticSearch or Logstash. Winlogbeat allows one to ship windows events to ElasticSearch or Logstash, check out the beats offered by Elastic; you can also write your own beat using the ibbeat library, and not to mention that beats are actually written in GoLang. If you are interested in using Golang with VSCode check out the channel 9 video I did for golang and vscode.

So here we sum up the main components of Elastic Stack, I will go through each component individually in upcoming blog post, going through install process to configuration.