In February of this year, Elastic launched ECS (Elastic Common Schema) and since it was introduced to the public we’ve been finding it extremely useful for a large variety of use cases such as infrastructure monitoring, security analytics, APM or AIOps. But first, let’s see what is ECS.
Elastic defines ECS as “ an open source specification that defines a common set of document fields for data ingested into Elasticsearch”. Let’s analyze their definition a bit.
The fact that ECS is open source is no surprise since Elastic has been an open source project from its beginnings. This means that anyone can check out ECS without having to pay anything in return. The second part of the definition is the really important one, however.
ECS is a specification and we need to stress this fact. Anyone expecting ECS to be something like an Elasticsearch plugin that you just install on all your nodes and it’s up and running will have a surprise. The common schema which Elastic suggests is a common set of guidelines that can (but are NOT required) be used when defining fields and field names for data ingested into Elasticsearch. And this brings us to the third part of the definition.
ECS only applies to field types and field names without having any impact on the raw data, meaning that your data is not changed it is just organized differently. So what’s the big deal then? Having a common set of rules for defining field types and names opens up a lot of opportunities when it comes to data analysis, team collaboration, scalability, or machine learning.
Below are some of the advantages that we found while using ECS.
- Event correlation: one of the important aspects when using Elasticsearch to analyze data is the incredible speed with which it can retrieve data. By having near-real-time search capabilities we can easily interact with dashboards and visuals to quickly drill down to granular events. Now imagine that you also have the same field names across multiple data sets and that those data sets are organized into visuals into a single dashboard. Suddenly you can apply a single filter such as user.name : Dave and you can see Dave’s activity everywhere in the infrastructure whether it’s network activity, file access, applications used, and so on. With ECS in place, you can now correlate events across multiple data sets with a single click.
- Machine learning: when creating machine learning jobs using the Elastic Stack the results are stored back into Elasticsearch and they can be visualized and analyzed in dashboards and visuals. With ECS in place the machine learning results are even more relevant because they can be correlated across multiple machine learning jobs. Imagine discovering an anomaly with regards to accessing one of your sensible files. Now also imagine then when you filter for events connected to that file you also identify an anomaly regarding the time of day in which an admin account logged into the file server. Suddenly it is very easy to correlate anomalies across multiple jobs.
- Collaboration: when it comes to solving issues in the IT world, collaboration between teams is critical. At the same time, we all know that each environment has its own terminology and every log entry is in a different format and each element of those environments and logs have different names. This usually leads to prolonged meetings and long email chains in which each team tries to make sense of what the other team is saying. With ECS however, all the logs and all the terminology is translated into a common set of fields thus enabling all IT teams to speak the same language and have an easier time collaborating on various projects and incidents.
- Efficiency: leveraging all the data that infrastructures generate is essential for businesses nowadays and as such being efficient in the process makes things go a little faster. Making use of ECS all the saved objects from Kibana such as saved views, visuals or dashboards are easier to replicate for different data sets because they all make use of the same fields or make use of existing visuals and saved views for new projects.
ECS is no doubt a powerful tool and a welcome addition to the Elastic Stack which will enable users to discover more insights into their data.
Partner with an expert company like Arcanna.ai who can lead your business through the ECS process.
