background preloader

ELK

Facebook Twitter

Elastic Search en production - Le Blog d'Eric Vidal. Ça y’est, j’ai mis mon premier cluster Elastic Search en production : plus d’un milliards de documents et chaque jour environ 500000 nouveaux documents ajoutés.

Elastic Search en production - Le Blog d'Eric Vidal

Ça tourne sur 3 nœuds identiques, chacun contient une instance Elastic Search et les applications l’alimentant et l’exploitant. Avoir plusieurs instances d’un même système facilite la montée en charge car le système supporte les “scale out” pour absorber plus de données ou plus de charge. Cluster best practices - Elasticsearch - Discuss the Elastic Stack.

ElasticSearch Cluster: Configuration & Best Practices. I was recently working on setting up an elasticsearch cluster with apache whirr.

ElasticSearch Cluster: Configuration & Best Practices

Setting up a cluster is one thing and running it is entirely different. Running a cluster is far more complex than setting one up. Things are no different for an elasticsearch cluster. There are several things one needs to be aware of and take care of. Some of such things with ES cluster are: Type of nodes in the cluster: Master, Data, Controller, Load BalancersPicking up the right configuration - Initial number of machinesNumber of shardsNumber of replicasNumber of indexesLoading initial data set into the cluster: how to make it fast? ElasticSearch Node Types Even though elasticsearch documentation talks about a Node, it is not very clear and helpful. Master and Data - This is the default setting. As you may have guessed, this is controlled by 2 values in the elasticsearch.yml file in config directory. node.data: true node.master: true Another important setting is http.enabled.

Getting Configuration Right. How many shards should Elasticsearch indexes have? One of the best features of Elasticsearch is that it comes with support for clustering built in.

How many shards should Elasticsearch indexes have?

However, because of this, when you set up an index, you're faced with an often intimidating question: just how many shards/replicas should my index have? Unfortunately, there's no one definitive answer to this question, but I'll give some guidance that should help finding an answer for your particular setup a bit easier.

What is a shard, anyways? 10 Things to Consider When Planning Your Elasticsearch Project. I am seeing a lot of interest in Elasticsearch from clients and colleagues.

10 Things to Consider When Planning Your Elasticsearch Project

Elasticsearch is an open source search engine that is commercially supported by a company called Elastic. It’s used for web search, log analysis, and big data analytics. You’ll often see it compared with Apache Solr. Both depend on Apache Lucene for low-level indexing and analysis. People like Elasticsearch because it is easy to install, scales out to hundreds of nodes with no additional software needed, and is easy to work with thanks to its built-in RESTful API. Multiple folks have asked me what they need to think about when leveraging Elasticsearch as part of their solution, so I thought I’d summarize those thoughts and share them here. 1. Centralized logging with an ELK stack (Elasticsearch-Logstash-Kibana) on Ubuntu. Update 22/12/2015 I’ve reviewed the book Learning ELK stack by Packt Publishing, it’s available online for 5$ only: I’ve recently setup an ELK stack in order to centralize the logs of many services in my company, and it’s just amazing !

Centralized logging with an ELK stack (Elasticsearch-Logstash-Kibana) on Ubuntu

I’ve used the following versions of the softwares on Ubuntu 12.04 (also works on Ubuntu 14.04): Elasticsearch 1.4.1Kibana 3.1.2Logstash 1.4.2Logstash-forwarder 0.3.1. Astuces et retour d’expérience sur Elasticsearch. La mise en place d’Elasticsearch comme énoncé dans l’article précédent nous a permis de creuser certains points essentiels d’Elasticsearch.

Astuces et retour d’expérience sur Elasticsearch

Voici donc un premier retour d’expérience avec la présentation d’astuces et des réponses à diverses questions que nous pouvons nous poser lorsque nous installons Elasticsearch pour la première fois. Le mapping Toujours créer un alias d’un index Si vous voulez pouvoir facilement modifier le mapping sans arrêt de service, il faut créer un alias sur un index et l’utiliser pour accéder à cet index. En effet, Il n’est pas possible de modifier un mapping à la volée (contrairement à ajouter des nouveaux champs, ou “upgrader” un champ en multi_field), pour cela, la façon de faire est la suivante : 1 – Créer un nouvel index “index2″ avec un même type mais un nouveau mapping associé 2 – Ré-indexer les données dans ce nouvel index (voir lien :

How To Set Up a Production Elasticsearch Cluster on Ubuntu 14.04. Introduction Elasticsearch is a popular open source search server that is used for real-time distributed search and analysis of data.

How To Set Up a Production Elasticsearch Cluster on Ubuntu 14.04

When used for anything other than development, Elasticsearch should be deployed across multiple servers as a cluster, for the best performance, stability, and scalability. This tutorial will show you how to install and configure a production Elasticsearch cluster on Ubuntu 14.04, in a cloud server environment. Learning Elasticsearch by Gary Gao. Data Searching Search Data:Search API & Query DSL How to query ElasticSearch using its Query DSL ?

Learning Elasticsearch by Gary Gao

How to use basic queries ? How to use compound queries ? How to filter your results and why it is important ? Osmose - Le blog Ebiznext - ElasticSearch : Architecture et Développement. ElasticSearch : Architecture et Développement Introduction ElasticSearch est un moteur de recherche Open Source (Apache 2).

Osmose - Le blog Ebiznext - ElasticSearch : Architecture et Développement

Il est basé sur la librairie Apache Lucene et masque la complexité de celle-ci. Les données sont indexées sous forme de documents. Highly Available ELK (Elasticsearch, Logstash and Kibana) Setup - Everything Should Be Virtual. Monitoring avec ELK – Windows et NXlog! On a vu comment dans un précédent article comment mettre en place la pile ELK pour recevoir nos logs.

Monitoring avec ELK – Windows et NXlog!

Maintenant, on va tenter de voir comment y balancer les événements produits par des machines sous windows. Je me suis inspiré des articles suivants : Au boulot! On commence par ajouter un nouvel input à logstash toutes les confs de logstash se gèrent dans /etc/logstash/conf.d/ : Using NXLog with Elasticsearch and Kibana. Elasticsearch Tutorial & Getting Started (course preview)

Analyse des logs applicatifs avec Logstash, Kibana et Elasticsearch.