ElasticSearch - Preventing Split Brain With a Cluster!

Elasticsearch - Brain Cluster !

After reading this about article about elastic search cluster --Split - brain problem I wanted to be careful about setting an elastic search cluster.

 I have to setup a 3 node cluster of elastic search. After reading a few articles about it, it comes down to how the cluster will be used mostly. Here are some of the questions that i asked when setting the cluster up. I hope that this can be a good starting point for anybody setting up elastic search cluster.

1. How many nodes are you going to use for the cluster ?
   We only have 3 physical servers that is why i decided to use a 3 node cluster. 
     Here are the settings that i used
         Gave the cluster a name [cluster.name: brain] - brain cluster
          Gave each node a name [node.name: "brain_1"']
                                               [node.name: "brain_2"'] 
                                               [node.name: "brain_3"']
        a master node [node.master: true]
        a data node [node.data: true]
        http enabled
     I decided all nodes to be master and data node because I only have 3 servers. If I have more 10-12 servers I might have employed a different strategy. I would have done at least 3 master nodes. 3-8 data nodes and 3-4 exclusive search nodes (not a master node nor a data node)

 2. Are you going to be doing more searching or indexing ? How often are you going to index ?
   - This will help determine if a tribal node is going to be good for you because you might encounter some issues indexing a tribal node specially when your index names are the same across clusters. Here is a link that explains how Tribe node works for elasticsearch
Tribe Node Elasticsearch

 3. Do you want the nodes to auto discover each other ?
    Elastic search uses zen discovery to automatically cluster any elastic search instances running in the same network. I have to deal with a head splitting issue while indexing but as it turns out another developer is running an elastic search instance using the default settings and our instances automatically clustered together.
To turn off multicast discovery set this configuration attribute to false
[discovery.zen.ping.multicast.enabled: false]
and if you want your instance to just try and cluster with certain servers you can define this attribute
discovery.zen.ping.unicast.hosts: ["<hostname>:<port>"] - where is the hostname of the other you want elastic search instance that you want this current instance to cluster with. <port> is the [transport.tcp.port] setting on the other server NOT the http.port


 After going through this exercise I have my clusters working. I installed marvel and wikipedia river on all my instances

 bin/plugin -install elasticsearch/marvel/latest

 bin/plugin -install elasticsearch/elasticsearch-river-wikipedia/2.0.0

 After installing the river- I have to run this in Sense against one of the master nodes.

 Start pulling data to a new my_wikipedia_river index
PUT /_river/my_wikipedia_river/_meta
{
   "type" : "wikipedia"
}

Search it
GET /my_wikipedia_river/_search

POST /my_wikipedia_river/_search
{
 "query": {
   "term": {
     "text": "computer"
   }
 }
}

Stop pulling data
DELETE /_river/my_wikipedia_river/

 This was so easy to setup.
Here is a screen capture of marvel after setting all up. Everything is green!


Comments

Post a Comment

Popular posts from this blog

OAuth 1.0a Request Signing and Verification - HMAC-SHA1 - HMAC-SHA256

Image
This is a security project that I did this Fall semester. I was interested in why the OAuth1.0a specification did not require you to sign a request using RSA-SHA1 or any RSA algorithm. The first issue that I found is where would you store your private keys in a javascript client? - as a javascript encrypted file ? from the browser ?  - that will be a security hole if you allow javascript to get private keys. These are some of the questions that got me asking and which I cannot find answers to for now. OAuth 1.0a Signature Signing and Verification using JavaScript Implementation Background The OAuth specification tries to define an API authentication method that allows consumer websites or applications to access protected API Resources from a Service provider. The advantage that OAuth provides is that the user of the website or application is not required to disclose their credentials to their service provider. Another advantage of using OAuth to authenticate is that it does not
Read more

Spark DataFrame - Array[ByteBuffer] - IllegalAurmentException

IllegalArgumentException - ByteBuffer - Spark DataFrame I was processing a several million documents (~ 20 million) in which we need to extract the NLP features using NLP4J, OpenNLP, and WordNet. The combination of the three NL features blows up each record to 11 times its original size. We are using all three because we do not know yet what feature sets will be helpful to us. The original dataset is in parquet files in HDFS (16 partitions). I thought that was convenient just use withColumn and pass a UDF (User Defined Function) on the column where it needs those features. withColumn adds the calculated column back to the DataFrame. So I created the spark job (I am on Spark 1.5.2-cdh5.5.2)for the above, and things started to get nasty. I am blowing up the ByteBuffer array on the in-memory columnar storage. This is the exception that I am getting. There seems to be no reference in my code in this stack trace. java.lang.IllegalArgumentException at java.nio.ByteB
Read more

Gensim Doc2Vec on Spark - a quest to get the right Vector

Ever since I joined the R & D group we have been doing a lot of cool things, like trying IBM Watson (see previous blog entry). Now we are doing a lot of Natural language processing.  We wanted to compare the similarity of two documents. There is this excellent project Gensim ( doc2vec ) that easily allow to you to translate large blocks of text to a fixed length feature vector to make comparisons. Here is the link to the  original paper  from some people from Google that explains the approach. In essence, they wanted to find a representation that will overcome the weaknesses of the bag of words model. The doc2vec approach proves to be a reliable approach to comparing similarities of documents because it takes into consideration the semantics and the order of the word in context. So with that, we wanted to use it for a corpus of 26 million documents. Calculating the doc2vec for 26 million documents is not a small task, so we need to process it in Spark. The problem is that there
Read more