ElasticSearch - Sniff ! Sniff!

Note: Below is a brief discussion of the issues i have encountered using Elasticsearch's Client behind a VIP.

 As I had written previously, I had to setup an Elasticsearch cluster. In order to take advantage of balancing the load between the nodes of the cluster. I used a DNS pointed to VIP that does a round-robin load balancing among the nodes. The reason i choose this is because I can add servers to the VIP without changing the Transport Addresses in the Elasticsearch Client java code which uses a TransportClient. Without the VIP I need a coding change and deploy to add the new Transport Address.

 The cluster would not be really useful if I cannot take away nodes and add nodes to the cluster as I wish without any coding change.

 The drawback in hiding the cluster behind the VIP is that you don't know which node the java code client is going to get in order to process a search request. 

 This is causing an issue in the client code.  Since I only have one Transport Address defined in my TransportClient it is throwing a NoNodeAvailableException when the TransportClient hits the only TransportAddress i have added to it and this server's request queue is filled up.

 The TransportClient is not forwarding or using the other nodes to process the request. The reason for this is that if there is only one TransportAddress in the TransportClient a second attempt to reconnect or retry the request will throw the exception (NoNodeAvailableException).  In order for the TransportClient forward the request to any node of the cluster you can do one of the following things:

 1. Add all the Transport addresses for each of the nodes to the TransportClient.
     - TransportClient uses a SimpleNodeSampler that basically gets the list of TransportAddress added to the TransportClient and try to forward the request to any of the transport Addresses. (BUT this does not work for my setup because it defeats the purpose of my cluster hiding  behind a VIP - enter the sniff) 

 2. Set the"client.transport.sniff" property in the TransportClient Settings equal to true 
          Sample Settings:
          Settings elasticSettings = ImmutableSettings.settingsBuilder()
                .put("cluster.name", this.clusterName)
                .put("client.transport.sniff", true)
                .put("client.transport.ping_timeout", "30s")
                .put("client.transport.nodes_sampler_interval", "10s")
                .build();

      When this property is set to true, the TransportClient will try and sniff out the rest of the nodes' addresses so that it can be used to forward request to without explicitly adding the addresses to a TransportClient.

 Tip: As you can see above I have set the setting of the ping timeout and nodes sample interval. This is to make sure that the TransportClient is going to be able to forward request to the nodes properly.

Comments

Post a Comment

Popular posts from this blog

OAuth 1.0a Request Signing and Verification - HMAC-SHA1 - HMAC-SHA256

Image
This is a security project that I did this Fall semester. I was interested in why the OAuth1.0a specification did not require you to sign a request using RSA-SHA1 or any RSA algorithm. The first issue that I found is where would you store your private keys in a javascript client? - as a javascript encrypted file ? from the browser ?  - that will be a security hole if you allow javascript to get private keys. These are some of the questions that got me asking and which I cannot find answers to for now. OAuth 1.0a Signature Signing and Verification using JavaScript Implementation Background The OAuth specification tries to define an API authentication method that allows consumer websites or applications to access protected API Resources from a Service provider. The advantage that OAuth provides is that the user of the website or application is not required to disclose their credentials to their service provider. Another advantage of using OAuth to authenticate is that it does not
Read more

Spark DataFrame - Array[ByteBuffer] - IllegalAurmentException

IllegalArgumentException - ByteBuffer - Spark DataFrame I was processing a several million documents (~ 20 million) in which we need to extract the NLP features using NLP4J, OpenNLP, and WordNet. The combination of the three NL features blows up each record to 11 times its original size. We are using all three because we do not know yet what feature sets will be helpful to us. The original dataset is in parquet files in HDFS (16 partitions). I thought that was convenient just use withColumn and pass a UDF (User Defined Function) on the column where it needs those features. withColumn adds the calculated column back to the DataFrame. So I created the spark job (I am on Spark 1.5.2-cdh5.5.2)for the above, and things started to get nasty. I am blowing up the ByteBuffer array on the in-memory columnar storage. This is the exception that I am getting. There seems to be no reference in my code in this stack trace. java.lang.IllegalArgumentException at java.nio.ByteB
Read more

Gensim Doc2Vec on Spark - a quest to get the right Vector

Ever since I joined the R & D group we have been doing a lot of cool things, like trying IBM Watson (see previous blog entry). Now we are doing a lot of Natural language processing.  We wanted to compare the similarity of two documents. There is this excellent project Gensim ( doc2vec ) that easily allow to you to translate large blocks of text to a fixed length feature vector to make comparisons. Here is the link to the  original paper  from some people from Google that explains the approach. In essence, they wanted to find a representation that will overcome the weaknesses of the bag of words model. The doc2vec approach proves to be a reliable approach to comparing similarities of documents because it takes into consideration the semantics and the order of the word in context. So with that, we wanted to use it for a corpus of 26 million documents. Calculating the doc2vec for 26 million documents is not a small task, so we need to process it in Spark. The problem is that there
Read more