Messaging for the Internet of Things

Messaging for the Internet of Things  


As we embed processing power to a lot of our everyday things. The thing will need to communicate with a server to send data or get data . The thing needs data in order to make something happen. The thing needs to send data so that we can be smarter on how we use the thing. But How do we send data to a central server that is robust, asynchronous and reliable ?

MQ, what is old is new Again 


MQ or Message Qeueing is new again in the age of the Internet of Things. Messaging Patterns for the enterprise has been around for ages. What were the major reasons that messaging addresses ?
1. Networks are unreliable - data needs to be transported over the network. These networks are a conglemeration of different types of network. Since we do not have control of over most of the network that we use to access the internet. The network will present a lot of problem specially when you are sending messages across the internet.
2. Networks are slow - The network that we are using to use the internet is mostly aging. Again, not having control how some of the networks that we might use to send data over the internet it can have performance implications.
3.Application are different - Using messaging any application can listen or subscribe to the message as long as it can be parsed or consumed by the application subscribing to that message.
4. Change is inevitable - in the internet of things, we can put more processing power in more things. There will probably be a need to add more devices that will send data over time. We might need to change the servers where it send data to as well as it gets updated.

These are the same issues that the Internet of Things will encounter.


Messaging Patterns 

Since messaging allows high-speed, asynchronous, program to program communication with reliable delivery it is only but natural to be adopted in the internet of things. But why use messaging for the Internet of Things ?
1. Asynchronous communication - Messaging enables a send and forget approach to communication. Since most of the connected devices would be somewhere remote. These devices does not have to wait for the receiver to receive the message. 2. Throttling - The issue with synchronous communication or call to a server like HTTP. A lot of connected devices can
overload the receiver which can cause performance degradation on the server. Messaging will help control the rate at which the receiver can consume the requests.
3. Disconnected operation - most of the time an internet of thing device will be used in a low-power autonomous setting. Low power consumption can be achieved by not doing a lot of communication with the server. During those time that the device tries to communicate and the network is not available it can queue up the messages and synchronize when the network becomes available.

Asynchronous Communication

Since messaging using MQ can be done asynchronously. A different kind of approach is required for designing applications like the use of callbacks and other result notification patterns.

Since messaging has been around for a while we can adapt certain patterns that have been used in the enterprise to the internet of things. Sample patterns [for a complete list please go to website in the reference at the end of this article]:
1. Guaranteed Delivery - make messaging persistent so that they are not lost if the messaging system crashes. The messages are stored locally until it can be successfully delivered.
2. Datatype Channel - provide a separate channels for each type of message. The sender knowing what message to send can choose the appropriate channel to send it thru. The receiver knowing what channel the data was received from will know the type of message it receives.
3. Publish-Subscribe Channel - delivers a copy of the message to multiple subscribers.

Conclusion:

Messaging systems like MQTT will be an essential infrastructure to the internet of things. We should learn from the patterns that we learned from creating enterprise software that leverages messaging and apply the experience when rolling out devices for the internet of things.


Reference:
1[http://www.enterpriseintegrationpatterns.com/patterns/messaging/Introductio
n.html]




Comments

Post a Comment

Popular posts from this blog

OAuth 1.0a Request Signing and Verification - HMAC-SHA1 - HMAC-SHA256

Image
This is a security project that I did this Fall semester. I was interested in why the OAuth1.0a specification did not require you to sign a request using RSA-SHA1 or any RSA algorithm. The first issue that I found is where would you store your private keys in a javascript client? - as a javascript encrypted file ? from the browser ?  - that will be a security hole if you allow javascript to get private keys. These are some of the questions that got me asking and which I cannot find answers to for now. OAuth 1.0a Signature Signing and Verification using JavaScript Implementation Background The OAuth specification tries to define an API authentication method that allows consumer websites or applications to access protected API Resources from a Service provider. The advantage that OAuth provides is that the user of the website or application is not required to disclose their credentials to their service provider. Another advantage of using OAuth to authenticate is that it does not
Read more

Spark DataFrame - Array[ByteBuffer] - IllegalAurmentException

IllegalArgumentException - ByteBuffer - Spark DataFrame I was processing a several million documents (~ 20 million) in which we need to extract the NLP features using NLP4J, OpenNLP, and WordNet. The combination of the three NL features blows up each record to 11 times its original size. We are using all three because we do not know yet what feature sets will be helpful to us. The original dataset is in parquet files in HDFS (16 partitions). I thought that was convenient just use withColumn and pass a UDF (User Defined Function) on the column where it needs those features. withColumn adds the calculated column back to the DataFrame. So I created the spark job (I am on Spark 1.5.2-cdh5.5.2)for the above, and things started to get nasty. I am blowing up the ByteBuffer array on the in-memory columnar storage. This is the exception that I am getting. There seems to be no reference in my code in this stack trace. java.lang.IllegalArgumentException at java.nio.ByteB
Read more

Gensim Doc2Vec on Spark - a quest to get the right Vector

Ever since I joined the R & D group we have been doing a lot of cool things, like trying IBM Watson (see previous blog entry). Now we are doing a lot of Natural language processing.  We wanted to compare the similarity of two documents. There is this excellent project Gensim ( doc2vec ) that easily allow to you to translate large blocks of text to a fixed length feature vector to make comparisons. Here is the link to the  original paper  from some people from Google that explains the approach. In essence, they wanted to find a representation that will overcome the weaknesses of the bag of words model. The doc2vec approach proves to be a reliable approach to comparing similarities of documents because it takes into consideration the semantics and the order of the word in context. So with that, we wanted to use it for a corpus of 26 million documents. Calculating the doc2vec for 26 million documents is not a small task, so we need to process it in Spark. The problem is that there
Read more