Elastic Search : Basic concepts and how to do setup

About Elastic search:

-Near real time search platform

-Each index can be horizontally splitted into multiple shards.

Each shards can have own replicas.

Replication helps in

-High availability

-Scale out search /throughput, as  search can be executed on replicas in parallel

Steps to setup elastic search:

1. Adding Schema:

PUT / INDEX_NAME

"mappings": {

         "Index_name": {

}

,

      "settings": {

         "index": {

            "number_of_replicas": "1",

            "number_of_shards": "1"

         }

      }

   }

2. Add Domain Schema (equivalent to table)

PUT /index_name

{

      "mappings": {

         "index_type": {

            "properties": {

               "firstName": {

                  "type": "string"

               },

 "secondName": {

                  "type": "string"

               },

 "location": {

                  "type": "string"

                  "index": "not_analyzed"

               }

}

}

3.Add data

PUT index_name/index_type/documentId1

{

                     "firstName" :" hello",

                     "secondName": "world"

                           "location"": "Malesia-Singapore"

}

GET INDEX_NAME/ INDEX_TYPE/Id

PUT INDEX_NAME / INDEX_TYPE /1

{

   "body": "here"

}

DELETE index_name/index_type/<id>

   Query search  :           

  GET INDEX_NAME 

  {

  "query" : {

    <json_query>

  }

}

All search:

GET INDEX_NAME/INDEX_TYPE/_search{

GET _search

{

  "query": {

    "match_all": {}

  }

}

Executing term query

get index_name/index_type/_search

{

  "query": {

    "term": {

      "location": "Malesia-Singapore"

    }

  }

}

Springboot User doc

Running in debug mode:

export MAVEN_OPTS="-Xms2g -Xmx2g -XX:MaxPermSize=2g -Duser.timezone=IST -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=8091,server=y,suspend=n"

Quick Gemfire Setup

In this blog, we will walk through Gemfire setup and issues encountered  in running on production.

Steps to install in local sandbox:

-Download Pivotal_GemFire_820_b17919_Linux

-Search for any process name with locator running , kill it

-Start gfsh client

  gfsh

-Add locator

start locator --name=locatorTest

Configure pdx serialization rule

configure pdx --auto-serializable-classes=com\.thetechiehouse\.blogspot\.services\.sampleapp\.vo\..* --read-serialized=true --disk-store
Start server:

start server --name=server1 --classpath=<absolutepath>sample-app-service.jar --server-port=40411

Create a replicated, persistent region:

create region --name=region1 --type=PARTITION_PERSISTENT

List regions:

list regions

To view specific about regions:
describe region --name= region1

List members of your cluster:
list members

-Start gemfire pulse monitoring tool:
start pulse

Issue debugging on Production :

Important commands:

-show missing-disk-storeslist members

-list regions

-query --query= “select * from /regionForSampleService”

Issue:

Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined.

Exception in thread "main" java.lang.NullPointerException

    at com.gemstone.gemfire.internal.InternalDataSerializer.readPdxSerializable(InternalDataSerializer.java:3157)

    at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2979)

    at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3210)

Solution :-Delete all locator and server related folders, if you are already having.

More:http://gemfire.docs.pivotal.io/gemfire/getting_started/15_minute_quickstart_gfsh.html

Multi threaded Ping Service

Let's create a service that keeps on pinging multiple services and give their status like live,down and so on..

public class PingService {

private static String url1 = "http://localhost:8082";

private static String url2 = "http://localhost:8083";

private static String url3 = "http://localhost:8084";

public List<ServerStatusVO> collectStatus() {

List<ServiceInfo> serviceList = getRunningServiceList();

List<ServerStatusVO> statusList = new ArrayList<>();

ScheduledExecutorService execService = Executors.newScheduledThreadPool(3);

List<Callable<ServerStatusVO>> callableList = new ArrayList<Callable<ServerStatusVO>>();

for (ServiceInfo service : serviceList) {

callableList.add(new PingHost(service.getId(), service.getUrl()));

}

try {

List<Future<ServerStatusVO>> resultFuture = execService.invokeAll(callableList);

for (Future<ServerStatusVO> future : resultFuture) {

try {

statusList.add(future.get());

} catch (ExecutionException e) {

e.printStackTrace();

}

}

} catch (InterruptedException e1) {

e1.printStackTrace();

}

execService.shutdown();

try {

if (execService.awaitTermination(10, TimeUnit.SECONDS)) {

System.out.println("All threads done with their jobs");

}

} catch (InterruptedException e) {

e.printStackTrace();

}

System.out.println("Final Sum : ");

return statusList;

}

class PingHost implements Callable<ServerStatusVO> {

RestTemplate restTemplate = new RestTemplate();

String id;

String host;

public PingHost(String id, String host) {

this.host = host;

this.id = id;

}

@Override

public ServerStatusVO call() throws Exception {

ServerStatusVO serverStatusVO = new ServerStatusVO();

serverStatusVO.setUrl(host);

serverStatusVO.setId(id);

try {

String widget = restTemplate.getForObject(host, String.class);

if (StringUtils.isBlank(widget)) {

serverStatusVO.setStatus("success");

System.out.println("success......");

} else {

serverStatusVO.setStatus("success");

}

} catch (Exception e) {

serverStatusVO.setStatus("failure");

System.out.println("failure......" + e);

}

return serverStatusVO;

}

}

private List<ServiceInfo> getRunningServiceList() {

List<String> hostList = new ArrayList<>();

hostList.add(url1);

hostList.add(url2);

hostList.add(url3);

List<ServiceInfo> serviceList = new ArrayList<>();

int i = 1;

for (String service : hostList) {

serviceList.add(new ServiceInfo(i + "", service));

i++;

}

return serviceList;

}

}

Websocket integration with Angular js -II

In the previous blog we created a server which serves web socket based service, here we angular js based client will establish connection and show live status in the form of service dashboard.

<!DOCTYPE html>

<html>

<head>

  <title>Service Dashboard application</title>

  <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">

  <link href="bootstrap-combined.no-icons.min.css" rel="stylesheet">

  <script type="text/javascript" src="jquery.min.js"></script>

  <script src="angular.js"></script>

<link rel="stylesheet" href="ng-table.min.css">

<script src="ng-table.js"></script>

</head>

<body>

  <div class="container">

    <h1>Service status Live</h1>

    <hr>

      Service status : <span id="message"></span>

    <hr>

    <h3>Running Services</h3>

  <div ng-app="serviceApp" > 

    <div id="outer" ng-controller="customersCtrl">

 <div class="row">

        <div class="span4">

        

<table class="table table-striped">

  <thead class="thead-inverse">

    <tr>

      <th>#</th>

      <th>Server</th>

      <th>Status</th>

    </tr>

  </thead>

  

<tr ng-repeat="user in customers.names track by $index" >

       <td>{{user.id}}</td>

    <td>{{user.url}}</td>

        <td>{{user.status}}</td>

    </tr>

    

          </table>

        </div>

</div>

</div>

  <script>

      var $message = $('#message');

   var app = angular.module('serviceApp', []);

   app.controller('customersCtrl', function ($scope, $rootScope, websocketService) {

      $scope.customers= {};

     websocketService.start("ws://localhost:8888/ws", function (evt) {

        var obj = JSON.parse(evt.data);

         $rootScope.$apply(function () {

         console.log(typeof obj);       

     $scope.customers.names =JSON.parse(obj);;

        });

    });

});

app.factory('websocketService', function () {

        return {

            start: function (url, callback) {

                var websocket = new WebSocket(url);

                websocket.onopen = function () {

                 $message.attr("class", 'label label-success');

      $message.text('open');

                };

                websocket.onclose = function () {

                 $message.attr("class", 'label label-important');

      $message.text('closed');

                };

                websocket.onmessage = function (evt) {

                 $message.attr("class", 'label label-info');

      $message.hide();

      $message.fadeIn("slow");

     $message.text('recieved message');

                    callback(evt);

                };

                 websocket.onerror = function(ev){

      $message.attr("class", 'label label-warning');

      $message.text('error occurred');

    };

            }

        }

    }

);

</script>

</body>

</html>

Websocket integration with Rest API : sample application -I

Websocket is an independent TCP based protocol. Web socket establishes a persistent connection between client and server, both can send data at  any time.

Server side implementation responsible for keeping large number of connections open, with high concurrency and low performance cost.Mostly based on non blocking  IO or threading. 

In this blog, we will go though Tornado, a Python implementation  for web socket connections.

In this example , we create a websocket server which keeps pinging a rest api at fixed interval and it will push the response whoever is connected with.

from tornado import websocket, web, ioloop, gen

import json

import urllib

import urllib2

import threading

cl = []

class IndexHandler(web.RequestHandler):

    def get(self):

        self.render("index.html")

class SocketHandler(websocket.WebSocketHandler):

    def check_origin(self, origin):

        return True

    def doWork(self):

        print "in  f()..."

        url = 'http://localhost:8080/rest/v1/echo/11?'

        username = 'admin'

        password = 'admin'

        p = urllib2.HTTPPasswordMgrWithDefaultRealm()

        p.add_password(None, url, username, password)

        handler = urllib2.HTTPBasicAuthHandler(p)

        opener = urllib2.build_opener(handler)

        urllib2.install_opener(opener)

        value = urllib2.urlopen(url).read()

       

        data = json.dumps(value)

        for c in cl:

            c.write_message(data)

        # call doWork() again in 230 seconds

        threading.Timer(230, self.doWork).start()

    def open(self):

        if self not in cl:

            cl.append(self)

        print "in  self()..."

        self.f();

        

    def on_close(self):

        if self in cl:

            cl.remove(self)

            

    def synchronous_fetch():

        print "Doing stuff..."

        url = 'http://localhost:8080/rest/v1/pingserver'

        http_client = HTTPClient()

        response = http_client.fetch(url)

        return response.body

    

class ApiHandler(web.RequestHandler):

#approach II

    @web.asynchronous

    def get(self, *args):

        self.finish()

        url = 'http://localhost:8080/rest/v1/pingserver'

        value = urllib2.urlopen(url).read()

        id = self.get_argument("id")

        value=response

        data = {"id": id, "value" : value}

        data = json.dumps(data)

        for c in cl:

            c.write_message(data)

    @web.asynchronous

    def post(self):

        pass

app = web.Application([

    (r'/', IndexHandler),

    (r'/ws', SocketHandler),

    (r'/api', ApiHandler),

    (r'/(favicon.ico)', web.StaticFileHandler, {'path': '../'}),

    (r'/(web_socket_rest_api.png)', web.StaticFileHandler, {'path': './'}),

])

if __name__ == '__main__':

    app.listen(8888)

    ioloop.IOLoop.instance().start()

Node Tools

node : single queue per node enviornment
process: gives node configuration
Event loop :

Loupe:http://latentflip.com/loupe

Most useful Opensource Tools

Diagram:
-creately
-lucidchart
-sequencediagram.com
-googledrawing

Injection Proxy
 - Vaurien

RabbitMQ : Most Useful Commands

brew install rabbitmq

Admin: rabbitmq-plugins enable rabbitmq_management

------------------------------------------------------------------------

Start rabbitmq server

rabbitmq-server

Start rabbitmq application

rabbitmqctl start_app

Delete an existing user

rabbitmqctl delete_user vhostusersampleservice

Delete a virtual host  

rabbitmqctl delete_vhost vhostsampleservice

Add a virtual host  

rabbitmqctl add_vhost vhostsampleservice

Add a user

rabbitmqctl add_user vhostuser vhostpassword

Assign tags

rabbitmqctl  set_user_tags vhostuser administrator

Giving permissions:
 For all hosts /

rabbitmqctl set_permissions -p / vhostuser ".*" ".*" ".*”

For host /vhostsampleservice
rabbitmqctl set_permissions -p /vhostsampleservice vhostuser ".*" ".*" ".*”

Enable Management plugin:
For managing nodes and cluseters
rabbitmq-plugins enable rabbitmq_management
-->http://localhost:15672/#/

Mobile appliction using PhoneGap

We have talked a lot about mobile development, here this article will walk us through the basics of Corodova formarly known as PhoneGap.

Originally developed by Adobe, is a development framework that works for all mobile OS. It’s an open source framework, once the code is written with this , can run on any Os like iOS, android,blackberry and windows mobile.

Phonegap is powered by the Corodova engine.

The good part here is the program just need to know the basic of web application development and don’t require mobile specific programming. PhoneGap works like a bridge between the web application api and mobile OS.Essentially it provides a wrapper over the native api.

Corodova can be considered as an application container with a web view, used by the native OS. For eg  iOS usage objective c wevview ui class, android web view on android.

How to install phonegap

sudo npm install -g cordova

cordova create sampleapp

cordova platforms add android

cordova platforms add ios

sampleapp/

 |-- hooks/

 |-- platforms/

 |-- plugins/

 |-- www/

 -- config.xml

Please note that in case of android, android sdk should be available on your OS , and for iOS ,Xcode requires to be installed in your mac.

Corodova comes with the basic plugin, like camera, GPS, file system. It does provide creating a plugin for anything specific we want which is not available with the cordova.

Build: Corodova build the application and package the deployable as per the OS. For eg. Android os will have in apk format, iOS will produce IPA file, XAP file for  windows  phone.

Advantages of using cordova

-Very fast to use it, least pre requisite of learning

-Quick to prototype

-Managing images for multiple devices easier with using css and media queries.

Disadvantages:

-Not supported on all mobile browsers

-Not fit very well with varying  hardware sizes , screen size

-Integrate with third party api for eg google map is not stable, gives sometimes error

-For large application , it’s not easy to maintain it.

Spark : Best Use Case

The world is moving towards data oriented app that the end user quickly get the analysed result and business decisions are made subsequently.

Below are the use case that requires spark based fast computing platform.

• Early alert: eg Earthquake alert from the tweets.
• Finance industry: identify fraud detection system.
• E commerce: Analyse the users/ review and customer comments and take a business decision.
• Sports: Identify the patterns and responds accordingly.
• Wether forecasting: Use the capability to process huge amount of data.

Users: Amazon, eBay, and Yahoo.

Largest known cluster to run scala is 8000.

Spark How to use?

Spark is 

-Light weight fast cluster computing

-A fast in memory data processing engine.

-Developmet api available in scala, java , python

-Can be used when we need efficiently execute machine learning algorithm

-Written in scala , Sacala is one implementation of Spark

-Integrates well with the Hadoop ecosystem and data sources (HDFS, Amazon S3, Hive, HBase, Cassandra, etc.)

Here we will go through a sample that primarily scan a log file and then it will give stats based on the need.

public class ErrorScanner {

    public static void main(String[] args) {

        String logFile = "/../../error_sample.log";

        SparkConf conf = new SparkConf().setAppName("The Techie house spark app").setMaster("local");

        JavaSparkContext sc = new JavaSparkContext(conf);

        JavaRDD<String> inputFile = sc.textFile(logFile);

        JavaRDD<String> errors = inputFile.filter(new Function<String, Boolean>() {

            public Boolean call(String s) {

                return s.contains("ERROR");

            }

        });

        errors.count();

        errors.filter(new Function<String, Boolean>() {

            public Boolean call(String s) {

                return s.contains("MySQL");

            }

        }).count();

        errors.filter(new Function<String, Boolean>() {

            public Boolean call(String s) {

                return s.contains("MySQL");

            }

        }).collect();

        for (String word : errors.collect()) {

            System.out.println(word);

        }

    }



}

Get earthquake alert!!

TwitterUtils.createStream(...)
            .filter(

_.getText.contains("Earthquake now!") 
|| _.getText.contains("earthquake") && _.getText.contains("shaking"))

Code coverage

Code coverage plays an essential part in development and helps to ensure is code is delivered with quality.

Below are the code coverage tool that can be used during development:

1. eclipse coverage plugin : http://eclemma.org/