Investigating Performance issue in Live Environment

1. Take a thread dump when the issue is with CPU
2. Go for heap dump if this is affecting high memory usage
3. Replicate similar load testing in some other environment.
4. Do performance Baselining/Benchmarking.

List of java process running :

jcmd -l

Java Histogram : To get java object statistics 

jmap -histo <process_id> > histogram_current_time.txt

*Strings are not part of object size in the above results

Get Threaddump:

jstack -l <pid>

Sometimes threads are stuck and other threads are blocked , this causes shooting up heap uses

HeapDump:

jmap -dump:format=b,file=<relative>/heap.bean

Profiling Tool:
Jconsole:
Open jconsole
adding remote host:port and credential
It will show dashboards for heap memory, thread, class,cpu usage, VM summary,Mbeans
Mbeans is useful to do lot of java exposed operation
-By seeing the memory usage diff between before and after an operation, the increase in memory tells the memory utilisation in an activity or set of objects.
-You can trigger GC run to clean up unused objects.

visualVM:
Add new jmx connection:hostname:port/credentials
we can take snapshot, do cpu sampling
Both the above tools allow perform GC,
visual VM has option to get heap dump, it saves on the m/c java is running.

JVisualVM

- jvosualvm, JDK provided tool

Spring annotations: What When and How?

@Configuration:

Used as a replacement for bean definition in xml. Usually @Bean is used to register beans.

@Configurable // To inject beans within object created using new ()

@Configurable(preConstruction = true)  -Inject dependencies before object creation 

<context:spring-configured/> To enable load time weaving 
<context:annotation-config> To activate annotations in beans already registered
<context:component-scan> To activate annotations in beans already registered +   also scans packages to find and register beans within the application context.

@ContextConfiguration:
-Used for Spring based test class
-By default it will look to load testclass-context.xml if not specified in class .
@ContextConfiguration("configFile.xml")
Or @ContextConfiguration({TestConfig.class,HelperClass.class})

@Import

-Aggregating @Configuration classes
-Can be used to import other @Configuration class

@ Retryable

@Service
public class TestService {

  @Retryable(value = {SomeException.class, MyException.class}, 
      maxAttempts = 3,backoff = @Backoff(2000))
  public void retryWhenException() {
try{
// perform operation that can fail
}catch(SomeException se){
// will be retried
throw se;
}catch(MyException me){
// will be retried
throw me;
}catch(Exception e){
// will not be retried
throw e;
}
  }

  @Recover
  public void recover(SomeException exception) {
    // recover from SomeException
  }
}

Less Commands

Forward Search

• ./ – search for a pattern which will take you to the next occurrence.
• n – for next match in forward
• N – for previous match in backward

Backward Search

• ? – search for a pattern which will take you to the previous occurrence.
• n – for next match in backward direction
• N – for previous match in forward direction

• CTRL+F – forward one window
• CTRL+B – backward one window
• CTRL+D – forward half window
• CTRL+U – backward half window
• j – navigate forward by one line
• k – navigate backward by one line
• q or ZZ – exit the less pager

• 10j – 10 lines forward.
• 10k – 10 lines backward.
• tail -f =>F

less application.* | grep 'search_key'

JVM Profiling

JVM profiling in highly useful in identifying what in happening in JVM.

There are many tools available for profiling.

-Visual VM

-JProfiler

Profiling with VisualVM

-Open VisualVM

-Select the processId

-Click 'Sampler' tab

-Click settings. Select profile only package, com.thetechiehouse.*

-Click CPU

-Hit the URL on browser

-Once we get the response, Stop the CPU profiling.

Profiling with Jprofiler:

- It will provide  CPU Views, Heap Walker, Live Memory, recording capability, saving Snapshot etc.

We should be seeing the profiling details along with method level/self/CPU/ total execution time.

Snapshot can be downloaded, nps file will have all the profiling details captured.
Click the combined tab to view the overall method execution time.

Elastic Search Sample Query

http://localhost:5601/app/kibana#/dev_tools/console?_g=()

user:pwd elastic/changeme

PUT /customer?pretty

GET /_cat/indices?v

PUT /customer/external/1?pretty

{

  "name": "John Doe"

}

GET /customer/external/1?pretty

PUT /customer

PUT /customer/external/1

{

  "name": "John Doe"

}

GET /customer/external/1

DELETE /customer

PUT /customer/external/1?pretty

{

  "name": "John Doe"

}

PUT /customer/external/2?pretty

{

  "name": "Sam Stint"

}

CURL Hit

 curl -X POST "http://<$ES_HOST>/INDEX_NAME/OPERATION?conflicts=proceed&wait_for_completion=false" -H 'Content-Type: application/json' -d'

{

 ES_QUERY

 }'

POST /customer/external/1/_update?pretty

{

  "doc": { "name": "Jane Doe" }

}

POST /customer/external/1/_update?pretty

{

  "doc": { "name": "Jane Doe", "age": 20 }

}

POST /customer/external/2/_update?pretty

{

  "doc": { "name": "Jane Doe2", "age": 20 }

}

POST /customer/external/1/_update?pretty

{

  "script" : "ctx._source.age += 5"

}

DELETE /customer/external/2?pretty

#batch operation

POST /customer/external/_bulk?pretty

{"index":{"_id":"1"}}

{"name": "John Doe" }

{"index":{"_id":"2"}}

{"name": "Jane Doe" }

POST /customer/external/_bulk?pretty

{"update":{"_id":"1"}}

{"doc":{"name":"John Doe becomes Jane Doe"}}

{"delete":{"_id":"2"}}

#Bank sample

accounts.json

{"index":{"_id":"1"}}

{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}

curl --user elastic:changeme -H "Content-Type: application/json" -XPOST 'localhost:9200/bank/account/_bulk?pretty&refresh' --data-binary "@accounts.json"

curl 'localhost:9200/_cat/indices?v' --user elastic:changeme

GET /bank/_search?q=*&sort=account_number:asc&pretty

GET /bank/_search

{

  "query": { "match_all": {} },

  "sort": [

    { "account_number": "asc" }

  ]

}

GET /bank/_search

{

  "query": { "match_all": {} },

  "size": 2

}

GET /bank/_search

{

  "query": { "match_all": {} },

  "from": 10,

  "size": 10

}

GET /bank/_search

{

  "query": { "match_all": {} },

  "sort": { "balance": { "order": "desc" } }

}

GET /bank/_search

{

  "query": { "match_all": {} },

  "_source": ["account_number", "balance"]

}

GET /bank/_search

{

  "query": { "match": { "address": "mill lane" } }

}

GET /bank/_search

{

  "query": {

    "bool": {

      "must": [

        { "match": { "address": "mill" } },

        { "match": { "address": "lane" } }

      ]

    }

  }

}

GET /bank/_search

{

  "query": {

    "bool": {

      "should": [

        { "match": { "address": "mill" } },

        { "match": { "address": "lane" } }

      ]

    }

  }

}

GET /bank/_search

{

  "query": {

    "bool": {

      "must_not": [

        { "match": { "address": "mill" } },

        { "match": { "address": "lane" } }

      ]

    }

  }

}

GET /bank/_search

{

  "query": {

    "bool": {

      "must": [

        { "match": { "age": "40" } }

      ],

      "must_not": [

        { "match": { "state": "ID" } }

      ]

    }

  }

}

Term query based search:

GET employee/_search

{

  "query": {

    "bool": {

      "must": [

        {

          "term": {

            "name": {

              "value": "John"

            }

          }

        },

        {

          "term": {

            "department": {

              "value": "Sales"

            }

          }

        }

        

      ]

    }

  }

}

GET /bank/_search

{

  "query": {

    "bool": {

      "must": { "match_all": {} },

      "filter": {

        "range": {

          "balance": {

            "gte": 20000,

            "lte": 30000

          }

        }

      }

    }

  }

}

Find distinct values

GET /cars/transactions/_search?search_type=count

{

  "aggs": {

    "distinct_colors": {

      "terms": {

        "field": "color",

        "size": 1000

      }

    }

  }

}

More: https://www.elastic.co/guide/en/elasticsearch/reference/current/_executing_filters.html