Documenting Rest api using Swagger

Rest apis documentation has great significance when it comes to publishing and testing them. Swagger is a specification for documenting the Rest APIs. It comes with UI that can be used as a GUI to view and test the api.

Annotations provided :

 @Api
@ApiClass
@ApiError
@ApiErrors
@ApiOperation
@ApiParam
@ApiParamImplicit
@ApiParamsImplicit
@ApiProperty
@ApiResponse
@ApiResponses
@ApiModel

/api/api-docs. It is used by Swagger UI which itself is embedded into finalJAR archive and served by Jetty as static web resource.

sample demo :http://petstore.swagger.io/

Spring core : Annotations when to use what

 @Configuration: 
Annotation based configuration alternative to xml based configuration .This will be mostly used with @Bean to define bean at run time 
 for annotation-based configuration, as opposed to xml-based configuration. 

 @Bean:
 define bean at run time, should be under class annotated with @configuration 
 It is used to explicitly declare a single bean, rather letting spring to do it while classpath scanning.
 It decouple declaration from the class definition
 When we don't own the class, it might make sense to use @bean wherever its being used

 @Component:
Define bean at run time, should be under class annotated with @configuration 
 Requires @ComponentScan to allow autoscanning

By default, the bean instances of this class have the same name as the class name with a lowercase initial. On top of that, we can specify a different name using the optional value argument of this annotation.

@ConditionalOnMissingBean:
 conditional that only matches when the specified bean / with name is not already in the bean container.Typically available in spring boot applciation

@ConditionalOnClass:
Will create bean if com.thetechiehouse.MyServiceClass class exist in the class path.To be used in Spring boot env.
@Bean
@ConditionalOnClass(name ="com.thetechiehouse.MyServiceClass")

@ConditionalOnProperty:

@ConditionalOnProperty("app.somekey") // Bean registration hapen only when app.somekey is present
@ConditionalOnProperty(name='com.myproperty',  havingValue = "true")

@ConditionalOnProperty(prefix = "notification", name = "service", havingValue = "sms") 

    //notification.service=sms
@Bean
Create bean only if a specific  property value is true.To be used in Spring boot env.
@ConditionalOnExpression("${xxx.YYY.enabled:true}")
similar to @ConditionalOnProperty but it's a bit slower

Uses 
//@Resources

@Resource (name="myBean")

//@Bean

@Bean(name = "myBean")

    public SomeClass someMethod() {

        return new SomeClass();

    }

//@ Autowired

@Autowired

or

@Autowired

@Qualifier("myBean")

What are differences among @Autowired vs @Inject vs @Resource

Annotataion              package

@Resource	javax.annotation
@Inject	javax.inject
@Qualifier	javax.inject
@Autowired	org.springframework.bean.factory

@Resource:
annotation based on JSR-250. @Resource is quite similar to @Autowired 
 @Autowired (Spring specific)

 @Autowired:
It is fundamentally about type-driven injection ie  that  a resource by-type,
with optional semantic qualifiers.
The bean name is considered a default qualifier value.
 It does  have the required field so this can be used if the field is null and required so.

@Inject:
Annotation based on JSR-330
replacement for Spring’s @Autowired annotation
It  does not have the required field so if it does not find a suitable object it will fail.
It has @Named similar to qualifier in case of autowired.

@Resource will narrow down the search first by name then by type and finally by Qualifiers (ignored if match is found by name). @Autowired and @Inject will narrow down the search first by type then by qualifier and finally by the name.

Since @Repository, @Service, @Configuration, and @Controller are all meta-annotations of @Component, they share the same bean naming behaviour

GC configuration

In this blog, we'll understand the  GC configuration.

-Xmx2560m
-Xms2560m
-XX:CMSInitiatingOccupancyFraction
-Xloggc
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps  

GC settings  and significance

-Xmx2560m  // Max heap memory
-Xms2560m // min heap memory

Setting -Xms and -Xmx to the same value increases predictability by removing the most important sizing decision from the virtual machine. On the other hand, the virtual machine can't compensate if you make a poor choice.

-XX:NewSize=786m   //size of new generation
-XX:MaxNewSize=786m //Max size of new generation
-XX:NewRatio=2 Ratio of old/new generation sizes  // either of two NewSize/NewRatio needed.

-XX:PermSize 
-XX:MaxPermSize=512m // is the range of the size of the permanent generation, which is the non-heap memory

-XX:CMSInitiatingOccupancyFraction=70  // CMS will run after old generation reaches  that pecentage CMS (2560-786)*.7
 -XX:+UseCMSInitiatingOccupancyOnly // optional
CMSInitiatingOccupancyFraction is only used for the 1st collection unless -XX:+UseCMSInitiatingOccupancyOnly is set. 
 default value is approximately 68%. 

-XX:SurvivorRatio=8 // surviver space will be 1/8th of eden space

-XX:+UseCMSInitiatingOccupancyOnly // if not set the little contiguous free space (i.e. fragmentation) in oldgen a variable which could trigger CMS prior to the set occupancy fraction? 

I.e. is CMS smart enough to take into account the possibility of of clearing out dead objects (thus, reducing fragmentation) before hitting a promotion failure

XX:+CMSScavengeBeforeRemark // This flag allows the JVM to perform a minor GC on the young generation before re-marking, so that the number of objects remaining to be marked in the young gen will be much lower than before the GC. By default, this flag is disabled.

If you don't set the latter switch then, after the first one, the usual heuristics
 (which are based on allocation stats gathered during runtime) are used to determine when CMS starts. 

-Xloggc:<gc_file_location>/logs/gc.log // specify gc log file location
-XX:+PrintTenuringDistribution

XX:+PrintGCApplicationStoppedTime:Total time for which application threads were stopped 
 -XX:+PrintGCApplicationConcurrentTime:Application time
sample output:
Application time: 0.2863875 seconds
Total time for which application threads were stopped: 0.0225087 seconds
Application time: 0.1476791 seconds

Total time for which application threads were stopped: 0.0255697 seconds
-XX:+PrintGCDetails // activate detailed gc logging 
eg.
[GC
    [PSYoungGen: 142816K->10752K(142848K)] 246648K->243136K(375296K),
    0,0935090 secs
]
[Times: user=0,55 sys=0,10, real=0,09 secs]

[Full GC
    [PSYoungGen: 10752K->9707K(142848K)]
    [ParOldGen: 232384K->232244K(485888K)] 243136K->241951K(628736K)
    [PSPermGen: 3162K->3161K(21504K)],
    1,5265450 secs
]
[Times: user=10,96 sys=0,06, real=1,53 secs]

XX:+PrintGCTimeStamps and -XX:+PrintGCDateStamps// each line starts with absolute date and time when the logging was written

-XX:+UseConcMarkSweepGC //enable concurrentMarkSweep used for old generation collections
with this most of the gc works happens in the background without stopping application threads.In this applications threads getting stopped kept minimum.

======================================================
-XX:ConcGCThreads=<> //2,3...Number of threads with which the concurrent phase CMS will run.
It is related to ParallelGCThreads

-XX:ParallelGCThreads

The number of garbage collector threads in the young and old generations 
-XX:+UseParallelG // enable throughput collectors

Frequent GC keeps happening at very short interval . This can happen  each times when it is serving request . This is required to cleanup  the short-lived (=new) object  from each request. as they required to to get cleaned up.
 The pause rate is actually very low 0.2/sec and the max pause time of <20ms is good.

Be sure to increase the memory as you increase the number of processors, since allocation can be parallelized.

First we should have the settings using the default settings provided by jvm, then the tuning can be started so that the improvement can be noticed on production. GC viewer:https://github.com/chewiebug/GCViewer

How to troubleshoot GC run

https://www.facebook.com/100006792001047/videos/1717300525172998/

Memory setting is very important for a java application.If properly set it will give significant clue when there is a performance issue from jvm side.

Whenever there is a memory issue occurring in a java application first thing that comes in our mind is whether the memory setting given is sufficient.

We specify memory configuration with -mx argument while running the java process.

-Investigate JVM memory usage in case of out of memory

-Xmx<max_memory_value>

Garbage collection logging flag:

• -XX:+PrintGC or -verbose:gc 
• -XX:+PrintGCDetails
• -XX:+PrintGCTimeStamps and -XX:+PrintGCDateStamps
• -Xloggc //-Xloggc:<file>

How to debug GC run:

Log diagnostic information after each garbage collector run

Run application with -verbose:gc 

 [GC 69068K->56180K(116544K), 0.0054920 secs]

 [GC 69068K->56180K(116544K), 0.0054920 secs] 

 // Young generation usage before 69068K

  and after run 56180K

         Size of JVM heap  116544K

  [Full GC 99119K->50845K(116544K), 0.0137490 secs] //

  The above shows full garbage collection. When application is having frequent full GC running means there is a problem.

  And even after full GC run memory utilisation is not going down that needs to be addressed on priority.

  

 How to get heapdump?

1. Using jmap

jmap -F -dump:format=b,file=heap.bin <process_id>

2. Getting a heap dump on OutOfMemoryError

java -Xmx64m  -XX:+HeapDumpOnOutOfMemoryError  -XX:HeapDumpPath=/tmp/heap.bin  

How to read Heap dump?

I. Using jhat 

jhat heap.bin

Access to http://localhost:7000/ to explore the heap dump

II. Using visual tool

  jvisualvm

   upload the heap dump

Manageable Flags:
All the above flags that w start with “PrintGC” belong to the “manageable category.
 eg. The verbose:gc option can be enabled at run time using java management API or JVM TI.Jconsole can also be used to enable this option as it is complaint with JMX extension.

  

More about JVM diagnostic :http://www.oracle.com/technetwork/java/javase/clopts-139448.html

What are all different memory areas in JVM

Permanent Generation:

-Contains the application metadata required by the JVM to describe the classes and methods used in the application. Note that Perm Gen is not part of Java Heap memory.

Method Area:

- Part of space in the Perm Gen and used to store class structure (runtime constants and static variables) and code for methods and constructors

Runtime Constant Pool

-per-class runtime representation of constant pool in a class. 

It contains class runtime constants and static methods. 

Runtime constant pool is the part of method area.

Memory Pool

-Created by JVM memory managers to create a pool of immutable objects, if implementation supports it

-Memory Pool can belong to Heap or Perm Gen, depending on the JVM memory manager implementation.

Eg. String pool

Young Generation

- where all the new objects are created. 

 This garbage collection is called Minor GC. Young Generation is divided into three parts – Eden Memory and two Survivor Memory spaces.

Since Young generation keeps short-lived objects, Minor GC is very fast and the application doesn’t get affected by this.

Old Generation:

contains the objects that are long lived and survived after many rounds of Minor GC. 

 Old Generation Garbage Collection is called -Major GC and usually takes longer time.

Stop the World Event

All the Garbage Collections are “Stop the World” events because all application threads are stopped until the operation completes.

Minor GC is very fast and the application doesn’t get affected by this.

Major GC takes longer time because it checks all the live objects

 Perm Gen objects are garbage collected in a full garbage collection.

http://jvmmemory.com/

How to monitor java memory

Java memory management is very important as this will give the health of our application.

Basic options:

Heap: -Xms512m –Xmx512m

Permanent generation:-XX:PermSize=128m -XX:MaxPermSize=256m  

Heap free ratio:-XX:MinHeapFreeRatio=40 -XX:MaxHeapFreeRatio=70  

Garbage collectiom

Young generation:

NewRatio:8  -XX:newRatio=8 

Survivor ratio:32   -XX:survivorRatio=32

Garbage collector type: 

G1 --> -XX:+UseG1GC  

ConcurrentMarkAndSweep(old) -> -XX:+UseConcMarkSweepGC 

Garbage collector:-XX:+UseG1GC  

Visual tool : jvisualvm

The below command gives the memory allocation

 jstat -gc <process_id> <time_interval>

 jstat -gc 6781 30000

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT   

264192.0 278016.0 180332.1  0.0   842240.0 578974.5  292864.0   214754.4  38400.0 36880.1     18    2.663   1      0.423    3.087

264192.0 278016.0 180332.1  0.0   842240.0 813343.0  292864.0   214754.4  38400.0 36884.6     18    2.663   1      0.423    3.087

where columns are

S0C and S1C:  Current size of the Survivor0 and Survivor1 areas in KB.

S0U and S1U: Current usage of the Survivor0 and Survivor1 areas in KB. One of the survivor areas are empty all the time.

EC and EU:  Current size and usage of Eden space in KB. EU size will keep increasing and as soon as it crosses the EC, Minor GC is called and EU size is decreased.

OC and OU: Current size and current usage of Old generation in KB.

PC and PU: Current size and current usage of Perm Gen in KB.

YGC and YGCT: YGC tells the number of GC event occurred in young generation. YGCT gives the accumulated time for GC operations for Young generation. Both YGC and YGCT will increase in the same row where EU value is dropped because of minor GC.

FGC and FGCT: FGC column is the number of Full GC event occurred. FGCT column gives accumulated time for Full GC operations.Full GC time will be too high when compared to young generation GC timings.

GCT: The total accumulated time for GC operations. Notice that it’s sum of YGCT and FGCT column values.


 If lot of Full GC operations, > increase Old generation memory space
java.lang.OutOfMemoryError: PermGen space errors ==> increase the Perm Gen memory space using -XX:PermGen and -XX:MaxPermGen JVM options More:http://jvmmemory.com/

Countdown latches: concurrency

Countdown latches are concurrency construct that cause one or more thread to wait until a set of thread execution completes

Using Countdown Latches:

A main thread creates a countdown Latch, that we can use as a starting gate to start all thread simultaneously

Main thread decrement this latch to let all workers proceed.

-Main thread waits for another countdown latch initialised with number of worker thread 

-When all worker threads completes, it decrease the countdown and when it reaches zero the main /waiting thread proceeds and gather results

Concurrency : Thread Executor basics

package : Java.lang.concurrent.ExecutorService

Thread Pool naming convention :pool-<N_GLOBAL_GROUP_COUNT>-thread-<M_THREAD_COUNT_WITHIN_A_GROUP>,

ExecutorService executor = Executors.newSingleThreadExecutor;

executor = Executors.newSingleThreadExecutor;//single thread size

 executor  = Executors.newFixedThreadPool(5);// Pool size fixed

executor.shutdown(); //  Will not take a new request ,

 does not wait for previously submitted tasks to

     * complete execution

executor.shutdownNow(); //  returns list of tasks that never commenced execution/ ignored and were in waiting state, attempt to stop the running thread;

Callable<T> callable= executor.submit(<Runnable >);

 Future<?> future = executor.submit(callable);

 Future<?> future = future.get(5, TimeUnit.SECONDS);//Timeout exception execution_time>5

Java-7 features : quick sample

1. Multi Catch statement:Multiple exceptions type to be handled by the same catch clause:
sample:
Catch(ParseException pe | IOException ioe){
//log error
}
Catch(Exception e){

}

Why dockerizing your application is essential?

Running our app on virtual box is a bit costly operation, as VMs are heavy and slow.This also requires effort in setting upend provisioning.

Whereas in case of Docker, It is lightweight, fast and very easy to configure/provision using a docker file where application specific instruction can be added.

Docker 
-Has a central repository of docker images, which developer can pull from different location.
-Can run many container running on a single machine.

commands:

Creating image :Create image : docker build -t <image_name>:1.0 .

Running app :    docker run -d -p 8080:8080 <image_name>:<v_1.0>
get automatic assigned ip :docker-machine ip default

#Setting env variable
ENV abc=hello

Dockerfile sample:

FROM <docker_repository>

#Install java8

FROM java:8

#Install java7 + maven

FROM jamesdbloom/docker-java7-maven

#Install only Maven

RUN apt-get update && \

    apt-get install -y maven

ADD your_app_root/trunk /opt/your_app

COPY your_app_root/settings.xml /usr/share/maven/conf/

#changing working directory

WORKDIR /opt/your_app/

#Exposing port to be available to the external machine.

EXPOSE 8080

#How to start the image

RUN ["mvn", "install"]

CMD ["mvn", "jetty:run"]

Thread Basics - I

Thread: Why to use thread?

• To run task parallel and concurrently

• To utilise the hardware resource efficiently multi core/mullti cpu+ core  processor

Thread name :

-Is mutable

-Multiple thread can share same name.

Volatile: When to use volatile?

shared variable: one thread read/write + multiple thread read .

Will not work for : thread 1 read/write + thread2 read//write 

Synchronization:
thread 1 read/write + thread2 read/write 

Thread Priorities:

• Each java threads open a new native thread at OS level. Setting priorities translates to native priorities based on the OS platform

• Not recommended to rely solely on priorities

Thread Local Storage:

-Exist as long as the thread is alive

public static class PaymentResponseData

{

public int transactionId;

public String transactionStatus;

public int userId;

}

public static final ThreadLocal<PaymentResponseData> globalData =

new ThreadLocal<PaymentResponseData>();

globalData.set(tranacationData) 

 globalData.get().

 myThread.setUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {

   public void uncaughtException(Thread t, Throwable e) {

   PaymentResponseData paymentResponseData=  globalData.get()

System.err.println("Error occurred with transaction ID " + paymentResponseData.get().transactionId);   }

   });

Demon thread:

Provide service to the user thread. Eg gc

Collecting statistics and performing the status monitoring tasks 

Http scripting

 curl is an open source http scripting tool that can be used to make/ gets the data.  it sends data and it retrieves the information.

curl --trace-ascii d.txt --trace-time http://thetechiehouse.blogspot.com/

curl -o blog.html http://thetechiehouse.blogspot.com/

http authentication curl --user name:password http://www.google.com
post curl --data "birthyear=1905&press=%20OK%20" http://www.example.com/when.cgi

 curl --basic --user-agent “<user_agent>"  —<user_name>:<pwd>  "<url>"

curl -i 

 -H "Accept: application/json"

 -H "Content-Type: application/x-www-form-urlencoded"

  -H "User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"

   -X POST "http://www.mysite.com/form" -d"param1=val1"

Getting data from secured service:

curl -H "Authorization: Bearer <service_auth_token>" 'secured_service_Url'

Getting IP configuration

ipconfig//win

ifconfig// mac

Querying DNS to get IP address mapping

nslookup google.com

network statistics

netstat -n

TCP packet dump

tcpdump -l -x

tcpdump allows us to save the packets that are captured, so that we can use it for future analysis. The saved file can be viewed by the same tcpdump command.

nslookup google.com

dig google.com

curl google.com

curl -L -V google.com

curl-L google.com

curl for sftp:

curl  -k ---user user:pwd sftp://abc.com/path

ping vs curl

Ping only checks if the remote host is answering ICMP packets, which (usually) means it's up and running;
- This doesn't give you any information about which services the host is actually offering.

An HTTP GET request checks that there is a web server running on the host, that it answers to a given IP/port/hostname combo
 that you asked it for a valid URL and that the web site is able to answer your request.


If IIS (or Apache) is stopped on the host, it will very well answer a Ping request, but a HTTP GET will fail.