Analysing Network Packets

Sometime it is essential to dig into how packets are being transmitted from source to destination to identify issues. Let's say imagine client is getting frequent timeout and from services side network admin would like to take a glance at how the machine is serving the network request.
Some of the useful commands:

List all networking hardware connected
networksetup -listallhardwareports

Taking tcp dump:
sudo tcpdump -ttttnnr ~/tcp_dumpFile.pcap

Reading tcpdump file on command prompt:
sudo tcpdump -ttttnnr tcp_dumpFile.pcap

Analysing the tcp dump:
22:24:18.910372 IP (tos 0x10, ttl 64, id 9792, offset 0, flags [DF], proto TCP (6), length 88)
78.47.105.76.ssh > 82.132.219.219.55495: Flags [P.], cksum 0xcb29 (correct), seq 497880562:497880610(48), ack 1593322765, win 379, length 48

Source IP address and Port: 78.47.105.76.ssh
Destination IP address and Port: 82.132.219.219.55495
Length 88 – the IP packet length, including all headers, in Bytes (16 bits, 3rd and 4th octets)
proto TCP (6) – the higher layer (four) protocol and it’s number (8 bits, 10th octet)
offset 0 – the fragment offset, used with fragmented packets, should be 0 or a multiple of 8, displayed in bytes (13 bits of the 7th and 8th octets)
cksum 0xcb29 (correct) – the packet’s TCP checksum value
win 379 – the source host’s TCP window
ack 1593322765 – the TCP packet’s acknowledgement number
seq 497880562:497880610(48) – the TCP packet’s starting and ending sequence numbers, the value in brackets indicates the difference and thus the amount of data carried (in Bytes); this should match the length field

Timeouts in Http Connection

This blog talks about how to configure timeout parameters in creating http connection. Irrespective of the the destination, the configuration is generic and applicable for any type of remote host internal or external.

http.connection.timeout-  Time to establish connection with the Remote host

http.socket.timeout: It's the waiting time for the data. Max time it will wait for between two packets

http.connection-manager.timeout: Time to wait for getting connection from connection manger/ Pool

Hard timeout: We don’t want the request to be executed for more than x sec. Eg. if file download request is taking time more than 5s

HttpGet getMethod = new HttpGet(   "http://www.domainname/respurces"); int hardTimeout = 5; // seconds TimerTask task = new TimerTask() {     @Override     public void run() {         if (getMethod != null) {             getMethod.abort();         }     } }; new Timer(true).schedule(task, hardTimeout * 1000); HttpResponse response = httpClient.execute(getMethod); System.out.println(   "HTTP Status of response: " + response.getStatusLine().getStatusCode());

Timeout and DNS Round Robin : when the same domain mapped to multiple IP addresses.
HttpClient gets the list of IP routes to that domain
It tries the first one – that times out (with the timeouts we configure)
It tries the second one – that also times out
- and so on …

DEBUG o.a.h.i.c.HttpClientConnectionOperator - Connecting to www.google.com/173.194.34.212:81
DEBUG o.a.h.i.c.HttpClientConnectionOperator -
Connect to www.google.com/173.194.34.212:81 timed out. Connection will be retried using another IP address

How to Read the GC Logs

Tools: http://gceasy.io/

Correlation data in RabbitMQ

Customising RabbitMQ Data:

rabbitTemplate.convertAndSend("", QUEUE, "foo", new CorrelationData("Correlation for message 1"));

/*

* Replace the correlation data with one containing the converted message in case

* we want to resend it after a nack.

*/

rabbitTemplate.setCorrelationDataPostProcessor((message, correlationData) ->

new CompleteMessageCorrelationData(correlationData != null ? correlationData.getId() : null, message));

rabbitTemplate.setConfirmCallback((correlation, ack, reason) -> {

if (correlation != null) {

System.out.println("Received " + (ack ? " ack " : " nack ") + "for correlation: " + correlation);

}

});

Static class CompleteMessageCorrelationData extends CorrelationData {
private final Message message;

CompleteMessageCorrelationData(String id, Message message) {
super(id);
this.message = message;
}

public Message getMessage() {
return this.message;
}

@Override
public String toString() {
return "CompleteMessageCorrelationData [id=" + getId() + ", message=" + this.message + "]";
}

}

The Basics of RMQ

What is an Exchange?
  -Exchange works like a message routing agent, responsible for  routing the message to queue.
  -Publisher  sends the message to exchange.
  -Exchange deliver the messages based on the header attributes, binding and routing keys.
- Publisher can send messages directly to exchange with declaring the routing key. No need to bind with any queue specifically.
- Exchange can be declared programmatically  from the connection template(RabbitMQTemplate)

Exchange types:
  -temporary, durable, auto-delete
 -Clients can can use predefined  default exchange or create a new one.
 -Exchange created when the server starts.

Direct exchange:
Topic exchanges route messages to queues based on wildcard matches between the routing key and something called the routing patternHere a message goes to the queues whose binding key exactly matches the routing key of the message.

Topic Exchange specified by the queue binding. Messages are routed to one or many queues based on a matching between a message routing key and this pattern.

Fanout Exchange
The fanout copies and routes a received message to all queues that are bound to it regardless of routing keys or pattern matching as with direct and topic exchanges. Keys provided will simply be ignored.

Virtual Hosts:
AMQP provide isolated environment in which the messages entities (exchange/queue/users/...) live. This allows one broker to have multiple hosts similar to webs servers.

Channels:
 Channels are the next protocol layer on top of Connections. Our application uses it to communicate with the RabbitMQ server.

Consumer Acknowledgement And Publisher Confirms

Publisher (delivery) Confirms and Returns (Processing confirmation)

RMQ supports  Publisher Confirms and Returns.

In the distributed architecture, the protocol method is not guaranteed that message reaches the peers. Both message broker and consumer needs a mechanism for delivery and processing confirmation

For returned messages, the template’s mandatory property must be set to true, or the mandatory-expression must evaluate to true for a particular message.

 This feature requires a CachingConnectionFactory that has its publisherReturns property set to true (see the section called “Publisher Confirms and Returns”). Returns are sent to the client by it registering a RabbitTemplate.ReturnCallback by calling setReturnCallback(ReturnCallback callback). The callback must implement this method:

Publisher confirmation:

Network can fail due to many reason and detecting network failure may take time.
Messages can be lost on its way or delayed significantly.
-To guarantee the delivery, it requires to make it transactional. 
-In exceptional case when broker is not able to send the message, it will send basic.nack.
-Confirms are when the broker sends an ack back to the publisher, indicating that a message was successfully routed.

-Only one ConfirmCallback is supported by a RabbitTemplate.

-void confirm(CorrelationData correlationData, boolean ack, String cause);

-void returnedMessage(Message message, int replyCode, String replyText,

          String exchange, String routingKey);

         

 rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {

            @Override

            public void confirm(CorrelationData correlationData, boolean b) {

                System.err.println("confirmedMessage");

            }

        });

ReturnCallback:

Returns are when the broker returns a message because it's undeliverable (no matching bindings on the exchange to which the message was published, and the mandatory bit is set)

        rabbitTemplate.setReturnCallback(new RabbitTemplate.ReturnCallback() {

            @Override

            public void returnedMessage(message, replyCode, replyText, exchange, routingKey) {

              

            }

        });

Positively Acknowledging Deliveries:

   

 channel.basicAck(deliveryTag, false);

Acknowledging Multiple Deliveries at Once:

  channel.basicAck(deliveryTag, true);

  

  For example, given that there are delivery tags 5, 6, 7, and 8 unacknowledged on channel Ch, when an acknowledgement frame arrives on that channel with delivery_tag set to 8 and multiple set to true, all tags from 5 to 8 will be acknowledged. If multiple was set to false, deliveries 5, 6, and 7 would still be unacknowledged.

                     

 Negative Acknowledgement and Requeuing of Deliveries

  

  // negatively acknowledge, the message will be discarded

                    channel.BasicReject(ea.DeliveryTag, false);

                                      

  // requeue the delivery

                  channel.BasicReject(ea.DeliveryTag, true);

                                           

  --requeue all unacknowledged deliveries up to this delivery tag

             channel.basicNack(deliveryTag, true, true);

         }