headers
Cheng Huang | 4 Feb 2011 19:11
Picon
Favicon

[CFP] HotMD 2011 -- Hot Topics in Multimedia Delivery

HotMD 2011 -- Hot Topics in Multimedia Delivery

 

Barcelona, Spain, July 2011

 

Deadline: Feb. 20, 2011

 

http://sites.google.com/site/hotmd2011/

 

Workshop co-chairs

      • Cheng Huang, Microsoft Research, USA

      • Yong Liu, Polytechnic Institute of NYU, USA

      • Francesc Pinyol Margalef, La Salle - Universitat Ramon Llull, Spain

      • Gabriel Fernàndez Ubiergo, La Salle - Universitat Ramon Llull, Spain

      • Alejandro López González, La Salle - Universitat Ramon Llull, Spain

 

Program committee

      • Federico Alvarez, Polytechnic University of Madrid

      • Yunnan Wu, Facebook

      • Wei Tsang Ooi, National University of Singapore

      • Antonio Navarro, Instituto de Telecomunicações

      • Songqing Chen, George Mason University

      • Yang Guo, Bell Labs

      • Zhi-Li Zhang, University of Minnesota

      • Ali Begen, Cisco

      • Jesus Alcober, Polytechnic University of Catalonia

      • Baochun Li, University of Toronto

      • Thomas Schierl, Fraunhofer HHI - Heinrich Hertz Institute

      • Stewart T. Worrall, University of Surrey

      • David Gómez-Barquero, Polytechnic University of Valencia

      • Zhu Liu, AT&T Research Labs

      • John Lui, Chinese University of Hong Kong

      • Bo Li, Hong Kong University of Science and Technology

      • Jiangchuan Liu, Simon Fraser University

      • Anees Shaikh, IBM Research

      • Richard Yang, Yale University

      • Sanjeev Mehrotra, Microsoft Research

      • Sanjay Rao, Purdue University

      • Georg Thallinger, Joanneum Research

      • Andy Berkheimer, Google

Permalink | Reply | 0 comments |
headers
Ratul Mahajan | 7 Feb 2011 18:55
Picon
Favicon

CFP: MobiArch 2011

 

Call for Papers: The 6th ACM International Workshop on Mobility
in the Evolving Internet Architecture (MobiArch 2011)

http://mobiarch11.cs.ucl.ac.uk

June 28, 2011

Co-located with ACM MobiSys 2011 in Washington, D.C.

Technical Area

With recent advances in technologies for wireless access (e.g., WiFi, 3G, and 4G) and mobile devices (e.g., smartphones, netbooks, and tablets), mobility has become a fundamental characteristic of today's Internet. Yet, basic architectural issues related to mobility such as efficient mobility management, the locator-identifier split, multi-homing, security, and various operational, deployment concerns are still in their early stages of exploration. Moreover, the Internet architecture itself, its end-to-end principles, and business models require rethinking due to the massive penetration of mobility into the Internet. For instance, an appropriate system that allows communicating with a mobile host requires addressing several fundamental issues with the Internet architecture, such as ability to locate the mobile host/service, preserving ongoing communications upon changes of locations, as well as seamless and secure handover management. As another example, the emerging wireless technologies such as those in the 60 GHz and whitespace bands may pose additional challenges since they may introduce design principles different from the packet-switched Internet.

MobiArch 2011 welcomes submissions, from both researchers and practitioners, in exploration of recent advances in architectures, protocols, and experiences with emerging technologies on various mobility issues over the Internet, with an emphasis on wireless infrastructures and mobility patterns for mobility support, new mobility protocols, service discovery, routing and location management, mobile network performance evaluation and modelling, multi-homing, security, architectural impacts and deployment considerations. We encourage submission of early, in-progress work as well as position papers.

The workshop will address all architectural issues and system support for mobility in the Internet, including but not limited to:

  • Impacts of new wireless technologies/services, networking technologies, and mobility patterns on Internet architecture.

  • Architectures and protocols for mobility support in the Internet, ranging from approaches in the link, network, and transport layers, to the application layer and cross-layer design.

  • Architectures and protocols for service partitioning, code offloading, and data center management to support mobile devices in the compute cloud.

  • Location management, robustness, routing, locator/identifier split, multi-homing and load sharing issues.

  • Security, dependability, and privacy issues in mobility networks, and their impacts on Internet architecture.

  • Architectures and mechanisms for wireless/mobile connectivity in remote areas and developing countries.

  • Performance issues with mobility in the Internet.

  • QoS and middlebox issues in mobility networks and impacts on Internet architecture.

  • Economic and deployment issues of mobility solutions (infrastructure and devices).

  • Impact of social aspects on mobility architectures, mobile application and protocol design.


Paper format and submission instructions

Submitted papers must be no more than 6 pages long, with no characters in smaller than 10 point fonts.  Paper submission will be handled via HotCRP.  Papers will be reviewed single-blind.

Important Dates

Submission Deadline:                  March 1, 2011, 11:59PM Eastern Standard Time (EST)
Acceptance Notification:                March 31, 2011
Camera Ready Due:                     April 24, 2011


Program committee

Kyle Jamieson, University College London, UK (co-chair)
Ratul Mahajan, Microsoft Research Redmond, USA (co-chair)
Ashok Agrawala, University of Maryland, USA
Rajesh Balan, Singapore Management University, Singapore
Nilanjan Banerjee, University of Arkansas, USA
Olivier Bonaventure, Université Catholique de Louvain, Belgium
Ranveer Chandra, MSR Redmond, USA       
Jakob Eriksson, University of Illinois at Chicago, USA    
Jaeyeon Jung, Intel Labs Seattle, USA
Li Li, Bell Labs, USA
Costin Raiciu, Politehnica U. of Bucharest, Romania
Dipankar Raychaudhuri, Rutgers University, USA
Injong Rhee, NC State University, USA      
Mahadev Satyanarayan, CMU, USA

Jonathan Smith, University of Pennsylvania, USA
Alex Snoeren, UCSD, USA        
Oliver Spatscheck, AT&T Labs, USA
Peter Steenkiste, CMU, USA        
Kun Tan, MSR Asia, USA
Mike Walfish, UT Austin, USA
Lixia Zhang, UC Los Angeles, USA

Steering Committee

Jon Crowcroft, University of Cambridge, UK (chair)
Xiaoming Fu, University of Goettingen, Germany (ex-officio)
Katherine Guo, Bell Laboratories, USA
Henning Schulzrinne, Columbia University, USA
Lars Eggert, Nokia Research Center, Finland
Jörg Ott, Helsinki University of Technology, Finland

Marco Gruteser, Rutgers University, USA

 

 

 

Permalink | Reply | 0 comments |
headers
Renata.Teixeira | 16 Feb 2011 04:11
Picon
Favicon

CFP: ACM Internet Measurement Conference 2011

---------------------------------------------------------------------
			   Call for Papers
		 Internet Measurement Conference 2011
	   Sponsored by ACM SIGCOMM and ACM SIGMETRICS and
		      in cooperation with USENIX
			  November 2-4, 2011
			   Berlin, Germany
	       http://conferences.sigcomm.org/imc/2011
---------------------------------------------------------------------

The eleventh Internet Measurement Conference is a three day event
focusing on all aspects of measurement-based research pertaining to
today's Internet. Building on the success of past IMCs, we invite
submissions of papers that contribute to our understanding of all
aspects  related to the Internet through the collection, analysis, or
visualization of all types of  network-related measurements. Examples
of relevant topics are:

* Internet traffic and topology measurement
* Internet-oriented wireless, and mobility measurement
* Internet performance measurements
* Inter-domain and intra-domain routing
* Measurement-based network management such as traffic engineering
* Network applications such as multimedia streaming, gaming and
on-line social networks
* Measurement of sensing applications, including home networking and
green networking
* Measurements of content distribution, peer-to-peer, overlay, and
social networks
* Internet data-specific issues, including anonymization, querying,
and storage
* Measurement-based inference of network properties and structures
* Design of and experiences with new monitoring systems
* Algorithms for network measurements, including sampling and
statistical inference methods
* Network anomaly detection and troubleshooting
* Software tools for analyzing and visualizing high-volume Internet-
related datasets
* Measurement related to network security and privacy
* Measurement-based assessment of simulation/testbeds
* Measurement-based modeling and workload generation
* Reappraisal of previous measurement findings
* Measurement related to data centers and the cloud
* Measurement of technological and socio-economic aspects of networking
* Relevant end-host and end-user measurements

Papers that do not in some fashion relate to measuring aspects of the
Internet or Internet-like systems are out of scope. Moreover, papers
describing original ideas and novel validation methods will be given
preference over submissions that report on incremental improvements of
the existing  state-of-the-art. Authors can contact the Program
Co-Chairs  for clarification if they are unsure whether their paper is
in scope.

Ethical standards for measurement must be considered by all IMC
authors. In particular, authors  must be aware of and conform to
acceptable use policies for individual domains that are probed  or
monitored, data privacy and anonymity for all personally identifiable
information, and etiquette  for using shared measurement data (see
Allman and Paxson, IMC '07). If applicable, authors are also  urged to
notify parties of security flaws in their products or services in
advance of publication.  Adherence to ethical standards for
measurement will be a criteria for all submissions and
violations---including ambiguous situations not well described---will
be grounds for rejection.

Submission Guidelines
---------------------

There are two forms of submissions:

   1. Full papers (up to 14 two-column pages) describing original
      research, with succinctness appropriate to the topics and themes
      they discuss.

   2. Short papers (up to 6 two-column pages for text and figures + 1
      page for references) conveying work that is less mature but shows
      promise, articulating a high-level vision, describing challenging
      future directions, critiquing current measurement wisdom or offering
      results that do not merit a full submission.  Short papers will be
      subject to a 7-page limit in the Proceedings (with the last page for
      references only).

Note: Previous IMCs have utilized a "reject to short" notion
whereby full paper submissions have been accepted as short papers. IMC
2011 will not be using this procedure.  Any submission longer than
allowed by the above short guidelines will be considered a full paper.

Note: Authors are encouraged to think carefully about whether to
submit a full or short paper.  In the past there have been many instances
whereby a full submission has been found to be lacking enough technical
contribution for a full paper, but for which the PC finds an interesting
portion of the paper that would have been a very nice contribution as
a short paper.

Submissions must be in electronic form, as PDF documents. The
submission must conform to the page limits stated above, and with text
written in at least a 10-point font (Fonts used in Figures etc should
be no smaller than 9 pt) satisfying the requirements
specified below.

   * use double column format
   * the size of each column should be at most 9.25" by 3.33"
   * the space between columns should be at least 0.33"
   * use 10pt font
   * use up to 55 lines of text per column

The http://conferences.sigcomm.org/imc/2009/sig-alternate-10pt.cls
style file should satisfy these requirements.

Please use -t letter when converting the .dvi file to pdf.

All manuscripts must be in English and do not need to be anonymized.
Submissions that do not comply with these requirements will not be read.

All full papers and short papers accepted for presentation will be
published in the Conference Proceedings produced by ACM. A few accepted
papers may be forwarded for fast-track submission to the IEEE/ACM
Transactions on Networking.

To encourage broader data sharing in the community, the conference will
present a best paper award for the top paper that makes its data sets
publically available by the time of camera ready submission. For
example, wireless-network data sets may be published through CRAWDAD.
Authors that would like their paper to be considered for this award
should add a footnote on the first page of their submission (and indicate
this by selecting the appropriate button on the paper registration/submission
page).

A limited number of travel grants may be available to students who
are unable to secure funding from their advisors.

Important Dates
---------------

   * May 6, 2011, mid-night EDT: Registration of title and 250-word abstract
   * May 13, 2011, mid-night EDT: Hard submission deadline
   * July 22, 2011: Notification
   * November 2-4, 2011:</b> Conference held in Berlin, Germany

Program Co-Chairs
-----------------

Patrick Thiran, EPFL
Walter Willinger, AT&T Labs-Research

Program Committee
-----------------

Katerina Argyraki (EPFL, Switzerland)
Martin Arlitt (HP Labs and University of Calgary, Canada)
Suman Banerjee (University of Wisconsin, USA)
Bobby Bhattacharjee (University of Maryland, USA)
Fabian Bustamante (Northwestern University, USA)
Augustin Chaintreau (Columbia University, USA)
Xenofontas Dimitropoulos (ETH Zurich, Switzerland)
Lixin Gao (University of Massachusetts, USA)
Alexandre Gerber (AT&T Labs-Research, USA)
Srikanth Kandula (Microsoft Research, USA)
Thomas Karagiannis (Mirosoft Research, UK)
Sachin Katti (Stanford University, USA)
Ramana Kompella (Purdue University, USA)
Christian Kreibich (ICSI, USA)
Nikolaos Laoutaris (Telefonica Research, Spain)
Olaf Maennel (Loughborough University, UK)
Ratul Mahajan (Microsoft Research, USA)
Z. Morley Mao (University of Michigan, USA)
Hung X. Nguyen (University of Adelaide, Australia)
Antonio Nucci (Narus, USA)
Konstantina (Dina) Papagiannaki (Intel Labs, USA)
Lili Qiu (University of Texas, USA)
Reza Rejaie (University of Oregon, USA)
Pablo Rodriguez (Telefonica Research, Spain)
Matthew Roughan (University of Adelaide, Australia)
Vyas Sekar (Intel Labs, USA)
Rob Sherwood (Deutsche Telekom Labs, USA)
Georgios Smaragdakis (TU Berlin and Deutsche Telekom Labs, Germany)
Joel Sommers (Colgate University, USA)
Neil Spring (University of Maryland, USA)
Renata Teixeira (CNRS and UPMC Sorbonne Universites, France)
Steve Uhlig (TU Berlin and Deutsche Telekom Labs, Germany)
David Wetherall (University of Washington, USA)
Lixia Zhang (UCLA, USA)

Sponsors
--------

Gold: Akamai
Silver: HP and Technicolor
Permalink | Reply | 0 comments |
headers
Zama Ques | 22 Feb 2011 10:24
Picon
Favicon

query on behaviour of tcp_keepalive and tcp retransmit on Linux based systems

We need some clarifications on TCP_keepalive .  We are facing some issues on our Prod servers related to TCP functionality .

The issue is like this.

We have some machines at one end sending data in real time to another group of machines on the other hand .  Now due to some hardware issues on the other hand , some of the machines becomes unresponsive/crashes. The client system which pumps data never came to know that the server went unresponsive . The connection remains in
ESTABLISHED state and the client always tries to send data thinking that the connection is alive because of which we are seeing backlog on client sides.

Our understanding is like this on how TCP will handle the connection.


Q 1) Since  the server went down , the client will try to the retransmit the data until it times out. What is the behavior of TCP after the timeout? Need clarification on
the following things.
a) Will the kernel will close the established connection after the timeout . Looks like no in our case as we still see the connection still in ESTABLISHED state after around more
than 2 hours.
b) Are there any kernel parameters which decides the when the client is timeout after retransmission fails. What is the behavior of TCP after the client retransmission timeouts.


Q 2 ) There is something called tcp_keepalive which if implemented in the kernel , by default it's there and comes to be around 2 hrs 2 minsutes , i think  ,  the client will send some TCP probes after the keepalive time ineterval and if it cannot reach the server , then the established connection in the client side will be closed by the kernel . This is my understanding. But I can see that the connection still remains in established after the tcp_keepalive time . We waited for around 2 hrs 30 minutes but the connection remains in established state only. Tried reducing the keepalive time to be around 10 minutes , but the connection remains in ESTABLISHED state in client side .


Where I went wrong .Please clarify my doubts raised above . What should we do to resolve the problem we are seeing above . Any help will be highly appreciated as we are going through a hard time to resolve the issue .

Thanks in Advance

Permalink | Reply | 2 comments |
headers
Yan Cai | 22 Feb 2011 15:55
Picon

Re: query on behaviour of tcp_keepalive and tcp retransmit on Linux based systems

Hi

According to your description, the expected behavior should be as follows.
At the beginning senders at one side can send data to the receivers at the other side, and the receivers can receive data without any problem. When some of the receivers become off-line, the affected senders should no long receive positive acknowledgments, therefore, lowering their congestion windows (i.e., sending rate). Since in your case the receiver is off forever, some senders should further experience timeout events. After a few timeouts, the sender should CLOSE this connection itself.

As far as I know, the whole procedure above should be automatically invoked in the sender side. This is how TCP (sender) handles exceptions.

My suggestion is that you run a simple experiment on your side to see if TCP in your machine can work that way. The test can be done using i-perf to send a long long live TCP flow, and then take off the receiver in the middle of the transmission. The connection is expected to be closed very soon after the receiver is off.

Hope it helpful.
Yan
On 2/22/2011 4:24 AM, Zama Ques wrote:
We need some clarifications on TCP_keepalive .  We are facing some issues on our Prod servers related to TCP functionality .

The issue is like this.

We have some machines at one end sending data in real time to another group of machines on the other hand .  Now due to some hardware issues on the other hand , some of the machines becomes unresponsive/crashes. The client system which pumps data never came to know that the server went unresponsive . The connection remains in
ESTABLISHED state and the client always tries to send data thinking that the connection is alive because of which we are seeing backlog on client sides.

Our understanding is like this on how TCP will handle the connection.


Q 1) Since  the server went down , the client will try to the retransmit the data until it times out. What is the behavior of TCP after the timeout? Need clarification on
the following things.
a) Will the kernel will close the established connection after the timeout . Looks like no in our case as we still see the connection still in ESTABLISHED state after around more
than 2 hours.
b) Are there any kernel parameters which decides the when the client is timeout after retransmission fails. What is the behavior of TCP after the client retransmission timeouts.


Q 2 ) There is something called tcp_keepalive which if implemented in the kernel , by default it's there and comes to be around 2 hrs 2 minsutes , i think  ,  the client will send some TCP probes after the keepalive time ineterval and if it cannot reach the server , then the established connection in the client side will be closed by the kernel . This is my understanding. But I can see that the connection still remains in established after the tcp_keepalive time . We waited for around 2 hrs 30 minutes but the connection remains in established state only. Tried reducing the keepalive time to be around 10 minutes , but the connection remains in ESTABLISHED state in client side .


Where I went wrong .Please clarify my doubts raised above . What should we do to resolve the problem we are seeing above . Any help will be highly appreciated as we are going through a hard time to resolve the issue .

Thanks in Advance


Permalink | Reply | 0 comments |
headers
Ning Weng | 21 Feb 2011 18:03

ANCS CFP

CALL FOR PAPERS

             The 7th ACM/IEEE Symposium on Architectures
              for Networking and Communications Systems

                              ANCS 2011
                       http://www.ancsconf.org

                         October 3-4, 2011
                    Polytechnic Institute of NYU
                      Brooklyn, New York, USA

Sponsored by (pending):
ACM Special Interest Group on Computer Architecture (SIGARCH) ACM Special
Interest Group on Communications (SIGCOMM) IEEE Computer Society Tech.
Committee on Computer Architecture (TCCA)

IMPORTANT DATES:

Paper registration and abstract:        May 9, 2011
Submission deadline:                    May 16, 2011
Author notification:                    July 29, 2011

CONFERENCE OVERVIEW:

ANCS is a systems-oriented research conference, presenting original work
that explores the relationship between the architecture of modern computer
networks and the architecture of the individual hardware and software
elements from which these networks are built. This year's conference will
emphasize in its paper selection research on computer and network systems
that provide the foundations of emerging network technologies and the future
Internet.

Topics of interest include, but are not limited to:

*  System design for future network architectures
*  Network architectures enabled by converged platforms
*  Virtualized infrastructure architectures, rationale, and devices
*  Converged router, server, and storage platforms
*  Content-centric architectures, platforms, and mechanisms
*  Scalable programming and application frameworks
*  High-performance / high-function packet processing platforms
*  Power- and size-optimized computer and communications platforms
*  High-speed networking mechanisms and algorithms
*  Network security architectures and security anchor/enhancement
   devices
*  Single-chip platform integration
*  Network measurement techniques, architectures, and devices
*  Techniques and systems for large-scale data analysis
*  Host-network interface issues
*  Architectures for data centers and cloud computing
*  Router and switch architectures
*  Network on chip
*  Software radios
*  Wireless-networking hardware and related software

SUBMISSION PROCEDURES:

Papers must be registered, with a title and abstract, no later than May 9,
2011 at 11:59PM EST (US). The deadline for submissions of full papers is May
16, 2011 at 11:59PM EST (US). There will be no extensions of this deadline.

Please see the conference web site for detailed submission guidelines
<http://www.ancsconf.org/>

Notification of Acceptance: July 29, 2011

The Program Committee may choose to accept certain papers conditionally,
based on a shepherding process.

SUBMISSION REQUIREMENTS:

ANCS 2011 will use single-blind reviewing, so submitted papers should
include the authors' names.

Submit papers using a 10pt font on 12pt leading formatted for printing on
Letter-sized (8.5" by 11") paper. Paper text blocks must follow ACM
guidelines: double-column, with each column 9.25" by 3.33", 0.33" space
between columns. Each column must contain no more than 55 lines of text.
Latex and Word templates, and additional submission requirements, are
available via <http://www.ancsconf.org/>. ANCS submissions must be no longer
than 12 pages. The PC may choose to reject any paper that violates these
format requirements, and no deadline extensions will be allowed for
re-formatting.

We encourage submissions containing original ideas that are relevant to the
scope of ANCS. Like other conferences, ANCS requires that papers not be
submitted simultaneously to any other conferences or publications; that
submissions not be previously published in peer-reviewed conferences or
journals; Papers describing work that was previously published in a peer-
reviewed workshop are allowed, if the authors clearly describe what
significant new content has been included.

All submissions will be treated as confidential prior to publication on the
ANCS 2011 Web site; rejected submissions will be permanently treated as
confidential.

BEST PAPER AWARD AND FAST-TRACK TO TRANSACTIONS ON NETWORKING

ANCS 2011 will feature a best paper award, which will be announced at the
conference. Some of the accepted papers will be fast-tracked as submissions
to the ACM/IEEE Transactions on Networking.

POSTER SESSION:

ANCS 2011 will include a poster session. Submission deadlines and guidelines
will be announced at a later date on the conference web site
<http://www.ancsconf.org/>.

GENERAL CHAIR:
  H. Jonathan Chao, Polytechnic Institute of NYU

PROGRAM CHAIRS:
  Jose Duato, Polytechnic University of Valencia
  Tilman Wolf, University of Massachusetts, Amherst

FINANCE CHAIR:
  N. Sertac Artan, Polytechnic Institute of NYU

PUBLICITY CHAIR:
  Maria Engracia Gomez, Polytechnic University of Valencia
  Bin Liu, Tsinghua University
  Ning Weng, Southern Illinois University at Carbondale

CONTACT INFORMATION:
For questions, please contact the ANCS 2011 program chairs.

---------------------------------------------
Ning Weng, Assistant Professor
Dept. of Electrical & Computer Engineering
Southern Illinois University Carbondale

Office: Engineering E-119
Phone: (618) 453-7645
Fax: (618) 453-7972
Email: nweng <at> siu.edu
WWW: http://www.engr.siu.edu/~weng
---------------------------------------------
Permalink | Reply | 0 comments |
headers
Zama Ques | 23 Feb 2011 07:52
Picon
Favicon

Re: end2end-interest Digest, Vol 83, Issue 4

Hi Yan,

Thanks for your suggestion .  I am familiar with iperf but the issue with us that it is a prod network and it is advisable for me not to pump data on the network . Will try to the experiment between two desktops connected by a cross over cable.

What I was trying earlier was that I started FTP server on one end  and connected to the server from the client side.

 $ ftp 10.66.X.X
Connected to 10.66.X.X
220 (vsFTPd 2.2.2)
Name (10.66.74.141:zama): anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.


After that I disconnected the network cable from the server and was monitoring the status of the connection on the client side .
The status of the connection was like this before and after disconnecting the network cable.

---
$  for i in {1..1000} ; do netstat -at | egrep "ftp" ; date  ; sleep 60  ; done
tcp        0      0 edgebeauty.c:50179 shopfrobsoon.c:ftp ESTABLISHED
Wed Feb 23 11:47:53 IST 2011

tcp        0      0 edgebeauty.c:50179 shopfrobsoon.c:ftp ESTABLISHED
Wed Feb 23 11:48:53 IST 2011
tcp        0      0 edgebeauty.c:50179 shopfrobsoon.c:ftp ESTABLISHED
Wed Feb 23 11:49:53 IST 2011
...
...
Wed Feb 23 12:14:03 IST 2011
tcp        0      0 edgebeauty.c:50179 shopfrobsoon.c:ftp ESTABLISHED
Wed Feb 23 12:15:03 IST 2011
===

If we see that the time is more than 25 minutes when the server went down and the client has still maintained the connection in established state.

My understanding is that the client should close the connection after TCP restarsmit timeout happens or my understanding is wrong.

Please clarify .

--Zaman


Message: 2
Date: Tue, 22 Feb 2011 09:55:13 -0500
From: Yan Cai <ycai <at> ecs.umass.edu>
Subject: Re: [e2e] query on behaviour of tcp_keepalive and tcp
    retransmit on    Linux based systems
To: end2end-interest <at> postel.org
Message-ID: <4D63CE50.8050606 <at> ecs.umass.edu>
Content-Type: text/plain; charset="iso-8859-1"

Hi

According to your description, the expected behavior should be as follows.
At the beginning senders at one side can send data to the receivers at
the other side, and the receivers can receive data without any problem.
When some of the receivers become off-line, the affected senders should
no long receive positive acknowledgments, therefore, lowering their
congestion windows (i.e., sending rate). Since in your case the receiver
is off forever, some senders should further experience timeout events.
After a few timeouts, the sender should CLOSE this connection itself.

As far as I know, the whole procedure above should be automatically
invoked in the sender side. This is how TCP (sender) handles exceptions.

My suggestion is that you run a simple experiment on your side to see if
TCP in your machine can work that way. The test can be done using i-perf
to send a long long live TCP flow, and then take off the receiver in the
middle of the transmission. The connection is expected to be closed very
soon after the receiver is off.

Hope it helpful.
Yan
On 2/22/2011 4:24 AM, Zama Ques wrote:
> We need some clarifications on TCP_keepalive .  We are facing some
> issues on our Prod servers related to TCP functionality .
>
> The issue is like this.
>
> We have some machines at one end sending data in real time to another
> group of machines on the other hand .  Now due to some hardware issues
> on the other hand , some of the machines becomes unresponsive/crashes.
> The client system which pumps data never came to know that the server
> went unresponsive . The connection remains in
> ESTABLISHED state and the client always tries to send data thinking
> that the connection is alive because of which we are seeing backlog on
> client sides.
>
> Our understanding is like this on how TCP will handle the connection.
>
>
> Q 1) Since  the server went down , the client will try to the
> retransmit the data until it times out. What is the behavior of TCP
> after the timeout? Need clarification on
> the following things.
> a) Will the kernel will close the established connection after the
> timeout . Looks like no in our case as we still see the connection
> still in ESTABLISHED state after around more
> than 2 hours.
> b) Are there any kernel parameters which decides the when the client
> is timeout after retransmission fails. What is the behavior of TCP
> after the client retransmission timeouts.
>
>
> Q 2 ) There is something called tcp_keepalive which if implemented in
> the kernel , by default it's there and comes to be around 2 hrs 2
> minsutes , i think  ,  the client will send some TCP probes after the
> keepalive time ineterval and if it cannot reach the server , then the
> established connection in the client side will be closed by the kernel
> . This is my understanding. But I can see that the connection still
> remains in established after the tcp_keepalive time . We waited for
> around 2 hrs 30 minutes but the connection remains in established
> state only. Tried reducing the keepalive time to be around 10 minutes
> , but the connection remains in ESTABLISHED state in client side .
>
>
> Where I went wrong .Please clarify my doubts raised above . What
> should we do to resolve the problem we are seeing above . Any help
> will be highly appreciated as we are going through a hard time to
> resolve the issue .
>
> Thanks in Advance
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.postel.org/pipermail/end2end-interest/attachments/20110222/50be8540/attachment-0001.html

------------------------------

_______________________________________________
end2end-interest mailing list
end2end-interest <at> postel.org
http://mailman.postel.org/mailman/listinfo/end2end-interest


End of end2end-interest Digest, Vol 83, Issue 4
***********************************************

Permalink | Reply | 0 comments |
headers
Detlef Bosau | 24 Feb 2011 12:06
Picon

Re: query on behaviour of tcp_keepalive and tcp retransmit on Linux based systems

First of all, I'm not quite sure whether this is the right list for Linux specific issues.

Second, you should have a look at the basic function of TCP. When a listening socket dies, the sender will continuously face timeouts. With the following consequences:
1. The sending window shrinks to 1 segment.
2. The RTO is doubled, each time a sent packet is not acknowledged in time. Whether RTO is limited or not depends on the implementation.
3. The sending socket is shut down after some period of time. (Have a look at the various timeouts in TCP.)

We need some clarifications on TCP_keepalive .  We are facing some issues on our Prod servers related to TCP functionality .

The issue is like this.

We have some machines at one end sending data in real time to another group of machines on the other hand .  Now due to some hardware issues on the other hand , some of the machines becomes unresponsive/crashes. The client system which pumps data never came to know that the server went unresponsive . The connection remains in
ESTABLISHED state and the client always tries to send data thinking that the connection is alive because of which we are seeing backlog on client sides.

Our understanding is like this on how TCP will handle the connection.


Q 1) Since  the server went down , the client will try to the retransmit the data until it times out. What is the behavior of TCP after the timeout? Need clarification on
the following things.
a) Will the kernel will close the established connection after the timeout . Looks like no in our case as we still see the connection still in ESTABLISHED state after around more
than 2 hours.
b) Are there any kernel parameters which decides the when the client is timeout after retransmission fails. What is the behavior of TCP after the client retransmission timeouts.


Q 2 ) There is something called tcp_keepalive which if implemented in the kernel , by default it's there and comes to be around 2 hrs 2 minsutes , i think  ,  the client will send some TCP probes after the keepalive time ineterval and if it cannot reach the server , then the established connection in the client side will be closed by the kernel . This is my understanding. But I can see that the connection still remains in established after the tcp_keepalive time . We waited for around 2 hrs 30 minutes but the connection remains in established state only. Tried reducing the keepalive time to be around 10 minutes , but the connection remains in ESTABLISHED state in client side .


Where I went wrong .Please clarify my doubts raised above . What should we do to resolve the problem we are seeing above . Any help will be highly appreciated as we are going through a hard time to resolve the issue .

Thanks in Advance



-- ------------------------------------------------------------------ Detlef Bosau Galileistraße 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau <at> web.de http://www.detlef-bosau.de ------------------------------------------------------------------
Permalink | Reply | 0 comments |
headers
Zama Ques | 24 Feb 2011 12:06
Picon
Favicon

Re: end2end-interest Digest, Vol 83, Issue 4

Hi Yan,

I tried testing with iperf today.

Started server on one side and connected to the client from another host and after sometime disconnected the cable on the server host.

Also  , reduced tcp_keepalive to 1200 sec , so timeout value should be like 32 minutes with the other two tcp_keepalive related kernel parameter (probes and interval) .


The following are my findings .

I can see that client terminates the ESTABLISHED connection after around 16 minutes since the server is not reachable , that is before the TCP keepalive timeout.

Looks to me like this minutes is somehow related to TCP retransmission timeout which probably is determined by the following 3 parameters which comes to be around 18 minutes. .

$ cat /proc/sys/net/ipv4/tcp_retries1
3
$ cat /proc/sys/net/ipv4/tcp_retries2
15
$ cat /proc/sys/net/ipv4/tcp_fin_timeout
60


Is my assumption correct here ?


The following is the netstats connection flow during my experiment

$  for i in {1..1000} ; do netstat -atn | egrep "5001" ; date  ; sleep 60  ; done
tcp        0 447432 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
tcp        0      0 10.66.X.Y:43531            10.66.A.B:5001           TIME_WAIT  
Thu Feb 24 14:47:16 IST 2011
tcp        0 3311576 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:48:16 IST 2011
(Network Cable removed during this time from the server)
tcp        0 3317368 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:49:16 IST 2011
tcp        0 3021976 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:50:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:51:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:52:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:53:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:54:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:55:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:56:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:57:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:58:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:59:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 15:00:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 15:01:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 15:02:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 15:03:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 15:04:17 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
.. (comes out be arnd 15 minutes from the server went unreachable when the connection status changed in client) 

Thu Feb 24 15:05:17 IST 2011
Thu Feb 24 15:06:17 IST 2011
Thu Feb 24 15:07:17 IST 2011


The following packet flow can be seen on client as sniffed by tcpdump during while I removed the network cable .

=====
14:50:20.158794 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1801285561:1801350721, ack 0, win 92, options [nop,nop,TS val 172872553 ecr 177048109], length 65160
14:50:20.164550 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1801350721:1801415881, ack 0, win 92, options [nop,nop,TS val 172872558 ecr 177048115], length 65160

14:50:20.394916 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172872992 ecr 177048116], length 1448
14:50:21.258921 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172873856 ecr 177048116], length 1448
14:50:22.986922 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172875584 ecr 177048116], length 1448
14:50:26.442922 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172879040 ecr 177048116], length 1448
14:50:33.354923 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172885952 ecr 177048116], length 1448
14:50:47.178932 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172899776 ecr 177048116], length 1448
14:51:14.826929 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172927424 ecr 177048116], length 1448
14:52:10.122922 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172982720 ecr 177048116], length 1448
14:54:00.714934 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 173093312 ecr 177048116], length 1448
14:56:00.714921 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 173213312 ecr 177048116], length 1448

14:58:00.714920 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 173333312 ecr 177048116], length 1448
15:00:00.714921 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 173453312 ecr 177048116], length 1448
15:02:00.714921 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 173573312 ecr 177048116], length 1448
15:04:00.714936 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 173693312 ecr 177048116], length 1448


 Does my TCP stack look fine based on the experiments above .


Thanks
Zaman






--- On Wed, 23/2/11, Yan Cai <ycai <at> ecs.umass.edu> wrote:

From: Yan Cai <ycai <at> ecs.umass.edu>
Subject: Re: end2end-interest Digest, Vol 83, Issue 4
To: "Zama Ques" <queszama <at> yahoo.in>
Date: Wednesday, 23 February, 2011, 2:04 PM

Hi Zaman,

I guess there might be some unknown ftp configuration at CLIENT side that causes this issue. You can isolate the problem first. I-perf can be used to test functionality of tcp stack on your machine. If it works as expected, then there is nothing wrong with tcp stack. Next check the settings of the ftp client (not the ftp server) to see if there is any specific configuration that causes this problem. If it is hard to do that, my suggestion is to install a third party ftp client application and test with that.

If none of them works, you might have to trace the traffic over the cable attached to the client machine and determine what is going on.

Best wishes,
Yan

On 2/23/2011 1:52 AM, Zama Ques wrote:
Hi Yan,

Thanks for your suggestion .  I am familiar with iperf but the issue with us that it is a prod network and it is advisable for me not to pump data on the network . Will try to the experiment between two desktops connected by a cross over cable.

What I was trying earlier was that I started FTP server on one end  and connected to the server from the client side.

 $ ftp 10.66.X.X
Connected to 10.66.X.X
220 (vsFTPd 2.2.2)
Name (10.66.74.141:zama): anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.


After that I disconnected the network cable from the server and was monitoring the status of the connection on the client side .
The status of the connection was like this before and after disconnecting the network cable.

---
$  for i in {1..1000} ; do netstat -at | egrep "ftp" ; date  ; sleep 60  ; done
tcp        0      0 edgebeauty.c:50179 shopfrobsoon.c:ftp ESTABLISHED
Wed Feb 23 11:47:53 IST 2011

tcp        0      0 edgebeauty.c:50179 shopfrobsoon.c:ftp ESTABLISHED
Wed Feb 23 11:48:53 IST 2011
tcp        0      0 edgebeauty.c:50179 shopfrobsoon.c:ftp ESTABLISHED
Wed Feb 23 11:49:53 IST 2011
...
...
Wed Feb 23 12:14:03 IST 2011
tcp        0      0 edgebeauty.c:50179 shopfrobsoon.c:ftp ESTABLISHED
Wed Feb 23 12:15:03 IST 2011
===

If we see that the time is more than 25 minutes when the server went down and the client has still maintained the connection in established state.

My understanding is that the client should close the connection after TCP restarsmit timeout happens or my understanding is wrong.

Please clarify .

--Zaman


Message: 2
Date: Tue, 22 Feb 2011 09:55:13 -0500
From: Yan Cai <ycai <at> ecs.umass.edu>
Subject: Re: [e2e] query on behaviour of tcp_keepalive and tcp
    retransmit on    Linux based systems
To: end2end-interest <at> postel.org
Message-ID: <4D63CE50.8050606 <at> ecs.umass.edu>
Content-Type: text/plain; charset="iso-8859-1"

Hi

According to your description, the expected behavior should be as follows.
At the beginning senders at one side can send data to the receivers at
the other side, and the receivers can receive data without any problem.
When some of the receivers become off-line, the affected senders should
no long receive positive acknowledgments, therefore, lowering their
congestion windows (i.e., sending rate). Since in your case the receiver
is off forever, some senders should further experience timeout events.
After a few timeouts, the sender should CLOSE this connection itself.

As far as I know, the whole procedure above should be automatically
invoked in the sender side. This is how TCP (sender) handles exceptions.

My suggestion is that you run a simple experiment on your side to see if
TCP in your machine can work that way. The test can be done using i-perf
to send a long long live TCP flow, and then take off the receiver in the
middle of the transmission. The connection is expected to be closed very
soon after the receiver is off.

Hope it helpful.
Yan
On 2/22/2011 4:24 AM, Zama Ques wrote:
> We need some clarifications on TCP_keepalive .  We are facing some
> issues on our Prod servers related to TCP functionality .
>
> The issue is like this.
>
> We have some machines at one end sending data in real time to another
> group of machines on the other hand .  Now due to some hardware issues
> on the other hand , some of the machines becomes unresponsive/crashes.
> The client system which pumps data never came to know that the server
> went unresponsive . The connection remains in
> ESTABLISHED state and the client always tries to send data thinking
> that the connection is alive because of which we are seeing backlog on
> client sides.
>
> Our understanding is like this on how TCP will handle the connection.
>
>
> Q 1) Since  the server went down , the client will try to the
> retransmit the data until it times out. What is the behavior of TCP
> after the timeout? Need clarification on
> the following things.
> a) Will the kernel will close the established connection after the
> timeout . Looks like no in our case as we still see the connection
> still in ESTABLISHED state after around more
> than 2 hours.
> b) Are there any kernel parameters which decides the when the client
> is timeout after retransmission fails. What is the behavior of TCP
> after the client retransmission timeouts.
>
>
> Q 2 ) There is something called tcp_keepalive which if implemented in
> the kernel , by default it's there and comes to be around 2 hrs 2
> minsutes , i think  ,  the client will send some TCP probes after the
> keepalive time ineterval and if it cannot reach the server , then the
> established connection in the client side will be closed by the kernel
> . This is my understanding. But I can see that the connection still
> remains in established after the tcp_keepalive time . We waited for
> around 2 hrs 30 minutes but the connection remains in established
> state only. Tried reducing the keepalive time to be around 10 minutes
> , but the connection remains in ESTABLISHED state in client side .
>
>
> Where I went wrong .Please clarify my doubts raised above . What
> should we do to resolve the problem we are seeing above . Any help
> will be highly appreciated as we are going through a hard time to
> resolve the issue .
>
> Thanks in Advance
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.postel.org/pipermail/end2end-interest/attachments/20110222/50be8540/attachment-0001.html

------------------------------

_______________________________________________
end2end-interest mailing list
end2end-interest <at> postel.org
http://mailman.postel.org/mailman/listinfo/end2end-interest


End of end2end-interest Digest, Vol 83, Issue 4
***********************************************



Permalink | Reply | 1 comment |
headers
Yan Cai | 24 Feb 2011 15:54
Picon

Re: end2end-interest Digest, Vol 83, Issue 4

Hi Zama,

Thanks for your detailed info. Yes. the tcp stack looks good. One quick question, what kind of FTP client application is used in your test, the previous one (where the connection issue is found) or a new third party one?

If it is the previous one, then your FTP client should also work well. If you still encounter that issue, there must be something else wrong.

If your test is done with a new third party client, repeat the procedure with the default client you were using.

Another thing you can check is the congestion window size during the period from the time the cable is disconnected to the time the ESTABLISHED connection is done. Notice that the congestion window size is hard to measure directly, but you can do this by analyzing the trace file for the period mentioned above. Use tcpdump to monitor a particular NIC port and analyze the throughput for that particular FTP flow. The amount of transmitted data over that PERIOD should be very small because the congestion window during timeout is around 1 (in terms of packets).

If all the things above work normally, you should not have seen the issues observed before.

Best wishes,
Yan

On 2/24/2011 6:06 AM, Zama Ques wrote:
Hi Yan,

I tried testing with iperf today.

Started server on one side and connected to the client from another host and after sometime disconnected the cable on the server host.

Also  , reduced tcp_keepalive to 1200 sec , so timeout value should be like 32 minutes with the other two tcp_keepalive related kernel parameter (probes and interval) .


The following are my findings .

I can see that client terminates the ESTABLISHED connection after around 16 minutes since the server is not reachable , that is before the TCP keepalive timeout.

Looks to me like this minutes is somehow related to TCP retransmission timeout which probably is determined by the following 3 parameters which comes to be around 18 minutes. .

$ cat /proc/sys/net/ipv4/tcp_retries1
3
$ cat /proc/sys/net/ipv4/tcp_retries2
15
$ cat /proc/sys/net/ipv4/tcp_fin_timeout
60


Is my assumption correct here ?


The following is the netstats connection flow during my experiment

$  for i in {1..1000} ; do netstat -atn | egrep "5001" ; date  ; sleep 60  ; done
tcp        0 447432 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
tcp        0      0 10.66.X.Y:43531            10.66.A.B:5001           TIME_WAIT  
Thu Feb 24 14:47:16 IST 2011
tcp        0 3311576 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:48:16 IST 2011
(Network Cable removed during this time from the server)
tcp        0 3317368 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:49:16 IST 2011
tcp        0 3021976 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:50:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:51:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:52:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:53:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:54:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:55:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:56:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:57:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:58:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 14:59:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 15:00:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 15:01:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 15:02:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 15:03:16 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
Thu Feb 24 15:04:17 IST 2011
tcp        0 2511048 10.66.X.Y:43533            10.66.A.B:5001           ESTABLISHED
.. (comes out be arnd 15 minutes from the server went unreachable when the connection status changed in client) 

Thu Feb 24 15:05:17 IST 2011
Thu Feb 24 15:06:17 IST 2011
Thu Feb 24 15:07:17 IST 2011


The following packet flow can be seen on client as sniffed by tcpdump during while I removed the network cable .

=====
14:50:20.158794 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1801285561:1801350721, ack 0, win 92, options [nop,nop,TS val 172872553 ecr 177048109], length 65160
14:50:20.164550 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1801350721:1801415881, ack 0, win 92, options [nop,nop,TS val 172872558 ecr 177048115], length 65160

14:50:20.394916 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172872992 ecr 177048116], length 1448
14:50:21.258921 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172873856 ecr 177048116], length 1448
14:50:22.986922 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172875584 ecr 177048116], length 1448
14:50:26.442922 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172879040 ecr 177048116], length 1448
14:50:33.354923 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172885952 ecr 177048116], length 1448
14:50:47.178932 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172899776 ecr 177048116], length 1448
14:51:14.826929 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172927424 ecr 177048116], length 1448
14:52:10.122922 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 172982720 ecr 177048116], length 1448
14:54:00.714934 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 173093312 ecr 177048116], length 1448
14:56:00.714921 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 173213312 ecr 177048116], length 1448

14:58:00.714920 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 173333312 ecr 177048116], length 1448
15:00:00.714921 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 173453312 ecr 177048116], length 1448
15:02:00.714921 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 173573312 ecr 177048116], length 1448
15:04:00.714936 IP edgebeauty-dr.43533 > shopfrobsoon-dr.commplex-link: Flags [.], seq 1798969993:1798971441, ack 0, win 92, options [nop,nop,TS val 173693312 ecr 177048116], length 1448


 Does my TCP stack look fine based on the experiments above .


Thanks
Zaman






--- On Wed, 23/2/11, Yan Cai <ycai <at> ecs.umass.edu> wrote:

From: Yan Cai <ycai <at> ecs.umass.edu>
Subject: Re: end2end-interest Digest, Vol 83, Issue 4
To: "Zama Ques" <queszama <at> yahoo.in>
Date: Wednesday, 23 February, 2011, 2:04 PM

Hi Zaman,

I guess there might be some unknown ftp configuration at CLIENT side that causes this issue. You can isolate the problem first. I-perf can be used to test functionality of tcp stack on your machine. If it works as expected, then there is nothing wrong with tcp stack. Next check the settings of the ftp client (not the ftp server) to see if there is any specific configuration that causes this problem. If it is hard to do that, my suggestion is to install a third party ftp client application and test with that.

If none of them works, you might have to trace the traffic over the cable attached to the client machine and determine what is going on.

Best wishes,
Yan

On 2/23/2011 1:52 AM, Zama Ques wrote:
Hi Yan,

Thanks for your suggestion .  I am familiar with iperf but the issue with us that it is a prod network and it is advisable for me not to pump data on the network . Will try to the experiment between two desktops connected by a cross over cable.

What I was trying earlier was that I started FTP server on one end  and connected to the server from the client side.

 $ ftp 10.66.X.X
Connected to 10.66.X.X
220 (vsFTPd 2.2.2)
Name (10.66.74.141:zama): anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.


After that I disconnected the network cable from the server and was monitoring the status of the connection on the client side .
The status of the connection was like this before and after disconnecting the network cable.

---
$  for i in {1..1000} ; do netstat -at | egrep "ftp" ; date  ; sleep 60  ; done
tcp        0      0 edgebeauty.c:50179 shopfrobsoon.c:ftp ESTABLISHED
Wed Feb 23 11:47:53 IST 2011

tcp        0      0 edgebeauty.c:50179 shopfrobsoon.c:ftp ESTABLISHED
Wed Feb 23 11:48:53 IST 2011
tcp        0      0 edgebeauty.c:50179 shopfrobsoon.c:ftp ESTABLISHED
Wed Feb 23 11:49:53 IST 2011
...
...
Wed Feb 23 12:14:03 IST 2011
tcp        0      0 edgebeauty.c:50179 shopfrobsoon.c:ftp ESTABLISHED
Wed Feb 23 12:15:03 IST 2011
===

If we see that the time is more than 25 minutes when the server went down and the client has still maintained the connection in established state.

My understanding is that the client should close the connection after TCP restarsmit timeout happens or my understanding is wrong.

Please clarify .

--Zaman


Message: 2
Date: Tue, 22 Feb 2011 09:55:13 -0500
From: Yan Cai <ycai <at> ecs.umass.edu>
Subject: Re: [e2e] query on behaviour of tcp_keepalive and tcp
    retransmit on    Linux based systems
To: end2end-interest <at> postel.org
Message-ID: <4D63CE50.8050606 <at> ecs.umass.edu>
Content-Type: text/plain; charset="iso-8859-1"

Hi

According to your description, the expected behavior should be as follows.
At the beginning senders at one side can send data to the receivers at
the other side, and the receivers can receive data without any problem.
When some of the receivers become off-line, the affected senders should
no long receive positive acknowledgments, therefore, lowering their
congestion windows (i.e., sending rate). Since in your case the receiver
is off forever, some senders should further experience timeout events.
After a few timeouts, the sender should CLOSE this connection itself.

As far as I know, the whole procedure above should be automatically
invoked in the sender side. This is how TCP (sender) handles exceptions.

My suggestion is that you run a simple experiment on your side to see if
TCP in your machine can work that way. The test can be done using i-perf
to send a long long live TCP flow, and then take off the receiver in the
middle of the transmission. The connection is expected to be closed very
soon after the receiver is off.

Hope it helpful.
Yan
On 2/22/2011 4:24 AM, Zama Ques wrote:
> We need some clarifications on TCP_keepalive .  We are facing some
> issues on our Prod servers related to TCP functionality .
>
> The issue is like this.
>
> We have some machines at one end sending data in real time to another
> group of machines on the other hand .  Now due to some hardware issues
> on the other hand , some of the machines becomes unresponsive/crashes.
> The client system which pumps data never came to know that the server
> went unresponsive . The connection remains in
> ESTABLISHED state and the client always tries to send data thinking
> that the connection is alive because of which we are seeing backlog on
> client sides.
>
> Our understanding is like this on how TCP will handle the connection.
>
>
> Q 1) Since  the server went down , the client will try to the
> retransmit the data until it times out. What is the behavior of TCP
> after the timeout? Need clarification on
> the following things.
> a) Will the kernel will close the established connection after the
> timeout . Looks like no in our case as we still see the connection
> still in ESTABLISHED state after around more
> than 2 hours.
> b) Are there any kernel parameters which decides the when the client
> is timeout after retransmission fails. What is the behavior of TCP
> after the client retransmission timeouts.
>
>
> Q 2 ) There is something called tcp_keepalive which if implemented in
> the kernel , by default it's there and comes to be around 2 hrs 2
> minsutes , i think  ,  the client will send some TCP probes after the
> keepalive time ineterval and if it cannot reach the server , then the
> established connection in the client side will be closed by the kernel
> . This is my understanding. But I can see that the connection still
> remains in established after the tcp_keepalive time . We waited for
> around 2 hrs 30 minutes but the connection remains in established
> state only. Tried reducing the keepalive time to be around 10 minutes
> , but the connection remains in ESTABLISHED state in client side .
>
>
> Where I went wrong .Please clarify my doubts raised above . What
> should we do to resolve the problem we are seeing above . Any help
> will be highly appreciated as we are going through a hard time to
> resolve the issue .
>
> Thanks in Advance
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.postel.org/pipermail/end2end-interest/attachments/20110222/50be8540/attachment-0001.html

------------------------------

_______________________________________________
end2end-interest mailing list
end2end-interest <at> postel.org
http://mailman.postel.org/mailman/listinfo/end2end-interest


End of end2end-interest Digest, Vol 83, Issue 4
***********************************************




Permalink | Reply | 0 comments |

Gmane