Erik Hugne | 23 Jan 17:35 2015
Picon

[erik.hugne <at> ericsson.com: [PATCH net-next 1/2] tipc: fix excessive network event logging]

If anyone feel that they will be missing the "Established/lost link, Established/lost contact with node"
traces, i have pushed a userspace daemon that does just this to sf/master:
https://sourceforge.net/p/tipc/tipcutils/ci/e68aeecde508bff1b85b12f7360658655a2c7eba/

(it's not release tagged)

//E

----- Forwarded message from erik.hugne <at> ericsson.com -----

Date: Thu, 22 Jan 2015 17:10:31 +0100
From: erik.hugne <at> ericsson.com
To: richard.alpe <at> ericsson.com, netdev <at> vger.kernel.org, jon.maloy <at> ericsson.com, ying.xue <at> windriver.com
CC: tipc-discussion <at> lists.sourceforge.net, Erik Hugne <erik.hugne <at> ericsson.com>
Subject: [PATCH net-next 1/2] tipc: fix excessive network event logging
X-Mailer: git-send-email 2.1.3

From: Erik Hugne <erik.hugne <at> ericsson.com>

If a large number of namespaces is spawned on a node and TIPC is
enabled in each of these, the excessive printk tracing of network
events will cause the system to grind down to a near halt.
The traces are still of debug value, so instead of removing them
completely we fix it by changing the link state and node availability
logging debug traces.
[...]

------------------------------------------------------------------------------
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
(Continue reading)

Jon Maloy | 22 Jan 20:47 2015
Picon

[PATCH net-next v6 0/9] tipc: resolve message disordering problem

When TIPC receives messages from multi-threaded device drivers it may
occasionally deliver messages to their destination sockets in the wrong
order. This happens despite correct resequencing at the link layer,
because the upcall path from link to socket is not protected by any
locks.

These commits solve this problem by introducing an 'input' message
queue in each link, though which messages must be delivered to the
upper layers.

v3:
   - Commit #1:    Changed names on some stack variables, as suggested by
                   Ying
   - Commit #2/3:  Moved introduction of function -tipc_sk_enqueue_msg()
                   to #2, to make it cleaner. I did NOT re-introduce the
                   'exit' label in tipc_sk_rcv(), because it would end up
                   inside a loop, something I consider an abomination.
   - Commit #6:    Realizing that we may have to kick several input
                   queues at the same reception occasion, I introduced a
                   new bit map 'inputq_map' in node::bclink, indicating
                   which input queues need to be delivered. This is
                   because messages extracted from the same bundle may
                   have different originating port, and will end up
                   in different queues.

v4: - Introduced new commit #3, where we avoid allocating and initializing
'     the socket buffer list for each sent message.
    - Rebased on Yingś series "Standardise TIPC SKB queue operations.
    - Some minor code changes in #4 and #5 based on Ying's feedback
    - Commit #7: Removed the multiqueue approach from v3, and introduced
(Continue reading)

Jon Maloy | 22 Jan 19:52 2015
Picon

[PATCH net-next 0/4] tipc: some minor fixes

During extensive testing and analysis of running dual links between
nodes, we have encountered some issues that potentially may cause
problems. We choose to fix those proactively in this series. 

Jon Maloy (4):
  tipc: avoid stale link after aborted failover
  tipc: add reference count to struct tipc_link
  tipc: eliminate race during node creation
  tipc: separate link starting event from link timeout event

 net/tipc/discover.c |  5 +++--
 net/tipc/link.c     | 15 +++++++++++----
 net/tipc/link.h     |  2 ++
 net/tipc/node.c     |  9 ++++-----
 4 files changed, 20 insertions(+), 11 deletions(-)

--

-- 
1.9.1

------------------------------------------------------------------------------
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet
Jon Maloy | 22 Jan 16:29 2015
Picon

[PATCH net-next 1/1] tipc: Add Ying Xue to TIPC maintainers list

We remove Allans Stephens, who has moved on to other tasks, from
the TIPC maintainers list. We replace him with Ying Xue, who has
been doing the maintenance on behalf of WindRiver since almost
three years.

Acked-by: Ying Xue <ying.xue <at> windriver.com>
Signed-off-by: Jon Maloy <jon.maloy <at> ericsson.com>
---
 MAINTAINERS | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index e1ff4ce..6b5a4de 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
 <at>  <at>  -9633,7 +9633,7  <at>  <at>  F:	include/linux/wl12xx.h

 TIPC NETWORK LAYER
 M:	Jon Maloy <jon.maloy <at> ericsson.com>
-M:	Allan Stephens <allan.stephens <at> windriver.com>
+M:	Ying Xue <ying.xue <at> windriver.com>
 L:	netdev <at> vger.kernel.org (core kernel code)
 L:	tipc-discussion <at> lists.sourceforge.net (user apps, general discussion)
 W:	http://tipc.sourceforge.net/
--

-- 
1.9.1

------------------------------------------------------------------------------
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
(Continue reading)

Erik Hugne | 22 Jan 13:00 2015
Picon

[davem <at> davemloft.net: Re: [PATCH net-next] tipc: ratelimit network event traces]

I'm going to change the pr_xxx_ratelimited to pr_debug instead.
Let me know if you oppose this.

    tipc: fix excessive network event logging

    If a large number of namespaces is spawned on a node and TIPC is
    enabled in each of these, the excessive printk tracing of network
    events will cause the system to grind down to a near halt.
    The link/node state events can be retreived using topology
    subscriptions, but the traces are still of debug value, so
    instead of removing them completely we fix it by changing the link
    state and node availability logging to pr_debug. 

[...]

----- Forwarded message from David Miller <davem <at> davemloft.net> -----

Date: Mon, 19 Jan 2015 19:46:31 -0500
From: David Miller <davem <at> davemloft.net>
To: erik.hugne <at> ericsson.com
CC: richard.alpe <at> ericsson.com, ying.xue <at> windriver.com, jon.maloy <at> ericsson.com,
netdev <at> vger.kernel.org, tipc-discussion <at> lists.sourceforge.net
Subject: Re: [PATCH net-next] tipc: ratelimit network event traces
X-Mailer: Mew version 6.6 on Emacs 24.4 / Mule 6.0 (HANACHIRUSATO)

From: <erik.hugne <at> ericsson.com>
Date: Mon, 19 Jan 2015 10:02:44 +0100

> From: Erik Hugne <erik.hugne <at> ericsson.com>
> 
(Continue reading)

Ying Xue | 21 Jan 04:13 2015

[PATCH] rps: support tipc

TIPC protocol introduces an abstract link layer which provides a
reliable message delivery mechanism for connection-based,
connectionless, as well as multicast messages, so this means that
message sequence control and retransmission are conducted on link
layer instead of common transport layer like TCP or SCTP doing.
Therefore, the target CPU for RPS is determined through calculating
a flow hash over 2-tuple (i.e, previous and destination node address).

However, TIPC message header has several different formats, so we
have to consider them separately:

- Multicast message. In multicast message header, previous node
  address is varied, but destination node address is always 0. To
  make RPS deem all multicast messages as one flow to steer them to
  one CPU, 0xffffffff and 0 are assumed as their previous and
  destination node address respectively.

- Unicast message. Unicast message has two different types:
  connection-oriented and connectionless message. The former contains
  valid source node address, but doesn't include destination node
  address; the latter contains both. To ensure all unicast messages
  with the same destinations to steer one target CPU, source node
  address and 0 are deemed as flow source and destination addresses
  respectively.

Signed-off-by: Ying Xue <ying.xue <at> windriver.com>
---
 net/core/flow_dissector.c |   22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

(Continue reading)

erik.hugne | 19 Jan 16:57 2015
Picon

[PATCH] flow_dissector: add tipc support

From: Erik Hugne <erik.hugne <at> ericsson.com>

Support for (0x88ca) receive hashing. The hash is calculated
over source/dest node. If port information is present it is
also included. Since TIPC ports are 32bit we xor the source
and destport into flow->ports, which should generate consistent
hashes inbound/outbound.
Based on Ying Xue's TIPC/RPS patchset.

Signed-off-by: Erik Hugne <erik.hugne <at> ericsson.com>
CC: Ying Xue <ying.xue <at> windriver.com>
---
On systems with single-queue NIC's such as virtio-net based VM's, all
TIPC softirq load (packet processing) is serialized to the same core
that received the interrupt. This is a huge bottleneck and RPS have
shown to resolve this to a large extent.

 MAINTAINERS               |  1 +
 include/linux/tipc_msg.h  | 91 +++++++++++++++++++++++++++++++++++++++++++++++
 net/core/flow_dissector.c | 27 ++++++++++++++
 net/tipc/msg.h            | 48 +------------------------
 4 files changed, 120 insertions(+), 47 deletions(-)
 create mode 100644 include/linux/tipc_msg.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 9de9005..e833d92 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
 <at>  <at>  -9631,6 +9631,7  <at>  <at>  L:	tipc-discussion <at> lists.sourceforge.net (user apps, general discussion)
 W:	http://tipc.sourceforge.net/
(Continue reading)

erik.hugne | 19 Jan 10:02 2015
Picon

[PATCH net-next] tipc: ratelimit network event traces

From: Erik Hugne <erik.hugne <at> ericsson.com>

If a large number of namespaces is spawned on a node and TIPC is
enabled in each of these, the excessive printk tracing of network
events will cause the system to grind down to a near halt.
We fix this by adding ratelimiting to the info/warning logs
regarding link state and node availability.

Signed-off-by: Erik Hugne <erik.hugne <at> ericsson.com>
Reviewed-by: Ying Xue <ying.xue <at> windriver.com>
---
 net/tipc/link.c | 21 +++++++++++----------
 net/tipc/node.c | 24 +++++++++++++-----------
 2 files changed, 24 insertions(+), 21 deletions(-)

diff --git a/net/tipc/link.c b/net/tipc/link.c
index 193bc15..bedb590 100644
--- a/net/tipc/link.c
+++ b/net/tipc/link.c
 <at>  <at>  -538,8 +538,8  <at>  <at>  static void link_state_event(struct tipc_link *l_ptr, unsigned int event)
 			link_set_timer(l_ptr, cont_intv / 4);
 			break;
 		case RESET_MSG:
-			pr_info("%s<%s>, requested by peer\n", link_rst_msg,
-				l_ptr->name);
+			pr_info_ratelimited("%s<%s>, requested by peer\n",
+					    link_rst_msg, l_ptr->name);
 			tipc_link_reset(l_ptr);
 			l_ptr->state = RESET_RESET;
 			l_ptr->fsm_msg_cnt = 0;
(Continue reading)

Ying Xue | 19 Jan 09:03 2015

[3.19.0-rc4+] rhashtable: BUG kmalloc-2048 (Not tainted): Poison overwritten

On 3.19.0-rc4+, I encountered below error with attached test
case(bind_netlink.c). Please execute the following commands to reproduce
the error:

gcc -Wall -o bind_netlink bind_netlink.c
./bind_netlink 1000

By the way, if we run another test case(bind_tipc.c), the similar issue
will happen on TIPC socket.

Therefore, it seems that the issue is closely associated with rhashtable
instead of specific stacks like netlink or tipc.

Regards,
Ying

root <at> localhost:/mnt# [  135.472382]
=============================================================================
[  135.473303] BUG kmalloc-2048 (Not tainted): Poison overwritten
[  135.473905]
-----------------------------------------------------------------------------
[  135.473905]
[  135.474894] INFO: 0xffff8800115d86d0-0xffff8800115d86d7. First byte
0x11 instead of 0x6b
[  135.475715] INFO: Allocated in sk_prot_alloc+0xcb/0x1b0 age=87 cpu=5
pid=4536
[  135.476381] 	__slab_alloc+0x453/0x4ba
[  135.476745] 	__kmalloc+0x23f/0x370
[  135.477115] 	sk_prot_alloc+0xcb/0x1b0
[  135.477477] 	sk_alloc+0x39/0x120
(Continue reading)

erik.hugne | 16 Jan 17:19 2015
Picon

[PATCH] tipc: ratelimit network event traces

From: Erik Hugne <erik.hugne <at> ericsson.com>

If a large number of namespaces is spawned on a node and TIPC is
enabled in each of these, the excessive printk tracing of network
events will cause the system to grind down to a near halt.
We fix this by adding ratelimiting to the info/warning logs
regarding link state and node availability.

Signed-off-by: Erik Hugne <erik.hugne <at> ericsson.com>
---
 net/tipc/link.c | 21 +++++++++++----------
 net/tipc/node.c | 24 +++++++++++++-----------
 2 files changed, 24 insertions(+), 21 deletions(-)

diff --git a/net/tipc/link.c b/net/tipc/link.c
index 193bc15..bedb590 100644
--- a/net/tipc/link.c
+++ b/net/tipc/link.c
 <at>  <at>  -538,8 +538,8  <at>  <at>  static void link_state_event(struct tipc_link *l_ptr, unsigned int event)
 			link_set_timer(l_ptr, cont_intv / 4);
 			break;
 		case RESET_MSG:
-			pr_info("%s<%s>, requested by peer\n", link_rst_msg,
-				l_ptr->name);
+			pr_info_ratelimited("%s<%s>, requested by peer\n",
+					    link_rst_msg, l_ptr->name);
 			tipc_link_reset(l_ptr);
 			l_ptr->state = RESET_RESET;
 			l_ptr->fsm_msg_cnt = 0;
 <at>  <at>  -549,7 +549,8  <at>  <at>  static void link_state_event(struct tipc_link *l_ptr, unsigned int event)
(Continue reading)

Ying Xue | 16 Jan 09:38 2015

Re: Socket hashtable BUG

Additionally, please try to verify it on the latest net-next tree again
because there are several important new patches:

3721e9c7c194f576fbd30926e98e0abb13c641b5
tipc: remove redundant timer defined in tipc_sock struct

57699a40b4f2694d3ee63fd5e6465ec8f600b620
rhashtable: Fix race in rhashtable_destroy() and use regular work_struct

Of course, the two patches may not definitely connect with the issue.

Regards,
Ying

On 01/16/2015 04:21 PM, Richard Alpe wrote:
> On Fri, Jan 16, 2015 at 11:04:59AM +0800, Ying Xue wrote:
>> On 01/14/2015 09:10 PM, Richard Alpe wrote:
>>> Hi Ying,
>>>
>>> Looks like we have a nasty BUG in the socket list hastables.
>>>
>>> What I did to reproduce this is to create more than 1024 sockets (normally needs
>>> ulimit increase) and then release them. The BUG occurs upon release or when
>>> trying to create new sockets after the release.
>>>
>>> Can you have a look at it? Let me know if you need anything else from my side :)
>>>
>>
>> Thank you for the report.
>>
(Continue reading)


Gmane