Is there a Throttling feature available or planned in EventLogAppender.

If we log to event log a message that could potentially be logged at frequency of one per incoming request to my web application, it could flood the event log with millions of messages if/when things go wrong.  If there was a = way to roll them up every x seconds into a single event log message (with a count of how many messages where rolledup), it would be useful to me.

Is there any such feature? Or workaround?

Regards

Pranav

Thanx.

Dupe filtering sounds very close. Is there a way to “append” a summary to the message itself ? Something like

Event id 1, Msg: Invalid input value : 45 [2 in last 5 sec]

Also I am thinking it would be nicer (though  more complicated) to collapse dupe across a time window. If we check just for the last event then it might still cause event log flooding. For example if an error causes 2 different events (different ids) to be logged and if this error happens too often (say per request) then the dupe checker that checks only with previous error will allow these messages and end up with log flooding.

So here is an example  of what I have in mind.

Example:

Input:

Event id 1, Msg: Invalid input value : 45

Event id 1, Msg: Invalid input value : 45

Event id 1, Msg: Invalid input value : 800                 <= Message is different though event id is same

Event id 2, Msg: Permission denied for user: xyz     <= Different event id

Event id 2, Msg: Permission denied for user: xyz

Output:

Event id 1, Msg: Invalid input value : 45 [2 in last 5 sec]

Event id 1, Msg: Invalid input value : 800 [1 in last  5 sec]

Event id 2, Msg: Permission denied for user: xyz [2 in last 5 sec]

If I can get help on how to add summary to a message, I can implement the dupe check part using a hash.

Thanx

Regards

Pranav

Pranav,

I recently posted a code snippet for a throttled / caching SMTP appender.  I’ve copied it here.  It might serve as the basis for what you need:

Ken Parrish

Gomez, Inc.

__________________________

Ole,

I recently solved a very similar problem.  I had a situation where in the event of a serious error in our server infrastructure, I needed to 'throttle' the volume of e-mail messages sent to our operations team.  In our case, regardless of the number of serious error messages generated, we wanted to limit to 1 e-mail per 5 minutes, and restrict the size of each e-mail to the most recent 10 events plus some context information (e.g. INFO log events).

I created a log4net extension which is an extension to the SMTPAppender class named SMTPCachingAppender.  This appender has a number of control parameters which establish it's behavior:  interval between flush, event count between flush, whether or not to include context information, maximum buffer size.

This appender might not exactly suit your needs, but it does provide a skeleton for creating a similar appender that is fine-tuned for your application.

Below is a copy of the source code and a sample configuration.

Hope this helps.

Ken Parrish

Gomez, Inc.

__________________________

Sample log4net configuration:

<!-- Caching SMTP appender for operational notifications. -->

<appender name="OperationsLogSmtpAppender" type="log4net.Extensions.SmtpCachingAppender, Utilities">

      <to value="ops-notifications <at> myplace.com" />

      <from value=" ops-notifications <at> myplace.com " />

      <subject value="Buffered Operational Notification" />

      <smtpHost value="mySMTPServer.com" />

      <bufferSize value="0" />

      <lossy value="false" />

      <priority value="high" />

      <FlushInterval value="00:05:00" />

      <FlushCount value="0" />

      <MaxBufferSize value="10" />

      <layout type="log4net.Layout.PatternLayout">

            <param name="ConversionPattern" value="%date{yyyy-MM-dd HH:mm:ss.fff} [%-2thread] [%-5level] [%logger] %message%newline%exception" />

      </layout>

</appender>

__________________________

Source code:

using System;

using System.Collections.Generic;

using log4net.Appender;

using log4net.Core;

namespace log4net.Extensions

{

      /// <summary>

      ///         Caching version of the standard log4net SmtpAppender.  This appender will

      ///         cache log events that are to be sent out via Smtp and send them in block.

      ///

      ///         Configuration options:

      ///

      ///         <FlushInterval value="hh:mm:ss" />

      ///               Indicates the periodic interval for log events flushing (e.g. sending and e-mail), specified

      ///               as a time span.  If the value of FlushInterval is 0:0:0 (zero), periodic interval flushing

      ///               is surpressed.  Default value:  0:5:0 (5 minutes).

      ///

      ///         <FlushCount value="x" />

      ///               Indicates the number of log events received by the appender which will trigger flushing

      ///               (e.g. sending and e-mail).  If the value FlushCount is 0, buffer flush triggering based

      ///               the number of log events received is surpressed.  Default value:  20 events.

      ///

      ///         <IncludeContextLogEvents value="[true/false]"/>"

      ///               True if context log events are included in the buffered log event cache.  Cache events are

      ///               those that are of a lower level than that specified for triggering the SmtpCachingAppender.

      ///               False if no context log events are included.  Default value:  true.

      ///

      ///         <MaxBufferSize value="x"/>

      ///               Maximum number of log events, both SmtpCachingAppender events and context events to save

      ///               in the cache buffer.  If more than MaxBufferSize events are received before the flushing

      ///               criteria is met, only the newest MaxBufferSize events are saved in the buffer.  A value

      ///               of 0 indicates no limit in the size of the cache.  Default value:  0 (no limit)

      ///

      ///         Sample SmtpCachingAppender configuration (in addition to all standard SmtpAppender options).

      ///               Note the namespace and assembly name for the appender.

      ///

      ///         <appender name="SmtpCachingAppender" type="log4net.Extensions.SmtpCachingAppender, Utilities">

      ///               . . .

      ///               <FlushInterval value="00:05:00" />

      ///               <FlushCount value="20" />

      ///               <IncludeContextLogEvents value="false"/>"

      ///               <MaxBufferSize value="3"/>

      ///         </appender>

      ///

      /// </summary>

      public class SmtpCachingAppender : SmtpAppender

      {

            // Configuration options

            int _flushCount = 20;

            TimeSpan _flushInterval = new TimeSpan(0, 5, 0);

            bool _includeContextLogEvents = true;

            int _maxBufferSize = 0;

            // Appender state data

            System.Threading.Timer _timer;

            bool _timedFlush = false;

            int _numCachedMessages = 0;

            List<LoggingEvent> _loggingEvents = new List<LoggingEvent>();

            /// <summary>

            ///         TimeSpan indicating the maximum period of time elapsed before sending

            ///         any cached SMTP log events.

            /// </summary>

            public TimeSpan FlushInterval

            {

                  get { return this._flushInterval; }

                  set { this._flushInterval = value; }

            }

            /// <summary>

            ///         Maximium number of SMTP events to cache before sending via SMTP.

            /// </summary>

            public int FlushCount

            {

                  get { return this._flushCount; }

                  set { this._flushCount = value; }

            }

            public int MaxBufferSize

            {

                  get { return this._maxBufferSize; }

                  set { this._maxBufferSize = value; }

            }

            public bool IncludeContextLogEvents

            {

                  get { return this._includeContextLogEvents; }

                  set { this._includeContextLogEvents = value; }

            }

            /// <summary>

            ///         Create a timer that fires to force flushing cached log events

            ///         via SMTP at a specified interval.

            /// </summary>

            public override void ActivateOptions()

            {

                  if (this._flushInterval > TimeSpan.Zero)

                  {

                        this._timer = new System.Threading.Timer(

                              new System.Threading.TimerCallback(OnTimer),

                              null,

                              this._flushInterval,

                              this._flushInterval);

                  }

                  base.ActivateOptions();

            }

            void OnTimer(Object stateInfo)

            {

                  this._timedFlush = true;

                  Flush(true);

            }

            protected override void SendBuffer(LoggingEvent[] events)

            {

                  if (this._includeContextLogEvents)

                  {

                        this._loggingEvents.AddRange(events);

                  }

                  else

                  {

                        this._loggingEvents.Add(events[events.Length - 1]);

                  }

                  if (this._maxBufferSize != 0)

                  {

                        int numRemoved = this._loggingEvents.Count - this._maxBufferSize;

                        if ((numRemoved > 0) && (numRemoved <= this._loggingEvents.Count))

                        {

                              this._loggingEvents.RemoveRange(0, numRemoved);

                        }

                  }

                  this._numCachedMessages++;

                  if (((this._flushCount != 0) && (this._numCachedMessages >= this._flushCount))

                        || this._timedFlush)

                  {

                        if (this._loggingEvents.Count > 0)

                        {

                              LoggingEvent[] bufferedEvents = this._loggingEvents.ToArray();

                              base.SendBuffer(bufferedEvents);

                              this._loggingEvents.Clear();

                        }

                        // Reset cache buffer conditions.

                        this._numCachedMessages = 0;

                        this._timedFlush = false;

                  }

            }

      }

}

__________________________

-----Original Message-----
From: heinztomato [mailto:ole <at> nocarrier.de]
Sent: Tuesday, April 22, 2008 5:32 AM
To: log4net-user <at> logging.apache.org
Subject: Counting number of Errors or Emails sent

Hi there,

I got a task that SHOULD be quite easy but I did not find any solution yet.

Maybe the reason is that google finds a lot of stuff when searching for

count... (impossible to get a non-sql-related site :-D)

But to my problem:

log4net works fine, sending emails when an Error occurs. But sometimes it

happens that because of a major bug thousands of Mails are sent because of

thousands of error messages. That ist NOT very nice especially because those

mails are sent to an recipientgroup: My collegues nearly cut my head off

last time they received 10.000 mails in a day B-)

So I want to create a Watchdog which stops my service (which sends those

mails) if a number of Errors had occured in a specific time (or as an

alternative if a number of Mails had been sent)

Any Hints how to achieve that goal? (A really good link would be helpful,

too)

Thanks in advance.

Ole

--

View this message in context: http://www.nabble.com/Counting-number-of-Errors-or-Emails-sent-tp16823745p16823745.html

Sent from the Log4net - Users mailing list archive at Nabble.com.

Thanx.

Dupe filtering sounds very close. Is there a way to “append” a summary to the message itself ? Something like

Event id 1, Msg: Invalid input value : 45 [2 in last 5 sec]

Also I am thinking it would be nicer (though  more complicated) to collapse dupe across a time window. If we check just for the last event then it might still cause event log flooding. For example if an error causes 2 different events (different ids) to be logged and if this error happens too often (say per request) then the dupe checker that checks only with previous error will allow these messages and end up with log flooding.

So here is an example  of what I have in mind.

Example:

Input:

Event id 1, Msg: Invalid input value : 45

Event id 1, Msg: Invalid input value : 45

Event id 1, Msg: Invalid input value : 800                 <= Message is different though event id is same

Event id 2, Msg: Permission denied for user: xyz     <= Different event id

Event id 2, Msg: Permission denied for user: xyz

Output:

Event id 1, Msg: Invalid input value : 45 [2 in last 5 sec]

Event id 1, Msg: Invalid input value : 800 [1 in last  5 sec]

Event id 2, Msg: Permission denied for user: xyz [2 in last 5 sec]

If I can get help on how to add summary to a message, I can implement the dupe check part using a hash.

Thanx

Regards

Pranav