How to override a gt score rule

Hello,

I have the following rule being hit:

[Thu Sep 01 10:50:00 2011] [error] [client 123.123.123.123] ModSecurity: Warning. Pattern match
"([\\\\~\\\\!\\\\ <at> \\\\#\\\\$\\\\%\\\\^\\\\&\\\\*\\\\(\\\\)\\\\-\\\\+\\\\=\\\\{\\\\}\\\\[\\\\]\\\\|\\\\:\\\\;\\"\\\\'\\\\\\xc2\\xb4\\\\\\xe2\\x80\\x99\\\\\\xe2\\x80\\x98\\\\`\\\\<\\\\>].*){4,}"
at REQUEST_FILENAME. [file
"/usr/local/httpd-2.2.19/modsecurity/rules/modsecurity_crs_41_sql_injection_attacks.conf"]
[line "523"] [id "981173"] [rev "2.2.2"] [msg "Restricted SQL Character Anomaly Detection Alert - Total
# of special characters exceeded"] [data "-oops/"] [hostname "www.somedomain.com"] [uri
"/case-studies/text/this-is-a-long-path-oops/"] [unique_id "Tl <at> bmH8eCIcAAC3jAbwAAAAC"]

What is the best way to override it without having to completely disable it ? I assume there is a way to
increase the  <at> gt score without modifying the rule directly ?
--

-- 
Thanks, OS

------------------------------------------------------------------------------
Special Offer -- Download ArcSight Logger for FREE!
Finally, a world-class log management solution at an even better 
price-free! And you'll get a free "Love Thy Logs" t-shirt when you
download Logger. Secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsisghtdev2dev
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php

Re: How to override a gt score rule

Is one right to assume the rule below triggers when say a '-' appears four or more times ? How can this be
increased to say six by using an override as I do not see a variable to change?
-- 
Thanks, OS
----- Original Message ----- 

From: "Organic Spider" <webmaster <at> organicspider.co.uk> 
To: mod-security-users <at> lists.sourceforge.net 
Sent: Thursday, 1 September, 2011 3:53:12 PM 
Subject: [mod-security-users] How to override a gt score rule 

Hello, 

I have the following rule being hit: 

[Thu Sep 01 10:50:00 2011] [error] [client 123.123.123.123] ModSecurity: Warning. Pattern match
"([\\\\~\\\\!\\\\ <at> \\\\#\\\\$\\\\%\\\\^\\\\&\\\\*\\\\(\\\\)\\\\-\\\\+\\\\=\\\\{\\\\}\\\\[\\\\]\\\\|\\\\:\\\\;\\"\\\\'\\\\\\xc2\\xb4\\\\\\xe2\\x80\\x99\\\\\\xe2\\x80\\x98\\\\`\\\\<\\\\>].*){4,}"
at REQUEST_FILENAME. [file
"/usr/local/httpd-2.2.19/modsecurity/rules/modsecurity_crs_41_sql_injection_attacks.conf"]
[line "523"] [id "981173"] [rev "2.2.2"] [msg "Restricted SQL Character Anomaly Detection Alert - Total
# of special characters exceeded"] [data "-oops/"] [hostname "www.somedomain.com"] [uri
"/case-studies/text/this-is-a-long-path-oops/"] [unique_id "Tl <at> bmH8eCIcAAC3jAbwAAAAC"] 

What is the best way to override it without having to completely disable it ? I assume there is a way to
increase the  <at> gt score without modifying the rule directly ? 

------------------------------------------------------------------------------
Special Offer -- Download ArcSight Logger for FREE!
Finally, a world-class log management solution at an even better 
price-free! And you'll get a free "Love Thy Logs" t-shirt when you

Re: How to override a gt score rule

On 9/1/11 2:56 PM, "Organic Spider" <webmaster <at> organicspider.co.uk> wrote:

>Is one right to assume the rule below triggers when say a '-' appears
>four or more times ? How can this be increased to say six by using an
>override as I do not see a variable to change?

We have added capabilities to externally update both the TARGET and ACTION
lists for rules (SecRuleUpdateTargetById and SecRuleUpdateActionByID) but
not for the OPERATOR argument.

For this type of issue, may have to edit the rule itself to increase the
regex repetition threshold to an appropriate limit.

One thing that we might look to update is to have different meta-chars
listed when inspected the REQUEST_FILENAME variable as - and _ don't
really cause any problems.

-Ryan

>--
>Thanks, OS
>----- Original Message -----
>
>From: "Organic Spider" <webmaster <at> organicspider.co.uk>
>To: mod-security-users <at> lists.sourceforge.net
>Sent: Thursday, 1 September, 2011 3:53:12 PM
>Subject: [mod-security-users] How to override a gt score rule
>
>Hello,

Re: How to override a gt score rule

Or, re-write the rule into a custom list ? One item I fell fowl of today was downloading latest SVN release of
the sql_injection rules; where I had overrides in place yet the original rule IDs had changed :( When that
happens is it possible for it to be documented ?
-- 
Thanks, OS
----- Original Message ----- 

From: "Ryan Barnett" <RBarnett <at> trustwave.com> 
To: "Organic Spider" <webmaster <at> organicspider.co.uk>, mod-security-users <at> lists.sourceforge.net 
Sent: Thursday, 1 September, 2011 8:03:55 PM 
Subject: Re: [mod-security-users] How to override a gt score rule 

On 9/1/11 2:56 PM, "Organic Spider" <webmaster <at> organicspider.co.uk> wrote: 

>Is one right to assume the rule below triggers when say a '-' appears 
>four or more times ? How can this be increased to say six by using an 
>override as I do not see a variable to change? 

We have added capabilities to externally update both the TARGET and ACTION 
lists for rules (SecRuleUpdateTargetById and SecRuleUpdateActionByID) but 
not for the OPERATOR argument. 

For this type of issue, may have to edit the rule itself to increase the 
regex repetition threshold to an appropriate limit. 

One thing that we might look to update is to have different meta-chars 
listed when inspected the REQUEST_FILENAME variable as - and _ don't 
really cause any problems. 

-Ryan 

PCRE Limits

Hello list,

is there a way to raise the PCRE Limits for a particular
request/argument, something like :

SecRule ARGS_NAMES|ARGS " <at> streq ARGUMENT"
"phase:2,log,ctl:PCREMATCHLIMIT=99999"

--
Slobodan

------------------------------------------------------------------------------
Special Offer -- Download ArcSight Logger for FREE!
Finally, a world-class log management solution at an even better 
price-free! And you'll get a free "Love Thy Logs" t-shirt when you
download Logger. Secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsisghtdev2dev
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php

Re: PCRE Limits

------------------------------------------------------------------------------
Special Offer -- Download ArcSight Logger for FREE!
Finally, a world-class log management solution at an even better 
price-free! And you'll get a free "Love Thy Logs" t-shirt when you
download Logger. Secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsisghtdev2dev

_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php

Re: Failed to access DBM file (data/ip): Resource deadlock avoided

Dear all,
It's been a while since I posted this issue and I want to share with
you some of my experiences as I'm seeing issues when the IP collection
grows in size.

My setup is Apache 2.2.20, ModSecurity 2.6.1 and CRS 2.2.1.
After a lot of performance testing and configuring ModSecurity to do 1
job (protect against a DoS against the web servers [which in turn
would protect against the real assets that ModSecurity setup is
proxying for]) on a "2CPU", 7Gb box (in EC2, m1.large) I get somewhere
between 200 and 300 hits per second against a static resource served
from Apache.  Not great, but acceptable.  What isn't acceptable is the
wildly fluctuating figures I get and load hitting 200+ when the IP
data collection grows in size.

My performance tests are replaying existing logs with real IP and URIs
in, but I use the IP to set an X-Forwarded-For header (so ModSec sees
it as a new IP), it uses the hash of the UA to give a level of
uniqueness to the IP as per the CRS configs and I rewrite all URLs to
a static file (to ensure I'm not running stuff against live in any
shape or form).

All is good at the beginning of the run and my IP collection database
is at 0Mb.  I get around 300 hits per second at a load avg of around
0.4.  Over a short period of time (at the rate mentioned above), this
IP collection builds up and when it hits a certain size, its game
over.  High load starts around 5-10 minutes.  Leaving this running for
20 mins sees load shoot up to 200+.  This isn't a gradual increase, it
hits a point and fails.  This seems to be file size as interrogating
the SDBM file lists for the IP collections it is a small figure.  For
example, I have a 700Mb SDBM file and inside are only 28 IP
collections.

I've played with TIMEOUT values to see if keeping less is helpful and
won't affect things too much, but ideally for DoS protection I'd hope
that keeping the IP in the database for as long as possible will be
more beneficial.

Any help is appreciated - as a potential mitigation step against a
Layer 7 DDoS (network and other good stuff in place to protect against
this aside) - what can I do from here?

Cheers,

Kev

On 19 August 2011 13:37, Ryan Barnett <RBarnett <at> trustwave.com> wrote:
>
> On 8/19/11 6:55 AM, "Kevin Jackson" <kevin <at> linuxservices.co.uk> wrote:
>
>>Dear all,
>>After a number of weeks fine tuning the CRS rules appropriate for a
>>busy website, we did a full live test pushing traffic through Apache
>>with ModSecurity 2.6.1 proxying requests to evaluate the performance
>>of ModSecurity through the backend service.
>>
>>Traffic is around 1k/second and we had Apache running on 18 servers
>>sitting behind a load balancer and Apache coping without issue.
>>
>>ModSecurity on the other hand didn't fare so well - it struggled to
>>keep pace causing resource deadlock issues on the ip database.  The
>>upshot was incredibly high load averages for obvious reasons and poor
>>user experience.
>>
>>--ba66410d-H--
>>Message: Failed to access DBM file "/usr/local/apache/data/ip":
>>Resource deadlock avoided
>>Apache-Handler: proxy-server
>>Stopwatch: 1313660329409820 283469 (- - -)
>>Stopwatch2: 1313660329409820 283469; combined=99526, p1=3446,
>>p2=12838, p3=3, p4=0, p5=41666, sr=3272, sw=41573, l=0, gc=0
>>Producer: ModSecurity for Apache/2.6.1 (http://www.modsecurity.org/);
>>core ruleset/2.2.1.
>>Server: Apache
>>
>>--ba66410d-Z--
>>
>>My question is what to suggest here? People must be running
>>ModSecurity on high volume websites? Have you encountered this issue?
>>This traffic is a fraction of our peak traffic at a quiet time during
>>the day and the environment only coped for about 10 minutes before
>>keeling over.
>
> Hey Kevin,
> Question for you - are you using the IP persistent collection to track
> data about clients across requests?  If not, then you may want to disable
> the initcol rules at the end of the modsecurity_crs_10_config.conf file.
>
> See if that helps.
>
> -Ryan
>
>>
>>Cheers,
>>
>>Kev
>>
>>--------------------------------------------------------------------------
>>----
>>Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
>>user administration capabilities and model configuration. Take
>>the hassle out of deploying and managing Subversion and the
>>tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>>_______________________________________________
>>mod-security-users mailing list
>>mod-security-users <at> lists.sourceforge.net
>>https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>ModSecurity Services from Trustwave's SpiderLabs:
>>https://www.trustwave.com/application-security.php
>>
>
>
> This transmission may contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution, or use of the information contained herein (including any reliance
thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately
contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.
>
>

------------------------------------------------------------------------------
Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage 
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php

Re: Failed to access DBM file (data/ip): Resource deadlock avoided

------------------------------------------------------------------------------
Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage 
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/

_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php

Re: Failed to access DBM file (data/ip): Resource deadlock avoided

Hello,

the size is also a problem for the session collection, i need for the
persistence of 15000 sessions around 1GB Storage (around 64k per
session).
Is it possible to reduce this or is this really needed?
I use a modified version of modsecurity_crs_16_session_hijacking.conf
from crs 2.2.1 where i also save the data to a backend and there the
size is not that big.

Thanks in advance
Michael

2011/9/7 Breno Silva <breno.silva <at> gmail.com>:
> Hi Kevin,
>
> We are planning to implement  a DDoS learning algorithm for ModSecurity. I
> know using data collection in heavy systems can cause performance issues.
> Most of Apache modules which works for  bandwidth limitation also stores
> data into temporary files, and it will also cause performance problems. So
> we need some code to store network information only in memory, persist only
> the learned data for recovery. It will also requires a time-to-time cleanup
> of internal modsecurity data structs. So i think in the future we will have
> a good solution.
>
> Until we do not have a better solution, maybe you could try limit per ip
> connections in a another network device (firewall, sw) ?
>
> Thanks
>
> Breno
>
> On Wed, Sep 7, 2011 at 10:42 AM, Kevin Jackson <kevin <at> linuxservices.co.uk>
> wrote:
>>
>> Dear all,
>> It's been a while since I posted this issue and I want to share with
>> you some of my experiences as I'm seeing issues when the IP collection
>> grows in size.
>>
>> My setup is Apache 2.2.20, ModSecurity 2.6.1 and CRS 2.2.1.
>> After a lot of performance testing and configuring ModSecurity to do 1
>> job (protect against a DoS against the web servers [which in turn
>> would protect against the real assets that ModSecurity setup is
>> proxying for]) on a "2CPU", 7Gb box (in EC2, m1.large) I get somewhere
>> between 200 and 300 hits per second against a static resource served
>> from Apache.  Not great, but acceptable.  What isn't acceptable is the
>> wildly fluctuating figures I get and load hitting 200+ when the IP
>> data collection grows in size.
>>
>> My performance tests are replaying existing logs with real IP and URIs
>> in, but I use the IP to set an X-Forwarded-For header (so ModSec sees
>> it as a new IP), it uses the hash of the UA to give a level of
>> uniqueness to the IP as per the CRS configs and I rewrite all URLs to
>> a static file (to ensure I'm not running stuff against live in any
>> shape or form).
>>
>> All is good at the beginning of the run and my IP collection database
>> is at 0Mb.  I get around 300 hits per second at a load avg of around
>> 0.4.  Over a short period of time (at the rate mentioned above), this
>> IP collection builds up and when it hits a certain size, its game
>> over.  High load starts around 5-10 minutes.  Leaving this running for
>> 20 mins sees load shoot up to 200+.  This isn't a gradual increase, it
>> hits a point and fails.  This seems to be file size as interrogating
>> the SDBM file lists for the IP collections it is a small figure.  For
>> example, I have a 700Mb SDBM file and inside are only 28 IP
>> collections.
>>
>> I've played with TIMEOUT values to see if keeping less is helpful and
>> won't affect things too much, but ideally for DoS protection I'd hope
>> that keeping the IP in the database for as long as possible will be
>> more beneficial.
>>
>> Any help is appreciated - as a potential mitigation step against a
>> Layer 7 DDoS (network and other good stuff in place to protect against
>> this aside) - what can I do from here?
>>
>> Cheers,
>>
>> Kev
>>
>> On 19 August 2011 13:37, Ryan Barnett <RBarnett <at> trustwave.com> wrote:
>> >
>> > On 8/19/11 6:55 AM, "Kevin Jackson" <kevin <at> linuxservices.co.uk> wrote:
>> >
>> >>Dear all,
>> >>After a number of weeks fine tuning the CRS rules appropriate for a
>> >>busy website, we did a full live test pushing traffic through Apache
>> >>with ModSecurity 2.6.1 proxying requests to evaluate the performance
>> >>of ModSecurity through the backend service.
>> >>
>> >>Traffic is around 1k/second and we had Apache running on 18 servers
>> >>sitting behind a load balancer and Apache coping without issue.
>> >>
>> >>ModSecurity on the other hand didn't fare so well - it struggled to
>> >>keep pace causing resource deadlock issues on the ip database.  The
>> >>upshot was incredibly high load averages for obvious reasons and poor
>> >>user experience.
>> >>
>> >>--ba66410d-H--
>> >>Message: Failed to access DBM file "/usr/local/apache/data/ip":
>> >>Resource deadlock avoided
>> >>Apache-Handler: proxy-server
>> >>Stopwatch: 1313660329409820 283469 (- - -)
>> >>Stopwatch2: 1313660329409820 283469; combined=99526, p1=3446,
>> >>p2=12838, p3=3, p4=0, p5=41666, sr=3272, sw=41573, l=0, gc=0
>> >>Producer: ModSecurity for Apache/2.6.1 (http://www.modsecurity.org/);
>> >>core ruleset/2.2.1.
>> >>Server: Apache
>> >>
>> >>--ba66410d-Z--
>> >>
>> >>My question is what to suggest here? People must be running
>> >>ModSecurity on high volume websites? Have you encountered this issue?
>> >>This traffic is a fraction of our peak traffic at a quiet time during
>> >>the day and the environment only coped for about 10 minutes before
>> >>keeling over.
>> >
>> > Hey Kevin,
>> > Question for you - are you using the IP persistent collection to track
>> > data about clients across requests?  If not, then you may want to
>> > disable
>> > the initcol rules at the end of the modsecurity_crs_10_config.conf file.
>> >
>> > See if that helps.
>> >
>> > -Ryan
>> >
>> >>
>> >>Cheers,
>> >>
>> >>Kev
>> >>
>>
>> >> >>--------------------------------------------------------------------------
>> >>----
>> >>Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
>> >>user administration capabilities and model configuration. Take
>> >>the hassle out of deploying and managing Subversion and the
>> >>tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>> >>_______________________________________________
>> >>mod-security-users mailing list
>> >>mod-security-users <at> lists.sourceforge.net
>> >>https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> >>ModSecurity Services from Trustwave's SpiderLabs:
>> >>https://www.trustwave.com/application-security.php
>> >>
>> >
>> >
>> > This transmission may contain information that is privileged,
>> > confidential, and/or exempt from disclosure under applicable law. If you are
>> > not the intended recipient, you are hereby notified that any disclosure,
>> > copying, distribution, or use of the information contained herein (including
>> > any reliance thereon) is STRICTLY PROHIBITED. If you received this
>> > transmission in error, please immediately contact the sender and destroy the
>> > material in its entirety, whether in electronic or hard copy format.
>> >
>> >
>>
>>
>> ------------------------------------------------------------------------------
>> Using storage to extend the benefits of virtualization and iSCSI
>> Virtualization increases hardware utilization and delivers a new level of
>> agility. Learn what those decisions are and how to modernize your storage
>> and backup environments for virtualization.
>> http://www.accelacomm.com/jaw/sfnl/114/51434361/
>> _______________________________________________
>> mod-security-users mailing list
>> mod-security-users <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>> ModSecurity Services from Trustwave's SpiderLabs:
>> https://www.trustwave.com/application-security.php
>
>
> ------------------------------------------------------------------------------
> Using storage to extend the benefits of virtualization and iSCSI
> Virtualization increases hardware utilization and delivers a new level of
> agility. Learn what those decisions are and how to modernize your storage
> and backup environments for virtualization.
> http://www.accelacomm.com/jaw/sfnl/114/51434361/
> _______________________________________________
> mod-security-users mailing list
> mod-security-users <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/application-security.php
>
>

------------------------------------------------------------------------------
Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage 
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php

Rules effectiveness in case of use inside reverse proxy with J2EE container

Dear mod security readers,

We are doing tests on Modsecurity in a reverse proxy configuration, Apache standing in 
the front of J2EE servers.

Modsecurity is working efficiently in applying rules to detect eventual security 
problems.

My question is about the effectiveness of these rules in the context of our 
configuration.

In other words, Modsecurity could have some rules built for an apache/php context that 
do not fit with a reverse proxy to a J2EE container ?

I think about the detection of some characters inside ARGS NAME like "|" for example.

Thanks your reading and time !

Jean

Une messagerie gratuite, garantie à vie et des services en plus, ça vous tente ?
Je crée ma boîte mail www.laposte.net

------------------------------------------------------------------------------
Doing More with Less: The Next Generation Virtual Desktop 
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops?   How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php