Excessive Bandwidth Usage Issues

[and04]

Thanks for your help.  It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings.  The theme files are in a sub folder.

[cybernut]

Thanks for your help.  It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings.  The theme files are in a sub folder.

The .htaccess should catch everything before they get to you, but could be you just had a wave of crawlers that were not included in the .htaccess rules. Some "search crawlers" can really be a nuisance, and may appear in numbers reaching a hundred in a small timeframe.

[and04]

So some of those 81 visitors could have been crawlers, hmmm.  But then again, some could  have been genuine...  I'll do a search for more bad crawlers and add those.  Thanks for the heads up!  I've already just gone through one army of bots, I don't want another.

[and04]

Just a mini update...  The bandwidth issue I had has definitely been cleared up and has been very stable for about two weeks now, so thanks once again to everyone that helped in this thread.   

[McGaskil]

Thanks for your help.  It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings.  The theme files are in a sub folder.

The .htaccess should catch everything before they get to you, but could be you just had a wave of crawlers that were not included in the .htaccess rules. Some "search crawlers" can really be a nuisance, and may appear in numbers reaching a hundred in a small timeframe.

hundreds?

I've experienced the china crawlers literally in the tens of thousands... the htaccess blocks took care of the majority of them but still in the late Jan to early Mar timeframe they return... every year... this year and last they were in the 4k range at peak, but before the htaccess blocking (which someone provided here- and thank you for that whomever you were) I would hit 32,000+ crawlers atop of my 300 or so users.   i was watching the metrics like a hawk at the time- half willing to kill them right there, and the winning half morbidly curious to see if the server could handle it... it did... but man, they came in FORCE.

the ALL originated from China- and not a one of them give a damn what you request or suggest - they just bear down on you... if they crash your server? they still don't care.... they stack up to bum rush it again just as soon as you're back up.... they crawl every. single. page. over and over... i despise them.  data harvesting is what they're doing- and it's amazing what they can put together by doing so- an innocuous comment here or there, a mention of job title/position there, a bit of information that means nothing by itself but when in aggregate of other comments both from the same user over time and then other sources? boom- they get a complete picture of whatever the subject matter is be it technical or personal.  they are something else...... and.... we (US) do it too... we do it as good as they do.  nothing is 'private', and with AI it's easier to make sense of the pile of formless data.... and forums are gold mines as rich or more so than social media.

[ATH019]

We've had some experience of similar numbers from that part of the world ourselves.  A bit concerning.

[cybernut]

Thanks for your help.  It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings.  The theme files are in a sub folder.

The .htaccess should catch everything before they get to you, but could be you just had a wave of crawlers that were not included in the .htaccess rules. Some "search crawlers" can really be a nuisance, and may appear in numbers reaching a hundred in a small timeframe.

hundreds?

I've experienced the china crawlers literally in the tens of thousands... the htaccess blocks took care of the majority of them but still in the late Jan to early Mar timeframe they return... every year... this year and last they were in the 4k range at peak, but before the htaccess blocking (which someone provided here- and thank you for that whomever you were) I would hit 32,000+ crawlers atop of my 300 or so users.  i was watching the metrics like a hawk at the time- half willing to kill them right there, and the winning half morbidly curious to see if the server could handle it... it did... but man, they came in FORCE.

the ALL originated from China- and not a one of them give a damn what you request or suggest - they just bear down on you... if they crash your server? they still don't care.... they stack up to bum rush it again just as soon as you're back up.... they crawl every. single. page. over and over... i despise them.  data harvesting is what they're doing- and it's amazing what they can put together by doing so- an innocuous comment here or there, a mention of job title/position there, a bit of information that means nothing by itself but when in aggregate of other comments both from the same user over time and then other sources? boom- they get a complete picture of whatever the subject matter is be it technical or personal.  they are something else...... and.... we (US) do it too... we do it as good as they do.  nothing is 'private', and with AI it's easier to make sense of the pile of formless data.... and forums are gold mines as rich or more so than social media.

I do believe that happens, but must say in my years online I have never seen a single crawler come in with a force quite like that. 

[cybernut]

We've had some experience of similar numbers from that part of the world ourselves.  A bit concerning.

Very!  To say the least...

[blag-it Admin]

Thanks for your help.  It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings.  The theme files are in a sub folder.

The .htaccess should catch everything before they get to you, but could be you just had a wave of crawlers that were not included in the .htaccess rules. Some "search crawlers" can really be a nuisance, and may appear in numbers reaching a hundred in a small timeframe.

hundreds?

I've experienced the china crawlers literally in the tens of thousands... the htaccess blocks took care of the majority of them but still in the late Jan to early Mar timeframe they return... every year... this year and last they were in the 4k range at peak, but before the htaccess blocking (which someone provided here- and thank you for that whomever you were) I would hit 32,000+ crawlers atop of my 300 or so users.   i was watching the metrics like a hawk at the time- half willing to kill them right there, and the winning half morbidly curious to see if the server could handle it... it did... but man, they came in FORCE.

the ALL originated from China- and not a one of them give a damn what you request or suggest - they just bear down on you... if they crash your server? they still don't care.... they stack up to bum rush it again just as soon as you're back up.... they crawl every. single. page. over and over... i despise them.  data harvesting is what they're doing- and it's amazing what they can put together by doing so- an innocuous comment here or there, a mention of job title/position there, a bit of information that means nothing by itself but when in aggregate of other comments both from the same user over time and then other sources? boom- they get a complete picture of whatever the subject matter is be it technical or personal.  they are something else...... and.... we (US) do it too... we do it as good as they do.  nothing is 'private', and with AI it's easier to make sense of the pile of formless data.... and forums are gold mines as rich or more so than social media.

Seems to be part-and-parcel of running websites these days, especially forums and there's only so much you can do.  It's a constant juggling act! If there's too many hurdles with the sign-up process or too many posting conditions, then genuine visitors and members get put off.



Admin

[McGaskil]

Thanks for your help.  It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings.  The theme files are in a sub folder.

The .htaccess should catch everything before they get to you, but could be you just had a wave of crawlers that were not included in the .htaccess rules. Some "search crawlers" can really be a nuisance, and may appear in numbers reaching a hundred in a small timeframe.

hundreds?

I've experienced the china crawlers literally in the tens of thousands... the htaccess blocks took care of the majority of them but still in the late Jan to early Mar timeframe they return... every year... this year and last they were in the 4k range at peak, but before the htaccess blocking (which someone provided here- and thank you for that whomever you were) I would hit 32,000+ crawlers atop of my 300 or so users.  i was watching the metrics like a hawk at the time- half willing to kill them right there, and the winning half morbidly curious to see if the server could handle it... it did... but man, they came in FORCE.

the ALL originated from China- and not a one of them give a damn what you request or suggest - they just bear down on you... if they crash your server? they still don't care.... they stack up to bum rush it again just as soon as you're back up.... they crawl every. single. page. over and over... i despise them.  data harvesting is what they're doing- and it's amazing what they can put together by doing so- an innocuous comment here or there, a mention of job title/position there, a bit of information that means nothing by itself but when in aggregate of other comments both from the same user over time and then other sources? boom- they get a complete picture of whatever the subject matter is be it technical or personal.  they are something else...... and.... we (US) do it too... we do it as good as they do.  nothing is 'private', and with AI it's easier to make sense of the pile of formless data.... and forums are gold mines as rich or more so than social media.

I do believe that happens, but must say in my years online I have never seen a single crawler come in with a force quite like that.

They tidied up their yearly assault not long ago... Now it's something like between 600 - 1700 a day... That should go down to about 500 by Summer, then next January they'll all horde in again.

What I should do is make copy of their IP's and adjust accordingly.  It 'should' be that simple.

2018 was 'the most', and I misspoke- it was 31k not 32... 2019 was just a hundred or so short of that and while i was blocking one range (before whomever it was left the post identifying the ranges they had encountered)... 2020 I had the htaccess blocks set up and same this year... i want to say it was just over 4k was peak this year.