"blag-it" Forum
Software related topics => Software general discussion => Topic started by: and04 on 18 March, 2021, 01:11:23 AM
-
I suppose this is software related in a sense, but sorry anyway if this is in the wrong section of the forum. Incidentally, I think there needs to be a 'general computing' section or similar...
Basically, a fair amount of bandwidth has been used up for the current month for my site, over 86%, and so I'm looking for ways to reduce further excessive bandwidth usage. What things should I be doing? I've discovered that spider and crawlers can use up bandwidth a lot and that a robots.txt file in the route directory can help with this, but I'm not that familiar with the syntax.
To stop more bandwidth being used up, I put my site offline for maintenance about a day ago, but the usage has gone up to 89.45%. How can more bandwidth be used up when it's offline?
-
Sounds like it's the spiders that are causing a lot of the problem.* Banning the bad ones via .htacess is the way to go. That will nail the ones that ignore robots.txt. They'll still be able to ping your domain, but they won't get any bytes back (just a 403 error message).
The ,htaccess for the root directory of my domain looks like this:
#BlockBotsByUserAgent
SetEnvIfNoCase User-Agent (Ahrefs|Baidu|BLEXBot|Brandwatch|DotBot|Garlik|Knowledge|libwww-perl|Linkdex|MJ12bot|omgili|PetalBot|Proximic|Semrush|Seznam|Sogou|Tweetmeme|Trendiction|Wordpress) bad_bot
<RequireAll>
Require all Granted
Require not env bad_bot
</RequireAll>
<Files .htaccess>
order allow,deny
deny from all
</Files>
<Files 403.shtml>
order allow,deny
allow from all
</Files>
Adding more bots is easy. I only threw in the ones that were giving me trouble.
*Spiders will often index everything on your site, multiple times in succession, when they are on the rampage. This can chew masses of bandwidth, particularly if you have a lot of images and/or downloadable zips.
-
Sounds like it's the spiders that are causing a lot of the problem.* Banning the bad ones via .htacess is the way to go. That will nail the ones that ignore robots.txt. They'll still be able to ping your domain, but they won't get any bytes back (just a 403 error message).
The ,htaccess for the root directory of my domain looks like this:
#BlockBotsByUserAgent
SetEnvIfNoCase User-Agent (Ahrefs|Baidu|BLEXBot|Brandwatch|DotBot|Garlik|Knowledge|libwww-perl|Linkdex|MJ12bot|omgili|PetalBot|Proximic|Semrush|Seznam|Sogou|Tweetmeme|Trendiction|Wordpress) bad_bot
<RequireAll>
Require all Granted
Require not env bad_bot
</RequireAll>
<Files .htaccess>
order allow,deny
deny from all
</Files>
<Files 403.shtml>
order allow,deny
allow from all
</Files>
Adding more bots is easy. I only threw in the ones that were giving me trouble.
*Spiders will often index everything on your site, multiple times in succession, when they are on the rampage. This can chew masses of bandwidth, particularly if you have a lot of images and/or downloadable zips.
Thanks. :) Crawlers and spiders was the main thing my host mentioned, so this rings a bell. Where exactly in my htaccess file do I paste this code and should I also use a robots.txt file in the route directory, would I need both or is it one or the other?
-
Opcache if you're not using it. My processor usage plummeted the most, but the bandwidth followed. Having a process staged keeps the server from round tripping constantly.
Also... set up your expires and caches. If you dont change a lot of things with presentation of your forum, you can cache the images, css, and other static items for as much as a year- not having to make the users browser make return trips for more data... in this same vane, if you're using http2 you can push everything at once if a visitor is new, and if they aren't? You can check in one call whether they have the file and NOT send it if they do.
If you aren't using those, and you implement them? You're looking at likely 80% less processor usage and 50% less bandwidth consumed.
-
OK, thanks for helping, but this code that A.Jenery mentions:
#BlockBotsByUserAgent
SetEnvIfNoCase User-Agent (Ahrefs|Baidu|BLEXBot|Brandwatch|DotBot|Garlik|Knowledge|libwww-perl|Linkdex|MJ12bot|omgili|PetalBot|Proximic|Semrush|Seznam|Sogou|Tweetmeme|Trendiction|Wordpress) bad_bot
<RequireAll>
Require all Granted
Require not env bad_bot
</RequireAll>
<Files .htaccess>
order allow,deny
deny from all
</Files>
<Files 403.shtml>
order allow,deny
allow from all
</Files>
Where in the htaccess file do I put it, and if I use this, do I also need a robots.txt file as well and if so does it go in the route directory or under public_html?
My htaccess file currently has this in it:
RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301,NE]
# php -- BEGIN cPanel-generated handler, do not edit
# This domain inherits the “PHP” package.
# php -- END cPanel-generated handler, do not edit
Also, how do I know which crawlers and spiders are using up b/width?
-
I suppose this is software related ... . Incidentally, I think there needs to be a 'general computing' section or similar...
Will mention this to the admin ppl that I think intends to make certain editions to the forum in near future.
Global Moderator
-
I suppose this is software related ... . Incidentally, I think there needs to be a 'general computing' section or similar...
Will mention this to the admin ppl that I think intends to make certain editions to the forum in near future.
Global Moderator
That's good to know :) @ and04 - drop it all in at the end.
Using robots.txt can be a good thing, but... the bad bots could'nt care less that it's there or not. They're going to do what they do regardless- only legitimate bots adhere to your request, which is basically what your robots.txt is- a request to robots.
Take a look at the visitor logs on your server- if you're using cpanel there are generally several to choose from- that show you the statistics of your bandwidth usage... and by IP address...
-
Thanks! :) This suggests that the .htaccess method is better than the robots.txt one which doesn't stop bad bots, but what about 'good bots' that also take up a lot of b/width such as indexing bots? I suppose I shouldn't exclude those, but then how do I minimize the amount of b/width they use up? What would I put in the .htaccess file for this? Lastly, do I need a robots.txt file at all? As I understand it, ppl can find out what is in the code just by entering mysitename.com/robots.txt so not exactly secure...
-
Right, just an update here... I've put my forum back online today after being in maintenance mode and edited my htaccess file with the anti-bot code suggested by A.Jenery, so that's all good.
I've noticed though that the forum stats at the bottom is saying 'Most Online Today: 81' which is a bit odd seeing as the forum has only been online for about an hour at the time of posting. Confused... ???
-
So your forum is in the main public directory or is it in a sub directory?
Even if you are bouncing bad bots off the site with htaccess, it may still register that they were online for half a second or so.
If your forum is in a sub directory, try putting the code in the .htaccess of the main public directory.
-
Thanks for your help. It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings. The theme files are in a sub folder.
-
Thanks for your help. It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings. The theme files are in a sub folder.
The .htaccess should catch everything before they get to you, but could be you just had a wave of crawlers that were not included in the .htaccess rules. Some "search crawlers" can really be a nuisance, and may appear in numbers reaching a hundred in a small timeframe.
-
So some of those 81 visitors could have been crawlers, hmmm. But then again, some could have been genuine... I'll do a search for more bad crawlers and add those. Thanks for the heads up! I've already just gone through one army of bots, I don't want another.
-
Just a mini update... The bandwidth issue I had has definitely been cleared up and has been very stable for about two weeks now, so thanks once again to everyone that helped in this thread. :)
-
Thanks for your help. It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings. The theme files are in a sub folder.
The .htaccess should catch everything before they get to you, but could be you just had a wave of crawlers that were not included in the .htaccess rules. Some "search crawlers" can really be a nuisance, and may appear in numbers reaching a hundred in a small timeframe.
hundreds?
I've experienced the china crawlers literally in the tens of thousands... the htaccess blocks took care of the majority of them but still in the late Jan to early Mar timeframe they return... every year... this year and last they were in the 4k range at peak, but before the htaccess blocking (which someone provided here- and thank you for that whomever you were) I would hit 32,000+ crawlers atop of my 300 or so users. i was watching the metrics like a hawk at the time- half willing to kill them right there, and the winning half morbidly curious to see if the server could handle it... it did... but man, they came in FORCE.
the ALL originated from China- and not a one of them give a damn what you request or suggest - they just bear down on you... if they crash your server? they still don't care.... they stack up to bum rush it again just as soon as you're back up.... they crawl every. single. page. over and over... i despise them. data harvesting is what they're doing- and it's amazing what they can put together by doing so- an innocuous comment here or there, a mention of job title/position there, a bit of information that means nothing by itself but when in aggregate of other comments both from the same user over time and then other sources? boom- they get a complete picture of whatever the subject matter is be it technical or personal. they are something else...... and.... we (US) do it too... we do it as good as they do. nothing is 'private', and with AI it's easier to make sense of the pile of formless data.... and forums are gold mines as rich or more so than social media.
-
We've had some experience of similar numbers from that part of the world ourselves. A bit concerning. :-X
-
Thanks for your help. It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings. The theme files are in a sub folder.
The .htaccess should catch everything before they get to you, but could be you just had a wave of crawlers that were not included in the .htaccess rules. Some "search crawlers" can really be a nuisance, and may appear in numbers reaching a hundred in a small timeframe.
hundreds?
I've experienced the china crawlers literally in the tens of thousands... the htaccess blocks took care of the majority of them but still in the late Jan to early Mar timeframe they return... every year... this year and last they were in the 4k range at peak, but before the htaccess blocking (which someone provided here- and thank you for that whomever you were) I would hit 32,000+ crawlers atop of my 300 or so users. i was watching the metrics like a hawk at the time- half willing to kill them right there, and the winning half morbidly curious to see if the server could handle it... it did... but man, they came in FORCE.
the ALL originated from China- and not a one of them give a damn what you request or suggest - they just bear down on you... if they crash your server? they still don't care.... they stack up to bum rush it again just as soon as you're back up.... they crawl every. single. page. over and over... i despise them. data harvesting is what they're doing- and it's amazing what they can put together by doing so- an innocuous comment here or there, a mention of job title/position there, a bit of information that means nothing by itself but when in aggregate of other comments both from the same user over time and then other sources? boom- they get a complete picture of whatever the subject matter is be it technical or personal. they are something else...... and.... we (US) do it too... we do it as good as they do. nothing is 'private', and with AI it's easier to make sense of the pile of formless data.... and forums are gold mines as rich or more so than social media.
I do believe that happens, but must say in my years online I have never seen a single crawler come in with a force quite like that. (https://static.simplemachinesweb.com/smf/smileys/default/shocked.gif)
-
We've had some experience of similar numbers from that part of the world ourselves. A bit concerning. :-X
Very! To say the least...
-
Thanks for your help. It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings. The theme files are in a sub folder.
The .htaccess should catch everything before they get to you, but could be you just had a wave of crawlers that were not included in the .htaccess rules. Some "search crawlers" can really be a nuisance, and may appear in numbers reaching a hundred in a small timeframe.
hundreds?
I've experienced the china crawlers literally in the tens of thousands... the htaccess blocks took care of the majority of them but still in the late Jan to early Mar timeframe they return... every year... this year and last they were in the 4k range at peak, but before the htaccess blocking (which someone provided here- and thank you for that whomever you were) I would hit 32,000+ crawlers atop of my 300 or so users. i was watching the metrics like a hawk at the time- half willing to kill them right there, and the winning half morbidly curious to see if the server could handle it... it did... but man, they came in FORCE.
the ALL originated from China- and not a one of them give a damn what you request or suggest - they just bear down on you... if they crash your server? they still don't care.... they stack up to bum rush it again just as soon as you're back up.... they crawl every. single. page. over and over... i despise them. data harvesting is what they're doing- and it's amazing what they can put together by doing so- an innocuous comment here or there, a mention of job title/position there, a bit of information that means nothing by itself but when in aggregate of other comments both from the same user over time and then other sources? boom- they get a complete picture of whatever the subject matter is be it technical or personal. they are something else...... and.... we (US) do it too... we do it as good as they do. nothing is 'private', and with AI it's easier to make sense of the pile of formless data.... and forums are gold mines as rich or more so than social media.
Seems to be part-and-parcel of running websites these days, especially forums and there's only so much you can do. It's a constant juggling act! If there's too many hurdles with the sign-up process or too many posting conditions, then genuine visitors and members get put off.
Admin
-
Thanks for your help. It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings. The theme files are in a sub folder.
The .htaccess should catch everything before they get to you, but could be you just had a wave of crawlers that were not included in the .htaccess rules. Some "search crawlers" can really be a nuisance, and may appear in numbers reaching a hundred in a small timeframe.
hundreds?
I've experienced the china crawlers literally in the tens of thousands... the htaccess blocks took care of the majority of them but still in the late Jan to early Mar timeframe they return... every year... this year and last they were in the 4k range at peak, but before the htaccess blocking (which someone provided here- and thank you for that whomever you were) I would hit 32,000+ crawlers atop of my 300 or so users. i was watching the metrics like a hawk at the time- half willing to kill them right there, and the winning half morbidly curious to see if the server could handle it... it did... but man, they came in FORCE.
the ALL originated from China- and not a one of them give a damn what you request or suggest - they just bear down on you... if they crash your server? they still don't care.... they stack up to bum rush it again just as soon as you're back up.... they crawl every. single. page. over and over... i despise them. data harvesting is what they're doing- and it's amazing what they can put together by doing so- an innocuous comment here or there, a mention of job title/position there, a bit of information that means nothing by itself but when in aggregate of other comments both from the same user over time and then other sources? boom- they get a complete picture of whatever the subject matter is be it technical or personal. they are something else...... and.... we (US) do it too... we do it as good as they do. nothing is 'private', and with AI it's easier to make sense of the pile of formless data.... and forums are gold mines as rich or more so than social media.
I do believe that happens, but must say in my years online I have never seen a single crawler come in with a force quite like that. (https://static.simplemachinesweb.com/smf/smileys/default/shocked.gif)
They tidied up their yearly assault not long ago... Now it's something like between 600 - 1700 a day... That should go down to about 500 by Summer, then next January they'll all horde in again.
What I should do is make copy of their IP's and adjust accordingly. It 'should' be that simple.
2018 was 'the most', and I misspoke- it was 31k not 32... 2019 was just a hundred or so short of that and while i was blocking one range (before whomever it was left the post identifying the ranges they had encountered)... 2020 I had the htaccess blocks set up and same this year... i want to say it was just over 4k was peak this year.
-
Thanks for your help. It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings. The theme files are in a sub folder.
The .htaccess should catch everything before they get to you, but could be you just had a wave of crawlers that were not included in the .htaccess rules. Some "search crawlers" can really be a nuisance, and may appear in numbers reaching a hundred in a small timeframe.
hundreds?
I've experienced the china crawlers literally in the tens of thousands... the htaccess blocks took care of the majority of them but still in the late Jan to early Mar timeframe they return... every year... this year and last they were in the 4k range at peak, but before the htaccess blocking (which someone provided here- and thank you for that whomever you were) I would hit 32,000+ crawlers atop of my 300 or so users. i was watching the metrics like a hawk at the time- half willing to kill them right there, and the winning half morbidly curious to see if the server could handle it... it did... but man, they came in FORCE.
the ALL originated from China- and not a one of them give a damn what you request or suggest - they just bear down on you... if they crash your server? they still don't care.... they stack up to bum rush it again just as soon as you're back up.... they crawl every. single. page. over and over... i despise them. data harvesting is what they're doing- and it's amazing what they can put together by doing so- an innocuous comment here or there, a mention of job title/position there, a bit of information that means nothing by itself but when in aggregate of other comments both from the same user over time and then other sources? boom- they get a complete picture of whatever the subject matter is be it technical or personal. they are something else...... and.... we (US) do it too... we do it as good as they do. nothing is 'private', and with AI it's easier to make sense of the pile of formless data.... and forums are gold mines as rich or more so than social media.
I can identify with 'late Jan to early Mar timeframe...' as it was February when the sudden spike in b/width usage happened to me, but it's notable that in your case they were all from China and for some years. I wonder who they're harvesting data on behalf of?
-
Thanks for your help. It's definitely in the public folder or directory, that is index php and index php~, as is htaccess and settings. The theme files are in a sub folder.
The .htaccess should catch everything before they get to you, but could be you just had a wave of crawlers that were not included in the .htaccess rules. Some "search crawlers" can really be a nuisance, and may appear in numbers reaching a hundred in a small timeframe.
hundreds?
I've experienced the china crawlers literally in the tens of thousands... the htaccess blocks took care of the majority of them but still in the late Jan to early Mar timeframe they return... every year... this year and last they were in the 4k range at peak, but before the htaccess blocking (which someone provided here- and thank you for that whomever you were) I would hit 32,000+ crawlers atop of my 300 or so users. i was watching the metrics like a hawk at the time- half willing to kill them right there, and the winning half morbidly curious to see if the server could handle it... it did... but man, they came in FORCE.
the ALL originated from China- and not a one of them give a damn what you request or suggest - they just bear down on you... if they crash your server? they still don't care.... they stack up to bum rush it again just as soon as you're back up.... they crawl every. single. page. over and over... i despise them. data harvesting is what they're doing- and it's amazing what they can put together by doing so- an innocuous comment here or there, a mention of job title/position there, a bit of information that means nothing by itself but when in aggregate of other comments both from the same user over time and then other sources? boom- they get a complete picture of whatever the subject matter is be it technical or personal. they are something else...... and.... we (US) do it too... we do it as good as they do. nothing is 'private', and with AI it's easier to make sense of the pile of formless data.... and forums are gold mines as rich or more so than social media.
I can identify with 'late Jan to early Mar timeframe...' as it was February when the sudden spike in b/width usage happened to me, but it's notable that in your case they were all from China and for some years. I wonder who they're harvesting data on behalf of?
i don't think they're targeting anything in particular or for anyone/thing in particular... i think they have a toy and by damn they're going to use it.
there was a time, pre-internet, there were three entities you had to concern about collecting information... it was (and this is likely going to surprise you if you didn't know) Financial industry- particularly credit cards... they collected all kinds of information about your spending habits... then, Churches- The Vatican first but in a close second was Mormons.... they didn't reach out into the world, but what they knew about their followers was startling and astounding... then, the usual suspects- gov'ts... particularly the IRS but it didn't stay there. then other gov'ts...
this interweb thing is a boon for data collection. you're already well documented no matter how well you hide from it. AI makes it even more complicated as it collects a heaping pile of raw data without form and creates relationships between those datums on the fly- and that is where google and the social media's come in... followed closely by the US and China who are neck and neck as to who collects more and the applications it's applied.... they actually sell it, for one, after targeting known users of products... your, say, kayak makers and marketers don't have to take out super bowl ads at millions of dollars to target the perhaps 1.5% of watchers that may be interested in their product... now, they just pay the big harvesters and have precision target marketing.
.... the fear is where ever else it goes... one thing is for certain- it never goes away.
maybe five years ago, now, i got a FB message asking me "Is this you in this picture?" - and there i sat in my drunken stupor at a vegas blackjack table in the background of someone's vacation photo... i said "nope"... some months later i saw a picture in 'photos of you' section and there i sat- 'tagged'- but it wasn't me who confirmed it. i wonder if someone i know did, or if they said "yeah right- that's you ya drunken clown"....
these robots/worms/crawlers aren't our friend... unless you're selling something- and then they are... somewhat... but not for long...
there ain't no hiding from it. it's the way it is and the way its going to be and there is nothing you can do about it even if you unplug- because your friends have facebook and twitter, and you and they carry phones, and those phones keep up with where you are and who you're around..... unless you bounce completely off the grid they know... and then they know you've bounced...
wonderful, no?
-
Not wonderful at all >:(. It seems cyberspace has become such a jungle of crawlers and spiders, that it's getting more and more difficult to know what to expect next. As to pre-internet info gatherers, I didn't know about The Vatican but I did know about the practices of the Mormons; they wanted to know your life story as a prerequisite for joining. I was literally 'this far' from being baptized when I luckily came to my senses and walked straight back out of their European head quarters opposite Exhibition Road, London. :-X
-
Not wonderful at all >:(. It seems cyberspace has become such a jungle of crawlers and spiders, that it's getting more and more difficult to know what to expect next. As to pre-internet info gatherers, I didn't know about The Vatican but I did know about the practices of the Mormons; they wanted to know your life story as a prerequisite for joining. I was literally 'this far' from being baptized when I luckily came to my senses and walked straight back out of their European head quarters opposite Exhibition Road, London. :-X
I couldn't agree more with much of what's said here. It's a digital jungle alright!
Admin
-
It gets a bit like that here sometimes. Ooops. :P :-X
Global Moderator
-
Yea, some strange topics here sometimes :-X Not this particular one though ;)
-
Not wonderful at all >:(. It seems cyberspace has become such a jungle of crawlers and spiders, that it's getting more and more difficult to know what to expect next. As to pre-internet info gatherers, I didn't know about The Vatican but I did know about the practices of the Mormons; they wanted to know your life story as a prerequisite for joining. I was literally 'this far' from being baptized when I luckily came to my senses and walked straight back out of their European head quarters opposite Exhibition Road, London. :-X
I couldn't agree more with much of what's said here. It's a digital jungle alright!
Admin
Thanks Admin. Now I know I'm not the only one that experiences these things, and at least the b/width issues I was having has stabilized.
-
That's good to hear. Issues with crawlers and spiders using up bandwidth is fairly universal now, almost every site owner or administrator will have dealt with them at some point...