URGENT READ ME - Unacceptable Forum Activity

See this forum for the rules that govern users engaging in discussions on this website. Includes links to the Ohioans For Concealed Carry privacy policy.

Moderators: 3FULLMAGS+1, djmac1964, sabalo, Stryker74, MeanStreaker, Derek D., oldmic, JediSkipdogg, jgarvas, Chuck, Mustang380gal, OldManTod, Jake, JustaShooter, Bryan Torok, MrMagoo, NavyChief, evan price, schmieg, Morne, Coordinators

URGENT READ ME - Unacceptable Forum Activity

Postby jgarvas » Thu Dec 18, 2008 11:12 am

It has been brought to my attention that users of the forums have discussed using the open source tool 'wget' and windows variants of this tool to mass-download the forums to their local computers for off-line reading and archival purposes.

Effective immediately, anyone caught loading a substantially large number of pages at a rapid pace will have their account suspended regardless of what software they use to do so.

This recent activity has put a substantial burden on the forums and has quite possible caused all of our "You May Not Use The Search Function" errors as individual users consume substantial memory and database resources to rapidly download significant quantities of forums activity.

If we determine that users continue to do this their accounts will be terminated indefinitely. If the problem persists, we will configure the forums to lock out those people who load more than a reasonable amount of pages in a specific amount of time.

We appreciate the fact that many of you find the forums so useful that you wish to archive them for personal use, but we must protect the integrity of the entire system
Jeff Garvas, President
Ohioans For Concealed Carry

Contrary to a popular belief when I brag about OFCC accomplishments I'm not looking for your thank you or personal recognition. I'd much prefer you send me an email telling me when you are going to get involved in doing what I've been doing since 1999. We are only as effective as we make ourselves. We need the next generation of OFCC to step to the plate.

Is that you?

To Contact Me: Use This Form and pick my name.
jgarvas
OFCC Director
OFCC Director
 
Posts: 3129
Joined: Sun Jun 26, 2005 10:35 pm
Location: Northern Summit County

Re: URGENT READ ME - Unacceptable Forum Activity

Postby corrsk » Thu Dec 18, 2008 3:00 pm

i have not done this but i have considered archiving the forums on my hard drive. is it possible that someone could do this and post the download on the website? that may solve this problem
NRA Certified Basic Pistol Instructor
PM for class info

http://www.facebook.com/pages/Patriots- ... 1429217356
User avatar
corrsk
 
Posts: 1393
Joined: Wed Dec 06, 2006 3:43 pm
Location: toledo ohio

Re: URGENT READ ME - Unacceptable Forum Activity

Postby SMMAssociates » Thu Dec 18, 2008 3:30 pm

Jeff:

Won't IE7's "Work Offline" click do something like this? (I think it's "page level" only, but I'm not sure how deep it'll go.)

Regards,
Stu.

(Why write a quick note when you can write a novel?)

(Why do those who claim to wish to protect me feel that the best way to do that is to disarm me?)

יזכר לא עד פעם
User avatar
SMMAssociates
 
Posts: 9488
Joined: Mon Jul 04, 2005 1:36 am
Location: Youngstown OH

Re: URGENT READ ME - Unacceptable Forum Activity

Postby Cruiser » Thu Dec 18, 2008 4:38 pm

It is doing it again! :x
Sorry but you cannot use search at this time. Please try again in a few minutes.
Abandon ye all HOPE!
User avatar
Cruiser
OFCC Member
OFCC Member
 
Posts: 10793
Joined: Mon Jul 04, 2005 6:41 pm
Location: Mercer County, Ohio - what is yours?

Re: URGENT READ ME - Unacceptable Forum Activity

Postby NordicRX8 » Thu Dec 18, 2008 5:39 pm

Would opening twenty tabs put the system under the same strain? I normally open each sub forum in its own tab, and then open each topic of interest in its own tab... I sometime have over 2 dozen tabs open at the same time.

Am I contributing to this error?

Let me know and I can stop doing this.
Suppose the Second amendment said "A well-educated electorate being necessary for self-governance in a free state, the right of the people to keep and read books shall not be infringed." Is there anyone who would suggest that means only registered voters have a right to read? – Robert Levy

A gun in the hands of a bad man is a very dangerous thing. A gun in the hands of a good person is no danger to anyone except the bad guys... - Charlton Heston
User avatar
NordicRX8
 
Posts: 2154
Joined: Sat Apr 21, 2007 10:40 pm
Location: Lost In Middle America, Ohio

Re: URGENT READ ME - Unacceptable Forum Activity

Postby SMMAssociates » Thu Dec 18, 2008 6:24 pm

Nordic:

I don't remember IE, but FireFox will let you indicate a group in your "favorites" and open all of the members of the group automatically. If you're doing that, you may have a spike for a minute or so, if that long. I wouldn't worry about it - especially if you're opening them manually.

"Open" doesn't count. Once the page is loaded most of the "action" is inside your browser unless you do something to agitate the server. You can't do that fast enough to bother anything unless you leave a brick on the "Enter" key....

Regards,
Stu.

(Why write a quick note when you can write a novel?)

(Why do those who claim to wish to protect me feel that the best way to do that is to disarm me?)

יזכר לא עד פעם
User avatar
SMMAssociates
 
Posts: 9488
Joined: Mon Jul 04, 2005 1:36 am
Location: Youngstown OH

Re: URGENT READ ME - Unacceptable Forum Activity

Postby Splat!! » Thu Dec 18, 2008 8:57 pm

Dang , I had you scheduled for this weekend...................I got to earn my "forum leech" title some way :P

I have discussed doing other forums but I haven't done this one..................
"Our government... teaches the whole people by its example. If the government becomes the lawbreaker, it breeds contempt for law; it invites every man to become a law unto himself; it invites anarchy."
Splat!!
 
Posts: 1383
Joined: Sun Nov 05, 2006 9:43 am
Location: Middletown, Ohio

Re: URGENT READ ME - Unacceptable Forum Activity

Postby cmarks03 » Thu Dec 18, 2008 11:42 pm

Not sure how the server that runs these forums is configured, but I'm sure that there is a way to monitor the access logs for crawlers (either a crawler like Google uses or `wget` in recursive mode). Knowing the open source community, there's probably a ton of configuration options for that kind of software (people mentioned multiple tabs, which I'm guilty of doing a bit of, so it would need look for a constant stream of requests). But once you have the info, you can either ban a forum user or ban an IP address (which I personally recommend against unless the ban is temporary).

It would all be on the server side though, and might require more control than our webmaster(s) have.
- Chris
"I are a engineer"
"I'm not fat - I'm grossly overdesigned" (Morne)
"People can see your gun. Just be sure to reserve the view of the muzzle for the really bad ones." (muxtech)

Though I do have Progressive insurance, IANAL and I didn't stay at a Holiday Inn Express last night. This is my $.02; remember you paid less than that for it.
User avatar
cmarks03
 
Posts: 348
Joined: Mon Oct 06, 2008 9:21 pm
Location: Cuyahoga County

Re: URGENT READ ME - Unacceptable Forum Activity

Postby jhartbarger » Thu Dec 18, 2008 11:56 pm

Your typical browsing no matter how many tabs you have open will not affect the site, the wget app is designed to (in the simplest form of explaining it) download an offline copy of the entire site by crawling every aspect of the site much like a search engine spider crawls the site.
jhartbarger
 
Posts: 189
Joined: Sun Nov 16, 2008 11:36 pm
Location: Toledo, OH

Re: URGENT READ ME - Unacceptable Forum Activity

Postby jgarvas » Sat Dec 20, 2008 1:15 am

cmarks03 wrote:Not sure how the server that runs these forums is configured, but I'm sure that there is a way to monitor the access logs for crawlers (either a crawler like Google uses or `wget` in recursive mode). Knowing the open source community, there's probably a ton of configuration options for that kind of software (people mentioned multiple tabs, which I'm guilty of doing a bit of, so it would need look for a constant stream of requests). But once you have the info, you can either ban a forum user or ban an IP address (which I personally recommend against unless the ban is temporary).

It would all be on the server side though, and might require more control than our webmaster(s) have.


We have full access and the environment is apache, and for the most part I know what I'm doing but I focus more on Cisco lately than unix/lamp. We've identified the culprits (and a very aggressive spider) and have rectified the problem. If you're using the forums to read them live (even opening a bunch of tabs) I really don't see that comparing to the volume of rapid data an aggressive depth wget would generate.

The problem for us isn't bandwidth at all. The problem is resource related. Since everything is database driven (every post) a very large and rapid demand for post after post after post causes the system to become taxed -- and quite honestly we didn't design it to support that kind of behavior because that isn't our target audience and it would be cost prohibitive.

If the issue keeps up, we'll install checks and balances that react if they see a specific session requesting too many sites in a very rapid succession instead of the more broad shutting down the forums, but recent changes seem to have nipped this issue for now.

I should clarify, if someone is reading the forums in a manner that is hurting us, we won't kick you out and shut off your account and throw away the key. We'll try to talk to you and determine what you're doing.

We will however take more aggressive action with people who intentionally and/or maliciously harm the forums for their own personal satisfaction and refuse to work with us if their access is detrimental to the system and/or it's resources.
Jeff Garvas, President
Ohioans For Concealed Carry

Contrary to a popular belief when I brag about OFCC accomplishments I'm not looking for your thank you or personal recognition. I'd much prefer you send me an email telling me when you are going to get involved in doing what I've been doing since 1999. We are only as effective as we make ourselves. We need the next generation of OFCC to step to the plate.

Is that you?

To Contact Me: Use This Form and pick my name.
jgarvas
OFCC Director
OFCC Director
 
Posts: 3129
Joined: Sun Jun 26, 2005 10:35 pm
Location: Northern Summit County

Re: URGENT READ ME - Unacceptable Forum Activity

Postby shdwfx » Tue Dec 23, 2008 10:46 am

If someone is using wget or another archival software, it's unlikely that client is bothering with the step of logging in (or in wget's case, spoofing the login with a browser cookie file).
So, the only identifying info available from such harmful activity is going to be an IP address.
You might then scan logs to see which forum ID(s) hail from those IP addresses, but it's not guaranteed to be 1 to 1 due to Network Address Translation (NAT).
User avatar
shdwfx
 
Posts: 187
Joined: Sat Feb 02, 2008 11:48 pm
Location: NW Ohio

Re: URGENT READ ME - Unacceptable Forum Activity

Postby armond21 » Sun Dec 28, 2008 3:27 am

Cruiser wrote:It is doing it again! :x
Sorry but you cannot use search at this time. Please try again in a few minutes.


Is this why I get this message when I search? Well, here comes a bucket of stupid questions from me that would have been answered through the search function. Sorry in advance.
Heavy is good, heavy is reliable. If it doesn't work you can always hit him with it.
-Boris 'The Blade' Yurinov
User avatar
armond21
 
Posts: 268
Joined: Fri Jun 20, 2008 1:39 am
Location: Lake County


Return to Forum Rules, Policies, & How-to's

Who is online

Users browsing this forum: No registered users and 1 guest