Misplaced Pages

This is an old revision of this page, as edited by Draicone (talk | contribs) at 13:35, 3 July 2007 (→The role of the BAG: Do it from toolserver?). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

• ]

Overview Dictionary Bot policy Bot Approvals Group Guide Noticeboard Newsletter Bot Requests Requests for Approval Adminbots {{BAG Tools}} Creating a Bot List of bots Activity monitor Status report History Types

This is a message board for coordinating and discussing bot-related issues on Misplaced Pages (also including other programs interacting with the mediawiki software). Although its target audience is bot owners, any user is welcome to leave a message or join the discussion here.

This is not the place for requests for bot approvals or requesting that tasks be done by a bot. It is also not the place for general questions about the mediawiki software (such as the use of templates, etc.), which have generally a best chance of being answered at WP:VPT.

Archives

2006 2007

Ralbot's unmatched small tags

Recently when delivering the Signpost, Ralbot has been missing off the final <small> HTML tags, causing talk-page errors. Should this bot be blocked or not?? I'm not certain whether it should or not... --SunStar Net 09:47, 7 April 2007 (UTC)

No need for a block. Ral315 has been informed of the problem and it will be fixed for the next time that the bot delivers the signpost (hopefully). Until the bot is actually running with the error, a block is unwarranted. Martinp23 09:50, 7 April 2007 (UTC)

Agreed. I ran in to this, and found that it was already announced and that corrective measures would be in place. Blocking is not warranted at this time, but can be done if there are future malfunctions until they are repaired. — xaosflux 13:53, 7 April 2007 (UTC)

A recent software fix caused the malformed HTML to fail; otherwise, it would have been gracefully ignored. Will be fixed next week. Ral315 » 20:39, 7 April 2007 (UTC)

Great! —METS501 (talk) 21:14, 7 April 2007 (UTC)

Replace a whole page

Anyone happen to have a pywikipedia mod that replaces a whole page (as opposed to a regex edit)? If not, I'll almost certainly code this myself. -- Rick Block (talk) 04:48, 15 April 2007 (UTC)

What do you mean? Once you generate the new pagetext for a page, you might use page.put(newtext, msg) to place it. Or do you need something else? Gimmetrow 04:56, 15 April 2007 (UTC)

I'm thinking something callable from a shell, like replace.py, that takes as input the page name and a file that has what I'd like the new contents to be (or, alternatively, the name of an external filter to run the existing content through to obtain the new content). -- Rick Block (talk) 05:03, 15 April 2007 (UTC)

The first part is easy enough to code. If you have any knowledge of python you should be able to do it in five minutes, or I could if you wanted. Although, why exactly would it be faster for you to tell a program the name of a page and a file that you want posted to it instead of just copying the file into the page yourself normally? As for the second, what precisely do you mean? What kind of external filter are you talking about, and what would it do?--Dycedarg ж 09:30, 15 April 2007 (UTC)

I have a number of scripts I currently run (see Misplaced Pages:Bots/Requests for approval/Rick Bot), that produce page content that I have been yanking and putting into the page manually. I'm intending to make some of these fully automatic (cron jobs). I do most things in unix shell, so like curl (or wget) can fetch a page (to stdout) what I really want is a pywikipediabot command (probably putpage.py) to put a page (from stdin or a file). The surround from replace.py (show the diff, prompt if ok, various params) would be good to preserve as well, so this isn't just a 5 line wrapper around page.put. The external filter idea would be another way to accomplish the same thing. What it would do is set up the existing page content as the filter's stdin and capture the filter's stdout as the "new" page content. This would allow the content manipulation to be done by an arbitrary external program (like awk or sed or perl or ...), extending the pywikipediabot capabilities to any string manipulation language. -- Rick Block (talk) 16:30, 15 April 2007 (UTC)

I have a modified version of replace.py that takes a new parameter (-content:filename) and replaces the content of a page with the content in the named file. I think doing this as a filter (per above) would actually be better (e.g. a new parameter like -filter:program), but this will suffice for now. If anyone wants my modified version or codes up a version invoking a filter, please let me know. -- Rick Block (talk) 03:51, 17 April 2007 (UTC)

Mass Link Replace

I've recently gotten a request that sorta scares me, so I wanted to get opinions from other bot owners. I should first say that I haven't gotten permission to run my bot in any namespace but "Talk", which this procedure would require — but assume for the moment that the bot has that permission.

The request is to change all wikilinks of WP:EA that currently point to the (inactive) Misplaced Pages:Esperanza so that the shortcut can be used by a new (active) Misplaced Pages:Editor assistance. It was followed up by a discussion at the Village Pump that yielded almost no comments.

A quick calculation showed a total of 47,000 pages that currently have links to WP:EA. At the moment I can't remember if that's just to WP:EA or also to Misplaced Pages:Esperanza. In any case, the sheer numbers make me nervous.

Any thoughts or comments? Thanks! -- SatyrBot 21:17, 16 April 2007 (UTC)

As long as you have a flag and edit lass than 10 times/minute, everything should be fine. You should also make sure the bot stops if you or the bot get a post on their talk page. Cbrown1023 talk 21:20, 16 April 2007 (UTC)

Not without specific approval for the task first, please, as it's a potentially controversial one. --kingboyk 21:21, 16 April 2007 (UTC)

Well, I assume that that would have been a given, but you are right, I should have mentioned it. Cbrown1023 talk 21:29, 16 April 2007 (UTC)

Definitely - as I said, assume I've gone through the process to get that permission. And I will before doing anything different from talk page edits. -- SatyrTN (talk | contribs) 21:33, 16 April 2007 (UTC)

I think it would be the easier if the new project chose a redirect name that wasn't already in use on 47,000 pages! (notwithstanding the fact that Esperanza is no longer active). --kingboyk 21:21, 16 April 2007 (UTC) (e/c)

The project's direct page is different - it's the shortcut that currently points to the inactive project. -- SatyrTN (talk | contribs) 21:33, 16 April 2007 (UTC)

(added) Sorry - I misread your comment. You're right, but you have to admit, "WP:EA" is easy and logically should point to something like "Editor assistance". But that's not my concern - that's up to the project :) -- SatyrTN (talk | contribs) 21:35, 16 April 2007 (UTC)

WP:EA is without doubt a good redirect for them, but I'm not sure how the community would feel about the usurpation and resultant 47,000 edits (mostly because Editor Assistance is new). Maybe we should just wait a few short weeks and see how the new project does? If, as I hope, it's successful, I can't see there being too much objection to this task. Better to use the redirect for a project that helps editors in distress than for a glorified MySpace, right? :) --kingboyk 22:21, 16 April 2007 (UTC)

This should go through a specific bot task approval due to it's huge scale. — xaosflux 01:13, 17 April 2007 (UTC)

Given the potential controversy involved in changing these redirects, I thought it best to seek consensus on the matter at WP:RfD - see Misplaced Pages:Redirects for discussion/Log/2007 April 16. At present it seems unlikely that a change will be supported so the above discussion may be moot. WjBscribe 01:53, 17 April 2007 (UTC)

Prolonged absence of Bot owner

Hi, I seem to remember that Bot policy requires Bots to have an owner active on Misplaced Pages- how long does an owner have to be away for this to become an issue? Hagerman has now not edited since Feb 6. HagermanBot generates a lot of questions from people who don't understand how it functions. Also, a number of requests for possible improvements to the Bot's operation have gone unanswered over the last couple of months. It's a useful Bot and is functioning as expected but it is a concern that there is no one to address issues with it. Just wanted your thoughts on the matter. Cheers, WjBscribe 14:01, 16 April 2007 (UTC)

(note: the above was copied from my talk page) I've emailed the bot operator to get a status on them. Bots need to have respondant operators. — xaosflux 01:15, 17 April 2007 (UTC)

Technically, a bot user is active if it responds to the queries brought up by his bot. (However, Hagerman bot is a different case then...). Cbrown1023 talk 02:10, 17 April 2007 (UTC)

February 6 is long ago in wiki time. Unfortunately we may have to consider this bot operator-less. I say "unfortunately" because HagermanBot is immensely useful.

Let's see if he responds to xaosflux's email; perhaps if the op isn't coming back soon he can allow a clone to operate? --kingboyk 13:04, 17 April 2007 (UTC)

I've received notice that this operator has NOT left the project, and will be responding to feedback soon, reported as being away for work. Thank you, — xaosflux 01:06, 19 April 2007 (UTC)

I say let the bot run until there is an issue the requires the operator's attention. _InBC 01:25, 19 April 2007 (UTC)

Cydebot Block

(moved from my talk — xaosflux 05:22, 22 April 2007 (UTC))

(Copied from User_talk:Cyde)

Your bot has been blocked

Your bot has been blocked as a malfunctioning bot. Your bot was editing in excess of 50 edits/min, in violation of the Bot policy. Bot accounts typically should run at up to 6 edits/min, unless performing tasks deemed "urgent", when a top rate of 15 edits/min has been approved. As blocks are not meant to be punitive and you are an administrator, please feel free to remove this block and resume bot edits once you have adjusted your bot's edit rate. You should also feel free to immediately remove any autoblocks or other collateral damage related blocks hit as a result of this block. If feel you must edit at this rate, please talk page me and we can bring this to a forum where more community input can be gathered. Thank you, — xaosflux 01:32, 22 April 2007 (UTC)

Note from Mindspillage

Hey, I noticed you blocked Cydebot. So, my question is: why—what harm was it doing?

Yes, I've seen the guidelines. But what are they for?

The bot has about 45000 more edits to make this run. A massively large-scale renaming effort with almost no potential for misapplication seems to justify the high rate.

What reason is there to block it? It's already done about 40,000 without performing incorrectly (that many would surely have given people enough edits to go by to lodge complaints, if there were any to make), and it's not significantly affecting server resources. If anything, the higher speed seems like a good idea to make sure it will finish during Misplaced Pages's lower-traffic days where it won't affect load so much.

The renaming of the image license templates to reflect the renaming of the guideline pages is an issue very near and dear to my heart... or my spleen, or something... :-) and due to the large amount of time it will take to do, the lack of need for supervision, and the minimal effect on resources, I am strongly in favor of an unblock so it can continue. If this isn't something you are willing to do, I'd like to know why, and to get more input. Cheers, Kat Walsh (spill your mind?) 03:07, 22 April 2007 (UTC)

Reply

Mindspillage, I've got a few comments on this bot and my block:

1. I am not inclined to unblock this account until the operator is available to discuss it.
2. As to the block reason, it is editing in violation of the bot policy, as for the reasoning (besides OMG POLICY VIO!), it is because bots editing at extreme speeds make it harder for independant edit validation, and can cause flooding of logs.
3. Misplaced Pages:Non-free content/templates is only ~5 days old, but alread has rules in place requiring naming standards. The bot's replacements, while not modifying the displyed content of the page, do not appear to have been put through the normal discusion used when replacing template instances (especially 85000 of them), on WP:TFD (at least not one where I can find it).

   I find isssue with this all together if there hasn't been wide spread agreement on this new style, but not to the point where I'd block the account. Nonetheless, as the end result on the page is nothing other then renaming the template, I don't even see this as an urgent need waranting the 15e/min rate (but also would not have blocked if it was limited to that).

4. I've briefly looked through the prior bot approval's pages and can not find where Cydebot has been approved to process template replacements on the Image: namespace; (found one for the user: and userspace namespace), but as this bot has successfully processed countless CFD's and generally operates without much error I would not have blocked for that either.

— xaosflux 03:32, 22 April 2007 (UTC)

Reply to reply

100,000 edits isn't going to make a small footprint however you consider it. At such a scale, I simply don't see the point. No one is going to go through and independently verify those edits to any degree that makes the difference in speed matter. As for flooding logs, no one has complained of it as far as I can tell; it is bot flagged, of course, so stays off of recent changes.

The template name change was proposed about a month ago on the talk page of the Non-free content page (formerly called "fair use"), and no one has contested it; it's largely motivated by the rename of the image licensing pages to make the names consistent and adhere with the requirement that non-free content be easily machine-identifiable.

The gist is this: I don't see any substantive difference at this scale between 6 edits a minute and 50. If it edits at all, any difference in editing rate isn't going to have any effect that anyone will appreciate. While it'd be nice if Cyde were around, I don't think it makes a difference. If it actually malfunctions, sure, cut it off and go message him about it, but if not I just don't see the point. Kat Walsh (spill your mind?) 04:22, 22 April 2007 (UTC)

This bot is approved to make several other types of edits, are these being made simultaneously? Or are they being suspended during this run, if not they will be hard to monitor. This and other reasons are behind the limit on bot edit speeds. I've left a message for the others in the bot approvals group to leave more input here on this. — xaosflux 04:47, 22 April 2007 (UTC)

Cydebot was still doing its approved tasks, see . Now it kinda can't update the speedy deletion page which is unfortunate. :-\ --Iamunknown 05:02, 22 April 2007 (UTC) Oh, and the contribs for which it is approved can be monitored by following that link (to answer your second question). --Iamunknown 05:11, 22 April 2007 (UTC)

Additions to reply to reply

Kat pointed out that the template rename discussion started almost a month ago, so the 5 day comment is a bit off the mark.. but it should also be noted that this is a move not a deletion, so you wouldn't expect a discussion at TFD. The /template subpage is a working page to keep track of the actual rename project. As you can see from the edit history it's had quite a few direct participants. Based on the lack of material objections to the rather simple and uncontroversial edits being performed, I'm going to recommend that the bot be unblocked right away so that it can complete its current tasks before it runs into heavy traffic times during the week. --Gmaxwell 04:29, 22 April 2007 (UTC)

I see the longer history on that page now, was only led to the /template page though, as that is what was in the edit summary. — xaosflux 05:21, 22 April 2007 (UTC)

I suppose RFD would be more approriate. — xaosflux 04:49, 22 April 2007 (UTC)

You mean RM? --Iamunknown 05:02, 22 April 2007 (UTC)

Except the redirects probably won't be deleted for a couple of months... not until people get used to the new templates. Until then we plan on having the bot go clean up all new use. --Gmaxwell 05:10, 22 April 2007 (UTC)

I meant RfD, as that is the action that is basically going on, if these don't need to be deleted, then this has become a redirect replacement only, and again, what's the hurry? Redirects are cheap. — xaosflux 05:21, 22 April 2007 (UTC)

I wish you'd read more of the pre-existing discussions. The reason the pages need to actually be changed is that a primary driver for the change is machine readability of the license status from the wikitext of the image pages. Unless there is some indicator of the license status in the actual image page wikitext, someone working off the dumps or a database replica must operate a complete copy if the mediawiki parser, which is very slow, in order to have any hope of figuring out the license status of the images. The change addresses that issue, but not if the images are not changed. "Whats the hurry"?? This is a change that was discussed for a month... it's one that in addition to cleaning up a bunch of misconceptions, is needed to fulfill enwikipedia's obligation under the foundation licensing guidelines. --Gmaxwell 05:43, 22 April 2007 (UTC)

Just wondering... why was such a large task not proposed at WP:BRFA? —— Eagle101 05:33, 22 April 2007 (UTC)

Good question. — xaosflux 05:35, 22 April 2007 (UTC)

Um, because the bot approval group has no oversight over image licensing procedures on english wikipedia, at least I can see no reason why it should or why I would have expected it to. ::shrugs:: --Gmaxwell 05:43, 22 April 2007 (UTC)

Indeed we don't, which is why the task would get approved whether or not I or other BAG members personally agree with changing, say, the album cover template (I don't). Provided there is commmunity consensus - or at least little likelihood of complaints - we approve; this would be easy to approve then because consensus is clear and it's Foundation policy too. Where we do have jurisdiction (see above for current ArbCom case where this seems to be in the process of being confirmed) is over bot activity. I daresay this task can be given a special waiver, but (as below) it has to be applied for first. Nobody except perhaps Jimbo has carte blanche to operate a bot at such high speeds without getting it approved first. --kingboyk 12:20, 22 April 2007 (UTC)

Comment

So, basically, y'all have blocked Cydebot from doing obviously valuable and necessary work simply because someone didn't file Form 27-BXT? Can we please stop wasting time filing TPS Reports and get back to writing an encyclopedia? Kelly Martin (talk) 05:45, 22 April 2007 (UTC)

Nope, it was blocked due to fast editing rate. See WP:BOT. —— Eagle101 05:46, 22 April 2007 (UTC)

Indeed. This is a demand for a shrubbery. Please get out of the way of getting real work done and unblock the bot already. If the devs have a problem with the edit rate, I'm sure they'll let us know. Kelly Martin (talk) 05:49, 22 April 2007 (UTC)

If that is the case, then I suggest that you make that recommendation to WT:BOT. I believe the edit rate is there for good reason... what would happen if the bot went nuts and did 1000 screwups before someone could grab an admin. If we trust bots not to screw up, and want them to go at faster rates, then bring it up on the bot policy page. —— Eagle101 05:52, 22 April 2007 (UTC)

If you don't know what the reason for the edit rate guidelines, I suggest you refrain from enforcing (or even arguing) them. This bot has already made thousands of edits on this task without error; expecting it to suddenly start doing so is slightly paranoid. Again, I believe that you are demanding a shrubbery. Kelly Martin (talk) 05:57, 22 April 2007 (UTC)

I may be, I don't know, lets see how this whole bit plays out :) —— Eagle101 06:00, 22 April 2007 (UTC)

You'd best ask your friends at ArbCom. BAG approval is needed, not to satisfy our "thirst for power" (do you think I actually enjoy this mundane task?!) but because it's technically necessary. --kingboyk 11:31, 22 April 2007 (UTC)

Point that bothers me

What bothers me about this, I suppose, is something that bothers me about discussing admin action in general that is often overlooked: lack of admin action is an admin action, too. How many admins saw the bot working or knew what it was going to do and thought "oh, there goes cydebot" and let it be? That's a decision, too. It made thousands of edits on this run already; there must have been at least a few who did. Those who don't act don't have to register their opinion or provide rationales for not acting, of course, which makes it hard to say if others care or not, and I don't want to undo someone's block in non-urgent circumstances without clearly showing that sentiment swings the other way. But I know of several admins who don't object, and currently of only xaosflux who does.

I don't, in general, follow bot goings-on; I'm paying attention to this one only because I care about the template renaming. What I see as someone not deeply involved in it is that something was happening that was outside the guidelines, but wasn't doing anything substantively rather than procedurally objectionable, and the reasons I've seen that the guidelines actually exist for don't seem to be a factor, as they either don't make a difference on this scale or, like the contributions by namespace filtering, aren't an issue.. Kat Walsh (spill your mind?) 05:47, 22 April 2007 (UTC)

I believe the block was made due to a violation of bot policy (fast edit rate, max is 15 edits per minute, it was going at 50). —— Eagle101 05:49, 22 April 2007 (UTC)

Eagle 101, I approved it editing at a rate of one operation interval sleep per operation or 1 edit per second, whichever is slower. --Gmaxwell 05:52, 22 April 2007 (UTC)

Errr... where did you approve it editing that fast? Generally bots don't edit that fast to prevent too much damage should something error out.... Then again I'm not a member of bag. —— Eagle101 05:55, 22 April 2007 (UTC)

If you run the bot at 15 edits per minute the task will complete in about 90 hours, which can be done easily if you host it on toolserv. Or you can just leave your computer on. —— Eagle101 05:57, 22 April 2007 (UTC)

Um. This batch. By the end of the transition every non-free image on enwiki will be be touched. Also add time lags from pipeline bubbles in the decisions to keep or get rid of templates that have to happen before the bot can run, and such.. and you're taking about making the process take months for no particular reason. It was my understanding that Cyde intended to run at higher speeds during low traffic times. The fact that this is unproblematic is demonstrated by the fact that it ran for many hours without incident, and also made similar edits at low speeds all week last week without trouble.

Cyde asked me via email what would be as safe rate to conduct this rather large set of edits from a server load perspective. As far as it 'going nuts' goes, with it simply doing a 1:1 template name replacement there is pretty much no risk of that, and with some 40k edits already made it's not likely to do any harm at this point. --Gmaxwell 06:04, 22 April 2007 (UTC)

Last I checked, bot operations were approved as a result of a bot approval request, how did this approval process run? — xaosflux 05:56, 22 April 2007 (UTC)

Additionaly, it appears the operator has set this to run even faster then that delay . — xaosflux 06:01, 22 April 2007 (UTC)

No it wasn't, it's running with a higher interval.. above.. i.e. slower. :) --Gmaxwell 06:07, 22 April 2007 (UTC)

Yes, that's what it says, but that doesn't address my comment. I know it did not follow the letter of procedure, but what I want to know is how it was actually causing harm, and if anyone who saw it edit has objected to any of the edits it was making or thought its going slower would be a substantive improvement. Kat Walsh (spill your mind?) 06:08, 22 April 2007 (UTC)

Note, This thread is getting in to many issues that may be easily resolved once the operator is available, is there an assertion of immediate urgency on why this can't wait? — xaosflux 06:01, 22 April 2007 (UTC)

Please let me know where to send the shrubbery. Kelly Martin (talk) 06:05, 22 April 2007 (UTC)

Gmaxwell how many images was this task going to hit? —— Eagle101 06:09, 22 April 2007 (UTC)

Most recent estimate I heard was that there were around 400,000 non-free images on enwiki... but that sounds a bit off to me but they are hard to count because our current tagging isn't very machine readable (I can't just query the database to find out). It's in that general ballpark. Cyde won't be editing them all right now, because some templates are up for deletion rather than being renamed. Only the ones that people are sure aren't changing are being renamed now.--Gmaxwell 06:15, 22 April 2007 (UTC)

Ok, I did not know, in any case, when Cyde gets on he can explain why he did not do a simple WP:BRFA, and went past what our current policy says is the fastest you can go. I'm sure its all just a mistake :). Unblocking the bot now probably won't do anything as it likely has errored out. —— Eagle101 06:31, 22 April 2007 (UTC)

At this point I'm tempted to put the BAG pages up for deletion, I do not believe that the community would approve of this sort of disruption of actions which are harmless and uncontested. Nor do I think the community would approve of the makeup of the bag or the methods for their selection. There is no technical or editoral reason for this disruption. And as far as I can tell at this point the BAG is simply obstructing valid activity in order to make a point about their authority to do so. If that is the case, it is reasoning made in error: no users group has the authority to disrupt valid edits just to show off their power to do so. --Gmaxwell 06:46, 22 April 2007 (UTC)

That's a bit much, considering I was in bed asleep at the time! You might want to see Misplaced Pages:Requests_for_arbitration/Betacommand/Proposed_decision#Automated_editing too. --kingboyk 11:05, 22 April 2007 (UTC)

I think it was the actions of one admin, I don't think he minds if it is unblocked, and was probably expecting cyde to be around. I really don't know much more then that. All I've been doing is explaining why he probably did what he did. Perhaps this was a case of WP:IAR, on the part of Cyde? I mean the block is valid per WP:BOT, but it might have been a case where WP:BOT should be ignored. —— Eagle101 06:56, 22 April 2007 (UTC)

As a random member of the Community passing through, I'd like to point out that as I see it the reason we have BAG is that most of us do not understand BOT policy (or more precisely the rationale behind it) and so defer to those who do. Unless someone can show that BAG has completely lost the plot I suspect it would survive MfD. In any event, as only one BAG member has commented in this thread it seems a little premature to draw sweeping conclusions. WjBscribe 07:02, 22 April 2007 (UTC)

I find myself with little option but to agree with you, Greg. There was no reason for Xaosflux to believe that Cydebot's edits were harmful; there is no reason for him to insist on waiting to "speak to the operator" before unblocking either. As far as I can tell, the reason for this block is nothing more than "I was not consulted and I have granted myself the right to be consulted". This is, quite simply, disruption of important and legitimate efforts to improve the encyclopedia, for the sole reason of protecting personal power. As such, it is totally unacceptable. If this is reflective of the normal behavior of the bot approval group, then the bot approval group needs to be either restructured or disbanded. I am reasonably certain that has never been any demonstrated community consensus for the bot approval group's rules, procedures, or authority. I am also reasonably certain that many of the bot approval group's policies are arbitrary and have no technical or editorial merit, but exist merely to exist. The edit rate rule is quite certainly one of these arbitrary policies. Kelly Martin (talk) 07:05, 22 April 2007 (UTC)

Technical problems introduced by bots editing at extreme paces go beyond the processing power of the database and caching servers. Linus's Law teaches us that given enough watchers, all problems will be uncovered. Unfortuantley we do not have an infinite supply of watchers. When a multi-purpose bot goes off to make thousands and thousands of edits in a short time, it makes it even harder for the watchers to determine if it's other purposes are having issues. It also makes it hard to detect other problems (e.g. Recent Image Changes (warning large link) now is only availble for the last 7 hours of edits. Running at full speed (as listed above) would make this log max out at 83 mins.) While some options (i.e. hide bots in the RC log) are in place to help alleviate this, they have limits (i.e. you have to hide ALL bots). My block is not "just because I can" but becuase I have good faith concerns over this operation, and at the LEAST feel that further community input is warranted. — xaosflux 07:13, 22 April 2007 (UTC)

FWIW, I would have blocked this bot for editing at this speed even had I not been in the approvals group. — xaosflux 07:27, 22 April 2007 (UTC)

Alright, so we looked at it, it's fine, lets move on. -- Ned Scott 07:17, 22 April 2007 (UTC)

While you all get hot under the collar about your precious free image crusade, what exactly is wrong with {{albumcover}} anyway? Why waste server resources renaming it, when 99.999% of album covers are copyright?! --kingboyk 11:07, 22 April 2007 (UTC)

I'm still waiting to be informed why moving {{albumcover}} to {{Non-free album cover}} is worth the server resources. The bast majority of album covers aren't freely licenced. Why not leave albumcover as is and just create a new {{Free album cover}} template? Are people not capable of reading the blurb on the template or what?! Is this a prelude to removing all album covers from Misplaced Pages? --kingboyk 15:02, 22 April 2007 (UTC)

It allows third part users to remove non free images simply by killing anything with a template starting with "Non-free" on it.Geni 15:53, 22 April 2007 (UTC)

I see. Thanks. --kingboyk 15:54, 22 April 2007 (UTC)

Another bot for now?

While we're waiting for word from Cyde, would it be alright to have another bot work on the template renaming (at a much slower pace, of course)? -- Ned Scott 07:31, 22 April 2007 (UTC)

Is there another Bot flagged to do this sort of work? WjBscribe 07:36, 22 April 2007 (UTC)

This debate has brought up many things! I won't personally enforce any blocks on bots performing this task at the approved rate while this debate is ongoing, but others may if the proposed bot is not approved for the task. FWIW, new bot approval requests are handled at WP:RFBOT. — xaosflux 07:38, 22 April 2007 (UTC)

This is such a minor housekeeping task, though.. -- Ned Scott 07:41, 22 April 2007 (UTC)

If it is the request to be approved should go by quickly, and a case for going faster then what WP:BOT allows can be made. Anyway hopefully this will get sorted out by the time I wake up. ;) —— Eagle101 07:44, 22 April 2007 (UTC)

Hypothetically if Xaosflux speedy approved the replacement Bot there is a crat around who might be willing to flag it. But per Xaosflux's question above- where's the urgency if its just "minor housekeeping" that this Bot would do? WjBscribe 07:46, 22 April 2007 (UTC)

Like I said, I won't block it unless it's racing. I would not be inclided to speedy approve this task, but to maintain neutrality while this debate is in progress I wouldn't deny it either. Other bot approvers are availble though. — xaosflux 07:50, 22 April 2007 (UTC)

MartinBotIII is approved for "template substing and renaming per consensus" (or, it should be :)), so if it's urgent, and the bot is needed, I can have it complete the task at an edit rate of 9-10/min by Tuesday/Wednesday (guess). However, it would probably be better to wait for Cyde to get back and finish the job for himself. As for the 50epm rate - I find it surprising that a bot can possibly fetch the wikitext of a page, and put the new wikitext, at such a rate (though this is probably the fault of my internet connection). The bot was editing in deifance of policy, both on the maxed out edit rate and the lack of approval, both of which could be valid reasons for a block. However, it seems that the task was simple (and the operator experienced), and not something that one would expect any admin to block for (BAG or not is irrelevant - any admin has the power to block bots). This does not excuse the edit rate, which should have been specifically requested if required, but again it is at the discretion of the particular admin whether to block or not. For 40000 pages, an edit rate of just 8/9 per minute will get them done in a couple of days (MartinBotIII is doing such a lengthy task now) - the concern that it would take months is nonsense :) Martinp23 09:14, 22 April 2007 (UTC)

You missed a zero. It was 400,000 pages, which I had at 18 days at 15 e/m. I agree that 50 e/m should have been specifically requested, but at any rate, I had an idea that might fix some of the shrubbery concerns. Now, it's getting late where I am and I reserve the right to change my mind in the morning, but it's posted at Misplaced Pages Talk:Bot policy. --Selket 10:05, 22 April 2007 (UTC)

Ooops - sorry. I read 40k a few times above, and it stuck :) Martinp23 10:09, 22 April 2007 (UTC)

Yes, as for the speed, I do have a server on a nice campus network. I actually could go a lot faster than one edit per second ... but the devs assured me that wouldn't be such a good idea :-P Cyde Weys 13:55, 22 April 2007 (UTC)

Heh. /me is jealous. --kingboyk 13:58, 22 April 2007 (UTC)

Endorse block

I endorse the block. Nobody - except perhaps Jimbo Wales - is excluded from the bot rules. If Cydebot were running at a little over 15ppm, I would personally have turned a blind eye. 50ppm is, however, over 3 times the limit.

I didn't impose the limit; I don't know who did and on what basis. What I do know is that however noble the task, folks can't go and unilaterally decide to ignore it. By all means let's have a discussion and quite possibly agree that Cydebot can go faster for this task; for now I endorse the block. --kingboyk 11:35, 22 April 2007 (UTC)

I fully agree with kingboyk, whilst this task might be fine to run at a higher speed it should be discussed before it is allowed to continue doing so. Adambro 11:58, 22 April 2007 (UTC)

Good block; non-urgent task that doesn't need to be run faster than the agreed-on rate. I see no reason to break the rule for this specific case, and a change of the general rule (for which there are good non-technical reasons) should be discussed beforehand. Kusma (talk) 12:53, 22 April 2007 (UTC)

You people are being completely pompous idiots. The only reason for the rate limit is to protect the wiki. A developer - that is, someone competent to decide - has approved it. You can "endorse" the block all you like, but you are in fact not competent to decide. Stop acting like you know more about the systems than the people who run them - David Gerard 20:06, 22 April 2007 (UTC)

That's a personal attack. Please retract it. --kingboyk 20:08, 22 April 2007 (UTC)

Agreed, that is out of line. If you cannot make your point without name calling, then perhaps you need to think longer before posting. _InBC 20:10, 22 April 2007 (UTC)

Excuse me. You are acting like completely pompous idiots in this instance. You blocked it, he pointed out a dev okayed it as harmless, and rather than the obvious thing to do - "ok, no worries, let us know in advance next time" - you spend thousands of words defending the policy against the people who are actually responsible for the servers the policy is supposed to protect. It's spectacular, and evidence for the urgent need for severe rationalisation and ground-up rewrite of bot policies. Preferably by devs - David Gerard 20:30, 22 April 2007 (UTC)

Get your facts straight. I unblocked. Also, at no point in time did anybody say "gmaxwell is a dev and he approves it". Then it would have been discussion over, wouldn't it? --kingboyk 21:53, 22 April 2007 (UTC)

I agree that the "idiots" comment is uncivil and uncalled for. One of the big problems here has been lack of communication. Cyde didn't make it clear where he had gained approval for his bot and it wasn't made clear that Greg was a developer nearly soon enough. I find the assumption that members of BAG have some empire building agenda an incredible assumption of bad faith when they seem to have been nothing but helpful. Bot policy at present requires administrators to block Bots acting outside the scope of their permission or exceeding the present edit rate limit. If the Bot policy needs changing, fine- that can be discussed and done. But it appears to me that those who have responded from BAG have been amazingly civil and helpful given the barrage of critism and accusations they have been receiving. WjBscribe 01:19, 23 April 2007 (UTC)

I'm back

Okay, I'm back. What is needed of me now? As far as I can tell, I did everything correctly. If anything, I am only guilty of going over the BAG's heads and asking the CRO and devs directly about what kind of editing rate I should use. I'm sorry I didn't notify — I didn't realize it would somehow be a problem — but now that everyone knows, I think Cydebot should get back on its task. There is a lot more work to do and this is a change that is definitely wanted by the Foundation. --Cyde Weys 13:53, 22 April 2007 (UTC)

Good question. Best submit a task approval request I suppose; approval should be a formality if the Foundation want this done. As for the edit rate, we've not had a case like this before. If you have some "official" sanction perhaps provide details of that and we can rubber stamp it? BAG don't proclaim to have authority over the Foundation or the devs :), it's just as per above these things need to be cleared on wiki first otherwise other folks will think "Cyde can do it, my task is just as important, I'm going at an edit a second too").

In the meantime, I'll unblock your bot (if it's not been done already). --kingboyk 14:02, 22 April 2007 (UTC)

I've unblocked. Off the record etc etc, if I were you I'd carry on but keeping the edit rate down for now. --kingboyk 14:05, 22 April 2007 (UTC)

It appears that this bot is still racing. Currently running at an average speed of: 20e/min on this task alone, and an aggregate rate of 25e/min as it appears to be running multipe task threads. As this isn't to the extreme that it was at before I'm not going to wheel war over the block, but really would like to know why it needs to run at 166% of the maximum approved rate. — xaosflux 15:38, 22 April 2007 (UTC)

As I said on my user talk page, I unblocked it because Cyde was back (which is normal procedure for bot blocks unless they're totally rogue), and unofficially recommended he carried on at 15ppm max, pending a proper bot task application. If you reblock because it's racing that's fine by me (and presumably by the other editors who agreed with my endorsement of the original block). No wheel warring is necessary or implied if you reblock due to changed circumstances :) --kingboyk 16:00, 22 April 2007 (UTC)

(moved from my talk — xaosflux 19:25, 22 April 2007 (UTC))

Lets be clear, I think bot policy is important and needed. But what we do not need are meat-bots enforcing the bot policy. If the BAG is going to disrupt harmless activities because they run at 50e/min vs 15e/min, without any clear technical or editorial reason beyond "we said not to" then I think it doesn't deserve to exist and it should be replaced with something more competent and less dysfunctional. And there really is no technical reason the editing can't run at that rate.... Most tasks shouldn't run at that rate for editorial reasons and sometimes technical ones, but most tasks will not need to touch anywhere near this number of pages. Most tasks are also not making an utterly trivial change to image pages which is almost impossible to do incorrectly. --Gmaxwell 17:41, 22 April 2007 (UTC)

See my response on:WP:BOWN (ultra short summary: Technical issue:log flooding; blocking I would have blocked this even if I wasn't on WP:BAG for the same reason). — xaosflux 17:55, 22 April 2007 (UTC)

What log flooding? It is bot-flagged. And why would you think that something showing up as a continuous drone for a week is better than it showing up more frequently for a day? So if I push a temporary change to mediawiki to suppress the creation of RC entries in the image namespace for cydebot, you will have no more objections? --Gmaxwell 18:02, 22 April 2007 (UTC)

That would be one of the logs, another would be anyone trying to review the contributions of this bot during it's run's through Special:Contributions, as this bot does work on things other than this task. If this would have been brought to bot review I would have suggested that it run under a unique account for this special task, especially if it will be running at abnormal rates. Note, that these concerns are purley from the technical nautre, but the actual reasoning for the edits appears to be getting questioned by other above (I really don't care myself which template is used). — xaosflux 19:23, 22 April 2007 (UTC)

Regarding special:contributions, even 15e/min will quickly scroll them off unless you're looking by namespace. Once you look by namespace there is no issue.

"actual reasoning for the edits appears to be getting questioned by other above", difflink?

I think it's also interesting to note that relative to the overall editing rate rambot's runs were at a much greater speed than cydebot's edits here. I can't help but see Xaosflus's complaints as nothing more than a pointless power game.

Since I have already approved the higher rate the argument is moot. --Gmaxwell 19:44, 22 April 2007 (UTC)

You still haven't answered the simple question of what position gives you the authority to approve of it. I'm not meaning to be disrespectful, and am probably showing my ignorance in even asking, but if you're approving as a dev or a board member or whatever please just say so :) --kingboyk 19:55, 22 April 2007 (UTC)

A process to approve bots exists not only to ensure that they comply to technical specifications, but to ensure they are performing tasks that the community wants / needs. An important aspect of that is documenting the task so that anyone can understand it upon later review. That is why the bot approval process is done on-wiki, in the open, with a community discussion period. I am infering (perhaps wrongly) from your reply that you are declaring yourself to be an absolute authority when it comes to approving this, where has the community vested you with this role? As this is getting heated, I am going to go have a nice cup of tea and revist this later. — xaosflux 19:58, 22 April 2007 (UTC)

You appear to be empire-building, rather than protecting the wiki. Stop assuming competence of areas you are not competent to decide upon, especially when directly contradicted by a developer, who is competetent to decide upon said area - David Gerard 20:08, 22 April 2007 (UTC)

The edits per minute rule was never subject to the devs opinion, while it's clearly a techcnical matter. If the developers are saying that there's no harm in going over 15 epm, then the rule becomes automatically arbitrary without any strong argument other than the BAG feud wanting it to be. -- drini 20:11, 22 April 2007 (UTC)

And yes kingkboy, greg's is a dev. -- drini 20:11, 22 April 2007 (UTC)

Where on gmaxwell's user page does it say that he is a dev? Why have I had to ask several times in what capacity he approved it, before getting an answer? (and then get shot down in flames for asking). I can't be expected to know everybody and I've politely asked more than once. The real answer, of course, is that I'm not a member of the IRC cartel, no doubt this is playing out in that arena right now.

As for empire building, I can disprove that notion by resigning from the group, which I am happy to do right now. --kingboyk 20:15, 22 April 2007 (UTC)

I think this fuss might have been avoided had there been a note on Cydebot to the effect: "Approved by dev:whoever to run at 50edits/min for task whatever". This would have at least explained the situation to anyone questioning the unusual edit rate. Is that empire building or common-sense public notice? Gimmetrow 20:26, 22 April 2007 (UTC)

And, this is sort of tangential but way more important: can I go back up to the faster editing rate without being blocked again? Have we pretty much worked it out now, regardless of whatever prior notification could have been given previously? I'd really like to get this task finished within a week rather than within a month. --Cyde Weys 20:33, 22 April 2007 (UTC)

Cyde - consider the bot approved for the task and edit rate requested and and allowed by the developer above (who, if common sense previals, is the authoritiative figure here). I do invite any BAG member to overrule me, but for the time being, lets all try to de-escalate the situation. Martinp23 20:48, 22 April 2007 (UTC)

Of course that's correct. I don't want a turf war with devs, and would - in the absence of any evidence otherwise - assume them to have greater authority over the servers than the bot approvals group. --kingboyk 21:55, 22 April 2007 (UTC)

As a complete outsider to the bot groups and the Foundation group and the developer group and any other groups involved here, can I just agree with what Gimmetrow said above: "I think this fuss might have been avoided had there been a note on Cydebot to the effect: "Approved by dev:whoever to run at 50edits/min for task whatever". This would have at least explained the situation to anyone questioning the unusual edit rate." - it is painfully clear to anyone watching this from the outside that what has happened here is a lack of communication. Carcharoth 20:54, 22 April 2007 (UTC)

Indeed - more communication is needed. When this communication isn't present, we get several (often well respected) users finding out about the issue, thinking that the actions involved were completely reprehensible, and making comments attacking the process. Of course, all of these comments are made in hindsight, with the benefit of a lot of information not available to those taking the steps they felt neccessary at the time. This is a recurring pattern on Misplaced Pages, where one small action causes the wheels of outrage to trundle on, until the next slightly-controversial action comes along... Martinp23 21:45, 22 April 2007 (UTC)

Nail. Head. Hit squarely. --kingboyk 01:42, 23 April 2007 (UTC)

The role of the BAG

If everyone could stop bickering for a minute and actually have a real conversation we might actually accomplish something.

I can assure you one thing: the BAG is not trying to build an empire over the bots that run on Misplaced Pages. If everyone was perfect and no one would possible run a bot that doesn't following the bot policy and help the wiki, then there would be absolutely no need for the group, and I'm sure no one would have a problem deleting the bot approvals page and abolishing the bot approvals group. The group was established precisely because this is not the case. Everyone in the group has technical and policy knowledge enough to approve or reject bots. It is also an open forum to promote discussion and consensus. If any developers would like to join the BAG and have enough policy knowledge as well, they are more than welcome. Most people think developers can trump consensus, and this has some truth, but only in half the cases. The developers can stop actions that they deem harmful, but they cannot approve actions that they deem harmless; that role is delegated to the community.

One thing fundamental is ignored often on this wiki, and perhaps this should even be a policy. No one, and I mean no one, is immune to the rules. It just doesn't work like that. If some people were immune and others weren't, everything would collapse. It just doesn't work. Even if you think your bot task is uncontroversial and save, you still need approval from a member of the bot approvals group in a forum open for discussion. It is impossible to predict the opinion of everyone, so even if you and a developer think something is OK, the rest of the community may not. This task is very uncontroversial, and would no doubt have been speedily approved, but there would be a record of this approval for everyone to see. Also, there was strong opposition when discussed of raising the bot edit rate over 15 edits per minute. I pushed for this change a few months ago, and it was controversial raising it from 6 to 15. Now imagine 50, over 8 times the original maximum edit rate. What if the bot messed up for 10 minutes on little-used pages which meant no one noticed. That's 500 edits, a MASSIVE amount to clean up afterwards. 15 edits per minute is a good compromise, reached by community consensus. If you want a change in the policy to get 50 edits per minute, then discuss it, don't just do it. I hope I've made this point clear.

If George Bush went ahead and just suddenly declared war on Canada and said afterwards that he thought it was necessary, and that he had the agreement of one other high-level official, the American people would be insulted and feel violated. Cyde, even though you may not think it, you're trying to be the President of Misplaced Pages, and the rest of the people feel they have no control over your actions, and because you have a few people supporting you, you feel that you are in the right and everyone else is wrong. Sorry, but that's the truth. People are afraid to approach you because you don't talk constructively, and you think that you never err. Everyone has to obey consensus, and when there is none, they cannot simply create a 2-person consensus.

People claim there is too much bureaucracy here, but really, the problem is that people spend more time trying to get around it thereby causing a huge amount of discussion, during which which everyone cites the rules. If people would just follow the rules, and not feel like they are the king, things would just get done faster.

Sorry for the stupid analogy above, and I hope I haven't offended anyone; I certainly didn't mean to, I just needed to get my point across.

Now, back to my Wikibreak :-) —METS501 (talk) 21:04, 22 April 2007 (UTC)

But why was BAG immune to the rules when it was created without consensus? ... or was Essjay immune to the rules, but now he's gone so no one is? --Gmaxwell 21:20, 22 April 2007 (UTC)

It's funny and inappropriate that you blame cyde here, since he wasn't the one in all the discussion claiming that the bag was off the mark here... it's been a lot of other people, including myself. So if you're going to make statements like this you need to target them at the right people. --Gmaxwell 21:16, 22 April 2007 (UTC)

I have no idea how or when BAG was created; I joined some time after creation. Was it created without consensus? It's quite intruguing that the arbitrators don't seem to think so (see current Betacommand case, where they are coming to the view that BAG has authority over bot issues).

Why is it okay to slag us off for doing a volunteers' job that we were under the impression we had the mandate to do? Why is xaosflux getting stick for blocking a high speed bot that didn't have any approval, and why am I getting stick when the only action I took was to unblock it?

Where in Misplaced Pages policies and guidelines does it say that consensus is formed on IRC?

What is the precedent for saying that rules can be overridden because an influential group of Wikipedians say so? (I'm not talking of WP:IAR here, which AFAIC doesn't apply to automated operations.

All that said, clearly the process needs to be rethought, and I am no longer convinced the process has community support. Therefore I am going to resign from the group. I have better things to do with my time. --kingboyk 22:04, 22 April 2007 (UTC)

No responses. Speaks volumes. --kingboyk 01:47, 23 April 2007 (UTC)

On wiki notice is important, this whole thing would never have happened with it. Right now the method is through bag (to notify about new bot tasks whatever). If that is to change, someone needs to start some discussion somewhere, perhaps back to the old bot noticeboard that we used to have before bag took its place. —— Eagle101 21:19, 22 April 2007 (UTC)

I have always maintained that the maximum bot edit rate is mostly decided on non-technical considerations. While there may be a technical limit, I don't believe anyone has reached it yet. We routinely run bots on the server side that edit with a rate only limited by the speed of the database servers.

The main technical concern is that an excessive write rate will cause the slave servers to lag. To avoid this on the server side, all of our scripts monitor lag and pause if it goes above 5-10 seconds. If client-side bots did this too, I would have no problem with edit rates on the order of 100 per minute. -- Tim Starling 21:11, 22 April 2007 (UTC)

Server-side bots? Couldn't this image template renaming thing have been done by a server-side bot? Carcharoth 21:31, 22 April 2007 (UTC)

Hey, if they'll give me an account ... :-D Cyde Weys 21:33, 22 April 2007 (UTC)

That did occur to me too; if the task is so important and so massive why isn't it being run by devs on the server? I did wonder though whether the API is much less efficient than anything the devs have, unless it were done by way of an SQL update query. --kingboyk 21:58, 22 April 2007 (UTC)

We have better things to do. Editing via HTTP is good enough. - Tim Starling 22:23, 22 April 2007 (UTC)

Thanks for the reply. --kingboyk 22:24, 22 April 2007 (UTC)

Isn't that what toolserver is for? Actually, I don't know if Cyde has a toolserver account but I could run the bot from mine - 0.1 ms ping to the wikipedia servers, and no need to waste Tim's time. --Draicone 13:35, 3 July 2007 (UTC)

It appears to be "OK" for certain people to slag off the work of (other) volunteers, because evidently in our constitutional monarchy (with no constitution, and indeed no constitutional conventions), they've received some sort of implicit patronage to do so. If the powers that be think that the bot policy as currently written sucks in general, or simply needs to set aside in a particular case (and there might be an excellent argument for either position), couldn't this at least be communicated (to the community in general, and to the BAG in particular) in advance of such bunfights as this? Simply going ahead and doing it anyway, and justifying it after the fact on the basis of "IARing all rules because our empire trumps your empire-- er, because it's the right thing, that's it" is deeply unhelpful. I don't see much chance of "No one is immune to the rules" becoming policy, since the practical outworking of the IAR "policy" is precisely the reverse. Alai 23:27, 22 April 2007 (UTC)

Thank you Alai. I do believe you're right. BAG were never empire builders, we were/they still are working for the community. We should have been informed of this decision; communicating it to us would have avoided this mess, and we don't deserve the resultant criticism. If anything, criticism should be flowing the other way for making decisions off wiki and scapegoating us when it went pearshaped. We were doing a volunteers' job in good faith and get personal attacks for our efforts. Seems a tad unfair to me and I'm quite unhappy about it :( --kingboyk 01:47, 23 April 2007 (UTC)

Look, rules are irrelevant. What is important is the intentions behind those rules. Limiting edit rate is in place to ensure that nothing can go wrong and that bots can't go mad and make thousands of hard-to-reverse edits before someone stops them. But this is a professional developer who has been trusted with modifying the system managing the information on this wiki - heck, he could probably even figure out your password. Yet he cannot be trusted with the simple task of approving a trivial bot? Since when was BAG the all powerful community group with regards to bots? In fact, has anyone even looked at the source code in question? If we look at CydeBot's scripts and have twelve experienced developers approve them as perfectly harmless, would people agree to let the bot edit at a decent rate? While we're at it, why rate limit at all once we know that nothing could possibly go wrong, just to get it over and done with? --Draicone 13:28, 3 July 2007 (UTC)

Maxlag parameter

On thinking about the maximum edit rate problem, I've introduced a hidden parameter called maxlag. You can set it for any request to index.php, e.g. . If the specified lag is exceeded at the time of the request, a 500 503 status code is returned, content type text/plain with a response body with the following format:

Waiting for $host: $lag seconds lagged\n

Recommended usage is as follows:

• Use maxlag=5 (5 seconds). This is an appropriate non-aggressive value, used by most of our server-side scripts. Higher values mean more aggressive behaviour, lower values are nicer.
• If you get a lag error, pause your script for at least 5 seconds before trying again. Be careful not to go into a busy loop.
• It's possible that with this value, you may get a low duty cycle at times of high database load. That's OK, just let it wait for off-peak. We give humans priority at times of high load because we don't want to waste their time by rejecting their edits.
• Unusually high or persistent lag should be reported to #wikimedia-tech on irc.freenode.net.

-- Tim Starling 21:36, 22 April 2007 (UTC)

Tim, that is great, any chance of this on query.php? _InBC 21:39, 22 April 2007 (UTC)

On further thought, query.php is read only, so it is not the same issue. Great idea for index.php though. _InBC 21:43, 22 April 2007 (UTC)

The implementation of this functionality is such that it could be easily done with query.php if it made sense. Mike Dillon 22:23, 22 April 2007 (UTC)

I'll start experimenting with this in pyWiki shortly and will hopefully have it committed to the main CVS in not too long. --Cyde Weys 21:43, 22 April 2007 (UTC)

How about making it return 503 Service Unavailable to distinguish it from other servers errors (rare as they may be) and providing an X- header with the current lag? Mike Dillon 21:49, 22 April 2007 (UTC)

Actually, it looks like there is a purpose-specific Retry-After in the HTTP spec. Mike Dillon 21:49, 22 April 2007 (UTC)

OK, I'll make it 503. -- Tim Starling 21:59, 22 April 2007 (UTC)

I point out for the suggestion to pause for at least 5 seconds that bot coders might consider an aggressive backoff such as I've been using. When response time increases, my bots increase their delay-between-request time by more than the response time. The delay time is then decreased slightly by each request which gets a fast reply, so as to gradually approach the desired maximum rate. I'm not giving the exact values because it is best if each bot has different characteristics. (SEWilco 04:01, 23 April 2007 (UTC))

This is an excellent idea. Thank you for implementing it so quickly. --kingboyk 14:46, 23 April 2007 (UTC)

I've put some documentation for this feature here. -- Tim Starling 09:00, 25 April 2007 (UTC)

Bot policy rewrite

OK, I've read this whole page and am pretty much convinced that many people (if not most) are unhappy with the current bot policy and some have lost complete respect for the BAG. So I've created a straw-pole-like thing below. I ask that the BAG doesn't !vote on the first heading ("Too much power for BAG"), because it's really up to the community if they think that we have too much power, not us. Also, if you want to add any more headings feel free. —METS501 (talk) 22:19, 22 April 2007 (UTC)

Too much power for BAG

Comment below (preferably with an explanation) if you agree or disagree that the BAG has too much power (or thinks they have too much power) and that some of this power should be delegated to others or left as responsibilities of bot owners.

• BAG seems to do their job well, though I would like to see a more standardized way to join than just bringing it up on a talk page(something with more community input). _InBC 23:13, 22 April 2007 (UTC)
  • I agree with that entirely. I don't think BAG has "too much power", and I note that ArbCom seems to be recognising BAG as legitimate in the ArbCom case. However, if the group is to be taken seriously and it's opinions respected it needs a more formal process for membership. I'll toss out a first idea, it can surely be improved on: Perhaps applications could be made at WP:RFA and be closed by a bureacrat? (I'm not suggesting that only admins need apply). --kingboyk 23:17, 22 April 2007 (UTC)
    • The only problem with that is that most of the community is unable to judge whether a person has enough technical understanding to be in the group, and also many people think RfA is failing, so it might not be a good idea to go there... —METS501 (talk) 23:18, 22 April 2007 (UTC)
      • Yes, I know. In fact I've just been over there talking about whether RFB is failing or not (I think it is). Any other ideas? We can't have a situation where it's ok to call members "idiots" without it being a personal attack; that means, we need more rigorous recruitment standards and a more formal assignation of authority. --kingboyk 23:21, 22 April 2007 (UTC)

Approvals process is too strict

Comment below (preferably with an explanation) if you agree or disagree that the bot approvals process is too strict and that there is too much bureaucracy involved in BAG.

• I have never had a problem getting a well articulated project approved. If you follow the instructions then approval is very fast and reasonable. _InBC 23:11, 22 April 2007 (UTC)
• As an outsider stumbling across this discussion, I agree that the process is too strict. rspeer / ɹəədsɹ 00:26, 23 April 2007 (UTC)

Edit rate

Comment below (preferably with an explanation) if you agree or disagree that for well-established bots (which we can come up with a good definition for) who are running simply takes and have literally zero chance of making a mistake should be allowed to edit at a rate of 50 edits per minute.

• I think edit rates above the standard rate should need special approval, and should be taken on a case by case basis. _InBC 23:11, 22 April 2007 (UTC)
• I might also support a policy whereas with a broader consensus than a normal bot approval, a bot can run at very high edit rates. —METS501 (talk) 23:14, 22 April 2007 (UTC)

• Likewise. I don't know where the 15ppm rate came from, but if the devs are happy with higher rates in special cases, so am I. --kingboyk 23:19, 22 April 2007 (UTC)

• Lets be clear..., Are we talking about experienced bot operators operating bot flagged bots making safe edits (ones which are clearly uncontroversial and can be easily undone)? If so, I believe the rule should be that the bot should edit slowly (6epm or so) for the first 10 minutes of its run and after that it should be able to operate at whatever speed the experienced bot operator believes is needed to complete the task in a reasonable amount of time up to the limits imposed for technical reasons.. which is on the order of 1-2 edits per second while watching serverlag. As for the bots actual task, thats up to the community and the subject area editors to decide. Anyone running around making lots of bad edits will suffer the communities wrath already, so we don't need to add another layer of approval. Anyone who makes a large scale bot run which is later shown to be non-consensus and whom isn't willing undo the changes will be subject to blocking like anyone else who is harming the project.--Gmaxwell 23:23, 22 April 2007 (UTC)

• There are two issues here: the ability of the WMF software to sustain a very high edit load from one account is not tested. It would make sense to investigate this with the developers. I saw a few very strange bugs recently (deletion of an article that does not go to the log, diff that does not make sense etc). I wonder if it maybe a result of a racing condition on a server. The second is the probability of an error. If something goes wrong and you have to undone 400K edits then it require at least another 400K edits. Even if a single revert is trivial 400K reverts are not. I would suggest first to ensure complete consensus for the concept on WP:VP, WP:AN, WP:BAG, etc. Then, say, 8h of running at 6epm, then 50epm if no objections. I guess 400K edits will pollute the recent changes anyway whether we apply then at 3epm for a year or at 50epm for a month Alex Bakharev 23:46, 22 April 2007 (UTC)
  • What do you think a very high edit rate is? We routinely take editing rates in well excess of what were are talking about. Enwiki takes over 200epm over the entire 4 hour busiest time of day every weekday. The "servers can't handle it" argument is basically FUD and misinformation. :) As for 8h at 6epm thats still around 3k edits, if a problem isn't obvious and doesn't get noticed right away it's probably not going to get noticed until it is well into it in any case. --Gmaxwell 02:18, 23 April 2007 (UTC)

    Well, we can't start handing out 50epm rates to everyone, we have many bots running simultaneously - barring special circumstances, individual bots shouldn't be much higher than 20epm, if that - one bot shouldn't be compared to our peak rates, you're weighing an apple against a cherry tree. Weigh the apple tree against the cherry tree so we know how many edits our bots have to work with, and decide from there. ST47Talk 03:42, 23 April 2007 (UTC)

No, but if everyone used the new maxlag parameter above, that would not be an issue. —— Eagle101 04:06, 23 April 2007 (UTC)

Complete rewrite

Comment below (preferably with an explanation) if you agree or disagree that the bot policy needs a complete and total rewrite, and the bot approvals process needs a complete rewrite as well, possibly getting rid of the BAG.

• I have not seem a demonstration that the policy is failing, so any improvement should build and improve on the existing one, I see no fundamental flaw. _InBC 23:12, 22 April 2007 (UTC)

Return to noticeboard format

From one of the other bag noticeboards... (that I posted at and was suggested to post here), An idea, I'm sure there are others but, lets make this back to what it used to be, a simple noticeboard where people who want to run a bot can post a message. If ther are no complaints in say a day or two, just run the bot.—— Eagle101 22:28, 22 April 2007 (UTC)

• As a bot operator, I'm very tempted by that. As somebody who's seen various hair-brained approval requests (as a BAG member) and heated discussions on the admin boards when bots have gone wrong (as an admin), I'm not. On balance, I think we do need a formal process, alas. --kingboyk 23:23, 22 April 2007 (UTC)

  If people aren't going to notice it listed on a notice board, why are they going to notice in some formal process? Avoid-bot-paranoia. We need to avoid the situation where bot approval is the longest part of the process for making a change on wikipedia. --Gmaxwell 23:25, 22 April 2007 (UTC)

A second note, we need to address why User:Cyde seems to be unwilling to bring tasks for his bot to a WP:BRFA. Finding that reason may help you all in "fixing" this process. I still urge something back to what we used to have. Post tasks, if nobody complains or has questions in a day or so let them run the bot. No need to have "elections" or what not, if you know things from a technical standpoint you would be more then capable of commenting on a standard notice board. —— Eagle101 23:27, 22 April 2007 (UTC)

Maybe Cyde doesn't like process? :) (I'm quite sure he doesn't, actually, but let's wait for him to answer that :))

OK, User X proposes a task on the noticeboard. User Y objects. What happens next?

What about if User X is Cyde, and User Y is a casual editor, not an admin, doesn't have friends in high places, but has some kind of objection which might be reasonably valid? What happens?

What about if Cyde goes ahead anyway? What happens then?

Aren't these things better done within a framework of rules and guidelines, and a transparent process which everybody must follow? --kingboyk 23:34, 22 April 2007 (UTC)

I agree with kingboyk here: A self-determined group is the best way to proceed: Community-wide support isn't going to work for those very reasons, and even though I'm no fan of instruction creep, we need to prevent a washing-out of the process by people who may not be experienced in the procedure. A vote will simply suck - we'll end up with an RfA-like system that doesn't even address the issues that need to be addressed. As for elections into the group, our current system is also superior to votes: those who have been on the 'front lines', so to speak, know what needs to be done and are able to choose those who have shown than they can do it (can you imagine a system in which anyone who writes a featured article can approve bots?). Community-wide elections - while the system does currently allow anyone to comment - seems like the opinions of the BAG would be washed out and overridden - we need a small, mostly self-elected, group with members who know what they're doing and who have the trust of the community - that is the main thing, the community needs to be able to trust that the people were chosen for a reason. ST47Talk 00:47, 23 April 2007 (UTC)

Well, it need not be self-elected, I've tried to get outsiders to look at some of the elections but few bothered. I'd agree that elections are probably better than if it was just like RfA and we need to discuss the issues rather than dissolve and have a huge vote. And we do encourage outside opinions on bots, in fact that often is why it took so long, perhaps we could cut down on that. If a bot is possibly controversial we can wait for outside views for 2 days, posting on AN/I or such. Otherwise, things should be pretty linear and in-and-out, especially for trusted bot operators. Voice-of-All 01:09, 23 April 2007 (UTC)

I think I said self-governing where I meant self-contained, as in, BAG IS the BAG, they are responsible for electing their own - it's more of semantics than anything, but it should be considered to be a BAG process in which the community is involved (Like BRFA - BAG makes decision, community expresses opinion) rather than a community process (Like RfA, where the outcome is directly tied to the set of all users). ST47Talk 01:58, 23 April 2007 (UTC)

I'm not sure whether you are implying this to be a good or bad thing. The point is that the community has given BAG the authority to make these decisions and does not want to be otherwise bothered. There are times when BAG specifically seeks outside community input, but otherwise works autonomously. Community opinion is given significant weight and definitely not ignored. Afterall, BAG has its commission from the community and exists to serve it. It could be a disaster if people were approving bots who didn't understand the implications. My favorite example is the automated spell check bot. If you think it is a good idea that can be properly implemented, chances are you should not be approving bots. -- RM 13:47, 23 April 2007 (UTC)

• I prefer a notice board to a formal approval process with designated approvers. If there is any more process it should be reserved for obtaining bot-flag. Once someone has proven themselves to be generally trustworthy with the bot bit, we should give them enough rope to accomplish whatever they want to accomplish. Sometimes that will also be enough rope to hang themselves, but thats their problem, not ours. The community is able to police the actions of editors, automated ones are no exception. --Gmaxwell 23:30, 22 April 2007 (UTC)

  I think you make a good point; I could be swayed by that. What would you propose for the initial process though? (The process whereby a bot flag is procured). --kingboyk 23:44, 22 April 2007 (UTC)

Simple, a post to the B-crat noticeboard, with evidence of why you need the flag and that community accepts the tasking. Do we really need anything more then that? —— Eagle101 23:46, 22 April 2007 (UTC)

Do bureaucrats want that task? Are they capable of it? Do we even have enough bureaucrats? I don't know if you're aware or not, but it's just about impossible to get anybody promoted to the role at the moment. --kingboyk 23:48, 22 April 2007 (UTC)

There's an issue here that selection of the BAG gets very little community input; RFB, if one were being cynical, could be said to suffer from getting too much. One possible approach would be to split up the current BC roles at the technical level (if the devs wouldn't mind), and have them be requested separately (if the community'd be OK with that), so at least the different trust (and "plaform", and what have you) issues can be factored out a bit more. Alai 00:06, 23 April 2007 (UTC)

King, bcrats already have to flag bots anyway, that simple load won't increase. —— Eagle101 00:56, 23 April 2007 (UTC)

Not as measured by number of clicks on the BC tools, no. But they'd then have the (sole, as opposed to in theory joint with the BAG, and in practice rather secondary) responsibility of questioning the operator to determine their competence and suitability, assessing consensus for the task, etc. Alai 01:02, 23 April 2007 (UTC)

Right. At the moment, all the crat does is check that a BAG member gave the approval, and click the button (or whatever they do to add a flag). --kingboyk 01:05, 23 April 2007 (UTC)

Right, but its not that hard for the crat to look and see... bot needs to edit fast, here is proof of the tasking, and nobody complained at the board after XX days (say 5). Clickety click, approved for the flag. This is what it was before BAG came in and attempted to simplify things. (around when essjay started this) Also its not like bots get flags every day or anything. —— Eagle101 01:09, 23 April 2007 (UTC)

I think that "no one complained after X days" is a pretty decent model, in fact (though not infallible, a trait it'd have in common with all others). Faster approval for genuinely simples cases can be handled by acclaim, invocation of precedent, or by explicit approval by designated individuals (whether that be a BC or BAG (or equivalent). It's desirable that a reasonable number of eyes see any given request, though, and ask the obvious (or less obvious) questions, flag any gotchas that a single-point-of-failure might miss, etc. Alai 02:24, 23 April 2007 (UTC)

(undent) I'm not saying that going back to the board style is perfect, but it would put the onus for good bot operations on the bot operator, rather then an authorization group. Really either way is fine with me, and I will continue to comment on new bots in whatever format that is chosen. —— Eagle101 12:40, 23 April 2007 (UTC)

The onus has always been on the operator to perform good actions. The BAG is there to verfiy it and in the case of new bots there is theoretically a bureaucrat to perform another verification step. -- RM 14:22, 23 April 2007 (UTC)

How the French do it. --kingboyk 13:10, 27 April 2007 (UTC)

Simple suggestion from an Insider.

I was part of BAG and I think that BAG or some form of simi-tight control is needed. I have seen some completely ass-backward bot request for Approval/Bot request. On the other hand it is a real pain in the ass as a bot op to come back for every task. I propose a idea. Bot op's MUST apply for their first approval (and get bot flag). once a bot and operator is trusted let them do their work and any other similar work. if a task is a major change in what it does have the bot op leave a note on the BRFA talk and if there are any issue file a BrfA otherwise let the bot op work. (Keep It Simple, Stupid) the motto of all bot writers Betacommand 00:01, 23 April 2007 (UTC) (I will crosspost this)

I agree with that totally. It's the bot flagging which I think needs scrutiny; we should then trust flagged ops to do a responsible job, or have a self-policing noticeboard for new tasks. --kingboyk 00:11, 23 April 2007 (UTC)

Why is the oversight even needed, B-crats already have to flag bots, so that won't change. (in reply to above section). Guys lets remember that this used to be a simple noticeboard, you posted your task here, waited a day or so, and then provided nobody found any problems, ran the task. There is no need for oversight, thats every admins job (not all of them do focus here, but out of 1200 admins enough do). If you are qualified to be a "bag" member, then you are more then qualified to raise issues at a simple noticeboard. —— Eagle101 00:55, 23 April 2007 (UTC)

Eagle: B-crats flag bots for technical reasons, they are the ones who can. While in RfA the bcrats determine an outcome and there is no group other than them who is tasked with closing RfAs, in this case those powers are split: bots aren't as powerful as admins, it's a community task to run a bot, so we don't need the busy bcrats to need to handle and process all these requests, it can be done by a group of trusted users. I don't see anything wrong with the process or with the people performing it, so let's just leave it alone. ST47Talk 02:03, 23 April 2007 (UTC)

B-crats don't see the majority of the requests now anyway, and under the proposal below they won't see the majority of the requests should bag disappear. Do remember that the community never really discussed bag in the first place ;) —— Eagle101 02:28, 23 April 2007 (UTC)

Is there any need to try and score points? "Do remember that the community never really discussed bag in the first place" What? Policy develops organically on wiki. People have been free to comment here whenever they like, too. I'm also told that only a few days ago you were asking about joining BAG! Let's keep the discussion on target and discuss rather than attempt to impress our IRC friends, alright? --kingboyk 10:49, 23 April 2007 (UTC)

Ah nah, that point is irrelevant, but what about the others that I have raised? I don't think switching formats would matter all that much, at least from the standpoint of flagging bots with a bot flag. B-crats don't seem to care about non bot flagged bots anyway. Its just an impression, and just so you know I do agree with the original block, but I also wonder if there is a better way then bag. —— Eagle101 12:37, 23 April 2007 (UTC)

Thanks for clearing that up, I appreciate it. I know where you're coming from, and deciding if there's a better way is why we're in this discussion after all :) Cheers. --kingboyk 14:35, 23 April 2007 (UTC)

Bag alternative

Well here it is, I dug out a copy of the old noticeboard out of WP:BRFA. This is what we did before BAG came into place. As bag never had a formal !vote, I suggest that we comment on this alternitive method and perhaps adopt it. I don't recall any problems under the old system. Check out my sandbox for the header, and see if you can think of any decent modifications User:Eagle_101/Sandbox/4. I'd like to emphasize that the community never really approved of bag to start with vie any large community discussion, so I'd like to see a justification of why the older noticeboard format is invalid, and why this current format of BAG is better if that is the case. :) —— Eagle101 01:34, 23 April 2007 (UTC)

Err well that one actually is part of the BAG as it started, but it looks like User:Betacommand has modified it so that mentions of BAG are not there. Again this is an alternitive to what we do now, I welcome any suggestions, and hope that this at least provokes thought. What is really needed when it comes to bots? Any bag member can comment on bots at a simple noticeboard, and heck I've been commenting on bots under this current format anyway. This is a community discussion, we must not forget that. The point of this is not to see "will it get approved" its "will the community accept my task?". —— Eagle101 01:49, 23 April 2007 (UTC)

With all this talk about a "BAG alternative", I thought I should weigh in, but I have not yet had time to read anything of the above discussion. Let me first post by saying that I've written on Misplaced Pages talk:Bots/Approvals group a little about the history of BAG and why I think it has always had consensus. I'll read the rest of the discussion and post more. -- RM 13:03, 23 April 2007 (UTC)

RM, I appreciate the history bit, there were things in there I did not know. I'm starting to think that this process can be improved... at least for existing bot operators. There was nothing wrong with the task that User:Cyde chose to run, but the bot was blocked as it was going at 50 edits per minute without prior notification. So... that makes me wonder why did Cyde not notify BAG, was it too time consuming to draft up a BRFA? I'm wondering if for experianced operators they could simply post to a noticeboard like we did in the old days... Again just food for thought :) —— Eagle101 13:08, 23 April 2007 (UTC)

Historically speaking, I've always been in favor of letting bot operators do whatever they feel like doing without any approval at all. Cyde just did what I would have done a couple years ago. My bot runs consisted of approximately 30,000 edits at a time and editing as quickly as possible was the only way to perform the tasks in a reasonable amount of time. Having a bot run for, say, a month really is just too long. I agree that the process could be changed or improved, but I don't think there is an overwhelming need to rush to a change. I've read the comments from the dev above regarding edit speed and our policy should reflect those thoughts and instructions as appropriate. I would agree that the approvals process is currently broken due to lack of people approving. It takes too long. But more members would solve that problem. Requests for new tasks really doesn't otherwise take that long and we allow established existing bot operators to ignore the process for certain urgent tasks. I like to think of it this way: putting a bot through approvals not only shows good will but shows that the operator wants to get constructive input to make sure that things get done right. -- RM 13:36, 23 April 2007 (UTC)

Problem - timeliness of repsonse

THe old system suffered from requests that did not get answered at all, the new system still has overly long delays. The principles of both are sound

1. disclosure of intent
2. testing
3. approval

These protect the wiki, the community and the bot owner. However we could perhaps find a more active solution by increasing the numbers in BAG, we can't expect those few meebers to be constantly available, I would volunteer, and I'm sure there are others with experience. How they are appointed doesn't really matter that much, as the process of task approval is fairly objective (but perhaps could be made clearer). Rich Farmbrough, 13:10 23 April 2007 (GMT).

I was also thinking about this overnight, that perhaps the task approval process could be made more objective. e.g. Commnunity consensus? Yes, check. No, go get it and come back. Operator has a reasonable contribs history/block log? Check. --kingboyk 14:44, 23 April 2007 (UTC)

Interesting, I never really thought of it this way... in any case I'm wondering if it would be a decent idea to ask Cyde why he did not bother with the BRFA before running the task. As I mentioned above there was nothing wrong with the task that he chose to run. But it got blocked for reasons of high edit rate, something that would not have happened had an announcement of intent (through bag or a noticeboard whatever) had occurred. In either case the block was correct per our existing WP:BOT. —— Eagle101 13:16, 23 April 2007 (UTC)

You're absolutely right. He only needed to notify us. We're not empire builders, and we have always acted in what we perceived to be the best interests of the wiki. There was nothing wrong with the task (I personally am a bit skeptical about it but would have approved because of clear consensus). There's nothing wrong with the edit rate if a dev has okayed it. What was wrong was going ahead without telling anybody on wiki, and then blaming the uninformed parties for questioning it. --kingboyk 14:44, 23 April 2007 (UTC)

The major problem I see with bot approvals is the speed because there are too few members. Having a productive member resign over this is only going to make the problem worse. I think the process works really well when there are enough people working on the process and there isn't a backlog. Only in the last few months have has the backlog grown substantially. I used to approve about 60% of all bot requests before I became less active for reasons off-wiki. We need more members in a way that we did not in the past. That would help solve most of our problems. Finding qualified individuals is complicated because approving bots generally requires broad policy understanding to approve bots correctly. If someone wasn't paying adequate attention to policy they might, for example, approve an automated spell checking bot which would be VERY BAD. As objective as they are, if you don't have a good policy understanding, you'll make bad approvals. I think that Rich Farmbrough should nominate himself on Misplaced Pages talk:Bots/Approvals group. I'll support the nomination. -- RM 13:28, 23 April 2007 (UTC)

As I see it, BRFAs go through 2 stages, questioning and testing, then most are approved. The first stage should only take a day or two, and then the bot is able to run. Perhaps I can throw something together with BAGBot that can contact BAG members who choose to be contacted under certain situations - such as after the trial or when one of the {{BAGAssistanceNeeded}} templates is placed on the page - or something. Also, when and if my nomination passes - I nominated to try to speed things up - I will help with that. The LAST thing we need is people resigning, and the BEST way to speed up the questioning phase and ensure that what needs to be asked is asked is community input. I think that most of the community knows about BRFA, less so about this page, and most of them either don't care, don't think they have the knowledge, or don't know they're allowed do comment. We regularly have 100+ people comment on RfA, but if someone posts one line on an admin's talk page, the admin doesn't blank every page he edits. If a bot has a line that says $newtext="";, then we have a bit of a problem, no? (I'm not saying that bots are more important than admins, but if someone watches WP:RFA and the Village Pump, BRFA is the next logical place.) ST47Talk 14:34, 23 April 2007 (UTC)

Just to clear one thing up RM: my resignation is until such time as community support for the process and it's mandate is clear and unambiguous, at which point (be it next week, next year, or next decade) I may well stand again if I have the spare time. I have specifically ruled out just adding myself back to the list, though.

I certainly agree BAG (if it continues) needs more members. I think we need to make a decision about how those members are appointed, however.

I would certainly approve of Rich joining the group, and tentatively (as I haven't researched the guy yet) Eagle101 too. Indeed, anybody with a modicum of common sense and a knowledge of bot operations is welcome to apply and will likely get my support. --kingboyk 14:40, 23 April 2007 (UTC)

I don't get it, I have gotten approval for several bots, and they all were very fast. If you want approval faster, get all your ducks lined up in a row before making your request. State exactly what it does, providing source code helps, get consensus ahead of time and provide a link. Approval is most often delayed due to people not reading the instructions before asking for approval. _InBC 14:47, 23 April 2007 (UTC)

All my approvals have been really fast except for one comtroversial one which ended up being a bad idea anyway. ST47Talk 14:53, 23 April 2007 (UTC)

Which one was that? Birthday Bot or something else? --kingboyk 14:54, 23 April 2007 (UTC)

I was referring to one which essentially cleaned up internal links - foos to foos and other stuff that doesn't need doing. ST47Talk 15:03, 23 April 2007 (UTC)

Ok, well while we are all guessing... should someone ask Cyde what he feels is wrong with the BAG approval process? I mean this whole thing started because of a bot running very fast without on wiki notification after all. I really do think that Cyde might be able to bring another very good point of view to this discussion. There has to be some way to make it so the bot operators don't "skip" this, whatever format it ends up in. —— Eagle101 15:02, 23 April 2007 (UTC)

Cyde is (without meaning to offend, and I'm quite sure he wouldn't take offence at this) a bit of a renegade, a bit of a shoot first and ask questions later kind of character :) That's cool, and ordinarily WP:IAR applies to actions here, but I think there has to be a limit to how far IAR applies to bots. No harm in asking him though; on the contrary I'd like to see some input from him here as he's conspicuous by his absence :) --kingboyk 15:07, 23 April 2007 (UTC)

In terms of approval, the process seeks to establish several things, I think:

1. No technical harm
2. No content harm
3. No community harm
4. Some measure of improvement however tiny.

There are things we look for which we see as certificates of the above, but we don't necessarily need the same set of certificates in each case.

1. Approver knowledge of WP (including the community)
2. Comments by third parties
3. Requests for the run by projects, or editors
4. Results of test runs
5. Users previous record

Clearly factors which "worry" us and make us look for higher standards of proof include number of edits, edit speed, difficulty of reversing, near "dangerous ground", maverick operators.

But really, absent other contraindications, a small test run, followed by progressively larger runs should flush out problems of most types without causing significant or irreparable problems.

Just thinking outloud... Rich Farmbrough, 16:20 23 April 2007 (GMT).

edit rate

(undent) Also not to get picky or anything but I'm digging for where this bot started editing... and the only way to figure that out is to click "previous 500" about 100 times or so... I'm still going, but I noticed this section of the bots diffs where the bot is going at a speed of 71 edits per minute, over the developer set criteria of slower then one operations's pause or 1 edit per second, whichever is slower. See ( Notice on this one there is approx 6 minutes of edits in here, which lends to an edit rate of ~83. , ~7 minutes , ~7 minutes , ~7 minutes , ~7 minutes)

Do note that those are 2500 consecutive edits, for a total time of about 38 minutes in which it was going in excess of the developer set speeds. (See the approval by gmaxwell here). To qoute him "I approved it editing at a rate of one operation interval sleep per operation or 1 edit per second, whichever is slower".

Please do notice I was not really looking for this, just happened to notice it while digging for the start time. Probably now its a moot point, but these speeds were in excess of the developer set amounts that were reported on wiki. Note I'm still looking for that elusive start time! :) —— Eagle101 15:24, 23 April 2007 (UTC)

Not to be picky or anything, but why say this here? This section is about bot approvals being too slow, not cyde being too fast. ST47Talk 15:29, 23 April 2007 (UTC)

Yeah yeah I know, I was too lazy to make a new section, feel free to refactor I will refactor. In any case this bot was running from 15:40, April 19, 2007 (hist) (diff) m Image:1943 arcade.png (Replacing template name per Misplaced Pages:Non-free content/templates.). So its been going for about 4 days now, Xaosflux only noticed it on day 3 of the tasking. Of course nobody could have known the start time without digging it out like I did :), but this may or may not be useful. (See: for the starting contribs )

—— Eagle101 15:43, 23 April 2007 (UTC)

I don't see why it matters when it was spotted? We don't sit with recent changes refreshing all the time (actually, I often watch recent changes to Talk: to keep an eye on usage of my plugin but that's a different issue). We have WP:ANI, WP:AIV etc to keep an eye on these things. --kingboyk 15:54, 23 April 2007 (UTC)

Tangential to the main point, once you have gone back to "older edits" once, you can see a clearly encoded date and time in the URl, which you can modify to your hearts content. Rich Farmbrough, 16:08 23 April 2007 (GMT).

Figures, I just happened to notice it and was not sure if it would be relevant to the discussion now or not. —— Eagle101 16:16, 23 April 2007 (UTC)

Manually approved edits and welcome bot

User:Matthew Yeager is interested in running a bot-assisted welcome campaign (but he insists the edits are manually approved, so no bot approval is needed). The bot account was blocked and the blocking admin asked for comments here. The people watching this noticeboard might be in a better position to comment. CMummert · talk 02:28, 26 April 2007 (UTC)

Isn't a welcome bot the perpetually mentioned archetype of a really, really bad bot idea? --Cyde Weys 03:29, 26 April 2007 (UTC)

If it's a bot. If the user were to do it under his own account, and somehow review the contribs, I would not be opposed to a script-assisted welcomer. ST47Talk 03:58, 26 April 2007 (UTC)

That's the complicating factor. It's apparently some AWB-like setup where he approves a large set of messages and then the script goes through and commits them. I don't know how things like that have historically been treated. CMummert · talk 04:02, 26 April 2007 (UTC)

Resolution has been reached, thank you all for your input. Matthew Yeager 04:43, 26 April 2007 (UTC)

Well, more comments are always better. The suggested resolution is to run the setup with a non-bot name and a slow edit rate. CMummert · talk 04:48, 26 April 2007 (UTC)

I used to do some rather heavy welcoming. I made sure for every person I welcomed I:

1. Signed/edited with my own name/account
2. Put their page on my watchlist
3. Reviewed their existing contributions
4. Responded to any vandalism
5. Gave advice regarding any mistakes
6. Gave encouragement for being useful
7. Answered the questions they leave on my talk page afterwards

As long as you don't skip these things I don't see why any reason why you cannot welcome 200 people a day. _InBC 04:59, 26 April 2007 (UTC)

Policy rewrite proposal

I have started a (hopefully insightful) discussion on the possible rewrite of our bot policy here. Comments are welcome. Миша13 09:51, 28 April 2007 (UTC)

Transmitting in UTF-8

I am having some trouble with UTF-8 encoding with my bot. I am trying to write text to wikipedia that contains UTF-8 characters (pulled from wikipedia in the first place), but am having inconsistent results – some characters are translating properly, whereas others are not (and are coming across as ?'s). All of the characters look proper in my browser on the original page, so it's not a browser issue.

I am using PHP. I tried calling utf8_decode() on the incoming text, and utf8_encode() on the outgoing text, which lead to the condition described above where only some of the characters come across properly. I'm not sure what I'm missing or what I can do to fix it; any help would be greatly appreciated.

For an example of the problem, see this diff. The left side is what it is supposed to look like (manually inserted), and the right is what the bot is putting. Note that while the à (in "U.S. Città di Palermo") is being retained, ā, ō, and ū are turned into ?'s. —Daniel Vandersluis 15:29, 1 May 2007 (UTC)

Never mind, I figured it out with help from the Village post (technical). Thanks anyways —Daniel Vandersluis 16:29, 1 May 2007 (UTC)

Cydebot's new task of GFDL standardization

Just a heads-up, Cydebot is now taking on the large task of Misplaced Pages:GFDL standardization at a rate of one edit per second. I should be able to get through all of this in two days, maximum. So don't worry, nothing's gone wrong, and I'll be able to handle all of it soon enough. This is one of those wiki-wide tasks where it's helpful if all of the edits came from a single account, for tracking purposes. --Cyde Weys 03:52, 7 May 2007 (UTC)

Sounds good, and thanks for alerting the community beforehand :-). I'll trust that there's consensus somewhere. Also, don't forget the #Maxlag parameter. —METS501 (talk) 04:32, 7 May 2007 (UTC)

Roomba to run again.

Just FYI: I haven't run roomba for a long time due to the lack of a suitable live slave database to pull its input data from. I've fixed that issue tonight, so I'll soon be running it again to tag all the non-free images which are not used in the main namespace as orphaned. This is the first step in the deletion process for non-used non-free images. This is the same task that roomba has always done but since it has been so long since I've done it roughly 7.5% (26k) of our non-free images will need to be tagged. It won't be running at high speed or anything like that, though I check maxlag. --Gmaxwell 04:47, 7 May 2007 (UTC)

Ah, betacommand tells me that he's finally got around to coding similar functionality. I'm always glad to let someone else take heat, so I'll leave it to him for now. --Gmaxwell 05:16, 7 May 2007 (UTC)

Children of Curpsbot

Please see Misplaced Pages:Administrators'_noticeboard/Incidents#Children_of_Curpsbot regarding the secret use of adminbots. Dragons flight 03:58, 8 May 2007 (UTC)

Logins seems to stop working in Perlwikipedia

It looks like logins stopped working in Perlwikipedia (probably because of latest changes in security).

In sub _get

when doing

my $res = $self->{mech}->get($url);

where $url is http://en.wikipedia.org/search/?title=Special%3AUserlogin&action=edit It gives 403 Forbidden.

Any ideas how to make it work?

Alex Bakharev 10:33, 9 May 2007 (UTC)

The devs have blocked the Perlwikipedia user-agent. A dev (User:Simetrical) tells me it could be because it doesn't contain descriptive contact information, so I've changed mine to 'Bot/WP/EN/ST47/BotName', as that gives necessary identification and contact information. ST47Talk 10:43, 9 May 2007 (UTC)

Sorry, I have to change

$self->{mech}->agent("Perlwikipedia/$VERSION")

to

$self->{mech}->agent("Bot/....)?

Is this correct? Alex Bakharev 11:05, 9 May 2007 (UTC)

It works! Thank you very much! You are great! Alex Bakharev 11:09, 9 May 2007 (UTC)

LDBot creating pages from....where?

Note: I asked this on lightdarkness's talk page, but he does not edit often anymore so I am crossposting here.

Where is LDBot pulling the source for the next AFD page from? It doesn't appear to be importing Misplaced Pages:Articles for deletion/current, because these edits occurred hours before this page creation. Please advise. -- nae'blis 15:42, 11 May 2007 (UTC)

It might just have memorized what the correct layout of the page is, like Sandbot. --ais523 16:09, 11 May 2007 (UTC)

Oof, that seems like a shortsighted maneuver when the bot runs automatically. -- nae'blis 17:46, 11 May 2007 (UTC)

CAPTCHA

What is exactly sets off a CAPTCHA requirement to log in? Sometimes I log in no problem, but then suddenly my bot is being challenged with a CAPTCHA even though it never got the pass wrong.

Does a single wrong password make it so you need to pass a CAPTCHA for X minutes? Is there no way bots can be exempt from this? _InBC 18:52, 11 May 2007 (UTC)

just log in with the web interface and you'll fix the CAPTCHA. Betacommand 18:54, 11 May 2007 (UTC)

That is what I thought, but I found I was still being challenged after a successful log in(just for a few minutes), so I am thinking now there is a timer. _InBC 23:11, 11 May 2007 (UTC)

You have to confirm your e-mail address. Then it shouldn't bug you anymore. – Quadell 16:24, 24 May 2007 (UTC)

Incidentally, either I'm overlooking something or e-mail confirmation is down right now. When logged in as my bot User:polbot, I tried confirming my e-mail, but I never got an e-mail. I left a message at Help talk:Email confirmation, but I don't know if anyone ever looks there. Does anybody know what's up with this? – Quadell 16:28, 24 May 2007 (UTC)

Exclusion code for User talk pages

From looking through some of my bot's edits and the edits of other bots that primarily issue notifications on user talk pages, I every so often see a user removing these notices. Sometimes they even put in their edit comment something like: "remove unwanted bot notices AGAIN". I know that a few bots already have their own exclusion codes, but I think it would be convenient if a standard way is devised to exclude certain, or all bots, from user talk pages. This would only apply to bots issuing notices, such as duplicate image warnings, image copyright warnings, speedy deletion warnings, etc. Bots that are warning a user to stop vandalizing or something of that nature should probably ignore any exclusion codes for the most part. My proposal for the exclusion code is a simple HTML comment of the form:

<!--Diallow:BotName-->

and to disallow all bots:

<!--Diallow:ALL-->

How does this idea sound? I created a topic on this here but this is probably a better place to have the discussion. --Android Mouse 17:01, 21 May 2007 (UTC)

{{bots}} is what your looking for but not all bots are compliant. (mine is not on purpose) Betacommand 17:24, 21 May 2007 (UTC)

Thanks, I'll make mine compatible when I get the time. But why are there so little bots compliant? Lack of publicity to this template? How would a user even find out about this? --Android Mouse 17:40, 21 May 2007 (UTC)

most bots need to leave their messages, Mine has to per policy. Betacommand 17:46, 21 May 2007 (UTC)

Not wanting a bot to post a message to your page is often not reason enough to not allow for the posting that message. 17:57, 21 May 2007 (UTC)

You have a point, but how do you distinguish what should be disallowable and what should not? --Android Mouse 21:40, 21 May 2007 (UTC)

I would say that messages intended to help the user should be disallowed, while warning and similar messages telling the user something should not. For example, a warning about an orphaned fairuse image is not necessary, as the user is not mandated to take any action. On the other hand, an antivandalbot message shouldn't be disallowed. Tizio 16:21, 22 May 2007 (UTC)

That seems pretty reasonable. Anything where the party MUST be notified cannot be blocked, but most bots should allow users to opt out through {{nobots}}. -- nae'blis 16:24, 22 May 2007 (UTC)

That is true, H, but it's reason enough to remove it. Gracenotes § 16:11, 13 June 2007 (UTC)

Sandbot

Sandbot (talk · contribs) appears to have ceased editing. The last time that this happened, I enabled sandbox raking in User:Uncle G's 'bot (which rakes sandboxes on several projects). I have done so again. Uncle G 15:52, 22 May 2007 (UTC)

Interwiki question

Hi, are there any bots active doing a full namespace sweep starting from this wikipedia? I noticed it took quite some time (about 3 months) for an interwiki link to propagate from the English wikipedia to the Dutch wikipedia. multichill 10:19, 30 May 2007 (UTC)

As it takes about two weeks to do the 300k articles in nl.wp, it will take a little less than three months to complete the whole en.wp namespace. I guess it's frustrating for the interwiki namespace count, so that no one actually tries to complete a full en.wp namespace sweep. Will you be one of the first to try? :) Siebrand 10:23, 30 May 2007 (UTC)

I could give the category namespace a try to see how much new links i encounter. multichill 10:49, 30 May 2007 (UTC)

Server Issues

We just found out that we're having some server issues. If you run a non-essential bot (ie: If it's not anti-vandalism, or working with AIV, etc), please take it offline for now to help. Archiving, etc can wait a few hours. ^demon 02:07, 4 June 2007 (UTC)

Alright, I think everything's back to normal. —METS501 (talk) 02:37, 4 June 2007 (UTC)

the servers have been fixed, but lets give the servers ~5 hours to get back to normal. Betacommand 02:39, 4 June 2007 (UTC)

Preemptive open sourcing

I've seen a few bots disappear over time because their owner had to take a wikibreak for whatever reason. Generally this means to other users that some expected functionality no longer works. It would be kind of neat if, when bot X drops out, some other bot takes over its functionality; in order to make this work best, it would be nice if bot owners posted the source code to their bot somewhere on wiki. What do people think of this? >Radiant< 08:17, 19 June 2007 (UTC)

I think it's a fine idea; I do it myself. Example: User:MadmanBot/Source/Task2. It would also help new bot programmers learn some of the basic techniques. — Madman bum and angel (talk – desk) 08:26, 19 June 2007 (UTC)

I do this as well. At Misplaced Pages:Bot_policy#Good_form, our policy suggests that bot-owners "Publish the source code of their bot (unless it's a clone)". – Quadell 10:39, 19 June 2007 (UTC)

Does the BAG ask for a bot source before approving it? >Radiant< 11:52, 19 June 2007 (UTC)

Not generally. But if the user is inexperienced, and doing something complex, regex's and such may be requested. Reedy Boy 12:28, 19 June 2007 (UTC)

There are various options for this. Here you can find a wiki dedicated to the development of bots that serve MediaWiki and its users. Once a bot has been found to be stable there, and if it was written in Python, we/I usually commit it to the pywikipedia bot framework, as was recently done with welcome.py, imagecopy.py and several other tools. Proper documentation and readable code is a requirement before code is committed. Please participate there if you require feedback on your bot code and if you would eventually like to have your bot code committed to the SourceForge project pywikipediabot. Bots in all languages are welcome, although we currently only have a formal project for bots in Python. As a last resort: each toolserver user can commit code to a publically available Subversion respository. You could ask any tollserver user to add your code to it, so that you would have a stable distribution platform for it. Cheers! Siebrand 12:35, 19 June 2007 (UTC)

I dont openly post my code because I know in the wrong hands it can be very dangerous. But if a trusted user would like the code I will give it to them, under the condition that they dont pass the code out. Betacommand 15:29, 19 June 2007 (UTC)

Half the code for my bot Bot523 (the bit that does the editing) is available on the wiki as User:Bot523/monobook.js (one of the advantages of a JavaScript bot). The other half is nonportable and a bit of a mess (it generates a shortlist of pages that might need editing and opens the relevant page so that the monobook.js side can edit it if needed), so I haven't published it online, but would email it on a good-faith request. --ais523 16:32, 19 June 2007 (UTC)

Are there any essential daily/weekly bots that aren't open-source, where if the operator (God forbid) were to die suddenly, we'd be up a creek? – Quadell 16:39, 19 June 2007 (UTC)

Well, the loss of HagermanBot has made an impact, and it seems he did not publish his source code; I believe E is trying to get in touch with him now. As I understand it, the code for MartinBot is available... is the code for the MiszaBots and the AIV helperbots available? Most of them seem to be available on demand, and not published, which doesn't solve the problem in your hypothetical situation. — Madman bum and angel (talk – desk) 17:10, 19 June 2007 (UTC)

I plan to do a re-write of MartinBot at some point, and will make that open-ish source (ie - access will be freely available to those who request it, and who have a positive history). I need to get the source of RefDeskBot out there too soon, and MartinBotIV perhaps? Anyway - if anyone ever wants source, I am usually happy to pass it on when I have the time to clean it up. Martinp23 19:40, 19 June 2007 (UTC)

BAG Joining

Hey, I have been asked to post a notification of my request to join the Bot Approvals Group on here. It can be found at Misplaced Pages talk:Bots/Approvals group#Joining. Thanks! Matt/TheFearow (Talk) (Contribs) (Bot) 02:08, 22 June 2007 (UTC)

Oh good. We need more BAGgers. – Quadell 03:10, 22 June 2007 (UTC)

Python help

I'm interested in creating a bot called SpebiBot, but I can hardly understand Python. The bot's tasks are to substitute unsigned and vandalism warning templates. This doesn't seem like such a complicated idea, however, if you can help out, any help will be much appreciated. Thanks, –Sebi ~ 02:11, 23 June 2007 (UTC)

I can write this, But it will be a few days. Betacommand 02:14, 23 June 2007 (UTC)

If you don't know python and want to learn programming, then I'd recommend you learn C instead. It seperates the real men from the boys :p --Android Mouse 02:18, 23 June 2007 (UTC)

No, that's puberty. Mike Dillon 02:32, 23 June 2007 (UTC)

yep the boys use C and men use python. I use the m:Pywikipedia bot framework Betacommand 02:26, 23 June 2007 (UTC)

A good language for learning programming in general may or may not be the language best suited to do some specific task. So in a sense you're both right. Gimmetrow 02:59, 23 June 2007 (UTC)

Thanks for your help :) –Sebi ~ 03:15, 23 June 2007 (UTC)

Edit token structure changed

The edit token string now includes a plus sign near the end; that is, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa+\" instead of "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\". This means that the edit token must be encoded when sent to the server in a POST request, whereas previously an edit would have worked without encoding. If not encoded (+ → %2B), the edit will not be made due to "loss of session data". Gracenotes § 22:08, 1 July 2007 (UTC)

Ah, I see that. I encoded it anyway, but thanks for the heads-up! I wonder why the change and why there wasn't any discussion that I could see. — Madman bum and angel (talk – desk) 03:13, 2 July 2007 (UTC)

Ah!!! That must be it!! I've been killing myself trying to figure out why I keep getting a damned loss of session data message on every edit! Thanks. —METS501 (talk) 03:15, 2 July 2007 (UTC)

• Misplaced Pages bots
• Misplaced Pages noticeboards