Misplaced Pages

This is an old revision of this page, as edited by Crotalus horridus (talk | contribs) at 07:18, 31 January 2006 (Revert to consensus version.). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

See the recent comments on the talk page for current discussions about the relevancy of this page.

There's a lot of talk about this 'policy' which attempts to divine meaning from things other people have said rather than just asking for details.

Complicated templates-within-templates generally ought to be thought twice about before being used, because they can be confusing and fragile. There are some good notes about that on this page; please don't go all willy-nilly with illegible code just because it sort-of works.

There are other notes on this page about server performance which are not necessarily clear or well-supported. In particular, there's no known evidence that moderate usage of meta-templates has any noticeable impact on server performance.

While there are potential issues with cache invalidation, that's a separate issue which can be separately solved -- and is little better with "regular" templates.

I'd like to ask that anyone fighting against ugly, fragile meta templates at this time do so based on their ugliness and fragility. Please don't go around claiming "the developers" laid down the law and said nobody can use meta-templates because they hurt the servers; that just isn't true.

--Brion 03:25, 21 January 2006 (UTC)

• ]

Template messages allow certain standard text to be included on many pages, usually with the idea that in the future, any changes to that text block can be changed in one place. The term "meta-templates", as used in this article, refers to nested templates which are created and used to add functions or formatting to other templates. They are extremely useful and convenient for many purposes.

However, meta-templates can also be fragile, hard to understand. they increase the number of database reads per page view affect back-end server performance; though the developers are not in agreement over whether such server impacts are significant. For these reasons, it is better to avoid meta-templates when clearer and more robust alternatives are available, such as those described below. Contrary to some perceptions, there is not a developer mandate that all meta-templates must be eliminated, but they should not be abused when more sensible alternatives are available.

In the drive to eliminate meta templates that occoured from the aforementioned perceptions of "no meta templates period" sometimes meta template based hacks were replaced with other possiblly worse hacks. An example is the "hiddenstructure" css class which was introduced to hide empty sections of infoboxes without using the meta template based if system. Unfortunately, it only works for browsers supporting css, reducing Misplaced Pages's accessibility. Another proposed soloution was to use an extra parameter together with the optional parameters system to supress the structure but this relies on adding extra code to every page that uses the template. Ideally, hacks (whether through meta templates or otherwise) should be avoided but the slow response of the small and overworked development team makes a hack that can be implemented immediately by a normal editor or even a Misplaced Pages "admistrator" look very attractive.

Finally if there is a choice between messy unintuitive structures in articles and messy unintuitive structures in templates messy structures in templates are preferable as templates are generally maintained by experianced wikipedians whereas articles are the target of most newbie edits.

Use of meta-templates

The use of metatemplates should be avoided in almost all circumstances. If you are considering using a meta-template, ask these questions.

1. Is the end product essential to Misplaced Pages, or is it a primarily decorative feature? Meta-templates that are not essential should be avoided.
2. Is the template likely to be high-profile? High-profile templates cause more server load, and so are less appropriate for meta-templates.
3. Is the desired effect only achievable through a meta template, or can a template of basically the same appearance be made without them? If the same effect can be achieved differently, even if it is more difficult, a meta-template should be avoided.

You should not use metatemplates unless you have a really good reason that you have to.

Harmful effects

The impacts of meta-templates include not only direct server load effects but also indirect effects, such as creating vandalism vulnerabilities.

Complexity

Some nested template schemes are so complex that they are prohibitively difficult for the average editor to grasp. This is particulary true in non-technical areas where the subject expert, who knows best what information should be presented in a template, is not able to edit the template due to the use of esoteric coding. As a result, routine maintenance and changes are neglected, improper usage can proliferate, and innovation suffers. The solution, meant to become an easy replacement, becomes more difficult than the function it was meant to improve. Clear article source and clear template source are most preferred, though it is understandable if complex code is moved to the template. If this is the case, steps should be taken to reduce complexity to a minimum.

Server load

The developers have noted that nested templates increase the number of back-end database calls, as each nested template is fetched and evaluated internally. As is typical of websites that use a database and cache scheme, extra database processing is more "costly" in terms of resource availability than the page caching mechanism. How much more costly meta-templates are than templates, and whether this has a noticable impact on server load, is (as of early 2006) a matter of debate. Also, when a nested template is edited, that change results in the cached version of every page using that template, whether directly or indirectly, to be marked invalidated. Pages using nested templates must be rebuilt, bringing a lot of activity onto the database servers. In some cases, editing meta-templates has caused enough server stress to temporarily lock the database.

Vandalism

As with any template which is used on a very high percentage of pages, a nested template is an excellent denial-of-service attack vector, since changing it or any component used in it would flush a substantial percentage of the site caches, which are critical to site performance and normally serve some 75–80% of all hits. Making even one subtle change, like the addition of a space, causes the effect. This has led administrators to protect some of the highest-risk meta-templates. Sometimes, though, a sub-sub-sub-template could be vandalised with offensive content. These could be difficult to notice immediately, because the sub-template is not edited often and appears on only one or a few Watchlists, and difficult to track down in a timely manner.

How a page is built and cached

Here's some technical background which may be of use. Jamesday 07:52, 13 Feb 2005 (UTC) and clarified by LarryLACa 00:22, 11 November 2005 (UTC) (Clarifications show in 's).

[Re: Rendering impact: The first time a page is viewed, i.e. after it has been removed from caches, the page request causes several steps:]

• Each item in the base page portion is requested from the database (images and CSS aren't in the main part). The page you edit, each template, each template included in the template and so on. Two templates, two database records to be retrieved. One template on its own, one read, one template including another, two. Plus the one for the base page.
• Once that and the rest of what is called the parsing is done, the page is saved in the parser cache. That's kept in RAM in memcached.
• Finally the skin is applied and the page is passed on to the Squids, which cache it in RAM and on disk (to get larger capacity but at slower access time) for all who aren't logged in (will only be useful if it's the normal skin) and send it on to the person who originally requested it.

• Whenever any part of a page is changed, be it the page itself or a template or image used in it:
  • The page is marked as changed ("touched") and will be regenerated next time it is requested. Both the Squid and parser caches have it removed. Necessary so people see the correct version.
  • The marking process involves a database update to every affected page, which for many pages can produce "replication lag" , with outdated information displayed to those using the page. This (the marking process) also shows up as slower response times when the lag is less than ten seconds. The effect is minimised by affecting small numbers of pages and maximised by affecting a large number, in part because the wait of up to ten seconds makes batches in the few thousand range effectively invisible except for delay in page load times. Touching about 18,000 pages currently takes a database slave with 4GB of RAM, 6 drive RAID 10 array and write caching disk controller some 90 seconds (that's from a real touch operation).
    • Assuming even distribution across 18,000 pages, Each of the 8 edits would flush from cache one eighth of the pages in each edit. On the database side, lets look at that 90 second case:
    • 90/8 = 11.25 seconds, call it 12.
    • Those who view in the first 2 seconds will wait ten seconds then see out of date information.
    • Those who view in the remaining ten seconds will see delay of up to ten seconds and then completely current data.
    • So, splitting it has removed most of the visible lag and visible problem.

The Squids, because of the limitations in the way they can work, with much less work per page, are inherently the fastest way of serving the pages and just 4 machines can serve some 75% of all hits to the site. But they are restricted in what they can serve. Next step is using the parser cache via the apache web servers. That allows all of the user settings for logged in people but uses more web server CPU so it's much less efficient.

We could switch everyone to using the apaches but that would be far less efficient and would require something like 4-5 times as many apaches and database servers as we have today, far more than the 4 machines gained by not using them as squids. And the page views would be slower, because it's an inherently slower process.

While all template use causes an extra database read and flush all pages using it, meta-templates are a special case because they use twice the number of database queries and can cause flushing of many more pages than other templates. If a meta-template is used in only a few or perhaps a few dozen templates which are fundamentally unrelated, it's debatable whether the extra equipment costs are worth it, compared to the relatively modest work involved in updating the individual templates. The replication lag issue can't so easily be addressed — that work is done by however many database servers are purchased. We're trying to reduce the effect but updates affecting many pages are inherently more problematic in this area than those affecting fewer.

Alternatives

1. MediaWiki needs developers. Metatemplates that did not cause the server hit would be of great benefit to the project, if someone can work out how to implement them in such a fashion. m:How to become a MediaWiki hacker. Code is equally needed to replace the meta-template based If hack.
   • Please read "The touching process involves a database update to every affected page". There are changes in 1.5 which will help. How much is currently not known. Jamesday 14:55, 6 August 2005 (UTC)
2. Design, document, and implement — To give an example in the case of Misplaced Pages:Sister projects, a proposal was made to use a meta-template. In this area, it is much better to decide on a common look and format, document that standard for easy reference in case new related templates are needed, and then implement it across the few templates being used. When changes are needed, this gives one central place for discussion and revision. After there is consensus for the change, interested editors can quickly apply them. Creating a page which displays each template also helps to locate templates which don't follow the agreed upon format.
3. Make use of CSS — Some meta-templates serve only to produce a specific visual format — such as size, position, color. If these were identified, CSS classes could be added to the site's global stylesheets. The meta-templates could then be replaced with the CSS classes in relevant pages. This would accomplish the same purpose — maintaining uniform style across the site — without placing a burden on the server. This also allows the visual style to vary depending on the skin or user agent.
   • This would be very useful. I've previously discussed difficulties with using CSS, notably lack of project and page-specific CSS and the inability to use the span tag, which is part of the blocked HTML set. Jamesday 14:55, 6 August 2005 (UTC)
     • Hmm?? Span tags were added to the whitelist in December 2004. — Omegatron 01:47, 17 December 2005 (UTC)
4. Use lists, not templates or categories — Some templates and categories are used with the goal of helping editors find articles under a specific topic area of interest. For this use, it is often more valuable to create a list, which can be annotated and prioritized. Many WikiProjects already maintain an area for reporting articles that need work. (Note that categories also heavily load the servers, for similar reasons.) See Misplaced Pages:Categories, lists, and series boxes
5. Use {{subst: }} when creating the daughter templates. This copies the text message to the daughter template (substitution) rather than causing a transclusion, thus eliminating the second template call. (See the descriptions of the use of subst: here and here.) Of course, this means that when the meta-template is changed, the daughter templates won't update, and won't be tracked by the What links here feature, which probably defeats the point of using a meta-template in the first place: If all of the daughter templates are to be updated, this would then have to be done by hand, expending a considerable amount of time, work and bandwidth. It's very unclear that this is better than just listing the templates and documenting the format in use on a project page for easy copy-and-pasting — see "Design, document and implement", above.
   • You can also do this on the end user pages. Where it is unimportant that the latest version is displayed this is a useful approach. Having the very latest version of say a stub message on a page, when that status will change only if the page is edited and the message can then be updated, doesn't seem to matter very much, since the message that it is a stub is the important part. Jamesday 14:55, 6 August 2005 (UTC)
   • Actually using subst with stub templates is a bad idea, see discussion at Misplaced Pages Talk:subst (archived here). Since changes to the stub categories can require revisiting all articles currently tagged with a particular stub. and for several other reasons, these should not be placed with subst. However subst can be and is used to generate new stub templates from the metastub template. DES 01:53, 17 December 2005 (UTC)
6. Protection — Meta-templates that are used in many instances but rarely changed can be protected, so that they can only be edited by administrators. This prevents vandalism and reduces the server load problem; if the meta-template is never changed, the daughter templates don't need to be updated. Note that this creates a permanantly protected page, which is also to be avoided.

See also

• Misplaced Pages:Transclusion costs and benefits
• Misplaced Pages:High-risk templates
• Misplaced Pages:HiddenStructure