[Discussion] - Interwiki Sidebar Replacement - O5 Command

https://scuttle.atlassian.net/browse/TT-48

EN’s current interwiki (which is developed and hosted by RU, and will be referred to in this document as the ‘RU interwiki’) has worked without issue for a long time but falls short of being perfect in that it isn’t able to be styled with CSS. CN have developed a new version of the interwiki (referred to in this document as the ‘CN interwiki’), pioneered by 7happy7 and sekai_s, which is able to be styled by CSS. It is currently in use on SCP-CN and WL.

This document will be a record of EN’s investigations into the CN interwiki, what it does well and what its shortfalls are, along with attempts to remedy those shortfalls, with a view to creating a new interwiki that can be used by all branches across SCP and WL (referred to in this document as the ‘new interwiki’).

CN interwiki research

In order to effectively rework the interwiki in an informed manner, it’s important to understand how it works.

Supplementary iframes

The CN interwiki uses supplementary iframes calling styleFrame.html to set the appearance of the interwiki. There are three kinds of supplementary iframe:

On http://scp-wiki-cn.wikidot.com/theme:anon, (Parawatch Anon theme) both styleFrame and customStyleFrame are active and are called with different CSS to each other. However, in general usage via page include, only customStyleFrame is active. The two have CSS that are similar in structure, but do not share any declarations.

Found it - the Parawatch Anon theme is an extension of the Creepypasta theme. The Creepypasta theme's styleFrame is identical to the Anon theme's styleFrame, which is presumably just copied across as a partial fork even though the rest of the theme is an extension. The Anon theme's customStyleTheme is its own styling. This usage is inconsistent and confusing. The 'partial fork' is because the Anon theme page does not include the Creepypasta theme, only imports its CSS, and therefore the styleFrame has to be duplicated separately. In normal usage of the Anon theme, though, both the Creepypasta and Anon themes are fully included, so the Creepypasta styleFrame is present, meaning duplication is not necessary. The Anon theme could probably resolve the duplication by including the full Creepypasta theme, and hiding it, rather than importing just the CSS.

(After investigating: styleFrame causes CSS to be prepended and customStyleFrame causes CSS to be appended. These effects don't match the naming scheme at all, resulting in the confusing usage.)

However, all of this does highlight the importance of taking theme extensions into account when redesigning this module.

Perhaps one way to sort out sub-theme priority, rather than appending and prepending willy-nilly, would be to have each added CSS assign itself a weight (like a z-index) and sort them accordingly? (Yes: see Execution Order)

But it would be much better just to somehow get them to stack on top of each other one by one like a real cascading style sheet. Can I rely on Wikidot's execution order to do that for me? (No: see Making the iframes wait for each other)

Additional iframes executing builtin code

I encountered another kind of iframe on http://scp-wiki-cn.wikidot.com/theme:swirling-ashes: (Swirling Ashes theme):

So it looks like some themes have their own iframes. Investigating the code of the above iframe, which is stored on :scp-wiki-cn:component:bhl-dark-sidebar, it is an approximate copy-paste of styleFrame.html except its active ingredient is window.parent.window.interwikiFrame.bhlDark();, which is a hard-coded function in interwikiFrame.html. Not sure what the BHLDarkFrame name is used for just yet.

(Interestingly, the dark sidebar component already creates that iframe, meaning the identical iframe in the CN Swirling Ashes theme would be redundant - but it doesn't actually include the dark sidebar itself, it just copy-pastes most of the sidebar's source code to construct the theme page. No idea why.)

All bhlDark() does is insert css/style-bhl-dark.css, which is hardcoded and kept in this repository. bhlDarkCheck() is executed immediately and checks for an iframe with name="BHLDarkFrame" and, if found, executes bhlDark(). (So it’s an equivalent of customStyleCheck(), but just for this specific bit of CSS. See the section investigating that function)

style-bhl-dark.css, based on its usage in the Swirling Ashes Theme, is more related to BHL Dark Sidebar than to Extra Black Highlighter, which is a full dark CSS theme - which is fine, because EBHL actually extends BHL Dark Sidebar. However, style-bhl-dark.css contains much less CSS than the code in BHL Dark Sidebar.

I would like to remove specialised builtins where possible, as it means that there are more things to update when their source changes, it’s an additional technical debt to maintain, and it invites the possibility of later adding further builtins which would further increase that debt. See section Removing hardcoded CSS.

Tracking added CSS

There is a styleAry ("style array") that tracks CSS strings that have been added to the custom style element. This is used to prevent duplication in case later incoming CSS is identical.

This is a good idea. It would be better to instead insert each new CSS string into a new <style> element, and check incoming CSS against those. It would then be possible to instead move that CSS to the top of the pile when identical CSS is encountered, though I'm not sure if that's a good idea or not.

Also, I realise that by not putting each new CSS string in its own <style> element, @import statements are blocked for any new stylesheets except the most recent invocation of styleFrame, because @import must come first. It is essential that each CSS string gets its own <style> element.

See section Execution Order, which also addresses this issue.

interwikiFrame URL parameters

It occurs to me that not only is styleFrame.html called with query parameters, but interwikiFrame.html is also called with parameters. For context, interwikiFrame.html is executed in nav:side on SCP and _template on WL, both inside a ListPages module configured to list the current page.

See section New interwikiFrame URL parameters.

Differences between SCP-CN and WL implementations

It's worth noting that the component for WL's interwiki seems to have its own copy of the branches configuration, which is much more complete than the configuration in this repository: Wl Interwiki Inc - The Topiary

It's split up weirdly, no doubt to get it to work on Wikidot through page sources rather than hosting it anywhere, but it is pretty much the same as CN's version. (I know that 7happy7 pioneered both, but I'm not sure which came first.) It removes a lot of things to simplify it for WL's use case instead of trying to be agnostic and support SCP as well. It also adds an elemSort() function: Wl Interwiki End - The Topiary - which fires 6 times at the end of getDataAll(), at 1-second intervals, but I'm not sure yet what it does or why it's needed.

WL also uses Getwikimodule - The Topiary It is identical to this repository's getWikiModule.js, but allows any module to be specified rather than assuming the user wants PageLookupQModule. Will have to investigate why this is necessary.

siteNum

What is siteNum?

• It is initialised to the number of configured branches.

• It is reduced by 1 if the current site is present in the branches config (during getDataAll()).

• It is reduced by 1 for each translation info retrieved from remote.

• If after getting the data for any site it is equal to or less than 0, changeStyleCheck() is executed, which changes the style of the interwiki based on what iframes are present in the page.

I think it's a counter to see if the data for all sites has been gotten. siteNum should be removed and the code that executes after checking it should be placed at the end of getDataAll().

Ah! Actually not - the getWikiModule() callback is asynchronous, so firing it at the end of the synchronous process wouldn't work. First thing I should do is check whether it actually makes sense for it to be checked after the asynchronous process, and if it can't just be done beforehand (e.g. if it's CSS-only, it's fine to only do it beforehand).

But all getData() does is get the link to another translation, if it exists. There's no possible way that adding CSS after the fact could possibly be affected by this - right?

I have removed this flag, and will recover it or something like it if it turns out to be necessary.

Style containers

There's a differentiation between inStyle and cuStyle. What is it?

inStyle is the style element that contains the interwiki base styling, determined by switchStyleType(), the styleType flag, and the style interwikiFrame.html parameter.

cuStyle is the style element that contains any additional styling added by styleFrame.html. CSS can be either appended or prepended to this, but it can never be parsed before inStyle.

I'm not removing switchStyleType yet, but I'm going to rename inStyle to baseStyle or something.

cuStyle will become several style elements. (See section Storing styles in their own style elements)

I'm not yet sure how to handle appending vs prepending. Will need to investigate how reliable the cascade order for iframes is. (See section Making the iframes wait for each other)

Finding matches for fullnames near the truncation limit

The interwiki works by exact fullname match. There's a length limit on fullnames, something like 60 characters. This is irrelevant, because it's by exact match only, right?

Nope! The interwiki supports the ability to restrict its search to a certain category - e.g. while WL EN has its own site, WL CN is kept in the wanderers: category on SCP CN. This results in that string being appended to the effective fullname, and if the fullname was long enough, it might be truncated, causing the exact match to be lost.

The PageLookupQModule is helpfully naive, though, and actually returns pages whose fullname starts with a substring rather than exact matches for that substring. So we can probably leverage that functionality to overcome this issue.

The existing codebase handles this by checking if the current page fullname is greater than a given length and setting a shorten flag, indicating that the fullname might be shortened on other wikis. Then, when results are returned from the remote, if the shorten flag was set, it checks for pages in the returned list whose fullnames start with that value rather than exact matches.

However, I don't believe that the existing implementation here is correct.

• For one, the PageLookupQModule is always called with the fullname of the current page, and looks for pages starting with that fullname. It doesn't appear to take into account either the current branch's scoped category, or the target branch's scoped category. (Scoped category being either "" or "wanderers:".)

  • Because it doesn't take the current category into account, it won't find pages on other branches whose fullname matches, just without the category.

    • Caveat: The current category of the pagename parameter to interwikiFrame.html is straight-up removed, so actually all pages are considered to be in the default category for the purposes of the interwiki. That's helpful for e.g. EN adult pages, but is it desirable? Correction: No it's not - only _default: and the scoped category is removed.

    • On EN for every adult page there is a fullname and a adult:fullname. If the category is ignored, either page could come up in the translation, but the page without the category is preferred.

    • I think the interwiki should just go by exact fullname match, including category (obviously except where a static scoped category is configured).

  • Because it doesn't take the target category into account, it won't find pages on other branches whose fullname matches, just with a scoped category.

• Although the script makes an attempt to check for pages in the returned results that start with the current fullname but are not exact matches for the current fullname, the returned results will only include pages that already start with the current fullname - and that wouldn't include a scoped category. As a result I don't think that this check would ever accurately pass.

  • The changes of this error occurring are reduced by the fact that the check only happens if shorten is set, which is only if the fullname is near the truncation limit.

  • Correction: This check was actually checking for when the category was removed and the resultant fullname, while it was stripped when it had the category, would not have been stripped without the category, and therefore there wouldn’t be an exact match. I still need to account for this.

These issues need to be resolved in the new interwiki.

changeStyleCheck()

This function does the following:

1. Finds the iframe with name "typeFrame", if it exists, and sets the base theme to its value.

2. Finds the iframe with name "styleFrame", if it exists, and handles it (prepending the CSS to cuStyle).

3. Finds the iframe with name "customStyleFrame", if it exists, and handles it (appending the CSS to cuStyle).

Quite a few problems here.

• iframe names are not unique - perhaps they should be unique, theoretically, but I can't imagine that being guaranteeable in normal usage. There is a good chance there will be multiple customStyleFrames on a page, particularly when themes extend other themes - and by doing it this way, only one of those will be seen.

• You'd only get two batches of CSS this way - one styleFrame and one customStyleFrame. That's it. That's the limit.

• There's no indication of the difference between styleFrame and customStyleFrame - how does a theme know which to use, especially if the theme's creator isn't well-versed in these sorts of technical details? An arbitrary and poorly-indicated choice like this would be prohibitive.

• It's totally redundant. Those styles get added via styleFrame.html, which actively calls a function within interwikiFrame.html to inject its styles. This method makes sense only if the best way of CSS getting to the interwikiFrame.html is via it passively searching for styleFrames. (Note - actually not redundant - see the end of this section)

  • This may be a pretty good idea, however. A styleFrame only knows about itself, and can push its data to the interwikiFrame contextlessly, but the interwikiFrame can potentially know about all styleFrames and then use this information to determine the CSS cascade order. This could solve any concerns I had about the execution order. Regardless, the two methods should not coexist.

  • It's worth noting that if I choose to go that route, the method by which I order the iframes cannot simply be the order in which they appear in the document - for example, WL's base iframe always appears at the bottom of the template and therefore would always be seen last when actually it should come first.

• It requires [[embed]] in order to be able to add the name attribute to the iframe, which Wikidot normally doesn't allow. Nothing wrong with that, per se, I just don't like it. I'd much rather be able to use [[iframe]] or even [[html]] for more complex stuff. It feels unclean.

For sanity's sake, I'm going to make sure that my assumptions of redundancy are accurate by removing changeStyleCheck from the current version of the CN interwiki and see what happens. I commented out the internal code of the function but left everything else, so the function simply does nothing.

This happens, sometimes but not all the time: Uncaught TypeError: window.parent.window.interwikiFrame.changeStyle is not a function

I think that the styleFrame often loads before the interwikiFrame, but executes immediately anyway, even if the interwikiFrame isn't done loading yet. So... I need to make it wait. How do I do that? Is there a reliable way of doing that from the styleFrame? Or do I need to do it passively from the interwikiFrame, like changeStyleCheck() was doing? Bearing in mind, of course, that I need to be able to get styleFrame to worth both before and after interwikiFrame has loaded, and I think that's what changeStyleCheck and the function in styleFrame were for, respectively. Plus all the wildcard error catching - it was guaranteed that one of the two would fail, every time. But at least I get why changeStyleCheck exists now.

Changes needed to make the new interwiki

Making the iframes wait for each other

So, interwikiFrame and styleFrame are independent iframes and don't share a connection with each other except via window.parent. Either can load first, and when it does, it'll immediately execute.

I think the ideal here is for one iframe to secure a reference to another, wait for it to load with either the load or DOMContentLoaded events (both will be roughly equivalent because both iframes have so little work to do, but DOMContentLoaded will fire earlier). I need to check, though, at what point exactly an iframe exposes itself to window.parent, because I might not be able to secure a reference to it at all until it's finished loading, and then neither iframe will have anything to wait on and I'll be reduced to interval polling or something stupid.

There's window.frames, which is an array of the windows of the frames in the document (also accessible by just indexing window directly). I wrote a further iframe to check if it reliably counts all frames in the page, even if they're not loaded yet, and it sometimes successfully printed all the frames. But not always. About half the time it didn't print anything at all, even though it should have printed an empty array if it didn't find anything. It failed much more often when it was the first iframe in the document. I conclude that window.parent.frames is not a reliable way of securing a reference to another iframe.

Access to window.parent.addEventListener for load is also forbidden.

Something that seems to work very reliably for styleFrame is only calling changeStyle() when the interwikiFrame has finished loading (by immediately attaching an event listener for "load". "DOMContentLoaded" might work even better it doesn't). But, for that to work, it still needs to already have a direct reference to it via the iframe's name property.

I don't think there's a way to reliably secure a reference to the interwikiFrame without a name. But, that is the only one that needs a name - by making the relationship between the styleFrames and interwikiFrame one-way, the styleFrames can be anonymous, making their implementation easier for the end user.

• [ ] Make the styleFrame wait for the interwikiFrame to finish loading before doing its thing.

• [ ] Remove changeStyleCheck.

ISSUE: What if there is more than one interwikiFrame on the page (e.g. one in Ayers' Info Bar)? window.parent.interwikiFrame would refer to just one of them. A reliable way is needed of getting all of them.

Minimising update effort

The two HTML files will be uploaded to a Wikidot page or pages; the JS files will stay in the repo and update themselves via jsdelivr or similar. To minimise how often the HTML files will need to be reuploaded, I should move as much logic as possible out of them. interwikiFrame.html for example should get the query parameters and nothing more, and then immediately defer to a JS script.

Move everything superfluous out of the HTML files.

Execution order

When there are two styleFrames on a page, their execution order is not consistent at all - it's pretty much random. I need a reliable way of determining the final order for CSS, without limiting the amount of CSS that can be added like the current implementation does.

Assuming I have a potentially infinite number of batches of incoming CSS, I need a way of determining what their order should be. The only idea I've got at the minute is to make each user attach a weight number to their styling, and then order the styles by their weight. But that introduces a new set of complications - CSS designers need to be aware of the weight of the styling of the themes they hope to override (or otherwise), and the problem isn't solved if two themes have the same weight.

A CSS designer should have a pretty good idea of what the order of their theme is, though, right? It's just equal to the number of themes in your extension list. We can say base Wikidot styling is -1, Sigma-9 is 0, and then any theme on top of that is 1, including BHL. If your theme extends another theme, you just add 1 to its number. BHL themes get 2, etc.

Storing styles in their own style elements

CSS added to the interwiki should be stored in their own style elements. This prevents any errors resulting from string concatenation (e.g. @import statements only working at the top of the style, or syntax errors in one bit of CSS invalidating all CSS that follows it). It also enables easy re-ordering of the styles based on the priority parameter (see section Execution order).

Knowing which community we're on

I've defined a concept of "communities": a community is a collection of branches. There are two communities, SCP and WL.

I need a reliable way of detecting which community the current page is. The current interwiki just uses the page category, but this obviously won't work for e.g. WL EN, because it's a dedicated site so there is necessarily no category info.

I think I'm just going to add a URL parameter to the interwikiFrame.html. Best to be explicit about it right? Plus this would enable easy testing without needing to configure a whole wiki to match some obscure condition.

A site that has both communities on a single wiki can use different sidebars with different interwiki modules for the two categories; or, it could modify the ListPages and maybe add an [[#ifexpr]] to provide the expected value based on the tags/category of the current page. (No, that won’t work, because ifexpr doesn’t do strings, only numbers. But there can just be two listpages with conflicting filters instead.)

New interwikiFrame URL parameters

Existing parameters that will be removed:

New parameters to be added:

New styleFrame URL parameters

The styleFrame will no longer be ambiguously responsible for setting either the base theme by name, a high-priority style, or a low-priority style based on its name attribute - all of these will now be determined by explicit URL parameters.

A tool should be created to ease conversion of CN styleFrames to new styleFrames.

Removing hardcoded CSS

As noted in section Additional iframes executing builtin code, there are some hardcoded bits of CSS in the interwiki. Instead of having these, and needing to update them every time their source changes, it might be possible to get the styling direct from the source.

Of the hardcoded CSS, the BHL Dark Sidebar is the least complex. This could be kept in a code block on the BHL Dark Sidebar, and injected via a styleFrame on that same page. Pages using the BHL Dark Sidebar should [[include]] that page, and get the correct styling as a result.

Whatever approach ends up being taken here, consistent approaches will also need to be taken for the remaining hardcoded styles in css/. style-bhl-dark.css is the least complex of those.

Here's the plan for most (all?) themes: store the interwiki CSS for that theme on the page. Then, create a styleFrame to style the interwiki, just like a normal theme. This would apply even to base themes, like BHL Dark Sidebar, BHL, and Sigma-9. We will have to make sure that extensibility and CSS prioritisation handling are robust because this way they'll always be in use, without making any hardcoded assumptions about what a base theme is. (See section Execution order)

The base theme of a site (typically Sigma-9 for SCP, and Dustjacket for WL) would be added in a styleFrame on the same page as the interwikiFrame.

For BHL specifically, this means that its CSS would sit on top of Sigma-9 CSS, which differs from the standard set by the CN interwiki which allowed themes to choose their own base theme. Instead, BHL will need to normalise the Sigma-9 interwiki styling and then apply its own styles, just like it does for the rest of Sigma-9.

Translations with varying fullnames

I spoke to Grom about maintaining a database of pages whose fullname varies between translations, who agreed that this was a necessary feature and started work on it: Unix Name Variations - O5 Command

This solidifies my decision to only count translated pages as those with exact fullname matches (barring the circumstances detailed above; see section Finding matches for fullnames near the truncation limit)

Non-encoded CSS

Separate to the interwiki, a Wikidot component could be created that uses [[html]] to construct a styleFrame from Javascript (alternatively, it could call the endpoint in interwikiFrame directly, bypassing styleFrame altogether). This could be used with a plaintext CSS parameter to enable styling the interwiki without needing to encode CSS, making things much easier during CSS development. However, being [[html]], it could be a lot slower than instantiating an iframe directly, so mileage may vary.

Communicating between iframes

The order that iframes are executed in is effectively random: both because the order cannot be known before runtime for a given combination of pages on a given wiki, and because both load and execution time is so ridiculously fast on modern browsers. As a result, I don’t believe there’s a reliable way for inter-frame communication to work one-way (i.e. styleFrames push to interwikiFrame, OR interwikiFrame pulls from styleFrames), because in either case there’s no way to reliably guarantee that the target(s) actually exist. I think it needs to work both ways - the interwikiFrame pulls styles from any styleFrames that finished loading before it did, and any styleFrames that load thereafter push their styles to the interwikiFrame. This is exactly how the CN interwiki works (see section changeStyleCheck()).

(Having interwikiFrame pull styles is also essential if I want to be able to support click-to-refresh, as that would otherwise destroy any styling - assuming that it works by reloading the window, anyway.)

The CN interwiki’s interwikiFrame relies on there being no more than three styleFrames, though, and the styleFrames rely on there being only a single interwikiFrame. Neither of these assumptions are always true for the new interwiki.

I think what I’m going to do is:

1. Once interwikiFrame has finished, have it check for other iframes by iterating parent (or if you want to be verbose about it, window.parent.frames) and checking for styleFrames. I’m unsure how I’d reliably check that the iterated iframe is a styleFrame - I guess just check its URL, or have styleFrame.html expose a sentinel value or something and read it. Once I’ve secured a reference to a styleFrame, I pull the styling.

2. Once styleFrame has finished, have it check for other iframes by iterating parent and checking for interwikiFrames. For any that it finds, have it push its style to it.

These two methods together should ensure that no style is ever missed. Styles might be duplicated, but that shouldn’t be an issue so long as I’m careful about silently rejecting duplicates - which is something that the CN interwiki did, too (see section Tracking added CSS).