Missing Wiki RLEs

Has something gone haywire? Let us know about it!
Post Reply
Book
Posts: 385
Joined: August 28th, 2021, 2:38 pm
Location: California
Contact:

Missing Wiki RLEs

Post by Book » September 6th, 2021, 5:09 pm

Gathering patterns from the wiki for my retirement hobby (developing my own life viewer--GOLHobby.com), I have come across a large number of missing RLE files. That is, the info box shows an RLE link but you get a "file not found." Some examples: sidewinder, anura, soba (and many more). I have gotten the missing RLEs I wanted from the forums or Catagolue and I can upload them to the wiki, but before I do anything, as a new user I want to know: 1) should I? 2) what's the right way to do it? Thanks.
Phil Bookman

User avatar
dvgrn
Moderator
Posts: 10610
Joined: May 17th, 2009, 11:00 pm
Location: Madison, WI
Contact:

Re: Missing Wiki RLEs

Post by dvgrn » September 7th, 2021, 3:37 am

Book wrote:
September 6th, 2021, 5:09 pm
I have gotten the missing RLEs I wanted from the forums or Catagolue and I can upload them to the wiki, but before I do anything, as a new user I want to know: 1) should I? 2) what's the right way to do it?
Thanks for checking in on this! The short answer is that you've probably already done the only thing that you can do at this point, which is bring up the question here -- and luckily that's the only thing you really need to do.

Now for the longer answer, which is a bit complicated.

Complication Number One
The RLE for the three spaceships you mentioned is in fact on the LifeWiki already, but it's not in the Patterns folder yet, it's only in what's called the "RLE namespace":

https://conwaylife.com/wiki/RLE:Sidewinder
https://conwaylife.com/wiki/RLE:Anura
https://conwaylife.com/wiki/RLE:Soba

However, as you noticed, the actual links in the article don't work, and these patterns are not included in the "all.zip" that's repackaged every night by the LifeWiki server and made available on the main page.

This is a problem, but it's ordinarily just a short-term problem. I wrote a script a while back that scrapes new content from the RLE namespace, and automatically converts it into actual RLE files, with the right comments and URLs and so on. It's nice having everything in a standard format, so that's the way most pattern files are usually created for upload -- RLE files and also .cells files, though .cells creation currently happens in a separate follow-up cycle after the RLE files have been uploaded to the server.

However, if an RLE namespace file changes, the auto-upload script deliberately does not notice this or attempt to re-create and re-upload new RLE or .cells files. Normally changes like this don't happen often, so they're not much of a problem. When they do happen, it really requires some kind of human judgment call on whether the change is really one that needs to be propagated into all.zip -- so I don't want a script doing that kind of thing automatically anyway.

Complication Number Two
Only a few people are currently set up to be able to upload pattern files to the all.zip pattern collection. Nathaniel didn't set this up to be part of the regular wiki, on the grounds that it was a fairly serious security hole if J. Random LifeWikiUser could upload arbitrary files to the LifeWiki server -- and also it was possible for enthusiastic newbies with a lot of energy to upload a lot of pattern files that we might not really want included in all.zip, maybe overwriting versions of pattern files that we did want.

Thus far, we've kept the same rather conservative approach to the server-side pattern collection that was set up over a decade ago... but we've worked around it with the RLE namespace, which can be edited by anyone with the trusted flag.

The LifeViewer animation/display of a pattern will start working as soon as a new pattern has been added to the RLE namespace, and the RLE can be copied out directly from the LifeWiki article by clicking to launch the pattern in the infobox at the upper right, and then hitting Ctrl+C or the Select, All, and CPY buttons -- maybe resetting the pattern first (R or the |< button at the bottom) if there's an AUTOSTART animation set up for it.

Complication Number Three
Early last year, Golly moved from supporting Python 2 to supporting Python 3 instead. The LifeWiki auto-upload script is written in Python 2, and so it handles certain Unicode-related things like special characters in a significantly different way from the way Python 3 does. I've now made the jump to Golly 4.0 and Python 3, so I no longer have anything installed that can run the Python 2 script.

I have the auto-upload script almost completely converted to Python 3, but a few mysterious errors are still cropping up late in the run, with just a few apparently random articles, which is preventing the RLE file creation from happening properly. It's been a good long time, but I just haven't gotten around to troubleshooting these strange and annoying encoding problems.

In Conclusion
Long story short: I'm starting an auto-upload script run right now, to remind myself of what exactly is left that needs fixing. If the run is successful, I'll check in the updated code to the GitHub repository, and make sure all the relevant files are uploaded, and we'll be back a little closer to being up-to-date again. Further bulletins as events warrant.

User avatar
dvgrn
Moderator
Posts: 10610
Joined: May 17th, 2009, 11:00 pm
Location: Madison, WI
Contact:

Re: Missing Wiki RLEs

Post by dvgrn » September 7th, 2021, 9:50 am

The new Version 2.0 of lifewiki-rlescraper.py auto-upload script is now working in Python 3, and is checked in to GitHub. It hadn't been run successfully in over a year, so 125 new RLE files and 31 .cells files have been added. About 120 of the new RLEs will get .cells files generated in the next update.

Before the Python 3 hiccup came along I had been running the script roughly four to six times a year. I'll probably try to get back to a schedule like that, so that the LifeWiki pattern collection never gets too far out of date.

The script also checks for several other kinds of non-standard formatting, like capitalization in pnames and headerless RLE in the RLE namespace. I think I've fixed everything that the script reported, so we should be in good shape for now. The only synthesis-cost mismatches were the following, which have all been corrected.

apgcodes where LifeWiki synth is worse than Catagolue:

Code: Select all

'20P4', '20p4', 'xp4_2e0771zw8ee074', 14 worse than 10
'44P7.2', '44p7.2', 'xp7_g88bdggz01130147obbgzy54t1u246zy71', 124 worse than 122
'Beehive_test_tube_baby', 'beehivetesttubebaby', 'xp2_03544ozcid11', 9 worse than 8
'Blinkers_bit_pole', 'blinkersbitpole', 'xp2_k3g4gljz11', 36 worse than 35
'Blocked_p4-2', 'blockedp4-2', 'xp4_g88rb88gz11w1vtz33xvv3z6a8caba4zx33', 52 worse than 41
'Blocked_p4-3', 'blockedp4-3', 'xp4_oo0u1680sssss0861u0oozx346x4a4x643', 56 worse than 45
'Boat_test_tube_baby', 'boattesttubebaby', 'xp2_03544ozca511', 9 worse than 8
'Caterer_on_44P7.2', 'catereron44p7.2', 'xp21_g88bdggz01130147obbgzcg2228x4t1u246zx1fy21', 133 worse than 131
'Circle_of_fire', 'circleoffire', 'xp2_g48e0v0e84gz295t1u1t592zy11', 288 worse than 270
'Claw_test_tube_baby', 'clawtesttubebaby', 'xp2_3542sws2453zy111', 10 worse than 8
'Eater_3', 'eater3', 'xs31_6952gg2u08k8z064k5421zw121', 24 worse than 23
'Eater_plug', 'eaterplug', 'xp2_wo44871z6221', 8 worse than 7
'Ellison_p4_HW_emulator', 'ellisonp4hwemulator', 'xp4_g88bb88gz11wt118zbdwvx6zo8wb88hz011dd11', 51 worse than 50
'Hat_siamese_vase', 'hatsiamesevase', 'xs27_651u0u156zw17871', 9 worse than 6
'Hook_test_tube_baby', 'hooktesttubebaby', 'xp2_321egge123', 11 worse than 8
'Inflected_clips', 'inflected30greatsym', 'xs32_4a9b8b96z259d1d96', 10 worse than 8
'Negentropy', 'negentropy', 'xp2_gg0gba9jz1107ac87066zx3213', 91 worse than 90
'Sidewinder', 'sidewinder', 'xq20_27de6zw8smx2ceaz0ggz13673', 11 worse than 10
'Stillater', 'stillater', 'xp3_0ca1d2sgz321343', 82 worse than 74
'Xs15_3lkia4z32', '13loop', 'xs15_3lkia4z32', 6 worse than 5

Book
Posts: 385
Joined: August 28th, 2021, 2:38 pm
Location: California
Contact:

Re: Missing Wiki RLEs

Post by Book » September 7th, 2021, 1:59 pm

I'm glad I raised the issue! Bravo!
Phil Bookman

Post Reply