safeopenclip.lua usability issues

[rowett]

Possible actions that a user could choose would be

1) replace Golly's stored rule with the rule from the pattern comments, and open the pattern;
2) ignore the rule in the comments and use Golly's stored rule, and open the pattern;
3a) cancel opening the pattern;
3b) pop up an HTML window showing the two conflicting rule texts in two columns next to each other, and cancel opening the pattern.

I was thinking more about the following use case (when the rule in the pattern comments is different than Golly's stored rule):
4) ignore Golly's stored rule and use the rule from the pattern comments and open the pattern.

In this case on first opening it would need to do something like write a renamed version of rule from the pattern comments to the rule folder, modify the pattern header and @RULE name in the pattern comments accordingly, and then open the pattern.

It's not very tidy because we've modified the rule name.

[confocaloid]

edit (2024-04-16): see a newer post: viewtopic.php?p=182769#p182769

What follows is one of my wishes. It is related to the discussion of appended rule definitions. However, it is beyond what could be done by changes limited to safeopenclip.lua. Feel free to skip this post.

I wonder if it would be possible to add a new algorithm to Golly, to support "RuleLoader rules defined by rulestrings".
That means it would be possible to define the behaviour of any RuleLoader-supported CA by writing a single rulestring, instead of a separate rulefile.
It may be unnecessary/undesirable to show the actual rulestring in the "Set Rule" dialog (the rulestrings are going to be long, and are not going to be human-readable).
Likewise, instead of putting the rulestring in the title of the window, probably there would be something like "untitled [unnamed / RuleLoader]" in the title.

An important part is that the rulestring would be loaded directly into an in-memory representation of the CA rule, without needing a rulefile.
Then it would be possible to generate such rulestrings dynamically by a Golly script.

For example, safeopenclip.lua could be modified so that it reads the appended @RULE data from the snippet in the clipboard, converts that @RULE data to a long rulestring defining the same cellular automaton, and invokes

Code: Select all

g.setrule("EXTREMELY_LONG_RULESTRING_DEFINING_A_RULELOADER_SUPPORTED_CA")

effectively ignoring Golly's stored rulefile if any exists, and instead using the appended @RULE data whenever one is supplied.

That would also "solve" the problem of defining new rulespaces. To add a new rulespace/notation, it would suffice to write a script that takes the input rulestring and validates it, converts the rulestring into an EXTREMELY_LONG_RULESTRING_DEFINING_A_RULELOADER_SUPPORTED_CA, and does g.setrule("EXTREMELY_LONG_RULESTRING_DEFINING_A_RULELOADER_SUPPORTED_CA").

As an example (to give a very rough idea of the length of such rulestrings), the following is the result of compressing Rule:SAKASMILEY with "zip -9" and encoding the resulting 736-byte archive into base64. This could be shortened by discarding parts of the .rule file that do not define actual behaviour of the CA, and otherwise improving the uncompressed representation of the rule definition. Probably it makes sense to choose a simpler encoding (e.g. avoid zip compression) to simplify writing scripts that generate and/or parse this, at the cost of longer rulestrings.

Code: Select all

UEsDBBQAAgAIAPO7h1ghcvuPLAIAACoJAAAPABwAU0FLQVNNSUxFWS5ydWxlVVQJAAOK5hJmiuYS
ZnV4CwABBOoDAAAE6gMAAI1Wy27bMBC88ysI5CrEXJKSKANG4wI+FHV7SJpDT4VrK7GAShQsOUb+
viQl0o84S4MyxOXOzo5GD/rh8Xm5oE/z7/OnH9+Wi9+EbPu+7aaTyVo3h9X7v+qlvF/revKid/u6
m7xV5aHXbbW+b7ftl3aWslzxu9aeFCF3T/2qLynQqqP9tqSb3epQNa+0s8sJ7TTd6CGgnDz8elws
CGn29R+31M1Auqgpq9ftX73rZmOsNyaZAgHKDDdQ4c6Z+alhxS6CGPPc5CVNTT43+YICcwBuEXxk
uDqu1LMP9QKp55/nHLuKqsvQ+iJar9B6iNsDmD/CFNtSP3JbDj4M9rFP+C29QOnhlP1iEHDeR+Rn
OH+O8ss4f4Hbw1D+NMrPOc4vEH4k57rbHrH2uH0ctw/i94fj/gnMP+lYjIeKctNHUp5+OBQVJpVe
vMnsqIUdX2T8URQSlSLSa+2vHETeCBxkZ1ELhcJ1F6huyW7Xkke1SPxxlQLXIm/X4j7xETH4wytz
XIy6VUxKhRGSUZGPhzRfjvQkLMKcmE1u/nW5IM24xU3tFjdub1utN9M33fws9/WqaUj3Xtdlv6sM
qtFN6TdTTnjCxgHEz3giPECERZFIIjw0YR4gA0Am5lIDWwCYazpr4aPMAzKSWcKxyiTGee4BOckv
RA6x8gBF1CBqZDApNy88oCDFiTCXGtiCSjMDdtYlhAABZPY5ONPqEQnwAOIErplqFiHYav9PHM0E
8h9QSwECHgMUAAIACADzu4dYIXL7jywCAAAqCQAADwAYAAAAAAABAAAAtIEAAAAAU0FLQVNNSUxF
WS5ydWxlVVQFAAOK5hJmdXgLAAEE6gMAAATqAwAAUEsFBgAAAAABAAEAVQAAAHUCAAAAAA==

[Andrew]

I was thinking more about the following use case (when the rule in the pattern comments is different than Golly's stored rule):
4) ignore Golly's stored rule and use the rule from the pattern comments and open the pattern.

To handle this case the script could do steps like this:

* Call saverules = g.getdir("rules") to save the user's current rules directory.
* Call g.setdir("rules", g.getdir("temp")) to switch to a temporary rules directory.
* Write the .rule file in the temporary directory.
* Open the pattern.
* Call g.setdir("rules", saverules) to restore the user's rules directory.
* Warn the user that the pattern contained conflicting rule data and a temporary .rule file had to be created.

Things will break later if the user does some undo action that requires Golly to restore the rule but better than doing nothing.

If all that proves to be too messy then my preference would be for the script to simply exit with a suitable warning dialog saying that the pattern's rule data conflicts with an existing .rule file (and display that file's full path). I suspect that should be quite rare.

[dvgrn]

4) ignore Golly's stored rule and use the rule from the pattern comments and open the pattern.

In this case on first opening it would need to do something like write a renamed version of rule from the pattern comments to the rule folder, modify the pattern header and @RULE name in the pattern comments accordingly, and then open the pattern.

It's not very tidy because we've modified the rule name.

Yep, there's definitely some untidiness there. If we then edit and save the pattern and try to share it, unless we do even more extra processing, the modified rule name is likely to stick and travel with the new pattern. Can't see anything very good coming out of that.

I'm kind of thinking that option #1 is the "tidy" way to handle that situation -- more specifically,

1) After getting an explicit okay from the user, replace Golly's stored rule with the rule from the pattern comments, and open the pattern.

Option #1 definitely doesn't handle all possible problematic situations, like where two people have come up with different (and maybe unrelated) definitions of some named rule -- let's call it "GenericRule".

If we implemented the "temporarily rename, the modify pattern header" idea, then it would be vaguely possible to use @RULE pattern postfixes to share and run Person 1's GenericRule patterns, and also Person 2's GenericRule patterns, even though the rule is actually different.

Let's not do anything to encourage ambiguity
My thought is that it really wouldn't work out well to implement anything like option #4. All things considered, it's going to work better in the long run to never do anything to encourage different people to mean different things by the same rule name, when patterns are being shared on the forums.

Given the likely size of the rule-file-making community for the foreseeable future, it's not actually a hardship to invent unique rulenames. For example, if people append their own chosen ID string (like initials or a date) after the name, that will prevent pretty much all duplication in practice.

We need to be encouraging people to always do the necessary small amount of work of naming new rules uniquely as they go along, so that multi-GenericRule type confusion happens as rarely as is humanly possible. We can't stop people from doing things like that if they want to, but we can make Life much easier for people who don't do things like that.

So far we've already done exactly that! If someone tries to post patterns using a newly invented rule that's already used for something else, where a competing rule has already been uploaded to the LifeWiki Rule namespace, then LifeViewer will simulate those patterns incorrectly. Giving the new rule a new unique name just works so much better, that that's what everybody does naturally -- and the combination of LifeViewer and LifeWiki is so handy for sharing patterns that those definitions for uniquely-named rules tend to end up in the Rule namespace sooner rather than later.

Self-contained patterns vs. patterns with dependencies
It's tempting to follow to its logical conclusion the beautiful idea of patterns always being completely self-contained, with no dangerous dependencies.

Take Entity Valkyrie's StateInvestigator.zip pattern collection, for example. There's currently a ~40K file representing the StateInvestigator rule on LifeWiki -- we won't need it for Golly 4.3, but we'll need it for anyone who doesn't choose to upgrade. Looks like we could cut that to 17K by leaving out the helpful documentation at the top... but then any given copy wouldn't have the helpful documentation at the top.

120-ish patterns with 40K added data each isn't a big deal, anyway, these days -- in ZIP file format it wouldn't be much bigger at all. It would add something like 5MB to the uncompressed file size... but Golly can open ZIP files directly, so there wouldn't necessarily be a permanent larger footprint.

If patterns posted on the forums started including an extra 10-40K per small pattern, I do think we'd start seeing noticeably longer loading times for the average forum page, though.

So at least some rule table definitions are still big enough that our current way of handling them makes quite a bit of sense.

Table vs. tree
For smaller rule tables, and/or maybe rules with no extensive comments that need to travel along with them, confocaloid's idea is an interesting one. We already have MAP rules and other extraordinarily long rule strings. It wouldn't be impossible to define a single-line format for rule tables that would allow patterns to be completely self-contained.

One of the practical difficulties is that rule trees are much more efficient to simulate than rule tables -- so we usually start with a rule table and then generate a @TREE section for LifeViewer and Golly to use instead. Maybe that should be part of the idea: just encode a one-line @TREE section for the rule header, and leave everything else in pattern comments? We aren't reliably going to be able to generate @TREE from @TABLE on the fly, unless we put some serious limitations on how complex a rule can be for this kind of store-it-all-in-the-header-line trick.

Another question: for something like WireWorld, would that rule header still include the name "WireWorld", or would it be a functional description of the rule only? It seems really useful to put the name right up front if there is one, so that the pattern is still human-recognizable as a WireWorld pattern.

That brings back the name-ambiguity problem, though: how does a user know if this header-specified rule that's called "WireWorld" is the same as everybody else's header-specified rule that is called "WireWorld"?

[confocaloid]

If I understand right, it is currently impractical to reimplement support for self-contained patterns (with @RULE data appended to the pattern data, equivalent to the existing support in LifeViewer) in Golly using just a script.

It makes sense to aim for equivalent functionality in Golly and LifeViewer, whenever some functionality is supported in both.

Hence I think it is better to admit that Golly does not currently support @RULE data appended to the pattern data, and avoid any processing of the appended @RULE data (if any). The Golly user will still be able to install the rulefile manually from the pattern comments, in those cases when they want to do so.

(a) That simplifies the behaviour/observable logic of the script (less potential for Golly users getting confused / losing their data and/or time).
(b) That simplifies the code (less potential for bugs).
(c) That avoids the situation of having two different behaviours on the same input, when LifeViewer does one thing, and Golly [sometimes?] does a [subtly?] different thing.

Self-contained patterns with @RULE appended to the pattern data would certainly be useful, if they could be made to work the same way as in LifeViewer (use the appended @RULE whenever one is supplied, and avoid creating or reading any .rule files in that case). The rulename can be ignored (other than checking that it is the same in both headers); it is used for the sole purpose of linking together the pattern header and the appended @RULE.

It is convenient to be able to assign temporary single-use names without having to pause and invent an unique name. Always requiring unique names will lead to things that are worse than Rule:LiterallyAnEntireRuleForOneStupidSpaceship. (Which I guess happened only because someone could not either assign a simple temporary name like "temp", or avoid names entirely.)

[rowett]

A cleaner solution might be to make pattern comment rules transient (i.e. not saved). All of the valid comments in this thread above are handling the situation where we're trying to persist the rule and deal with name clashes and so on.

When opening a pattern if it contains an @RULE section in the pattern comments then:

1. Use the rule definition from the pattern comments (even if there is local rule with the same name).
2. Do not add this rule to the local rule folder.

If you modify and save the pattern then the rule is maintained since it's still in the comments.
The rule name is now irrelevant.

This would require some changes to Golly.

Self-contained patterns vs. patterns with dependencies
It's tempting to follow to its logical conclusion the beautiful idea of patterns always being completely self-contained, with no dangerous dependencies.

I think it's nice to have the ability to export a pattern in a self-contained format. This is exactly what showinviewer.lua does. I think the benefits of external rule files outweigh this beautiful idea.

Table vs. tree
One of the practical difficulties is that rule trees are much more efficient to simulate than rule tables -- so we usually start with a rule table and then generate a @TREE section for LifeViewer and Golly to use instead.

On a side note this is much less of a requirement now. For many types of @TABLE rules LifeViewer now creates a fast internal representation of the rule on pattern load.

Another question: for something like WireWorld, would that rule header still include the name "WireWorld", or would it be a functional description of the rule only? It seems really useful to put the name right up front if there is one, so that the pattern is still human-recognizable as a WireWorld pattern.

For transient rules it doesn't matter. It's worth calling them something human-recognizable.

[dvgrn]

If you modify and save the pattern then the rule is maintained since it's still in the comments.

Aha, that seems like a key point. Modifying then saving a pattern seems like it will be an important use case. Persisting the rule in the comments seems to handle that.

Copying a piece out of a pattern and opening it in a new layer seems like another important use case... but there I think we might run into a problem. Copying a selection from a pattern currently doesn't copy the comments, and I'm thinking it probably shouldn't -- it's way too easy to end up propagating irrelevant information into new mostly unrelated patterns. But if the @RULE section at least doesn't copy over, then we can end up with patterns in different Golly layers that are using the same base rule name, but they're actually running different rules.

Maybe one of them will be displayed as "GenericRule_TRANSIENT" and the other one (the copied selection) will just show up as "GenericRule".

That still seems quite painful and awkward, in practice. In 99.9% of cases, "GenericRule_TRANSIENT" will in fact be identical to "GenericRule". The GUI will be drawing a distinction when in fact there is no difference -- and this is a case where it would be good for the user to know that there is no difference, and really really good for the user to be made aware that there is a difference, in the 0.1% of cases where a local rule actually doesn't match a transient rule.

See the "Can we give users a useful new tool?" section below for more about this.

If I understand right, it is currently impractical to reimplement support for self-contained patterns (with @RULE data appended to the pattern data, equivalent to the existing support in LifeViewer) in Golly using just a script.

It makes sense to aim for equivalent functionality in Golly and LifeViewer, whenever some functionality is supported in both.

This is also a key point, I think!

My question is, is it even theoretically possible to ever end up with equivalent functionality in Golly and LifeViewer, as far as rules are concerned?

Golly has a local rule store, so it works when there is no Internet connection. LifeViewer has no local rule store, so it depends on an external central repository for anything that isn't natively supported.

We could support equivalent functionality by taking the local rule store out of Golly. But that's a significant loss of functionality. It seems unlikely that we're going to do that... We could get close to equivalent functionality by enabling automatic LifeWiki rule lookups in Golly, whenever the Internet is in fact available -- but we'd still end up having a lot of the same tricky questions to deal with, related to the local rule store.

Hence I think it is better to admit that Golly does not currently support @RULE data appended to the pattern data, and avoid any processing of the appended @RULE data (if any). The Golly user will still be able to install the rulefile manually from the pattern comments, in those cases when they want to do so.

This is a good point about installing manually. The rule in the pattern comments is currently in a format where it can be selected and copied out of the comments, and Ctrl+Shift+O will immediately install that rule. We've been reasonably happy with that functionality so far.

However, that's definitely functionality that works fine for experts, but new Golly users tend to not know about it, and they really don't have any likely path to find out. First there's the hurdle of figuring out about opening the pattern comments in the first place, and maybe scrolling way down to find the otherwise invisible @RULE section, and knowing what it means. Then the fact that File > Open Clipboard easily installs new rules as well as opening patterns... is not at all obvious from the current GUI. To fix that we'd have to change the menu to say something like "Open Clipboard (Pattern or Rule)".

Possible side effects of transient rules
I think that the local rule store in Golly is a useful feature, so we need to keep supporting that.

I'm a bit worried that having rules start traveling with patterns more often will produce one of two unfortunate side effects:

1) if transient rules are implemented in Golly with no attempt to reconcile the transient rules with stored local rules, then that will enable the proliferation of extremely confusing rule variants, where it very quickly becomes almost impossible to figure out what is (if anything) is intended to be different between patterns that are getting shared with the same rule name.

This happened most recently with the "B-Univ" rule: b-engine made various changes to the rule behind the scenes that broke some existing published patterns but fixed others. "B-Univ" development also brought up the tricky question of how to distinguish rules intended to be throwaway or experimental from rules that are intended to have permanent names associated with published patterns.

There were also some really tricky maintenance nightmares back in the days of "shapeloop rule" development. (The renaming to "shapeloop3" in the linked post was badly needed, for example, and involved a lot of confusion and mysteriously broken older posted patterns before it all got sorted out. There might still be some threads back there with patterns that no longer work as intended.)

These cases are worth thinking very slowly and carefully about, because they really did happen, and they weren't easy to fix. I agree that having the rule travel with each pattern might theoretically solve a lot of these cases. But it still seems possible that an implementation of throwaway transient rules in Golly might make similar confusions more likely to happen in the future (because non-uniquely-named rules will be able to proliferate a lot more easily) ... even though the intention is to make confusion less likely.

2) If we provide an option to update Golly's stored local rules using rules from pattern comments, then we're going to start being presented with choices that aren't really choices -- we'll end up getting asked something equivalent to "Do you want to replace the existing rule with the version in the [clipboard|pattern comments]?" in the 99.9% of cases where the existing rule is identical to the "new" version.

Can we give users a useful new tool?
From the point of view of usability, it seems really important to me to reserve that "Do you want to replace..." question, for cases where the replacement actually makes a difference. Maybe we can even give users a better idea of what the difference is.

We can easily check if the rule we're proposing to replace is character-for-character identical with the new rule that we're considering replacing it with. A lot of times it will be.

In that case, there's no point in asking about replacing anything -- it works fine to just let the user know that the rule in question is already installed (probably via the status bar, without a pop-up needed). That will get rid of a lot of annoying pop-ups while giving really useful information to the user -- an ongoing confirmation that there's no possibility of rule confusion.

When there _is_ a difference between rules, the user might want the new version, or might want to keep the old version -- but they might not know which. Most commonly they won't want to lose information. They might want whichever version has more up-to-date comments, or maybe a @COLORS or @ICONS or @TREE section that the other version of the rule is missing.

In that case, a side-by-side comparison of the two rules would be extremely handy. That's something that Golly doesn't currently provide. But it might be possible to provide it, even with our current Lua-script approach -- e.g., via a three-way choice:

"Replace? Yes / No / Display comparison".

[confocaloid]

edit (2024-04-16): see a newer post: viewtopic.php?p=182769#p182769
---

[...] We could support equivalent functionality by taking the local rule store out of Golly. [...]

I do not understand why having the local rule store might possibly prevent supporting self-contained patterns where @RULE definition is appended to the pattern data.

• When there is a @RULE definition appended to the pattern data (loaded from a file or opened from clipboard via Ctrl+O), interpret it as a complete definition of an unnamed (transient) RuleLoader-supported rule. This rule is not stored in any separate file, and is loaded directly into memory. There are no lookups into local rule store, or lookups into built-in rules in this case. Appending a @RULE definition to the pattern data can be used to override any existing rulename, without storing the new transient @RULE definition in a separate file.
• When there is no @RULE definition appended to the pattern data (loaded from a file or opened from clipboard via Ctrl+O), perform the usual lookup sequence (which is already implemented).
• When copying selection (part of a pattern) in an unnamed transient rule, copy the transient rule definition into the clipboard as well. When later pasting the copied part into a new layer, the rule definition from the clipboard is used to restore a transient RuleLoader-supported rule.
• When saving a pattern in a transient rule to a file, append the transient rule definition to the end of the pattern file.

Maybe one of them will be displayed as "GenericRule_TRANSIENT" and the other one (the copied selection) will just show up as "GenericRule".

Both of them could be displayed for example as "unnamed rule" in the title.
Alternatively, both could be displayed as "RuleName, transient" where "RuleName" is the name specified after @RULE (and in the RLE header).

When copying, the information from the appended @RULE definition would also be copied into the clipboard, along with the selected region of the pattern.
When pasting using "Open Clipboard" (into a new empty layer), the copied selection would be pasted, the algorithm would perhaps be switched to something like "TransientRuleLoader", and the appended @RULE definition from the clipboard would be used. The displayed title would become identical to the displayed title of the source layer from which the pattern was copied, as long as the displayed title is determined by the @RULE definition.

I think that the local rule store in Golly is a useful feature, so we need to keep supporting that.

I also think the ability to have a @RULE definitions in a separate reusable file is a useful feature.
I think having transient rules would be even more useful. Being able to use a transient rule does not prevent being able to use the local rule store.

f transient rules are implemented in Golly with no attempt to reconcile the transient rules with stored local rules, then that will enable the proliferation of extremely confusing rule variants, where it very quickly becomes almost impossible to figure out what is (if anything) is intended to be different between patterns that are getting shared with the same rule name.

I think for transient rules, the rule name is not important. It could be autogenerated for the sole purpose of linking the RLE header to the appended @RULE header. Any extended information about the transient rule could go to pattern comments.

If we provide an option to update Golly's stored local rules using rules from pattern comments, then we're going to start being presented with choices that aren't really choices -- we'll end up getting asked something equivalent to "Do you want to replace the existing rule with the version in the [clipboard|pattern comments]?" in the 99.9% of cases where the existing rule is identical to the "new" version.

I think safeopenclip.lua should not be concerned with appended @RULE definitions. I believe leaving them in the pattern comments (where they can be copied from and installed manually) is better than trying to guess and creating a competing implementation that is different from what LifeViewer does.

Regarding the pattern comments functionality being relatively inaccessible to new Golly users -- I think that is a good reason to make the pattern comments feature easier to find. Explain them in Golly Help. That by itself might solve most of issues, without trying to process those comments automatically / without trying to do something "smart".