catagolue is down

A forum where anything goes. Introduce yourselves to other members of the forums, discuss how your name evolves when written out in the Game of Life, or just tell us how you found it. This is the forum for "non-academic" content.
User avatar
cordership3
Posts: 129
Joined: August 23rd, 2016, 8:53 am
Location: Smome tomato
Contact:

Re: catagolue is down

Post by cordership3 » May 25th, 2018, 12:41 pm

Catagolue is down again, and I don't know why this time.
evil twin of cordership2

User avatar
calcyman
Moderator
Posts: 2932
Joined: June 1st, 2009, 4:32 pm

Re: catagolue is down

Post by calcyman » May 25th, 2018, 1:41 pm

cordership3 wrote:Catagolue is down again, and I don't know why this time.
It looks like b3678s25678/C1 is straining the server; there are over 500000 distinct objects in there and Anonymous has been uploading hauls every 2 minutes to that census.

EDIT: and is still doing so. Anonymous, please stop.

EDIT 2: I've modified Catagolue to ban this rule for the time being.
What do you do with ill crystallographers? Take them to the mono-clinic!

User avatar
77topaz
Posts: 1496
Joined: January 12th, 2018, 9:19 pm

Re: catagolue is down

Post by 77topaz » May 26th, 2018, 2:18 am

Catagolue appears to be down again, but now it's giving a 503 error instead of a 500 error like it was before.

wwei23

Re: catagolue is down

Post by wwei23 » May 26th, 2018, 9:23 am

Why didn't B3/S23/1x256X2+1 bring down the site in a few hours, despite me submitting hauls like twice as fast? ?? ??? ???? ????? ?????? ??????? ???????? ????????? ??????????

User avatar
Apple Bottom
Posts: 1034
Joined: July 27th, 2015, 2:06 pm
Contact:

Re: catagolue is down

Post by Apple Bottom » May 26th, 2018, 1:10 pm

wwei23 wrote:Why didn't B3/S23/1x256X2+1 bring down the site in a few hours, despite me submitting hauls like twice as fast? ?? ??? ???? ????? ?????? ??????? ???????? ????????? ??????????
As has been explained before, it's really about the number of operations Catagolue needs to perform to process a haul.

Suppose that a rulesym has a very rich repertoire of objects, and is very fast to search, and a user is submitting a haul with, say, 10,000 distinct objects every 2 minutes. That's (at least) 5,000 datastore operations Catagolue has to perform per minute, and since these cost money, the site's daily quota (2 USD, IIRC) will be used up relatively fast.

Suppose that a different rulesym is much less rich in objects, and a user submits hauls of, say, 100 distinct objects every minute. That's 100 datastore operations Catagolue has to perform per minute. I'm sure you can see the difference.

This isn't to say submitting hauls every minute is a good idea, even for less productive rules. IIRC, Calcyman's asked users to not submit hauls more often than once every 20 minutes from a single machine; so if you're running e.g. four searcher instances on that machine, you should adjust haul sizes that you're only submitting hauls once every 80 minutes at most from each.

Depending on the rule this'll reduce server load significantly. Suppose that for the same (second) rulesym considered, the number of distinct objects doubles if you increase the haul size by a factor of ten. Suppose now that you decide to submit a haul every 100 minutes instead of every minute. Hauls will now contain ~400 distinct objects on average, and Catagolue will have to perform ~4 datastore operations per minute and haul.

This is my best understanding of the situation anyway. Calcyman'd be able to tell you more. :)

(Oh, and you may want to reconsider how many punctuation marks you use.)
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

wwei23

Re: catagolue is down

Post by wwei23 » June 2nd, 2018, 12:09 am

How can I search B3678/S25678 without bringing down Catagolue? I don’t know how to create wrapper scripts.
Maybe Catagolue should allow hauls larger than 1 megabyte.

User avatar
77topaz
Posts: 1496
Joined: January 12th, 2018, 9:19 pm

Re: catagolue is down

Post by 77topaz » June 2nd, 2018, 4:11 am

wwei23 wrote:How can I search B3678/S25678 without bringing down Catagolue? I don’t know how to create wrapper scripts.
Maybe Catagolue should allow hauls larger than 1 megabyte.
Wait, you were the one submitting all those "anonymous" B3678/S25678 hauls?

Hauls larger than 1 megabyte would just require more datastore operations, though, so it wouldn't really help except in maybe a few specific rules (and it would make it easier for malicious users to crash Catagolue with rules like B/S012345678).

wwei23

Re: catagolue is down

Post by wwei23 » June 2nd, 2018, 10:17 am

77topaz wrote:
wwei23 wrote:How can I search B3678/S25678 without bringing down Catagolue? I don’t know how to create wrapper scripts.
Maybe Catagolue should allow hauls larger than 1 megabyte.
Wait, you were the one submitting all those "anonymous" B3678/S25678 hauls?

Hauls larger than 1 megabyte would just require more datastore operations, though, so it wouldn't really help except in maybe a few specific rules (and it would make it easier for malicious users to crash Catagolue with rules like B/S012345678).
Objective frequency drops off exponentially, so larger hauls tend to have less unique objects per megabyte. As a result, it would bee better for Catagolue. Also, when I tried to search B3678/S25678, whenever I put the number of soups high enough to follow 1 haul every 20 minutes, the hauls are too big.

User avatar
Majestas32
Posts: 549
Joined: November 20th, 2017, 12:22 pm
Location: 'Merica

Re: catagolue is down

Post by Majestas32 » June 2nd, 2018, 10:36 am

Search movero8 instead then
Searching:
b2-a5k6n7cs12-i3ij4k5j8
b2-a3c7cs12-i

Currently looking for help searching these rules.

User avatar
Apple Bottom
Posts: 1034
Joined: July 27th, 2015, 2:06 pm
Contact:

Re: catagolue is down

Post by Apple Bottom » June 2nd, 2018, 2:03 pm

wwei23 wrote:Also, when I tried to search B3678/S25678, whenever I put the number of soups high enough to follow 1 haul every 20 minutes, the hauls are too big.
And that should be telling you something.

Now, accepting that Catagolue's limits are in place for a reason, I think the salient question isn't "why can't you search B3678/S25678" as much as "why do you want to use Catagolue". Or, perhaps more to the point, "why do you want to use Catagolue to store a FULL census" --- every object, including the entirely of the long tail?

Calcyman's said before that while data fed to Catagolue *can* come from apgsearch, it doesn't have to. As a corollary, it doesn't have to come from an unmodified apgsearch; you could easily rig up a version that only submits part of the result, i.e., the objects considered interesting, whatever that means in practice. (You could also have it save full logs locally, so you'd not lose any data yourself.) So long as you use a rulestring/symmetry that communicates the fact the resulting census isn't a regular, full soup search census, this should be fine, right?

Or alternatively, modify apgsearch so it only saves logs locally and doesn't submit to Catagolue, and use your own tools to hunt for interesting objects.
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

User avatar
Majestas32
Posts: 549
Joined: November 20th, 2017, 12:22 pm
Location: 'Merica

Re: catagolue is down

Post by Majestas32 » June 2nd, 2018, 2:05 pm

The "UnbanMajestas" symetry
Searching:
b2-a5k6n7cs12-i3ij4k5j8
b2-a3c7cs12-i

Currently looking for help searching these rules.

dani
Posts: 1222
Joined: October 27th, 2017, 3:43 pm

Re: catagolue is down

Post by dani » July 13th, 2018, 6:27 pm

catagolue is down again. i have my own guesses as to who caused it but i'll leave that in the air.

User avatar
77topaz
Posts: 1496
Joined: January 12th, 2018, 9:19 pm

Re: catagolue is down

Post by 77topaz » July 13th, 2018, 7:44 pm

Hmm, I really think it would be useful to have some sort of feature on the site that showed how much of the daily quota has been used/remained. That would likely help avoid these kinds of outages in the future.

User avatar
77topaz
Posts: 1496
Joined: January 12th, 2018, 9:19 pm

Re: catagolue is down

Post by 77topaz » July 14th, 2018, 3:17 am

Catagolue is back! :) I ended up submitting a 78 million-soup haul in b3s23/C1, which is exactly the same as what I got the previous time Catagolue went down. :o

Oh, and also, looking at the hauls that processed immediately after Catagolue went back up, I think Heavpoot searching b2-a4z678s234w with hauls of ~100 KB and ~2600 distinct objects once per minute definitely contributed to Catagolue going down, and he's still searching that rule as of the time of writing; I recommend raising the threshold for that rule (he was actually warned on the Discord to increase his haul sizes multiple times before this outage).

User avatar
Apple Bottom
Posts: 1034
Joined: July 27th, 2015, 2:06 pm
Contact:

Re: catagolue is down

Post by Apple Bottom » July 14th, 2018, 4:15 am

77topaz wrote:searching b2-a4z678s234w with hauls of ~100 KB and ~2600 distinct objects once per minute
...yeah, that defies common sense, not to mention the ample warning given on the wiki.

Slow down cowboy!

EDIT: Calcyman tells me he's fixed this, by increasing the minimum required for that rule.
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

dani
Posts: 1222
Joined: October 27th, 2017, 3:43 pm

Re: catagolue is down

Post by dani » October 20th, 2018, 7:57 pm

Catagolus Descendus! :cry:

also wait did they just ad the :cry: emoticon i never noticed it

User avatar
Ian07
Moderator
Posts: 891
Joined: September 22nd, 2018, 8:48 am
Location: New Jersey, US

Re: catagolue is down

Post by Ian07 » October 20th, 2018, 8:10 pm

danny wrote:also wait did they just ad the :cry: emoticon i never noticed it
Pretty sure I've seen it on Discord for a while now, but not as an animated emoticon because that's restricted to Nitro users.

User avatar
cordership3
Posts: 129
Joined: August 23rd, 2016, 8:53 am
Location: Smome tomato
Contact:

Re: catagolue is down

Post by cordership3 » December 12th, 2018, 7:10 pm

:( catagolue is over quota quota-o-meter when
evil twin of cordership2

wwei23

Re: catagolue is down

Post by wwei23 » December 12th, 2018, 7:20 pm

cordership3 wrote::( catagolue is over quota quota-o-meter when
Actually, I just refreshed the page, it seems to be back up already!

User avatar
cordership3
Posts: 129
Joined: August 23rd, 2016, 8:53 am
Location: Smome tomato
Contact:

Re: catagolue is down

Post by cordership3 » January 14th, 2019, 12:19 pm

it's down again
evil twin of cordership2

User avatar
calcyman
Moderator
Posts: 2932
Joined: June 1st, 2009, 4:32 pm

Re: catagolue is down

Post by calcyman » January 14th, 2019, 1:02 pm

cordership3 wrote:it's down again
Ouch -- someone has been submitting hauls for g6b2ce3ar4a5ns013ij, which has lots of distinct objects. Storing the sample soups was causing a peak of 10000 datastore writes per minute. That incurred a total cost of $0.72 per hour, causing it to soon max out at the daily spending limit of $5.00.

I've now made a commit to the repository to ensure sample soups are not stored for objects in tabulations exceeding 500kB (except for b3s23, which is exempt). That should limit the amount of damage someone can do by intensive searching of a single census.
What do you do with ill crystallographers? Take them to the mono-clinic!

User avatar
muzik
Posts: 5614
Joined: January 28th, 2016, 2:47 pm
Location: Scotland

Re: catagolue is down

Post by muzik » January 14th, 2019, 1:30 pm

calcyman wrote:I've now made a commit to the repository to ensure sample soups are not stored for objects in tabulations exceeding 500kB (except for b3s23, which is exempt). That should limit the amount of damage someone can do by intensive searching of a single census.
Does this mean that all existing hauls which take up more than this amount (and, by extension, all rules whose only sample soups lie within such a haul) will be deleted?

User avatar
calcyman
Moderator
Posts: 2932
Joined: June 1st, 2009, 4:32 pm

Re: catagolue is down

Post by calcyman » January 14th, 2019, 1:39 pm

muzik wrote:
calcyman wrote:I've now made a commit to the repository to ensure sample soups are not stored for objects in tabulations exceeding 500kB (except for b3s23, which is exempt). That should limit the amount of damage someone can do by intensive searching of a single census.
Does this mean that all existing hauls which take up more than this amount (and, by extension, all rules whose only sample soups lie within such a haul) will be deleted?
No, it merely means that when (for instance) b3678s34678/C1 reached 500 kB of objects in the xp2 tabulation, it will stop including sample soups for that census on individual xp2_??? object pages. The tabulations themselves (including frequencies of each object) are unaffected by this.

The justification is that if a tabulation reaches that size, the individual objects aren't interesting enough to bother recording their sample soups.
What do you do with ill crystallographers? Take them to the mono-clinic!

wwei23

Re: catagolue is down

Post by wwei23 » January 14th, 2019, 5:08 pm

What if I'm searching for a certain object, and the tabulation is over 500KB? How will I post the soup in the soup search results in rules other than Conway's Life thread?

User avatar
Apple Bottom
Posts: 1034
Joined: July 27th, 2015, 2:06 pm
Contact:

Re: catagolue is down

Post by Apple Bottom » January 14th, 2019, 6:08 pm

wwei23 wrote:What if I'm searching for a certain object, and the tabulation is over 500KB? How will I post the soup in the soup search results in rules other than Conway's Life thread?
Look at the locally-saved log, in particular the @SAMPLE_SOUPIDS section.
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

Post Reply