catagolue is down
- cordership3
- Posts: 129
- Joined: August 23rd, 2016, 8:53 am
- Location: Smome tomato
- Contact:
Re: catagolue is down
Catagolue is down again, and I don't know why this time.
evil twin of cordership2
Re: catagolue is down
It looks like b3678s25678/C1 is straining the server; there are over 500000 distinct objects in there and Anonymous has been uploading hauls every 2 minutes to that census.cordership3 wrote:Catagolue is down again, and I don't know why this time.
EDIT: and is still doing so. Anonymous, please stop.
EDIT 2: I've modified Catagolue to ban this rule for the time being.
What do you do with ill crystallographers? Take them to the mono-clinic!
Re: catagolue is down
Catagolue appears to be down again, but now it's giving a 503 error instead of a 500 error like it was before.
Re: catagolue is down
Why didn't B3/S23/1x256X2+1 bring down the site in a few hours, despite me submitting hauls like twice as fast? ?? ??? ???? ????? ?????? ??????? ???????? ????????? ??????????
- Apple Bottom
- Posts: 1034
- Joined: July 27th, 2015, 2:06 pm
- Contact:
Re: catagolue is down
As has been explained before, it's really about the number of operations Catagolue needs to perform to process a haul.wwei23 wrote:Why didn't B3/S23/1x256X2+1 bring down the site in a few hours, despite me submitting hauls like twice as fast? ?? ??? ???? ????? ?????? ??????? ???????? ????????? ??????????
Suppose that a rulesym has a very rich repertoire of objects, and is very fast to search, and a user is submitting a haul with, say, 10,000 distinct objects every 2 minutes. That's (at least) 5,000 datastore operations Catagolue has to perform per minute, and since these cost money, the site's daily quota (2 USD, IIRC) will be used up relatively fast.
Suppose that a different rulesym is much less rich in objects, and a user submits hauls of, say, 100 distinct objects every minute. That's 100 datastore operations Catagolue has to perform per minute. I'm sure you can see the difference.
This isn't to say submitting hauls every minute is a good idea, even for less productive rules. IIRC, Calcyman's asked users to not submit hauls more often than once every 20 minutes from a single machine; so if you're running e.g. four searcher instances on that machine, you should adjust haul sizes that you're only submitting hauls once every 80 minutes at most from each.
Depending on the rule this'll reduce server load significantly. Suppose that for the same (second) rulesym considered, the number of distinct objects doubles if you increase the haul size by a factor of ten. Suppose now that you decide to submit a haul every 100 minutes instead of every minute. Hauls will now contain ~400 distinct objects on average, and Catagolue will have to perform ~4 datastore operations per minute and haul.
This is my best understanding of the situation anyway. Calcyman'd be able to tell you more.
(Oh, and you may want to reconsider how many punctuation marks you use.)
If you speak, your speech must be better than your silence would have been. — Arabian proverb
Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_
Proud member of the Pattern Raiders!
Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_
Proud member of the Pattern Raiders!
Re: catagolue is down
How can I search B3678/S25678 without bringing down Catagolue? I don’t know how to create wrapper scripts.
Maybe Catagolue should allow hauls larger than 1 megabyte.
Maybe Catagolue should allow hauls larger than 1 megabyte.
Re: catagolue is down
Wait, you were the one submitting all those "anonymous" B3678/S25678 hauls?wwei23 wrote:How can I search B3678/S25678 without bringing down Catagolue? I don’t know how to create wrapper scripts.
Maybe Catagolue should allow hauls larger than 1 megabyte.
Hauls larger than 1 megabyte would just require more datastore operations, though, so it wouldn't really help except in maybe a few specific rules (and it would make it easier for malicious users to crash Catagolue with rules like B/S012345678).
Re: catagolue is down
Objective frequency drops off exponentially, so larger hauls tend to have less unique objects per megabyte. As a result, it would bee better for Catagolue. Also, when I tried to search B3678/S25678, whenever I put the number of soups high enough to follow 1 haul every 20 minutes, the hauls are too big.77topaz wrote:Wait, you were the one submitting all those "anonymous" B3678/S25678 hauls?wwei23 wrote:How can I search B3678/S25678 without bringing down Catagolue? I don’t know how to create wrapper scripts.
Maybe Catagolue should allow hauls larger than 1 megabyte.
Hauls larger than 1 megabyte would just require more datastore operations, though, so it wouldn't really help except in maybe a few specific rules (and it would make it easier for malicious users to crash Catagolue with rules like B/S012345678).
- Majestas32
- Posts: 549
- Joined: November 20th, 2017, 12:22 pm
- Location: 'Merica
Re: catagolue is down
Search movero8 instead then
Searching:
b2-a5k6n7cs12-i3ij4k5j8
b2-a3c7cs12-i
Currently looking for help searching these rules.
b2-a5k6n7cs12-i3ij4k5j8
b2-a3c7cs12-i
Currently looking for help searching these rules.
- Apple Bottom
- Posts: 1034
- Joined: July 27th, 2015, 2:06 pm
- Contact:
Re: catagolue is down
And that should be telling you something.wwei23 wrote:Also, when I tried to search B3678/S25678, whenever I put the number of soups high enough to follow 1 haul every 20 minutes, the hauls are too big.
Now, accepting that Catagolue's limits are in place for a reason, I think the salient question isn't "why can't you search B3678/S25678" as much as "why do you want to use Catagolue". Or, perhaps more to the point, "why do you want to use Catagolue to store a FULL census" --- every object, including the entirely of the long tail?
Calcyman's said before that while data fed to Catagolue *can* come from apgsearch, it doesn't have to. As a corollary, it doesn't have to come from an unmodified apgsearch; you could easily rig up a version that only submits part of the result, i.e., the objects considered interesting, whatever that means in practice. (You could also have it save full logs locally, so you'd not lose any data yourself.) So long as you use a rulestring/symmetry that communicates the fact the resulting census isn't a regular, full soup search census, this should be fine, right?
Or alternatively, modify apgsearch so it only saves logs locally and doesn't submit to Catagolue, and use your own tools to hunt for interesting objects.
If you speak, your speech must be better than your silence would have been. — Arabian proverb
Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_
Proud member of the Pattern Raiders!
Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_
Proud member of the Pattern Raiders!
- Majestas32
- Posts: 549
- Joined: November 20th, 2017, 12:22 pm
- Location: 'Merica
Re: catagolue is down
The "UnbanMajestas" symetry
Searching:
b2-a5k6n7cs12-i3ij4k5j8
b2-a3c7cs12-i
Currently looking for help searching these rules.
b2-a5k6n7cs12-i3ij4k5j8
b2-a3c7cs12-i
Currently looking for help searching these rules.
Re: catagolue is down
catagolue is down again. i have my own guesses as to who caused it but i'll leave that in the air.
Re: catagolue is down
Hmm, I really think it would be useful to have some sort of feature on the site that showed how much of the daily quota has been used/remained. That would likely help avoid these kinds of outages in the future.
Re: catagolue is down
Catagolue is back! I ended up submitting a 78 million-soup haul in b3s23/C1, which is exactly the same as what I got the previous time Catagolue went down.
Oh, and also, looking at the hauls that processed immediately after Catagolue went back up, I think Heavpoot searching b2-a4z678s234w with hauls of ~100 KB and ~2600 distinct objects once per minute definitely contributed to Catagolue going down, and he's still searching that rule as of the time of writing; I recommend raising the threshold for that rule (he was actually warned on the Discord to increase his haul sizes multiple times before this outage).
Oh, and also, looking at the hauls that processed immediately after Catagolue went back up, I think Heavpoot searching b2-a4z678s234w with hauls of ~100 KB and ~2600 distinct objects once per minute definitely contributed to Catagolue going down, and he's still searching that rule as of the time of writing; I recommend raising the threshold for that rule (he was actually warned on the Discord to increase his haul sizes multiple times before this outage).
- Apple Bottom
- Posts: 1034
- Joined: July 27th, 2015, 2:06 pm
- Contact:
Re: catagolue is down
...yeah, that defies common sense, not to mention the ample warning given on the wiki.77topaz wrote:searching b2-a4z678s234w with hauls of ~100 KB and ~2600 distinct objects once per minute
Slow down cowboy!
EDIT: Calcyman tells me he's fixed this, by increasing the minimum required for that rule.
If you speak, your speech must be better than your silence would have been. — Arabian proverb
Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_
Proud member of the Pattern Raiders!
Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_
Proud member of the Pattern Raiders!
Re: catagolue is down
Catagolus Descendus!
also wait did they just ad the emoticon i never noticed it
also wait did they just ad the emoticon i never noticed it
Re: catagolue is down
Pretty sure I've seen it on Discord for a while now, but not as an animated emoticon because that's restricted to Nitro users.danny wrote:also wait did they just ad the emoticon i never noticed it
- cordership3
- Posts: 129
- Joined: August 23rd, 2016, 8:53 am
- Location: Smome tomato
- Contact:
Re: catagolue is down
Actually, I just refreshed the page, it seems to be back up already!cordership3 wrote: catagolue is over quota quota-o-meter when
- cordership3
- Posts: 129
- Joined: August 23rd, 2016, 8:53 am
- Location: Smome tomato
- Contact:
Re: catagolue is down
Ouch -- someone has been submitting hauls for g6b2ce3ar4a5ns013ij, which has lots of distinct objects. Storing the sample soups was causing a peak of 10000 datastore writes per minute. That incurred a total cost of $0.72 per hour, causing it to soon max out at the daily spending limit of $5.00.cordership3 wrote:it's down again
I've now made a commit to the repository to ensure sample soups are not stored for objects in tabulations exceeding 500kB (except for b3s23, which is exempt). That should limit the amount of damage someone can do by intensive searching of a single census.
What do you do with ill crystallographers? Take them to the mono-clinic!
Re: catagolue is down
Does this mean that all existing hauls which take up more than this amount (and, by extension, all rules whose only sample soups lie within such a haul) will be deleted?calcyman wrote:I've now made a commit to the repository to ensure sample soups are not stored for objects in tabulations exceeding 500kB (except for b3s23, which is exempt). That should limit the amount of damage someone can do by intensive searching of a single census.
Help wanted: How can we accurately notate any 1D replicator?
Re: catagolue is down
No, it merely means that when (for instance) b3678s34678/C1 reached 500 kB of objects in the xp2 tabulation, it will stop including sample soups for that census on individual xp2_??? object pages. The tabulations themselves (including frequencies of each object) are unaffected by this.muzik wrote:Does this mean that all existing hauls which take up more than this amount (and, by extension, all rules whose only sample soups lie within such a haul) will be deleted?calcyman wrote:I've now made a commit to the repository to ensure sample soups are not stored for objects in tabulations exceeding 500kB (except for b3s23, which is exempt). That should limit the amount of damage someone can do by intensive searching of a single census.
The justification is that if a tabulation reaches that size, the individual objects aren't interesting enough to bother recording their sample soups.
What do you do with ill crystallographers? Take them to the mono-clinic!
Re: catagolue is down
What if I'm searching for a certain object, and the tabulation is over 500KB? How will I post the soup in the soup search results in rules other than Conway's Life thread?
- Apple Bottom
- Posts: 1034
- Joined: July 27th, 2015, 2:06 pm
- Contact:
Re: catagolue is down
Look at the locally-saved log, in particular the @SAMPLE_SOUPIDS section.wwei23 wrote:What if I'm searching for a certain object, and the tabulation is over 500KB? How will I post the soup in the soup search results in rules other than Conway's Life thread?
If you speak, your speech must be better than your silence would have been. — Arabian proverb
Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_
Proud member of the Pattern Raiders!
Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_
Proud member of the Pattern Raiders!