apgsearch v2.2

For general discussion about Conway's Game of Life.
User avatar
Alexey_Nigin
Posts: 323
Joined: August 4th, 2014, 12:33 pm
Location: Ann Arbor, MI
Contact:

Re: apgsearch v2.2

Post by Alexey_Nigin » December 4th, 2015, 8:07 am

I have checked whether Catagolue is vulnerable to SQL injection.

Result: vulnerable.

Description of vulnerability: the chart breaks if the username ends with backslash (\).

Page I tried this on: b3s23/D2_+1.

Suggested solution: replace each backslash with double backslash (\\). Also, you replace each apostrophe (') with ('), which seems kinda clumsy to me. Don't you think it would be better to use backslash+apostrophe (\')?

Note to everybody: trying this on other pages before this is fixed is NOT a funny prank.
There are 10 types of people in the world: those who understand binary and those who don't.

User avatar
calcyman
Posts: 2198
Joined: June 1st, 2009, 4:32 pm

Re: apgsearch v2.2

Post by calcyman » December 4th, 2015, 10:09 am

Alexey_Nigin wrote:I have checked whether Catagolue is vulnerable to SQL injection.

Result: vulnerable.

Description of vulnerability: the chart breaks if the username ends with backslash (\).
Catagolue doesn't use SQL; the so-called 'vulnerability' is just a client-side display error. And you can't do a js injection since quotation marks are sanitised.

I've now resolved the problem by the means of the following line of code:

Code: Select all

String displayedName = parts[1].replaceAll("'","'").replaceAll("\\\\","\\\\\\\\");
Amusingly, it requires eight backslashes in the replaced string because:
  • Backslashes need to be escaped in string literals;
  • Backslashes need to be escaped in the regex-style replaceAll syntax;
  • Backslashes need to be escaped in the generated client-side js.
What do you do with ill crystallographers? Take them to the mono-clinic!

SuperJedi224
Posts: 25
Joined: November 25th, 2015, 5:26 pm

Re: apgsearch v2.2

Post by SuperJedi224 » December 4th, 2015, 12:36 pm

There should be a top 100 most common patterns page for each rule in the catalogue. There should also be a page explaining the apgcode syntax.

User avatar
calcyman
Posts: 2198
Joined: June 1st, 2009, 4:32 pm

Re: apgsearch v2.2

Post by calcyman » December 4th, 2015, 3:26 pm

SuperJedi224 wrote:There should be a top 100 most common patterns page for each rule in the catalogue.
You mean like this? https://catagolue.appspot.com/textcensus/b3s23/C1
There should also be a page explaining the apgcode syntax.
You mean like this? https://catagolue.appspot.com/help/wechsler.txt
What do you do with ill crystallographers? Take them to the mono-clinic!

SuperJedi224
Posts: 25
Joined: November 25th, 2015, 5:26 pm

Re: apgsearch v2.2

Post by SuperJedi224 » December 4th, 2015, 4:07 pm

calcyman wrote:
SuperJedi224 wrote:There should be a top 100 most common patterns page for each rule in the catalogue.
You mean like this? https://catagolue.appspot.com/textcensus/b3s23/C1
I forgot about those.
There should also be a page explaining the apgcode syntax.
You mean like this? https://catagolue.appspot.com/help/wechsler.txt
I had absolutely no idea that one existed!

User avatar
gameoflifeboy
Posts: 474
Joined: January 15th, 2015, 2:08 am

Re: apgsearch v2.2

Post by gameoflifeboy » December 5th, 2015, 3:52 am

Actually, both subpages on https://catagolue.appspot.com/help/ are pretty useful.
SuperJedi224 wrote:I had absolutely no idea that one existed!
There used to be a link to the help page next to the statistics page. I don't know why it got unlinked.

User avatar
Apple Bottom
Posts: 1033
Joined: July 27th, 2015, 2:06 pm
Contact:

Re: apgsearch v2.2

Post by Apple Bottom » December 5th, 2015, 4:20 pm

calcyman wrote:You mean like this? https://catagolue.appspot.com/help/wechsler.txt
Speaking of that file, could you by any chance be persuaded to amend it to describe the format used for the codes of linear-growth patterns as well? I'm a little puzzled over those, but I'm not quite ready to delve into the apgnano code to try and work out how they're generated just yet.
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

SuperJedi224
Posts: 25
Joined: November 25th, 2015, 5:26 pm

Re: apgsearch v2.2

Post by SuperJedi224 » December 5th, 2015, 5:59 pm

Is there a help page about the symmetry codes?

User avatar
Apple Bottom
Posts: 1033
Joined: July 27th, 2015, 2:06 pm
Contact:

Re: apgsearch v2.2

Post by Apple Bottom » December 5th, 2015, 7:14 pm

SuperJedi224 wrote:Is there a help page about the symmetry codes?
There was a thread about them recently, here, with both textual explanations and pictures.
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

thunk
Posts: 170
Joined: October 3rd, 2015, 8:50 pm
Location: Central USA

Re: apgsearch v2.2

Post by thunk » December 5th, 2015, 10:43 pm

Apple Bottom wrote:
SuperJedi224 wrote:Is there a help page about the symmetry codes?
There was a thread about them recently, here, with both textual explanations and pictures.
Speaking of which, couldn't be 32*32 initial seeds used in the symmetric searches be overlapped slightly, with the overlapping parts XOR'd to produce a single value? The resulting 28*28 pattern should still be symmetric (if I'm thinking straight), and will fit in the starting box of apgnano.

EDIT: I just realised that there would be a problem with orthogonal or diagonal blank strips for the odd symmetries. Not sure how much of an issue that would actually be, or whether they could be filled in somehow.
"What's purple and commutes?
The Evanston Express."

User avatar
Scorbie
Posts: 1445
Joined: December 7th, 2013, 1:05 am

Re: apgsearch v2.2

Post by Scorbie » December 16th, 2015, 2:46 am

thunk wrote:EDIT: I just realised that there would be a problem with orthogonal or diagonal blank strips for the odd symmetries. Not sure how much of an issue that would actually be, or whether they could be filled in somehow.
Actually, that problem would have been in the original python script regardless of soup size if symmetries were implemented that way. Fortunately, they weren't. The python script fills in only the half/quarter/semi-quarter, with edges/diagonals/centers included, and decalcomanies them. So it wouldn't have that annoying odd-strip problem.

The thing I'm curious is: Would Catagolue accept 28x28 soups if we did make and census one?
Best wishes to you, Scorbie

User avatar
Scorbie
Posts: 1445
Joined: December 7th, 2013, 1:05 am

Re: apgsearch v2.2

Post by Scorbie » December 19th, 2015, 11:36 pm

EEK! I get errors when compiling apgnano!
The error message I got was a bunch of:

Code: Select all

life128.h:1370:183: error: unknown register name ‘xmm14’ in ‘asm’
I use 32-bit Linux, with Intel Atom CPU (which is listed on CPUs that support SSE2, SSE3 instructions)
Googling told me to check if g++ enables -mmmx or -msse2 by default and add the flags if not, so I did.
Fewer errors popped out. Before all 'xmm1~xmm14' caused errors but now it's 'xmm8~xmm14'.
What could be the problem and how could I solve it? If the solution is merely adding compiler flags then I think it would be wise to add the flags to the makefile in apgnano.git.
Best wishes to you, Scorbie

User avatar
calcyman
Posts: 2198
Joined: June 1st, 2009, 4:32 pm

Re: apgsearch v2.2

Post by calcyman » December 20th, 2015, 7:08 am

Scorbie wrote:I use 32-bit Linux
That's the problem; apgnano only works with x86-64 machines:

https://en.wikipedia.org/wiki/Streaming ... #Registers
Wikipedia wrote:The AMD64 extensions from AMD (originally called x86-64) added a further eight registers XMM8 through XMM15, and this extension is duplicated in the Intel 64 architecture.
I didn't realise people still used 32-bit machines. Sorry!
What do you do with ill crystallographers? Take them to the mono-clinic!

User avatar
Scorbie
Posts: 1445
Joined: December 7th, 2013, 1:05 am

Re: apgsearch v2.2

Post by Scorbie » December 20th, 2015, 9:30 am

calcyman wrote:
Scorbie wrote:I use 32-bit Linux
That's the problem; apgnano only works with x86-64 machines.
Wikipedia wrote:The AMD64 extensions from AMD (originally called x86-64) added a further eight registers XMM8 through XMM15, and this extension is duplicated in the Intel 64 architecture.
I didn't realise people still used 32-bit machines. Sorry!
:o Didn't realize! I did see that apgnano only supports "x86-64, which is mostly the case except for really old hardware." And yes, I did try to install apgnano on a crappy old machine (my 5-yr-old netbook) but here I do see a lot of 32bit machines out there. (Even in my home there are two 32bit machines.) I thought you wanted to say x86/x86-64. Whoops.
Is there any obvious way to fix apgnano to work in 32bit machines? (Possibly at the cost of not using assembly?) Probably not, just hoping...
Best wishes to you, Scorbie

User avatar
Apple Bottom
Posts: 1033
Joined: July 27th, 2015, 2:06 pm
Contact:

Re: apgsearch v2.2

Post by Apple Bottom » December 20th, 2015, 9:08 pm

calcyman wrote:I didn't realise people still used 32-bit machines. Sorry!
Never underestimate the amount of legacy hardware (and software, for that matter) out in the wild - you'd be surprised what people still have running. And many folks got older machines they could dust off and repurpose as soup searchers given the opportunity.
Scorbie wrote:Is there any obvious way to fix apgnano to work in 32bit machines? (Possibly at the cost of not using assembly?) Probably not, just hoping...
Just tossing out a random thought: does apgnano run several soups in parallel the way apgsearch 1.x did? If so, perhaps it could be made to only search half as many at the same time, only requiring half the number of registers, seeing as how the first 8 are available in 32-bit mode as well.

- Apple Bottom, who's always happy to add to someone else's workload
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

User avatar
Scorbie
Posts: 1445
Joined: December 7th, 2013, 1:05 am

Re: apgsearch v2.2

Post by Scorbie » December 20th, 2015, 11:06 pm

Apple Bottom wrote:- Apple Bottom, who's always happy to add to someone else's workload
Laughing out loud :lol:
Apple Bottom wrote:Just tossing out a random thought: does apgnano run several soups in parallel the way apgsearch 1.x did? If so, perhaps it could be made to only search half as many at the same time, only requiring half the number of registers, seeing as how the first 8 are available in 32-bit mode as well.
Wait a minute!!! If apgnano doesn't accept soups sized > 28x28 then it must mean it doesn't run soups in parallel, does it? And running soups in parallel made a 4x speed boost in apgsearch.py!

EDIT: @wildmyron Ahh, you're right. I was mistaken. Yes, apgsearch does the censusing part parallel, rather than the generating part, and yeah, I also agree that calcyman had optimized apgnano pretty well. I'm pretty sure he does the censusing parallel.
Last edited by Scorbie on December 21st, 2015, 3:08 am, edited 1 time in total.
Best wishes to you, Scorbie

wildmyron
Posts: 1397
Joined: August 9th, 2013, 12:45 am

Re: apgsearch v2.2

Post by wildmyron » December 21st, 2015, 12:56 am

The python version doesn't (and has never) run the soups in parallel. It runs sqrtspp^2 soups serially until stabilisation and then performs the census in parallel on the ashes of all of the soups in one universe. I suspect this is primarily because the time to stabilisation is unknown and so it is difficult to determine how far apart to place soups to ensure there is no interaction between them. Once all the soups have turned to ash however, the maximum number of generations to stabilisation determines the required spacing between ashes in the census.

I haven't looked at apgnano, but in light of calcyman saying that the census is performed directly in C code (i.e. without the use of auxillary rules) I presume the census is not done in parallel. Even if it was possible to run the soups in parallel (in apgnano) I don't believe there would be any speed advantage because the Life128 algorithm is not hashtile based. Details are on the first page of this thread.
The latest version of the 5S Project contains over 226,000 spaceships. There is also a GitHub mirror of the collection. Tabulated pages up to period 160 (out of date) are available on the LifeWiki.

User avatar
Saka
Posts: 3400
Joined: June 19th, 2015, 8:50 pm
Location: In the kingdom of Sultan Hamengkubuwono X
Contact:

Re: apgsearch v2.2

Post by Saka » December 22nd, 2015, 12:43 am

I recently uploaded the first soup with xs22_g9f033zdl913, but when I visit the page, for the still life, it says there are no soups stored in the catagolue. why isn't it "There is one soup stored in the catagolue"?
(https://catagolue.appspot.com/object/xs ... l913/b3s23)

Code: Select all

o3b2ob2obo3b2o2b2o$bo3b2obob3o3bo2bo$2bo2b3o5b3ob4o$3o3bo2bo2b3o3b3o$
4bo4bobo4bo$2o2b2o2b4obo2bo3bo$2ob4o3bo2bo2bo2bo$b2o3bobob2o$3bobobo5b
obobobo$3bobobob2o3bo2bobo!
(Check gen 3)
Add your computer to the Table of Lifeenthusiast Computers!

User avatar
biggiemac
Posts: 504
Joined: September 17th, 2014, 12:21 am
Location: California, USA

Re: apgsearch v2.2

Post by biggiemac » December 22nd, 2015, 2:36 am

Saka wrote:I recently uploaded the first soup with xs22_g9f033zdl913, but when I visit the page, for the still life, it says there are no soups stored in the catagolue. why isn't it "There is one soup stored in the catagolue"?
(https://catagolue.appspot.com/object/xs ... l913/b3s23)

It lists one now. I think there is a 15 minute latency. Say you submit a haul at 15:11. At 15:15 that haul will enter limbo, where it awaits verification from some other instance. Almost always there will be verification before the next cron at 15:30. Then the objects will appear at 15:30.
Physics: sophistication from simplicity.

User avatar
calcyman
Posts: 2198
Joined: June 1st, 2009, 4:32 pm

Re: apgsearch v2.2

Post by calcyman » December 22nd, 2015, 7:45 pm

(No, apgnano doesn't run multiple soups in the same universe because there's no rule-switching overhead. The only reason to do parallel censusing in apgsearch v1.x is the time taken by Golly to switch between rules, which themselves were only necessary since manipulating lists of cells in Python is incredibly slow. In C++, these weird performance hacks are no longer effective.)

By the way, we've recently passed 2^45 objects in the Catagolue. #brimfulofhashes

One of the more recent objects was the thirteenth distinct p30 oscillator (found by @doctroid, earning a Monarchist badge in the process: https://catagolue.appspot.com/user/@doctroid ).
What do you do with ill crystallographers? Take them to the mono-clinic!

User avatar
Apple Bottom
Posts: 1033
Joined: July 27th, 2015, 2:06 pm
Contact:

Re: apgsearch v2.2

Post by Apple Bottom » January 4th, 2016, 5:43 am

calcyman wrote:By the way, we've recently passed 2^45 objects in the Catagolue. #brimfulofhashes
And, now, 40 trillion objects. That's a barnful!
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

User avatar
Saka
Posts: 3400
Joined: June 19th, 2015, 8:50 pm
Location: In the kingdom of Sultan Hamengkubuwono X
Contact:

Re: apgsearch v2.2

Post by Saka » January 9th, 2016, 6:49 pm

How fast does apgnano run on other people's computer? Is it just my computer being slow or what? People usually upload 20 million soups per haul, how long does processing 20 million soups take?

Code: Select all

o3b2ob2obo3b2o2b2o$bo3b2obob3o3bo2bo$2bo2b3o5b3ob4o$3o3bo2bo2b3o3b3o$
4bo4bobo4bo$2o2b2o2b4obo2bo3bo$2ob4o3bo2bo2bo2bo$b2o3bobob2o$3bobobo5b
obobobo$3bobobob2o3bo2bobo!
(Check gen 3)
Add your computer to the Table of Lifeenthusiast Computers!

thunk
Posts: 170
Joined: October 3rd, 2015, 8:50 pm
Location: Central USA

Re: apgsearch v2.2

Post by thunk » January 9th, 2016, 7:43 pm

Saka wrote:How fast does apgnano run on other people's computer? Is it just my computer being slow or what? People usually upload 20 million soups per haul, how long does processing 20 million soups take?
In my experience, running 10-million soup hauls on my two computers (desktop and laptop, both with core i7 processors), took 215 minutes per core for the former and 200 minutes for the latter. As stated in one of my posts above, this became inexplicably slightly slower with time (I have since stopped running soups, at least for now while I don't have access to the desktop.) At 8 cores in my desktop and 4 in my laptop, the total time to create 20 million soups was about 34 minutes.
"What's purple and commutes?
The Evanston Express."

User avatar
drc
Posts: 1664
Joined: December 3rd, 2015, 4:11 pm
Location: creating useless things in OCA

Re: apgsearch v2.2

Post by drc » January 9th, 2016, 10:33 pm

Saka wrote:How fast does apgnano run on other people's computer? Is it just my computer being slow or what? People usually upload 20 million soups per haul, how long does processing 20 million soups take?
That's probably me. It takes usually 6 hours (slow computer)
\100\97\110\105

User avatar
Saka
Posts: 3400
Joined: June 19th, 2015, 8:50 pm
Location: In the kingdom of Sultan Hamengkubuwono X
Contact:

Re: apgsearch v2.2

Post by Saka » January 20th, 2016, 3:51 am

Err...
Nevermind

Code: Select all

o3b2ob2obo3b2o2b2o$bo3b2obob3o3bo2bo$2bo2b3o5b3ob4o$3o3bo2bo2b3o3b3o$
4bo4bobo4bo$2o2b2o2b4obo2bo3bo$2ob4o3bo2bo2bo2bo$b2o3bobob2o$3bobobo5b
obobobo$3bobobob2o3bo2bobo!
(Check gen 3)
Add your computer to the Table of Lifeenthusiast Computers!

Post Reply