Scatter plot for still lifes (R help needed)

For discussion directly related to ConwayLife.com, such as requesting changes to how the forums or home page function.
Post Reply
User avatar
Apple Bottom
Posts: 1034
Joined: July 27th, 2015, 2:06 pm
Contact:

Scatter plot for still lifes (R help needed)

Post by Apple Bottom » July 27th, 2016, 7:43 pm

I was thinking it would be nice to have a scatter plot of (total) still life counts vs. populations in the Catagolue article on the LifeWiki (which is why I'm sticking this in "Website Discussion", BTW), akin to the upper plot on page 2 of the full results for Andrzej Okrasinski's census.

That sort of thing's best done programmatically, so I turned to R, which I vaguely recalled being told was useful; but I've never used that before myself, and found it rather unwieldy, so I was wondering if anyone here either a) was an R wizard who could help improve the following plot, or b) had suggestions for a better tool.

For now here's the current best attempt at a plot I have:
g5zCn4m.png
g5zCn4m.png (6.86 KiB) Viewed 186 times
Here's the R script used:

Code: Select all

library(bit64)
options(scipen=100)
stilllifes <- read.table("xs_scatterplot.dat", header=TRUE);
attach(stilllifes)
## svg("xs_scatterplot.svg", width=10, height=10, antialias="subpixel")
png("xs_scatterplot.png", width=640, height=640, antialias="cleartype")
plot(Population, Count, log="y", lab=c(length(Population), ceiling(log10(max(Count)))+1, 7), xaxs="i")
## text(Population, Count, Count, cex=0.6, pos=4, col="red")
legend("topright", inset=0.05, title="Total still lifes", c("Count", "Linear fit", expression(paste("LOWESS fit, ", alpha, "=0.3"))), pch=c(21, -1, -1), lty=c(0, 1, 1), col=c("black", "black", "blue"))
title(main="Total still lifes by population")
abline(lm(log10(Count)~Population))
lines(lowess(stilllifes, f=0.3), col="blue")
dev.off()
And finally, the data file (note that this was generated from somewhat older data; I'd use newer data when generating an image to upload to the LifeWiki):

Code: Select all

Population	Count
4	37230663244700
5	5217667971195
6	22811439059639
7	6028058512999
8	1151083489759
9	6130605168
10	4179800708
11	1236556937
12	186490260778
13	157670526
14	71439981537
15	423074176
16	2672307142
17	140665171
18	502771638
19	17640474
20	62870208
21	1460833
22	9283930
23	454126
24	409529
25	209479
26	142231
27	30761
28	2932260
29	8988
30	43911
31	17135
32	7052
33	177
34	300
35	23
36	51
37	9
38	402
39	4
40	5092
41	2
42	64
45	1
46	1
So, yeah. Any tips on how to improve this plot? R seems to offer precious little control over formatting, axes, labels, ticks etc., but maybe I'm just looking in the wrong places.

Alternatively, any other (better) tools to generate such plots? Programs that can be used non-interactively (i.e. called with a script as part of a shell script) would be very strongly preferred.
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

User avatar
Scorbie
Posts: 1692
Joined: December 7th, 2013, 1:05 am

Re: Scatter plot for still lifes (R help needed)

Post by Scorbie » July 28th, 2016, 5:34 am

Python + matplotlib?
Disclaimer: I have only heard of it.

User avatar
Apple Bottom
Posts: 1034
Joined: July 27th, 2015, 2:06 pm
Contact:

Re: Scatter plot for still lifes (R help needed)

Post by Apple Bottom » July 28th, 2016, 5:55 am

Scorbie wrote:Python + matplotlib?
Disclaimer: I have only heard of it.
"matplotlib tries to make easy things easy and hard things possible" -- that certainly sounds promising. (Also, they're taking more than a subtle cue from Perl there, aren't they.)

I'll look into it, thanks!

EDIT: not much luck so far. Cygwin doesn't provide a package, and the pip package can't even be installed:

Code: Select all

$ pip install matplotlib
[...]
    running build_ext
    building 'matplotlib.ft2font' extension
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-gwez0e7l/matplotlib/setup.py", line 277, in <module>
        **extra_args
      File "/usr/lib/python3.4/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.4/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.4/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib/python3.4/distutils/command/install.py", line 539, in run
        self.run_command('build')
      File "/usr/lib/python3.4/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.4/distutils/command/build.py", line 126, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.4/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.4/site-packages/setuptools/command/build_ext.py", line 50, in run
        _build_ext.run(self)
      File "/usr/lib/python3.4/distutils/command/build_ext.py", line 339, in run
        self.build_extensions()
      File "/usr/lib/python3.4/distutils/command/build_ext.py", line 448, in build_extensions
        self.build_extension(ext)
      File "/usr/lib/python3.4/site-packages/setuptools/command/build_ext.py", line 183, in build_extension
        _build_ext.build_extension(self, ext)
      File "/usr/lib/python3.4/distutils/command/build_ext.py", line 500, in build_extension
        include_dirs=ext.include_dirs,
      File "/tmp/pip-build-gwez0e7l/matplotlib/setupext.py", line 785, in __get__
        result = obj._hooks[self._name]() + result
      File "/tmp/pip-build-gwez0e7l/matplotlib/setupext.py", line 805, in include_dirs_hook
        import numpy
      File "/usr/lib/python3.4/site-packages/numpy/__init__.py", line 170, in <module>
        from . import add_newdocs
      File "/usr/lib/python3.4/site-packages/numpy/add_newdocs.py", line 13, in <module>
        from numpy.lib import add_newdoc
      File "/usr/lib/python3.4/site-packages/numpy/lib/__init__.py", line 17, in <module>
        from . import scimath as emath
    ImportError: cannot import name 'scimath'

    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-gwez0e7l/matplotlib/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-bb_8_3xg-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-gwez0e7l/matplotlib/
$ 
Google's not helpful, either. So, no dice with Cygwin.

There's apparently precompiled binaries for Python on Windows, but Python itself (3.5.2) can't be installed, failing with "0x80080005 server execution failed" and advising me to check its install log (which of course contains nothing whatsoever related). But it matters little, because there are not actually any precompiled windows packages anyway, either for 3.5 or the older 2.7 I still have installed for use with Golly.

So currently I'm downloading an 800 MiB bundle called Python(x, y) which bundles matplotlib and which I hope will finally actually work.

EDIT 2: matplotlib as bundled with Python(x, y) works, but Python(x, y) itself does not contain the python27.dll file needed for Python scripts in Golly. And unfortunately it also cannot coexist with a vanilla Python 2.7, so this isn't a solution either.

OTOH I was able to pip-install matplotlib on Cygwin's Python after installing the (unrelated, IMO, but what do I know) gtk 2.0 library/headers packages. After also installing statsmodels (which necessitated, among other things, LAPACK and a Fortran compiler) I whipped up this:
lFaLDip.png
lFaLDip.png (49.42 KiB) Viewed 187 times
Which is a good start. Script:

Code: Select all

#!/usr/bin/python3

import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm

# read data file
datafile=open("xs_scatterplot.dat","rb")
x, y=np.loadtxt(datafile,delimiter="\t",skiprows=1,unpack=True)

# generate a scatter plot
fig, ax=plt.subplots()
ax.scatter(x, y, label='Still lifes')
ax.set_yscale('symlog', linthreshy=1)
ax.set_xlabel('Population', fontsize=15)
ax.set_ylabel("Count", fontsize=15)
ax.set_title('Total still lifes', fontsize=20)
plt.ylim(ymin=0)
ax.grid(True)
fig.tight_layout()

# plot linear fit
logy=np.log(y)
coefficients=np.polyfit(x, logy, 1)
polynomial=np.poly1d(coefficients)
ys=polynomial(x)
plt.plot(x, np.exp(ys), label='Linear fit')

# plot LOWESS fit
lowess=sm.nonparametric.lowess
z=lowess(logy, x, frac=0.3, it=3, return_sorted=False)
plt.plot(x, np.exp(z), label=r'LOWESS fit, $\alpha=0.3$')

# add legend
plt.legend(loc='upper right')

# save plot to file
fig.set_size_inches(8, 8)
plt.savefig("xs_scatterplot.png", dpi=80)
Data:

Code: Select all

Population      Count
4       38443380059190
5       5387615762362
6       23554442686741
7       6224388388190
8       1188573288568
9       6330295738
10      4315921916
11      1276821800
12      192563548091
13      162807618
14      73766540016
15      436861751
16      2759365374
17      145246934
18      519150780
19      18213412
20      64918210
21      1507933
22      9586023
23      468929
24      422767
25      216249
26      146749
27      31728
28      3027923
29      9266
30      45332
31      17680
32      7284
33      181
34      304
35      24
36      55
37      9
38      410
39      4
40      5265
41      2
42      65
45      1
46      1
I'm fairly happy with how sane matplotlib's defaults are and how good its plots look without much tweaking -- most of the time required for this was installing Python modules, figuring out issues with the same, and trying to understand matplotlib's documentation. ;)
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

User avatar
Scorbie
Posts: 1692
Joined: December 7th, 2013, 1:05 am

Re: Scatter plot for still lifes (R help needed)

Post by Scorbie » July 31st, 2016, 1:25 am

Hmm... So you use python from cygwin with golly? That's very unique. Vanilla python for windows installs matplotlib without a problem:

Code: Select all

PS C:\Users\Scorbie> python -m pip install matplotlib
Collecting matplotlib
  Downloading matplotlib-1.5.1-cp35-none-win32.whl (6.2MB)
    100% |################################| 6.2MB 126kB/s
Collecting numpy>=1.6 (from matplotlib)
  Downloading numpy-1.11.1-cp35-none-win32.whl (6.6MB)
    100% |################################| 6.6MB 125kB/s
Collecting pytz (from matplotlib)
  Downloading pytz-2016.6.1-py2.py3-none-any.whl (481kB)
    100% |################################| 481kB 1.4MB/s
Collecting pyparsing!=2.0.4,>=1.5.6 (from matplotlib)
  Downloading pyparsing-2.1.5-py2.py3-none-any.whl (42kB)
    100% |################################| 51kB 3.7MB/s
Collecting python-dateutil (from matplotlib)
  Downloading python_dateutil-2.5.3-py2.py3-none-any.whl (201kB)
    100% |################################| 204kB 2.4MB/s
Collecting cycler (from matplotlib)
  Downloading cycler-0.10.0-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil->matplotlib)
  Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: numpy, pytz, pyparsing, six, python-dateutil, cycler, matplotlib
Successfully installed cycler-0.10.0 matplotlib-1.5.1 numpy-1.11.1 pyparsing-2.1.5 python-dateutil-2.5.3 pytz-2016.6.1 six-1.10.0

Edit That's a very decent short script indeed! couldn't get statsmodels library to work in windows, but except that it's kinda nice :)
Note for readers: the xs_scatterplot.dat file has its tabs converted to spaces by phpBB.

User avatar
Apple Bottom
Posts: 1034
Joined: July 27th, 2015, 2:06 pm
Contact:

Re: Scatter plot for still lifes (R help needed)

Post by Apple Bottom » July 31st, 2016, 5:49 am

Scorbie wrote:Hmm... So you use python from cygwin with golly? That's very unique. Vanilla python for windows installs matplotlib without a problem:[/code]

No, Golly's using the regular Windows version of Python. But when working on the command line I prefer Cygwin, and use Cygwin's python there.
Scorbie wrote:Edit That's a very decent short script indeed! couldn't get statsmodels library to work in windows, but except that it's kinda nice :)
Note for readers: the xs_scatterplot.dat file has its tabs converted to spaces by phpBB.
Thanks! And good to know about phpBB playing tricks there. (I suppose I could use commas instead of tabs when generating that file anyway; the only reason I used tabs is because that's what R liked best.)
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

User avatar
Apple Bottom
Posts: 1034
Joined: July 27th, 2015, 2:06 pm
Contact:

Re: Scatter plot for still lifes (R help needed)

Post by Apple Bottom » October 4th, 2016, 6:44 pm

BTW, just for fun I now generated images for not just B3/S23/C1 but B3/S23/!C1 (i.e. all other symmetries combined) as well. These are all using data from October 1st (right now; MediaWiki does not support linking to stable versions of the current revision of an uploaded file, so these will always point to the latest image):

Image
Image
Image
Image

I'm honestly not sure the LOWESS fit in particular is actually meaningful, but the plots sure are interesting to look at.

For the data and scripts used, see the file description pages on the wiki, linked e.g. from the Catagolue article.
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

User avatar
Scorbie
Posts: 1692
Joined: December 7th, 2013, 1:05 am

Re: Scatter plot for still lifes (R help needed)

Post by Scorbie » October 8th, 2016, 10:37 pm

Wow, seriously nice work with the plots and the wiki article!

User avatar
Apple Bottom
Posts: 1034
Joined: July 27th, 2015, 2:06 pm
Contact:

Re: Scatter plot for still lifes (R help needed)

Post by Apple Bottom » October 9th, 2016, 5:15 am

Scorbie wrote:Wow, seriously nice work with the plots and the wiki article!
Aww shucks, sugarcube. Thank you! :D
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

Post Reply