Morning progress so far:
I figured out how the values (and the noise) are generated.
The source image is an 8-bits per pixel color image, the source pixel value is chosen from one of the color channels using a bayer filter, with a blue filter at 0, 0.
The final value is given by: clamp(0, 2**24-1, (source_value * 65793 + uniform(0, 1676)))
, where uniform(x, y) is a uniformly chosen random value between x and y inclusive.
Without cracking the noise source, the best we can do to encode the noise itself is 465255 bytes.
...because 347475 pixels in the image have non-255 values, and log2(1677^347475) = 3722036.5 bits.
The best I could do to encode the bulk of the data is 276554 bytes, again with the general purpose zpaq compressor.
Aside on attempted alternative compression methods for the bulk of the data:
Image compression did quite poorly here, I thought adam7 interlacing would help compression due to the bayer filter pattern, but png did not perform well. With zopflipng + pngcrush the best I could achieve was 329939 bytes.
This gives an approximate lower bound of 741809 bytes without either modeling the actual data better, or cracking the noise source. This does not include the data needed to describe the decompressor and glue all the data together into the original bitstream.
I believe I had a good start analyzing the file, although I’m currently slightly stuck on the exact details of the encoding.
Spoilers ahead for people who want to try things entirely by themselves.
My initial findings were that the raw file easily compresses from 2100086 bytes to 846149 bytes with zpaq -m5, probably hard to beat its context modeling with other compressors or even manual implementations.
I wrote a Python script to reverse the intern’s transformation and analyzed the bits, looks like the file is largely organized into septets of data. I dumped those septets in all the ways that made sense (inverted bits, inverted order of bits) into a file. None of them compressed better than the source.
I opened those files with GIMP as raw data files, and noticed that at 4000x600 8-bit grayscale with a small offset for the header it looks like some sort of 2D data, but it’s extremely noisy. There are lots of very flat areas in the file, though, which is why it seems to compress so well. Likely a threshold of some sort on the sensor data.
By only taking every fourth septet I could form four 1000x600 grayscale images. One of them seems to have actual structure (I presume it’s the most significant septet), the rest are dominated by noise.
It’s recognizable as a picture of a piece of crystalized bismuth. (https://i.imgur.com/rP5sq56.png) :::
There also seems to be some sort of pattern in the picture pixels similar to the raw data behind a Bayer filter in a camera. It might be a 2x2 or a 4x4 filter, I can’t quite tell. It could also just be some sort of block artifacts from the picture source (which wouldn’t be present in raw sensor data from aliens, but the author needs to source this image somehow).
The binary encoding seems to involve a sign bit in each septet if I’m interpreting things correctly, but I’m not sure how to combine septets into a sensible value yet.
I’m currently stuck at this point and won’t have time to work on it more this evening, but seems like good progress for one and a half hour.