I sat down before going to bed and believe I have made some more progress.
I experimented with what I called the sign bit in the earlier post, and I’m certain I got it wrong. By ignoring the sign bit, I can reconstruct a much higher fidelity image. I can also do a non-obvious operation of rotating the bit to least significant place after inverting. I can’t visually distinguish the two approaches, though.
I wrote a naive debayering filter and got this image out: https://i.imgur.com/e5ydBTb.png (bit rotated version, 16-bit color RGB. Red channel on even rows/columns is exact, blue channel on odd rows/columns is exact, green channel is exact elsewhere.)
You can reverse image search that image to find that it’s a standard stock photo of bismuth, example larger but slightly cropped version: http://blog-imgs-98.fc2.com/s/c/i/scienceminestrone/Bismuth.jpg
Finding the original image would definitely help, even if it wouldn’t fit the spirit of this challenge.
I haven’t yet tried going in the reverse direction and seeing how much space this saves—doing this properly is tricky. I’m not aware of any good, ready-to-use libraries that provide things like context modeling and arithmetic coding, so writing a custom compressor is a lot of work.
In fact, I believe it may be worth trying to break the author’s noise source on the sensor. Most programming languages use a fairly breakable PRNG, either a xorshift variant or an LCG. But this may be a dead end if cryptographic randomness was used. Again, this wouldn’t fit the spirit of this challenge, but it would minimize description length if it worked.
I believe I had a good start analyzing the file, although I’m currently slightly stuck on the exact details of the encoding.
Spoilers ahead for people who want to try things entirely by themselves.
My initial findings were that the raw file easily compresses from 2100086 bytes to 846149 bytes with zpaq -m5, probably hard to beat its context modeling with other compressors or even manual implementations.
I wrote a Python script to reverse the intern’s transformation and analyzed the bits, looks like the file is largely organized into septets of data. I dumped those septets in all the ways that made sense (inverted bits, inverted order of bits) into a file. None of them compressed better than the source.
I opened those files with GIMP as raw data files, and noticed that at 4000x600 8-bit grayscale with a small offset for the header it looks like some sort of 2D data, but it’s extremely noisy. There are lots of very flat areas in the file, though, which is why it seems to compress so well. Likely a threshold of some sort on the sensor data.
By only taking every fourth septet I could form four 1000x600 grayscale images. One of them seems to have actual structure (I presume it’s the most significant septet), the rest are dominated by noise.
It’s recognizable as a picture of a piece of crystalized bismuth. (https://i.imgur.com/rP5sq56.png) :::
There also seems to be some sort of pattern in the picture pixels similar to the raw data behind a Bayer filter in a camera. It might be a 2x2 or a 4x4 filter, I can’t quite tell. It could also just be some sort of block artifacts from the picture source (which wouldn’t be present in raw sensor data from aliens, but the author needs to source this image somehow).
The binary encoding seems to involve a sign bit in each septet if I’m interpreting things correctly, but I’m not sure how to combine septets into a sensible value yet.
I’m currently stuck at this point and won’t have time to work on it more this evening, but seems like good progress for one and a half hour.