Thanks for sharing! I think the paper is cool (though massively buries the lede). My summary:
They create a synthetic dataset for lit and unlit rooms with styleGAN. They exploit the fact that the GAN has disentangled and meaningful directions in its latent space, that can be individually edited. They find a lighting latent automatically, by taking noise that produces rooms, editing each latent in turn and looking for big changes specifically on light pixels
StyleGAN does not have a text input, and there’s no mention of prompting (as far as I can tell—I’m not familiar with GANs). This is not a DALL-E style model. Its input is just Gaussian noise
This is a really cool result, and I am excited about it! The claim that GANs have disentangled latents (and that this is known), which makes this less exciting (man I wish this was true of LLMs). But it’s still solid!
This is in section 4.1
They manually create a dataset of lit and unlit rooms, which isn’t that interesting. They use this for benchmarking their method, not for actually training it (I don’t find this that exciting)
They use the GAN as a source of training data, to train a model specifically for lit → unlit rooms (I don’t find this that exciting)
Yeah, sorry, I should have made clear that the story that I tell in the post is not contained in the linked paper. Rather, it’s a story that David Bau sometimes tells during talks, and which I wish were wider-known. As you note, the paper is about the problem of taking specific images and relighting them (not of generating any image at all of an indoor scene with unlit lamps), and the paper doesn’t say anything about prompt-conditioned models. As I understand things, in the course of working on the linked project, Bau’s group noticed that they couldn’t get scenes with unlit lamps out of the popular prompt-conditioned generative image models.
Ah, thanks for the clarification! That makes way more sense. I was confused because you mentioned this in a recent conversation, I excitedly read the paper, and then couldn’t see what the fuss was about (your post prompted me to re-read and notice section 4.1, the good section!).
Thanks for sharing! I think the paper is cool (though massively buries the lede). My summary:
They create a synthetic dataset for lit and unlit rooms with styleGAN. They exploit the fact that the GAN has disentangled and meaningful directions in its latent space, that can be individually edited. They find a lighting latent automatically, by taking noise that produces rooms, editing each latent in turn and looking for big changes specifically on light pixels
StyleGAN does not have a text input, and there’s no mention of prompting (as far as I can tell—I’m not familiar with GANs). This is not a DALL-E style model. Its input is just Gaussian noise
This is a really cool result, and I am excited about it! The claim that GANs have disentangled latents (and that this is known), which makes this less exciting (man I wish this was true of LLMs). But it’s still solid!
This is in section 4.1
They manually create a dataset of lit and unlit rooms, which isn’t that interesting. They use this for benchmarking their method, not for actually training it (I don’t find this that exciting)
They use the GAN as a source of training data, to train a model specifically for lit → unlit rooms (I don’t find this that exciting)
Yeah, sorry, I should have made clear that the story that I tell in the post is not contained in the linked paper. Rather, it’s a story that David Bau sometimes tells during talks, and which I wish were wider-known. As you note, the paper is about the problem of taking specific images and relighting them (not of generating any image at all of an indoor scene with unlit lamps), and the paper doesn’t say anything about prompt-conditioned models. As I understand things, in the course of working on the linked project, Bau’s group noticed that they couldn’t get scenes with unlit lamps out of the popular prompt-conditioned generative image models.
Ah, thanks for the clarification! That makes way more sense. I was confused because you mentioned this in a recent conversation, I excitedly read the paper, and then couldn’t see what the fuss was about (your post prompted me to re-read and notice section 4.1, the good section!).