Korva is a scroll of the internet’s best ideas.
Meta image model searches the web
Earlier image models had no context for a prompt like that and could only work from what they were trained on. Plugged into the web, Meta's model grounds its images in factual, real-time information.
The model also edits by region. A user can circle one part of the sky and request changes to only that part, or pull individual assets out of an image or video, much like magic layers in Canva. Models from a couple of years ago took a prompt in a black box and returned an output with no way to change anything locally.
A third feature starts from a photo of a room. Meta AI can redesign the space using real products from the web and from Facebook Marketplace.
How does a model edit only one region?
The technique is called inpainting. The user's selection becomes a mask, and the model regenerates pixels only inside the mask while holding the rest of the image fixed as context. A diffusion model starts the masked area as noise and gradually removes it while conditioning on the visible pixels around it. Because the surroundings stay constant, the edit blends with the original image.
How does web search ground an image model?
The general method is called retrieval-augmented generation. Researchers at Facebook AI Research, led by Patrick Lewis, introduced it in a 2020 paper that combined a language model with a retrieval step. The system fetches relevant documents at request time and places them in the prompt, so the model can use facts that were not in its training data. For an image model, retrieved text such as a score or a lineup can guide what the picture shows.
Why did older image models garble text?
Early diffusion models, such as the 2022 release of Stable Diffusion, learned letters as visual patterns and not as spelling. Words therefore came out as lookalike shapes that were close to real letters but not quite right. Training captions also rarely recorded the text visible inside an image. Later models paired with stronger language encoders rendered text far more reliably.
More from Lightspeed Venture Partners
