DiffusionCraft transforms Minecraft scenes into AI images using Stable Diffusion. This works even in multiplayer.
Someone finally had to tackle it: Combining Minecraft with an AI image generator. After all, both platforms have something in common, namely a fairly low barrier to entry, and they open up practically endless possibilities for users to live out their creativity.
Since Stable Diffusion was released as open-source software in late August, countless enthusiasts have created their own Stable Diffusion projects. DiffusionCraft is another project that has been particularly well received on Reddit.
Minecraft image generation almost in real time
Reddit user Sean Simon aka "Lozmosis" has shared a video in which Minecraft buildings and scenes are converted into AI images by a Python script in near real-time. Currently, his RTX 3080-equipped computer takes about three to four seconds per image.
Real-time implementation is just a matter of hardware, Simon writes. The generation of a new image would run at a set interval or whenever a player places or removes a block. The developer explains the generation process like this:
- A script captures images from the Minecraft window, resizes or crops them to 512 x 512 pixels, and saves them to a folder.
- InvokeAI has been modified to run in a loop, targeting the last image in the folder and outputting the Stable Diffusion image.
- Another script displays the latest image in the Stable Diffusion folder via tkinter.
Minecraft becomes a multiplayer prompt for Stable Diffusion
While the AI image community as a whole relies heavily on collaboration, the actual input of a prompt is only a "single-player experience." DiffusionCraft allows multiple people to collaborate on an AI image at the same time by working together to build the Minecraft scene, which then serves as the template for image generation. Simon's script runs on a local server.
Reactions to the project have been overwhelmingly positive, as evidenced by the more than 1000 upvotes on the corresponding Reddit thread, among other things. Simon has not yet published his code, but updates are available on his Twitter account.
A major challenge, according to Simon, is that the current img2img model only accepts a single prompt. Therefore, he said, defining certain functions requires multiple computers running Stable Diffusion with the same image with separate prompts. Finally, there needs to be a final phase in which these images are reassembled.
Minecraft worlds made from AI images coming soon?
The transformation of Minecraft scenes into stable-diffusion images is already impressive - how far are we from reversing that order?
A Reddit account also asked if it would be possible to turn AI art into a Minecraft scene, meaning the whole process in reverse, to which DiffusionCraft developer Simon replied that he was working on something without going into detail.