Models that use human pointers and sketches to produce realistic fashion images

Artificial intelligence (AI) has recently begun to penetrate many creative industries, for example in the form of tools for digital artists, architects, interior designers and image editors. In this context, AI can automate tedious or time-consuming processes, while potentially inspiring artists and facilitating their creative process.

Researchers at the University of Florence, University of Modena and Reggio Emilia and University of Pisa have recently started exploring the potential of AI models in fashion design. In a pre-published paper on arXivthey introduced a new computer vision framework that could help fashion designers to visualize their designs, by showing them how they would look on the human body.

Most of the previous studies exploring the use of AI in the fashion industry focused on computational tools that can recommend clothing that is similar to that chosen by a user or models that can show customers online how clothing will look on their bodies (i.e., virtual try-on systems). This team of Italian researchers, on the other hand, set out to develop a framework that could support the work of designers, showing them how the clothes they design look like in real life, so that they can find new inspiration, identify potential problems and change their designs if necessary.

“Different from previous work which primarily focused on virtual trial clothing, we propose a multimodal conditioned fashion image editing task, guiding the creation of human-centred fashion images by following multimodal cues, such as text, human body poses, and garment sketches,” wrote Alberto Baldrati, Davide Morelli and their colleagues in their paper.

“We address this issue by proposing a new architecture based on a latent diffusion model, an approach that has not been used before in the fashion domain.”

Instead of using generative adversarial networks (GANs), an artificial neural network architecture often used to generate new text or images, the researchers decided to create a framework based on a latent diffusion model, or LDM. When they are trained in a compressed, lower-dimensional latent space, LDM can create high-quality synthetic images.

Although this promising model has been applied to many tasks requiring artificial image or video creation, it is rarely used in the context of fashion image editing. Much of the previous work in this area introduced GAN-based architectures, which produce lower quality images than LDM.

Most existing datasets for training AI models on fashion design tasks include only low-resolution images of clothing and do not include the information needed to create fashion images based on text requests and sketches. To train their model effectively, Baldrati, Morelli and their colleagues must first update existing data sets or create new ones.

“Given the lack of data sets suitable for the task, we also extended the two existing fashion data sets, namely Dress Code and VITON-HD, with multimodal annotations collected in a semi-automatic way,” Baldrati, Morelli and their colleagues explain in their paper. . . “Experimental results on this new data set demonstrate the effectiveness of our proposal, both in terms of realism and coherence with a given multimodal input.”

In initial evaluations, the model created by the research team achieved very promising results, creating realistic images of clothing on the human body inspired by human sketches and specific text cues. Their model source code and the multimodal annotations they add to the dataset will be released on GitHub soon.

In the future, these new models may be integrated into existing or new software tools for fashion designers. It can also inform the development of other AI architectures based on LDM for real-world creative applications.

“This is one of the first successful attempts to emulate the work of designers in the creative process of fashion design and could be a starting point for the adoption of the capillary diffusion model in the creative industries, control by human input,” Baldrati, Morelli and their colleagues conclude in their paper.

Science X Network 2023