A neural network may be needed between the UI design and the front-end engineers.
Recently, Uizard Technologies, a startup based in Copenhagen, developed a system called "pix2code". Using deep learning, the system can directly output the corresponding code based on the input graphical user interface (GUI) screenshots, eliminating the need for the front end to manually write code.
At present, the accuracy of the code generated by pix2code has reached 77%, and these codes are compatible with Android, iOS and Web interface platforms.
How amazing is it? Take a look at the video below to know.
Tony Beltramelli, founder of UIzard, said that in order to come up with this neural network, the R&D team had to overcome three major difficulties:
The first is the computer vision level—computers cannot automatically recognize, understand, and determine the location and characteristics of a given object or scene;
The second is the language level – they need to teach the neural network to understand the text so that it can create accurate samples;
Finally, neural networks need to understand the association between code, text, and corresponding images.
Beltramelli also said that in the future, they may further improve pix2code with the generation of confrontation networks (GANs). GANs have proven their advantages in generating sequences and images, but because research in this area is still at a relatively preliminary stage, it takes a lot of time to train neural networks.
Attached paper address:
Https://arxiv.org/abs/1705.07962
GitHub address:
Https://github.com/tonybeltramelli/pix2code
In addition, on GitHub, Beltramelli answers some common questions about pix2code. Qubits compile this part as follows:
Q: When will the data set be opened?
A: We have submitted the paper to NIPS this year. After they give the result of adoption or rejection, the data set will be open, which is about September. At that time we will provide a GUI screenshot of the dataset, related DSL code, and three object codes for iOS, Android, and the web interface.
Q: When is the source code open?
A: Originally, as written in the paper, we have no plans for open source. But I didn't expect this project to attract so much attention, so we decided to open the pix2code implementation code and dataset together in the paper.
Q: Does pix2code support other target platforms/language?
A: No, pix2code is just a research project that will maintain the state described in the paper. This project is really just a small showcase of what we did at Uizard Technologies. Of course, we welcome you to fork and experiment on other target platforms/language.
Q: Can I use pix2code in my front-end project?
A: No, pix2code is just an experimental project, and it is not currently available for you to apply in a specific case. But we are working hard to make it commercial.
Q: How is the performance of the model measured?
A: The exact or erroneous results reported in the paper are obtained at the DSL level by comparing the generated token with the expected token. If there is any difference in length between the two, it will also be considered an error.
Q: How long does it take to train this model?
A: On a NVIDIA Tesla K80 GPU, it takes less than 5 hours to optimize the 109 * 10^6 parameters included in a data set. So if you want to train this model on three target platforms, it will take 15 hours.
Q: Am I a front-end developer? Am I going to be unemployed soon? (I have asked this question very sincerely many times...)
A: AI doesn't replace front-end engineers so quickly.
Even assuming a mature pix2code version already exists, the code generated on each different platform/language can achieve 100% accuracy, and a good front end still requires logic, interaction, advanced graphics and animation, and all other users. The things I love.
The purpose of doing this is to fill the gap between UI/UX designers and front-end developers, rather than replacing them. We hope to allow designers to create better, while allowing developers to spend more of their time on those core features.
We believe that the future AI will work with humans, not humans.
CCTV Products,CCTV Parts, Video Surveillance Accessories, CCTV Camera Accessories
Chinasky Electronics Co., Ltd. , https://www.cctv-products.com