Nvidia releases stylegan2, the strongest image generator

Nvidia’s “realistic to scary” image generator has been upgraded.

Editor’s note: This article comes from WeChat public account “ 新智元 ” ( ID: AI_era), source arXiv, editor Xiao Qin.

Nvidia releases the strongest image generator StyleGAN2, which produces realistic and scary images

[Guide to Xinzhiyuan] StyleGAN is currently the most advanced high-resolution image synthesis method. The face photos it generated were once considered “realistic to scary.” Today, Nvidia researchers have released an upgraded version, StyleGAN2, which focuses on fixing feature artifacts and further improves the quality of generated images.

StyleGAN is a new image generation method released by NVIDIA last year and open sourced in February this year.

The image generated by StyleGAN is very realistic, it is to generate artificial images step by step, starting from very low resolution, all the way to high resolution (1024 × 1024). By modifying the input of each level in the network separately, it can control the visual features represented in that level, from rough features (pose, face shape) to fine details (hair color) without affecting other level.

Face generated by StyleGAN

StyleGAN is currently the most advanced high-resolution image synthesis method and has been proven to work reliably on a variety of data sets. In addition to realistic portraits, StyleGAN can also be used to generate other animals, cars and even rooms.

However, StyleGAN is not perfect. The most obvious flaw is that the generated images sometimes contain speckle-like artifacts (artifacts), and this flaw has been perfectly resolved today!

Today, NVIDIA researchers released an upgraded version of StyleGAN-StyleGAN2, which focuses on fixing artifacts and further improves the quality of generated images.

Images generated by StyleGAN2

Major improvements include:

The quality of the generated images is significantly better (higher FID score and fewer artifacts)

Propose a new method to replace progressive growing, with more perfect details such as teeth and eyes

Improved Style-mixing

Smoother interpolation (extra regularization)

Training is faster

</ </ iframe> </ p></p> <p class = "img-desc"> Nvidia StyleGAN2 </ p><br /> </ li><br /> </ ul></p> <h2> Redesigned StyleGAN image synthesis network </ h2></p> <p> The distinctive feature of StyleGAN is its unconventional generator architecture. The mapping network f not only inputs the entered latent code z ∈ Z to the beginning of the network, but also first converts it into an intermediate latent code w ∈ W. Affine transforms are then generatedStyles, which control the layers of the synthesis network g through adaptive instance normalization (AdaIN). </ p></p> <p> In this study, we focus all our analysis on W, because from the perspective of the synthetic network, W is the relevant latent space. </ p></p> <p> Many people have noticed feature artifacts in the images generated by StyleGAN. This study identifies two causes of these artifacts and describes how to eliminate them by changing architecture and training methods. </ p></p> <p> <img alt = "Nvidia releases the strongest image generator StyleGAN2, which generates realistic and scary images" src = "https://img.36krcdn.com/20191213/v2_86a6cfa006e74259a0e25209ca8a0f2b_img_000"> </ p></p> <p> <img alt = "Nvidia releases the strongest image generator StyleGAN2, generating realistic and scary images" src = "https://img.36krcdn.com/20191213/v2_3946c6b39b2e456b8a818f386364d7d7_img_000" data-img-size-val = "720,135 "> </ p></p> <p class = "img-desc"> Figure 1: Instance normalization will cause speckle-like artifacts in images generated by StyleGAN </ p></p> <p> First, we studied the origin of common speckled artifacts and found that the generator created them to circumvent design flaws in their architecture. We redesigned the normalization used in the generator, which removed artifacts. </ p></p> <p> Second, we analyzed artifacts related to progressive growing. Progressive growth has been very successful in stabilizing high-resolution GAN training. </ p></p> <p> We propose an alternative design that can achieve the same goal-focus on low-resolution images at the beginning of training, and then gradually shift attention to higher and higher resolutions-in The network topology is not changed during the training process. This new design also allows us to reason about the effective resolution of the generated images, with results that are lower than expected, motivating us to design larger models. </ p></p> <p> <img alt = "Nvidia releases the strongest image generator StyleGAN2, which produces realistic and scary images" src = "https://img.36krcdn.com/20191213/v2_f00092cf25fe432aa01348e44ca64afe_img_000 "data-img-size-val =" 720,301 "> </ p></p> <p class = "img-desc"> Figure 2: Redesigned StyleGAN image synthesis network </ p></p> <p> As shown in Figure 2, (a) is the original StyleGAN, where A represents the affine transformation learned from W, resulting in a style; (b) shows the details of the original StyleGAN architecture. Here, we decompose AdaIN into an explicit normalization and then modulation mode, and operate on the mean and standard deviation of each feature map. </ p></p> <p> We also annotated the weights (w), biases (b), and constant inputs (c) of the learning, and redrawn the gray boxes so that each box activates a style. The activation function (leaky ReLU) is always applied immediately after adding a bias. As shown in (c), we made several changes to the original architecture, including removing some redundant operations at the beginning, moving the addition of b and B outside the active area of style, and adjusting only each feature map Standard deviation. </ p></p> <p> (d) is a modified architecture that allows us to replace instance normalization with the “demodulation” operation. We apply the demodulation operation to the weights associated with each convolutional layer. </ p></p> <p> <img alt = "Nvidia releases the strongest image generator StyleGAN2, which produces realistic and scary images" src = "https://img.36krcdn.com/20191213/v2_8bbbf09edbb940eaa1b40b4cf2cfd336_img_000" data-img-size-val = "720,662 "> </ p></p> <p class = "img-desc"> Figure 3: Demodulation instead of instance normalization can remove feature artifacts in images and activations. </ p></p> <p> As shown in Figure 3, the redesigned StyleGAN2 architecture eliminates feature artifacts while retaining full controllability. </ p></p> <p> <img alt = "Nvidia releases the strongest image generator StyleGAN2, which produces images that are realistic and scary" class = "rich_pages __bg_gif" data- data-src = "https://mmbiz.qpic.cn/mmbiz_gif/UicQ7HgWiaUb04ic3c8P79z9U6y0RqHJJWWicrWoJaWwJaWwJaWwJaWwOjaxSMbR2DQVtxQ / 640? wx_fmt = gif & tp = webp & wxfrom = 5 & wx_lazy = 1 "data-type =" gif "_width =" 662px "src =" https://mmbiz.qpic.cn/mmbiz_gif/UicQ7HgWiaUb04ic3c8P79z9U6y0RqHJpaWoLIjay9NWDw059jvMzianAxHDSnvhTb206G888J8HsicxSMbR2DQVtxQ/640?wx_fmt=gif&tp=webp&wxfrom = 5 & wx_lazy = 1 "width =" 300 "height =" 150 "> </ p></p> <p> <img alt = "Nvidia releases the strongest image generator StyleGAN2, generating realistic and scary images" src = "https://img.36krcdn.com/20191213/v2_d0903a57dd664604861df0fa707d051a_img_000" data-img-size-val = "639,357 "> </ p></p> <p> Quantitative analysis of the image quality generated by GAN is still a challenging subject. Frechet inception distance (FID) measures the difference between the two distribution densities in the high-dimensional feature space of the InceptionV3 classifier. Precision and Recall (P & R) provide additional visibility by explicitly quantifying the percentage of images generated that are similar to the training data and the percentage of training data that can be generated. We use these metrics to quantify the improvements of StyleGAN2. </ p></p> <p> <img alt = "Nvidia releases the strongest image generator StyleGAN2, which produces realistic and scary images" src = "https://img.36krcdn.com/20191213/v2_cfd8bcfcf8ad46eead66289f368f7f25_img_000" data-img-size-val = "720,205 "> </ p></p> <p class = "img-desc"> Table 1: Main results </ p></p> <p> FID is basically unaffected (Table 1, rows A, B), but there is a significant change, from precision to FID. </ p></p> <p> FID and P & R are both based on classifier networks. Recent research has shown that classifier networks focus on textures rather than shapes. Therefore, these indicators cannot accurately represent the quality of images.All aspects. We use the Perceptual Path Length (PPL) index as a method to estimate the quality of potential spatial interpolation, which is related to the consistency and stability of the shape. </ p></p> <p> Based on this, we regularize the synthetic network to support smooth mapping and obtain significant quality improvements. To offset the computational overhead, we also recommend reducing the frequency of performing all regularizations, as doing so will not affect efficiency. </ p></p> <p> <img alt = "Nvidia releases the strongest image generator StyleGAN2, generating realistic and scary images" src = "https://img.36krcdn.com/20191213/v2_996be78a889142808413ddc5c0a9fdc8_img_000" data-img-size-val = "720,449 "> </ p></p> <p class = "img-desc"> Figure 4 </ p></p> <p> <img alt = "Nvidia releases the strongest image generator StyleGAN2, generating realistic and scary images" src = "https://img.36krcdn.com/20191213/v2_af6792e03ce74da99d5c3d204b14c16c_img_000" data-img-size-val = "720,452 "> </ p></p> <p class = "img-desc"> Figure 5 </ p></p> <h2> The new method replaces Progressive growing, the details are more perfect </ h2></p> <p> Progressive growing has proven to be very successful in stabilizing high-resolution image synthesis, but it produces its own feature artifacts. </ p></p> <p> The key issue is that the generators that appear to grow progressively have a strong position preference in detail, for example, when features such as teeth or eyes move smoothly on the image, they may stay in their original position and then jump Go to the next preferred location. </ p></p> <p> <img alt = "Nvidia releases the strongest image generator StyleGAN2, which produces realistic and scary images" src = "https://img.36krcdn.com/20191213/v2_a79f03f181044093b33921a1afcd56a1_img_000" data-img-size-val = "639,357 "> </ p></p> <p> Figure 6 shows a related artifact. We believe that the problem is that in progressive growing, each resolution temporarily serves as the output resolution, forcing it to produce the maximum frequency detail, which causes the trained network to be too high in the middle layer frequency, sacrificing translation invariance. </ p></p> <p> <img alt ="Nvidia releases the strongest image generator StyleGAN2, which generates realistic and scary images" src = "https://img.36krcdn.com/20191213/v2_b9dff45bd24d43a9848ff6a2e9a6a559_img_000" data-img-size-val = "720,361"> </ p></p> <p class = "img-desc"> Figure 6: Progressive growing results in a “phase” artifact. In this example, the teeth did not follow the posture change, the face turned to one side, and the teeth were still facing straight ahead, as shown by the blue line. </ p></p> <p> In order to solve these problems, we propose an alternative method that eliminates the defects while retaining the advantages of progressive growing. </ p></p> <p> Although StyleGAN uses simple feedforward design in generators (synthetic networks) and discriminators, there is still a lot of work dedicated to researching better network architectures. In particular, skip connections [34, 22], residual networks [17, 16, 31] and hierarchical methods [7, 46, 47], these methods have proven to be very successful. Therefore, we decided to re-evaluate StyleGAN’s network design and look for an architecture that can generate high-quality images without the need for progressive growing. </ p></p> <p> <img alt = "Nvidia releases the strongest image generator StyleGAN2, which generates realistic and scary images" src = "https://img.36krcdn.com/20191213/v2_c38c8d31b69c4d7aaeb094ef025dfbd3_img_000" data-img-size-val = "720,709 "> </ p></p> <p class = "img-desc"> Figure 7: Three generators (above the dotted line) and discriminator architecture. </ p></p> <p> Figure 7a shows MSG-GAN [22], which uses multiple skip connections to connect the matching resolution of the generator and discriminator. </ p></p> <p> In Figure 7b, we simplify this design by upsampling and summing the RGB outputs corresponding to different resolutions. In the discriminator, we also provide down-sampled images to each resolution block of the discriminator. We use bilinear filtering in all upsampling and downsampling operations. </ p></p> <p> In Figure 7c, we have further modified the design to use residual connections. This design is similar to LAPGAN [7]. </ p></p> <p> Table 2 compares the three generator and discriminator architectures: for StyleGAN, skip connectioThe original feedforward network of ns and residual network, they are both trained, but not using progressive growing. </ p></p> <p> <img alt = "Nvidia releases the strongest image generator StyleGAN2, which produces realistic and scary images" src = "https://img.36krcdn.com/20191213/v2_84ae404b4ad94df5b45a9101c9dd41a6_img_000" data-img-size-val = "720,324 "> </ p></p> <p class = "img-desc"> Table 2: Comparison of generator and discriminator structure without progressive growing. </ p></p> <p> For these 9 combinations, each provides FID and PPL results. We can see two big trends: the skip connections of the generator greatly improve the PPL of all configurations, and the residual discriminator network obviously benefits FID. </ p></p> <p> StyleGAN2 uses a skip generator and a residual discriminator, but does not use progressive growing. This corresponds to configuration E in Table 1, as can be seen from the table, switching to this setting significantly improves FID and PPL. </ p></p> <p> Finally, we found that using the new path length regularization generator to project the image onto the latent space W is significantly better than the original StyleGAN. </ p></p> <p> Paper address: </ p></p> <p> https://arxiv.org/pdf/1912.04958.pdf </ p></p> <p> Code and trained models are open source: </ p></p> <p> https://github.com/NVlabs/stylegan2 </ p></p> </div> <p class="post-tags"></p> </article> <script src='//unpkg.com/valine/dist/Valine.min.js'></script> <div id="vcomments"></div> <script> new Valine({ el: '#vcomments', appId: 'nFXLD31mFjDHlrEDx3OQWstS-gzGzoHsz', appKey: 'GhTuxYTHv3w2aqSJ0urXJov5', }) </script> <style> #veditor { background-image: url(https://image.bestzuo.cn/images/20200708160947.gif!getwebp); background-size: contain; background-repeat: no-repeat; background-position: right; background-color: rgba(255, 255, 255, 0); resize: vertical } img.vimg { transition: all 1s /* 旋转时间为 1s */ } img.vimg:hover { transform: rotate(360deg); -webkit-transform: rotate(360deg); -moz-transform: rotate(360deg); -o-transform: rotate(360deg); -ms-transform: rotate(360deg); } #vcomments .vcards .vcard { padding: 15px 20px 0 20px; border-radius: 10px; margin-bottom: 15px; box-shadow: 0 0 4px 1px rgba(0, 0, 0, .12); transition: all .3s } #vcomments .vcards .vcard:hover { box-shadow: 0 0 8px 3px rgba(0, 0, 0, .12) } #vcomments .vcards .vcard .vh .vcard { border: none; box-shadow: none; } </style> </main>  <aside class="sidebar col-md-3 col-sm-4 col-xs-12" style="float:right"> <aside id="polylang-2" class="widget widget_polylang"><ul> <li class="lang-item lang-item-17 lang-item-zh no-translation lang-item-first"><a lang="zh-CN" hreflang="zh-CN" href="https://www.ww01.net/"><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAALCAIAAAD5gJpuAAAABGdBTUEAAK/INwWK6QAAABl0RVh0U29mdHdhcmUAQWRvYmUgSW1hZ2VSZWFkeXHJZTwAAAFqSURBVHjaYrzOwPAPjJgYQEDAleHVbhADIvgHLPgHiQ0QQCxAlkR9NW8sw+cV/1gV/7Gb/hV4+vfzhj8Mv/78//Pn/+/f/8AkhH1t0yaAAAJp4I37zyz2lDfu79uqv/++/WYz+cuq/vvLxt8gdb+A5K9/v34B2SyyskBLAAII5JAva/7/+/z367a/f3/8ZuT9+//Pr78vQUrB6n4CSSj6/RuoASCAWEDO/fD3ddEfhv9/OE3/sKj8/n7k9/fDQNUIs/+DVf8HawAIIJCT/v38C3Hr95N/GDh/f94AVvT7N8RUBpjxQAVADQABBNLw/y/Ifwy/f/399ufTOpDBEPf8g5sN0QBEDAwAAQTWABEChgOSA9BVA00E2wAQQCANQBbEif/AzoCqgLkbbBYwWP/+//sXqBYggFhAkfL7D7OkJFCOCSj65zfUeFjwg8z++/ffX5AGoGKAAGI8jhSRyIw/SJH9D4aAYQoQYAA6rnMw1jU2vQAAAABJRU5ErkJggg==" title="中文 (中国) (zh_CN)" alt="中文 (中国)" width="16" height="11" style="width: 16px; height: 11px;" /><span style="margin-left:0.3em;">中文 (中国)</span></a></li> <li class="lang-item lang-item-29 lang-item-en current-lang"><a lang="en-AU" hreflang="en-AU" href="https://www.ww01.net/en/archives/34133"><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAALCAIAAAD5gJpuAAAABGdBTUEAAK/INwWK6QAAABl0RVh0U29mdHdhcmUAQWRvYmUgSW1hZ2VSZWFkeXHJZTwAAAIzSURBVHjaYjxe3nyAQ6Vyx7veD+tY/v3L/+dWKvXIyUTOc9Ybhq8fGBj+AJFnssv2uZsYGN4xMPwCCAAxAM7/AUtNjZ95PPsXHfjPzf/49wEfIhAVELzd+MzU5v38/vf6+1tNLQQEAd7j77fB3KOMjwIAMQDO/wHNCQkZhYYD7Or78vL++fkFDAv5/gH29/qJCD3w/AH6+PodGQ9GOyGJm8UgHRGrko8CiOmQjg+Ttj6HluZfYVEGWQUuM7Pfsop3ZfR+/Pnv56jCwMBw4/5roOrKdBsJYW4Ghm8AAcT0ISSJQVh4wz+F5zziL1gF1gmZMevofuQTcbZTlRXnLUyy+P7jd4SXFisLo6uVIgPDD4AAADEAzv8DLAEa6w0YwN/4+/b43/UCuNbx2/QDEP73rcbkJSIUq7fV6ev07O/3EQ8IqLXU3NDDAgAxAM7/A8veKS1ELvXw9N77Cd76BwT8+ujr9M/o+/3//8bN4+nt9P///1dLK6Cu0QkIBd7RvgKICRRwf/79/vMvyF6pNsX81++/f/7+Y/j39/evP//+/fv/9//Pn3965hz7+Onbv79/gYoBAgio4devP0Dj/psbSMtJ8gW4afz89fvX7z9g9BcYrNISfOWpVj9///379z9QA0AAsQA1AE0S4ufceeSuvprowZMPZCX4fv4Eyv778+f/9x+/ihLNtZTFfv76u2bnNaCnAQKIkYHBFxydP4A6kdAfZK6qY9nt/U0MDP+AoQwQYAAK+BukFnf4xAAAAABJRU5ErkJggg==" title="English (en_AU)" alt="English" width="16" height="11" style="width: 16px; height: 11px;" /><span style="margin-left:0.3em;">English</span></a></li> </ul> </aside><aside id="custom_html-2" class="widget_text widget widget_custom_html"><div class="textwidget custom-html-widget"><script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>  <ins class="adsbygoogle" style="display:block" data-ad-client="ca-pub-1204574674466407" data-ad-slot="3371268877" data-ad-format="auto" data-full-width-responsive="true"></ins> <script> (adsbygoogle = window.adsbygoogle || []).push({}); </script>  <script src="https://libs.baidu.com/jquery/2.0.0/jquery.min.js"></script> <script> $(function () { $(".search-form").submit(function(e){ e.preventDefault(); window.open("https://www.google.com/search?q=site:ww01.net%20"+$(".search-field").val()); }); }) </script></div></aside> <aside id="recent-posts-4" class="widget widget_recent_entries"> <h3 class="widget-title">Recent Posts</h3> <ul> <li> <a href="https://www.ww01.net/en/archives/174760">CISA plans to establish an iron ore Working Committee: strengthen the research on major issues related to iron ore</a> <span class="post-date">2022年8月1日</span> </li> <li> <a href="https://www.ww01.net/en/archives/174752">Hangzhou: when a family with three children purchases their first home, the first loan limit of the provident fund can be increased by up to 20%</a> <span class="post-date">2022年8月1日</span> </li> <li> <a href="https://www.ww01.net/en/archives/174747">COMAC announced that the domestic large aircraft C919 has completed its certification test flight</a> <span class="post-date">2022年8月1日</span> </li> <li> <a href="https://www.ww01.net/en/archives/174745">Investigation ｜ long term rent in hotels: it is a “new consumption trend”, but also an industry self-help</a> <span class="post-date">2022年8月1日</span> </li> <li> <a href="https://www.ww01.net/en/archives/174743">Liquidity outlook in August: can the overnight capital interest rate of about 1% continue</a> <span class="post-date">2022年8月1日</span> </li> </ul> </aside><aside id="text-8" class="widget widget_text"><h3 class="widget-title">contact us</h3> <div class="textwidget"><p>email：sparrww@vip.163.com</p> </div> </aside> </aside> </div>  </div>   <footer> <div class="footer-top"> <div class="container"> <div class="row"> <div class="col-sm-4"> <aside id="linkcat-6" class="widget widget_links"><h3 class="widget-title">Links</h3> <ul class='xoxo blogroll'> <li><a href="http://www.domeet.net/" rel="me noopener" target="_blank">domeet</a></li> <li><a href="https://www.kevinbobo.top/" rel="noopener" target="_blank">kevinBobo</a></li> <li><a href="https://www.yvyong.com/" rel="noopener" target="_blank">yvyong BLOG</a></li> </ul> </aside> </div> <div class="col-sm-4 md-footer-cl"> </div> <div class="col-sm-4"> </div> </div> </div> </div> <div class="blog-footer blog-footer-widgets"> <div class="container"> <p class="site-credits">© 2022 - Powered by <a href="https://jumpstarter.io/frenchtastic/coeur">Coeur</a></p> <p class="back-to-top"><a href="#"><i class="fa fa-angle-up"></i> Back to top<a></p> </div> </div> </footer> </div>  <script id="imwpf-this-page"> var g_this_page = { "type": "post", "id": 34133, "terms": [{"id":37,"tax":"category"},{"id":29,"tax":"language"}] };console.log(g_this_page); </script><div class="imwpcache_status_ok"></div><div style="display:none"> <script src="https://s5.cnzz.com/z_stat.php?id=1275404939&web_id=1275404939" language="JavaScript"></script> <script> (function(){ var src = "https://jspassport.ssl.qhimg.com/11.0.1.js?d182b3f28525f2db83acfaaf6e696dba"; document.write('<script src="' + src + '" id="sozz"><\/script>'); })(); </script> </div> <script>console.log('命中缓存\n缓存页加载耗时: 0.004s\n缓存页生成时间: 2022-08-06 04:47:14')</script></body> </html>

domeet webmaster