Encoder Comparison NVENC x264 Quicksync QSV VP9 and AV1

NVENC part 1

History

In 2012, NVIDIA released their own hardware encoder. One advantage was for users of non-QuickSync CPUs, since they could now have hardware encoding in the graphics card. As of 2019 with the Turing RTX 2080, 2070 and 2060 cards, compression has improved substantially. NVENC Part 2 will examine Turing chipsets separately.

To the best of my knowledge, no NVENC implementation offers hardware accelerated VP9 encoding. Kepler and Maxwell first generation had only H.264 encoding. Maxwell second generation and Pascal also have H.265 encoding capability, albeit without B-Frames. Turing can do H.265 with B-Frames.

It is as easy to setup basic NVENC use in OBS as it is to setup QuickSync. Pascal and Turing support encoding 10-bit HEVC but this is not easily accessible in neither OBS nor Handbrake. At least not at the time of writing this. Note that the Maxwell card used in this testing was a GTX 980TI, which is second generation Maxwell. Also note that Maxwell 1st generation is NVENC 2nd generation, and Maxwell 2nd generation is NVENC 3rd generation.

Results

NVENC 1 VMAF Scores on 2560x1440 60 fps
NVENC 1 VMAF Scores on 2560×1440 60 fps

Things to note for NVENC:

  • Kepler at 4 Mbps produced an extraordinarily poor quality video. The results exclude anything that scores below 60 on VMAF.
  • x265 Medium produces equal quality to Kepler at only half the bitrate.
  • Maxwell and Pascal NVENC preset “hq” is easy to use in OBS. However the “slow” preset is not readily available in OBS at the time of writing. It is possible to use it through the recording tab and custom options but is beyond most streamers’ ability. If you want a tutorial on how to do it, ask in the comments.
  • H.265/HEVC at the same bitrate is considerably better than the H.264/AVC version.

Special note for NVENC Part 1

For the rest of this section, keep in mind a few things. This test video is Overwatch gameplay in 2560×1440 at 60 fps. I chose my bitrates carefully to represent bandwidth starvation and abundance. In my opinion the mid-range of the 6Mbps tests look approximately as clear as most Twitch streamers’ live streams. The top scores look far better, and the bottom scores look far worse. The reasoning behind this, is so people can look at their available options and make a judgement call.

If you have limited bandwidth at 4Mbps, then perhaps you should limit your resolution to 1080p. If you operate a server farm that can encode AV1 no problem, and your source video is 2160p or 4k, AND 8Mbps is no issue, then perhaps you shouldn’t bother downscaling to 1440p since you would be over-spending bits on the smaller resolution when you could have had the larger one.

Essentially, if you have bandwidth to spare, then you might be able to afford a larger resolution or higher fps. If you have limited bandwidth then a lower resolution might be better. In my opinion, 6Mbps is the sweet spot for 1440p60. Average encoders do reasonably well. Poor quality encoders or old codecs should consider a lower res. Conversely, the high quality encoders gain very little with more bandwidth past this.

Conclusions for NVENC Part 1

Kepler

The worst encoder tested in this article is Kepler NVENC. Essentially, if you have any other option, you should use that instead. If you have a powerful CPU, use x264 or another codec entirely. If you have a CPU with internal graphics, you probably have QuickSync available. Kepler NVENC H.264 is a last resort. For offline video, see the second point above regarding x265 Medium producing equal quality at half the bitrate of Kepler.

Decisions decisions…

At 6Mbps x264 Faster beats Maxwell and Pascal NVENC in H.264. At 4 and 8 Mbps x264 needs to use Medium to win in live streaming. This means that depending on your streaming bandwidth capacity, your mileage may vary. If your situation limits you to 4Mbps or allows for 8Mbps then keep in mind the above notes. For live streaming, if your bandwidth means that Maxwell NVENC is better than x264 Faster, then consider lowering your resolution.

NVENC on Pascal or Maxwell in HEVC is a decent solution if you are encoding offline. If you have tonnes of video to process and bandwidth is not a major concern then this is for you. The H.265/HEVC results are competitive with x265 in bitrate abundance. Maxwell 2nd gen (NVENC 3rd gen) is about 2.5x faster than a stock i7-8086k at 100% CPU usage. Pascal (NVENC 3rd gen) is double that again at 5x faster. Also remember that it’s easy to use the “slow” preset in Handbrake for NVENC, for that extra boost over “hq”. However if you want to save on file size then x265 wins.

Final thoughts on Maxwell and Pascal H.264 AVC

Finally, your situation might call for H.264/AVC for reasons of compatibility. In this case, QuickSync may be the better option. During these tests, at 6Mpbs and 8Mbps, QuickSync H.264/AVC encoding outperforms all NVENC except Turing. At 4Mbps QuickSync takes a dive and Pascal/Maxwell make a comeback, but look at the VMAF scores. At this bitrate x264 clearly wins and is only 1/3rd the speed of Pascal on my i7-8086k CPU. Even then, the VMAF score is still under 70. If you desperately need H.264 but don’t have more than 4Mbps for 1440p60 you are sacrificing a lot of quality.

UNREAL AUSSIES ARE ALL OVER THE WEB

HELP SUPPORT
UNREAL AUSSIES

Unreal Aussies run many events over the year to help connect and build the Australian gaming community. If you are interested in helping out in any current or future planned events or wishing to offer some more ideas for us to explore - let us know!

About Us

Unreal Aussies is for passionate gamers from all walks of life. Games come and go, but the people still remain. From meetups to tournaments, hardcore teams to charity streams, Unreal Aussies core mission is to make gaming more fun as part of a community than it can ever be alone.