NVENC vs x264 on OBS

x264 has been the go-to encoder in OBS for a long time. With the release of the NVIDIA RTX 20 series, this has all changed. If you have a 2080TI, 2080, 2070, 2060 or 1660 then you are in luck. A lot of hype has been surrounding the Turing NVENC encoder and the upgrade to OBS to make better use of it.

Long story short, Turing wins.

NVENC vs x264 vs QuickSync
NVENC vs x264 vs QuickSync

Either pick your bitrate and scan bottom-up to see the quality you can get with each encoder. Or, pick your quality and scan left-to-right to see how much more bitrate the next encoder needs. For example, the quality from Turing “MaxQ” at 3000kbps, you would get the same from x264 “Medium” at about 3400kbps.

Source & test bench

For this test, I recorded a lossless copy of some Apex Legends gameplay at 1080p60. I ran around the training ground with some guns and shot some targets. My previous test was on Overwatch to see how codecs compared in 1440p60 but most questions I get asked are about 1080p60. Apex Legends is really popular right now among streamers, therefore it seemed appropriate to give it a shot.

Turing encodes were done on an RTX 2070. QuickSync was done on QuickSync version 6 (Coffee Lake). x264 encodes were done on an i7-8086k CPU. All were done in real-time or faster, with Turing @ 3xRT, QSV @ 2xRT and x264 between 1xRT-2xRT. x264 Slow was the slowest, averaging 70fps and dipping to about 63fps for approximately 7%-10% of the frames.

VMAF

The quality on the graph is a VMAF measurement. I used VMAF model vmaf_v0.6.1.pkl in Debian FFMPEG 4.0.1-static. There’s a brief description of what VMAF is here or you can read Netflix’s own description by the head dev Zhi Li. In summary, the higher the score, the better the video looks.

In my opinion, on this test video:

  • VMAF 90+ looks as good as the best looking Apex streams I could find on Twitch today.
  • VMAF 85 looks approximately as good as the top-tier streamers doing 1080p60. I found some streamers streaming VBR between 7Mbps and 10Mbps who look a little better. Usually, the channels that look good at 6Mbps are about this quality.
  • VMAF 80 looks like the average to poor end of the 1080p60 streamers, usually those with bandwidth of 5Mbps or under.
  • VMAF 70 is definitely on the poor quality side of things. The colours are blocky and terrain loses all its fine detail. Although you can tell what’s happening and at least it’s smooth.
  • VMAF 60 is where the terrain and some models tend to block and blur into each other a lot. So much so that it can be difficult to know what a player is shooting at. At this quality, it’s definitely wise to just go down to 720p instead.

VMAF examples

Turing MaxQ 10000kbps VMAF 91.959243
Looking good with Turing MaxQ 10000kbps VMAF 91.959243
Turing MaxQ 4000 7.5s VMAF 80.587836
Blurriness becomes obvious with Turing MaxQ 4000 7.5s VMAF 80.587836
Turing MaxQ 2800kbps VMAF 71.711495
Looking blocky with Turing MaxQ 2800kbps VMAF 71.711495

Turing RTX series

Setup

There are some more test results coming that show some interesting facts about Turing. For now, I’ve gone with imitating the OBS Max Quality settings. Therefore I used this command in FFMPEG:

ffmpeg -i Original.mkv -c:v h264_nvenc -preset:v slow -profile high -level 4.2 -b:v 6000k -rc-lookahead:v 32 -an -f matroska Turing_MaxQ_6000.mkv

In this thread, dodgepong states that OBS Max Quality is the same as Quality but with 2-pass via CUDA. The FFMPEG NVENC encoder preset “Slow” is the same as “High Quality” but with 2-pass. Lookahead is set to 32 because I couldn’t find what OBS uses in the Max Quality preset so I just guessed. Spoiler alert, increasing it to 64 doesn’t change the results much anyway. I can’t figure out what the psycho visual tuning is supposed to mean, so I just left it out. Therefore the real-world use might be even better than this test. Finally, -an excludes audio from the final test files.

Results

It’s obvious from the graph that Turing NVENC beats x264 and QuickSync at any realistic bitrate. It’s not complicated and it’s not confusing. You can see similar things in another recent test bench I did with more codecs.

The drawback is that “Max Quality” in OBS uses CUDA, therefore it requires some of your GPU’s main power. If your game needs more than 90% of your GPU capacity, then I don’t recommend using it. Just use the “Quality” preset instead. It’s basically just as good. If you want more tests including this aspect, please comment below. If you can cap your frame-rate in game, then do so to secure some % of your GPU for OBS to use. Apex Legends in particular makes this tough, but each game has their own fix if you look hard enough.

Previous generations of NVENC are not as good. You can see a comparison here. Turing is the first generation that decisively beats x264.

x264

x264 presets are self-explanatory. As long as you use profile=high and level=4.2 for 1080p60 then the only difference is the preset. Interestingly “Fast” performs nearly as good as “Medium” only losing at the low bitrates but breaking even for most of the graph.

Interestingly, “Slow” loses to both Medium and Fast. I suspect there’s a combination of things in play here. First, a lot of it could just be the game. In Overwatch for example, “Slow” was much closer to Medium at higher bitrates, and beat it in lower bitrates. Second, the slower presets use older methods within H.264 of finding more quality per bit. VMAF is a much newer quality measurement, and there will be disagreements between them as to what constitutes more quality. x264 “Slow” may search harder than “Medium” for quality, but end up finding things that humans don’t really notice, and then wasting bits on those ‘improvements’.

QuickSync

Didn’t see that coming

Here’s something I bet you didn’t see coming….. OBS makes great use of the NVENC encoder but only limited use of QuickSync. The Kaby Lake, Coffee Lake and Whiskey Lake versions of QuickSync (v6) can do much more.

The rate control of LA_ICQ is excellent for local VBR recordings. However OBS only offers 3 presets for QuickSync including “Quality”, “Balanced” and “Speed”. There are 7 presets in FFMPEG, the slowest of which is called “veryslow”. QuickSync using OBS preset “Quality” is the lowest scoring encoder in this test, just take another look at the graph to see it can’t compete. However using the “veryslow” preset in FFMPEG yielded results very close to, but a little BETTER THAN x264 could get.

Here’s the FFMPEG command I used:

ffmpeg -i Original.mkv -c:v h264_qsv -preset:v veryslow -profile high -level 4.2 -b:v 6000k -rdo 1 -an -f matroska Coffee_VSlow_6000.mkv

Pls improve QuickSync access in OBS!

I didn’t expect that either, but it’s there. The “veryslow” preset is currently available in FFMPEG and can be accessed through OBS using custom output in the recording tab. You can still stream with the recording tab if you set the output URL properly.

The drawback is different to that of NVENC listed above. Since you are using an entirely different memory space it is less efficient. However it’s still capable of 1080p at 130fps so it’s fine. It’s not going to occupy any CUDA power from your main graphics card so your game can have a larger share. However OBS will STILL NEED at least some % of your main GPU’s power to render/composite the scene. Don’t make the mistake of assuming OBS needs nothing, that would be silly and logically incoherent.

The Coffee Lake (QuickSync version 6) preset “veryslow” in FFMPEG did add CPU load to my system. This didn’t go higher than 30% of a stock clocked i7-8086k. This means it may not always be useful if your game needs heavy access to your CPU. However if you can stay under 75% overall, you’re totally fine. It’s still going to use less than x264, so I would argue if you have Kaby Lake or later Intel CPU, or a Turing RTX series GPU, there aren’t many reasons to use x264 any more. We just need it to be more accessible in OBS.

Please comment below if you’d like me to expand on these results with some more data.

Agamemnus
-Agamemnus
Credits:
Featured image by Ultimatrium

Data

NVENC or x264 or QuickSync Data
NVENC or x264 or QuickSync Data

UNREAL AUSSIES ARE ALL OVER THE WEB

HELP SUPPORT
UNREAL AUSSIES

Unreal Aussies run many events over the year to help connect and build the Australian gaming community. If you are interested in helping out in any current or future planned events or wishing to offer some more ideas for us to explore - let us know!

About Us

Unreal Aussies is for passionate gamers from all walks of life. Games come and go, but the people still remain. From meetups to tournaments, hardcore teams to charity streams, Unreal Aussies core mission is to make gaming more fun as part of a community than it can ever be alone.

Forums

General

Make a Challenge

Streamers Corner

Heroes of the Storm

Overwatch

Events

Weekly Community Events

Seasonal Community Events

Community Meet Ups

Charity Events

Tournaments & Competitions

Help & FAQ

Community Rules and Guidelines

Donations & Their Use

Contact Us

FAQ