View source for Debug NVIDIA Jetson nvv4l2av1enc VPR Violation (EMEM decode error)

=Debug NVIDIA Jetson nvv4l2av1enc VPR Violation (EMEM decode error)=

== Issue description==
When using NVIDIA's encoder AV1 GStreamer element nvv4l2av1enc with a non-standard resolution, such as 2712x1538, it may trigger unauthorized access to protected memory issues. You may see logs from the kernel reporting EMEM address decode error (EMEM decode error), route sanity error, and VPR violation issues.

==Replicate issue==
For replicating this issue do the following.  
 
To see kernel debug messages run: 
<syntaxhighlight lang="bash">
dmesg -wH
</syntaxhighlight>

On another console run the following GStreamer pipeline: 

<syntaxhighlight lang="bash">
gst-launch-1.0 videotestsrc is-live=true ! 'video/x-raw, width=2712, height=1538, framerate=90/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=2712, height=1538, framerate=90/1' ! queue ! 'video/x-raw(memory:NVMM), width=2712, height=1538, framerate=90/1' ! nvv4l2av1enc name=encoder1 bitrate=22000000 maxperf-enable=1 insert-seq-hdr=true idrinterval=15 iframeinterval=256 ! fakesink
</syntaxhighlight>

'''Important''': The issue does not occur every time. For getting the error cancel and execute the pipeline repeatedly until you see a log from the '''dmesg''' the looks like the following: 

<syntaxhighlight lang="bash">
arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ffded8000, fsynr=0x6f0013, cbfrsynra=0x835, cb=2
[  +0.000462] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ffded8000, fsynr=0x6f0003, cbfrsynra=0x835, cb=2
[  +0.000436] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ffded8000, fsynr=0x6f0003, cbfrsynra=0x835, cb=2
[  +0.000423] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ffded8000, fsynr=0x6f0013, cbfrsynra=0x835, cb=2
[  +0.000406] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ffded8000, fsynr=0x6f0013, cbfrsynra=0x835, cb=2
[  +0.000407] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ffded8000, fsynr=0x6f0003, cbfrsynra=0x835, cb=2
[  +0.000400] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ffded8000, fsynr=0x6f0003, cbfrsynra=0x835, cb=2
[  +0.000396] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ffded8000, fsynr=0x6f0003, cbfrsynra=0x835, cb=2
[  +0.000865] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ffded8000, fsynr=0x6f0003, cbfrsynra=0x835, cb=2
[  +0.001541] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x3ffded8000, fsynr=0x6f0013, cbfrsynra=0x835, cb=2
[  +0.008255] tegra-mc 2c00000.memory-controller: nvencsrd: secure read @0x000000ffffffff00: EMEM address decode error (EMEM decode error)
[  +0.009824] tegra-mc 2c00000.memory-controller: nvencswr: secure write @0x00000003ffffff00: VPR violation ((null))
[  +0.009101] tegra-mc 2c00000.memory-controller: nvencswr: secure write @0x00000003ffffff00: Route Sanity error ((null))
[  +0.016422] tegra-mc 2c00000.memory-controller: nvencsrd: secure read @0x000000ffffffff00: EMEM address decode error (EMEM decode error)
[  +0.004142] tegra-mc 2c00000.memory-controller: nvencswr: secure write @0x00000003ffffff00: VPR violation ((null))
[  +0.009097] tegra-mc 2c00000.memory-controller: nvencswr: secure write @0x00000003ffffff00: Route Sanity error ((null))
[  +0.019064] tegra-mc 2c00000.memory-controller: unknown: secure read @0x000000ffffffff00: EMEM address decode error (EMEM decode error)
[  +0.001501] tegra-mc 2c00000.memory-controller: nvencswr: secure write @0x00000003ffffff00: VPR violation ((null))
[  +0.009187] tegra-mc 2c00000.memory-controller: nvencswr: secure write @0x00000003ffffff00: Route Sanity error ((null))
[  +0.015548] tegra-mc 2c00000.memory-controller: nvencsrd: secure read @0x000000ffffffff00: EMEM address decode error (EMEM decode error)
</syntaxhighlight>

== What may be causing this issue? ==

NVIDIA's nvv4l2av1enc encoder uses NVENC hardware unit in Jetson platforms for hardware-accelerated processing on encoding. NVENC has strict constraints for resolution alignment, max resolution limits, frame rate limits and memory requirements. If any of this constraints is not met the encoder will silently fail and report the issue only in the kernel logs and not on GStreamer. In our case, resolution '''2712x1538''' is a non-standard resolution and it seems to trigger the resolution alignment constraint on the hardware causing the unauthorized access to protected memory issue(EMEM address decode error, route sanity error, and VPR violation). Even though NVIDIA does not have explicitly documented which alignment is required; through some extensive testing we have come up to the conclusion that a 64 pixel width alignment is necessary. What does this mean? This means that the resolution's width should be divisible by 64. In our case:

<syntaxhighlight lang="bash">
2712 % 64 = 24 --> Not aligned
</syntaxhighlight>

==Possible Solutions==

===Solution 1: Use an image resolution that aligns with 64 pixels===

* Use a lower resolution close to the original desired resolution

In this case the closest following resolution would be: '''2688 x 1526'''. For this new resolution the width respects the 64 pixel alignment required by the NVENC. 
<syntaxhighlight lang="bash">
2688 % 64 = 0 --> Aligned 
</syntaxhighlight>

Pipeline example: 
<syntaxhighlight lang="bash">
gst-launch-1.0 videotestsrc is-live=1 ! 'video/x-raw, width=2688, height=1526, framerate=90/1' ! nvvidconv ! queue ! nvv4l2av1enc name=encoder1 bitrate=22000000 maxperf-enable=1 insert-seq-hdr=true idrinterval=15 iframeinterval=256 ! fakesink
</syntaxhighlight>

* Use original resolution for capturing and add padding with black pixels

This approach allows to adjust the original resolution to a resolution that fits the 64 pixel width alignment required by the NVENC hardware unit to avoid getting the VPR violation and EMEM address decode errors. For that, you will need to increment the width resolution to the next number divisible by 64. In this case the next number would be: '''2752'''. 

<syntaxhighlight lang="bash">
2752 % 64 = 0 --> Aligned 
</syntaxhighlight>

The capturing process plus adding the padding would be the following: 

<syntaxhighlight lang="bash">
Camera capture @ 2712x1536 --> Add horizontal padding of 40 pixels (2752-2712) --> AV1 encoder
</syntaxhighlight>

With this approach we are taking into account the 64 pixel width alignment for the NVENC hardware unit; solving the issue.

==Experimental section==
This section includes the exploration of ideas for solving the width alignment encoder issue. 

=== Modify TEGRA_WIDTH_ALIGNMENT on tegra_camera_core.h file ===
The '''tegra_camera_core.h''' is a header file for the NVIDIA Tegra camera core driver. It includes constants, enums and structures used by the Tegra video input device driver in the Linux Kernel. In this file you can see the following line: 
<syntaxhighlight lang="c++">
/* Width alignment */
#define TEGRA_WIDTH_ALIGNMENT	1
</syntaxhighlight>

* '''Proposal''': Modify TEGRA_WIDTH_ALIGNMENT to 64 so that the encoder may process non-standard image resolutions.

* '''Veredict''': Not possible. 

The '''TEGRA_WIDTH_ALIGNMENT''' sets the required alignment for image width, in this case, to 1 pixel. This means the image width does not need to be aligned to any larger boundary. Essentially letting the camera core driver to capture with any resolution. In our proposal we suggested changing it to 64. The effect on this would be that for the camera driver to be able to capture, the requested resolution's width needs to be divisible by 64. Hence, the '''2712x1538''' resolution won't work since 2712 is not divisible by 64.

* '''Conclusion''': Modifying '''TEGRA_WIDTH_ALIGNMENT''' will only affect the camera driver instead of the encoder, so modifying it won't represent any change in the encoder behavior when receiving a width not aligned to 64 pixels, it would actually break the pipeline at the capture stage rather than on the encoding stage.