Unable to convert Side Aware Boundary Localization: [checkSanity.cpp::checkSanity::106] Error Code 2: Internal Error

[MeteoriteWeny]

Describe the bug
First, thank you very much for your contribution. When I tried to convert the custom Side-Aware Boundary Localization model, I encountered [checkSanity.cpp::checkSanity::106] Error Code 2: Internal Error. By the way, the custom model works fine when directly use init_detector, inference_detector of mmdet.api.inference. I also found that the implement of SABL in mmdetection 2.12.0 has change a lot compared to 2.19.1. Is the newer version SABL no longer supported? I am beginner of tensorrt and confused with the TRT Error. Could you tell me where the problem exists? Thank you!

Error Output

[03/19/2022-16:15:55] [TRT] [I] [MemUsageChange] Init CUDA: CPU +296, GPU +0, now: CPU 3107, GPU 2883 (MiB)
[03/19/2022-16:15:56] [TRT] [I] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 3124 MiB, GPU 2883 MiB
[03/19/2022-16:16:01] [TRT] [I] [MemUsageSnapshot] End constructing builder kernel library: CPU 3189 MiB, GPU 2883 MiB
[03/19/2022-16:16:20] [TRT] [E] 2: [checkSanity.cpp::checkSanity::106] Error Code 2: Internal Error (Assertion  regionNames.find(r->name) == regionNames.end() failed. Found duplicate region name  unsqueeze_tensor_after_(Unnamed Layer* 931) [ElementWise]_(Unnamed Layer* 931) [ElementWise]_output_out_tensor)
Traceback (most recent call last):
  File "/root/anaconda3/envs/tensorrt/bin/mmdet2trt", line 33, in <module>
    sys.exit(load_entry_point('mmdet2trt', 'console_scripts', 'mmdet2trt')())
  File "/home/mmdetection-to-tensorrt/mmdet2trt/mmdet2trt.py", line 339, in main
    torch.save(trt_model.state_dict(), args.output)
  File "/root/anaconda3/envs/tensorrt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1261, in state_dict
    hook_result = hook(self, destination, prefix, local_metadata)
  File "/root/anaconda3/envs/tensorrt/lib/python3.7/site-packages/torch2trt_dynamic/torch2trt_dynamic.py", line 443, in _on_state_dict
    state_dict[prefix + 'engine'] = bytearray(self.engine.serialize())
AttributeError: 'NoneType' object has no attribute 'serialize'

environment:

PyTorch version: 1.9.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.21.2
Libc version: glibc-2.17
Python version: 3.7.11 (default, Jul 27 2021, 14:32:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.10.60.1-microsoft-standard-WSL2-x86_64-with-debian-buster-sid
Is CUDA available: True
CUDA runtime version: 11.6.112
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2070
Nvidia driver version: 471.21
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.1
Versions of relevant libraries:
[pip3] mmcv-full==1.4.6
[pip3] mmdet==2.19.1
[pip3] mmdet2trt==0.5.0
[pip3] nvidia-tensorrt==8.4.0.6
[pip3] torch==1.9.0+cu111
[pip3] torch2trt-dynamic==0.5.0
[pip3] torchaudio==0.9.0
[pip3] torchvision==0.10.0+cu111
[conda] mmcv-full                 1.4.6                    pypi_0    pypi
[conda] mmdet                     2.12.0                   pypi_0    pypi
[conda] mmdet2trt                 0.5.0                     dev_0    <develop>
[conda] nvidia-tensorrt           8.4.0.6                  pypi_0    pypi
[conda] torch                     1.9.0+cu111              pypi_0    pypi
[conda] torch2trt-dynamic         0.5.0                     dev_0    <develop>
[conda] torchaudio                0.9.0                    pypi_0    pypi
[conda] torchvision               0.10.0+cu111             pypi_0    pypi

[grimoire]

The SABL model conversion succeeded on my device with master branch of mmdetection.
And my environment is a little bit different from yours, with CUDA==11.3 and TensorRT ==8.2.3

[MeteoriteWeny]

The SABL model conversion succeeded on my device with master branch of mmdetection. And my environment is a little bit different from yours, with CUDA==11.3 and TensorRT ==8.2.3

Thanks for your reply. My another conversion for YOLOX seemed to succeeded, but the converted inference model seems not working. Do I need to downgrade the TensorRT 8.4 ea to an earlier version?

YOLOX Conversion

(tensorrt) root@P775TM1:/home/mmdetection-2.19.1# mmdet2trt --min-scale 1 3 640 640 --opt-scale 1 3 640 640 --max-scale 1 3 640 640 ../mmdetection-2.19.1/work_dirs/yolox_l_8x8_300e_coco_cloth.py ../mmdetection-2.19.1/work_dirs/yolox_l_8x8_300e_coco_cloth/best_bbox_mAP_epoch_300.pth ../mmdetection-2.19.1/work_dirs/yolox_l_8x8_300e_coco_cloth/yolox_l_8x8_300e_coco_cloth.trt
Use load_from_local loader
The model and loaded state dict do not match exactly

unexpected key in source state_dict: ema_backbone_stem_conv_conv_weight, ema_backbone_stem_conv_bn_weight, ema_backbone_stem_conv_bn_bias, ema_backbone_stem_conv_bn_running_mean, ema_backbone_stem_conv_bn_running_var, ema_backbone_stem_conv_bn_num_batches_tracked, ema_backbone_stage1_0_conv_weight, ema_backbone_stage1_0_bn_weight, ema_backbone_stage1_0_bn_bias, ema_backbone_stage1_0_bn_running_mean, ema_backbone_stage1_0_bn_running_var, ema_backbone_stage1_0_bn_num_batches_tracked, ema_backbone_stage1_1_main_conv_conv_weight, ema_backbone_stage1_1_main_conv_bn_weight, ema_backbone_stage1_1_main_conv_bn_bias, ema_backbone_stage1_1_main_conv_bn_running_mean, ema_backbone_stage1_1_main_conv_bn_running_var, ema_backbone_stage1_1_main_conv_bn_num_batches_tracked, ema_backbone_stage1_1_short_conv_conv_weight, ema_backbone_stage1_1_short_conv_bn_weight, ema_backbone_stage1_1_short_conv_bn_bias, ema_backbone_stage1_1_short_conv_bn_running_mean, ema_backbone_stage1_1_short_conv_bn_running_var, ema_backbone_stage1_1_short_conv_bn_num_batches_tracked, ema_backbone_stage1_1_final_conv_conv_weight, ema_backbone_stage1_1_final_conv_bn_weight, ema_backbone_stage1_1_final_conv_bn_bias, ema_backbone_stage1_1_final_conv_bn_running_mean, ema_backbone_stage1_1_final_conv_bn_running_var, ema_backbone_stage1_1_final_conv_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_0_conv1_conv_weight, ema_backbone_stage1_1_blocks_0_conv1_bn_weight, ema_backbone_stage1_1_blocks_0_conv1_bn_bias, ema_backbone_stage1_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage1_1_blocks_0_conv1_bn_running_var, ema_backbone_stage1_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_0_conv2_conv_weight, ema_backbone_stage1_1_blocks_0_conv2_bn_weight, ema_backbone_stage1_1_blocks_0_conv2_bn_bias, ema_backbone_stage1_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage1_1_blocks_0_conv2_bn_running_var, ema_backbone_stage1_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_1_conv1_conv_weight, ema_backbone_stage1_1_blocks_1_conv1_bn_weight, ema_backbone_stage1_1_blocks_1_conv1_bn_bias, ema_backbone_stage1_1_blocks_1_conv1_bn_running_mean, ema_backbone_stage1_1_blocks_1_conv1_bn_running_var, ema_backbone_stage1_1_blocks_1_conv1_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_1_conv2_conv_weight, ema_backbone_stage1_1_blocks_1_conv2_bn_weight, ema_backbone_stage1_1_blocks_1_conv2_bn_bias, ema_backbone_stage1_1_blocks_1_conv2_bn_running_mean, ema_backbone_stage1_1_blocks_1_conv2_bn_running_var, ema_backbone_stage1_1_blocks_1_conv2_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_2_conv1_conv_weight, ema_backbone_stage1_1_blocks_2_conv1_bn_weight, ema_backbone_stage1_1_blocks_2_conv1_bn_bias, ema_backbone_stage1_1_blocks_2_conv1_bn_running_mean, ema_backbone_stage1_1_blocks_2_conv1_bn_running_var, ema_backbone_stage1_1_blocks_2_conv1_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_2_conv2_conv_weight, ema_backbone_stage1_1_blocks_2_conv2_bn_weight, ema_backbone_stage1_1_blocks_2_conv2_bn_bias, ema_backbone_stage1_1_blocks_2_conv2_bn_running_mean, ema_backbone_stage1_1_blocks_2_conv2_bn_running_var, ema_backbone_stage1_1_blocks_2_conv2_bn_num_batches_tracked, ema_backbone_stage2_0_conv_weight, ema_backbone_stage2_0_bn_weight, ema_backbone_stage2_0_bn_bias, ema_backbone_stage2_0_bn_running_mean, ema_backbone_stage2_0_bn_running_var, ema_backbone_stage2_0_bn_num_batches_tracked, ema_backbone_stage2_1_main_conv_conv_weight, ema_backbone_stage2_1_main_conv_bn_weight, ema_backbone_stage2_1_main_conv_bn_bias, ema_backbone_stage2_1_main_conv_bn_running_mean, ema_backbone_stage2_1_main_conv_bn_running_var, ema_backbone_stage2_1_main_conv_bn_num_batches_tracked, ema_backbone_stage2_1_short_conv_conv_weight, ema_backbone_stage2_1_short_conv_bn_weight, ema_backbone_stage2_1_short_conv_bn_bias, ema_backbone_stage2_1_short_conv_bn_running_mean, ema_backbone_stage2_1_short_conv_bn_running_var, ema_backbone_stage2_1_short_conv_bn_num_batches_tracked, ema_backbone_stage2_1_final_conv_conv_weight, ema_backbone_stage2_1_final_conv_bn_weight, ema_backbone_stage2_1_final_conv_bn_bias, ema_backbone_stage2_1_final_conv_bn_running_mean, ema_backbone_stage2_1_final_conv_bn_running_var, ema_backbone_stage2_1_final_conv_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_0_conv1_conv_weight, ema_backbone_stage2_1_blocks_0_conv1_bn_weight, ema_backbone_stage2_1_blocks_0_conv1_bn_bias, ema_backbone_stage2_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_0_conv1_bn_running_var, ema_backbone_stage2_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_0_conv2_conv_weight, ema_backbone_stage2_1_blocks_0_conv2_bn_weight, ema_backbone_stage2_1_blocks_0_conv2_bn_bias, ema_backbone_stage2_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_0_conv2_bn_running_var, ema_backbone_stage2_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_1_conv1_conv_weight, ema_backbone_stage2_1_blocks_1_conv1_bn_weight, ema_backbone_stage2_1_blocks_1_conv1_bn_bias, ema_backbone_stage2_1_blocks_1_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_1_conv1_bn_running_var, ema_backbone_stage2_1_blocks_1_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_1_conv2_conv_weight, ema_backbone_stage2_1_blocks_1_conv2_bn_weight, ema_backbone_stage2_1_blocks_1_conv2_bn_bias, ema_backbone_stage2_1_blocks_1_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_1_conv2_bn_running_var, ema_backbone_stage2_1_blocks_1_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_2_conv1_conv_weight, ema_backbone_stage2_1_blocks_2_conv1_bn_weight, ema_backbone_stage2_1_blocks_2_conv1_bn_bias, ema_backbone_stage2_1_blocks_2_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_2_conv1_bn_running_var, ema_backbone_stage2_1_blocks_2_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_2_conv2_conv_weight, ema_backbone_stage2_1_blocks_2_conv2_bn_weight, ema_backbone_stage2_1_blocks_2_conv2_bn_bias, ema_backbone_stage2_1_blocks_2_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_2_conv2_bn_running_var, ema_backbone_stage2_1_blocks_2_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_3_conv1_conv_weight, ema_backbone_stage2_1_blocks_3_conv1_bn_weight, ema_backbone_stage2_1_blocks_3_conv1_bn_bias, ema_backbone_stage2_1_blocks_3_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_3_conv1_bn_running_var, ema_backbone_stage2_1_blocks_3_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_3_conv2_conv_weight, ema_backbone_stage2_1_blocks_3_conv2_bn_weight, ema_backbone_stage2_1_blocks_3_conv2_bn_bias, ema_backbone_stage2_1_blocks_3_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_3_conv2_bn_running_var, ema_backbone_stage2_1_blocks_3_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_4_conv1_conv_weight, ema_backbone_stage2_1_blocks_4_conv1_bn_weight, ema_backbone_stage2_1_blocks_4_conv1_bn_bias, ema_backbone_stage2_1_blocks_4_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_4_conv1_bn_running_var, ema_backbone_stage2_1_blocks_4_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_4_conv2_conv_weight, ema_backbone_stage2_1_blocks_4_conv2_bn_weight, ema_backbone_stage2_1_blocks_4_conv2_bn_bias, ema_backbone_stage2_1_blocks_4_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_4_conv2_bn_running_var, ema_backbone_stage2_1_blocks_4_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_5_conv1_conv_weight, ema_backbone_stage2_1_blocks_5_conv1_bn_weight, ema_backbone_stage2_1_blocks_5_conv1_bn_bias, ema_backbone_stage2_1_blocks_5_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_5_conv1_bn_running_var, ema_backbone_stage2_1_blocks_5_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_5_conv2_conv_weight, ema_backbone_stage2_1_blocks_5_conv2_bn_weight, ema_backbone_stage2_1_blocks_5_conv2_bn_bias, ema_backbone_stage2_1_blocks_5_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_5_conv2_bn_running_var, ema_backbone_stage2_1_blocks_5_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_6_conv1_conv_weight, ema_backbone_stage2_1_blocks_6_conv1_bn_weight, ema_backbone_stage2_1_blocks_6_conv1_bn_bias, ema_backbone_stage2_1_blocks_6_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_6_conv1_bn_running_var, ema_backbone_stage2_1_blocks_6_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_6_conv2_conv_weight, ema_backbone_stage2_1_blocks_6_conv2_bn_weight, ema_backbone_stage2_1_blocks_6_conv2_bn_bias, ema_backbone_stage2_1_blocks_6_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_6_conv2_bn_running_var, ema_backbone_stage2_1_blocks_6_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_7_conv1_conv_weight, ema_backbone_stage2_1_blocks_7_conv1_bn_weight, ema_backbone_stage2_1_blocks_7_conv1_bn_bias, ema_backbone_stage2_1_blocks_7_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_7_conv1_bn_running_var, ema_backbone_stage2_1_blocks_7_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_7_conv2_conv_weight, ema_backbone_stage2_1_blocks_7_conv2_bn_weight, ema_backbone_stage2_1_blocks_7_conv2_bn_bias, ema_backbone_stage2_1_blocks_7_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_7_conv2_bn_running_var, ema_backbone_stage2_1_blocks_7_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_8_conv1_conv_weight, ema_backbone_stage2_1_blocks_8_conv1_bn_weight, ema_backbone_stage2_1_blocks_8_conv1_bn_bias, ema_backbone_stage2_1_blocks_8_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_8_conv1_bn_running_var, ema_backbone_stage2_1_blocks_8_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_8_conv2_conv_weight, ema_backbone_stage2_1_blocks_8_conv2_bn_weight, ema_backbone_stage2_1_blocks_8_conv2_bn_bias, ema_backbone_stage2_1_blocks_8_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_8_conv2_bn_running_var, ema_backbone_stage2_1_blocks_8_conv2_bn_num_batches_tracked, ema_backbone_stage3_0_conv_weight, ema_backbone_stage3_0_bn_weight, ema_backbone_stage3_0_bn_bias, ema_backbone_stage3_0_bn_running_mean, ema_backbone_stage3_0_bn_running_var, ema_backbone_stage3_0_bn_num_batches_tracked, ema_backbone_stage3_1_main_conv_conv_weight, ema_backbone_stage3_1_main_conv_bn_weight, ema_backbone_stage3_1_main_conv_bn_bias, ema_backbone_stage3_1_main_conv_bn_running_mean, ema_backbone_stage3_1_main_conv_bn_running_var, ema_backbone_stage3_1_main_conv_bn_num_batches_tracked, ema_backbone_stage3_1_short_conv_conv_weight, ema_backbone_stage3_1_short_conv_bn_weight, ema_backbone_stage3_1_short_conv_bn_bias, ema_backbone_stage3_1_short_conv_bn_running_mean, ema_backbone_stage3_1_short_conv_bn_running_var, ema_backbone_stage3_1_short_conv_bn_num_batches_tracked, ema_backbone_stage3_1_final_conv_conv_weight, ema_backbone_stage3_1_final_conv_bn_weight, ema_backbone_stage3_1_final_conv_bn_bias, ema_backbone_stage3_1_final_conv_bn_running_mean, ema_backbone_stage3_1_final_conv_bn_running_var, ema_backbone_stage3_1_final_conv_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_0_conv1_conv_weight, ema_backbone_stage3_1_blocks_0_conv1_bn_weight, ema_backbone_stage3_1_blocks_0_conv1_bn_bias, ema_backbone_stage3_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_0_conv1_bn_running_var, ema_backbone_stage3_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_0_conv2_conv_weight, ema_backbone_stage3_1_blocks_0_conv2_bn_weight, ema_backbone_stage3_1_blocks_0_conv2_bn_bias, ema_backbone_stage3_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_0_conv2_bn_running_var, ema_backbone_stage3_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_1_conv1_conv_weight, ema_backbone_stage3_1_blocks_1_conv1_bn_weight, ema_backbone_stage3_1_blocks_1_conv1_bn_bias, ema_backbone_stage3_1_blocks_1_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_1_conv1_bn_running_var, ema_backbone_stage3_1_blocks_1_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_1_conv2_conv_weight, ema_backbone_stage3_1_blocks_1_conv2_bn_weight, ema_backbone_stage3_1_blocks_1_conv2_bn_bias, ema_backbone_stage3_1_blocks_1_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_1_conv2_bn_running_var, ema_backbone_stage3_1_blocks_1_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_2_conv1_conv_weight, ema_backbone_stage3_1_blocks_2_conv1_bn_weight, ema_backbone_stage3_1_blocks_2_conv1_bn_bias, ema_backbone_stage3_1_blocks_2_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_2_conv1_bn_running_var, ema_backbone_stage3_1_blocks_2_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_2_conv2_conv_weight, ema_backbone_stage3_1_blocks_2_conv2_bn_weight, ema_backbone_stage3_1_blocks_2_conv2_bn_bias, ema_backbone_stage3_1_blocks_2_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_2_conv2_bn_running_var, ema_backbone_stage3_1_blocks_2_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_3_conv1_conv_weight, ema_backbone_stage3_1_blocks_3_conv1_bn_weight, ema_backbone_stage3_1_blocks_3_conv1_bn_bias, ema_backbone_stage3_1_blocks_3_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_3_conv1_bn_running_var, ema_backbone_stage3_1_blocks_3_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_3_conv2_conv_weight, ema_backbone_stage3_1_blocks_3_conv2_bn_weight, ema_backbone_stage3_1_blocks_3_conv2_bn_bias, ema_backbone_stage3_1_blocks_3_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_3_conv2_bn_running_var, ema_backbone_stage3_1_blocks_3_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_4_conv1_conv_weight, ema_backbone_stage3_1_blocks_4_conv1_bn_weight, ema_backbone_stage3_1_blocks_4_conv1_bn_bias, ema_backbone_stage3_1_blocks_4_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_4_conv1_bn_running_var, ema_backbone_stage3_1_blocks_4_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_4_conv2_conv_weight, ema_backbone_stage3_1_blocks_4_conv2_bn_weight, ema_backbone_stage3_1_blocks_4_conv2_bn_bias, ema_backbone_stage3_1_blocks_4_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_4_conv2_bn_running_var, ema_backbone_stage3_1_blocks_4_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_5_conv1_conv_weight, ema_backbone_stage3_1_blocks_5_conv1_bn_weight, ema_backbone_stage3_1_blocks_5_conv1_bn_bias, ema_backbone_stage3_1_blocks_5_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_5_conv1_bn_running_var, ema_backbone_stage3_1_blocks_5_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_5_conv2_conv_weight, ema_backbone_stage3_1_blocks_5_conv2_bn_weight, ema_backbone_stage3_1_blocks_5_conv2_bn_bias, ema_backbone_stage3_1_blocks_5_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_5_conv2_bn_running_var, ema_backbone_stage3_1_blocks_5_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_6_conv1_conv_weight, ema_backbone_stage3_1_blocks_6_conv1_bn_weight, ema_backbone_stage3_1_blocks_6_conv1_bn_bias, ema_backbone_stage3_1_blocks_6_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_6_conv1_bn_running_var, ema_backbone_stage3_1_blocks_6_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_6_conv2_conv_weight, ema_backbone_stage3_1_blocks_6_conv2_bn_weight, ema_backbone_stage3_1_blocks_6_conv2_bn_bias, ema_backbone_stage3_1_blocks_6_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_6_conv2_bn_running_var, ema_backbone_stage3_1_blocks_6_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_7_conv1_conv_weight, ema_backbone_stage3_1_blocks_7_conv1_bn_weight, ema_backbone_stage3_1_blocks_7_conv1_bn_bias, ema_backbone_stage3_1_blocks_7_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_7_conv1_bn_running_var, ema_backbone_stage3_1_blocks_7_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_7_conv2_conv_weight, ema_backbone_stage3_1_blocks_7_conv2_bn_weight, ema_backbone_stage3_1_blocks_7_conv2_bn_bias, ema_backbone_stage3_1_blocks_7_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_7_conv2_bn_running_var, ema_backbone_stage3_1_blocks_7_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_8_conv1_conv_weight, ema_backbone_stage3_1_blocks_8_conv1_bn_weight, ema_backbone_stage3_1_blocks_8_conv1_bn_bias, ema_backbone_stage3_1_blocks_8_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_8_conv1_bn_running_var, ema_backbone_stage3_1_blocks_8_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_8_conv2_conv_weight, ema_backbone_stage3_1_blocks_8_conv2_bn_weight, ema_backbone_stage3_1_blocks_8_conv2_bn_bias, ema_backbone_stage3_1_blocks_8_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_8_conv2_bn_running_var, ema_backbone_stage3_1_blocks_8_conv2_bn_num_batches_tracked, ema_backbone_stage4_0_conv_weight, ema_backbone_stage4_0_bn_weight, ema_backbone_stage4_0_bn_bias, ema_backbone_stage4_0_bn_running_mean, ema_backbone_stage4_0_bn_running_var, ema_backbone_stage4_0_bn_num_batches_tracked, ema_backbone_stage4_1_conv1_conv_weight, ema_backbone_stage4_1_conv1_bn_weight, ema_backbone_stage4_1_conv1_bn_bias, ema_backbone_stage4_1_conv1_bn_running_mean, ema_backbone_stage4_1_conv1_bn_running_var, ema_backbone_stage4_1_conv1_bn_num_batches_tracked, ema_backbone_stage4_1_conv2_conv_weight, ema_backbone_stage4_1_conv2_bn_weight, ema_backbone_stage4_1_conv2_bn_bias, ema_backbone_stage4_1_conv2_bn_running_mean, ema_backbone_stage4_1_conv2_bn_running_var, ema_backbone_stage4_1_conv2_bn_num_batches_tracked, ema_backbone_stage4_2_main_conv_conv_weight, ema_backbone_stage4_2_main_conv_bn_weight, ema_backbone_stage4_2_main_conv_bn_bias, ema_backbone_stage4_2_main_conv_bn_running_mean, ema_backbone_stage4_2_main_conv_bn_running_var, ema_backbone_stage4_2_main_conv_bn_num_batches_tracked, ema_backbone_stage4_2_short_conv_conv_weight, ema_backbone_stage4_2_short_conv_bn_weight, ema_backbone_stage4_2_short_conv_bn_bias, ema_backbone_stage4_2_short_conv_bn_running_mean, ema_backbone_stage4_2_short_conv_bn_running_var, ema_backbone_stage4_2_short_conv_bn_num_batches_tracked, ema_backbone_stage4_2_final_conv_conv_weight, ema_backbone_stage4_2_final_conv_bn_weight, ema_backbone_stage4_2_final_conv_bn_bias, ema_backbone_stage4_2_final_conv_bn_running_mean, ema_backbone_stage4_2_final_conv_bn_running_var, ema_backbone_stage4_2_final_conv_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_0_conv1_conv_weight, ema_backbone_stage4_2_blocks_0_conv1_bn_weight, ema_backbone_stage4_2_blocks_0_conv1_bn_bias, ema_backbone_stage4_2_blocks_0_conv1_bn_running_mean, ema_backbone_stage4_2_blocks_0_conv1_bn_running_var, ema_backbone_stage4_2_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_0_conv2_conv_weight, ema_backbone_stage4_2_blocks_0_conv2_bn_weight, ema_backbone_stage4_2_blocks_0_conv2_bn_bias, ema_backbone_stage4_2_blocks_0_conv2_bn_running_mean, ema_backbone_stage4_2_blocks_0_conv2_bn_running_var, ema_backbone_stage4_2_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_1_conv1_conv_weight, ema_backbone_stage4_2_blocks_1_conv1_bn_weight, ema_backbone_stage4_2_blocks_1_conv1_bn_bias, ema_backbone_stage4_2_blocks_1_conv1_bn_running_mean, ema_backbone_stage4_2_blocks_1_conv1_bn_running_var, ema_backbone_stage4_2_blocks_1_conv1_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_1_conv2_conv_weight, ema_backbone_stage4_2_blocks_1_conv2_bn_weight, ema_backbone_stage4_2_blocks_1_conv2_bn_bias, ema_backbone_stage4_2_blocks_1_conv2_bn_running_mean, ema_backbone_stage4_2_blocks_1_conv2_bn_running_var, ema_backbone_stage4_2_blocks_1_conv2_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_2_conv1_conv_weight, ema_backbone_stage4_2_blocks_2_conv1_bn_weight, ema_backbone_stage4_2_blocks_2_conv1_bn_bias, ema_backbone_stage4_2_blocks_2_conv1_bn_running_mean, ema_backbone_stage4_2_blocks_2_conv1_bn_running_var, ema_backbone_stage4_2_blocks_2_conv1_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_2_conv2_conv_weight, ema_backbone_stage4_2_blocks_2_conv2_bn_weight, ema_backbone_stage4_2_blocks_2_conv2_bn_bias, ema_backbone_stage4_2_blocks_2_conv2_bn_running_mean, ema_backbone_stage4_2_blocks_2_conv2_bn_running_var, ema_backbone_stage4_2_blocks_2_conv2_bn_num_batches_tracked, ema_neck_reduce_layers_0_conv_weight, ema_neck_reduce_layers_0_bn_weight, ema_neck_reduce_layers_0_bn_bias, ema_neck_reduce_layers_0_bn_running_mean, ema_neck_reduce_layers_0_bn_running_var, ema_neck_reduce_layers_0_bn_num_batches_tracked, ema_neck_reduce_layers_1_conv_weight, ema_neck_reduce_layers_1_bn_weight, ema_neck_reduce_layers_1_bn_bias, ema_neck_reduce_layers_1_bn_running_mean, ema_neck_reduce_layers_1_bn_running_var, ema_neck_reduce_layers_1_bn_num_batches_tracked, ema_neck_top_down_blocks_0_main_conv_conv_weight, ema_neck_top_down_blocks_0_main_conv_bn_weight, ema_neck_top_down_blocks_0_main_conv_bn_bias, ema_neck_top_down_blocks_0_main_conv_bn_running_mean, ema_neck_top_down_blocks_0_main_conv_bn_running_var, ema_neck_top_down_blocks_0_main_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_short_conv_conv_weight, ema_neck_top_down_blocks_0_short_conv_bn_weight, ema_neck_top_down_blocks_0_short_conv_bn_bias, ema_neck_top_down_blocks_0_short_conv_bn_running_mean, ema_neck_top_down_blocks_0_short_conv_bn_running_var, ema_neck_top_down_blocks_0_short_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_final_conv_conv_weight, ema_neck_top_down_blocks_0_final_conv_bn_weight, ema_neck_top_down_blocks_0_final_conv_bn_bias, ema_neck_top_down_blocks_0_final_conv_bn_running_mean, ema_neck_top_down_blocks_0_final_conv_bn_running_var, ema_neck_top_down_blocks_0_final_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_0_conv1_conv_weight, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_weight, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_bias, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_running_mean, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_running_var, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_0_conv2_conv_weight, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_weight, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_bias, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_running_mean, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_running_var, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_1_conv1_conv_weight, ema_neck_top_down_blocks_0_blocks_1_conv1_bn_weight, ema_neck_top_down_blocks_0_blocks_1_conv1_bn_bias, ema_neck_top_down_blocks_0_blocks_1_conv1_bn_running_mean, ema_neck_top_down_blocks_0_blocks_1_conv1_bn_running_var, ema_neck_top_down_blocks_0_blocks_1_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_1_conv2_conv_weight, ema_neck_top_down_blocks_0_blocks_1_conv2_bn_weight, ema_neck_top_down_blocks_0_blocks_1_conv2_bn_bias, ema_neck_top_down_blocks_0_blocks_1_conv2_bn_running_mean, ema_neck_top_down_blocks_0_blocks_1_conv2_bn_running_var, ema_neck_top_down_blocks_0_blocks_1_conv2_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_2_conv1_conv_weight, ema_neck_top_down_blocks_0_blocks_2_conv1_bn_weight, ema_neck_top_down_blocks_0_blocks_2_conv1_bn_bias, ema_neck_top_down_blocks_0_blocks_2_conv1_bn_running_mean, ema_neck_top_down_blocks_0_blocks_2_conv1_bn_running_var, ema_neck_top_down_blocks_0_blocks_2_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_2_conv2_conv_weight, ema_neck_top_down_blocks_0_blocks_2_conv2_bn_weight, ema_neck_top_down_blocks_0_blocks_2_conv2_bn_bias, ema_neck_top_down_blocks_0_blocks_2_conv2_bn_running_mean, ema_neck_top_down_blocks_0_blocks_2_conv2_bn_running_var, ema_neck_top_down_blocks_0_blocks_2_conv2_bn_num_batches_tracked, ema_neck_top_down_blocks_1_main_conv_conv_weight, ema_neck_top_down_blocks_1_main_conv_bn_weight, ema_neck_top_down_blocks_1_main_conv_bn_bias, ema_neck_top_down_blocks_1_main_conv_bn_running_mean, ema_neck_top_down_blocks_1_main_conv_bn_running_var, ema_neck_top_down_blocks_1_main_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_short_conv_conv_weight, ema_neck_top_down_blocks_1_short_conv_bn_weight, ema_neck_top_down_blocks_1_short_conv_bn_bias, ema_neck_top_down_blocks_1_short_conv_bn_running_mean, ema_neck_top_down_blocks_1_short_conv_bn_running_var, ema_neck_top_down_blocks_1_short_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_final_conv_conv_weight, ema_neck_top_down_blocks_1_final_conv_bn_weight, ema_neck_top_down_blocks_1_final_conv_bn_bias, ema_neck_top_down_blocks_1_final_conv_bn_running_mean, ema_neck_top_down_blocks_1_final_conv_bn_running_var, ema_neck_top_down_blocks_1_final_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_0_conv1_conv_weight, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_weight, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_bias, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_running_mean, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_running_var, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_0_conv2_conv_weight, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_weight, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_bias, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_running_mean, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_running_var, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_1_conv1_conv_weight, ema_neck_top_down_blocks_1_blocks_1_conv1_bn_weight, ema_neck_top_down_blocks_1_blocks_1_conv1_bn_bias, ema_neck_top_down_blocks_1_blocks_1_conv1_bn_running_mean, ema_neck_top_down_blocks_1_blocks_1_conv1_bn_running_var, ema_neck_top_down_blocks_1_blocks_1_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_1_conv2_conv_weight, ema_neck_top_down_blocks_1_blocks_1_conv2_bn_weight, ema_neck_top_down_blocks_1_blocks_1_conv2_bn_bias, ema_neck_top_down_blocks_1_blocks_1_conv2_bn_running_mean, ema_neck_top_down_blocks_1_blocks_1_conv2_bn_running_var, ema_neck_top_down_blocks_1_blocks_1_conv2_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_2_conv1_conv_weight, ema_neck_top_down_blocks_1_blocks_2_conv1_bn_weight, ema_neck_top_down_blocks_1_blocks_2_conv1_bn_bias, ema_neck_top_down_blocks_1_blocks_2_conv1_bn_running_mean, ema_neck_top_down_blocks_1_blocks_2_conv1_bn_running_var, ema_neck_top_down_blocks_1_blocks_2_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_2_conv2_conv_weight, ema_neck_top_down_blocks_1_blocks_2_conv2_bn_weight, ema_neck_top_down_blocks_1_blocks_2_conv2_bn_bias, ema_neck_top_down_blocks_1_blocks_2_conv2_bn_running_mean, ema_neck_top_down_blocks_1_blocks_2_conv2_bn_running_var, ema_neck_top_down_blocks_1_blocks_2_conv2_bn_num_batches_tracked, ema_neck_downsamples_0_conv_weight, ema_neck_downsamples_0_bn_weight, ema_neck_downsamples_0_bn_bias, ema_neck_downsamples_0_bn_running_mean, ema_neck_downsamples_0_bn_running_var, ema_neck_downsamples_0_bn_num_batches_tracked, ema_neck_downsamples_1_conv_weight, ema_neck_downsamples_1_bn_weight, ema_neck_downsamples_1_bn_bias, ema_neck_downsamples_1_bn_running_mean, ema_neck_downsamples_1_bn_running_var, ema_neck_downsamples_1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_main_conv_conv_weight, ema_neck_bottom_up_blocks_0_main_conv_bn_weight, ema_neck_bottom_up_blocks_0_main_conv_bn_bias, ema_neck_bottom_up_blocks_0_main_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_main_conv_bn_running_var, ema_neck_bottom_up_blocks_0_main_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_short_conv_conv_weight, ema_neck_bottom_up_blocks_0_short_conv_bn_weight, ema_neck_bottom_up_blocks_0_short_conv_bn_bias, ema_neck_bottom_up_blocks_0_short_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_short_conv_bn_running_var, ema_neck_bottom_up_blocks_0_short_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_final_conv_conv_weight, ema_neck_bottom_up_blocks_0_final_conv_bn_weight, ema_neck_bottom_up_blocks_0_final_conv_bn_bias, ema_neck_bottom_up_blocks_0_final_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_final_conv_bn_running_var, ema_neck_bottom_up_blocks_0_final_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_0_conv1_conv_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_bias, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_0_conv2_conv_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_bias, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_1_conv1_conv_weight, ema_neck_bottom_up_blocks_0_blocks_1_conv1_bn_weight, ema_neck_bottom_up_blocks_0_blocks_1_conv1_bn_bias, ema_neck_bottom_up_blocks_0_blocks_1_conv1_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_1_conv1_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_1_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_1_conv2_conv_weight, ema_neck_bottom_up_blocks_0_blocks_1_conv2_bn_weight, ema_neck_bottom_up_blocks_0_blocks_1_conv2_bn_bias, ema_neck_bottom_up_blocks_0_blocks_1_conv2_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_1_conv2_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_1_conv2_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_2_conv1_conv_weight, ema_neck_bottom_up_blocks_0_blocks_2_conv1_bn_weight, ema_neck_bottom_up_blocks_0_blocks_2_conv1_bn_bias, ema_neck_bottom_up_blocks_0_blocks_2_conv1_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_2_conv1_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_2_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_2_conv2_conv_weight, ema_neck_bottom_up_blocks_0_blocks_2_conv2_bn_weight, ema_neck_bottom_up_blocks_0_blocks_2_conv2_bn_bias, ema_neck_bottom_up_blocks_0_blocks_2_conv2_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_2_conv2_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_2_conv2_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_main_conv_conv_weight, ema_neck_bottom_up_blocks_1_main_conv_bn_weight, ema_neck_bottom_up_blocks_1_main_conv_bn_bias, ema_neck_bottom_up_blocks_1_main_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_main_conv_bn_running_var, ema_neck_bottom_up_blocks_1_main_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_short_conv_conv_weight, ema_neck_bottom_up_blocks_1_short_conv_bn_weight, ema_neck_bottom_up_blocks_1_short_conv_bn_bias, ema_neck_bottom_up_blocks_1_short_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_short_conv_bn_running_var, ema_neck_bottom_up_blocks_1_short_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_final_conv_conv_weight, ema_neck_bottom_up_blocks_1_final_conv_bn_weight, ema_neck_bottom_up_blocks_1_final_conv_bn_bias, ema_neck_bottom_up_blocks_1_final_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_final_conv_bn_running_var, ema_neck_bottom_up_blocks_1_final_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_0_conv1_conv_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_bias, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_0_conv2_conv_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_bias, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_1_conv1_conv_weight, ema_neck_bottom_up_blocks_1_blocks_1_conv1_bn_weight, ema_neck_bottom_up_blocks_1_blocks_1_conv1_bn_bias, ema_neck_bottom_up_blocks_1_blocks_1_conv1_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_1_conv1_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_1_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_1_conv2_conv_weight, ema_neck_bottom_up_blocks_1_blocks_1_conv2_bn_weight, ema_neck_bottom_up_blocks_1_blocks_1_conv2_bn_bias, ema_neck_bottom_up_blocks_1_blocks_1_conv2_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_1_conv2_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_1_conv2_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_2_conv1_conv_weight, ema_neck_bottom_up_blocks_1_blocks_2_conv1_bn_weight, ema_neck_bottom_up_blocks_1_blocks_2_conv1_bn_bias, ema_neck_bottom_up_blocks_1_blocks_2_conv1_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_2_conv1_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_2_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_2_conv2_conv_weight, ema_neck_bottom_up_blocks_1_blocks_2_conv2_bn_weight, ema_neck_bottom_up_blocks_1_blocks_2_conv2_bn_bias, ema_neck_bottom_up_blocks_1_blocks_2_conv2_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_2_conv2_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_2_conv2_bn_num_batches_tracked, ema_neck_out_convs_0_conv_weight, ema_neck_out_convs_0_bn_weight, ema_neck_out_convs_0_bn_bias, ema_neck_out_convs_0_bn_running_mean, ema_neck_out_convs_0_bn_running_var, ema_neck_out_convs_0_bn_num_batches_tracked, ema_neck_out_convs_1_conv_weight, ema_neck_out_convs_1_bn_weight, ema_neck_out_convs_1_bn_bias, ema_neck_out_convs_1_bn_running_mean, ema_neck_out_convs_1_bn_running_var, ema_neck_out_convs_1_bn_num_batches_tracked, ema_neck_out_convs_2_conv_weight, ema_neck_out_convs_2_bn_weight, ema_neck_out_convs_2_bn_bias, ema_neck_out_convs_2_bn_running_mean, ema_neck_out_convs_2_bn_running_var, ema_neck_out_convs_2_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_0_0_conv_weight, ema_bbox_head_multi_level_cls_convs_0_0_bn_weight, ema_bbox_head_multi_level_cls_convs_0_0_bn_bias, ema_bbox_head_multi_level_cls_convs_0_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_0_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_0_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_0_1_conv_weight, ema_bbox_head_multi_level_cls_convs_0_1_bn_weight, ema_bbox_head_multi_level_cls_convs_0_1_bn_bias, ema_bbox_head_multi_level_cls_convs_0_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_0_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_0_1_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_1_0_conv_weight, ema_bbox_head_multi_level_cls_convs_1_0_bn_weight, ema_bbox_head_multi_level_cls_convs_1_0_bn_bias, ema_bbox_head_multi_level_cls_convs_1_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_1_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_1_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_1_1_conv_weight, ema_bbox_head_multi_level_cls_convs_1_1_bn_weight, ema_bbox_head_multi_level_cls_convs_1_1_bn_bias, ema_bbox_head_multi_level_cls_convs_1_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_1_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_1_1_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_2_0_conv_weight, ema_bbox_head_multi_level_cls_convs_2_0_bn_weight, ema_bbox_head_multi_level_cls_convs_2_0_bn_bias, ema_bbox_head_multi_level_cls_convs_2_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_2_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_2_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_2_1_conv_weight, ema_bbox_head_multi_level_cls_convs_2_1_bn_weight, ema_bbox_head_multi_level_cls_convs_2_1_bn_bias, ema_bbox_head_multi_level_cls_convs_2_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_2_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_2_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_0_0_conv_weight, ema_bbox_head_multi_level_reg_convs_0_0_bn_weight, ema_bbox_head_multi_level_reg_convs_0_0_bn_bias, ema_bbox_head_multi_level_reg_convs_0_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_0_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_0_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_0_1_conv_weight, ema_bbox_head_multi_level_reg_convs_0_1_bn_weight, ema_bbox_head_multi_level_reg_convs_0_1_bn_bias, ema_bbox_head_multi_level_reg_convs_0_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_0_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_0_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_1_0_conv_weight, ema_bbox_head_multi_level_reg_convs_1_0_bn_weight, ema_bbox_head_multi_level_reg_convs_1_0_bn_bias, ema_bbox_head_multi_level_reg_convs_1_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_1_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_1_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_1_1_conv_weight, ema_bbox_head_multi_level_reg_convs_1_1_bn_weight, ema_bbox_head_multi_level_reg_convs_1_1_bn_bias, ema_bbox_head_multi_level_reg_convs_1_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_1_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_1_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_2_0_conv_weight, ema_bbox_head_multi_level_reg_convs_2_0_bn_weight, ema_bbox_head_multi_level_reg_convs_2_0_bn_bias, ema_bbox_head_multi_level_reg_convs_2_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_2_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_2_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_2_1_conv_weight, ema_bbox_head_multi_level_reg_convs_2_1_bn_weight, ema_bbox_head_multi_level_reg_convs_2_1_bn_bias, ema_bbox_head_multi_level_reg_convs_2_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_2_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_2_1_bn_num_batches_tracked, ema_bbox_head_multi_level_conv_cls_0_weight, ema_bbox_head_multi_level_conv_cls_0_bias, ema_bbox_head_multi_level_conv_cls_1_weight, ema_bbox_head_multi_level_conv_cls_1_bias, ema_bbox_head_multi_level_conv_cls_2_weight, ema_bbox_head_multi_level_conv_cls_2_bias, ema_bbox_head_multi_level_conv_reg_0_weight, ema_bbox_head_multi_level_conv_reg_0_bias, ema_bbox_head_multi_level_conv_reg_1_weight, ema_bbox_head_multi_level_conv_reg_1_bias, ema_bbox_head_multi_level_conv_reg_2_weight, ema_bbox_head_multi_level_conv_reg_2_bias, ema_bbox_head_multi_level_conv_obj_0_weight, ema_bbox_head_multi_level_conv_obj_0_bias, ema_bbox_head_multi_level_conv_obj_1_weight, ema_bbox_head_multi_level_conv_obj_1_bias, ema_bbox_head_multi_level_conv_obj_2_weight, ema_bbox_head_multi_level_conv_obj_2_bias

/root/anaconda3/envs/tensorrt/lib/python3.7/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
[03/21/2022-11:12:47] [TRT] [I] [MemUsageChange] Init CUDA: CPU +306, GPU +0, now: CPU 3100, GPU 2665 (MiB)
[03/21/2022-11:12:48] [TRT] [I] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 3117 MiB, GPU 2665 MiB
[03/21/2022-11:12:53] [TRT] [I] [MemUsageSnapshot] End constructing builder kernel library: CPU 3182 MiB, GPU 2665 MiB
[03/21/2022-11:12:56] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +519, GPU +222, now: CPU 4123, GPU 2785 (MiB)
[03/21/2022-11:12:57] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +117, GPU +52, now: CPU 4240, GPU 2837 (MiB)
[03/21/2022-11:12:57] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[03/21/2022-11:13:17] [TRT] [I] Some tactics do not have sufficient workspace memory to run. Increasing workspace size will enable more tactics, please check verbose output for requested sizes.
[03/21/2022-11:14:25] [TRT] [I] Detected 1 inputs and 4 output network tensors.
[03/21/2022-11:14:26] [TRT] [I] Total Host Persistent Memory: 351888
[03/21/2022-11:14:26] [TRT] [I] Total Device Persistent Memory: 18877440
[03/21/2022-11:14:26] [TRT] [I] Total Scratch Memory: 2402816
[03/21/2022-11:14:26] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 232 MiB, GPU 471 MiB
[03/21/2022-11:14:27] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 187.157ms to assign 13 blocks to 329 nodes requiring 70236160 bytes.
[03/21/2022-11:14:27] [TRT] [I] Total Activation Memory: 70236160
[03/21/2022-11:14:27] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 4775, GPU 3273 (MiB)
[03/21/2022-11:14:27] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 4776, GPU 3283 (MiB)
[03/21/2022-11:14:27] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +206, GPU +207, now: CPU 206, GPU 207 (MiB)
[03/21/2022-11:14:27] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 4774, GPU 3259 (MiB)
[03/21/2022-11:14:27] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +1, GPU +8, now: CPU 4775, GPU 3267 (MiB)
[03/21/2022-11:14:27] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +1, GPU +84, now: CPU 207, GPU 291 (MiB)

YOLOX inference

 (tensorrt) root@P775TM1:/home/mmdetection-2.19.1# python
Python 3.7.11 (default, Jul 27 2021, 14:32:16)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from mmdet.apis.inference import init_detector, inference_detector
>>> from mmdet2trt.apis import create_wrap_detector
>>> trt_detector = create_wrap_detector('work_dirs/yolox_l_8x8_300e_coco_cloth/yolox_l_8x8_300e_coco_cloth.trt', 'work_dirs/yolox_l_8x8_300e_coco_cloth.py', 'cuda:0')
Can not load dataset from config. Use default CLASSES instead.
>>> trt_detector
TRTDetector(
  (model): TRTModule()
)
>>> result = inference_detector(trt_detector, 'demo/cloth_flaw.jpg')
#assertion/home/amirstan_plugin/src/plugin/batchedNMSPlugin/batchedNMSPlugin.cpp,127
Aborted

[grimoire]

try add

cudaDeviceSynchronize();    // synchronize the device so we can read the log
printf("cuda error log: %s\n", cudaGetErrorString(cudaGetLastError()));    // print the log

before the assert, rebuild the plugin and see if there are any error log.

[MeteoriteWeny]

cuda error log: the provided PTX was compiled with an unsupported toolchain.

[grimoire]

OK, have you update your cuda or tensorrt? rebuild the plugin library might fix this.

[MeteoriteWeny]

I have update the cuda development environment to 11.6, and tensorrt is 8.4.0. The plugin is also rebuilt, but the same cuda error still exists.

(tensorrt) root@P775TM1:/home/mmdetection-2.19.1# dpkg -l | grep TensorRT
ii  libnvinfer-bin                                              8.4.0-1+cuda11.6                    amd64        TensorRT binaries
ii  libnvinfer-dev                                              8.4.0-1+cuda11.6                    amd64        TensorRT development libraries and headers
ii  libnvinfer-doc                                              8.4.0-1+cuda11.6                    all          TensorRT documentation
ii  libnvinfer-plugin-dev                                       8.4.0-1+cuda11.6                    amd64        TensorRT plugin libraries
ii  libnvinfer-plugin8                                          8.4.0-1+cuda11.6                    amd64        TensorRT plugin libraries
ii  libnvinfer-samples                                          8.4.0-1+cuda11.6                    all          TensorRT samples
ii  libnvinfer8                                                 8.4.0-1+cuda11.6                    amd64        TensorRT runtime libraries
ii  libnvonnxparsers-dev                                        8.4.0-1+cuda11.6                    amd64        TensorRT ONNX libraries
ii  libnvonnxparsers8                                           8.4.0-1+cuda11.6                    amd64        TensorRT ONNX libraries
ii  libnvparsers-dev                                            8.4.0-1+cuda11.6                    amd64        TensorRT parsers libraries
ii  libnvparsers8                                               8.4.0-1+cuda11.6                    amd64        TensorRT parsers libraries
ii  onnx-graphsurgeon                                           8.4.0-1+cuda11.6                    amd64        ONNX GraphSurgeon for TensorRT package
ii  python3-libnvinfer                                          8.4.0-1+cuda11.6                    amd64        Python 3 bindings for TensorRT
ii  python3-libnvinfer-dev                                      8.4.0-1+cuda11.6                    amd64        Python 3 development package for TensorRT
ii  tensorrt                                                    8.4.0.6-1+cuda11.6                  amd64        Meta package of TensorRT