Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting segmentation fault when running InferenceSlicer with OBB model with thread_workers > 1 #1632

Open
1 of 2 tasks
zxsk1974 opened this issue Oct 30, 2024 · 9 comments
Open
1 of 2 tasks
Labels
bug Something isn't working

Comments

@zxsk1974
Copy link

zxsk1974 commented Oct 30, 2024

Search before asking

  • I have searched the Supervision issues and found no similar bug report.

Bug

InferenceSlicer throws Segmentation fault with thread_workers = 4:

Segmentation fault (core dumped)

Environment

Supervision 0.24.0
Python 3.11

Minimal Reproducible Example

from ultralytics import YOLO
import cv2
import sys
import torch
import numpy as np
from pathlib import Path
import math
import supervision as sv

def callback(image_slice: np.ndarray) -> sv.Detections:
    result = model(image_slice, conf=0.6)
    print("Results:", result)
    return sv.Detections.from_ultralytics(result[0])

model = YOLO('yolo11x-obb.pt')

print("GPU:", torch.cuda.is_available())
if torch.cuda.is_available():
    device = torch.device("cuda")
    print("Using GPU:", torch.cuda.get_device_name())

# load image
file_name = Path(sys.argv[1])
image = cv2.imread(sys.argv[1]) #Image.open(file_name, mode='r')
print("Loaded image:", file_name)

image_wh = (image.shape[1], image.shape[0])
slice_wh = (1024, 1024)
overlap_ratio_wh = (0.2, 0.2)
overlap_ratio_w, overlap_ratio_h = overlap_ratio_wh
slice_w, slice_h = slice_wh
overlap_wh = (math.ceil(slice_w * overlap_ratio_w), math.ceil(slice_h * overlap_ratio_h))

slicer = sv.InferenceSlicer(
  callback=callback,
  overlap_filter=sv.OverlapFilter.NON_MAX_MERGE,
  iou_threshold=0.15,
  slice_wh=slice_wh,
  overlap_ratio_wh=None,
  overlap_wh=overlap_wh,
  thread_workers=2
)

detections = slicer(image)

labels = [
    f"{class_name} {confidence:.1f}"
    for class_name, confidence
    in zip(detections['class_name'], detections.confidence)
]

label_annotator = sv.LabelAnnotator(text_scale=0.2, text_thickness=1, text_padding=0, text_position=sv.Position.TOP_LEFT)
bbox_annotator = sv.BoxAnnotator(color=sv.ColorPalette.DEFAULT.colors[6], thickness=2)
obb_annotator = sv.OrientedBoxAnnotator(color=sv.ColorPalette.DEFAULT.colors[6], thickness=2)

print(f"Image shape: {image_wh[0]}w x {image_wh[1]}h")
print(f"Tile size: {slice_wh[0]}w x {slice_wh[1]}h")
print(f"Overlap: {overlap_wh[0]}w x {overlap_wh[1]}h. Ratio {overlap_ratio_wh}")
print(f"Overlap Filter: {sv.OverlapFilter.NON_MAX_MERGE}")
print(f"Found {len(detections)} objects")

annotated_image = obb_annotator.annotate(scene=image.copy(), detections=detections)
annotated_image = label_annotator.annotate(scene=annotated_image, detections=detections, labels=labels)

cv2.imwrite(file_name.stem + "-output.jpg", annotated_image)
![airplane-graveyard-zoomed](https://github.com/user-attachments/assets/0bf79b23-4775-43f3-8573-bfe0e96cbf91)

Additional

To reproduce:

python detect-image-slicer-bug.py airplane-graveyard-zoomed.jpg

airplane-graveyard-zoomed
.jpg

Are you willing to submit a PR?

  • Yes I'd like to help by submitting a PR!
@zxsk1974 zxsk1974 added the bug Something isn't working label Oct 30, 2024
@LinasKo
Copy link
Contributor

LinasKo commented Oct 30, 2024

Hi @zxsk1974 👋

Thank you for reporting it. Unfortunately, many models aren't made to be run from multiple threads. We plan to shift to running in batches (linked PR), but it is unfortunately relatively low on the priority list.

@zxsk1974
Copy link
Author

@LinasKo batching will work too, can I try it on pre-release branch?

@LinasKo
Copy link
Contributor

LinasKo commented Oct 30, 2024

Not as a pre-release branch. I don't expect to merge this any time soon.

However, I brought the branch up-to-date with the latest supervision version.
Feel free to install via
pip install git+https://github.com/LinasKo/supervision.git@feature/batched-inference-slicer.

Alternatively, you may fork it from my repo. Install it in the same way, but from your own namespace.

@LinasKo
Copy link
Contributor

LinasKo commented Oct 30, 2024

Let us know if it works - that'd give more reason to revisit the PR 🙂

@zxsk1974
Copy link
Author

zxsk1974 commented Oct 31, 2024

Interesting, but sometimes the Slicer with 2 threads works till the end of detection and than throws the error, but not segmentation fault. This is still released version of SV.
The error:

Traceback (most recent call last):
File "/home/sergey/Workspace/detection-tiling-custom-obb/detect-image-slicer.py", line 89, in
detections = slicer(image)
^^^^^^^^^^^^^
File "/home/sergey/anaconda3/envs/yolo/lib/python3.11/site-packages/supervision/detection/tools/inference_slicer.py", line 162, in call
detections_list.append(future.result())
^^^^^^^^^^^^^^^
File "/home/sergey/anaconda3/envs/yolo/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/home/sergey/anaconda3/envs/yolo/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/sergey/anaconda3/envs/yolo/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sergey/anaconda3/envs/yolo/lib/python3.11/site-packages/supervision/detection/tools/inference_slicer.py", line 191, in _run_callback
detections = self.callback(image_slice)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sergey/Workspace/detection-tiling-custom-obb/detect-image-slicer.py", line 50, in callback
result = model(image_slice, conf=0.6)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sergey/anaconda3/envs/yolo/lib/python3.11/site-packages/ultralytics/engine/model.py", line 176, in call
return self.predict(source, stream, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sergey/anaconda3/envs/yolo/lib/python3.11/site-packages/ultralytics/engine/model.py", line 547, in predict
self.predictor.setup_model(model=self.model, verbose=is_cli)
File "/home/sergey/anaconda3/envs/yolo/lib/python3.11/site-packages/ultralytics/engine/predictor.py", line 303, in setup_model
self.model = AutoBackend(
^^^^^^^^^^^^
File "/home/sergey/anaconda3/envs/yolo/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/sergey/anaconda3/envs/yolo/lib/python3.11/site-packages/ultralytics/nn/autobackend.py", line 145, in init
model = model.fuse(verbose=verbose)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sergey/anaconda3/envs/yolo/lib/python3.11/site-packages/ultralytics/nn/tasks.py", line 208, in fuse
delattr(m, "bn") # remove batchnorm
^^^^^^^^^^^^^^^^
File "/home/sergey/anaconda3/envs/yolo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1799, in delattr
super().delattr(name)
AttributeError: 'Conv' object has no attribute 'bn'

@LinasKo
Copy link
Contributor

LinasKo commented Oct 31, 2024

Could be an ultralytics version issue. Unless you need yolo11, I'd try ultralytics==8.2.103.

@zxsk1974
Copy link
Author

@LinasKo apologies for the delay - I got pulled into another area of the project. I just tried your branch build and it worked very well. I observed 25-30% speed up from batch size 1 to batch size 10, than almost no speedup up to batch size 100+, since GPU utilization became bottle neck. I noticed that your branch was forked from 0.24.0 version, so if you merge 0.25.1 changes into your branch, I can test it again. Please let me know when it will merged into release branch. Thank you again.

@SkalskiP
Copy link
Collaborator

Hi @zxsk1974 👋🏻 no worries! Which version of the code you just tested?

@zxsk1974
Copy link
Author

From this branch - batched-inference-slicer:
pip install git+https://github.com/LinasKo/supervision.git@feature/batched-inference-slicer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants