On This Page
What the model actually produces
Three outputs answer different questions. Classification assigns a category
to an image or crop. Object detection places a box around a suspected
feature and labels it. Segmentation identifies the pixels belonging to a
feature, providing a more detailed outline. Ultralytics'
segmentation documentation
illustrates outputs containing masks, classes, boxes, and confidence scores.
For inspection, a box may be sufficient to direct a reviewer to a suspected
crack. An outline can support assessment of visible extent. Neither output, by
itself, establishes crack depth, the cause of damage, or the urgency of repair.
Physical dimensions also require a justified image scale or geometric
measurement method. A pixel outline is still an image-space result.
The system boundary should therefore be explicit: is the supplier offering image
sorting, candidate detection, measured dimensions, or a reviewed inspection
report? Our explanation of
what drone-inspection evidence can prove
separates observation from measurement and diagnosis.
A useful deployment links each prediction back to its original photograph and
asset component. Otherwise, a promising demonstration can leave the maintenance
team with annotated pictures that are difficult to locate, verify, or compare at
the next inspection.
What must enter the workflow
Start with an inspection brief defining the assets, component identifiers,
defect classes, and smallest features the job needs to resolve. Include examples
of acceptable condition as well as damage. Decide how uncertain, obscured, and
out-of-scope areas will be recorded before collecting images.
Specify the capture conditions the proposed system has actually been evaluated
on: sensor type, image resolution, working distance, viewing angle, lighting,
and any resizing or compression. Request representative original files,
including difficult views, rather than selecting only the clearest demonstration
images.
The published
MBDD2025 building-defect dataset
shows why those details matter. Its creators describe UAV imagery from six
districts and counties in Hefei, China, covering five defect types. They
excluded blurred and defocused photographs and avoided extreme weather such as
rain or heavy fog. Those choices define the data represented in the research;
they do not establish performance on rejected images or different asset
populations.
For procurement, turn that limitation into a separate image usability check.
Require the workflow to identify inadequate views and request recapture. A frame
rejected for poor quality must remain visible as a coverage gap. Counting it as
a clean surface would confuse missing evidence with evidence of no defect.
The processing location is another integration choice. Onboard processing is
useful when a result must inform capture before the aircraft leaves; post-flight
processing can fit a workflow in which reviewers work from uploaded originals.
Ask when results become available, which files must move, what happens without
connectivity, and whether the same model and image settings are used in the
demonstration and delivered service. A processing-speed number has little
meaning without that boundary.
How to read an accuracy claim
Start with the denominator
Ask whether the score describes images, image crops, individual defects, pixels,
components, or entire assets. A model can classify a crop correctly without
finding every defect in the original photograph. Multiple views of one crack
must also be reconciled if the deliverable promises a count of unique defects.
For classification, accuracy is the share of all decisions that are correct.
Recall is the share of actual positive cases found; precision is the
share of positive predictions that are correct. Google's
metric definitions
explain why accuracy can mislead when positive cases are rare, and why changing
the classification threshold can trade missed cases against false alarms.
Consider an illustrative test of 1,000 component images, each assigned one
independently checked defect-present or defect-absent label. Suppose 20 contain
a target defect. A system calling every image defect-free would score 980 /
1,000 = 98% accuracy while finding none of the 20 positives.
Now suppose a different result contains 16 true positives, 4 false negatives, 24
false positives, and 956 true negatives. The calculated accuracy is (16 + 956) /
1,000 = 97.2%. Recall is 16 / 20 = 80%, and precision is 16 / 40 = 40%.
Reviewers receive 40 flags, of which 24 are false alarms, while four positive
images remain unflagged.
These invented inputs illustrate the arithmetic, not a product benchmark. They
show why a slightly lower accuracy score can coexist with much better defect
finding, and why the review workload still matters.
Check location quality and operating settings
For object detection, intersection over union, or IoU, compares the overlap
of a predicted region and a reference region with their combined area. Mean
average precision, or mAP, summarizes detection performance across classes.
The overlap requirement changes the task: mAP at IoU 0.50 is not directly
interchangeable with mAP averaged from 0.50 to 0.95. Ultralytics'
model-evaluation guide
explains those distinctions.
Ask for the matching rule, defect classes, image-processing settings, and
confidence threshold used for the reported result. Request results by class and
feature size, alongside the number of examples. An aggregate score can conceal a
weak category that drives your maintenance risk. A detector's box score also
does not establish the accuracy of a crack-width measurement.
Keep coverage outside that score. Recall measured on visible, labeled defects
says nothing about a surface the camera never captured. Require the report to
state which images and areas were excluded before calculating model performance.
NIST's
AI Risk Management Framework 1.0
calls for realistic test sets representing expected use, documented methods, and
evaluation under conditions similar to deployment. For a buyer, that means
requiring evidence from the intended asset and capture conditions before relying
on a benchmark percentage.
Training data that transfers to your assets
Define the labels before counting images
Ask who labeled the data, what qualifies as each defect, how disagreements were
resolved, and whether uncertain examples were retained as uncertain. Establish
whether a label describes visible appearance or a condition confirmed by another
inspection method. Those are different targets for a model to learn.
The MBDD2025 authors report cross-checking annotations and obtaining
industry-expert review. They also describe expanding an underrepresented bulge
class from 98 samples to 980 through rotation and cropping, while acknowledging
that augmentation may not capture real changes in viewpoint, light, and
environment. Extra versions of an image are not equivalent to observations of
additional independent defects.
For your project, request the number of distinct assets, capture sessions, and
confirmed examples per critical class, with augmented images counted separately.
Include intact surfaces and difficult lookalikes such as joints, stains,
shadows, and surface texture in the trial specification. If a consequential
defect class has few examples, record the resulting uncertainty rather than
claiming that a large total image count resolves it. Decide whether more
labeling is worthwhile by tracking performance on independent data as training
examples are added, especially for the weak classes.
Training data teaches the model; validation data supports development choices; a
held-out test evaluates the finished configuration. AWS's
guidance on data splits and leakage
recommends this separation and warns that duplicate samples can inflate results.
Drone collections need particular care because neighboring frames and crops can
show the same physical feature. Apply the grouped-data principle documented by
scikit-learn:
related samples should remain together when testing transfer to unseen groups.
For a trial claiming performance on new buildings, keep all images of each test
building outside model development. For repeat visits to known assets, test on
later capture sessions and state that narrower claim.
Keep original images and their augmented derivatives on the same side of the
split. Freeze the model and operating settings before scoring the final test. If
the team uses test failures to improve the model, it needs fresh independent
evidence for the next final evaluation.
Ask separately whether the service may retain your photographs, annotations, and
reviewer corrections for future training, including training used for other
customers. Resolve the permitted uses and export arrangements in the procurement
terms. Access to your inspection history should not depend on an undocumented
training-data arrangement.
Human review and measurable deliverables
Reviewers need original images, neighboring views, component context, and the
ability to correct a prediction. Preserve the model output and the reviewer
decision as separate records so that an edited report does not erase what the
automated system actually did.
Reviewing only the flagged queue cannot reveal all missed defects. For a pilot,
arrange independent assessment of the held-out set, including unflagged images.
In ongoing use, define how unflagged material will be sampled and which
high-consequence components need fuller examination. Set the extent of review
with the responsible inspection team and the consequences of an omission.
Use clear dispositions: suspected defect, reviewer-confirmed visual finding,
rejected flag, insufficient imagery, and further examination required. A
reviewer may confirm that a visible feature is present without confirming its
cause or structural significance. Urgent suspected conditions need an agreed
escalation route before routine report delivery.
The table below is a proposed buyer specification, synthesizing the cited
evaluation and data-quality guidance. It is not a reporting standard or a claim
that every service provides these outputs.
Scroll horizontally to compare all columns.
Measure elapsed time to a usable reviewed report as well as model processing
time. Include upload, review, duplicate reconciliation, recapture, and report
correction when comparing the proposed workflow with the current one. Compare
both workflows on the same inspection scope and required output quality. Present
any savings as trial results under the recorded conditions, rather than
extrapolating from frames processed per second.
Keep monitoring after rollout. NIST's framework includes production monitoring;
a practical inspection program should record changes in cameras, seasons,
surfaces, model versions, and processing settings, then reassess results when
those changes could affect performance.
What to require before buying
Request a bounded pilot with agreed defect classes, representative assets,
independent reference review, and a frozen model configuration. Set separate
requirements for missed critical defects, false-alarm workload, usable coverage,
and report delivery. The responsible inspection team should choose those
requirements from the intended maintenance decision; there is no universal
accuracy percentage that answers all four questions.
Ask the supplier to demonstrate three complete paths: a correct finding, an
unflagged defect discovered by review, and an image that cannot support a
decision. Follow each through the export and into the receiving team's workflow.
This exposes how the system handles the cases that a gallery of successful
detections leaves unanswered.
AI assistance fits best where the pilot establishes useful detection and a
manageable review process for a clearly defined task. If the evidence covers
only different assets, selected clean images, or rare defects with too few
independent examples, retain fuller conventional review while collecting better
evidence. Buy the demonstrated workflow and its stated limits, with
responsibility for unresolved findings assigned before scaling.
Source notes
- Ultralytics: Instance Segmentation.
Primary software documentation describing masks, classes, boxes, and scores.
- Zha and colleagues: A dataset of building surface defects collected by UAVs for machine learning-based detection.
Original Scientific Data research describing collection, screening,
annotation, and augmentation limits.
- Google: Accuracy, recall, precision, and related metrics.
Primary educational documentation for classification metrics and threshold
tradeoffs.
- Ultralytics: Model Evaluation and Fine-Tuning.
Primary documentation explaining IoU and mAP variants.
- NIST: AI Risk Management Framework 1.0.
Voluntary framework supporting representative testing, documented evaluation,
and monitoring; not an inspection certification.
- AWS: Splits and data leakage.
Primary technical guidance on held-out evaluation and duplicate samples.
- scikit-learn: Cross-validation for grouped data.
Primary software documentation explaining separation of dependent sample
groups.
Last checked: September 7, 2026.
