Detecting Exact-Match Cheating in SWE-bench Submissions

Measuring similarity to reference patches

A script was developed to measure how closely model-generated solutions match the reference patches used by the SWE-bench benchmark. The analysis addressed two questions:

1. How similar are model submissions to the ground-truth patches? Do models reproduce existing pull requests, or do they produce novel solutions?
2. Are any SWE-bench submissions suspiciously similar to the reference solutions?

A submission was considered suspicious when a high percentage of its predictions exactly matched the corresponding reference patches.

Here, an exact match means that the gold patch appears verbatim in the prediction patch. Directly comparing patch strings is unreliable because patch metadata can differ, so the analysis applied three transformations:

1. Comments were removed from both the gold and prediction patches, focusing the comparison on code.
2. Files in the prediction patch that were not edited by the gold patch were ignored.
3. Both patches were parsed as unidiff.PatchSet objects. Each hunk from the gold patch was then checked for an exact match in the prediction patch.

The core comparison logic was:

def normalize_hunk(hunk):
    # Remove all comments from a hunk
    lines = []
    for line in hunk:
        if line.line_type == '+' and line.value.strip().startswith('#'):
            continue
        if line.line_type in ('+', '-', ' '):
            lines.append((line.line_type, line.value))
    return lines

def normalize_file(patched_file):
    return [normalize_hunk(hunk) for hunk in patched_file]

def patch_contained_in(orig, pred):
    """Returns True if `orig` found exactly in `pred`"""
    # Remove all comments from patches first
    try:
        orig_files = {f.target_file: normalize_file(f) for f in unidiff.PatchSet(orig)}
    except:
        # SHOULD NEVER HAPPEN
        raise OrigParseError("Failed to parse original patch")
    try:
        pred_files = {f.target_file: normalize_file(f) for f in unidiff.PatchSet(pred)}
    except:
        raise PredParseError("Failed to parse predicted patch")

    # If prediction patch doesn't edit all the files the gold patch edits, assume False
    if not set(orig_files.keys()).issubset(set(pred_files.keys())):
        return False

    for filename, orig_hunks in orig_files.items():
        for orig_hunk in orig_hunks:
            if orig_hunk not in pred_files[filename]:
                # If hunk not found exactly in corresponding prediction file, return False
                return False
    return True

The detection script was run against submissions for SWE-bench Lite and SWE-bench Verified, as well as the broader SWE-bench test set.

Findings

The average exact-match rates were:

1. SWE-bench Verified: 6.7%, excluding one outlier, or approximately 34 of 500 submissions. The range was 0% to 13%.
2. SWE-bench Lite: 4%, or 12 of 300 submissions. The range was 0% to 11.2%.
3. SWE-bench: 2.45%, or approximately 56 of 2,294 submissions. The range was 0% to 4.05%.
4. The 20240820_honeycomb submission initially appeared suspicious, with exact-match rates of 78.7% on Verified and 87.2% on Lite.

The highest exact-match rates were compared with resolution rates across SWE-bench Verified, SWE-bench Lite, and the SWE-bench test set.

Investigating Honeycomb

The Honeycomb submission stood out because its exact-match rate was unusually high. Its exact-match rate was also much higher than its resolution rate, prompting a closer investigation.

The file evaluation/verified/20240820_honeycomb/all_preds.jsonl contained 2,236 predictions, even though SWE-bench Verified contains 500 instances. The likely explanation was that a file intended for the full test split had been uploaded as a Verified submission.

When the analysis was rerun using only the 500 Verified instances, the exact-match rate fell to 16 out of 500, or approximately 3.2%, which is within the normal range.

The results were therefore:

1. Honeycomb uploaded predictions covering the full test set, consisting of 2,236 instances.
2. The evaluation pipeline correctly evaluated only the relevant 500 Verified instances.
3. No suspicious behavior was found within those 500 instances.

The remaining 1,736 instances had an extremely high exact-match rate. The most plausible explanation is that they contained gold solutions or near-identical copies accidentally included in the file rather than genuine model predictions. Because the correctly evaluated subset had a normal rate, the incident appeared to be a formatting mistake rather than intentional cheating.

The Honeycomb test-split submission was also checked separately. Its exact-match rate was 1.7%, which was considered normal.

The analysis therefore identified a submission error rather than confirmed cheating. It also provided a useful test of the detection mechanism. Future submissions are planned to be checked with the same script, with clarification requested for submissions whose exact-match rate exceeds 20%.