The Same CCTV List, Uploaded Four Times: Finding Repeated Uploads in Korea's Open Data

Korea's national CCTV standard dataset (전국CCTV표준데이터) keeps an agency's old camera list when the agency uploads a new full list under a later reference date. I found out while checking one district's numbers on Viewcone, my map of Korea's public CCTV: in Daegu Seo-gu it was counting the same cameras up to four times. Row-level deduplication could not catch it, since every upload assigns fresh record IDs. What worked was comparing whole uploads: an earlier upload is dropped when at least 80% of its rows reappear, by address or by coordinate, in the same agency's later uploads. That rule removed 38 uploads and 29,223 rows.

Every count from the source file below comes from the file downloaded 2026-09-20 (381,036 rows). It is a nationwide CSV published by the Ministry of the Interior and Safety and assembled from lists that local governments and their departments upload on their own schedules, so the pattern may be familiar if you work with any merged government dataset.

Reading the file

The CSV is encoded in CP949, Microsoft's superset of EUC-KR. Read as UTF-8 it fails at the first byte; read as strict EUC-KR it gets further and then fails at byte offset 420,434, on a character only CP949 covers. In Python that means encoding="cp949". The headers are Korean; these are the columns this post uses:

Column	Meaning
개방자치단체코드	Local-government code of the publisher
관리기관명	Managing agency name (free text; one code can hold several departments)
데이터기준일자	Reference date of the upload
관리번호	Record ID
데이터갱신구분	Update type (I, U or blank)
데이터갱신시점	Update timestamp
소재지도로명주소, 소재지지번주소	Road address, lot-number address
WGS84위도, WGS84경도	Latitude, longitude
카메라대수	Number of cameras on the row

Five cameras per site in one district

I noticed it while putting together per-district figures for Daegu. The map showed Daegu's Seo-gu with 1,060 installation sites and 5,246 cameras, 4.9 cameras per site. Daegu's other districts were between 1.2 and 2.0, and the city as a whole 1.9. An average of five cameras per site is not a normal setup, so I opened the source file. The rows whose managing agency is 대구광역시 서구청, the Seo-gu district office, carried four different reference dates.

Reference date	Rows	Rows whose address or coordinate is in the last list	Record IDs shared with the last list
2023-12-18	1,000	99.7%	0
2024-07-18	1,300	99.8%	0
2025-01-01	1,350	100%	0
2025-11-06	1,599	(last list)	-

Almost every row of the first three lists reappears in the last one; comparing road addresses alone still matches 92 to 99%. The list grew a little each year, and each new version was stacked next to the old ones instead of replacing them.

Viewcone merges rows with the same coordinate, purpose and agency into one site and adds up their camera counts, because many agencies split the cameras at one coordinate across several rows. With the same list in the file four times, that sum counted the same cameras four times, and any row whose coordinate had shifted slightly became a separate dot. That is where 1,060 sites and 5,246 cameras came from.

Why IDs and the update flag do not help

My first idea was to deduplicate on the record ID. None of the IDs in Seo-gu's first three lists appear in the last list: each upload numbers its rows afresh, so by ID every row is a different camera.

The update-type column looked promising until I counted it. All four Seo-gu lists carry I and the same update timestamp, 2026-05-15. Across the 29,223 rows the rule eventually removed, 29,220 are I and 3 are blank. Nothing in the file says that one list replaces another.

So the comparison had to work on uploads, checking whether one agency's list reappears, nearly whole, in a later list from the same agency.

Matching uploads instead of rows

The rule has three steps:

1. Group rows by agency, where an agency is the local-government code plus the agency name, and split each agency's rows by reference date. Each group is one upload.
2. For every upload, take all later uploads of the same agency and count the share of its rows that share a key with them. A key is the road address, the lot-number address, or the latitude and longitude rounded to 5 decimals (about 1 m).
3. If that share is at least 80% and the later uploads together hold at least 80% as many rows, treat the upload as superseded and leave it out.

These are the thresholds and the key functions from Viewcone's data pipeline, copied from the source:

export const SUPERSEDED_MATCH_SHARE = 0.8;
export const SUPERSEDED_LATER_SIZE = 0.8;

function coordinateKey(row: Row, config: SourceConfig) {
  const latitude = finiteNumber(row[config.fields.latitude]); const longitude = finiteNumber(row[config.fields.longitude]);
  return latitude == null || longitude == null ? null : `${latitude.toFixed(5)},${longitude.toFixed(5)}`;
}

function rowKeys(row: Row, config: SourceConfig) {
  const keys: string[] = [];
  for (const field of [config.fields.roadAddress, config.fields.lotAddress]) {
    const value = field ? text(row[field]) : "";
    if (value) keys.push(`a:${value}`);
  }
  const coordinate = coordinateKey(row, config);
  if (coordinate) keys.push(`c:${coordinate}`);
  return keys;
}

And the decision part of the loop (the lines that record each dropped upload are left out):

for (const [agency, byDate] of uploads) {
  const dates = [...byDate.keys()].sort();
  for (let index = 0; index < dates.length - 1; index++) {
    const later = dates.slice(index + 1).flatMap(date => byDate.get(date)!);
    const laterKeys = new Set(later.flatMap(row => rowKeys(row, config)));
    const rows = byDate.get(dates[index])!;
    const matched = rows.filter(row => rowKeys(row, config).some(key => laterKeys.has(key))).length;
    const matchShare = matched / rows.length;
    if (matchShare < SUPERSEDED_MATCH_SHARE || later.length < rows.length * SUPERSEDED_LATER_SIZE) continue;
    for (const row of rows) dropped.add(row);
  }
}

One shared key is enough. Using a single kind of key misses uploads: addresses alone find 35 of the 38, coordinates alone 29. In some agencies' re-uploads the coordinates changed, in others the address spelling, so the rule looks at both.

The agency name matters as much as the code. Hanam, in Gyeonggi Province, has one local-government code but several departments (information and communications, resource recycling, safety policy, water supply and others) that each upload on their own schedules, sometimes on the same day: three of them uploaded on 2024-01-30. Grouping by code alone merges different departments' lists that share a date into one upload, and the nationwide result changes to 34 uploads and 29,584 rows.

How sensitive is the 80% threshold?

Counting an agency as a code plus a name, 60 of 1,836 agencies use more than one reference date. Their uploads other than the last one, 122 in all, were compared with what came after them.

The shares cluster at the two ends. Under 20%, the later uploads look like additions of other cameras rather than a repeat of the list, so the upload stays. Of the 34 at 90% or more, 33 are re-uploads; the other is the Yeonggwang case below. Only 10 fall between 20% and 80%. I re-ran the rule with other match thresholds:

Match threshold	Uploads left out	Rows left out
50%	42	30,048
60%	40	30,016
70%	38	29,223
80% (in use)	38	29,223
90%	33	27,934
95%	30	25,641

Anywhere from 50% to 80%, the rows left out move by at most 825, and 70% and 80% give the same result. Above that the result drops faster: at 90%, the five uploads in the 80-90% band would stay, and their cameras would be counted twice.

A high match share that is not a re-upload

The second condition, that the later lists must be about as large, catches uploads whose match share is high for the wrong reason. The 2021-10-20 upload of the Yeonggwang county office (영광군청) has 390 rows that all carry the same coordinate (35.2677, 126.5516). The next upload's 32 rows use that coordinate too, so the match share is 100%, yet all later uploads together hold only 192 rows. That is not a re-upload of the same list, and it stays.

Some cases slip through both conditions. If an agency re-uploaded only part of a list, or both its addresses and coordinates changed, the match share drops and the old list stays. The Taebaek city office's (태백시청) 2024-07-01 upload has 10,296 rows on only 534 distinct coordinates, its later uploads hold 343 rows, and the match share is 63%. It falls short of both conditions and stays, though it may still contain re-uploaded rows. The same office's 2023-04-01 upload (5,048 rows, 100% match) was left out. Each new download gets the same check of which uploads are left out and which stay.

A check against a separate count

The source file alone cannot prove the rule right, so I compared Daegu with a different publication from the same ministry: its count of CCTV connected to local integrated control centers, as of 2025-12-31. That table lists Daegu on four lines, one of them "남구 (일부 통합관제)" (Nam-gu, partly integrated) with 12,324 cameras. A 2022 Boannews report says the city's shared control center houses the centers of six districts, all of the districts Daegu had then except Suseong-gu and Dalseong-gun, so I compared that line with those six districts combined.

The gap went from 31% to under 2%. The two sources count different things, though. Suseong-gu (3,740 cameras against 3,149), Dalseong-gun (1,680 against 2,109) and Gunwi-gun (1,242 against 940), which the rule did not touch, are 19 to 32% apart. I read the Daegu match as supporting evidence, not proof.

What changed on the map

Viewcone's web map has applied the rule since 2026-10-19, and the Android app has shown the same data since the 1.1.0 update. The national map went from 258,376 sites and 697,562 cameras to 253,980 sites and 640,238 cameras, and 48 cities, counties and districts changed. The 57,324 cameras removed are the camera counts written in the superseded uploads.

The dataset's other quirks (many records on one coordinate, placeholder coordinates, blank install dates) are covered in How Accurate Is Korea's Public CCTV Dataset?, and how far reference dates spread across agencies in How Fresh Is Korea's Public CCTV Data?.

---

Method

The whole CSV downloaded 2026-09-20 (CP949, 381,036 rows) was recounted with an independent Python re-implementation of the rule, separate from Viewcone's pipeline. Addresses were compared after collapsing whitespace; coordinates were rounded to 5 decimals. The result equals what the pipeline counted when the rule was applied to the map on 2026-10-19. Map figures are the Viewcone region-page values before and after the rule.

Sources

• National CCTV standard dataset (전국CCTV표준데이터), Ministry of the Interior and Safety, Public Data Portal (downloaded 2026-09-20)
• CCTV connected to local integrated control centers, as of 2025-12-31 (지방정부 통합관제센터 CCTV 운영 현황), Ministry of the Interior and Safety, Disaster and Safety Data Sharing Platform (read in its file preview 2026-10-24)
• Daegu Buk-gu CCTV integrated control center site visit, Boannews, 2022-10-04 (Korean)

FAQ

How do I check another dataset for re-uploaded lists quickly? If each row carries a publisher ID and a reference date, count the distinct reference dates per publisher first. A publisher with a single date cannot have a superseded upload, so the candidates shrink fast. For this file:

import csv
from collections import defaultdict

dates = defaultdict(set)
with open("national-cctv.csv", encoding="cp949", newline="") as f:
    for row in csv.DictReader(f):
        agency = (row["개방자치단체코드"], row["관리기관명"].strip())
        dates[agency].add(row["데이터기준일자"])

several = {a: sorted(d) for a, d in dates.items() if len(d) > 1}
print(len(dates), "agencies,", len(several), "with more than one reference date")

On the 2026-09-20 file it prints 1836 agencies, 60 with more than one reference date.