UTF-8 vs UTF-8 BOM for CSV: A Tested Python Read/Write Matrix

Use the receiving system’s encoding contract first. In our Python CSV test, utf-8-sig read both plain UTF-8 and BOM-prefixed UTF-8 with a clean first header. A plain utf-8 reader kept the BOM in that header. The difference was a three-byte prefix, not different Korean text or a changed identifier.

All four combinations preserve cell values. The failure is an exact header lookup, not lost row data.

This reference compares four read/write combinations using one controlled fixture. It answers an export-design question: which UTF-8 variant should a producer write, and what should the consumer read? Tests ran on Python 3.12.13 on macOS on September 6, 2026. Excel behavior below is attributed to Microsoft, not presented as a local Excel test.

On this page

• The fixture and what was checked
• Observed results: four combinations
• Run the same comparison
• Choose the producer and consumer settings separately
• Writing a new CSV for an Excel recipient
• Limits: encoding is only one part of CSV compatibility

The fixture and what was checked

id,name,note
00123,서울,"comma, quote ""yes"""

The actual test string uses CRLF line endings. We check the first header, a leading-zero identifier, Korean text, and a quoted field containing a comma and doubled quotes. Each input is decoded and then parsed with csv.DictReader.

Observed results: four combinations

Python 3.12.13 CSV read/write results

Write encoding	Read encoding	First header (repr)	row["id"]
utf-8	utf-8	'id'	'00123'
utf-8	utf-8-sig	'id'	'00123'
utf-8-sig	utf-8	'\ufeffid'	KeyError
utf-8-sig	utf-8-sig	'id'	'00123'

All four combinations preserved the cell values, including 서울 and comma, quote "yes". Only the third combination failed the exact id lookup. In that row the value still existed under the prefixed key. Plain UTF-8 began with 69 64 2c (id,); BOM-prefixed UTF-8 began with ef bb bf.

Run the same comparison

import csv
import io

text = 'id,name,note\r\n00123,서울,"comma, quote ""yes"""\r\n'
for write_encoding in ("utf-8", "utf-8-sig"):
    raw = text.encode(write_encoding)
    for read_encoding in ("utf-8", "utf-8-sig"):
        with io.TextIOWrapper(io.BytesIO(raw), encoding=read_encoding,
                              newline="") as f:
            reader = csv.DictReader(f)
            row = next(reader)
            first = reader.fieldnames[0]
        assert row[first] == "00123"
        assert row["name"] == "서울"
        assert row["note"] == 'comma, quote "yes"'
        print(write_encoding, read_encoding, repr(first), "id" in row)

Choose the producer and consumer settings separately

• A documented data pipeline: follow its required encoding. If you control both sides and need no signature, plain UTF-8 on both sides passed this test.
• A Python importer accepting either UTF-8 variant: utf-8-sig passed both inputs. This is a narrow compatibility choice, not an encoding detector.
• A CSV intended for normal opening in Excel: Microsoft says UTF-8 CSV files can open normally when saved with a BOM. It also describes an import route for files without one. Test the intended Excel version and import workflow before promising compatibility.

Sources: Microsoft: opening UTF-8 CSV files in Excel; Python: UTF-8 signature codec.

Writing a new CSV for an Excel recipient

import csv

with open("export-for-excel.csv", "w", encoding="utf-8-sig", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["id", "name", "note"])
    writer.writerow(["00123", "서울", 'comma, quote "yes"'])

Run this in a folder where that output name is unused: write mode replaces an existing file. For a consumer requiring no BOM, change only the encoding to utf-8. The standard-library CSV writer handles the comma and embedded quotes; avoid building rows with a plain string join.

Limits: encoding is only one part of CSV compatibility

The Python result 00123 does not establish what a spreadsheet will infer when opening the file. This experiment tests decoding and CSV parsing, not spreadsheet cell types. We also checked that both readers reject an invalid UTF-8 byte, and that a U+FEFF inside a cell survives BOM-aware decoding. Delimiters, schema validation, and spreadsheet import choices remain separate decisions.

Already debugging a failed header lookup? Use the focused Python CSV KeyError diagnosis and fix to distinguish an initial BOM from whitespace or a wrong delimiter.

Continue with the next CSV problem

• Fix a missing id header — Diagnose an invisible BOM, a wrong delimiter, or a header mismatch.
• Read quoted commas correctly — Replace comma splitting with a CSV reader.
• Write and read quoted fields — Check commas, quotes, and embedded newlines with a round-trip example.

Python CSV KeyError: 'id' — Diagnose and Fix a UTF-8 BOM Header

If Python raises KeyError: 'id' even though the CSV appears to start with id, inspect the header before changing your data. A UTF-8 byte order mark (BOM) can turn that first key into '\ufeffid'. For that specific case, read the file with encoding="utf-8-sig".

The same bytes produce different first-header keys. Diagram based on the verified Python example.

This walkthrough reproduces the failure and the fix using Python 3.12.13 on macOS, tested on September 6, 2026. The example uses synthetic data and only the standard library.

Quick diagnosis

Check the first header with repr(). If it is \ufeffid, use utf-8-sig. If it is id;name or id, follow the delimiter or whitespace checks below.

Match the symptom to the fix

Inspect reader.fieldnames before changing the file

Observed header	Next step
\ufeffid	Read with encoding="utf-8-sig".
id;name	Confirm the separator, then set delimiter=";".
id or ID	Check spaces and case against the expected schema.
No headers	Check for an empty or failed export.

On this page

• 1. See the actual key, including invisible characters
• 2. Reproduce the error without downloading a file
• 3. Read the BOM-aware way and validate the header
• 4. If it still fails, follow the header evidence
• What this fix does not do

1. See the actual key, including invisible characters

Run this against your CSV. The repr() output makes the invisible prefix visible. Reading the first three bytes is a separate check: ef bb bf identifies the UTF-8 signature at the start of this file.

from pathlib import Path
import csv

path = Path("input.csv")
with path.open("rb") as f:
    print("First bytes:", f.read(3).hex(" "))
with path.open(encoding="utf-8", newline="") as f:
    reader = csv.DictReader(f)
    print("Headers:", [repr(x) for x in (reader.fieldnames or [])])

2. Reproduce the error without downloading a file

import csv
import io

raw = "id,name\r\n00123,서울\r\n".encode("utf-8-sig")
with io.TextIOWrapper(io.BytesIO(raw), encoding="utf-8", newline="") as f:
    row = next(csv.DictReader(f))
    print(repr(next(iter(row))))
    try:
        print(row["id"])
    except KeyError as error:
        print(type(error).__name__, str(error))

Observed output:

'\ufeffid'
KeyError 'id'

The row exists. The lookup fails because id and \ufeffid are different strings. In this reproduction, neither a missing row nor damaged Korean text causes the error.

3. Read the BOM-aware way and validate the header

import csv

with open("input.csv", encoding="utf-8-sig", newline="") as f:
    reader = csv.DictReader(f)
    headers = reader.fieldnames or []
    if "id" not in headers:
        raise ValueError(f"Expected an id column; got {headers!r}")
    for row in reader:
        print(row["id"])

For the sample above, the value is 00123. Our test also confirmed that the same reader handles the UTF-8 sample without a BOM. It does not turn the identifier into a number.

Python documents how utf-8-sig skips an initial UTF-8 signature. Its CSV documentation explains header-based dictionaries and the recommended newline="" file setting.

4. If it still fails, follow the header evidence

• ['id;name']: your comma reader sees one column. If the file uses semicolons, pass delimiter=";".
• [' id', 'name']: the first header has a space. Confirm the intended schema before trimming or renaming columns.
• ['ID', 'name']: the spelling differs by case. Use the actual key or an explicit, checked mapping.
• An empty list: there is no header to map. Check whether the file is empty or whether the export failed.

The wrong-delimiter and leading-space cases were included in our tests: switching to utf-8-sig did not fix either one. Avoid silently falling back to row.get("id", "") before diagnosing the schema; that would hide the failed lookup.

What this fix does not do

It does not detect every encoding, repair invalid UTF-8 bytes, or remove a U+FEFF character embedded in a cell. We tested invalid UTF-8 and an interior U+FEFF separately: decoding failed for the former and preserved the latter. Keep the original export while diagnosing the problem.

For choosing the encoding of a new export, see UTF-8 vs UTF-8 BOM for CSV: a tested Python read/write matrix.

Continue with the next CSV problem

• Choose UTF-8 or UTF-8 BOM — Compare Python read/write settings before exporting a file.
• Read quoted commas correctly — Replace comma splitting with a CSV reader.
• Write and read quoted fields — Check commas, quotes, and embedded newlines with a round-trip example.

어고 커플링 구매후기!!

최근에 압구정 현대백화점에서 진행 중인 어고 팝업 스토어에 다녀왔습니다. 온라인으로만 보던 브랜드였는데, 직접 보고 착용해볼 수 있는 기회라 방문하게 되었어요. 팝업 공간은 전체적으로 깔끔하고 정돈된 분위기였고, 브랜드 특유의 차분한 무드가 잘 느껴졌습니다.

가장 인상 깊었던 점은 직원분의 응대였습니다. 단순히 제품을 설명하는 데 그치지 않고, 반지의 디자인 의도나 착용 시 느낌 등을 차분하게 안내해주셔서 편하게 둘러볼 수 있었어요. 부담스럽게 구매를 권하는 분위기가 아니라, 궁금한 점을 물어보면 하나하나 친절하게 설명해주셔서 팝업 방문 자체에 대한 만족도가 높았습니다.

어고의 반지들은 전반적으로 과하지 않고, 일상에서 자연스럽게 착용하기 좋은 디자인이 많았습니다. 직접 착용해보니 사진으로 봤을 때보다 손에 올라갔을 때의 분위기가 더 잘 느껴졌고, 은은한 디테일들이 눈에 들어오더라고요. 이런 부분은 역시 오프라인에서 직접 보는 게 좋다는 생각이 들었습니다.

기념일이나 특별한 날을 앞두고 있다면, 압구정 현대백화점 팝업을 한 번 방문해보는 것도 괜찮을 것 같아요. 직접 착용해보고 비교해볼 수 있어서 여자친구선물을 고민 중인 분들께 특히 도움이 될 것 같았습니다. 전체적으로 편안한 분위기에서 브랜드를 경험할 수 있었던 방문이었어요.

조용하지만 탄탄한 인상을 남긴 팝업이었고, 디자인뿐 아니라 응대까지 포함해 브랜드 이미지가 잘 전달된 공간이었습니다. 의미 있는 여자친구선물이나 기념 아이템을 찾고 있다면, 이런 팝업 방문도 좋은 선택지가 될 수 있을 것 같습니다.

https://www.agojewelry.com/