2026년 10월 18일 일요일

Find files recursively with Path.rglob() while excluding hidden paths

Suppose a script needs every .csv file under data, except files whose names or parent directories begin with a dot. In Python 3.14, Path.rglob() finds candidates throughout that tree. A filter then decides which candidates belong in the result. If the script must skip hidden directories during the search itself, use Path.walk() to prune them.

Find files recursively with Path.rglob() while excluding hidden paths
A reading map of the checks explained below.

Filter every component below the root

Here is a complete collection step: from pathlib import Path root = Path("data") files = sorted( path for path in root.rglob("*.csv") if path.is_file() and not any(part.startswith(".") for part in path.relative_to(root).parts) ) The relative_to(root).parts expression examines only the portion beneath data. It rejects data/.cache/report.csv and data/reports/.draft.csv as well as data/.summary.csv. It does not reject every descendant merely because the chosen root itself has a dot-prefixed name. The is_file() check limits the collection to files; by default, it also accepts a symlink whose target is a regular file. See Python 3.14’s Path.rglob() and Path.is_file() documentation: https://docs.python.org/3.14/library/pathlib.html#pathlib.Path.rglob and https://docs.python.org/3.14/library/pathlib.html#pathlib.Path.is_file.

Path.rglob("*.csv") searches at the root and below it, and its results retain the root prefix. The sorted() call gives the script a stable presentation order for the results it receives; the search itself supplies no particular order. See https://docs.python.org/3.14/library/pathlib.html#pathlib.Path.rglob.

Sources: pathlib — Object-oriented filesystem paths

Prune hidden directories before entering them

The filter above still lets the recursive search visit a hidden directory before discarding matches found there. For a search that skips those directories during descent, walk from the same root and edit the directory list: from pathlib import Path root = Path("data") files = [] for directory, dirnames, filenames in root.walk(): dirnames[:] = [name for name in dirnames if not name.startswith(".")] files.extend( directory / name for name in filenames if name.endswith(".csv") and not name.startswith(".") and (directory / name).is_file() ) files.sort() Editing dirnames in place during the default top-down walk prevents Path.walk() from descending into removed directories. The filename check handles hidden files in directories that remain. This example uses a literal .csv ending rather than a glob pattern; if the intended rule is more complex, define it explicitly where filenames are selected. See https://docs.python.org/3.14/library/pathlib.html#pathlib.Path.walk.

Path.walk() does not follow directory symlinks by default. With that default, a link to a directory is placed among filenames, and the is_file() check above leaves it out. If the search needs a different link policy, decide that separately from the hidden-name policy. See https://docs.python.org/3.14/library/pathlib.html#pathlib.Path.walk.

Sources: pathlib — Object-oriented filesystem paths

Choose the exclusion rule deliberately

A hidden path here means a component whose name starts with a dot. A directory called build is visible under that rule, even if the script should ignore it. Add such names to the dirnames filter when using Path.walk(), or add a separate component check when filtering rglob() results. A name like report.v2.csv is also visible because its dot is not at the start.

The examples treat data as the search boundary and allow the caller to choose even a dot-prefixed root. That choice avoids making a relative name test depend on parent directories outside the requested search. If the caller instead wants to reject a hidden root, check root.name before starting. These are application rules, so writing them beside the search makes later changes easier to review.

Sources: pathlib — Object-oriented filesystem paths

Know what a result list establishes

Pathlib wildcards can match dot-prefixed names, and recursive double-star expansion does not follow directory symlinks by default. The recurse_symlinks option changes that traversal choice. See the comparison with glob at https://docs.python.org/3.14/library/pathlib.html#comparison-to-the-glob-module and Path.rglob() at https://docs.python.org/3.14/library/pathlib.html#pathlib.Path.rglob.

Path.rglob() suppresses filesystem scanning errors, including permission errors. Path.walk() also ignores scanning errors by default, but its on_error argument can receive them. A short result list therefore does not establish that every directory was readable. Use on_error when the script needs to detect an incomplete walk. See https://docs.python.org/3.14/library/pathlib.html#pathlib.Path.walk.

Sources: pathlib — Object-oriented filesystem paths

Primary sources checked: 2026-10-17. Prepared with AI assistance and a separate evidence review. This is documentation-based guidance, not a hands-on test or performance benchmark.

Read the companion guide

Reproduce the locally verified result

pathlib-local-v1

Recorded environment: 3.12.13 / Darwin. Checked: 2026-11-05T05:15:43.358631+00:00

from pathlib import Path
from tempfile import TemporaryDirectory
with TemporaryDirectory() as tmp:
    root = Path(tmp)
    for name in ("visible/a.txt", ".cache/b.txt", "visible/.hidden.txt"):
        p = root / name
        p.parent.mkdir(parents=True, exist_ok=True)
        p.write_text("fixture", encoding="utf-8")
    (root / "folder.txt").mkdir()
    all_files = sorted(str(p.relative_to(root)) for p in root.rglob("*.txt") if p.is_file())
    selected = sorted(str(p.relative_to(root)) for p in root.rglob("*.txt")
                      if p.is_file() and not any(part.startswith(".") for part in p.relative_to(root).parts))
    print("all:", all_files)
    print("without dot paths:", selected)

Observed output

all: ['.cache/b.txt', 'visible/.hidden.txt', 'visible/a.txt']
without dot paths: ['visible/a.txt']

Only this fixed synthetic fixture was tested on the recorded runtime. No production files, other OS/version combinations, UI applications, throughput or performance were tested.

2026년 9월 6일 일요일

How do csv.writer and csv.reader round-trip commas, quotes and newlines?

With Python’s default csv dialect, csv.writer and csv.reader recovered all four tested string values unchanged: plain text, a comma, a quote, and an embedded newline. The table below shows the observed round trips; the complete runnable example follows in the article.

Verified round-trip results

Fixed synthetic example rechecked on 2026-10-22, Python 3.12.13 on Darwin
Input value (Python representation)Round trip equals inputIdentifier stays "00123"
'plain'TrueTrue
'comma, inside'TrueTrue
'a "quote"'TrueTrue
'two\nlines'TrueTrue

Tests use standard-library csv and synthetic strings. No Excel, pandas or third-party parser was executed. Do not generalize to every CSV dialect.

CSV quoting and text encoding solve different problems. If your columns parse but row["id"] fails, use the BOM header diagnosis. When choosing an export encoding, compare the UTF-8 and UTF-8 BOM read/write results.

CSV fields are not just comma splits: Comma in quotes — One CSV field; Doubled quote — One literal quote; Newline in quotes — One CSV field; 00123 — String preserved
Diagram of verified synthetic examples; details and limitations below.

The round-trip mechanism

Round-tripping means encoding a value with csv.writer and then decoding the result with csv.reader to get back the original value. csv.writer, a callable in the csv module, returns a writer object; calling writerow on that object examines each field before writing it: if the field contains a comma, a quotation mark, or a newline, writerow wraps the entire field in quotation marks, first doubling any literal quotation mark already inside it. csv.reader, the counterpart callable, returns a reader object; iterating over that object reverses this by stripping the surrounding quotation marks and converting any doubled quotation marks back into a single quotation mark.

Testing four values through this sequence — plain text, text containing a comma, text containing a quotation mark, and text containing a newline — showed each value recovered exactly as it was written, confirming the round trip for these specific cases.

Comma and newline handling

A field containing a comma is wrapped in quotation marks by csv.writer because an unquoted comma would otherwise be read as separating two fields. When csv.reader encounters that quoted field, it recognizes that the comma inside the quotation marks is part of the field's value rather than a delimiter.

A field containing a newline is handled the same way: csv.writer wraps it in quotation marks so the newline is understood as part of the field's content rather than the end of a row. In testing, a value containing a newline was correctly read back as a single field by csv.reader, rather than being split into two rows.

Quote escaping inside quoted fields

When a field's value itself contains a quotation mark, calling writerow on the writer object returned by csv.writer doubles that character and wraps the whole field in quotation marks. Iterating over the reader object returned by csv.reader detects a doubled quotation mark inside a quoted field and converts it back to a single quotation mark in the value it returns.

This doubling convention is the default escaping behavior for csv.writer and csv.reader in the csv module. In the tested case, a value containing a quotation mark was written, read back, and matched the original value exactly.

File handling and the newline setting

The csv module's documentation recommends opening files with the newline parameter set to an empty string when reading or writing CSV data, so that line-ending characters are passed through to csv.reader and csv.writer without being translated by the file layer first. This matters especially when a field's value legitimately contains a newline inside quotation marks.

The testing described here used an in-memory text buffer configured the same way, rather than an actual file on disk, so the effect of this setting on real files across different operating systems was not directly exercised. The recommendation to use this setting for files comes from the module's documentation rather than from the tests performed.

Verification through testing

To check that a round trip works, a value is passed to csv.writer, the resulting text is captured, and that text is then passed to csv.reader; the parsed result is compared to the original value. This sequence was run for the four values described earlier, and each one matched its original after being written and read back.

This kind of check is useful for confirming behavior on specific inputs. If a round trip fails for a particular value, the mismatch points to either how the file or buffer was opened or how that value interacts with the quoting and escaping behavior of csv.writer and csv.reader.

Practical considerations and limitations

These results come from running the csv module's reader and writer callables against a small set of synthetic string values in a single Python 3.12 environment, using the module's default dialect settings. Other libraries such as pandas, spreadsheet applications, and third-party CSV parsers were not tested.

Because only the default dialect was exercised, these findings should not be treated as evidence that csv.writer and csv.reader comply with any formal CSV specification, or that every CSV parser handles quoting and escaping the same way. Pairing output from csv.writer with a differently configured reader could still produce mismatched results.

Run the verified example

Python

import csv
import io

cases = ["plain", "comma, inside", 'a "quote"', "two\nlines"]
for value in cases:
    buffer = io.StringIO(newline="")
    csv.writer(buffer).writerow(["00123", value])
    encoded = buffer.getvalue()
    decoded = next(csv.reader(io.StringIO(encoded, newline="")))
    assert decoded == ["00123", value]
    print(repr(value), "round-trip:", decoded == ["00123", value])

Observed output

'plain' round-trip: True
'comma, inside' round-trip: True
'a "quote"' round-trip: True
'two\nlines' round-trip: True

Test scope and sources

Executed 2026-09-06 with Python 3.12.13 on Darwin. Tests use standard-library csv and synthetic strings. No Excel, pandas or third-party parser was executed. Do not generalize to every CSV dialect.

Read the companion article on this topic