File | Haobin Tan

Working with Files

Thu, 19 Nov 2020 00:00:00 +0000

`with open(...) as ...` pattern

open() opens file for reading or writing and returns a file handle
Use appropriate methods to read or write this file handel

Example

Read

with open('data.txt', 'r') as f:
 data = f.read()

Write

with open('data.txt', 'w') as f:
 data = 'some data to be written to the file'
 f.write(data)

Directory listing

Directory Listing in Legacy Python Versions

os.listdir()

returns a Python list containing the names of the files and subdirectories in the directory given by the path argument:

Example

import os

entries = os.listdir('my_directory/')

Directory Listing in Modern Python Versions

`os.scandir()`

returns an iterator

ROOT = 'my_directory/'
entries = os.scandir(ROOT)
entries

<posix.ScandirIterator at 0x7f6db713c3f0>

The ScandirIterator points to all the entries in the current directory. You can loop over the contents of the iterator and print out the filenames:
```
with os.scandir(ROOT) as entries:
 for entry in entries:
 print(f'{entry.name:10}: {entry.path}')
```

`pathlib` module

pathlib.Path() objects have an .iterdir() method for creating an iterator of all files and folders in a directory. Each entry yielded by .iterdir() contains information about the file or directory such as its name and file attributes.
pathlib offers a set of classes featuring most of the common operations on paths in an easy, object-oriented way. Another benefit of using pathlib over os is that it reduces the number of imports you need to make to manipulate filesystem paths. 👏

Example

from pathlib import Path

entries = Path(ROOT)
for entry in entries.iterdir():
 print(f'{entry.name}, name: {entry.stem}, ext: {entry.suffix}')

Summary

Function	Description
`os.listdir()`	Returns a list of all files and folders in a directory
`os.scandir()`	Returns an iterator of all the objects in a directory including file attribute information
`pathlib.Path.iterdir()`	Returns an iterator of all the objects in a directory including file attribute information

Listing all files in a directory

Filter out directories and only list files from a directory listing

Use `os.listdir()`

import os

basepath = 'my_directory'
for entry in os.listdir(basepath):
 if os.path.isfile(os.path.join(basepath, entry)):
 print(entry)

Use `os.scandir()`

Using os.scandir() has the advantage of looking cleaner and being easier to understand than using os.listdir()

import os

basepath = 'my_directory'
with os.scandir(basepath) as entries:
 for entry in entries:
 if entry.is_file():
 print(entry.name)

Use `pathlib.Path()`

from pathlib import Path

basepath = 'my_directory'
files_in_basepath = Path(basepath).iterdir()
for item in files_in_basepath:
 if item.is_file():
 print(item.name)

The code above can be made more concise if we combine the for loop and the if statement into a single generator expression

from pathlib import Path

basepath = 'my_directory'
files_in_basepath = (entry for entry in Path(basepath).iterdir() if entry.is_file())
for item in files_in_basepath:
 print(item.name)

Listing subdirectories

Use `os.listdir()` and

import os

basepath = 'my_directory'
sub_dirs = (entry for entry in os.listdir(basepath)
 if os.path.isdir(os.path.join(basepath, entry)))

for sub_dir in sub_dirs:
 print(sub_dir)

Use `os.scandir()`

import os

basepath = 'my_directory'
with os.scandir(basepath) as entries:
 for entry in entries:
 if entry.is_dir():
 print(entry.name)

Use `pathlib.Path`

from pathlib import Path

basepath = 'my_directory'
sub_dirs = (item for item in Path(basepath).iterdir() if item.is_dir())
for sub_dir in sub_dirs:
 print(sub_dir.name)

Making directories

os and pathlib include functions for creating directories. We’ll consider these:

Function	Description
`os.mkdir()`	Creates a single subdirectory
`pathlib.Path.mkdir()`	Creates single or multiple directories
`os.makedirs()`	Creates multiple directories, including intermediate directories

Making single directories

Use `os.mkdir()`

To create a single directory, pass a path to the directory as a parameter to os.mkdir():

import os

os.mkdir('example_directory/')

Note: If a directory already exists, os.mkdir() raises FileExistsError.

Use `pathlib`

from pathlib import Path

p = Path('example_directory/')
p.mkdir()

If the path already exists, Path.mkdir() raises a FileExistsError.

To avoid the error, catch the error when it happens and let the user know:

from pathlib import Path

p = Path('example_directory/')
try:
 p.mkdir()
except FileExistsError as err:
 print(err)

Alternatively, you can ignore the FileExistsError by passing the exist_ok=True argument to .mkdir(). This will not raise an error if the directory already exists.

from pathlib import Path

p = Path('example_directory/')
p.mkdir(exist_ok=True)

Creating Multiple Directories

Use `os.makedirs`

os.makedirs()

similar to os.mkdir()
The difference between the two is that not only can os.makedirs() create individual directories, it can also be used to create directory trees. In other words, it can create any necessary intermediate folders in order to ensure a full path exists.
similar to running mkdir -p in Bash.

import os

os.makedirs('dir/sub_dir/sub_sub_dir')

This will create a nested directories with default permissions:

.
|
└── dir/
 └── sub_dir/
 └── sub_sub_dir/

Use `pathlib.Path`

import pathlib

p = pathlib.Path('dir/sub_dir/sub_sub_dir')
p.mkdir(parents=True, exist_ok=True)

I prefer using pathlib when creating directories because I can use the same function to create single or nested directories.

Filename Pattern Matching

Function	Description
`startswith()`	Tests if a string starts with a specified pattern and returns `True`or `False`
`endswith()`	Tests if a string ends with a specified pattern and returns `True` or `False`
`fnmatch.fnmatch(filename, pattern)`	Tests whether the filename matches the pattern and returns `True` or `False`
`glob.glob()`	Returns a list of filenames that match a pattern
`pathlib.Path.glob()`	Finds patterns in path names and returns a generator object

Traversing Directories and Processing Files

`os.walk()`

os.walk() defaults to traversing directories in a top-down manner.

To traverse the directory tree in a bottom-up manner, pass in a topdown=False keyword argument

os.walk() returns three values on each iteration of the loop:

The name of the current folder
A list of folders in the current folder
A list of files in the current folder

Deleting files

Function	Note
`os.remove()`	Will throw an `OSError` if the path passed to them points to a directory instead of a file
`os.unlink()`	semantically identical to `os.remove()`
`pathlib.Path.unlink()`

Deleting Directories

Delete single directory

Function	Note
`os.rmdir()`	Only work if the directory you’re trying to delete is empty. If the directory isn’t empty, an `OSError` is raised.
`pathlib.rmdir()`	semantically identical to `os.rmdir()`

Delete entire directory trees

shutil.rmtree(dir): Everything in dir is deleted when shutil.rmtree() is called on it.

Summary for deleting

Function	Description
`os.remove()`	Deletes a file and does not delete directories
`os.unlink()`	Is identical to `os.remove()` and deletes a single file
`pathlib.Path.unlink()`	Deletes a file and cannot delete directories
`os.rmdir()`	Deletes an empty directory
`pathlib.Path.rmdir()`	Deletes an empty directory
`shutil.rmtree()`	Deletes entire directory tree and can be used to delete non-empty directories

Copying

Copying files

Function Note

shutil.copy() comparable to the cp command in UNIX based systems.
shutil.copy(src, dst) will copy the file src to the location specified in dst. If dstis a file, the contents of that file are replaced with the contents of src. If dst is a directory, then src will be copied into that director
shutil.copy() ONLY copies the file’s contents and the file’s permissions. Other metadata like the file’s creation and modification times are not preserved.

shutil.copy2() preserve all file metadata when copying

Function	Note
`shutil.copy()`	comparable to the `cp` command in UNIX based systems. `shutil.copy(src, dst)` will copy the file `src` to the location specified in `dst`. If `dst`is a file, the contents of that file are replaced with the contents of `src`. If `dst` is a directory, then `src` will be copied into that director `shutil.copy()` ONLY copies the file’s contents and the file’s permissions. Other metadata like the file’s creation and modification times are not preserved.
`shutil.copy2()`	preserve all file metadata when copying

Copying directories

shutil.copytree()

Moving Files and Directories

To move a file or directory to another location, use shutil.move(src, dst).

src: file or directory to be moved
dst: destination. If dst does not exist, src will be renamed to dst

Renaming Files and Directories

os.rename(src, dst)
pathlib.Path.rename()

Colab Notebook

Reference

Working with Files in Python

pathlib

Sat, 26 Dec 2020 00:00:00 +0000

Why `patblib`?

The pathlib module, introduced in Python 3.4 (PEP 428), gathers the necessary functionality in one place and makes it available through methods and properties on an easy-to-use Path object.

from pathlib import Path

Createing paths

Use class methods:
- Path.cwd(): get Current Working Directory
- Path.home(): get home directory
Create from string representation explicitly

E.g.
```
Path("/content/src/demo.py")
```

Construct path by joining parts of the path

E.g.

Path.joinpath(Path.cwd(), "src", "demo.py")

Reading and writing files

The built-in open() function can use Path objects directly. E.g.:

with open(path, mode="r") as f:
 pass

For simple reading and writing of files, there are a couple of convenience methods in the pathlib library:

.read_text(): open the path in text mode and return the contents as a string.
.read_bytes(): open the path in binary/bytes mode and return the contents as a bytestring.
.write_text(): open the path and write string data to it.
.write_bytes(): open the path in binary/bytes mode and write data to it.

Each of these methods handles the opening and closing of the file, making them trivial to use.

Components of path

The different parts of a path are conveniently available as properties. Basic examples include:

.name: the file name without any directory
.parent: the directory containing the file, or the parent directory if path is a directory
.stem: the file name without the suffix
.suffix: the file extension
.anchor: the part of the path before the directories

path = Path.joinpath(Path.cwd(), "src", "hello-world.txt")

print(f"Full path: {path}")
print(f"Name: {path.name}")
print(f"Parent: {path.parent}")
print(f"Stem: {path.stem}")
print(f"Suffix: {path.suffix}")
print(f"Anchor: {path.anchor}")

Full path: /content/src/hello-world.txt
Name: hello-world.txt
Parent: /content/src
Stem: hello-world
Suffix: .txt
Anchor: /

Renaming files

When you are renaming files, useful methods might be .with_name() and .with_suffix(). They both return the original path but with the name or the suffix replaced, respectively.

Deleting files

Directories and files can be deleted using .rmdir() and .unlink() respectively.

Listing files

Path.iterdir() : iterates over all files in the given directory
More flexible file listing
- Path.glob()
- Path.rglob() (recursive glob)

Moving files

Path(current_location).rename(new_location)

Correspondence to tools in the `os` module

os and os.path	pathlib
`os.path.abspath()`	`Path.resolve()`
`os.chmod()`	`Path.chmod()`
`os.mkdir()`	`Path.mkdir()`
`os.makedirs()`	`Path.mkdir(parents=True)`
`os.rename()`	`Path.rename()`
`os.replace()`	`Path.replace()`
`os.rmdir()`	`Path.rmdir()`
`os.remove()`, `os.unlink()`	`Path.unlink()`
`os.getcwd()`	`Path.cwd()`
`os.path.exists()`	`Path.exists()`
`os.path.expanduser()`	`Path.expanduser()` and `Path.home()`
`os.listdir()`	`Path.iterdir()`
`os.path.isdir()`	`Path.is_dir()`
`os.path.isfile()`	`Path.is_file()`
`os.path.islink()`	`Path.is_symlink()`
`os.link()`	`Path.link_to()`
`os.symlink()`	`Path.symlink_to()`
`os.readlink()`	`Path.readlink()`
`os.stat()`	`Path.stat()`, `Path.owner()`, `Path.group()`
`os.path.isabs()`	`PurePath.is_absolute()`
`os.path.join()`	`PurePath.joinpath()`
`os.path.basename()`	`PurePath.name`
`os.path.dirname()`	`PurePath.parent`
`os.path.samefile()`	`Path.samefile()`
`os.path.splitext()`	`PurePath.suffix`

`pathlib` cheatsheet

pathlib cheatsheet

Google Colab Notebook

Open in Colab

Reference

Documentation: pathlib — Object-oriented filesystem paths
Python 3’s pathlib Module: Taming the File System

glob

Fri, 25 Dec 2020 00:00:00 +0000

The glob module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell. If you need a list of filenames that all have a certain extension, prefix, or any common string in the middle, use glob instead of writing code to scan the directory contents yourself.

Wildcards

* : matches zero or more characters in a segment of a name
? : matches any single character in that position in the name
[] : matches characters in the given range
- [1-9] : matches any single digit
- [a-z : mathces any single letter

Functions

`glob.glob()`

glob(file_pattern, recursive = False)

It retrieves the list of files matching the specified pattern in the file_pattern parameter.

The file_pattern can be an absolute or relative path. It may also contain wild cards such as * or ? symbols.
The recursive parameter is turn off (False) by default. When True, it recursively searches files under all subdirectories of the current directory.

For example, we have file structure like this:

- src
|- code
 |- main.py
 |- test0.py
 |- test1.py
 |- test2.py
 |- test-env.py
 |- test-prod.py
|- text
 |- file_a.txt
 |- file_b.txt
 |- file_c.txt
 |- file_d.txt
|- demo.py

We want to get all .py files:

import glob

for py_file in glob.glob("src/**/*.py", recursive=True):
 print(py_file)

src/demo.py
src/code/test1.py
src/code/test-env.py
src/code/main.py
src/code/test2.py
src/code/test0.py

We want to get test0.py, test1.py, test2.py:

import glob

for py_file in glob.glob("src/**/test?.py"):
 print(py_file)

src/code/test1.py
src/code/test2.py
src/code/test0.py

We want to get file_a.txt, file_b.txt, file_c.txt:

import glob

for txt_file in glob.glob("src/**/file_[a-c].txt"):
 print(txt_file)

src/text/file_a.txt
src/text/file_c.txt
src/text/file_b.txt

`glob.iglob()`

Return an iterator which yields the same values as glob() without actually storing them all simultaneously (good for large directories).

Example:

import glob

for py_file in glob.iglob("src/**/test?.py"):
 print(py_file)

src/code/test1.py
src/code/test2.py
src/code/test0.py

`glob.escape()`

Escape all special characters ('?', '*' and '['). “Escape” means treating special characters as normal character instead of wildcards.

This is useful if you want to match an arbitrary literal string that may have special characters in it.

For example, if we want to get test-env.py and test-prod.py (both contain special character - in the filename):

import glob

for py_file in glob.glob("src/code/*" + glob.escape("-") + "*.py"):
 print(py_file)

Another ways for filename pattern matching

`fnmatch.fnmatch()` ¹

fnmatch.fnmatch(filename, pattern)

Test whether the filename string matches the pattern string.

For example, we want to get test0.py, test1.py, test2.py:

import os
import fnmatch

for py_file in os.listdir("src/code"):
 if fnmatch.fnmatch(py_file, "test?.py"):
 print(py_file)

`pathlib.Path.glob()` ²

pathlib.Path.glob(pattern)

Glob the given relative pattern in the directory represented by this path, yielding all matching files (of any kind).

For example, if we want to list all python file:

from pathlib import Path

path = Path("src")

for py_file in path.glob("**/*.py"):
 print(py_file)

src/demo.py
src/code/test1.py
src/code/test-prod.py
src/code/test-env.py
src/code/main.py
src/code/test2.py
src/code/test0.py

The “**” pattern means “this directory and all subdirectories, recursively”. In other words, it enables recursive globbing. However, using the “**” pattern in large directory trees may consume an inordinate amount of time.

Path.glob() is similar to os.glob() discussed above. As you can see, pathlib combines many of the best features of the os, os.path, and glob modules into one single module, which makes it a joy to use.

`pathlib.Path.rglob()` ³

This is like calling Path.glob() with “**/” added in front of the given relative pattern.

E.g. list all python file:

for py_file in Path("src").rglob("*.py"):
 print(py_file)

src/demo.py
src/code/test1.py
src/code/test-prod.py
src/code/test-env.py
src/code/main.py
src/code/test2.py
src/code/test0.py

Summary

Function	Description
`fnmatch.fnmatch(filename, pattern)`	Tests whether the filename matches the pattern and returns `True` or `False`
`glob.glob()`	Returns a list of filenames that match a pattern
`glob.iglob()`	Returns an iterator of filenames that match a pattern
`pathlib.Path.glob()`	Finds patterns in path names and returns a generator object
`pathlib.Path.rglob()`	Finds patterns in path names recursively and returns a generator object

File | Haobin Tan

Working with Files

with open(...) as ... pattern

Directory listing

Directory Listing in Legacy Python Versions

Directory Listing in Modern Python Versions

os.scandir()

pathlib module

Summary

Listing all files in a directory

Use os.listdir()

Use os.scandir()

Use pathlib.Path()

Listing subdirectories

Use os.listdir() and

Use os.scandir()

Use pathlib.Path

Making directories

Making single directories

Use os.mkdir()

Use pathlib

Creating Multiple Directories

Use os.makedirs

Use pathlib.Path

Filename Pattern Matching

Traversing Directories and Processing Files

os.walk()

Deleting files

Deleting Directories

Delete single directory

Delete entire directory trees

Summary for deleting

Copying

Copying files

Copying directories

Moving Files and Directories

Renaming Files and Directories

Colab Notebook

Reference

pathlib

Why patblib?

Createing paths

Reading and writing files

Components of path

Renaming files

Deleting files

Listing files

Moving files

Correspondence to tools in the os module

pathlib cheatsheet

Google Colab Notebook

Reference

glob

Wildcards

Functions

glob.glob()

glob.iglob()

glob.escape()

Another ways for filename pattern matching

fnmatch.fnmatch() 1

pathlib.Path.glob() 2

pathlib.Path.rglob() 3

Summary

Reference

`with open(...) as ...` pattern

`os.scandir()`

`pathlib` module

Use `os.listdir()`

Use `os.scandir()`

Use `pathlib.Path()`

Use `os.listdir()` and

Use `os.scandir()`

Use `pathlib.Path`

Use `os.mkdir()`

Use `pathlib`

Use `os.makedirs`

Use `pathlib.Path`

`os.walk()`

Why `patblib`?

Correspondence to tools in the `os` module

`pathlib` cheatsheet

`glob.glob()`

`glob.iglob()`

`glob.escape()`

`fnmatch.fnmatch()` ¹

`pathlib.Path.glob()` ²

`pathlib.Path.rglob()` ³