Basics | Haobin Tan

Getting Started

Sun, 30 Aug 2020 00:00:00 +0000

Source

This is the summary of the book “A Whirlwind Tour of Python” by Jake VanderPlas.

You can view it in

nbviewer: A whirlwind Tour of Python, or
Github: A whirlwind Tour of Python

import this

The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Operators

Identity and Membership

The identity operators, is and is not check for object identity. Object identity is different than equality.

a = [1, 2, 3]
b = [1, 2, 3]

a == b

True

a is b

False

1 in a

True

4 in a

False

Built-in Types

Python’s simple types:

Type	Example	Description
`int`	`x = 1`	integers (i.e., whole numbers)
`float`	`x = 1.0`	floating-point numbers (i.e., real numbers)
`complex`	`x = 1 + 2j`	Complex numbers (i.e., numbers with real and imaginary part)
`bool`	`x = True`	Boolean: True/False values
`str`	`x = 'abc'`	String: characters or text
`NoneType`	`x = None`	Special object indicating nulls

Complex Numbers

complex(1, 2)

(1+2j)

Alternatively, we can use the “j” suffix in expressions to indicate the imaginary part:

1 + 2j

(1+2j)

c = 3 + 4j

c.real

3.0

c.imag

4.0

c.conjugate()

(3-4j)

abs(c)

5.0

String Type

msg = "what do you like?" # double quotes
response = 'spam' # single quotes

# length
len(response)

# Upper/lower case
response.upper()

'SPAM'

# Capitalize, see also str.title()
msg.capitalize()

'What do you like?'

# concatenation with +
msg + response

'what do you like?spam'

# multiplication is multiple concatenation
5 * response

'spamspamspamspamspam'

# Access individual characters (zero-based (list) indexing)

msg[0]

'w'

None Type

Most commonly used as the default return value of a function

type(None)

NoneType

ret_val = print("abc")

abc

print(ret_val)

None

Likewise, any function in Python with no return value is, in reality, returning None.

Boolean

Booleans can also be constructed using the bool() object constructor: values of any other type can be converted to Boolean via predictable rules

any numeric type is False if equal to zero, and True otherwise
The Boolean conversion of None is always False
For strings, bool(s) is False for empty strings and True otherwise
For sequences, the Boolean representation is False for empty sequences and True for any other sequences

# numeric type
bool(0)

False

bool(1)

True

a = 0
if not a:
 print("a")

bool(None)

False

bool("")

False

bool("Hello World!")

True

bool([])

False

bool([1])

True

l_1 = [1, 2, 3]
l_2 = []

def is_empty(l):
 if l:
 print("not empty")
 return False
 else:
 print("empty")
 return True

is_empty([1, 2, 3])

not empty
False

is_empty([])

empty
True

Built-In Data Structures

Type Name	Example	Description
`list`	`[1, 2, 3]`	Ordered collection
`tuple`	`(1, 2, 3)`	Immutable ordered collection
`dict`	`{'a':1, 'b':2, 'c':3}`	Unordered (key,value) mapping
`set`	`{1, 2, 3}`	Unordered collection of unique values

Defining and Using Functions

`*args` and `**kwargs`

Write a function in which we don’t initially know how many arguments the user will pass.

*args:
- * before a variable means “expand this as a sequence”
- args is short for “arguments”
**kwargs
- ** before a variable means “expand this as a dictionary”
- kwargs is short for “keyword arguments”

def catch_all(*args, **kwargs):
 print("args = ", args)
 print("kwargs = ", kwargs)

catch_all(1, 2, 3, a=4, b=5)

args = (1, 2, 3)
kwargs = {'a': 4, 'b': 5}

inputs = (1, 2, 3)
keywords = {"one": 1, "two": 2}

catch_all(*inputs, **keywords)

args = (1, 2, 3)
kwargs = {'one': 1, 'two': 2}

Iterators

`enumerate`

“Pythonic” way to enumerate the indices and values in a list.

l = [2, 4, 6, 8, 10]
for i, val in enumerate(l):
 print("index: {}, value: {}".format(i, val))

index: 0, value: 2
index: 1, value: 4
index: 2, value: 6
index: 3, value: 8
index: 4, value: 10

`zip`

Iterate over multiple lists simultaneously

L = [1, 3, 5, 7, 9]
R = [2, 4, 6, 8, 10]

for l_val, r_val in zip(L, R):
 print("L: {}, R: {}".format(l_val, r_val))

L: 1, R: 2
L: 3, R: 4
L: 5, R: 6
L: 7, R: 8
L: 9, R: 10

for i, val in enumerate(zip(L, R)):
 print("Index: {}, L: {}, R: {}".format(i, val[0], val[1]))

Index: 0, L: 1, R: 2
Index: 1, L: 3, R: 4
Index: 2, L: 5, R: 6
Index: 3, L: 7, R: 8
Index: 4, L: 9, R: 10

`map` and `filter`

map: takes a function and applies it to the values in an iterator

func = lambda x: x + 1
l = [1, 2, 3, 4, 5]

list(map(func, l))

[2, 3, 4, 5, 6]

filter: only passes-through values for which the filter function evaluates to True

is_even = lambda x: x % 2 == 0
list(filter(is_even, l))

[2, 4]

Iterators as function arguments

It turns out that the *args syntax works not just with sequences, but with any iterator:

print(*range(5))

0 1 2 3 4

list(range(3))

[0, 1, 2]

print(*map(lambda x: x + 1, range(3)))

1 2 3

L1 = [1, 2, 3, 4]
L2 = ["a", "b", "c", "d"]

z = zip(L1, L2)
print(*z)

(1, 'a') (2, 'b') (3, 'c') (4, 'd')

z = zip(L1, L2)
new_L1, new_L2 = zip(*z)
new_L1

(1, 2, 3, 4)

new_L2

('a', 'b', 'c', 'd')

Specialized Iterators: `itertools`

from itertools import permutations

p = permutations(range(3))
print(*p)

(0, 1, 2) (0, 2, 1) (1, 0, 2) (1, 2, 0) (2, 0, 1) (2, 1, 0)

<itertools.permutations at 0x10fb32710>

List Comprehensions

l = [1, 2, 3, 4, 5]

[2 * el for el in l if el > 3]

[8, 10]

which is equivalent to the loop syntax, but list comprehension is much easier to write and to understand!

L = []
for el in l:
 if el > 3:
 L.append(2 * el)

L

[8, 10]

[(i, j) for i in range(2) for j in range(3)]

[(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2)]

print(*range(10))

# Leave out multiples of 3, and negate all multiples of 2
[val if val % 2 else -val for val in range(10) if val % 3]

0 1 2 3 4 5 6 7 8 9
[1, -2, -4, 5, 7, -8]

L = []

for val in range(10):
 if val % 3 != 0: # conditional on iterator 
 # conditional on value
 if val % 2 != 0:
 L.append(val)
 else:
 L.append(-val)

L

[1, -2, -4, 5, 7, -8]

{n * 2 for n in range(5)}

{0, 2, 4, 6, 8}

{a % 3 for a in range(100)}

{0, 1, 2}

Generators

Difference between list comprehensions and generator expressions:

List comprehensions use square brackets, while generator expressions use parentheses

# list comprehension:
[n * 2 for n in range(5)]

[0, 2, 4, 6, 8]

# generator
g = (n * 2 for n in range(5))
list(g)

[0, 2, 4, 6, 8]

A list is a collection of values, while a generator is a recipe for producing values

When you create a list, you are actually building a collection of values, and there is some memory cost associated with that.

When you create a generator, you are not building a collection of values, but a recipe for producing those values.

Both expose the same iterator interface.

l = [n * 2 for n in range(5)]
for val in l:
 print(val, end=" ")

0 2 4 6 8

g = g = (n * 2 for n in range(5))
for val in g:
 print(val, end=" ")

0 2 4 6 8

The difference is that a generator expression does not actually compute the values until they are needed. This not only leads to memory efficiency, but to computational efficiency as well! This also means that while the size of a list is limited by available memory, the size of a generator expression is unlimited!

A list can be iterated multiple times; a generator expression is single-use

l = [n * 2 for n in range(5)]

for val in l:
 print(val, end=" ")

print("\n")

for val in l:
 print(val, end=" ")

0 2 4 6 8
0 2 4 6 8

g = g = (n * 2 for n in range(5))

list(g)

[0, 2, 4, 6, 8]

list(g)

[]

This can be very useful because it means iteration can be stopped and started:

g = g = (n ** 2 for n in range(12))

for n in g:
 print(n, end=" ")
 if n > 30:
 break

print("\nDoing something in between...")

for n in g:
 print(n, end=" ")

0 1 4 9 16 25 36
Doing something in between...
49 64 81 100 121

This is useful when working with collections of data files on disk; it means that you can quite easily analyze them in batches, letting the generator keep track of which ones you have yet to see.

Generator Functions: Using `yield`

# list comprehension

L1 = [n * 2 for n in range(5)]

L2 = []
for n in range(5):
 L2.append(n * 2)

print("L1:", L1)
print("L2:", L2)

L1: [0, 2, 4, 6, 8]
L2: [0, 2, 4, 6, 8]

# generator
G1 = (n * 2 for n in range(5))

# generator function
def gen():
 for n in range(5):
 yield n * 2

G2 = gen()

print(*G1)
print(*G2)

0 2 4 6 8
0 2 4 6 8

Example: Prime Number Generator

# Generate a list of candidates
L = [n for n in range(2, 40)]
print(L)

[2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39]

# Remove all multiples of the first value
L = [n for n in L if n == L[0] or n % L[0] > 0]
print(L)

[2, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37, 39]

# Remove all multiples of the second value
L = [n for n in L if n == L[1] or n % L[1] > 0]
print(L)

[2, 3, 5, 7, 11, 13, 17, 19, 23, 25, 29, 31, 35, 37]

If we repeat this procedure enough times on a large enough list, we can generate as many primes as we wish.

Encapsulate this logic in a generator function:

def gen_primes(N):
 """
 Generate primes up to N
 """
 primes = set()
 for n in range(2, N):
 # print("n = ", n, ":", *(n % p > 0 for p in primes))
 if all(n % p > 0 for p in primes):
 primes.add(n)
 yield n


print(*gen_primes(100))

2 3 5 7 11 13 17 19 23 29 31 37 41 43 47 53 59 61 67 71 73 79 83 89 97

args and kwargs

Mon, 06 Jul 2020 00:00:00 +0000

Passing multiple arguments to a function

*args and **kwargs allow you to pass multiple arguments or keyword arguments to a function.

args will collect extra positional arguments as a tuple because the parameter name has a * prefix.
kwargs will collect extra keyword arguments as a dictionary because the parameter name has a ** prefix

For example, we neede to take a various numbers of arguments and compute their sum.

The first way is often the most intuitive for people that have experience with collections: simply pass a list or a set of all the arguments to your function.

def my_sum(my_integers):
 result = 0
 for x in my_integers:
 result += x
 return result

list_of_integers = [1, 2, 3]
my_sum(list_of_integers)

This implementation works, but whenever you call this function you’ll also need to create a list of arguments to pass to it. This can be inconvenient, especially if you don’t know up front all the values that should go into the list. 🤪

This is where *args can be really useful, because it allows you to pass a varying number of positional arguments.

def my_sum(*args):
 result = 0
 # Iterating over the python `args` tuple
 for x in args:
 result += x
 return result

# *args = 1, 2, 3
# -> args = (1, 2, 3)
my_sum(1, 2, 3)

def print_args(*args):
 print(args)

print_args(1, 2, 3)

(1, 2, 3)

In this example, we’re no longer passing a list to my_sum(). Instead, we’re passing three different positional arguments. my_sum() takes all the parameters that are provided in the input and packs them all into a single iterable object named args.

The magic is that we use the unpacking operator (*). The iterable object you’ll get using the unpacking operator * is not a list but a tuple, which is NOT mutable.

Using the Python kwargs Variable in Function Definitions

**kwargs works just like *args, but instead of accepting positional arguments it accepts keyword (or named) arguments.

E.g.:

def concatenate(**kwargs):
 result = ""
 # Iterating over the Python kwargs dictionary
 for arg in kwargs.values():
 result += arg
 return result

print(concatenate(a="Real", b="Python",
 c="Is", d="Great", e="!"))

RealPythonIsGreat!

Like args, kwargs is just a name that can be changed to whatever you want. Again, what is important here is the use of the unpacking operator (**).

Note: If we iterate over the dictionary and want to return its values, we must use .values(). Otherwise it will iterates over the keys of the kwargs dictionary.

Ordering Arguments in a Function

The correct order for parameters is:

Standard arguments
*args arguments
**kwargs arguments

E.g.:

# correct
def my_function(a, b, *args, **kwargs):
 pass

Forwarding Optional or Keyword Arguments

To pass optional or keyword parameters from one function to another, use the argument-unpacking operators * and ** when calling the function you want to forward arguments to.

This also gives you an opportunity to modify the arguments before you pass them along. E.g.:

def foo(x, *args, **kwargs):
 kwargs['name'] = 'Alice'
 new_args = args + ('extra', )
 bar(x, *new_args, **kwargs)

One scenario where this technique is useful is writing wrapper functions such as decorators. There you typically also want to accept arbitrary arguments to be passed through to the wrapped function. If we can do it without having to copy and paste the original function’s signature, that might be more maintainable. E.g.:

def trace(func):
 @functools.wraps(func)
 def wrapper_trace(*args, **kwargs):
 print(func.__name__, args, kwargs)
 result = func(*args, **kwargs)
 print(result)

 return wrapper_trace


@trace
def greet(greeting, name):
 return(f"{greeting}, {name}!")

>>> greet("Hello", "Ecko")
greet ('Hello', 'Ecko') {}
Hello, Ecko!

Unpacking With the Asterisk Operators: * & **

In short, the unpacking operators are operators that unpack the values from iterable objects in Python.

The single asterisk operator * can be used on any iterable that Python provides
The double asterisk operator ** can only be used on dictionaries

`*`

The * operator works on any iterable object.

For example:

my_list = [1, 2, 3]
print(my_list)

[1, 2, 3]

The list is printed, along with the corresponding brackets and commas.

Now try to prepend the unpacking operator * to the name of the list:

print(*my_list)

1 2 3

Putting a * before an iterable in a function call will unpack it and pass its elements as separate positional arguments to the called function.

Here, the * operator tells print() to unpack the list first. In this case, the output is no longer the list itself, but rather the content of the list. The difference is: Instead of a list, print() has taken three separate arguments as the input.

Another thing should be noticed is that we used the unpacking operator * to call a function, instead of in a function definition. In this case, print() takes all the items of a list as though they were single arguments.

Other convenient uses of the unpacking operator

E.g. 1

Split a list into three different parts: The output should show the first value, the last value, and all the values in between.

With the unpacking operator, we can do this in just one line of code:

my_list = [1, 2, 3, 4, 5, 6]

a, *b, c = my_list

print(a)
print(b)
print(c)

1
[2, 3, 4, 5]
6

E.g. 2

Merge two lists:

list_1 = [1, 2, 3]
list_2 = [4, 5, 6]

merged_list = [*list_1, *list_2]
merged_list

[1, 2, 3, 4, 5, 6]

For strings

In Python, strings are iterable objects. So the * operators can also be used to unpack a string.

E.g.:

a = [*'HelloWorld']
a

['H', 'e', 'l', 'l', 'o', 'W', 'o', 'r', 'l', 'd']

The previous example seems great, but when you work with these operators it’s important to keep in mind the seventh rule of The Zen of Python by Tim Peters: Readability counts!

Consider the following example:

*a, = 'HelloWorld'
a

['H', 'e', 'l', 'l', 'o', 'W', 'o', 'r', 'l', 'd']

There’s the unpacking operator *, followed by a variable, a comma, and an assignment. That’s a lot packed into one line! In fact, this code is no different from the previous example. It just takes the string HelloWorld and assigns all the items to the new list a, thanks to the unpacking operator *.

The comma after the a does the trick. When you use the unpacking operator with variable assignment, Python requires that your resulting variable is either a list or a tuple. With the trailing comma, you have actually defined a tuple with just one named variable a.

While this is a neat trick, many Pythonistas would not consider this code to be very readable. As such, it’s best to use these kinds of constructions sparingly.

`**`

The ** operator works similarly as *, but only for dictionary.

my_first_dict = {"A": 1, "B": 2}
my_second_dict = {"C": 3, "D": 4}

my_merged_dict = {**my_first_dict, **my_second_dict}
my_merged_dict

{'A': 1, 'B': 2, 'C': 3, 'D': 4}

The ** operator also works for function argument unpacking.

For example:

def print_vector(x, y, z):
 print(f"<{x}, {y}, {z}>")

vec_dict = {
 "y": 1,
 "x": 0,
 "z": 2,
}

>>> print_vector(**vec_dict)
<0, 1, 2>

Because dictionaries are unordered, this matches up dictionary values and function arguments based on the dictionary keys: the x argument receives the value associated with the 'x' key in the dictionary.

If you use the single asterisk (*) operator to unpack the dictionary, keys would be passed to the function in random order instead.

zip

Mon, 06 Jul 2020 00:00:00 +0000

zip(): creates an iterator that will aggregate elements from two or more iterables.

According to the official documentation, Python’s zip() function behaves as follows:

Returns an iterator of tuples, where the i-th tuple contains the i-th element from each of the argument sequences or iterables. The iterator stops when the shortest input iterable is exhausted. With a single iterable argument, it returns an iterator of 1-tuples. With no arguments, it returns an empty iterator.

Python zip operations work just like the physical zipper on a bag or pair of jeans. Interlocking pairs of teeth on both sides of the zipper are pulled together to close an opening.

Use `zip()` in python

zip(*iterables):

takes in iterables as arguments and returns an iterator
generates a series of tuples containing elements from each iterable
can accept any type of iterable, such as files, lists, tuples, dictionaries, sets, and so on.

Pass n arguments

# Pass n arguments
numbers = [1, 2, 3]
letters = ['a', 'b', 'c']
upper_letters = ['A', 'B', 'C']
zipped = zip(numbers, letters, upper_letters)

zipped

<zip at 0x62479ae08>

list(zipped)

[(1, 'a', 'A'), (2, 'b', 'B'), (3, 'c', 'C')]

num_tuple = (1, 2)
lettet_tuple = ('a', 'b')
upper_letter_tuple = ('A', 'B')
list(zip(num_tuple, lettet_tuple, upper_letter_tuple))

[(1, 'a', 'A'), (2, 'b', 'B')]

Pass no arguments

# Passing no argument
zipped = zip()

zipped

<zip at 0x62473c908>

list(zipped)

[]

Pass one arguments

# Pass one argument
zipped = zip(numbers)
list(zipped)

[(1,), (2,), (3,)]

Pass arguments of unequal length:

the number of elements that zip() puts out will be equal to the length of the shortest iterable. The remaining elements in any longer iterables will be totally ignored by zip()

list(zip(range(5), range(100)))

[(0, 0), (1, 1), (2, 2), (3, 3), (4, 4)]

If trailing or unmatched values are important, then use itertools.zip_longest() instead of zip()

from itertools import zip_longest

longest = range(5)
zipped = zip_longest(numbers, letters, longest, fillvalue='?')

list(zipped)

[(1, 'a', 0), (2, 'b', 1), (3, 'c', 2), ('?', '?', 3), ('?', '?', 4)]

Loop Over Multiple Iterables

Looping over multiple iterables is one of the most common use cases for Python’s zip() function

Traverse lists in parallel

letters

['a', 'b', 'c']

numbers

[1, 2, 3]

for l,n in zip(letters, numbers):
 print(f'letter: {l}')
 print(f'number: {n} \n')

letter: a
number: 1
letter: b
number: 2
letter: c
number: 3

Traverse dictionaries in parallel

dict_one = {'name': 'John', 'last_name': 'Doe', 'job': 'Python Consultant'}
dict_two = {'name': 'Jane', 'last_name': 'Doe', 'job': 'Community Manager'}

for (k1, v1), (k2, v2) in zip(dict_one.items(), dict_two.items()):
 print(k1, '->', v1)
 print(k2, '->', v2, '\n')

name -> John
name -> Jane
last_name -> Doe
last_name -> Doe
job -> Python Consultant
job -> Community Manager

Unzip a sequence

zip(*zipped)

numbers

[1, 2, 3]

letters

['a', 'b', 'c']

zipped = zip(numbers, letters)

zipped_list = list(zipped)
zipped_list

[(1, 'a'), (2, 'b'), (3, 'c')]

We have a list of tuples and want to separate the elements of each tuple into independent sequences. To do this, we can use zip() along with the unpacking operator *,

list(zip(*zipped_list))

[(1, 2, 3), ('a', 'b', 'c')]

Sorting in parallel

Combine two lists and sort them at the same time.

numbers = [2, 4, 3, 1]

letters = ['b', 'a', 'd', 'c']

data1 = list(zip(numbers, letters))
data1

[(2, 'b'), (4, 'a'), (3, 'd'), (1, 'c')]

data1.sort() # sort by numbers
data1

[(1, 'c'), (2, 'b'), (3, 'd'), (4, 'a')]

data2 = list(zip(letters, numbers))
data2

[('b', 2), ('a', 4), ('d', 3), ('c', 1)]

data2.sort() # sort by letters
data2

[('a', 4), ('b', 2), ('c', 1), ('d', 3)]

Use sorted() and zip() together to achieve a similar result

data = sorted(zip(letters, numbers))
data

[('a', 4), ('b', 2), ('c', 1), ('d', 3)]

Calculating in pairs

total_sales = [52000.00, 51000.00, 48000.00]
prod_cost = [46800.00, 45900.00, 43200.00]

for sales, costs in zip(total_sales, prod_cost):
 profit = sales - costs
 print(f'Profit: {profit}')

Profit: 5200.0
Profit: 5100.0
Profit: 4800.0

Building Dictionaries

fields = ['name', 'last_name', 'age', 'job']
values = ['John', 'Doe', '45', 'Python Developer']

a_dict = dict(zip(fields, values))

a_dict

{'name': 'John', 'last_name': 'Doe', 'age': '45', 'job': 'Python Developer'}

Update an existing dictionary by combining zip() with dict.update().

new_job = ['Python Consultant']
field = ['job']

a_dict.update(zip(field, new_job))
a_dict

{'name': 'John', 'last_name': 'Doe', 'age': '45', 'job': 'Python Consultant'}

Modules and Packages

Thu, 29 Oct 2020 00:00:00 +0000

Modular programming refers to the process of breaking a large, unwieldy programming task into separate, smaller, more manageable subtasks or modules. Individual modules can then be cobbled together like building blocks to create a larger application.

Advantages to modularizing code in a large application:

Simplicity

Rather than focusing on the entire problem at hand, a module typically focuses on one relatively small portion of the problem. If you’re working on a single module, you’ll have a smaller problem domain to wrap your head around. This makes development easier and less error-prone.
Maintainability

Modules are typically designed so that they enforce logical boundaries between different problem domains. If modules are written in a way that minimizes interdependency, there is decreased likelihood that modifications to a single module will have an impact on other parts of the program. (You may even be able to make changes to a module without having any knowledge of the application outside that module.) This makes it more viable for a team of many programmers to work collaboratively on a large application.
Reusability

Functionality defined in a single module can be easily reused (through an appropriately defined interface) by other parts of the application. This eliminates the need to duplicate code.
Scoping

Modules typically define a separate namespace, which helps avoid collisions between identifiers in different areas of a program. (One of the tenets in the Zen of Python is Namespaces are one honking great idea—let’s do more of those!)

Functions, modules and packages are all constructs in Python that promote code modularization.

Python modules

Different ways to define a module in Python:
- A module can be written in Python itself.
- A module can be written in C and loaded dynamically at run-time, like the re (regular expression) module.
- A built-in module is intrinsically contained in the interpreter, like the itertools module.
A module’s contents are accessed the same way in all three cases: with the importstatement.
Build a module written in python:
- Create a file that contains legitimate Python code
- Give the file a name with a .py extension.

Example:

mod.py

s = "If Comrade Napoleon says it, it must be right."
a = [100, 200, 300]

def foo(arg):
 print(f'arg={arg}')

class Foo:
 pass

Several objects are defined in mod.py:

s (a string)
a (a list)
foo() (a function)
Foo (a class)

These objects can be accessed by importing the module as follows (assuming mod.py is in an appropriate location):

>>> import mod
>>> print(mod.s)
If Comrade Napoleon says it, it must be right.
>>> mod.a
[100, 200, 300]
>>> mod.foo(['quux', 'corge', 'grault'])
arg = ['quux', 'corge', 'grault']
>>> x = mod.Foo()
>>> x
<mod.Foo object at 0x03C181F0>

The Module Search Path

When the interpreter executes the

import mod

statement, it searches for mod.py in a list of directories assembled from the following sources:

The directory from which the input script was run or the current directory if the interpreter is being run interactively
The list of directories contained in the PYTHONPATH environment variable, if it is set. (The format for PYTHONPATH is OS-dependent but should mimic the PATHenvironment variable.)
An installation-dependent list of directories configured at the time Python is installed

The resulting search path is accessible in the Python variable sys.path, which is obtained from a module named sys.

Thus, to ensure your module is found, you need to do one of the following:

Put mod.py in the directory where the input script is located or the current directory, if interactive
Modify the PYTHONPATH environment variable to contain the directory where mod.pyis located before starting the interpreter
- Or: Put mod.py in one of the directories already contained in the PYTHONPATH variable
Put mod.py in one of the installation-dependent directories, which you may or may not have write-access to, depending on the OS

One additional option: you can put the module file in any directory of your choice and then modify sys.path at run-time so that it contains that directory.

The `import` statement

`import <module_name>`

Note that this does not make the module contents directly accessible to the caller.

Each module has its own private symbol table, which serves as the global symbol table for all objects defined in the module. Thus, a module creates a separate namespace
The statement import <module_name> only places <module_name> in the caller’s symbol table. The objects that are defined in the module remain in the module’s private symbol table.
From the caller, objects in the module are only accessible when prefixed with <module_name> via dot notation

In our example, after the following import statement, mod is placed into the local symbol table. But s and foo remain in the module’s private symbol table and are not meaningful in the local context:

>>> s
NameError: name 's' is not defined
>>> foo('quux')
NameError: name 'foo' is not defined

To be accessed in the local context, names of objects defined in the module must be prefixed by mod:

>>> mod.s
'If Comrade Napoleon says it, it must be right.'
>>> mod.foo('quux')
arg = quux

`from <module_name> import <name(s)>`

Because this form of import places the object names directly into the caller’s symbol table, any objects that already exist with the same name will be overwritten:

>>> a = ['foo', 'bar', 'baz']
>>> a
['foo', 'bar', 'baz']

>>> from mod import a
>>> a
[100, 200, 300]

Indiscriminately import everything from a module at one fell swoop:

from <module_name> import *

This will place the names of all objects from <module_name> into the local symbol table, with the exception of any that begin with the underscore (_) character.

This isn’t necessarily recommended in large-scale production code. It’s a bit dangerous because you are entering names into the local symbol table en masse. Unless you know them all well and can be confident there won’t be a conflict, you have a decent chance of overwriting an existing name inadvertently.

`from <module_name> import <name> as <alt_name>[, <name> as <alt_name> …]`

This makes it possible to place names directly into the local symbol table but avoid conflicts with previously existing names

>>> s = 'foo'
>>> a = ['foo', 'bar', 'baz']

>>> from mod import s as string, a as alist
>>> s
'foo'
>>> string
'If Comrade Napoleon says it, it must be right.'
>>> a
['foo', 'bar', 'baz']
>>> alist
[100, 200, 300]

`import <module_name> as <alt_name>`

Import an entire module under an alternate name

>>> import mod as my_module
>>> my_module.a
[100, 200, 300]
>>> my_module.foo('qux')
arg = qux

The `dir()` function

The built-in function dir() returns a list of defined names in a namespace. Without arguments, it produces an alphabetically sorted list of names in the current local symbol table

Executing a Module as a Script

Distinguish between when the file is loaded as a module and when it is run as a standalone script:

When a .py file is imported as a module, Python sets the special dunder variable __name__ to the name of the module.
If a file is run as a standalone script, __name__ is (creatively) set to the string '__main__'.

Using this fact, we can discern which is the case at run-time and alter behavior accordingly:

s = "If Comrade Napoleon says it, it must be right."
a = [100, 200, 300]

def foo(arg):
 print(f'arg = {arg}')

class Foo:
 pass

if (__name__ == '__main__'):
 print('Executing as standalone script')
 print(s)
 print(a)
 foo('quux')
 x = Foo()
 print(x)

Now, if we run as a script, we get output:

>>> python mod.py
Executing as standalone script
If Comrade Napoleon says it, it must be right.
[100, 200, 300]
arg = quux
<__main__.Foo object at 0x03450690>

But if you import as a module, you don’t:

>>> import mod
>>> mod.foo('grault')
arg = grault

Modules are often designed with the capability to run as a standalone script for purposes of testing the functionality that is contained within the module. This is referred to as unit testing.

Reloading a module

For reasons of efficiency, a module is only loaded once per interpreter session. A module can contain executable statements as well, usually for initialization. Be aware that these statements will only be executed the first time a module is imported. If you make a change to a module and need to reload it, you need to either restart the interpreter or use a function called reload() from module importlib.

Python packages

Packages allow for a hierarchical structuring of the module namespace using dot notation. In the same way that modules help avoid collisions between global variable names, packages help avoid collisions between module names.

Creating a package is quite straightforward, since it makes use of the operating system’s inherent hierarchical file structure. Consider the following arrangement:

Here, there is a directory named pkg that contains two modules, mod1.py and mod2.py. The contents of the modules are:

mod1.py

def foo():
 print('[mod1] foo()')

class Foo:
 pass

mod2.py

def bar():
 print('[mod2] bar()')

class Bar:
 pass

Given this structure, if the pkg directory resides in a location where it can be found (in one of the directories contained in sys.path), we can refer to the two modules with dot notation (pkg.mod1, pkg.mod2) and import them with the syntax we are already familiar with

import <module_name>[, <module_name> ...]

>>> import pkg.mod1, pkg.mod2
>>> pkg.mod1.foo()
[mod1] foo()
>>> x = pkg.mod2.Bar()
>>> x
<pkg.mod2.Bar object at 0x033F7290>

from <module_name> import <name(s)>

>>> from pkg.mod1 import foo
>>> foo()
[mod1] foo()

from <module_name> import <name> as <alt_name>

>>> from pkg.mod2 import Bar as Qux
>>> x = Qux()
>>> x
<pkg.mod2.Bar object at 0x036DFFD0>

We can import modules with these statements as well:

from <package_name> import <modules_name>[, <module_name> ...]
from <package_name> import <module_name> as <alt_name>

Importing the package doesn’t do much of anything useful. In particular, it does not place any of the modules in pkg into the local namespace

Package Initialization

If a file named __init__.py is present in a package directory, it is invoked when the package or a module in the package is imported. This can be used for execution of package initialization code, such as initialization of package-level data.

Consider the following __init__.py file:

__init__.py:

print(f'Invoking __init__.py for {__name__}')
A = ['quux', 'corge', 'grault']

Now when the package is imported, the global list A is initialized.

__init__.py can also be used to effect automatic importing of modules from a package. If __init__.py in the pkg directory contains the following:

print(f'Invoking __init__.py for {__name__}')
import pkg.mod1, pkg.mod2

then when we execute import pkg, modules mod1 and mod2 are imported automatically:

>>> import pkg
Invoking __init__.py for pkg
>>> pkg.mod1.foo()
[mod1] foo()
>>> pkg.mod2.bar()
[mod2] bar()

Note:

Before Python 3.3, an __init__.py file must be present in the package directory when creating a package. It used to be that the very presence of __init__.py signified to Python that a package was being defined. The file could contain initialization code or even be empty, but it had to be present.
Starting with Python 3.3, Implicit Namespace Packages were introduced. These allow for the creation of a package without any __init__.py file. Of course, it can still be present if package initialization is needed. But it is no longer required.

Importing `*` from a package

from <package_name> import *

Python follows this convention:

If the __init__.py file in the package directory contains a list named __all__, it is taken to be a list of modules that should be imported when the statement from <package_name> import * is encountered.

For example, consider the following structure:

Suppose we create an __init__.py in the pkg directory like this:

pkg/__init__.py

__all__ = [
 'mod1',
 'mod2',
 'mod3',
 'mod4'
 ]

Now from pkg import * imports all four modules:

>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__']

>>> from pkg import *
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__',
'__package__', '__spec__', 'mod1', 'mod2', 'mod3', 'mod4']
>>> mod2.bar()
[mod2] bar()
>>> mod4.Qux
<class 'pkg.mod4.Qux'>

In summary, __all__ is used by both packages and modules to control what is imported when import * is specified. But the default behavior differs:

For a package, when __all__ is not defined, import * does not import anything.
For a module, when __all__ is not defined, import * imports everything (except—you guessed it—names starting with an underscore).

Subpckages

Packages can contain nested subpackages to arbitrary depth. For example:

Importing still works the same as shown previously. Syntax is similar, but additional dot notation is used to separate package name from subpackage name:

import <package_name>.<subpackage_name>.<module_name>

Reference

Python Modules and Packages – An Introduction

Underscores in Python

Fri, 01 May 2020 00:00:00 +0000

TL;DR

Pattern	Example	Meaning
Single Leading Underscore	`_var`	Naming convention indicating a name is meant for internal use. Generally not enforced by the Python interpreter (except in wildcard imports) and meant as a hint to the programmer only.
Single Trailing Underscore	`var_`	Used by convention to avoid naming conflicts with Python keywords.
Double Leading Underscore	`__var`	Triggers name mangling when used in a class context. Enforced by the Python interpreter.
Double Leading and Trailing Underscore	`__var__`	Indicates special methods defined by the Python language. Avoid this naming scheme for your own attributes.
Single Underscore	`_`	Sometimes used as a name for temporary or insignificant variables (“don’t care”). Also: The result of the last expression in a Python REPL.

Single Leading Underscore: `_var`

When it comes to variable and method names, the single underscore prefix has a meaning by convention only.

It is meant as a hint to another programmer that a variable or method starting with a single underscore is intended for internal use.
It does NOT affect the behavior of your programs and this isn’t enforced by Python.

Example

class Test:
 def __init__(self):
 self.foo = 11
 self._bar = 23

t = Test()
print(t.foo)
print(t._bar) # we can still access _bar

11
23

However, leading underscores do impact how names get imported from modules. If you use a wildcard import (should be avoided!) to import all names from the module, Python will NOT import names with a leading underscore (unless the module defines an __all__ list that overrides this behavior).

Wildcard imports should be avoided as they make it unclear which names are present in the namespace. It’s better to stick to regular imports for the sake of clarity.

Example

my_module.py

def external_func():
 print("external")

def _internal_func():
 print("internal")

from my_module import *

_internal_func()

---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-6-3757d8ee357f> in <module>()
----> 1 _internal_func()

NameError: name '_internal_func' is not defined

Unlike wildcard imports, regular imports are not affected by the leading single underscore naming convention:

import my_module

my_module._internal_func()

internal

Single Trailing Underscore: `var_`

Sometimes the most fitting name for a variable is already taken by a keyword. Therefore names like class or def cannot be used as variable names in Python. In this case you can append a single underscore to break the naming conflict.

Example

def make_obj(name, class_):
 pass

Double Leading Underscore: `__var`

“dunder” in python:

Double underscores are often referred to as “dunders” in the Python community. The reason is that double underscores appear quite often in Python code and to avoid fatiguing their jaw muscles Pythonistas often shorten “double underscore” to “dunder.”

For example, you’d pronounce __baz as “dunder baz”. Likewise __init__ would be pronounced as “dunder init”.

A double underscore prefix causes the Python interpreter to rewrite the attribute name in order to avoid naming conflicts in subclasses. This is also called name mangling — the interpreter changes the name of the variable in a way that makes it harder to create collisions when the class is extended later.

Example

class Test:
 def __init__(self):
 self.foo = 11
 self._bar = 23
 self.__baz = 42

>>> t = Test()
>>> dir(t) # show the list with the object's attributes
['_Test__baz', '__class__', '__delattr__', '__dict__',
'__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_bar', 'foo']

This gives us a list with the object’s attributes. There’re some interesting findings:

The self.foo variable appears unmodified as foo in the attribute list.
self._bar behaves the same way—it shows up on the class as _bar.
However with self.__baz, things look a little different. When you search for __baz in that list you’ll see that there is no variable with that name.

If you look closely you’ll see there’s an attribute called _Test__baz on this object. This is the name mangling that the Python interpreter applies. It does this to protect the variable from getting overridden in subclasses.

Let’s create another class that extends the Test class and attempts to override its existing attributes added in the constructor:

class ExtendedTest(Test):
 def __init__(self):
 super().__init__()
 self.foo = 'overridden'
 self._bar = 'overridden'
 self.__baz = 'overridden'

>>> t2 = ExtendedTest()
>>> t2.foo
'overridden'
>>> t2._bar
'overridden'
>>> t2.__baz
AttributeError:
"'ExtendedTest' object has no attribute '__baz'"

It turns out this object doesn’t even have a __baz attribute. Let’s check the list of object’s attributes:

>>> dir(t2)
['_ExtendedTest__baz', '_Test__baz', '__class__',
'__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_bar', 'foo', 'get_vars']

__baz got turned into _ExtendedTest__baz to prevent accidental modification.

>>> t2._ExtendedTest__baz
'overridden'

The original _Test__baz is also still around:

>>> t2._Test__baz
42

Double underscore name mangling is fully transparent to the programmer. Name mangling affects ALL names that start with two underscore characters (“dunders”) in a class context. E.g.

class ManglingTest:
 def __init__(self):
 self.__mangled = "hello"

 def get_mangled(self):
 return self.__mangled

 def __method(self):
 return 42

 def call_it(self):
 return self.__method()

>>> ManglingTest().get_mangled()
'hello'

>>> ManglingTest().__mangled
AttributeError:
"'ManglingTest' object has no attribute '__mangled'"

>>> MangledMethod().call_it()
42

>>> MangledMethod().__method()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-29-3a27f66f344d> in <module>()
----> 1 MangledMethod().__method()
AttributeError: 'MangledMethod' object has no attribute '__method'

Double Leading and Trailing Underscore: `var`

Name mangling is NOT applied if a name starts and ends with double underscores.
Names that have both leading and trailing double underscores are reserved for special use in the language.
- This rule covers things like __init__ for object constructors, or __call__ to make an object callable. These dunder methods are often referred to as magic methods.
- It’s best to stay away from using names that start and end with double underscores (“dunders”) in your own programs to avoid collisions with future changes to the Python language.

Single Underscore `_`

Ignoring Values

If you don’t want to use specific values while unpacking, just assign that value to underscore(_).

# Ignoring a value
a, _, b = (1, 2, 3)
print(f"a={a}, b={b}")

a=1, b=3

You can also use *_ to ignore multiple values. (Only available in Python 3.x)

a, *_, b = (1, 2, 3, 4, 5, 6, 7)
print(f"a={a}, b={b}")
print(*_)

a=1, b=7
2 3 4 5 6

Use in Looping

for _ in range(5):
 print(_)

Use in Interprerter

_ is a special variable in most Python REPLs that represents the result of the last expression evaluated by the interpreter.

>>> 20 + 3
23
>>> _
23

Reference

Basic Input and Output

Thu, 28 Apr 2022 00:00:00 +0000

Source: RealPython

Reading Input From the Keyboard

input([<prompt>]): Reads a line from the keyboard. (Documentation)

pauses program execution to allow the user to type in a line of input from the keyboard
Once the user presses the Enter key, all characters typed are read and returned as a string
- Note: the return string doesn’t include the newline generated when the user presses the Enter key.
- Example
```
>>> user_input = input()
foo bar baz
>>> user_input
'foo bar baz'
```
If you include the optional <prompt> argument, then input() displays it as a prompt so that your user knows what to input
- Example
```
>>> name = input("What is your name? ")
What is your name? Winston Smith
>>> name
'Winston Smith'
```
input() always returns a string. If you want a numeric type, convert the string with the appropriate built-in fuctions (int(), float(), complex())

Writing Output to the Console

You can display program data to the console in Python with print().

print() is capable of taking the following arguments:

print() function

The values (value1, value2) mentioned above can be any string or any of the data types like list, float, string, etc.
sep: Divides the values given as arguments
end: String appended to the end (\n by default)

Example

>>> arr = [1, 2, 3, 4 ,5]
>>> _ = [print("num", el, sep=": ", end="; ") for el in arr]
num: 1; num: 2; num: 3; num: 4; num: 5;

>>> d = {"foo": 1, "bar": 2, "baz": 3}
>>> for k, v in d.items():
... print(k, v, sep=" -> ")
...
foo -> 1
bar -> 2
baz -> 3

Reference

String

Sun, 01 May 2022 00:00:00 +0000

Source: Real Python

String = object that contains a sequence of character data.

Python provides a rich set of operators, functions, and methods for working with strings.

String Operators

The `+` Operator

The + operator concatenates strings. It returns a string consisting of the operands joined together

Example

>>> str_1 = "Hello"
>>> str_2 = "World"
>>> str_1 + str_2
HelloWorld

The `*` Operator

The * operator creates multiple copies of a string. If s is a string and n is an integer, either of the following expressions returns a string consisting of n concatenated copies of s:

s * n
n * s

If n <= 0, then the result is an empty string.

Example

>>> s = 'foo.'

>>> s * 4 # n > 0
'foo.foo.foo.foo.'
>>> 4 * s
'foo.foo.foo.foo.'

n <= 0:

>>> 'foo' * -8 # n <= 0
''

The `in` Operator

As string is essentially a list of characters, the membership operator in can also be used with string.

Example

>>> s = 'foo'

>>> s in 'That\'s food for thought.'
True
>>> s in 'That\'s good for now.'
False

>>> 'z' not in 'abc'
True
>>> 'z' not in 'xyz'
False

Build-in String Functions

Function	Description
`chr()`	Converts an integer to a character
`ord()`	Converts a character to an integer
`len()`	Returns the length of a string
`str()`	Returns a string representation of an object

`ord(c)`

Returns an integer value (ASCII or Unicode) for the given character.

Example

>>> ord('a')
97
>>> ord('#')
35

>>> ord('€')
8364
>>> ord('∑')
8721

`chr(n)`

chr() does the reverse of ord(). Given a numeric value n, chr(n) returns a string representing the character that corresponds to n.

Example

>>> chr(97)
'a'
>>> chr(35)
'#'

>>> chr(97)
'a'
>>> chr(35)
'#'

`len(s)`

Returns the length (number of characters) of a string.

Example

>>> s = 'I am a string.'
>>> len(s)
14

`str(obj)`

Returns a string representation of an object.

Example

>>> str(49.2)
'49.2'
>>> str(3+4j)
'(3+4j)'
>>> str(3 + 29)
'32'
>>> str('foo')
'foo'

List Operations for String

In Python, strings are ordered sequences of character data. Therefore, list operations (indexing, slicing, negative indexing) also work with string.

Interpolating Variables into a String

Since Python 3.6, a new powerful formatting mechanism was introduced.

This feature is formally named the Formatted String Literal, but is more usually referred to by its nickname f-string.

More about f-string see: f-string

Modify Strings

In a nutshell, you can’t. Strings are one of the data types Python considers immutable, meaning not able to be changed.

You can usually easily accomplish what you want by generating a copy of the original string that has the desired change in place. Two possibilities:

Assign a new string to the variable

>>> s = 'foobar'
>>> s = s[:3] + 'x' + s[4:]
>>> s
'fooxar'

Use built-in string method

>>> s = 'foobar'
>>> s = s.replace('b', 'x')
>>> s
'fooxar'

Built-in String Methods

Python provides a lot of useful built-in methods for string objects.

Function: a callable procedure that you can invoke to perform specific tasks.

Method: a specialized type of callable procedure that is tightly associated with an object. Like a function, a method is called to perform a distinct task, but it is invoked on a specific object and has knowledge of its target object during execution.

In the following method definitions, arguments specified in square brackets ([]) are optional.

Case Conversion

Methods in this group perform case conversion on the target string.

`s.capitalize()`

Returns a copy of s with the first character converted to uppercase and all other characters converted to lowercase. Non-alphabetic characters are unchanged.

Example

>>> s = 'foo123#BAR#.'
>>> s.capitalize()
'Foo123#bar#.'

`s.lower()`

Returns a copy of s with all alphabetic characters converted to lowercase.

`s.swapcase()`

Returns a copy of s with uppercase alphabetic characters converted to lowercase and vice versa.

Example

>>> 'FOO Bar 123 baz qUX'.swapcase()
'foo bAR 123 BAZ Qux'

`s.title()`

Returns a copy of s in which the first letter of each word is converted to uppercase and remaining letters are lowercase.

This method uses a fairly simple algorithm. It does NOT attempt to distinguish between important and unimportant words, and it does NOT handle apostrophes, possessives, or acronyms gracefully.

Example

>>> "what's happened to ted's IBM stock?".title()
"What'S Happened To Ted'S Ibm Stock?"

`s.upper()`

Returns a copy of s with all alphabetic characters converted to uppercase.

Find and Replace

These methods provide various means of searching the target string for a specified substring.
Each method in this group supports optional <start> and <end> arguments.
- The action of the method is restricted to the portion of the target string starting at character position <start> and proceeding up to but NOT including character position <end>
- If <start> is specified but <end> is not, the method applies to the portion of the target string from <start> through the end of the string.

`s.count([, <start>[, <end>]])`

Counts occurrences of a substring in the target string.
returns the number of non-overlapping occurrences of substring  in s:

Example

>>> 'foo goo moo'.count('oo')
3

>>> 'foo goo moo'.count('oo', 0, 8)
2

`s.endswith(<suffix>[, <start>[, <end>]])`

Determines whether the target string ends with a given substring
returns True if s ends with the specified <suffix> and False otherwise

Example

>>> 'foobar'.endswith('bar')
True
>>> 'foobar'.endswith('baz')
False

>>> 'foobar'.endswith('oob', 0, 4)
True
>>> 'foobar'.endswith('oob', 2, 4)
False

`s.find([, <start>[, <end>]])`

Searches the target string for a given substring. You can use .find() to see if a Python string contains a particular substring.
Returns the lowest index in s where substring  is found
Returns -1 if the specified substring is not found

Example

>>> 'foo bar foo baz foo qux'.find('foo')
0

>>> 'foo bar foo baz foo qux'.find('grault')
-1

>>> 'foo bar foo baz foo qux'.find('foo', 4)
8
>>> 'foo bar foo baz foo qux'.find('foo', 4, 7)
-1

`s.index([, <start>[, <end>]])`

Identical to .find(), except that it raises an exception if  is not found rather than returning -1

`s.rfind([, <start>[, <end>]])`

Searches the target string for a given substring starting at the end.
Returns the highest index in s where substring  is found
Returns -1 if the substring is not found

Example

>>> 'foo bar foo baz foo qux'.rfind('foo')
16

>>> 'foo bar foo baz foo qux'.rfind('grault')
-1

>>> 'foo bar foo baz foo qux'.rfind('foo', 0, 14)
8
>>> 'foo bar foo baz foo qux'.rfind('foo', 10, 14)
-1

`s.rindex([, <start>[, <end>]])`

Identical to .rfind(), except that it raises an exception if  is not found rather than returning -1

`s.startswith(<prefix>[, <start>[, <end>]])`

Determines whether the target string starts with a given substring.
Returns True if s starts with the specified <prefix> and False otherwise

Example

>>> 'foobar'.startswith('foo')
True
>>> 'foobar'.startswith('bar')
False

>>> 'foobar'.startswith('bar', 3)
True
>>> 'foobar'.startswith('bar', 3, 2)
False

Character Classification

Classify a string based on the characters it contains.

`s.isalnum()`

Determines whether the target string consists of alphanumeric characters
Returns True if s is nonempty and all its characters are alphanumeric (either a letter or a number), and False otherwise

Example

>>> 'abc123'.isalnum()
True
>>> 'abc$123'.isalnum()
False
>>> ''.isalnum()
False

`s.isalpha()`

Determines whether the target string consists of alphabetic characters.
s.isalpha() returns True if s is nonempty and all its characters are alphabetic, and False otherwise

Example

>>> 'ABCabc'.isalpha()
True
>>> 'abc123'.isalpha()
False

`s.isdigit()`

Determines whether the target string consists of digit characters. You can use the .isdigit() Python method to check if your string is made of only digits.
Returns True if s is nonempty and all its characters are numeric digits, and False otherwise

Example

>>> '123'.isdigit()
True
>>> '123abc'.isdigit()
False

`s.isidentifier()`

Determines whether the target string is a valid Python identifier.
Returns True if s is a valid Python identifier according to the language definition, and False otherwise

Example

>>> 'foo32'.isidentifier()
True
>>> '32foo'.isidentifier()
False
>>> 'foo$32'.isidentifier()
False

.isidentifier() will return True for a string that matches a Python keyword even though that would not actually be a valid identifier, e.g.,

>>> 'and'.isidentifier() # and is a keyword in python
True

To test whether a string matches a Python keyword, use keyword.iskeyword():

>>> from keyword import iskeyword
>>> iskeyword('and')
True

If you really want to ensure that a string would serve as a valid Python identifier, you should check that .isidentifier() is True and that iskeyword() is False

`s.islower()`

Determines whether the target string’s alphabetic characters are lowercase.
returns True if s is nonempty and all the alphabetic characters it contains are lowercase, and False otherwise. Non-alphabetic characters are ignore.

Example

>>> 'abc'.islower()
True
>>> 'abc1$d'.islower()
True
>>> 'Abc1$D'.islower()
False

`s.isprintable()`

Returns True if s is empty or all the alphabetic characters it contains are printable.
Returns False if s contains at least one non-printable character.
Non-alphabetic characters are ignored

Example

>>> 'a\tb'.isprintable()
False
>>> 'a b'.isprintable()
True
>>> ''.isprintable()
True
>>> 'a\nb'.isprintable()
False

This is the only .isxxxx() method that returns True if s is an empty string. All the others return False for an empty string.

`s.isspace()`

Determines whether the target string consists of whitespace characters.
Returns True if s is nonempty and all characters are whitespace characters, and False otherwise.
The most commonly encountered whitespace characters are
- space ' '
- tab '\t'
- newline '\n'

Example

>>> ' \t \n '.isspace()
True
>>> ' a '.isspace()
False

`s.istitle()`

Determines whether the target string is title cased.
Returns
- True if s is nonempty, the first alphabetic character of each word is uppercase, and all other alphabetic characters in each word are lowercase. (more intuitive: “Uppercase characters may only follow uncased characters and lowercase characters only cased ones.”)
- False, otherwise

Example

>>> 'This Is A Title'.istitle()
True
>>> 'This is a title'.istitle()
False
>>> 'Give Me The #$#@ Ball!'.istitle()
True

`s.isupper()`

Determines whether the target string’s alphabetic characters are uppercase.
Returns True if s is nonempty and all the alphabetic characters it contains are uppercase, and False otherwise.
Non-alphabetic characters are ignored

Example

>>> 'ABC'.isupper()
True
>>> 'ABC1$D'.isupper()
True
>>> 'Abc1$D'.isupper()
False

String Formatting

Modify or enhance the format of a string

`s.center(<width>[, <fill>])`

Centers a string in a field.
Returns a string consisting of s centered in a field of width <width>. By default, padding consists of the ASCII space character
```
>>> 'foo'.center(10)
' foo '
```
If the optional <fill> argument is specified, it is used as the padding character
```
>>> 'bar'.center(10, '-')
'---bar----'
```
If s is already at least as long as <width>, it is returned unchanged
```
>>> 'foo'.center(2)
'foo'
```

`s.expandtabs(tabsize=8)`

Replaces each tab character ('\t') with spaces.
tabsize is an optional keyword parameter specifying alternate tab stop columns. By default, spaces are filled in assuming a tab stop at every eighth column

Example

>>> 'a\tb\tc'.expandtabs()
'a b c'
>>> 'aaa\tbbb\tc'.expandtabs()
'aaa bbb c'

>>> 'a\tb\tc'.expandtabs(4)
'a b c'
>>> 'aaa\tbbb\tc'.expandtabs(tabsize=4)
'aaa bbb c'

`s.ljust(<width>[, <fill>])`

Left-justifies a string in field.
Returns a string consisting of s left-justified in a field of width <width>
```
>>> 'foo'.ljust(10)
'foo '
```
If the optional <fill> argument is specified, it is used as the padding character
```
>>> 'foo'.ljust(10, '-')
'foo-------'
```
If s is already at least as long as <width>, it is returned unchanged
```
>>> 'foo'.ljust(2)
'foo'
```

`s.lstrip([<chars>])`

Trims leading characters from a string.

returns a copy of s with any whitespace characters removed from the left end

>>> ' foo bar baz '.lstrip()
'foo bar baz '
>>> '\t\nfoo\t\nbar\t\nbaz'.lstrip()
'foo\t\nbar\t\nbaz'

If the optional <chars> argument is specified, it is a string that specifies the set of characters to be removed
```
>>> 'http://www.realpython.com'.lstrip('/:pth')
'www.realpython.com'
```

`s.replace(<old>, <new>[, <count>])`

Replaces occurrences of a substring within a string.

Returns a copy of s with all occurrences of substring <old> replaced by <new>

>>> 'foo bar foo baz foo qux'.replace('foo', 'grault')
'grault bar grault baz grault qux'

If the optional <count> argument is specified, a maximum of <count> replacements are performed, starting at the left end of s
```
>>> 'foo bar foo baz foo qux'.replace('foo', 'grault', 2)
'grault bar grault baz foo qux'
```

`s.rjust(<width>[, <fill>])`

Right-justifies a string in a field.
Works similarly to s.ljust()

`s.rstrip([<chars>])`

Trims trailing characters from a string.
Works similarly to s.lstrip()

`s.strip([<chars>])`

Strips characters from the left and right ends of a string.
Equivalent to s.lstrip().rstrip()
As with .lstrip() and .rstrip(), the optional <chars> argument specifies the set of characters to be removed

`s.zfill(<width>)`

Pads a string on the left with zeros.
Returns a copy of s left-padded with '0' characters to the specified <width>
```
>>> '42'.zfill(5)
'00042'
```
If s contains a leading sign, it remains at the left edge of the result string after zeros are inserted
```
>>> '+42'.zfill(8)
'+0000042'
>>> '-42'.zfill(8)
'-0000042'
```
If s is already at least as long as <width>, it is returned unchanged

Converting Between Strings and Lists

Convert between a string and some composite data type by either pasting objects together to make a string, or by breaking a string up into pieces.

`s.join(<iterable>)`

Concatenates strings from an iterable.

Example

>>> ', '.join(['foo', 'bar', 'baz', 'qux'])
'foo, bar, baz, qux'

In the following example, <iterable> is specified as a single string value. When a string value is used as an iterable, it is interpreted as a list of the string’s individual characters.

>>> list('corge')
['c', 'o', 'r', 'g', 'e']

>>> ':'.join('corge')
'c:o:r:g:e'

`s.partition(<sep>)`

Splits s at the first occurrence of string <sep>.
The return value is a three-part tuple consisting of:
- The portion of s preceding <sep>
- <sep> itself
- The portion of s following <sep>

Example

>>> 'foo.bar'.partition('.')
('foo', '.', 'bar')
>>> 'foo@@bar@@baz'.partition('@@')
('foo', '@@', 'bar@@baz')

If <sep> is not found in s, the returned tuple contains s followed by two empty strings:

>>> 'foo.bar'.partition('@@')
('foo.bar', '', '')

s.rpartition()

Works exactly like s.partition(<sep>), except that s is split at the last occurrence of <sep> instead of the first occurrence

`s.rsplit(sep=None, maxsplit=-1)`

Splits a string into a list, starting from the right

Without arguments, s.rsplit() splits s into substrings delimited by any sequence of whitespace and returns the substrings as a list

>>> 'foo bar baz qux'.rsplit()
['foo', 'bar', 'baz', 'qux']
>>> 'foo\n\tbar baz\r\fqux'.rsplit()
['foo', 'bar', 'baz', 'qux']

If <sep> is specified, it is used as the delimiter for splitting
```
>>> 'foo.bar.baz.qux'.rsplit(sep='.')
['foo', 'bar', 'baz', 'qux']
```
- When <sep> is explicitly given as a delimiter, consecutive delimiters in s are assumed to delimit empty strings, which will be returned
```
>>> 'foo...bar'.rsplit(sep='.')
['foo', '', '', 'bar']
```
If the optional keyword parameter <maxsplit> is specified, a maximum of that many splits are performed, starting from the right end of s
```
>>> 'www.realpython.com'.rsplit(sep='.', maxsplit=1)
['www.realpython', 'com']
```

`s.split(sep=None, maxsplit=-1)`

Behaves exactly like s.rsplit(), except that if <maxsplit> is specified, splits are counted from the left end of s rather than the right end

`s.splitlines([<keepends>])`

Splits s at line boundaries up into lines and returns them in a list.

Any of the following characters or character sequences is considered to constitute a line boundary:

Escape Sequence	Character
`\n`	Newline
`\r`	Carriage Return
`\r\n`	Carriage Return + Line Feed
`\v` or `\x0b`	Line Tabulation
`\f` or `\x0c`	Form Feed
`\x1c`	File Separator
`\x1d`	Group Separator
`\x1e`	Record Separator
`\x85`	Next Line (C1 Control Code)
`\u2028`	Unicode Line Separator
`\u2029`	Unicode Paragraph Separator

Advanced

Substring count with overlapping occurrences

Let’s say we want to count the occurrence of substring 11 in the string 1011101111.

Note that in Python, the count() method returns the number of substrings in a given string, but it does not give correct results when two occurrences of the substring overlap. However, we still have different solution for this problem.

string = "1011101111"
sub_string = "11"

Use built-in `re` module

import re

Use [re.findall()](https://docs.python.org/3/library/re.html#re.findall):

>>> len(re.findall(f"(?={sub_string})", string))
5

Use re.subn :

>>> re.subn(f"(?={sub_string})", "", string)[1]
5

Use built-in string methods

Use startswith():

def count_substring(string, sub_string):
 count = 0
 for pos in range(len(string)):
 if string[pos:].startswith(sub_string):
 count += 1
 return count

>>> count_substring(string, sub_string)
5

Or in a more pythonic way, use list comprehension:

sum([string.startswith(sub_string, i) for i in range(len(string))])

Reference

String count with overlapping occurrences

Reference

Strings and Character Data in Python

f-string

Wed, 21 Oct 2020 00:00:00 +0000

Python f-string is the newest Python syntax to do string formatting. Python f-strings provide a faster, more readable, more concise, and less error prone way of formatting strings in Python. 👏

The f-strings have the f prefix and use {} brackets to evaluate values.

Python string formatting

The following example summarizes string formatting options in Python:

name = 'Peter'
age = 23

print('%s is %d years old' % (name, age)) # C style
print('{} is {} years old'.format(name, age)) # python format function
print(f'{name} is {age} years old') # f-string

Peter is 23 years old
Peter is 23 years old
Peter is 23 years old

f-string usage

f-string expressions

We can put expressions between the {} brackets, and the expression will be evaluated inside f-string

bags = 3
apples_in_bag = 12

print(f'There are total of {bags * apples_in_bag} apples')

There are total of 36 apples

f-string dictionaries

user = {'name': 'John Doe', 'occupation': 'gardener'}

print(f"{user['name']} is a {user['occupation']}")

John Doe is a gardener

f-string debug

Python 3.8 introduced the self-documenting expression with the = character.

import math

x = 0.8

print(f'{math.cos(x) = }')
print(f'{math.sin(x) = }')

math.cos(x) = 0.6967067093471654
math.sin(x) = 0.7173560908995228

multiline f-string

The f-strings are placed between round brackets
Each of the strings is preceded with the f character

name = 'John Doe'
age = 32
occupation = 'gardener'

msg = (
 f'Name: {name}\n'
 f'Age: {age}\n'
 f'Occupation: {occupation}'
)

print(msg)

Name: John Doe
Age: 32
Occupation: gardener

f-string calling function

def mymax(x, y):
 return x if x > y else y

a = 3
b = 4

print(f'Max of {a} and {b} is {mymax(a, b)}')

Max of 3 and 4 is 4

f-string objects

Python f-string accepts objects as well; the objects must have either __str__() or __repr__() magic functions defined.

class User:
 def __init__(self, name, occupation):
 self.name = name
 self.occupation = occupation

 def __repr__(self):
 return f"{self.name} is a {self.occupation}"

u = User('John Doe', 'gardener')

print(f'{u}')

John Doe is a gardener

f-string format

floats

Floating point values have the f suffix. We can also specify the precision: the number of decimal places. The precision is a value that goes right after the dot character.

val = 12.3

print(f'{val:.2f}')
print(f'{val:.5f}')

12.30
12.30000

width

The width specifier sets the width of the value. The value may be filled with spaces or other characters if the value is shorter than the specified width.

for x in range(1, 11):
 print(f'{x:02} {x*x:3} {x*x*x:4}')

01 1 1
02 4 8
03 9 27
04 16 64
05 25 125
06 36 216
07 49 343
08 64 512
09 81 729
10 100 1000

We can combine floats precision and width, for example:

import numpy as np

a = np.random.rand(20)

for index, value in enumerate(a):
 if index < 10:
 print(f'{index:2}: {value:4.4f}')
 else:
 print(f'{index:2}: {value:4.3f}')

{value:4.4f} means

We will use a width of 4-character to print value
The precision of value is 4: we will keep 4 decimal places

`!r`, `!s`, and `!a`

The conversions !r, !s, and !a are equivalent to calling repr(), str(), and ascii(), respectively.

Example:

>> a = 'some string'
>>> f'{a!r}'
"'some string'"

is identical to

>>> f'{repr(a)}'
"'some string'"

The usage of these conversion are not strictly required, because explicitly calling the corresponding functions is arguably more clear in its intent. In other words, they are redundant.

!r (repr), !s (str) and !a (ascii) are kept around just to ease compatibility with the str.format() alternative, because it does not allow the execution of arbitrary expressions. You don’t need to use them with f-strings.

Reference:

Reference

Python f-string tutorial

Sorting

Mon, 02 May 2022 00:00:00 +0000

Python provides two built-in functions for sorting:

list.sort(): modifies the list in-place
sorted(): builds a new sorted list from an iterable

Difference between these methods:

list.sort() method is only defined for lists. In contrast, the sorted() function accepts any iterable.
list.sort() modifies the list in-place and returns None. sorted() returns a new sorted list. Usually sort() is less convenient than sorted().

Sorting Basics

A simple ascending sort is very easy: just call the sorted() function. It returns a new sorted list.

Example

>>> a = [5, 2, 1, 3, 4]
>>> b = sorted(a)

>>> a # the original list is not affected
[5, 2, 1, 3, 4]

>>> b
[1, 2, 3, 4, 5]

Key

Both list.sort() and sorted() have a key parameter.

The value of the key parameter should be a function (or other callable) that serves as a key for the sort comparison. In other word, key decides the comparison criteria for sorting.

One way to specify key is to use the lambda function.

Example:

students_dict = [
 {"name": "John", "age": 12, "grade": "A"},
 {"name": "Mary", "age": 14, "grade": "B"},
 {"name": "Ben", "age": 13, "grade": "A"}
]

sorted(students_dict, key=lambda student: student.get("age")) # sort by age

Output:

[{'age': 12, 'grade': 'A', 'name': 'John'},
 {'age': 13, 'grade': 'A', 'name': 'Ben'},
 {'age': 14, 'grade': 'B', 'name': 'Mary'}]

This also works for objects with named attributes.

Example:

class Student:
 def __init__(self, name, age, grade):
 self.name = name
 self.age = age
 self.grade = grade

 def __repr__(self):
 return repr((self.name, self.age, self.grade))

students = [
 Student("John", 12, "A"),
 Student("Mary", 14, "B"),
 Student("Ben", 13, "A")
]

sorted(students, key=lambda student: student.age)

Output:

[('John', 12, 'A'), ('Ben', 13, 'A'), ('Mary', 14, 'B')]

Operator Module Functions

Python operator module provides convenience functions to make accessor functions easier and faster:

itemgetter()
attrgetter()
methodcaller()

`itemgetter()`

Example:

students_tuple = [
 ("John", 12, "A"),
 ("Mary", 14, "B"),
 ("Ben", 13, "A")
]

sorted(students_tuple, key=itemgetter(1))

Output:

[('John', 12, 'A'), ('Ben', 13, 'A'), ('Mary', 14, 'B')]

students_dict = [
 {"name": "John", "age": 12, "grade": "A"},
 {"name": "Mary", "age": 14, "grade": "B"},
 {"name": "Ben", "age": 13, "grade": "A"}
]

sorted(students_dict, key=itemgetter("age"))

Output:

[{'age': 12, 'grade': 'A', 'name': 'John'},
 {'age': 13, 'grade': 'A', 'name': 'Ben'},
 {'age': 14, 'grade': 'B', 'name': 'Mary'}]

`attrgetter()`

Example:

class Student:
 def __init__(self, name, age, grade):
 self.name = name
 self.age = age
 self.grade = grade

 def __repr__(self):
 return repr((self.name, self.age, self.grade))

students = [
 Student("John", 12, "A"),
 Student("Mary", 14, "B"),
 Student("Ben", 13, "A")
]

[('John', 12, 'A'), ('Ben', 13, 'A'), ('Mary', 14, 'B')]

order

Both list.sort() and sorted() accept a reverse parameter with a boolean value. This is used to flag descending sorts" reverse=False and reverse=True represent ascending and descening order, respectively.

Advance

Multiple levels of sorting

Multiple level of sorting can be easily achieved with operator module functions.

For example, to sort by grade then by age:

>>> sorted(students_tuple, key=itemgetter(2, 1))
[('John', 12, 'A'), ('Ben', 13, 'A'), ('Mary', 14, 'B')]

>>> sorted(students, key=attrgetter("grade", "age"))
[('John', 12, 'A'), ('Ben', 13, 'A'), ('Mary', 14, 'B')]

>>> sorted(students_dict, key=itemgetter("grade", "age"))
[{'age': 12, 'grade': 'A', 'name': 'John'},
 {'age': 13, 'grade': 'A', 'name': 'Ben'},
 {'age': 14, 'grade': 'B', 'name': 'Mary'}]

Reference

Sorting HOW TO

Assertion

Sat, 03 Dec 2022 00:00:00 +0000

The proper use of assertions is to inform developers about unrecoverable errors in a program.
Assertions are not intended to signal expected error conditions.
Assertions are meant to be internal self-checks for your program.

keep in mind that Python’s assert statement is a debugging aid, not a mechanism for handling run-time errors.

The goal of using assertions is to let developers find the likely root cause of a bug more quickly. An assertion error should NEVER be raised unless there’s a bug in your program.

Syntax

assert_stmt ::= "assert" expression1 ["," expression2]

expression1 is the condition we test
the optional expression2 is an error message that’s displayed if the assertion fails.

At execution time, the Python interpreter transforms each assert state- ment into roughly the following sequence of statements:

if __debug__:
 if not expression1:
 raise AssertionError(expression2)

Before the assert condition is checked, there’s an additional check for the __debug__ global variable. It’s a built-in boolean flag that’s true under normal circumstances and false if optimizations are requested.

Two caveats when using asserts

Don’t Use Asserts for Data Validation

Assertions can be globally disabled3 with the -0 and -00 command line switches, as well as the PYTHONOPTIMIZE environment variable in CPython!

$\rightarrow$ This turns any assert statement into a null-operation: the assertions simply get compiled away and won’t be evaluated 🤪, which means that none of the conditional expressions will be executed. As a side-effect, it becomes extremely dan- gerous to use assert statements as a quick and easy way to validate input data.

The solution to avoid this problem is: NEVER use assertions to do data validation. Instead, we could do our validation with regular if-statements and raise validation exceptions if necessary

Example

❌ Don’t do

def delete_product(prod_id, user):
 assert user.is_admin(), 'Must be admin'
 assert store.has_product(prod_id), 'Unknown product'
 store.get_product(prod_id).delete()

✅ Do

def delete_product(product_id, user):
 if not user.is_admin():
 raise AuthError('Must be admin to delete')
 if not store.has_product(product_id):
 raise ValueError('Unknown product id')
 store.get_product(product_id).delete()

Asserts That Never Fail

When you pass a tuple as the first argument in an assert statement, the assertion always evaluates as true and therefore never fails.

For example, the following assertion will never fail:

assert(1 == 2, 'This should fail')

Reason is that non-empty tuples is always true in Python.

A good countermeasure you can apply to prevent this syntax quirk from causing trouble is to use a code linter. Newer versions of Python 3 will also show a syntax warning for these dubious asserts.

Decorator: Basics

Sat, 21 May 2022 00:00:00 +0000

TL;DR

Decorators define reusable building blocks you can apply to a callable to modify its behavior without permanently modifying the callable itself.
The @ syntax is just a shorthand (syntax sugar) for calling the decorator on an input function.
- Multiple decorators on a single function are applied bottom to top (decorator stacking).
Use the functools.wraps helper in every decorator to carry over metadata from the undecorated callable to the decorated one.
Decorators are not a cure-all and they should not be overused

What is decorator?

Decorators provide a simple syntax for calling higher-order functions.

By definition, a decorator is a function that takes another function and extends the behavior of the latter function without explicitly modifying it.

In other words, decorators “decorate” or “wrap” another function and let you execute code before and after the wrapped function runs.

Allow you to define reusable building blocks that can change or extend the behavior of other functions
Let you do that without permanently modifying the wrapped function itself. The function’s behavior changes only when it’s decorated.

Under the hood, a decorator is a callable that takes a callable as input and returns another callable.

Functions

Before you can understand decorators, you must first understand how functions work.

In Python, functions

return a value based on the given arguments
are first-class objects. I.e., functions can be passed around and used as arguments.

Inner Functions

Inner functions are functions defined inside other functions.

Example:

def parent():
 print("Printing from the parent() function")

 def first_child():
 print("Printing from the first_child() function")

 def second_child():
 print("Printing from the second_child() function")

 second_child()
 first_child()

The order in which the inner functions are defined does not matter.
The inner functions are not defined until the parent function is called. They are locally scoped to parent(), i.e., they only exist inside the parent() function as local variables.

Returning Functions From Functions

Python also allows you to use functions as return values. For example:

def parent(num):
 def first_child():
 return "Hi, I am the first child."

 def second_child():
 return "Hi, I am the second child."

 # Return function name without the parentheses means returning a reference to the function
 # In contrast, function name with parentheses refers to the result of evaluating the functions.
 return first_child if num == 1 else second_child

Simple Decorators

Put simply: decorators wrap a function, modifying its behavior.

Example:

def my_decorator(func):
 def wrapper():
 print("Before function")
 func()
 print("After function")
 return wrapper

def say_hi():
 print("Hi")

say_hi = my_decorator(say_hi)

>>> say_hi()
Before function
Hi
After function

Syntactic Sugar

The way we decorated say_hi() above is a little clunky.

We type the function name say_hi() three times
The decoration gets a bit hidden away below the definition of the function.

To facilitate, Python allows you to use decorators in a simpler way with the @ symbol (sometimes called the “pie” syntax).

The followings are equivalent:

def my_decorator(func):
 def wrapper():
 print("Before function")
 func()
 print("After function")
 return wrapper

def say_hi():
 print("Hi")

say_hi = my_decorator(say_hi)

def my_decorator(func):
 def wrapper():
 print("Before function")
 func()
 print("After function")
 return wrapper

@my_decorator # an easier way of say_hi = my_decorator(say_hi)
def say_hi():
 print("Hi")

>>> say_hi()
Before function
Hi
After function

Naming of the Inner Function

You can name your inner function whatever you want, and a generic name like wrapper() is usually okay. You can also name the inner function with the same name as the decorator but with a wrapper_ prefix.

Example:

def do_twice(func):
 def wrapper_do_twice():
 func()
 func()
 return wrapper_do_twice

Decorating Functions With Arguments

To allow the decorator accept an arbitrary number of positional and keyword arguments, use *args and **kwargs in the inner wrapper function.

It use the * and ** operators in the wrapper closure definition to collect all positional and keyword arguments and stores them in variables (args and kwargs).
The wrapper closure then forwards the collected arguments to the original input function using the * and ** “argument unpacking” operators.

Example:

def do_twice(func):
 def wrapper_do_twice(*args, **kwargs):
 func(*args, **kwargs)
 func(*args, **kwargs)
 return wrapper_do_twice

@do_twice
def say_hi():
 print("Hi")

@do_twice
def greet(name):
 print(f"Hello {name}")

>>> say_hi()
Hi
Hi
>>> greet("World")
Hello World
Hello World

Returning Values From Decorated Functions

Make sure the wrapper function returns the return value of the decorated function.

Example

def do_twice(func):
 def wrapper_do_twice(*args, **kwargs):
 func(*args, **kwargs)
 return func(*args, **kwargs)
 return wrapper_do_twice

@do_twice
def return_greeting(name):
 print("Creating greeting")
 return f"Hi {name}"

>>> hi_adam = return_greeting("Adam")
Creating greeting
Creating greeting
>>> print(hi_adam)
Hi Adam

Another example

def uppercase(func):
 def wrapper():
 original_result = func()
 modified_result = original_result.upper()
 return modified_result
 return wrapper

@uppercase
def greet():
 return "Hello!"

Getting the Indentity Information

What a decorator do is replacing one function with another. One downside of this process is that it “hides” some of the metadata attached to the original (undecorated) function.

For example, the original function name, its docstring, and parameter list are hidden by the wrapper closure:

def greet():
 """Return a friendly greeting."""
 return 'Hello!'

decorated_greet = uppercase(greet)

If you try to access any of that function metadata, you’ll see the wrap- per closure’s metadata instead:

>>> greet.__name__
'greet'
>>> greet.__doc__
'Return a friendly greeting.'

>>> decorated_greet.__name__
'wrapper'
>>> decorated_greet.__doc__
None

This makes debugging and working with the Python interpreter awkward and challenging 🤪.

A quick fix for this is to use the @functools.wraps decorator in Python’s standard library, which will preserve information about the original function.

Technical Detail

The @functools.wraps decorator uses the function functools.update_wrapper() to update special attributes like __name__ and __doc__ that are used in the introspection.

Example

import functools

def do_twice(func):
 @functools.wraps(func)
 def wrapper_do_twice(*args, **kwargs):
 func(*args, **kwargs)
 return func(*args, **kwargs)
 return wrapper_do_twice

>>> say_hi
<function __main__.say_hi>
>>> say_hi.__name__
say_hi
>>> help(say_hi)
Help on function say_hi in module __main__:

say_hi()

As a best practice, you should use functools.wraps in all of the decorators you write yourself. It doesn’t take much time and it will save you (and others) debugging headaches down the road.

Real World Examples

A good boilerplate template for building more complex decorators:

import functools

def decorator(func):
 @functools.wraps(func)
 def wrapper_decorator(*args, **kwargs):
 # Do something before
 value = func(*args, **kwargs)
 # Do something after
 return value
 return wrapper_decorator

Timing Functions

import time

def timer(func):
 """Print the runtime of the decorated function"""
 @functools.wraps(func)
 def wrapper_timer(*args, **kwargs):
 start_time = time.perf_counter()
 value = func(*args, **kwargs)
 end_time = time.perf_counter()
 runtime = end_time - start_time
 print(f"Finished {func.__name__!r}: in {runtime:.4f} secs.")
 return value

 return wrapper_timer

@timer
def waste_some_time(num_times):
 for _ in range(num_times):
 sum([i * 2 for i in range(100)])

>>> waste_some_time(1000)
Finished 'waste_some_time': in 0.0130 secs.

Debugging Code

import functools

def debug(func):
 """Print the function signature and return value"""
 @functools.wraps(func)
 def wrapper_debug(*args, **kwargs):
 args_repr = [repr(arg) for arg in args]
 kwargs_repr = [f"{k}={v!r}" for k, v in kwargs.items()]
 signature = ", ".join(args_repr + kwargs_repr)
 print(f"Calling {func.__name__}({signature})")
 value = func(*args, **kwargs)
 print(f"{func.__name__!r} returned {value!r}")
 return value

 return wrapper_debug

@debug
def make_greeting(name, age=None):
 if not age:
 return f"Howdy {name}!"
 else:
 return f"Whoa {name}! {age} already, you are growing up!"

>>> make_greeting("Richard", age=20)
Calling make_greeting('Richard', age=20)
'make_greeting' returned 'Whoa Richard! 20 already, you are growing up!'
Whoa Richard! 20 already, you are growing up!

Registering Plugins

Decorators don’t have to wrap the function they’re decorating. They can also simply register that a function exists and return it unwrapped.

E.g., to create a light-weight plug-in architecture:

import random
PLUGINS = dict()

def register(func):
 """Register a function as a plug-in"""
 # Simply stores a reference to the decorated function in the global PLUGINS dict
 # Note that you do not have to write an inner function or use @functools.wraps in this example, 
 # as you are returning the original function unmodified.
 PLUGINS[func.__name__] = func
 return func


@register
def say_hello(name):
 print(f"Hello {name}")


@register
def be_awesome(name):
 return f"Yo {name}, together we are the awesomest!"


def randomly_greet(name):
 """Randomly chooses one of the registered functions to use."""
 greeter, greeter_func = random.choice(list(PLUGINS.items()))
 print(f"Using {greeter!r}")
 return greeter_func(name)

The `@register` decorator simply stores a reference to the decorated function in the global `PLUGINS` dict.

Because

@register
def be_awesome(name):

is equivalent to

be_awesome = register(be_awesome)

Therefore, whenever adding @register on top of the function, this function is registered in PLUGINS. In other words, the PLUGINS dictionary contains references to thefunction object.

Note that you do not have to write an inner function or use @functools.wraps in this example because you are returning the original function unmodified.

>>> PLUGINS
{'be_awesome': <function __main__.be_awesome>,
 'say_hello': <function __main__.say_hello>}
>>> randomly_greet("Alice")
Using 'say_hello'
Hello Alice

👍 Main benefit: You do not need to maintain a list of which plugins exist. That list is created when the plugins register themselves. This makes it trivial to add a new plugin: just define the function and decorate it with @register.

Reference

Primer on Python Decorators

Function: First Class Object

Sat, 03 Dec 2022 00:00:00 +0000

TL;DR

Everything in Python is an object, including functions. You can
- assign them to variables
- store them in data structures
- pass or return them to and from ohter functions
First-class functions allow you to abstract away and pass around behavior in your programs.
Functions can be nested and they can capture and carry some of the parent function’s state with them.
Objects can be made callable, which allows you to treat them like functions.

Python’s functions are first-class objects. You can

assign them to variables
store them in data structures
pass them as arguments to other functions
return them as values from other functions

Example throughout:

def yell(text):
 return text.upper() + '!'

>>> yell('hello')
'HELLO!'

Functions Are Objects

Like strings, lists, modules, functions in Python are also just objects.

Becaus the yell function is an object in Python, we can assign it to another variable:

bark = yell

This line doesn’t call the function. It takes the function object referenced by yell and creates a second name, bark, that points to it.

You could now also execute the same underlying function object by calling bark:

>>> bark('woof')
'WOOF!'

Function objects and their names are two separate concerns. We can delete the function’s original name (yell). Since another name (bark) still points to the underlying function, you can still call the function through it:

>>> del yell

>>> yell('hello?')
NameError: "name 'yell' is not defined"

>>> bark('hey')
'HEY!'

For debugging purposes, Python attaches a string identifier to every function at creation time. We can access this internal identifier with the __name__ attribute:

>>> bark.__name__
'yell'

Functions Can Be Stored in Data Structures

We can store functions in data structures, just like we can do with other objects.

Example:

>>> funcs = [bark, str.lower, str.capitalize]

>>> funcs
[<function yell at 0x10ff96510>,
 <method 'lower' of 'str' objects>,
 <method 'capitalize' of 'str' objects>]

Accessing the function objects stored inside the list works like it would with any other type of object:

>>> for f in funcs:
... print(f, f('hey there'))

<function yell at 0x10ff96510> 'HEY THERE!'
<method 'lower' of 'str' objects> 'hey there'
<method 'capitalize' of 'str' objects> 'Hey there'

We can do the lookup and then immediately call the resulting “disembodied” function object within a single expres- sion:

>>> funcs[0]('heyho')
'HEYHO!'

Functions Can Be Passed to Other Functions

We can pass functions as arguments to other functions.

Example:

def greet(func):
 greeting = func('Hi, I am a Python program')
 print(greeting)

We pass the bark functon to greet:

>>> greet(bark)
'HI, I AM A PYTHON PROGRAM!'

The ability to pass function objects as arguments to other functions is powerful. It allows you to abstract away and pass around behavior in your programs. 👍

Functions that can accept other functions as arguments are also called higher-order functions. They are a necessity for the functional programming style. The classical example for higher-order functions in Python is the built-in map function: It takes a function object and an iterable, and then calls the function on each element in the iterable, yielding the results as it goes along.

Example

>>> list(map(bark, ['hello', 'hey', 'hi']))
['HELLO!', 'HEY!', 'HI!']

Functions Can Be Nested

Python allows functions to be defined inside other functions. These are often called nested functions or inner functions.

Example:

def speak(text):
 def whisper(t):
 return t.lower() + '...'
 return whisper(text)

>>> speak('Hello, World')
'hello, world...'

Notice that the inner function is “nested”. In other words, whisper does NOT exist outside speak.

>>> whisper('Yo')
NameError:
"name 'whisper' is not defined"

>>> speak.whisper
AttributeError:
"'function' object has no attribute 'whisper'"

Is it possible to access that nested whisper? Yes. Remember that function are just objects - we can return the inner function to the caller of the parent function.

Example:

def get_speak_func(volume):
 def whisper(text):
 return text.lower() + "..."

 def yell(text):
 return text.upper() + "!"

 return yell if volume > 0.5 else whisper

Our get_speak_funct does NOT actually call any of its inner functions. Instead, it simply selects the appropriate inner function based on the volume argument and then returns the function object.

>>> get_speak_func(0.3)
<function get_speak_func.<locals>.whisper at 0x10ae18>

>>> get_speak_func(0.7)
<function get_speak_func.<locals>.yell at 0x1008c8>

>>> speak_func = get_speak_func(0.7)
>>> speak_func('Hello')
'HELLO!'

Functions Can Capture Local State

Not only can functions return other functions, these inner functions can also capture and carry some of the parent function’s state with them.

Let’s slightly rewrite the get_speak_func above:

def get_speak_func(text, volume):
 def whisper():
 return text.lower() + "..."

 def yell():
 return text.upper() + "!"

 return yell if volume > 0.5 else whisper

>>> get_speak_func('Hello, World', 0.7)()
'HELLO, WORLD!'

Notice that the inner functions whisper and yell do not have a text parameter. But somehow they can still access the text parameter defined in the parent function. They seem to capture and “remember” the value of that argument.

Functions that do this are called lexical closures (or just closures). A closure remembers the values from its enclosing lexical scope even when the program flow is no longer in that scope.

This means not only can functions return behaviors but they can also pre-configure those behaviors. Example:

def make_adder(n):
 def add(x):
 return x + n
 return add

>>> plus_3 = make_adder(3)

>>> plus_3(4)
7

>>> make_adder(5)(4)
9

Here make_adder serves as a factory to create and configure “adder” functions. Notice that the inner add function can still access the n argument of the make_adderfunction, since it is in thee enclosing scope.

Object can behave like functions

Objects can be made callable, which allows us to treat them like functions in many cases.

If an object is callable, we can use the round parentheses function call syntax on it and even pass in function call arguments. This is powered by the __call__ dunder method. Under the hood, “calling” an object instance as a function attempts to execute the object’s __call__ method.

Example:

class Adder:
 def __init__(self, n):
 self.n = n

 def __call__(self, x):
 return self.n + x

>>> plus_3 = Adder(3)
>>> plus_3(4)
7

Notice that NOT all objects are callable. We can use the built-in callable function to check whether an object appears to be callable or not.

Decorator: Advance

Mon, 23 May 2022 00:00:00 +0000

In Decorator: Basics, we have seen the basics of decorators.

import functools

def decorator(func):
 @functools.wraps(func)
 def wrapper_decorator(*args, **kwargs):
 # Do something before
 value = func(*args, **kwargs)
 # Do something after
 return value
 return wrapper_decorator

The key is to remember the following two expressions are equivalent (the latter uses the syntactic sugar):

def function():
 pass

function = decorator(function)

@decorator
def function():
 pass

Here we will explore more advanced features

Decorating classes
Nesting decorators
Decorators with arguments
Stateful decorators
Classes as Decorators

We will resue some custom decorators defined before:

import functools

def debug(func):
 """Print the function signature and return value"""
 @functools.wraps(func)
 def wrapper_debug(*args, **kwargs):
 args_repr = [repr(arg) for arg in args]
 kwargs_repr = [f"{k}={v!r}" for k, v in kwargs.items()]
 signature = ", ".join(args_repr + kwargs_repr)
 print(f"Calling {func.__name__}({signature})")
 value = func(*args, **kwargs)
 print(f"{func.__name__!r} returned {value!r}")
 return value

 return wrapper_debug


def timer(func):
 """Print the runtime of the decorated function"""
 @functools.wraps(func)
 def wrapper_timer(*args, **kwargs):
 start_time = time.perf_counter()
 value = func(*args, **kwargs)
 end_time = time.perf_counter()
 runtime = end_time - start_time
 print(f"Finished {func.__name__!r}: in {runtime:.4f} secs.")
 return value

 return wrapper_timer


def do_twice(func):
 @functools.wraps(func)
 def wrapper_do_twice(*args, **kwargs):
 func(*args, **kwargs)
 return func(*args, **kwargs)
 return wrapper_do_twice

Decorating Classes

There are two different ways to use decorators on classes

Decorate the methods of a class
Decorate the whole class

Decorate the methods of a class

Some commonly used decorators (that are even built-ins in Pythons)

@classmethod, @staticmethod

Define methods inside a class namespace that are not connected to a particular instance of that class
@property

Customize getters and setters for class attribute

Example

class Circle:

 def __init__(self, radius):
 self._radius = radius

 # radius is a mutable property: it can be set to a different value.
 @property
 def radius(self):
 """Get value of radius"""
 return self._radius

 @radius.setter
 def radius(self, value):
 """Set radius, raise error if negative"""
 if value >= 0:
 self._radius = value
 else:
 raise ValueError("Radius must be positive")

 # area is an immutable property 
 # since preperties without setter methods can not be changed
 @property
 def area(self):
 """Calculate area inside circle"""
 return self.pi() * self.radius ** 2

 # regular method
 def cylinder_volume(self, height):
 """Calculate volume of cylinder with circle as base"""
 return self.area * height

 # Class method, which is not bound to one particular instance of Circle.
 # Class methods are often used as factory methods that can create specific instances of the class.
 @classmethod
 def unit_circle(cls):
 """Factory method creating a circle with radius 1"""
 return cls(1)

 # Static method.
 # It’s not really dependent on the Circle class, except that it is part of its namespace. 
 # Static methods can be called on either an instance or the class.
 @staticmethod
 def pi():
 """Value of π, could use math.pi instead though"""
 return 3.14155926535

>>> c = Circle(5)
>>> c.radius
5

>>> c.area
78.5398163375

>>> c.radius = 2
>>> c.area
12.566370614

>>> c.area = 100
AttributeError: can't set attribute

>>> c.cylinder_volume(height=4)
50.265482456

>>> c.radius = -1
ValueError: Radius must be positive

>>> c = Circle.unit_circle()
>>> c.radius
1

>>> c.pi()
3.1415926535

>>> Circle.pi()
3.1415926535

We can also apply custom decoraters to decorate methods.

Example

class TimeWaster:
 @debug
 def __init__(self, max_num):
 self.max_num = max_num

 @timer
 def waste_time(self, num_times):
 for _ in range(num_times):
 sum([i**2 for i in range(self.max_num)])

>>> tw = TimeWaster(1000)
Calling __init__(<time_waster.TimeWaster object at 0x7efccce03908>, 1000)
'__init__' returned None

>>> tw.waste_time(999)
Finished 'waste_time' in 0.3376 secs

Decorate the whole class

A common use of class decorators is to be a simpler alternative to some use-cases of metaclasses. For example, the new dataclasses module in Python 3.7

from dataclasses import dataclass

@dataclass
class PlayingCard:
 rank: str
 suit: str

The meaning of the syntax is similar to the function decorators: In the example above, you could have done the decoration by writing PlayingCard = dataclass(PlayingCard).

Writing a class decorator is very similar to writing a function decorator. The only difference is that the decorator will receive a class and not a function as an argument.

Example

@timer # a shorthand for TimeWaster = timer(TimeWaster)
class TimeWaster:
 def __init__(self, max_num):
 self.max_num = max_num

 def waste_time(self, num_times):
 for _ in range(num_times):
 sum([i**2 for i in range(self.max_num)])

Decorating a class does NOT decorate its methods. Here, @timer only measures the time it takes to instantiate the class:

>>> tw = TimeWaster(1000)
Finished 'TimeWaster' in 0.0000 secs

>>> tw.waste_time(999)
>>>

Nesting Decorators

You can apply more than one decorator to a function. This accumulates their effects and it’s what makes decorators so helpful as reusable building blocks.

The order of application of decorators is from bottom to top. It is sometimes called decorator stacking: You start building the stack at the bottom and then keep adding new blocks on top to work your way upwards.

Example

@debug
@do_twice
def greet(name):
 print(f"Hello {name}")

The decorators will be executed in the order they are listed. In other words, @debug calls @do_twice, which calls greet(), or debug(do_twice(greet())).

>>> greet("Eva")
Calling greet('Eva')
Hello Eva
Hello Eva
'greet' returned None

Change the order of @debug and @do_twice:

from decorators import debug, do_twice

@do_twice
@debug
def greet(name):
 print(f"Hello {name}")

In this case, @do_twice will be applied to @debug as well (do_twice(debug(greet()))):

>>> greet("Eva")
Calling greet('Eva')
Hello Eva
'greet' returned None
Calling greet('Eva')
Hello Eva
'greet' returned None

Example

We define two decorators which wrap the output string of the decorated function in HTML tags.

def strong(func):
 def wrapper():
 return f"<strong> {func()} </strong>"
 return wrapper


def emphasis(func):
 def wrapper():
 return f"<em> {func} </em>"
 return wrapper

Apply them to the greet() function at the sme time:

@strong
@emphasis
def greet():
 return "Hello!"

>>> greet()
'<strong><em>Hello!</em></strong>'

Decorators With Arguments

Syntax for decorators with parameters:

@decorator(params)
def func_name():
 ''' Function implementation'''

The above code is equivalent to

def func_name():
 ''' Function implementation'''

func_name = (decorator(params))(func_name)

To create a decorator that accepts arguments, you need to create a “meta-decorator” function that

takes arguments and
returns a regular decorator (which in turns returns a function)

Example: Extend @do_twice to a @repeat(num_times) decorator, where the number of times to execute the decorated function could then be given as an argument.

import functools

def repeat(num_times):
 # By passing num_times, a closure is created, 
 # where the value of num_times is stored until it will be used later by wrapper_repeat()

 ### Just a common regular decorator ###
 def decorator_repeat(func):
 @functools.wraps(func)
 def wrapper_repeat(*args, **kwargs):
 for _ in range(num_times):
 value = func(*args, **kwargs)
 return value
 return wrapper_repeat
 #######################################

 return decorator_repeat

@repeat(num_times=4)
def greet(name):
 print(f"Hello {name}")

>>> greet("World")
Hello World
Hello World
Hello World
Hello World

More flexible: optionally take arguments

We can also define decorator that can be used both with and without arguments.

import functools

def name(
 original_func=None,
 *, # enforce that following parameters are keyword-only
 kw1=val1,
 kw2=val2,
 ...
):
 def decorator_name(function):
 @functools.wraps(function)
 def wrapper_function(*args, **kwargs):
 ...

 return wrapper_function

 if original_func:
 return decorator_name(original_func)

 return decorator_name

Explanation

When the name is called with no optional arguments
```
@name
def function():
 ...
```
The function is passed as the first argument and the decorated function will be returned:
```
function = name(original_func=function)
```
As name(original_func=function) returns decorator_name(function), the code above is equivalent to
```
function = decorator_name(function)
```
which is the same as a regular decorator.
When the decorator is called with one or more optional arguments
```
@name(kw1="some value")
def function():
 ...
```
The name is called with original_func=None and kw1="some value":
```
function = (name(original_func=None, * kw1="some value"))(function)
```
As name(original_func=None, * kw1="some value") returns decorator_name, the code above is equivalent to
```
function = decorator_name(function)
```
, as expected.

Example: Apply this boilerplate on the @repeat decorator

def repeat(original_func=None, *, num_times=2):
 def decorator_repeat(func):
 @functools.wraps(func):
 def wrapper_repeat(*args, **kwargs):
 for _ in range(num_times):
 value = func(*args, **kwargs)
 return value
 return wrapper_repeat

 if original_func:
 return decorator_repeat(original_func)

 return decorator_repeat

@repeat
def say_whee():
 print("Whee!")

@repeat(num_times=3)
def greet(name):
 print(f"Hello {name}")

>>> say_whee()
Whee!
Whee!

>>> greet("Penny")
Hello Penny
Hello Penny
Hello Penny

Stateful Decorators

Sometimes it’s useful to have a decorator that can keep track of state. We can use the function attribute of the wrapper function to store the state.

Example: create a decorator that counts the number of times a function is called.

import functools

def count_calls(func):
 @functools.wraps(func)
 def wrapper_count_calls(*args, **kwargs):
 wrapper_count_calls.num_calls += 1 # update the state
 print(f"Call {wrapper_count_calls.num_calls} of {func.__name__!r}")
 return func(*args, **kwargs)
 wrapper_count_calls.num_calls = 0 # store the initialized state using the function attribute
 return wrapper_count_calls

@count_calls
def say_whee():
 print("Whee!")

>>> say_whee()
Call 1 of 'say_whee'
Whee!

>>> say_whee()
Call 2 of 'say_whee'
Whee!

>>> say_whee.num_calls
2

Classes as Decorators

The typical way to maintain state is by using classes.

A typical implementation of a decorator class needs to implement

.__init__()
- stores a reference to the function
- do any other necessary initialization
.__call__(): the class instance needs to be callable so that it can stand in for the decorated function

Example: implement count_calls in the previous section

import functools

class CountCalls:
 def __init__(self, func):
 functools.update_wrapper(self, func)
 self.func = func
 self.num_calls = 0

 def __call__(self, *args, **kwargs):
 self.num_calls += 1
 print(f"Call {self.num_calls} of {self.func.__name__!r}")
 return self.func(*args, **kwargs)

@CountCalls
def say_whee():
 print("Whee!")

>>> say_whee()
Call 1 of 'say_whee'
Whee!

>>> say_whee()
Call 2 of 'say_whee'
Whee!

>>> say_whee.num_calls
2

Real World Examples / Use Cases

Slowing Down Code

import functools
import time

def slow_down(_func=None, *, rate=1):
 def decorator_slow_down(func):
 @functools.wraps(func)
 def wrapper_slow_down(*args, **kwargs):
 time.sleep(rate)
 print(f"Sleep for {rate} second.")
 value = func(*args, **kwargs)
 return value
 return wrapper_slow_down
 if _func:
 return decorator_slow_down(_func)

 return decorator_slow_down

@slow_down(rate=2)
def countdown(from_number):
 if from_number < 1:
 print("Lift off!")
 else:
 print(from_number)
 return countdown(from_number - 1)

>>> countdown(5)
Sleep for 2 second.
5
Sleep for 2 second.
4
Sleep for 2 second.
3
Sleep for 2 second.
2
Sleep for 2 second.
1
Sleep for 2 second.
Lift off!

Creating Singletons

A singleton is a class with only ONE instance.

The idea of turning a class into a singleton

Store the first instance of the class as an attribute
Simply freturn the stored instance if later attempts at creating an instance

Example:

import functools

def singleton(cls):
 """Make a class a Singleton class (only one instance)"""
 @functools.wraps(cls)
 def wrapper_singleton(*args, **kwargs):
 if not wrapper_singleton.instance:
 wrapper_singleton.instance = cls(*args, **kwargs)
 return wrapper_singleton.instance

 wrapper_singleton.instance = None
 return wrapper_singleton

@singleton
class TheOne:
 pass

>>> first_one = TheOne()
>>> another_one = TheOne()
>>> id(first_one) == id(another_one)
True
>>> first_one is another_one
True

We can also implement singleton with class decorator:

import functools

class Singleton:

 def __init__(self, cls):
 functools.update_wrapper(self, cls)
 self.cls = cls
 self.instance = None

 def __call__(self, *args, **kwargs):
 if not self.instance:
 self.instance = self.cls(*args, **kwargs)

 return self.instance

@Singleton
class OnlyOne:
 pass

>>> first_one = OnlyOne()
>>> another_one = OnlyOne()
>>> id(first_one) == id(another_one)
True
>>> first_one is another_one
True

Caching Return Values

Decorators can provide a nice mechanism for caching and memoization. We can use the function attribute of the inner wrapper function to store a cache lookup table.

Example: Fibonacci

def cache(func):
 """Keep a cache of previous function calls"""
 @functools.wraps(func)
 def wrapper_cache(*args, **kwargs):
 # args_repr = [repr(arg) for arg in args]
 # kwargs_repr = [f"{k}={v!r}" for k, v in kwargs.items()]
 # signature = ", ".join(args_repr + kwargs_repr)
 # print(f"{func.__name__}({signature}) cache: {wrapper_cache.cache}")

 cache_key = args + tuple(kwargs.items())
 if cache_key not in wrapper_cache.cache:
 wrapper_cache.cache[cache_key] = func(*args, **kwargs)
 return wrapper_cache.cache[cache_key]
 wrapper_cache.cache = dict()
 return wrapper_cache

@cache
@count_calls
def fibonacci(num):
 if num < 2:
 return num
 return fibonacci(num - 1) + fibonacci(num - 2)

The function attribute wrapper_cache.cache works a s a lookup table, which use the function argument(s) as the key. So now fibonacci() only does the necessary calculations ONCE.

For example, if we have called fibonacci(2) (which returns 1) before, the item {(2, ): 1} will be stored in wrapper_cache.cache, the lookup table. Next time when we call fibonacci(2), we do not need to execute the whole function again. Instead, we just retrieve the value from the lookup table.

>>> fibonacci(10)
Call 1 of 'fibonacci'
...
Call 11 of 'fibonacci'
55

>>> fibonacci(8)
21

Note: in the final call to fibonacci(8), no new calculations were needed, since the eighth Fibonacci number had already been calculated for fibonacci(10).

In the standard library, a Least Recently Used (LRU) cache is available as @functools.lru_cache.

has more features
You should use @functools.lru_cache instead of writing your own cache decorator

import functools

@functools.lru_cache(maxsize=4)
def fibonacci(num):
 print(f"Calculating fibonacci({num})")
 if num < 2:
 return num
 return fibonacci(num - 1) + fibonacci(num - 2)

Reference

Primer on Python Decorators: detailed and clear tutorial for Python decorators
Decorators with arguments
- Python Tutorial: Decorators With Arguments
- Python Decorators 2: Decorators with arguments
- Making decorators with optional arguments

Lambda Function

Sat, 03 Dec 2022 00:00:00 +0000

The lambda keyword in Python provides a shortcut for declaring small anonymous functions. Lambda functions behave just like regular functions declared with the def keyword.

Example:

>>> add = lambda x, y: x + y
>>> add(5, 3)
8

It is equivalent to declaring the same add function with the def keyword (which would be slightly moer verbose):

>>> def add(x, y):
... return x + y
>>> add(5, 3)
8

We can also define a lambda function and then immediately evaluated it inline, without binding the function object to a name:

>>> (lambda x, y: x + y)(5, 3)
8

The syntactic difference between lambdas and regular function definitions is: Lambda functions are restricted to a single expression. This means a lambda function can’t use statements or annotations—not even a return statement.

Executing a lambda function evaluates its expression and then automatically returns the expression’s result, so there’s always an implicit return statement.

When to use `lambda`?

Lambdas provide a handy and “unbureaucratic” shortcut to defining a function in Python. A typical use case for lambdas is writing short and concise key funcs for sorting iterables by an alternate key:

>>> tuples = [(1, 'd'), (2, 'b'), (4, 'a'), (3, 'c')]
>>> sorted(tuples, key=lambda x: x[1])
[(4, 'a'), (2, 'b'), (3, 'c'), (1, 'd')]

Just like regular nested functions, lambdas also work as lexical closures.

Example:

>>> def make_adder(n):
... return lambda x: x + n

>>> plus_3 = make_adder(3)

>>> plus_3(4)
7

The x + n lambda can still access the value of n even though it was defined in the make_adder function (the enclosing scope).

When NOT to use `lambda`?

Lambda functions should be used sparingly and with extraordinary care!!!

If you’re tempted to use a lambda, spend a few seconds (or minutes) to think if it is really the cleanest and most maintainable way to achieve the desired result.

For example:

# Bad readablility with lambda:
>>> list(filter(lambda x: x % 2 == 0, range(16))) [0, 2, 4, 6, 8, 10, 12, 14]

# Better readablility with list comprehension
>>> [x for x in range(16) if x % 2 == 0] [0, 2, 4, 6, 8, 10, 12, 14]

Do not use lambda just for show-off purpose.

If you find yourself doing anything remotely complex with lambda expressions, consider defining a standalone function with a proper name instead.
Saving a few keystrokes won’t matter in the long run, but your col- leagues (and your future self) will appreciate clean and readable code more than terse wizardry.

Function: Return `None`

Sun, 04 Dec 2022 00:00:00 +0000

Python adds an implicit return None statement to the end of any function. In other words, if a function doesn’t specify a return value, it returns None by default.

This means you can replace return None statements with bare return statements or even leave them out completely and still get the same result. E.g.:

def foo1(value):
 if value:
 return value
 else:
 return None


def foo2(value):
 """Bare return statement implies `return None`"""
 if value:
 return value
 else:
 return


def foo3(value):
 """Missing return statement implies `return None`"""
 if value:
 return value

>>> type(foo1(0))
<class 'NoneType'>

>>> type(foo2(0))
<class 'NoneType'>

>>> type(foo3(0))
<class 'NoneType'>

Rule of thumb:

If a function does NOT have a return value (other languages would call this a procedure), then leave out the return statement.
If a function does have a return value from a logical point of view, then you need to decide whether to use an im- plicit return or not.

Looping and Iterations

Sat, 14 Jan 2023 00:00:00 +0000

TL;DR

Iterators provide a sequence interface to Python objects that’s memory efficient and considered Pythonic.
To support iteration an object needs to implement the iterator protocol by providing the __iter__ and __next__ dunder methods.

Writing Pythonic Loops

in Python, for-loops are really “for-each” loops that can iterate directly over items from a container or sequence, without having to look them up by index.

Example:

my_items = ['a', 'b', 'c']

for item in my_items:
 print(item)

Advantages

Remains nice and clean and almost reads like pseudo code from a programming textbook
We don’t need to take care of the container’s size and looping index
The container itself now takes care of handing out the elements so they can be processed.
- If the container is ordered, the resulting sequence of elements will be too.
- If the container isn’t ordered, it will return its elements in arbitrary order but the loop will still cover all of them.

If we need to keep track of the running index, we can use the enumerate() built-in.

Example:

for i, item in enumerate(my_items):
 print(f"{i}: {item}")

If we inevitablely need to write a Java- or C-style loop (e.g., we must contrl the step size for the index),:

for (int i = a; i < n; i += s) {
 // ...
}

We can make use of the range() function: it accepts optional parameters to control the start value for the loop (a), the stop value (n), and the step size (s).

for i in range(a, n, s):
 # ...

Comprehensions

Comprehensions are a key feature in Python. Under the hood, comprehension are for`-loops over a collection but expressed in a more terse and compact syntax.

# list comprehension
values = [expression for item in collection]

It is equivalent to

# plain for-loop
values = []
for item in collection:
 values.append(expression)

List comprehensions can filter values based on some arbitrary condi- tion that decides whether or not the resulting value becomes a part of the output list.

values = [expression
 for item in collection
 if condition]

Example:

>>> even_squares = [x * x for x in range(10) if x % 2 == 0]
>>> even_squares
[0, 4, 16, 36, 64]

Python also supports comprehensions for sets and dictionaries.

Set comprehension:

>>> { x * x for x in range(-9, 10) }
set([64, 1, 36, 0, 49, 9, 16, 81, 25, 4])

Unlike lists, which retain the order of the elements in them, Python sets are an unordered collection type.

Dictionary comprehension:

>>> { x: x * x for x in range(5) }
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

As you get more proficient at using them, it becomes easier and easier to write code that’s difficult to read. If you’re not careful, you might have to deal with monstrous list, set, and dict comprehensions soon.

Remember, too much of a good thing is usually a bad thing. And readabilty always has the first priority!

List Slicing Tricks

List slicing

We can view it as an extension of the square-brackets indexing syntax
Commonly used to access ranges of elements within an ordered col- lection.

List slicing pattern

list[start:stop:step]

Note: To avoid off-by-one errors, the upper bound stop is always exclusive.

Example

>>> lst = [1, 2, 3, 4, 5]
>>> lst
[1, 2, 3, 4, 5]

# lst[start:end:step]
>>> lst[1:3:1]
[2, 3]

Tricks with step parameter (a.k.a. stride)

If you leave out the step size, it defaults to one.
```
>>> lst[1:3]
[2, 3]
```
You can also control the step size to skip element.
```
>>> lst[::2]
[1, 3, 5]
```
If you ask for a [::-1] slice, you’ll get a copy of the original list, but in the reverse order:
```
>>> numbers[::-1]
[5, 4, 3, 2, 1]
```
This is pretty neat, but in most cases we should still stick with list.reverse() and the built-in reversed() function to reverse a list.

You can use the :-operator to clear all elements from a list without destroying the list object itself.

>>> lst = [1, 2, 3, 4, 5]
>>> del lst[:]
>>> lst
[]

In Python 3 you can also use lst.clear() for the same job, which might be the more readable patter.

Besides clearing lists, you can also use slicing to replace all elements of a list without creating a new list object.

>>> original_lst = lst
>>> lst[:] = [7, 8, 9]
>>> lst
[7, 8, 9]
>>> original_lst
[7, 8, 9]

Another use case for hte :-operator creating (shallow) copies of existing lists:

>>> copied_lst = lst[:]
>>> copied_lst
[7, 8, 9]
>>> copied_lst is lst
False

Beautiful Iterators

According to Python’s iterator protocol: Objects that support the __iter__ and __next__ dunder methods automatically work with for-in loops.

To understand how iterators work under the hood, let’s take a look at an example.

class Repeater:
 def __init__(self, value):
 self.value = value

 def __iter__(self):
 return RepeaterIterator(self)


class RepeaterIterator:
 def __init__(self, source):
 self.source = source

 def __next__(self):
 return self.source.value

The two dunder method, __iter__ and __next__, are the keys to making a Python object iterable.

repeater = Repeater("Hello")

for item in repeater:
 print(item)

You will see lots of "Hello" printed to the screen, and this loop will never complete. But in other words, this iterator works!

To dispel some of that “magic,” we can expand this loop into a slightly longer code snippet that gives the same result:

repeater = Repeater('Hello')
iterator = repeater.__iter__()
while True:
 item = iterator.__next__()
 print(item)

As we can see, the for-in was just syntactic sugar for a simple while loop:

It first prepared the repeater object for iteration by calling its __iter__ method → This returns the actual iterator object.
Then the loop repeatedly called the iterator object’s __next__ method to retrieve values from it.

(Because there’s never more than one element “in flight,” this approach is highly memory-efficient.)

On more abstract terms, iterators provide a common interface that allows you to process every element of a container while being completely isolated from the container’s internal structure.

→ In other words, whether you’re dealing with a list of elements, a dictionary, or an infinite sequence, it does not matter, because every single one of these objects can be traversed in the same way with the power iterators.

We can also manually “emulate” how the loop uses the iterator protocol:

>>> repeater = Repeater('Hello')
>>> iterator = iter(repeater)
>>> next(iterator)
'Hello'
>>> next(iterator)
'Hello'
>>> next(iterator)
'Hello'

Notice that here we replace the __iter__ and __next__ with Python’s built-in functions, iter() and next().

Internally, these built-ins invoke the same dunder methods, but they make this code a little prettier and easier to read by providing a clean “facade” to the iterator protocol.

Python offers these facades for other functionality as well. E.g.

len(x) is a shortcut for calling x.__len__.

Generally, it’s a good idea to use the built-in facade functions rather than directly accessing the dunder methods implementing a protocol. It just makes the code a little easier to read.

A simple iterator

In the previous example, Repeater and RepeaterIterator correpsonded directly to the two phases used by Python’s iterator protocol:

Repeater : setting up and retrieving the iterator object with an iter() call,
RepeaterIterator: repeatedly fetching values from it via next().

Both of these responsibilities can be shouldered by a single class. it doesn’t really matter where __next__ is defined. In the iterator protocol, all that matters is that __iter__ returns any object with a __next__ method on it.

→ We can add the __next__ method directly to the Repeater class!

class SimpleRepeater:
 def __init__(self, value):
 self.value = value

 def __iter__(self):
 return self

 def __next__(self):
 return self.value

Iterators that can stop

Iterators use exceptions to structure control flow. To signal the end of iteration, a Python iterator simply raises the built-in StopIteration exception.

Python iterators normally can NOT be “reset”—once they’re exhausted they’re supposed to raise StopIteration every time next() is called on them. To iterate anew you’ll need to request a fresh iterator object with the iter() function.

Having this idea in mind, we can implement a BoundedRepeater that will stop after a predefined number of repetitions:

class BoundedRepeater:
 def __init__(self, value, max_reps):
 self.value = value
 self.max_reps = max_reps
 self.count = 0

 def __iter__(self):
 return self

 def __next__(self):
 if self.count >= self.max_reps:
 raise StopIteration
 self.count += 1
 return self.value

>>> repeater = BoundedRepeater('Hello', 3)
>>> for item in repeater:
 print(item)
Hello
Hello
Hello

Generator

Fri, 20 Jan 2023 00:00:00 +0000

In Python, a generator is a function that returns an iterator that produces a sequence of values when iterated over.

Generators are useful when we want to produce a large sequence of values, but we don’t want to store all of them in memory at once.

Understanding Generators

Generator functions look and act just like regular functions, but with one defining characteristic. Generator functions use the Python yield keyword instead of return.

For example: generating infinite sequence:

def infinite_sequence():
 num = 0
 while True:
 yield num
 num += 1

This looks like a typical function definition, except for the Python yield statement and the code that follows it.

yield indicates where a value is sent back to the caller, but unlike return, you don’t exit the function afterward.
Instead, the state of the function is remembered. That way, when next() is called on a generator object (either explicitly or implicitly within a for loop), the previously yielded variable num is incremented, and then yielded again.

Building generators

Define as function

Similar to defining a normal function, we can define a generator function using the def keyword, but instead of the return statement we use the yield statement.

def generator_name(arg):
 # statements
 yield something

When the generator function is called, it does not execute the function body immediately. Instead, it returns a generator object that can be iterated over to produce the values.

Define using comprehension

Like list comprehensions, generator expressions allow you to quickly create a generator object in just a few lines of code.

(expression for item in iterable)

The generator expression creates a generator object that produces the values of expression for each item in the iterable, one at a time, when iterated over.

Generator comprehensions are also useful in the same cases where list comprehensions are used, with an added benefit: you can create them without building and holding the entire object in memory before iteration.

Example: squaring numbers

num_squared_list = [num ** 2 for num in range(5)] # list comprehension
num_squared_generator = (num ** 2 for num in range(5)) # generator comprehension

>>> num_squared_list
[0, 1, 4, 9, 16]

>>> num_squared_generator
<generator object <genexpr> at 0x7f9c604c8430>

Profiling generator performance

Memory:

>>> import sys
>>> nums_squared_lc = [i ** 2 for i in range(10000)]
>>> sys.getsizeof(nums_squared_lc) # in bytes
87624
>>> nums_squared_gc = (i ** 2 for i in range(10000))
>>> print(sys.getsizeof(nums_squared_gc)) # in bytes
120

The list is over 700 times larger than the generator object!

Speed:

>>> import cProfile
>>> cProfile.run('sum([i * 2 for i in range(10000)])')
 5 function calls in 0.001 seconds

 Ordered by: standard name

 ncalls tottime percall cumtime percall filename:lineno(function)
 1 0.001 0.001 0.001 0.001 <string>:1(<listcomp>)
 1 0.000 0.000 0.001 0.001 <string>:1(<module>)
 1 0.000 0.000 0.001 0.001 {built-in method builtins.exec}
 1 0.000 0.000 0.000 0.000 {built-in method builtins.sum}
 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}


>>> cProfile.run('sum((i * 2 for i in range(10000)))')
 10005 function calls in 0.003 seconds

 Ordered by: standard name

 ncalls tottime percall cumtime percall filename:lineno(function)
 10001 0.002 0.000 0.002 0.000 <string>:1(<genexpr>)
 1 0.000 0.000 0.003 0.003 <string>:1(<module>)
 1 0.000 0.000 0.003 0.003 {built-in method builtins.exec}
 1 0.001 0.001 0.003 0.003 {built-in method builtins.sum}
 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}

Summing across all values in the list comprehension took about a third of the time as summing across the generator.

If the list is smaller than the running machine’s available memory, then list comprehensions can be faster to evaluate than the equivalent generator expression.

Memory matters → Use generator comprehension.

Runtime matters → Use list comprehension.

Understanding the `yield` Statement

On the whole, yield is a fairly simple statement. Its primary job is to control the flow of a generator function in a way that’s similar to return statements.

When the yield statement is hit/arrived, the program suspends function execution and returns the yielded value to the caller. (In contrast, return stops function execution completely.) When a function is suspended, the state of that function is saved (including variables binding local to the generator, the instruction point, the internal stack, and any exception handling).
This allows you to resume function execution whenever you call one of the generator’s methods. In this way, all function evaluation picks back up right after yield.

Example:

def infinite_sequence():
 num = 0
 while True:
 yield num
 num += 1
 yield f"Next number to yield: {num}"

>>> inf_seq_generator = infinite_sequence()
>>> inf_seq_generator
<generator object infinite_sequence at 0x7f9c604c8b30>

# The `inifinite_sequence` runs and arrives line 4,
# the execution is suspended by yield statement. 
# The `num` (0 in this case) is yielded.
>>> next(inf_seq_generator)
0

# The funtion execution picks back up right after `yield num`.
# I.e., the program continues from line 5, `num += 1`. `num` is incremented to 1.
# Then the `yield` statement at line 6 is arrived
# --> It suspends the program and yield "Next number to yield: 1"
>>> next(inf_seq_generator)
'Next number to yield: 1'

>>> next(inf_seq_generator)
1

>>> next(inf_seq_generator)
'Next number to yield: 2'

>>> next(inf_seq_generator)
2

Use of Python Generators

There are several reasons that make generators a powerful implementation.

Easy to Implement

Generators can be implemented in a clear and concise way as compared to their iterator class counterpart.

Memory Efficient

A normal function to return a sequence will create the entire sequence in memory before returning the result. This is an overkill, if the number of items in the sequence is very large.

In contrast, Generator implementation of such sequences is memory friendly and is preferred since it only produces one item at a time.

Represent Infinite Stream

Generators are excellent mediums to represent an infinite stream of data. Infinite streams cannot be stored in memory, and since generators produce only one item at a time, they can represent an infinite stream of data.

E.g., the following generator function can generate all the even numbers (at least in theory).

def all_even():
 n = 0
 while True:
 yield n
 n += 2

Pipelining Generators

Multiple generators can be used to pipeline a series of operations.

E.g., suppose we have a generator that produces the numbers in the Fibonacci series. And we have another generator for squaring numbers. If we want to find out the sum of squares of numbers in the Fibonacci series, we can do it in the following way by pipelining the output of generator functions together.

def fibonacci_numbers(nums):
 x, y = 0, 1
 for _ in range(nums):
 x, y = y, x+y
 yield x

def square(nums):
 for num in nums:
 yield num**2

print(sum(square(fibonacci_numbers(10))))

# Output: 4895

Reference

Import

Sat, 11 Feb 2023 00:00:00 +0000

In Python, we use the import keyword to make code in one module available in another.

Imports in Python are important for structuring your code effectively. Using imports properly will make you more productive, allowing you to reuse code while keeping your projects maintainable.

Basic Python Import

Modules and packages

Module

Definition (according to Python.org glossary)

An object that serves as an organizational unit of Python code. Modules have a namespace containing arbitrary Python objects. Modules are loaded into Python by the process of importing. (Source)
Usually corresponds to one .py file containing Python code

Example

>>> import math # import the code in the `math` module
>>> math.pi # access the `pi` variable within the math module
3.141592653589793

Acts as a namespace that keeps all the attributes of the module together.

List the content of a namespace with dir():

>>> import math
>>> dir() # shows what’s in the global namespace
['__annotations__', '__builtins__', ..., 'math']

>>> dir(math) # shows what's in the `math` namespace
['__doc__', ..., 'nan', 'pi', 'pow', ...]

Ways to import

Direct import
```
import math
```

Import specific parts of a module

>>> from math import pi
>>> pi
3.141592653589793

>>> math.pi
NameError: name 'math' is not defined

In this case pi is placed in the global namespace and NOT within math namespace.

Rename modules and attributes as they’re imported

>>> import math as m
>>> m.pi
3.141592653589793

>>> from math import pi as PI
>>> PI
3.141592653589793

Importing a module both loads the contents and creates a namespace containing the contents.

Package

Definition

A Python module which can contain submodules or recursively, subpackages. Technically, a package is a Python module with an __path__ attribute. (Source)

(A package is still a module. As a user, you usually don’t need to worry about whether you’re importing a module or a package.)
package typically corresponds to a file directory containing Python files and other directories.
To create a Python package yourself, you create
- a directory and
- a file named __init__.py inside it
  - contains the contents of the package when it’s treated as a module
  - can be left empty.
In general, submodules and subpackages ar NOT imported when you import a package.
- You can use __init__.py to include any or all submodules and subpackages if you want. It’s fairly common to import subpackages and submodules in an __init__.py file to make them more readily available to your users.
- Submodules and subpackages included in __init__.py will be then imported along with the package
- Good Example: __init__.py of the request package

Absolute and relative imports

Let’s say we have a package called world and it includes a subpackage called africa.

In world/__init__.py:

from . import africa

The dot (.) refers to the current package, and the statement is an example of a relative import. You can read it as “From the current package, import the subpackage africa.”

There’s an equivalent absolute import statement in which you explicitly name the current package:

from world import africa

The PEP 8 style guide recommends using absolute imports in general. However, relative imports are an alternative for organizing package hierarchies. (For more information, see Absolute vs Relative Imports in Python.)

Python’s import path

Python looks for modules and packages in its import path, which is a list of locations that are searched for modules to import.

When you type import something, Python will look for something a few different places before searching the import path.

In particular, it’ll look in a module cache to see if something has already been imported, and it’ll search among the built-in modules.

You can inspect Python’s import path by printing sys.path. This list will contain three different kinds of locations:

The directory of the current script (or the current directory if there’s no script, such as when Python is running interactively). This location is always the first in the searching list.
The contents of the PYTHONPATH environment variable
Other, installation-dependent directories

Python will start at the beginning of the list of locations and look for a given module in each location until the first match.

You should always be careful that you don’t create modules that shadow, or hide, other important modules.

For example, we define our custom math module:

# math.py

def double(number):
 return 2 * number

>>> import math
>>> math.double(3.14)
6.28

But this module also shadows the standard math module that’s included in the standard library! That means our earlier example of looking up the value of $\pi$ no longer works:

>>> import math
>>> math.pi
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
AttributeError: module 'math' has no attribute 'pi'

>>> math
<module 'math' from 'math.py'>

In other words, Python now searches your new math module for pi instead of searching the math module in the standard library.

To avoid these kinds of issues, you should be careful with the names of your modules and packages. In particular, your top-level module and package names should be unique.

Namespace packages

Namespace packages

less dependent on the underlying file hierarchy
can be split across multiple directories
A namespace package is created automatically if you have a directory containing a .py file but no __init__.py.

Import style guide

PEP 8, the Python style guide, has a couple of recommendations about imports.

To keep your code both readable and maintainable, here are a few general rules of thumb for how to style your imports:

Keep imports at the top of the file.
Write imports on separate lines.
Organize imports into groups: first standard library imports, then third-party imports, and finally local application or library imports.
Order imports alphabetically within each group.
Prefer absolute imports over relative imports.
Avoid wildcard imports like from module import *.

isort and reorder-python-imports are great tools for enforcing a consistent style on your imports.

Resource Imports

Sometimes you’ll have code that depends on data files or other resources. If the resource file is important for your package and you want to distribute your package to other users, then a few challenges will arise:

You won’t have control over the path to the resource since that will depend on your user’s setup as well as on how the package is distributed and installed. You can try to figure out the resource path based on your package’s __file__ or __path__ attributes, but this may not always work as expected.
Your package may reside inside a ZIP file or an old .egg file, in which case the resource won’t even be a physical file on the user’s system.

With the introduction of importlib.resources into the standard library in Python 3.7, there’s now one standard way of dealing with resource files.

`importlib.resources`

Gives access to resources within packages
- A resource is any file located within an importable package. The file may or may not correspond to a physical file on the file system.
Advantages
- More consistent way to deal with the files inside your packages
- Gives you easier access to resource files in other packages
Requirement when using importlib.resources:
- Your resource files must be available inside a regular package. Namespace packages aren’t supported.
  
  → In practice, this means that the file must be in a directory containing an __init__.py file.

Example

Assume you have resources inside a package like this:

books/
│
├── __init__.py
├── alice_in_wonderland.png
└── alice_in_wonderland.txt

__init__.py is just an empty file necessary to designate books as a regular package.

You can then use open_text() and open_binary() to open text and binary files, respectively:

>>> from importlib import resources
>>> with resources.open_text("books", "alice_in_wonderland.txt") as fid:
... alice = fid.readlines()

>>> with resources.open_binary("books", "alice_in_wonderland.png") as fid:
... cover = fid.read()

open_text() and open_binary() are equivalent to the built-in open() with the mode parameter set to rt and rb, respectively.

importlib.resources became part of the standard library in Python 3.7. However, on older versions of Python, a backport is available as importlib_resources. This backport is compatible with Python 2.7 as well as Python 3.4 and later versions. To use the backport, install it from PyPI:

$ python -m pip install importlib_resources

To seamlessly fall back to using the backport on older Python versions, you can import importlib.resources as follows:

try:
 from importlib import resources
except ImportError:
 import importlib_resources as resources

See the tips and tricks section of this tutorial for more information.

Dynamic imports

Using `importlib`

The whole import machinery is available in the importlib package, and this allows you to do your imports more dynamically.

Python Import Systems

Import Internals

The details of the Python import system are described in the official documentation.

At a high level, three things happen when you import a module (or package). The module is:

Searched for
Loaded
Bound to a namespace

For the usual imports—those done with the import statement—all three steps happen automatically. When you use importlib, however, ONLY the first two steps are automatic. You need to bind the module to a variable or namespace yourself.

Even when you import only one attribute from a module, the whole module is loaded and executed. The rest of the contents of the module just aren’t bound to the current namespace.

The module cache plays a very important role in the Python import system. The first place Python looks for modules when doing an import is in sys.modules. If Python finds a module in the module cache, then it won’t bother searching the import path for the module. If a module is already available, then it isn’t loaded again.

Reloading modules

Use importlib.reload() to reload a module.

Example:

>>> import number
>>> number.answer
24

>>> # Update number.py in your editor. Now `number.answer` is changed to 42.

>>> import importlib
>>> importlib.reload(number)
<module 'number' from 'number.py'>

>>> number.answer
42

Import Tips and Tricks

Handle packages across Python versions

Sometimes you need to deal with packages that have different names depending on the Python version. As long as the different versions of the package are compatible, you can handle this by renaming the package with as.

Example:

try:
 from importlib import resources
except ImportError:
 import importlib_resources as resources

In the rest of the code, you can refer to resources and not worry about whether you’re using importlib.resources or importlib_resources. 👏

Handle missing packages: Use an alternative

This case is similar to the one above. Also use try...except to handle.

try:
 # import the desired package
except ImportError:
 # import an alternative

Reference

Python import: Advanced Techniques and Tips

Big O Notation

Sun, 04 Apr 2021 00:00:00 +0000

Definition

In general, a better algorithm means a FASTER algorithm. In other words, it costs less runtime or it has lower time complexity.

Big O notation is one of the most fundamental tools to analyze the cost of an algorithm. It is about finding an asymptotic upper bound of the time complexity of an algorithm. I.e., Big O notation is about the worst-case scenario.

Basically, it answers the question: “How the runtime of an algorithm grows as the input grow?”

Here we use number of operations to evaluate the runtime of an algorithm, Instead of second/microsecond.

Rule of Thumbs

Different steps get added

Example

def func():
 step1() # O(a)
 step2() # O(b)

Overall complexity is $O(a + b)$

Constants do NOT matter

Constant are inconsequential, when input size $n$ is getting massively huge.

Example

$O(2n) = O(n)$
$O(500) = O(1)$

Different inputs $\Rightarrow$ different variables

Example

def intersection_size(list_A, list_B):
 # Assumes that len(list_A) is lenA, len(len_listB) is lenB
 count = 0
 for a in list_A:
 for b in list_B:
 if a == b:
 count += 1

 return count

Overall complexity is $O(\text{lenA} \times \text{lenB})$

Smaller terms do NOT matter

Smaller terms are non-dominant, when input size $n$ is getting massively huge.

Example

$O(n + 100) = O(n)$
$O(n^2 + 200n) = O(n^2)$

Shorthands

Arithmetric operations are constant
Variable assignment is constant
Accessing elements in an array (by index) or object (by key) is constant
In a loop:
$$ \text{overall complexity} = (\text{length of loop}) \times (\text{complexity inside of the loop}) $$

Summary of Complexity Classes

Reference

Big O Notation

Complete Beginner’s Guide to Big O Notation

Binary Search

Sun, 04 Apr 2021 00:00:00 +0000

Binary search only works when your list is in sorted order.

Code

def binary_search(list, item):
 # keep track of low and high
 low = 0
 high = len(list) - 1

 while low <= high: # when we haven't narrowed down to one element
 mid = (low + high) // 2 # index of the middle element
 guess = list[mid] # value of the middle element

 if guess == item:
 return mid
 elif guess > item:
 # Guess is too high,
 # we narrow our search on the "left" part
 high = mid - 1
 else:
 # Guess is too low,
 # we narrow our search on the "right" part
 low = mid + 1

 # If we have anrrowed down to one element but still can not find the item
 # => Item is not in the list 
 return -1

Complexity

Time complexity: $O(\log n)$

Binary Vs Linear Search

Average Case

Binary Search Worst Case

Binary Search Best Case

Recursion

Tue, 06 Apr 2021 00:00:00 +0000

Every recursive function has two parts

base case: the function does not call itself again, so it will not go into an infinite loop
recursive case: the function calls itself

Sometimes recursion could be also re-written using loops. In fact, loops are sometimes better for performance. But recursion is used when it makes the solution clearer.

Examples

Count down

def countdown(i):
 print(i)
 if i <= 0: # base case
 return
 else: # recursive case
 countdown(i-1)

countdown(5)

Factorial

factorial(3) is defined as $3! = 3 \cdot 2 \cdot 1 = 3$

def factorial(x):
 # base case
 if x == 1:
 return 1 # 1! = 1
 else:
 return x * factorial(x-1)

factorial(3)

factorial(5)

Divide and Conquer

Divide and conquer (D&C) is a well-known recursive technique for solving problems. To solve a problem using D&C, there’re 2 steps:

Figure out the base case. This should be the simplest possible case.
Divide or decrease the problem until it becomes the base case.

Sum of Array

Let’s calculate the sum of an array using D&C.

Figure out the base case.

Base case is that the array is empty. In this situation, sum of this array is 0.

Decrease the problem until it becomes the base case.

sum = a[0] + a[1] + a[2]
 = a[0] + (a[1] + a[2])
 = a[0] + (a[1] + (a[2] + 0))

Example:

As we can see, in the 2nd and 3rd step, we’re passing a smaller (sub)array into the sum function. That is, we’re decreasing the size of our problem!

When we have figured out the two steps of D&C, the implementation is pretty simple:

def sum(arr):
 if not arr: # arr is empty
 return 0
 else:
 return arr[0] + sum(arr[1:])

When you’re writing a recursive function involving an array, the base case is often an empty array or an array with one element. If you’re stuck, try that first.

Count Number of Elements

def length(arr):
 if not arr:
 # base case: length of empty array is 0
 return 0

 # recursive case: 1 + length of the remaining subarray
 return 1 + length(arr[1:])

Find Maximum

def max(arr):
 if len(arr) == 2:
 return arr[0] if arr[0] > arr[1] else arr[1]
 return arr[0] if arr[0] > max(arr[1:]) else max(arr[1:])

Recursive Binary Search

def binary_search(arr, low, high, target):
 if low <= high:
 mid = (low + high) // 2
 if arr[mid] > target:
 # If element is smaller than mid, 
 # then it can only be present in left subarray
 return binary_search(arr, low, mid-1, target)
 elif arr[mid] == target:
 return mid
 else:
 # If element is greater than mid, 
 # then it can only be present in right subarray
 return binary_search(arr, mid+1, high, target)

 else: # Element is not present in the array
 return -1

Data Structures and Collections

Sun, 01 May 2022 00:00:00 +0000

Array and Linked List

Mon, 05 Apr 2021 00:00:00 +0000

TL;DR

	Array	Linked List
Store in memory	Elements are stored in contiguous locations in memory.	Elements can be stored anywhere in memory
Size	Fixed, must be specified at the time of initialization	Changeable, can grow/shrink by insertion/deletion
Access element	Random access $O(1)$	Sequence access $O(n)$
Insert element	$O(n)$	$O(1)$
Delete element	$O(n)$	$O(1)$

Storing In Memory

Array store elements in contiguous memory locations/blocks.

By Linked list, elements (also called nodes) can be stored anywhere in memory. Each node stores its next node’s address (link/pointer to next node).

Operations

Access

Array

Accessing an element in Array is simple. As array is essentially a block of continuous memory, we can directly access an element using index.

In other words, accessing an element in the array is independent of the size of the array. Hence, time complexity is $O(1)$.

Linked List

However, accessing a node in Linked list is not that easy. Since nodes are scattered out in memory, we have to traverse the linked list from the head node until we reach the target node.

The worst case is that our target node is at the tail of the linked list. In this case we have to traverse the whole linked list. Time complexity is thus O(n).

Insert

Array

Inserting an element in the Array is a pain.

For example, we want to insert 11 in the 3rd position of array [1, 3, 5, 7, 9]. We need to

allocate a new memory block of suitable size (in our case, 6)，
put new element 11 in the 3rd position, and copy other elements to the right place.

Linked list

Key of insertion is to maintain the order/sequence of nodes using links.

Insert in the middle

Let’s say we want to insert a new_node between left_node and right_node.

Before insertion:

left_node --> right_node

After insertion, the linked should look like:

left_node --> new_node --> right_node

Keeping this in mind, we have the idea of insertion:

new_node.next = left_node.next
left_node.next = new_node

Insert at head

By inserting a new node at the head of linked list, we need to specify the new head after the insertion.

Assume that we want to insert new_node before linked_list.head:

new_node.next = linked_list.head
linked_list.head = new_node

Insert at tail

By inserting a new node at the tail of linked list, we need to specify that the new node is the tail.

Assume that we want to insert new_node after tail_node.

Before insertion:

tail_node --> None

After insertion:

tail_node --> new_node --> None

Thus, the code of insertion should be:

# traverse linked list until reaching the tail node
tail_node.next = new_node
new_node.next = None

Delete

Array

Deleting of element from an array is similar to insertion. We need to allocate a new memory of suitable size, and copy remaining elements to the right place.

Linked list

Similar to insertion, we just need to maintain the order/sequence of nodes using links.

Insert a node in the middle

Let’s say we want to delete del_node between left_node and right_node.

Before deletion:

left_node --> del_node --> right_node

After deletion:

left_node --> right_node

Therefore, we just need to do the followings

left_node.next = right_node

Example:

Delete head node

Let’s say we want to delete linked_list.head. After deletion, we need to announce that the new head is linked_list.head.next.

linked_list.head = linked_list.head.next

Delete tail node

After deleting the tail node, the second last node becomes the new tail node.

Before deletion:

left_node --> tail_node --> None

After deletion:

left_node --> None

Therefore, we just need to do the followings

left_node.next = None

Array vs. Linked List

	Array	Linked List
Access element	Fast	Slow
Insert element	Slow	Fast
Delete element	Slow	Fast

By frequent retrieval/accessing, use Array
By frequent insertion/deletion, use Linked List

LinkedList in Python

Implement our own Linked List

To implement the linked list, we firstly need to implment a class for node, element of the list:

class Node:

 def __init__(self, data):
 self.data = data
 self.next = None

 def __repr__(self):
 return self.data

Then we implment the linked list:

class LinkedList:

 def __init__(self, data_list=None):
 self.head = None
 if data_list is not None:
 node = Node(data=data_list.pop(0))
 self.head = node
 for element in data_list:
 node.next = Node(data=element)
 node = node.next

 def __repr__(self):
 node = self.head
 nodes = list()
 while node is not None:
 nodes.append(node.data)
 node = node.next
 nodes.append("None")
 return " -> ".join(nodes)

 def __iter__(self):
 node = self.head
 while node is not None:
 yield node
 node = node.next

 def add_first(self, new_node):
 """
 Insert new_node at head
 """
 new_node.next = self.head
 self.head = new_node

 def add_last(self, new_node):
 """
 Insert new_node at tail
 """

 # If linked list is empty,
 # new node will be the only node in linked list,
 # i.e. head and tail node are the same
 if self.head is None:
 self.head = new_node
 return

 # If linked list is not empty:
 # 1. we need to traverse the whole list until we reach the current last node
 # 2. we add the new node as the next node of the current last node
 for current_node in self:
 pass
 current_node.next = new_node

 def add_after(self, target_node_data, new_node):
 """
 Insert new_node after the node whose data is target_node_data
 """
 if self.head is None:
 raise Exception("Linked list is empty.")

 for node in self:
 if node.data == target_node_data:
 new_node.next = node.next
 node.next = new_node
 return

 raise Exception(f"Node with data '{target_node_data}' not found.")

 def remove_node(self, target_node_data):
 """
 Remove node whose data is target_node_data
 """

 # If linked list is empty, then raise an exception
 if self.head is None:
 raise Exception("Linked list is empty.")

 # If the node to be removed is the current head,
 # then we want the next node in the list to become the new head
 if self.head.data == target_node_data:
 self.head = self.head.next
 return

 # If list is not empty and node to be removed is not the current head,
 # we traverse the list looking for the node to be removed.
 # If we find it, then we need to update its previous node to point to its next node
 previous_node = self.head
 for node in self:
 if node.data == target_node_data:
 previous_node.next = node.next
 return

 previous_node = node

 # If we traverse the whole list without finding the node to be removed,
 # then raise an exception
 raise Exception(f"Node with data '{target_node_data}' not found.")

Traverse

llist = LinkedList(["a", "b", "c", "d", "e"])
llist

a -> b -> c -> d -> e -> None

for node in llist:
 print(node)

a
b
c
d
e

Insert

llist = LinkedList()
llist

None

Insert at head:

llist.add_first(Node("a"))
llist

a -> None

Insert at tail:

llist.add_last(Node("b"))
llist

a -> b -> None

llist.add_last(Node("d"))
llist

a -> b -> d -> None

Insert in the middle:

llist.add_after("b", Node("c"))
llist

a -> b -> c -> d -> None

Remove

llist = LinkedList(["a", "b", "c", "d", "e"])
llist

a -> b -> c -> d -> e -> None

Remove head:

llist.remove_node("a")
llist

b -> c -> d -> e -> None

Remove tail:

llist.remove_node("e")
llist

b -> c -> d -> None

Remove node in the middle:

llist.remove_node("c")
llist

b -> d -> None

Double-ended Queue (Deque)

collections.deque uses an implementation of a linked list in which you can access, insert, or remove elements from the beginning or end of a list with constant 𝑂(1) performance.

Append/Remove element from the right side: append() / pop()
Append/Remove element from the left side: appendleft() / popleft()

Implment Queue using `deque`

For a queue, we use a First-In/First-Out (FIFO) approach. I.e., the first element inserted in the list is the first one to be retrieved.

queue = deque()
queue

deque([])

queue.append("Mary")
queue.append("John")
queue.append("Susan")
queue

deque(['Mary', 'John', 'Susan'])

Retrieve: (Order should be Mary --> John --> Susan)

queue.popleft()

'Mary'

queue.popleft()

'John'

queue.popleft()

'Susan'

Implment Stack using `deque`

For a stack, we use a Last-In/Fist-Out (LIFO) approach, meaning that the last element inserted in the list is the first to be retrieved.

queue = deque()
queue

deque([])

queue.append("Mary")
queue.append("John")
queue.append("Susan")
queue

deque(['Mary', 'John', 'Susan'])

Retrieve: (Order should be Susan --> John --> Mary)

queue.pop()

'Susan'

queue.pop()

'John'

queue.pop()

'Mary'

Reference

Linked Lists in Python: An Introduction

Hash Table

Thu, 08 Apr 2021 00:00:00 +0000

Hash Table

The Hash table data structure stores elements in key-value pairs where

Key - unique integer that is used for indexing the values
Value - data that are associated with keys.

Hash table is also called hash map, map, dictionary, and associative arrays.

Python has built-in hash tables, they’re called dictionary. E.g. we model a dictionary, where key and value correspond to fruit and price, respectively.

catalog = dict() # we use fruit name as key, and price as value
catalog["apple"] = 1.5
catalog["watermelon"] = 3.2
catalog["banana"] = 3

print(catalog)

{'apple': 1.5, 'watermelon': 3.5, 'milk': 3}

Hash Functions (Hashing)

In a hash table, a new index is processed using the key. And the value corresponding to that key is stored in this index. This process is called hashing.

Let $k$ be a key and $h(x)$ be a hash function. $h(k)$ will give us a new index in the table to store the value linked with $k$.

Requirements of a hash function:

Consistent. Every time we input the same key, we should get the same index.
Different keys should be mapped to different indexes.

Example

In the example above, let’s say for apple:

hash_function("apple") = 4

Therefore, the price of apple, 1.5, will be stored in table[4].

When we want to retrieve the price of apple (catalog.get("apple")), the hash function will tell us that its price is stored in the index 4 in table. So table[4] will be returned.

Hash Colliison

When the hash function generates the same index for multiple keys, there will be a conflict (what value to be stored in that index). This is called a hash collision.

We can resolve the hash collision using one of the following techniques.

Collision resolution by chaining
Open Addressing

Collision resolution by chaining

In chaining, if a hash function produces the same index for multiple elements, these elements are stored in the same index by using a doubly-linked list.

If j is the slot for multiple elements, it contains a pointer to the head of the list of elements.

Open Addressing

Linear Probing

In linear probing, collision is resolved by checking the next slot.

$$ h(k, i) = (h^{\prime}(k) + i) \mod m $$

where

$i = \{0 ,1, ...\}$
$h^{\prime}(k)$ is a new hash function.

Quadratic Probing

It works similar to linear probing but the spacing between the slots is increased (greater than one) by using the following relation.

$$ h(k, i) = (h^{\prime}(k) + c\_1 i + c\_2 i^2) \mod m $$

where

$i = \{0 ,1, ...\}$
$c\_1$ and $c\_2$ are positive auxiliary constants

Double Hashing

If a collision occurs after applying a hash function h(k), then another hash function is applied for finding the next slot.

$$ h(k, i) = (h\_1(k) + ih\_2(k)) \mod m $$

Avoid Hash Collision

To avoid hash collisions, we need

a low load factor
a good hash function

Load Factor

Load factor of a hash table is defined as

$$ \text{load factor} = \frac{\text{number of items in hash table}}{\text{total number of slots}} $$

For example

Having a load factor greater than 1 means you have more items than slots in your array. Once the load factor starts to grow, you need to add more slots to your hash table. This is called resizing. With a lower load factor, you’ll have fewer collisions, and your table will perform better.

Rule of thumb

Resize when load factor is greater than 0.7
Make an array that is twice the size.

Good Hash Function

A good hash function distributes values in the array evenly. It can reduce the number of collisions.

We have different methods to find a good hash function (Let $k$ be a key and m be the size of the hash table):

Division method
$$ h(k) = k \mod m $$
Multiplication Method
$$ h(k) = \lfloor m(kA \mod 1)\rfloor $$
- $\lfloor \rfloor$ gives the floor value
- $A \in (0, 1)$. $kA \mod 1$ gives the fractional part of $kA$. A suggested optimal choice for $A$ is $\frac{\sqrt{5} - 1}{2}$

Performance

Best case

The hash function maps keys evenly all over the hash table. In this case, hash table takes $O(1)$ for everything.

Worst case

The hash function maps all keys to the SAME index. In this case, hash table takes $O(n)$ for everything.

Comparison with Arrays and Linked Lists

	Hash table (average)	Hash table (worst)	Array	Linked list
Search	$O(1)$	$O(n)$	$O(1)$	$O(n)$
Insert	$O(1)$	$O(n)$	$O(n)$	$O(1)$
Delete	$O(1)$	$O(n)$	$O(n)$	$O(1)$

Reference

Dictionaries, Maps, and Hashtables

Mon, 19 Dec 2022 00:00:00 +0000

TL;DR

Dictionaries are the central data structure in Python
THe built-in dict type is “good enough” most of the time
Specialized implementations, like read-only or ordered dicts, are also available in the Python standard library.

In Python, dictionaries (or “dicts” for short)

store an arbitrary number of objects, each identified by a unique dictionary key.
also called maps, hashmaps, lookup tables, or associative arrays.
allow for the efficient lookup, insertion, and deletion of any object associated with a given key.

`dict`

Python features a robust dictionary implementation that’s built directly into the core language: the dict data type.

“Suyntactic sugar” for working with dicts:

curly-braces dict expression syntax

phonebook = {
 "bob": 7387,
 "alice": 3719,
 "jack": 7052,
}

>>> phonebook['alice']
3719

dictionary comprehensions

squares = {x: x * x for x in range(6)}

>>> squares
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25}

Python’s dictionaries are indexed by keys that can be of any hashable type.

A hashable object has a hash value which never changes during its lifetime (see __hash__), and it can be compared to other objects (see __eq__).
Hashable objects which compare as equal must have the same hash value.

For most use cases, Python’s built-in dictionary implementation will do everything you need.

Are highly optimized and underlie many parts of the language (e.g. class attributes and variables in a stack frame are both stored internally in dictionaries)
Are based on a well-tested and finely tuned hash table implementation that provides $O(1)$ time complexity for lookup, insert, update, and delete operations in the average case.

`collections.OrderedDict`

Remembers the insertion order of keys added to it
Must be imported from the collections module

Example

>>> from collections import OrderedDict
>>> d = OrderedDict(one=1, two=2, three=3)
>>> d
OrderedDict([('one', 1), ('two', 2), ('three', 3)])

>>> d["four"] = 4
>>> d
OrderedDict([('one', 1), ('two', 2), ('three', 3), ('four', 4)])

>>> d.keys()
odict_keys(['one', 'two', 'three'])

`collections.defaultdict`

Accepts a callable in its constructor whose return value will be used if a requested key cannot be found

Example: Use list as the default_factory

>>> from collections import defaultdict
>>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
>>> d = defaultdict(list)
>>> for k, v in s:
... d[k].append(v)
...
>>> sorted(d.items())
[('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]

When each key is encountered for the first time, it is not already in the mapping; so an entry is automatically created using the default_factory function which returns an empty list. The list.append() operation then attaches the value to the new list.

When keys are encountered again, the look-up proceeds normally (returning the list for that key) and the list.append() operation adds another value to the list.

Example: Setting the default_factory to int makes the defaultdict useful for counting

from collections import defaultdict

s = "mississippi"
d = defaultdict(int)

for k in s:
 d[k] += 1

>>> d.items()
dict_items([('m', 1), ('i', 4), ('s', 4), ('p', 2)])

When a letter is first encountered, it is missing from the mapping, so the default_factory function calls int() to supply a default count of zero.

The increment operation then builds up the count for each letter.

`collections.ChainMap`

Quickly links a number of mappings so they can be treated as a single unit. It is often much faster than creating a new dictionary and running multiple update() calls.
Is an updatable view over multiple dicts, and it behaves just like a normal dict (All of the usual dictionary methods are supported).
Can be used to simulate nested scopes and is useful in templating.
Use cases
- Searching through multiple dictionaries
  
  Example:
```
>>> toys = {'Blocks': 30, 'Monopoly': 20}
>>> computers = {'iMac': 1000, 'Chromebook': 800, 'PC': 400}
>>> clothing = {'Jeans': 40, 'T-Shirt': 10}
```
```
>>> from collections import ChainMap
>>> inventory = ChainMap(toys, computers, clothing)
```
  Test some normal dict operations:
```
>>> inventory['Monopoly']
20

>>> inventory.get('Mario Bros.')
None

>>> inventory.pop('Blocks')
200

>>> inventory['Blocks']
# KeyError: 'Blocks'
```
  If we now add a toy to the toys dictionary, it will also be made available in the inventory. This is the updatable aspect of a ChainMap.
```
>>> toys['Nintendo'] = 200
>>> inventory['Nintendo']
200
```
  A nice feature is that while in our example toys, computers and clothing are all in the same context (the interpreter), they could come from totally different modules or packages. This is because ChainMap stores the underlying dictionaries by reference.
- Providing a chain of default values
- Performance-critical applications that frequently compute subsets of a dictionary

More see:

A practical usage of ChainMap in Python
ChainMap objects

`types.MappingProxyType`

A wrapper around a standard dictionary that provides a read-only view into the wrapped dictionary’s data. It can be used to create immutable proxy versions of dictionaries.
Using MappingProxyType allows you to put these read-only restrictions in place without first having to create a full copy of the dictionary

Example

from types import MappingProxyType

writable = {"one": 1, "two": 2}
read_only = MappingProxyType(writable)

>>> read_only["one"]
1

The proxy is read-only. Trying to modify the proxy will cause an error:

>>> read_only["one"] = 23
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [19], line 1
----> 1 read_only["one"] = 23

TypeError: 'mappingproxy' object does not support item assignment

Updates to the original are reflected in the proxy:

>>> writable["one"] = 42
>>> read_only["one"]
42

Conclusion

If you’re looking for a general recommendation on which mapping type to use in your programs, it is recommended to use the built-in dict data type. It’s a versatile and optimized hash table implementation that’s built directly into the core language.

It is ONLY recommended that you use one of the other data types listed here if you have special requirements that go beyond what’s provided by dict.

Array Data Structure

Wed, 04 Jan 2023 00:00:00 +0000

TL;DR

You need to store arbitrary objects, potentially with mixed data types? $\rightarrow$ Use list (mutable) or tuple (immutable)
You have numeric (integer or floating point) data and tight packing and performance is important? $\rightarrow$ Try array.array. Also consider going beyond the standard library and try out packages like NumPy or Pandas.
You have textual data represented as Unicode characters? $\rightarrow$ Use Python’s built-in str. If you need a “mutable string,” use a list of characters.
You want to store a contiguous block of bytes? $\rightarrow$ Use byte (immutable) or bytearray (mutable).

In most cases, start out with a simple list. Only specialize later on if performance or storage space becomes an issue. (Most of the time, using a general-purpose array data structure like list gives you the fastest development speed and the most programming conve- nience.)

Arrays consist of fixed-size →records that allow each element to be efficiently located based on its index. Arrays store information in adjoining blocks of memory, therefore they’re considered contiguous data structures (as opposed to linked datas structure like linked lists, for example.)

Performance-wise, it’s very fast to look up an element contained in an array given the element’s index. A proper array implementation guarantees a constant O(1) access time for this case.

`list` – Mutable Dynamic Arrays

Python’s lists are implemented as dynamic arrays. $\rightarrow$ A list allows elements to be added or removed, and the list will automatically adjust the backing store that holds these elements by allocating or releasing memory.

Python lists can hold arbitrary elements. Therefore, you can mix and match different kinds of data types and store them all in a single list. However, the downside is that supporting multiple data types at the same time means that data is generally less tightly packed.

Example:

>>> arr = ['one', 'two', 'three']
>>> arr[0]
'one'

# Lists have a nice repr:
>>> arr
['one', 'two', 'three']

# Lists are mutable:
>>> arr[1] = 'hello'
>>> arr
['one', 'hello', 'three']

>>> del arr[1]
>>> arr
['one', 'three']

# Lists can hold arbitrary data types:
>>> arr.append(23)
>>> arr
['one', 'three', 23]

`tuple` - Immutable Containers

Python’s tuple objects are immutable.

Elements can NOT be added or removed dynamicall.
All elements in a tuple must be defined at creation time.

Just like lists, tuples can hold elements of arbitrary data types.

Example:

>>> t = ("one", "two", "three")
>>> t
('one', 'two', 'three')

>>> t[0]
"one"

# Tuples are immutable:
>>> t[1] = "Hello"
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [3], line 1
----> 1 t[1] = "Hello"

TypeError: 'tuple' object does not support item assignment

>>> del t[1]
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [4], line 1
----> 1 del t[1]

TypeError: 'tuple' object doesn't support item deletion

# Tuples can hold arbitrary data types:
# (Adding elements creates a copy of the tuple)
>>> t + (23, )
('one', 'two', 'three', 23)

`array.array` - Basic Typed Arrays

Python’s array module provides space-efficient storage of basic C- style data types like bytes, 32-bit integers, floating point numbers, etc.

Arrays created with the array.array class

mutable
behave similarly to lists
“typed arrays” constrained to a single data type $\rightarrow$ more space-efficient than lists and tuples. This can be useful if you need to store many elements of the same type.

Example:

>>> import array
>>> arr = array.array('f', (1.0, 1.5, 2.0, 2.5))
>>> arr[1]
1.5


# Arrays are mutable:
>>> arr[1] = 23.0
>>> arr
array('f', [1.0, 23.0, 2.0, 2.5])

>>> del arr[1]
>>> arr
array('f', [1.0, 2.0, 2.5])

>>> arr.append(42.0)
>>> arr
array('f', [1.0, 2.0, 2.5, 42.0])


# Arrays are "typed":
>>> arr[1] = 'hello'
TypeError: "must be real number, not str"

`str` - Immutable Arrays of Unicode Characters

A str is an *** immutable*** array of characters.

It’s also a recursive data structure—each character in a string is a str object of length 1 itself.

String objects are space-efficient because they’re tightly packed and they specialize in a single data type.

Example:

>>> arr = 'abcd'
>>> arr[1]
'b'

>>> arr
'abcd'


# Strings are immutable:
>>> arr[1] = 'e'
TypeError:
"'str' object does not support item assignment"

>>> del arr[1]
TypeError:
"'str' object doesn't support item deletion"


# Strings can be unpacked into a list to
# get a mutable representation:
>>> list('abcd')
['a', 'b', 'c', 'd']
>>> ''.join(list('abcd'))
'abcd'


# Strings are recursive data structures:
>>> type('abc')
"<class 'str'>"
>>> type('abc'[0])
"<class 'str'>"

`bytes` – Immutable Arrays of Single Bytes

Bytes objects are immutable sequences of single bytes (integers in the range of 0<=x<=255).

They are similar to str objects and they are also space-efficient. But unlike strings, there’s a dedicated “mutable byte array” data type called bytearray that they can be unpacked into.

Example:

>>> arr = bytes((0, 1, 2, 3))
>>> arr[1]
1


# Bytes literals have their own syntax:
>>> arr
b'x00x01x02x03'
>>> arr = b'x00x01x02x03'


# Only valid "bytes" are allowed:
>>> bytes((0, 300))
ValueError: "bytes must be in range(0, 256)"


# Bytes are immutable:
>>> arr[1] = 23
TypeError:
"'bytes' object does not support item assignment"
>>> del arr[1]
TypeError:
"'bytes' object doesn't support item deletion"

`bytearray` – Mutable Arrays of Single Bytes

The bytearray type is a mutable sequence of integers in the range 0 <= x <= 255.
They’re closely related to bytes objects. But they can be modified freely — you can overwrite elements, remove existing elements, or add new ones. The bytearray object will grow and shrink accordingly.
Bytearrays can be converted back into immutable bytes objects. But this involves copying the stored data in full—a slow operation taking $O(n)$ time.

Example:

>>> arr = bytearray((0, 1, 2, 3))
>>> arr[1]
1


# The bytearray repr:
>>> arr
bytearray(b'x00x01x02x03')


# Bytearrays are mutable:
>>> arr[1] = 23
>>> arr
bytearray(b'x00x17x02x03')
>>> arr[1]
23


# Bytearrays can grow and shrink in size:
>>> del arr[1]
>>> arr
bytearray(b'x00x02x03')
>>> arr.append(42)
>>> arr
bytearray(b'x00x02x03*')


# Bytearrays can only hold "bytes"
# (integers in the range 0 <= x <= 255) 
>>> arr[1] = 'hello'
TypeError: "an integer is required"
>>> arr[1] = 300
ValueError: "byte must be in range(0, 256)"


# Bytearrays can be converted back into bytes objects: # (This will copy the data)
>>> bytes(arr)
b'x00x02x03*'

Records, Structs, and Data Transfer Objects

Thu, 05 Jan 2023 00:00:00 +0000

TL;DR

You only have a few (2-3) fields $\rightarrow$ Using a plain tuple object may be okay if the field order is easy to remember or field names are superfluous.
You need immutable fields $\rightarrow$ plain tuples, collections.namedtuple, and typing.NamedTuple would all make good options
You need to lock down field names to avoid typos $\rightarrow$ Use collections.namedtuple and typing.NamedTuple
You want to keep things simple $\rightarrow$ Try plain dictionary object
You need full control over your data structure $\rightarrow$ Write a custom class with @property setters and getters.
You need to add behavior (methods) to the object $\rightarrow$ You should write a custom class, either from scratch or by extending collections.namedtuple or typing.NamedTuple.
You need to pack data tightly to serialize it to disk or to send it over the network $\rightarrow$ Use struct.Struct

Record data structures provide a fixed number of fields, where each field can have a name and may also have a different type.

`dict` – Simple Data Objects

Python dictionaries store an arbitrary number of objects, each identified by a unique key. Dictionaries are also often called maps or associative arrays and allow for the efficient lookup, insertion, and deletion of any object associated with a given key.

Dictionaries are easy to create in Python, as they have their own syntactic sugar built into the language in the form of dictionary literals.

Data objects created using dictionaries are mutable, and there’s little protection against misspelled field names, as fields can be added and removed freely at any time. $\rightarrow$ These properties introduce suprising bugs.

Example:

car1 = {
 'color': 'red',
 'mileage': 3812.4,
 'automatic': True,
}
car2 = {
 'color': 'blue',
 'mileage': 40231,
 'automatic': False,
}

# Dicts have a nice repr:
>>> car2
{'color': 'blue', 'automatic': False, 'mileage': 40231}

# Get mileage:
>>> car2['mileage']
40231

# Dicts are mutable:
>>> car2['mileage'] = 12
>>> car2['windshield'] = 'broken'
>>> car2
{'windshield': 'broken', 'color': 'blue',
 'automatic': False, 'mileage': 12}

# No protection against wrong field names,
# or missing/extra fields:
car3 = {
 'colr': 'green',
 'automatic': False,
 'windshield': 'broken',
}

`tuple` – Immutable Groups of Objects

Python’s tuples are simple data structures for grouping arbitrary objects. They are immutable—they can NOT be modified once they’ve been created.

Performance-wise, tuples take up slightly less memory than lists in CPython,17 and they’re also faster to construct. However, In practice, the performance difference will often be negligible. Trying to squeeze extra performance out of a program by switching from lists to tuples will likely be the wrong approach.

Downside of plain tuple:

The data stored in tuples can only be pulled out by accessing it through integer indexes. ou can’t give names to individual properties stored in a tuple. $\rightarrow$ This can impact code readability.
A tuple is always an ad-hoc structure: It’s difficult to ensure that two tuples have the same number of fields and the same properties stored on them. $\rightarrow$ This makes it easy to introduce “slip-of-the-mind” bugs, such as mixing up the field order.

Example:

# Fields: color, mileage, automatic
>>> car1 = ('red', 3812.4, True)
>>> car2 = ('blue', 40231.0, False)

# Tuple instances have a nice repr:
>>> car1
('red', 3812.4, True)
>>> car2
('blue', 40231.0, False)

# Get mileage:
>>> car2[1]
40231.0

# Tuples are immutable:
>>> car2[1] = 12
TypeError:
"'tuple' object does not support item assignment"

# No protection against missing/extra fields # or a wrong order:
>>> car3 = (3431.5, 'green', True, 'silver')

Writing a Custom Class – More Work, But More Control

Classes allow you to define reusable “blueprints” for data objects to ensure each object provides the same set of fields.

However, this approach has some limitations:

It takes manual work to get the convenience features of other implementations. E.g., adding new fields to the __init__ constructor is verbose and takes time.
The default string representation for objects instantiated from custom classes is not very helpful. To fix that you may have to add your own __repr__ method, which again is usually quite verbose and must be updated every time you add a new field.
Although it’s possible to provide more access control and to create read-only fields using the @property decorator, this requires writing more glue code.

When should we write a custom class?

Writing a custom class is a great option whenever you’d like to add business logic and behavior to your record objects using methods.

Example: we implement the Car above with custom class:

class Car:
def __init__(self, color, mileage, automatic):
 self.color = color
 self.mileage = mileage
 self.automatic = automatic

>>> car1 = Car('red', 3812.4, True)
>>> car2 = Car('blue', 40231.0, False)

# Get the mileage:
>>> car2.mileage
40231.0

# Classes are mutable:
>>> car2.mileage = 12
>>> car2.windshield = 'broken'

# String representation is not very useful
# (must add a manually written __repr__ method): >>> car1
<Car object at 0x1081e69e8>

`collections.namedtuple` – Convenient Data Objects

Similar to defining a custom class, using namedtuple allows you to define reusable “blueprints” for your records that ensure the correct field names are used.

namedtuple is an extension of the built-in tuple

Namedtuples are immutable - you can NOT add new fields or modify existing fields after the namedtuple instance was created.
Each object stored in them can be accessed through a unique identifier.

Namedtuple objects are implemented as regular Python classes internally. When it comes to memory usage, they are also “better” than regular classes and just as memory efficient as regular tuples. 👍

Moreover, Namedtuples can be an easy way to clean up your code and make it more readable by enforcing a better structure for your data. It can also helps you express the intent of your code more clearly.

Example: Implement the Car above with namedtupule

>>> from collections import namedtuple
>>> Car = namedtuple('Car' , 'color mileage automatic')
>>> car1 = Car('red', 3812.4, True)

# Instances have a nice repr:
>>> car1
Car(color='red', mileage=3812.4, automatic=True)

# Accessing fields:
>>> car1.mileage
3812.4

# Fields are immtuable:
>>> car1.mileage = 12
AttributeError: "can't set attribute"
>>> car1.windshield = 'broken' AttributeError:
"'Car' object has no attribute 'windshield'"

`typing.NamedTuple` – Improved Namedtuples

typing.NamedTuple is very similar to collections.namedtuple, the main difference being an updated syntax for defining new record types and added support for type hints.

Example:

>>> from typing import NamedTuple
class Car(NamedTuple):
 color: str
 mileage: float
 automatic: bool
>>> car1 = Car('red', 3812.4, True)

# Instances have a nice repr:
>>> car1
Car(color='red', mileage=3812.4, automatic=True)

# Accessing fields:
>>> car1.mileage
3812.4

# Fields are immutable:
>>> car1.mileage = 12
AttributeError: "can't set attribute"
>>> car1.windshield = 'broken' AttributeError:
"'Car' object has no attribute 'windshield'"

# Type annotations are not enforced without
# a separate type checking tool like mypy:
>>> Car('red', 'NOT_A_FLOAT', 99)
Car(color='red', mileage='NOT_A_FLOAT', automatic=99)

`types.SimpleNamespace` – Fancy Attribute Access

SimpleNamespace is basically a glorified dictionary that allows attribute access and prints nicely. Attributes can be added, modified, and deleted freely.

SimpleNamespace instances expose all of their keys as class attributes.

$\rightarrow$ You can use obj.key “dotted” attribute access instead of the obj['key'] square-brackets indexing syntax that’s used by regular dicts.
All instances also include a meaningful __repr__ by default.

Example:

>>> from types import SimpleNamespace
>>> car1 = SimpleNamespace(color='red',
... mileage=3812.4,
... automatic=True)

# The default repr:
>>> car1
namespace(automatic=True, color='red', mileage=3812.4)

# Instances support attribute access and are mutable:
>>> car1.mileage = 12
>>> car1.windshield = 'broken'
>>> del car1.automatic
>>> car1
namespace(color='red', mileage=12, windshield='broken')

Sets

Sat, 07 Jan 2023 00:00:00 +0000

TL;DR

Use the built-in set type when looking for a mutable set.
frozenset objects are hashable and can be used as dictionary or set keys.
collections.Counter implements multiset or “bag” data structures.

Overview

A set is an unordered collection of objects that does NOT allow duplicate elements.

Typically, sets are used

to quickly test a value for membership in the set
to insert or delete new values from a set
to compute the union or intersection of two sets

In a “proper” set implementation, membership tests are expected to run in fast O(1) time. Union, intersection, difference, and subset operations should take O(n) time on average.

Sets in Python

In Python, the curly-braces set expression syntax and set comprehensions allow you to conveniently define new set instances:

vowels = {'a', 'e', 'i', 'o', 'u'}
squares = {x * x for x in range(10)}

To create an empty set you’ll need to call the set() constructor. Using empty curly-braces {} is ambiguous and will create an empty dictionary instead.

`set` – Your Go-To Set

This is the built-in set implementation in Python. The set type is mutable and allows for the dynamic insertion and deletion of elements.

Example

>>> vowels = {'a', 'e', 'i', 'o', 'u'}
>>> 'e' in vowels
True

>>> letters = set('alice')
>>> letters.intersection(vowels)
{'a', 'e', 'i'}

>>> vowels.add('x')
>>> vowels
{'i', 'a', 'u', 'o', 'x', 'e'}

>>> len(vowels)
6

`frozenset` – Immutable Sets

An immutable version of set that cannot be changed after it has been constructed.
Frozensets are static and only allow query operations on their elements (no inserts or deletions.)
As frozensets are static and hashable, they can be used as dictionary keys or as elements of another set.

Example

>>> vowels = frozenset({'a', 'e', 'i', 'o', 'u'})
>>> vowels.add('p')
AttributeError:
"'frozenset' object has no attribute 'add'"

# Frozensets are hashable and can
# be used as dictionary keys:
>>> d = { frozenset({1, 2, 3}): 'hello' }
>>> d[frozenset({1, 2, 3})]
'hello'

`collections.Counter` – Multisets

The collections.Counter class implements a multiset (or bag) type that allows elements in the set to have more than one occurrence.

$\rightarrow$ This is useful if you need to keep track of not only if an element is part of a set, but also how many times it is included in the set

Example

>>> from collections import Counter
>>> inventory = Counter()

>>> loot = {'sword': 1, 'bread': 3}
>>> inventory.update(loot)
>>> inventory
Counter({'bread': 3, 'sword': 1})

>>> more_loot = {'sword': 1, 'apple': 1}
>>> inventory.update(more_loot)
>>> inventory
Counter({'bread': 3, 'sword': 2, 'apple': 1})

You’ll want to be careful when counting the number of elements in a Counter object.

Calling len() returns the number of unique elements in the multiset
Calling sum() returns the total number of elements

>>> len(inventory)
3 # Number of unique elements

>>> sum(inventory.values())
6 # Total number of elements

Stacks

Sun, 08 Jan 2023 00:00:00 +0000

TL;DR

collections.deque provides a safe and fast general-purpose stack implementation. First choice!
The built-in list type can be used as a stack, but be careful to only append and remove items with append() and pop() in order to avoid slow performance.
Use queue module for parallel processing cases

A stack is a collection of objects that supports fast last-in, first-out (LIFO) semantics for inserts and deletes.

The insert and delete operations are also often called push and pop. Performance-wise, a proper stack implementation is expected to take O(1) time for insert and delete operations.

`list` – Simple, Built-In Stacks

Python’s built-in list type makes a decent stack data structure as it supports push and pop operations in amortized $O(1)$ time. To get the amortized $O(1)$ performance for inserts and deletes, new items must be added to the end of the list with the append() method and removed again from the end using pop().

For optimum performance, stacks based on Python lists should grow towards higher indexes and shrink towards lower ones. Adding and removing from the front is much slower and takes O(n) time, as the existing elements must be shifted around to make room for the new element. This is a performance antipattern that you should avoid as much as possible!

Example

>>> s = []
>>> s.append('eat')
>>> s.append('sleep')
>>> s.append('code')

>>> s
['eat', 'sleep', 'code']

>>> s.pop()
'code'
>>> s.pop()
'sleep'
>>> s.pop()
'eat'
>>> s.pop()
IndexError:
"pop from empty list"

`collections.deque` – Fast & Robust Stacks

The deque class implements a double-ended queue that supports adding and removing elements from either end in $O(1)$ time (non-amortized). $\rightarrow$ deque can serve both as queues and as stacks.

Python’s deque objects have excellent and consistent performance for inserting and deleting elements, but poor $O(n)$ performance for randomly accessing elements in the middle of a stack.

Overall, collections.deque is a great choice if you’re looking for a stack data structure in Python’s standard library 👏.

Example

>>> from collections import deque
>>> s = deque()
>>> s.append('eat')
>>> s.append('sleep')
>>> s.append('code')

>>> s
deque(['eat', 'sleep', 'code'])

>>> s.pop()
'code'
>>> s.pop()
'sleep'
>>> s.pop()
'eat'
>>> s.pop()
IndexError:
"pop from an empty deque"

`queue.LifoQueue` – Locking Semantics for Parallel Computing

This stack implementation in the Python standard library is synchronized and provides locking semantics to support multiple concurrent producers and consumers.

Besides LifoQueue, the queue module contains several other classes that implement multi-producer/multi-consumer queues that are use- ful for parallel computing.

Depending on your use case, the locking semantics might be helpful, or they might just incur unneeded overhead. In this case you’d be better off with using a list or a deque as a general-purpose stack.

Example

>>> from queue import LifoQueue
>>> s = LifoQueue()
>>> s.put('eat')
>>> s.put('sleep')
>>> s.put('code')

>>> s
<queue.LifoQueue object at 0x108298dd8>
>>> s.get()
'code'
>>> s.get()
'sleep'
>>> s.get()
'eat'

>>> s.get_nowait()
queue.Empty
>>> s.get()
# Blocks / waits forever...

Stack Implementations Comparison

If you’re not looking for parallel processing support (or don’t want to handle locking and unlocking manually), your choice comes down to the built-in list type or collections.deque.

Difference between list and collections.deque:

list is backed by a dynamic array which makes it great for fast random access, but requires occasional resizing when elements are added or removed.
- You get an amortized $O(1)$ time complexity for push and pop operations.
- But you do need to be careful to only insert and remove items “from the right side” using append() and pop(). Otherwise, perfor- mance slows down to $O(n)$.
collections.deque is backed by a doubly-linked list which optimizes appends and deletes at both ends and provides consistent $O(1)$ performance for these operations.
- More stable performance and also easier to use (you don’t have to worry about adding or removing items from “the wrong end.”)

In summary: collections.deque is an excellent choice for implementing a stack (LIFO queue) in Python.

Queues

Sun, 08 Jan 2023 00:00:00 +0000

TL;DR

collections.deque is an excellent default choice for implementing a FIFO queue data structure in Python
list objects can be used as queues, but this is generally NOT recommended due to slow performance.

A queue is a collection of objects that supports fast first-in, first-out (FIFO) semantics for inserts and deletes.

The insert and delete operations are sometimes called enqueue and dequeue. Unlike lists or arrays, queues typically do NOT allow for random access to the objects they contain.

Performance-wise, a proper queue implementation is expected to take $O(1)$ time for insert and delete operations.

`list` — Terribly Sloooow Queues

It’s possible to use a regular list as a queue but this is NOT ideal from a performance perspective. Because inserting or deleting an element at the beginning requires shifting all of the other elements by one, requiring O(n) time.

Therefore, it is NOT recommend using a list as a makeshift queue in Python (unless you’re only dealing with a small number of elements).

`collections.deque` – Fast & Robust Queues

A double-ended queue that supports adding and removing elements from either end in $O(1)$ time (non-amortized).
Implemented as doubly-linked lists
- Excellent and consistent performance for inserting and deleting elements,
- but poor $O(n)$ performance for randomly accessing elements in the middle of the stack.
Is a great default choice if you’re look- ing for a queue data structure in Python’s standard library.

`queue.Queue` – Locking Semantics for Parallel Computing

Synchronized and provides locking semantics to support multiple concurrent producers and consumers.

Example

>>> from queue import Queue
>>> q = Queue()
>>> q.put('eat')
>>> q.put('sleep')
>>> q.put('code')

>>> q
<queue.Queue object at 0x1070f5b38>

>>> q.get()
'eat'
>>> q.get()
'sleep'
>>> q.get()
'code'
>>> q.get_nowait()
queue.Empty
>>> q.get()
# Blocks / waits forever...

`multiprocessing.Queue` – Shared Job Queues

A shared job queue implementation that allows queued items to be processed in parallel by multiple concurrent workers.
Makes it easy to distribute work across multiple processes in order to work around the GIL limitations.
Can store and transfer any pickle-able object across process boundaries.

Example

>>> from multiprocessing import Queue
>>> q = Queue()
>>> q.put('eat')
>>> q.put('sleep')
>>> q.put('code')

>>> q
<multiprocessing.queues.Queue object at 0x1081c12b0>

>>> q.get()
'eat'
>>> q.get()
'sleep'
>>> q.get()
'code'
>>> q.get()
# Blocks / waits forever...

Priority Queues

Sun, 08 Jan 2023 00:00:00 +0000

TL;DR

queue.PriorityQueue stands out from the pack with a nice object-oriented interface and a name that clearly states its in- tent. It should be your preferred choice.
If you’d like to avoid the locking overhead of queue.PriorityQueue, using the heapq module directly is also a good option.

A priority queue is a container data structure that manages a set of records with totally-ordered keys (e.g., a numeric weight value) to provide quick access to the record with the smallest or largest key in the set.

We can think of a priority queue as a modified queue: instead of retrieving the next element by insertion time, it retrieves the highest-priority element. The priority of individual elements is decided by the ordering applied to their keys.

Priority queues are commonly used for dealing with scheduling problems.

`list` – Maintaining a Manually Sorted Queue

You can use a sorted list to quickly identify and delete the smallest or largest element.

Downsides

Inserting new elements into a list is a slow $O(n)$ operation.
You must manually take care of re-sorting the list when new elements are inserted. It’s easy to introduce bugs by missing this step.

`heapq` – List-Based Binary Heaps

A binary heap implementation usually backed by a plain list, and it supports insertion and extraction of the smallest element in $O(log n)$ time.
A good choice for implementing priority queues in Python.
Extra steps must be taken to ensure sort stability and other features typically expected from a “practical” priority queue.

Example

import heapq
q = []
heapq.heappush(q, (2, 'code'))
heapq.heappush(q, (1, 'eat'))
heapq.heappush(q, (3, 'sleep'))

while q:
 next_item = heapq.heappop(q)
 print(next_item)

# Result:
# (1, 'eat')
# (2, 'code')
# (3, 'sleep')

`queue.PriorityQueue` – Beautiful Priority Queues

Uses heapq internally and shares the same time and space complexities
Is synchronized and provides locking semantics to support multiple concurrent producers and consumers.
You might prefer the class-based interface provided by PriorityQueue over using the function-based interface provided by heapq.

Example

from queue import PriorityQueue
q = PriorityQueue()
q.put((2, 'code'))
q.put((1, 'eat'))
q.put((3, 'sleep'))

while not q.empty():
 next_item = q.get()
 print(next_item)

# Result:
# (1, 'eat')
# (2, 'code')
# (3, 'sleep')

Dictionary Tricks

Sat, 04 Feb 2023 00:00:00 +0000

Sorting Dictionary

Python dictionaries don’t have an inherent order. You can iterate over them just fine but there’s no guarantee that iteration returns the dictionary’s elements in any particular order。

However, it is frequently useful to get a sorted representation of a dictionary to put the dictionary’s items into an arbitrary order based on their key, value, or some other derived property.

To sort a dictionary, we can apply the built-in sorted() function on the items. E.g.:

xs = {'a': 4, 'c': 2, 'b': 3, 'd': 1}

>>> sorted(xs.items())
[('a', 4), ('b', 3), ('c', 2), ('d', 1)]

The key/value tuples are ordered using Python’s standard lexicographical ordering for comparing sequences.

To get complete control over how items are ordered, we can specify the key argument of the sorted() function.

E,g,, sort by value:

>>> sorted(xs.items(), key=lambda x: x[1])
[('d', 1), ('c', 2), ('b', 3), ('a', 4)]

Another apporach i to use the Python built-in operator module. This module implements some of the most frequently used key funcs as plug-and-play building blocks, such as operator.itemgetter and operator.attrgetter.

Example: replace the lambda-based index lookup with operator.itemgetter:

>>> import operator
>>> sorted(xs.items(), key=operator.itemgetter(1))
[('d', 1), ('c', 2), ('b', 3), ('a', 4)]

Using the operator module might communicate your code’s intent more clearly in some cases. On the other hand, using a simple lambda expression might be just as readable and more explicit. Another benefit of using lambdas as a custom key func is that you get to control the sort order in much finer detail.

To reverse the sort order, we can use the reverse=True keyword argument.

Example:

>>> sorted(xs.items(),
 key=lambda x: x[1],
 reverse=True)
[('a', 4), ('b', 3), ('c', 2), ('d', 1)]

Emulating Switch/Case Statements With Dicts

Python doesn’t have switch/case statements so it’s sometimes neces- sary to write long if...elif...else chains as a workaround. Long if chains tend to be a code smell that makes programs more difficult to read and maintain.

One way to deal with long if...elif...else statements is to replace them with dictionary lookup tables that emulate the behavior of switch/case statements. The idea here is to leverage the fact that Python has first-class functions.

We define a dictionary that maps lookup keys for the input conditions to functions that will carry out the intended operations. And we also make use of the get() method of dict, which return a default value if the key can’t be found, to handle the else case.

# long if...elif...else
if cond == "cond_a":
 handle_a()
elif cond == "cond_b":
 handle_b()
else:
 handle_default()

# use dict lookup
func_dict = {
 "cond_a": handle_a,
 "cond_b": handle_b,
}

handle = func_dict.get(cond, handle_default)

Example:

def dispatch_if(operator, x, y):
 if operator == 'add':
 return x + y
 elif operator == 'sub':
 return x - y
 elif operator == 'mul':
 return x * y
 elif operator == 'div':
 return x / y

Use dictionary lookup:

>>> def dispatch_dict(operator, x, y):
 return {
 'add': lambda: x + y,
 'sub': lambda: x - y,
 'mul': lambda: x * y,
 'div': lambda: x / y,
 }.get(operator, lambda: None)()

this technique won’t apply in every situation and sometimes you’ll be better off with a plain if-statement.

Merging Dictionaries

Merge two or more dictionaries into one, so that the resulting dictionary contains a combination of the keys and values of the source dicts.

Use built-in `update()`

>>> xs = {'a': 1, 'b': 2}
>>> ys = {'b': 3, 'c': 4}

>>> zs = {}
>>> zs.update(xs)
>>> zs.update(ys)
>>> zs
{'a': 1, 'b': 3, 'c': 4}

a naive implementation of update() is to simply iterate over all of the items of the right-hand side dictionary and add each key/value pair to the left-hand side dictionary, overwriting existing keys as we go along:

def update(dict1, dict2):
 for key, value in dict2.items():
 dict1[key] = value

Use `dict()` built-in combined with the `**`-operator

>>> zs = dict(xs, **ys)
>>> zs
{'a': 1, 'c': 4, 'b': 3}

Use the `**`-operator

zs = {**xs, **ys}

This expression has the exact same result as a chain of update() calls. This is an arguably prettier way to merge an arbitrary number of dictionaries. Using the **-operator is also faster than using chained update() calls

Dictionary Pretty-Printing

Example:

>>> mapping = {'a': 23, 'b': 42, 'c': 0xc0ffee}
>>> str(mapping)
{'b': 42, 'c': 12648430, 'a': 23}

Use `json.dumps()`

>>> import json
>>> json.dumps(mapping, indent=4, sort_keys=True)
{
 "a": 23,
 "b": 42,
 "c": 12648430
}

While this looks nice and readable, it isn’t a perfect solution. There are some limitations:

Printing dictionaries with the json module only works with dicts that contain primitive types—you’ll run into trouble trying to print a dictionary that contains a non-primitive data type, like a function.
Using json.dumps() is that it can’t stringify complex data types, like sets.
In some cases you won’t be able to take the output from json.dumps and copy and paste it into a Python interpreter session to reconstruct the original dictionary object.

Use `pprint`

>>> import pprint
>>> pprint.pprint(mapping)
{'a': 23, 'b': 42, 'c': 12648430, 'd': set([1, 2, 3])}

However, compared to json.dumps(), it doesn’t represent nested structures as well visually.

Namedtuple

Sat, 02 May 2020 00:00:00 +0000

TL;DR

collection.namedtuple is a memory-efficient shortcut to defining an immutable class in Python manually.
Namedtuples can help clean up your code by enforcing an easier to understand structure on your data, which also improves readability.
Namedtuples provide a few useful helper methods that all start with an _ underscore—but are part of the public interface.

What is `namedtuple`?

Python’s tuple is a simple immutable data structure for grouping arbitrary objects. It has two shortcomings:

The data you store in it can only be pulled out by accessing it through integer indexes. You can’t give names to individual properties stored in a tuple. This can impact code readability.
It’s hard to ensure that two tuples have the same number of fields and the same properties stored on them.

namedtuple aims to solve these two problems. namedtuple is a factory function for making a tuple class. With that class we can create tuples that are callable by name also.

namedtuple is immutable just like regular tuple
- Once you store data in top-level attribute on a namedtuple, you can’t modify it by updating the attribute.
- All attributes on a namedtuple object follow the “write once, read many” principle.
Each object stored in them can be accessed through a unique (human-readable) string identifier.

→ This frees you from having to remember integer indexes, or resorting to workarounds like defining integer constants as mnemonics for your indexes. 👏

How `namedtuple` works?

Let’s take a look at an example:

from collections import namedtuple

User = namedtuple("User", ["name", "id", "gender"])
user = User("Ecko", 1, "male")

>>> user
User(name='Ecko', id=1, gender='male')

"User" as the first argument to the namedtuple factory function is referred to as the “typename” in the Python docs. It’s the name of the new class that’s being created by calling the namedtuple function, which needs to be explicitly specified.

The code above is equivalent to

class User:
 def __init__(self, name, id, gender):
 self.name = name
 self.id = id
 self.gender = gender

user = User("Ecko", 1, "male")

Why `namedtuple`?

A good way to view them is to think that namedtuples are a memory-efficient shortcut to defining an immutable class in Python manually.

Operations

Unpacking

Tuple unpacking and the *-operator for function argument unpacking also work as expected:

name, id, gender = user
print(f"name: {name}, id: {id}, gender: {gender}")

name: Ecko, id: 1, gender: male

print(*user)

Ecko 1 male

Accessing Values

Values can be accessed either by identifier or by index.

By identifier:

>>> user.name
Ecko

By index:

>>> user[0]
Ecko

Built-in Helper Functions

namedtuple has some useful helper methods.

Their names all start with an underscore character _
- With namedtuples the underscore naming convention has a different meaning though: These helper methods and properties are part of namedtuple’s public interface.
- The helpers were named that way to avoid naming collisions with user-defined tuple fields.

`_asdict`

Returns the contents of a namedtuple as a dictionary

>>> user._asdict()
OrderedDict([('name', 'Ecko'), ('id', 1), ('gender', 'male')])

This is great for avoiding typos in the field names when generating JSON-output, for example:

>>> import json
>>> json.dumps(user._asdict())
'{"name": "Ecko", "id": 1, "gender": "male"}'

We can convert a dictionary into a namedtuple with ** operator

>>> user_dict = {"name": "Ben", "id": 2, "gender": "male"}
>>> User(**user_dict)
User(name='Ben', id=2, gender='male')

`_replace()`

Creates a (shallow) copy of a tuple and allows you to selectively replace some of its fields.

user._replace(id=2)

User(name='Ecko', id=2, gender='male')

`_make()`

Classmethod can be used to create new instances of a namedtuple from a sequence or iterable

User._make(["Ilona", 3, "female"])

User(name='Ilona', id=3, gender='female')

When to Use `namedtuple`?

If you’re going to create a bunch of instances of a class and NOT change the attributes after you set them in __init__, you can consider to use namedtuple.
- Example: Definition of classes in torchvision.datasets.cityscapes
Namedtuples can be an easy way to clean up your code and to make it more readable by enforcing a better structure for your data. You should use namedtuple anywhere you think object notation will make your code more pythonic and more easily readable
But try NOT to use namedtuples for their own sake if they don’t help you write “cleaner” and more maintainable code.

Reference

[Issues] List

Wed, 08 Jul 2020 00:00:00 +0000

Change Value in List Based on Conditions

E.g. 1

>>> a = ["a", "b", "c", "d", "e"]
>>> b = ["a", "c", "e"]

We want a list vector with the same length as a. If the element also occurs in b, value in the corresponding position is 1, else 0.

I.e., target output should be [1, 0, 1, 0, 1]

Solution:

>>> l = [1 if el in b else 0 for el in a]
>>> l

[1, 0, 1, 0, 1]

E.g. 2

>>> a = np.arange(5)
>>> a

array([0, 1, 2, 3, 4])

>>> a[a > 2] = 5
>>> a

array([0, 1, 2, 5, 5])

List Comprehension

Define the list and its contents at the same time by following this format:

new_list = [expression for member in iterable]

expression: the member itself, a call to a method, or any other valid expression that returns a value.
member: the object or value in the list or iterable
iterable: a list, set, sequence, generator, or any other object that can return its elements one at a time

Example

Get the even numbers in 0 to 10 (10 is not inlcuded):

>>> even_nums = [el for el in range(10) if el % 2 == 0]
>>> even_nums

[0, 2, 4, 6, 8]

Advantages

More pythonic
More declarative than loops: list comprehensions are easier to read and understand

Combined with conditional logic

Simple filtering: Place the conditional to the end
```
new_list = [expression for member in iterable (if conditional)]
```
Here, the conditionals allow list comprehensions to filter out unwanted values.

Change a member value: Place the conditional near the beginning

new_list = [expression (if conditional) for member in iterable]

Example: Change all odd numbers in 0 - 10 to 0

>>> arr = [el if el % 2 == 0 else 0 for el in range(10)]
>>> arr
[0, 0, 2, 0, 4, 0, 6, 0, 8, 0]

Nested list comprehension

Nested List Comprehensions are nothing but a list comprehension within another list comprehension which is quite similar to nested for loops.

Example

Let’s say we want to obtain all possible combination of two lists: a = [1, 2, 3] and b = [4, 5].

Use list comprehension:

>>> combinations = [[i, j] for j in b for i in a]
>>> combinations
[[1, 4], [2, 4], [3, 4], [1, 5], [2, 5], [3, 5]]

This is equvalent to nested loops:

combinations = []

for i in a:
 for j in b:
 combinations.append([i, j])

Apparently, using list comprehension is more succinct and more pythonic.

Reference

When to Use a List Comprehension in Python

Print Lists

Using `for`-loop

a = [1, 2, 3, 4, 5]
for x in a:
 print(x)

Output:

Using `*` symbol

>>> a = [1, 2, 3, 4, 5]
>>> print(*a)
1 2 3 4 5

Using this way, we can also easily control the separator and ending.

>>> print(*a, sep=", ") # print list separted by comma
1, 2, 3, 4, 5

print(*a, sep="\n") # print list in new line

Output:

Convert list to string for display

If it is a list of strings we can simply join them using join() function.

>>> a = ["Hello", "World"]
>>> print(" ".join(a))
Hello World

Reference

Print lists in Python (4 Different Ways)

List Comprehension Vs. `*`-operator

List comprehension and using the *-operator are two common methods to create list containing repeating elements. However, there is a minor difference between them, which is often overlooked and may cause unexpected bug.

The *-operator can NOT make independent objects
- Reason: the multiplication operator * operates on objects, without seeing expressions.
- E.g.: [x] * 3 creates a list [x, x, x], which is essentially a list with 3 references to the same x. I.e., when you modify one single x, all references will be modified.
List comprehension can make independent objects
- It re-evaluates the element expression on every iteration. Every evaluation generates a new independent object. I.e., Modifying one of them will not affect the others.
- E.g.: [x for _ in range(3)] creates a like [x, copy.copy(x), copy.copy(x)]

Using the *-operator may be inconsistent. In contrast, list comprehension would be a safer option.

Example 1: List containing objects

class Student:

 def __init__(self, age):
 self.age = age

Create list with *:

>>> students = [Student(12)] * 3
>>> _ = [print(student.age) for student in students]
12
12
12

As is essentially a list with 3 references to the same Student, modifying one of them will affect others.

>>> students[0].age = 15
>>> _ = [print(student.age) for student in students]
15
15
15

In contrast, using list comprehension will create a list containing three independent Students.

>>> students = [Student(12) for _ in range(3)]
>>> _ = [print(student.age) for student in students]
12
12
12

Modifying one of them will NOT affect others:

>>> students[0].age = 15
>>> _ = [print(student.age) for student in students]
15
12
12

Example 2: Multidimensional Array/List

Using *-operator:

>>> matrix = [[0, 0]] * 2
>>> matrix
[[0, 0], [0, 0]] # Two reference to the same [0, 0]
>>> matrix[0][1] = 1
>>> matrix
[[0, 1], [0, 1]]

Using list comprehension:

>>> matrix = [[0] * 2 for _ in range(2)]
>>> matrix
[[0, 0], [0, 0]] # Two independent [0, 0]
>>> matrix[0][1] = 1
>>> matrix
[[0, 1], [0, 0]]

Reference

List of lists changes reflected across sublists unexpectedly

[Issues] Dictionary

Wed, 08 Jul 2020 00:00:00 +0000

Get Dictionary Items with Specified Initialization

dict.get(key, default = None))

returns the value of the item with the specified key.

Parameters

key − This is the Key to be searched in the dictionary.
default − This is the Value to be returned in case key does not exist.

See: Python dictionary get() Method

We can use dict.get(key, init_value) for initializing key-value pair in dictionary

E.g.: Counting how many times each element occurs in a list

 a = ["a", "b", "a", "a", "c", "c","a"]
 d = {}
 for el in a:
 d[el] = d.get(el, 0) + 1

 {'a': 4, 'b': 1, 'c': 2}

Remove Key from Python Dictionary

If you want to remove keys from Python dictionary, there’re different ways to do it:

Use Python built-in del
Use dict.pop()
Use dictionary comprehension to create a new dictionary without the keys that need to be removed

Let’s say we have a dict like this:

person_info = {
 "name": "Ecko",
 "age": 20,
 "gender": "male"
}

and we want to remove the key age.

Built-in `del`

del person_info["age"]
print(person_info)

{'name': 'Ecko', 'gender': 'male'}

However, if the key doesn’t exist, del will raise an KeyError error. For example:

del person_info["address"]

---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-14-8391b7d3a4c8> in <module>()
----> 1 del person_info["address"]

KeyError: 'address'

Therefore, we should check existence before removing:

key = "age"
if key in person_dict:
 del key

`dict.pop()`

To delete a key regardless of whether it is in the dictionary, use dict.pop()

pop(key[, default])

If key is in the dictionary, remove it and return its value, else return default. If default is not given and key is not in the dictionary, a KeyError is raised.

The advantage of this way is that you can get and delete an entry from a dict in one line of code. 👏

For example, we remove the existed key age:

age = person_info.pop("age")

print(age)
print(person_info)

We remove the key address which does NOT exist:

addr = person_info.pop("address", "earth")

print(addr)
print(person_info)

earth
{'name': 'Ecko', 'age': 20, 'gender': 'male'}

Use dictionary comprehension

{key:val for key, val in dict.items() if key != remove_key}

For our example:

{key:val for key,val in person_info.items() if key != "age"}

print(person_info)

{'gender': 'male', 'name': 'Ecko'}

Using dictionary comprehension, the a new dict will be created, and the original dict won’t be affected.

Reference

How to remove a key from a Python dictionary?

Files

Sun, 01 May 2022 00:00:00 +0000

Working with Files

Thu, 19 Nov 2020 00:00:00 +0000

`with open(...) as ...` pattern

open() opens file for reading or writing and returns a file handle
Use appropriate methods to read or write this file handel

Example

Read

with open('data.txt', 'r') as f:
 data = f.read()

Write

with open('data.txt', 'w') as f:
 data = 'some data to be written to the file'
 f.write(data)

Directory listing

Directory Listing in Legacy Python Versions

os.listdir()

returns a Python list containing the names of the files and subdirectories in the directory given by the path argument:

Example

import os

entries = os.listdir('my_directory/')

Directory Listing in Modern Python Versions

`os.scandir()`

returns an iterator

ROOT = 'my_directory/'
entries = os.scandir(ROOT)
entries

<posix.ScandirIterator at 0x7f6db713c3f0>

The ScandirIterator points to all the entries in the current directory. You can loop over the contents of the iterator and print out the filenames:
```
with os.scandir(ROOT) as entries:
 for entry in entries:
 print(f'{entry.name:10}: {entry.path}')
```

`pathlib` module

pathlib.Path() objects have an .iterdir() method for creating an iterator of all files and folders in a directory. Each entry yielded by .iterdir() contains information about the file or directory such as its name and file attributes.
pathlib offers a set of classes featuring most of the common operations on paths in an easy, object-oriented way. Another benefit of using pathlib over os is that it reduces the number of imports you need to make to manipulate filesystem paths. 👏

Example

from pathlib import Path

entries = Path(ROOT)
for entry in entries.iterdir():
 print(f'{entry.name}, name: {entry.stem}, ext: {entry.suffix}')

Summary

Function	Description
`os.listdir()`	Returns a list of all files and folders in a directory
`os.scandir()`	Returns an iterator of all the objects in a directory including file attribute information
`pathlib.Path.iterdir()`	Returns an iterator of all the objects in a directory including file attribute information

Listing all files in a directory

Filter out directories and only list files from a directory listing

Use `os.listdir()`

import os

basepath = 'my_directory'
for entry in os.listdir(basepath):
 if os.path.isfile(os.path.join(basepath, entry)):
 print(entry)

Use `os.scandir()`

Using os.scandir() has the advantage of looking cleaner and being easier to understand than using os.listdir()

import os

basepath = 'my_directory'
with os.scandir(basepath) as entries:
 for entry in entries:
 if entry.is_file():
 print(entry.name)

Use `pathlib.Path()`

from pathlib import Path

basepath = 'my_directory'
files_in_basepath = Path(basepath).iterdir()
for item in files_in_basepath:
 if item.is_file():
 print(item.name)

The code above can be made more concise if we combine the for loop and the if statement into a single generator expression

from pathlib import Path

basepath = 'my_directory'
files_in_basepath = (entry for entry in Path(basepath).iterdir() if entry.is_file())
for item in files_in_basepath:
 print(item.name)

Listing subdirectories

Use `os.listdir()` and

import os

basepath = 'my_directory'
sub_dirs = (entry for entry in os.listdir(basepath)
 if os.path.isdir(os.path.join(basepath, entry)))

for sub_dir in sub_dirs:
 print(sub_dir)

Use `os.scandir()`

import os

basepath = 'my_directory'
with os.scandir(basepath) as entries:
 for entry in entries:
 if entry.is_dir():
 print(entry.name)

Use `pathlib.Path`

from pathlib import Path

basepath = 'my_directory'
sub_dirs = (item for item in Path(basepath).iterdir() if item.is_dir())
for sub_dir in sub_dirs:
 print(sub_dir.name)

Making directories

os and pathlib include functions for creating directories. We’ll consider these:

Function	Description
`os.mkdir()`	Creates a single subdirectory
`pathlib.Path.mkdir()`	Creates single or multiple directories
`os.makedirs()`	Creates multiple directories, including intermediate directories

Making single directories

Use `os.mkdir()`

To create a single directory, pass a path to the directory as a parameter to os.mkdir():

import os

os.mkdir('example_directory/')

Note: If a directory already exists, os.mkdir() raises FileExistsError.

Use `pathlib`

from pathlib import Path

p = Path('example_directory/')
p.mkdir()

If the path already exists, Path.mkdir() raises a FileExistsError.

To avoid the error, catch the error when it happens and let the user know:

from pathlib import Path

p = Path('example_directory/')
try:
 p.mkdir()
except FileExistsError as err:
 print(err)

Alternatively, you can ignore the FileExistsError by passing the exist_ok=True argument to .mkdir(). This will not raise an error if the directory already exists.

from pathlib import Path

p = Path('example_directory/')
p.mkdir(exist_ok=True)

Creating Multiple Directories

Use `os.makedirs`

os.makedirs()

similar to os.mkdir()
The difference between the two is that not only can os.makedirs() create individual directories, it can also be used to create directory trees. In other words, it can create any necessary intermediate folders in order to ensure a full path exists.
similar to running mkdir -p in Bash.

import os

os.makedirs('dir/sub_dir/sub_sub_dir')

This will create a nested directories with default permissions:

.
|
└── dir/
 └── sub_dir/
 └── sub_sub_dir/

Use `pathlib.Path`

import pathlib

p = pathlib.Path('dir/sub_dir/sub_sub_dir')
p.mkdir(parents=True, exist_ok=True)

I prefer using pathlib when creating directories because I can use the same function to create single or nested directories.

Filename Pattern Matching

Function	Description
`startswith()`	Tests if a string starts with a specified pattern and returns `True`or `False`
`endswith()`	Tests if a string ends with a specified pattern and returns `True` or `False`
`fnmatch.fnmatch(filename, pattern)`	Tests whether the filename matches the pattern and returns `True` or `False`
`glob.glob()`	Returns a list of filenames that match a pattern
`pathlib.Path.glob()`	Finds patterns in path names and returns a generator object

Traversing Directories and Processing Files

`os.walk()`

os.walk() defaults to traversing directories in a top-down manner.

To traverse the directory tree in a bottom-up manner, pass in a topdown=False keyword argument

os.walk() returns three values on each iteration of the loop:

The name of the current folder
A list of folders in the current folder
A list of files in the current folder

Deleting files

Function	Note
`os.remove()`	Will throw an `OSError` if the path passed to them points to a directory instead of a file
`os.unlink()`	semantically identical to `os.remove()`
`pathlib.Path.unlink()`

Deleting Directories

Delete single directory

Function	Note
`os.rmdir()`	Only work if the directory you’re trying to delete is empty. If the directory isn’t empty, an `OSError` is raised.
`pathlib.rmdir()`	semantically identical to `os.rmdir()`

Delete entire directory trees

shutil.rmtree(dir): Everything in dir is deleted when shutil.rmtree() is called on it.

Summary for deleting

Function	Description
`os.remove()`	Deletes a file and does not delete directories
`os.unlink()`	Is identical to `os.remove()` and deletes a single file
`pathlib.Path.unlink()`	Deletes a file and cannot delete directories
`os.rmdir()`	Deletes an empty directory
`pathlib.Path.rmdir()`	Deletes an empty directory
`shutil.rmtree()`	Deletes entire directory tree and can be used to delete non-empty directories

Copying

Copying files

Function Note

shutil.copy() comparable to the cp command in UNIX based systems.
shutil.copy(src, dst) will copy the file src to the location specified in dst. If dstis a file, the contents of that file are replaced with the contents of src. If dst is a directory, then src will be copied into that director
shutil.copy() ONLY copies the file’s contents and the file’s permissions. Other metadata like the file’s creation and modification times are not preserved.

shutil.copy2() preserve all file metadata when copying

Function	Note
`shutil.copy()`	comparable to the `cp` command in UNIX based systems. `shutil.copy(src, dst)` will copy the file `src` to the location specified in `dst`. If `dst`is a file, the contents of that file are replaced with the contents of `src`. If `dst` is a directory, then `src` will be copied into that director `shutil.copy()` ONLY copies the file’s contents and the file’s permissions. Other metadata like the file’s creation and modification times are not preserved.
`shutil.copy2()`	preserve all file metadata when copying

Copying directories

shutil.copytree()

Moving Files and Directories

To move a file or directory to another location, use shutil.move(src, dst).

src: file or directory to be moved
dst: destination. If dst does not exist, src will be renamed to dst

Renaming Files and Directories

os.rename(src, dst)
pathlib.Path.rename()

Colab Notebook

Reference

Working with Files in Python

File I/O

Thu, 19 Nov 2020 00:00:00 +0000

What is a File?

At its core, a file is a contiguous set of bytes used to store data. This data is organized in a specific format and can be anything as simple as a text file or as complicated as a program executable. In the end, these byte files are then translated into binary 1and 0 for easier processing by the computer.

Files on most modern file systems are composed of three main parts:

Header: metadata about the contents of the file (file name, size, type, and so on)
Data: contents of the file as written by the creator or editor
End of file (EOF): special character that indicates the end of the file

Opening and closing files

Use with statement

with open('dog_breeds.txt', 'r') as reader:
 # Further file processing goes here
 pass

The second positional argument, mode, is a string that contains multiple characters to represent how you want to open the file. The default and most common is 'r', which represents opening the file in read-only mode as a text file

Other options for modes are fully documented online, but the most commonly used ones are the following:

Character	Meaning
`'r'`	Open for reading (default)
`'w'`	Open for writing, truncating (overwriting) the file first
`'rb'` or `'wb'`	Open in binary mode (read/write using byte data)
`'a'`	open for writing, appending to the end of the file if it exists

Reading opened files

Method	What It Does
`.read(size=-1)`	This reads from the file based on the number of `size`bytes. If no argument is passed or `None` or `-1` is passed, then the entire file is read.
`.readline(size=-1)`	This reads at most `size` number of characters from the line. This continues to the end of the line and then wraps back around. If no argument is passed or `None` or `-1` is passed, then the entire line (or rest of the line) is read.
`.readlines()`	This reads the remaining lines from the file object and returns them as a list.

Example:

with open('dog_breeds.txt', 'r') as reader:
 for line in reader.readlines():
 print(line, end='')

The end='' is to prevent Python from adding an additional newline to the text that is being printed and only print what is being read from the file.

The code above can be further simplified by iterating over the file object itself:

with open('dog_breeds.txt', 'r') as reader:
 for line in reader:
 print(line, end='')

This approach is more Pythonic and can be quicker and more memory efficient. Therefore, it is suggested you use this instead.

Writing opened files

Method	What It Does
`.write(string)`	This writes the string to the file.
`.writelines(seq)`	This writes the sequence to the file. No line endings are appended to each sequence item. It’s up to you to add the appropriate line ending(s)

Tips and tricks

Working With Two Files at the Same Time

d_path = 'dog_breeds.txt'
d_r_path = 'dog_breeds_reversed.txt'

with open(d_path, 'r') as reader, open(d_r_path, 'w') as writer:
 dog_breeds = reader.readlines()
 writer.writelines(reversed(dog_breeds))

Get rid of `\n` when reading lines

Reference: How to read a file without newlines?

with open(filename) as f:
 mylist = f.read().splitlines()

Google Colab Notebook

Colab Notebook

Reference

Reading and Writing Files in Python (Guide)

pathlib

Sat, 26 Dec 2020 00:00:00 +0000

Why `patblib`?

The pathlib module, introduced in Python 3.4 (PEP 428), gathers the necessary functionality in one place and makes it available through methods and properties on an easy-to-use Path object.

from pathlib import Path

Createing paths

Use class methods:
- Path.cwd(): get Current Working Directory
- Path.home(): get home directory
Create from string representation explicitly

E.g.
```
Path("/content/src/demo.py")
```

Construct path by joining parts of the path

E.g.

Path.joinpath(Path.cwd(), "src", "demo.py")

Reading and writing files

The built-in open() function can use Path objects directly. E.g.:

with open(path, mode="r") as f:
 pass

For simple reading and writing of files, there are a couple of convenience methods in the pathlib library:

.read_text(): open the path in text mode and return the contents as a string.
.read_bytes(): open the path in binary/bytes mode and return the contents as a bytestring.
.write_text(): open the path and write string data to it.
.write_bytes(): open the path in binary/bytes mode and write data to it.

Each of these methods handles the opening and closing of the file, making them trivial to use.

Components of path

The different parts of a path are conveniently available as properties. Basic examples include:

.name: the file name without any directory
.parent: the directory containing the file, or the parent directory if path is a directory
.stem: the file name without the suffix
.suffix: the file extension
.anchor: the part of the path before the directories

path = Path.joinpath(Path.cwd(), "src", "hello-world.txt")

print(f"Full path: {path}")
print(f"Name: {path.name}")
print(f"Parent: {path.parent}")
print(f"Stem: {path.stem}")
print(f"Suffix: {path.suffix}")
print(f"Anchor: {path.anchor}")

Full path: /content/src/hello-world.txt
Name: hello-world.txt
Parent: /content/src
Stem: hello-world
Suffix: .txt
Anchor: /

Renaming files

When you are renaming files, useful methods might be .with_name() and .with_suffix(). They both return the original path but with the name or the suffix replaced, respectively.

Deleting files

Directories and files can be deleted using .rmdir() and .unlink() respectively.

Listing files

Path.iterdir() : iterates over all files in the given directory
More flexible file listing
- Path.glob()
- Path.rglob() (recursive glob)

Moving files

Path(current_location).rename(new_location)

Correspondence to tools in the `os` module

os and os.path	pathlib
`os.path.abspath()`	`Path.resolve()`
`os.chmod()`	`Path.chmod()`
`os.mkdir()`	`Path.mkdir()`
`os.makedirs()`	`Path.mkdir(parents=True)`
`os.rename()`	`Path.rename()`
`os.replace()`	`Path.replace()`
`os.rmdir()`	`Path.rmdir()`
`os.remove()`, `os.unlink()`	`Path.unlink()`
`os.getcwd()`	`Path.cwd()`
`os.path.exists()`	`Path.exists()`
`os.path.expanduser()`	`Path.expanduser()` and `Path.home()`
`os.listdir()`	`Path.iterdir()`
`os.path.isdir()`	`Path.is_dir()`
`os.path.isfile()`	`Path.is_file()`
`os.path.islink()`	`Path.is_symlink()`
`os.link()`	`Path.link_to()`
`os.symlink()`	`Path.symlink_to()`
`os.readlink()`	`Path.readlink()`
`os.stat()`	`Path.stat()`, `Path.owner()`, `Path.group()`
`os.path.isabs()`	`PurePath.is_absolute()`
`os.path.join()`	`PurePath.joinpath()`
`os.path.basename()`	`PurePath.name`
`os.path.dirname()`	`PurePath.parent`
`os.path.samefile()`	`Path.samefile()`
`os.path.splitext()`	`PurePath.suffix`

`pathlib` cheatsheet

pathlib cheatsheet

Google Colab Notebook

Open in Colab

Reference

Documentation: pathlib — Object-oriented filesystem paths
Python 3’s pathlib Module: Taming the File System

glob

Fri, 25 Dec 2020 00:00:00 +0000

The glob module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell. If you need a list of filenames that all have a certain extension, prefix, or any common string in the middle, use glob instead of writing code to scan the directory contents yourself.

Wildcards

* : matches zero or more characters in a segment of a name
? : matches any single character in that position in the name
[] : matches characters in the given range
- [1-9] : matches any single digit
- [a-z : mathces any single letter

Functions

`glob.glob()`

glob(file_pattern, recursive = False)

It retrieves the list of files matching the specified pattern in the file_pattern parameter.

The file_pattern can be an absolute or relative path. It may also contain wild cards such as * or ? symbols.
The recursive parameter is turn off (False) by default. When True, it recursively searches files under all subdirectories of the current directory.

For example, we have file structure like this:

- src
|- code
 |- main.py
 |- test0.py
 |- test1.py
 |- test2.py
 |- test-env.py
 |- test-prod.py
|- text
 |- file_a.txt
 |- file_b.txt
 |- file_c.txt
 |- file_d.txt
|- demo.py

We want to get all .py files:

import glob

for py_file in glob.glob("src/**/*.py", recursive=True):
 print(py_file)

src/demo.py
src/code/test1.py
src/code/test-env.py
src/code/main.py
src/code/test2.py
src/code/test0.py

We want to get test0.py, test1.py, test2.py:

import glob

for py_file in glob.glob("src/**/test?.py"):
 print(py_file)

src/code/test1.py
src/code/test2.py
src/code/test0.py

We want to get file_a.txt, file_b.txt, file_c.txt:

import glob

for txt_file in glob.glob("src/**/file_[a-c].txt"):
 print(txt_file)

src/text/file_a.txt
src/text/file_c.txt
src/text/file_b.txt

`glob.iglob()`

Return an iterator which yields the same values as glob() without actually storing them all simultaneously (good for large directories).

Example:

import glob

for py_file in glob.iglob("src/**/test?.py"):
 print(py_file)

src/code/test1.py
src/code/test2.py
src/code/test0.py

`glob.escape()`

Escape all special characters ('?', '*' and '['). “Escape” means treating special characters as normal character instead of wildcards.

This is useful if you want to match an arbitrary literal string that may have special characters in it.

For example, if we want to get test-env.py and test-prod.py (both contain special character - in the filename):

import glob

for py_file in glob.glob("src/code/*" + glob.escape("-") + "*.py"):
 print(py_file)

Another ways for filename pattern matching

`fnmatch.fnmatch()` ¹

fnmatch.fnmatch(filename, pattern)

Test whether the filename string matches the pattern string.

For example, we want to get test0.py, test1.py, test2.py:

import os
import fnmatch

for py_file in os.listdir("src/code"):
 if fnmatch.fnmatch(py_file, "test?.py"):
 print(py_file)

`pathlib.Path.glob()` ²

pathlib.Path.glob(pattern)

Glob the given relative pattern in the directory represented by this path, yielding all matching files (of any kind).

For example, if we want to list all python file:

from pathlib import Path

path = Path("src")

for py_file in path.glob("**/*.py"):
 print(py_file)

src/demo.py
src/code/test1.py
src/code/test-prod.py
src/code/test-env.py
src/code/main.py
src/code/test2.py
src/code/test0.py

The “**” pattern means “this directory and all subdirectories, recursively”. In other words, it enables recursive globbing. However, using the “**” pattern in large directory trees may consume an inordinate amount of time.

Path.glob() is similar to os.glob() discussed above. As you can see, pathlib combines many of the best features of the os, os.path, and glob modules into one single module, which makes it a joy to use.

`pathlib.Path.rglob()` ³

This is like calling Path.glob() with “**/” added in front of the given relative pattern.

E.g. list all python file:

for py_file in Path("src").rglob("*.py"):
 print(py_file)

src/demo.py
src/code/test1.py
src/code/test-prod.py
src/code/test-env.py
src/code/main.py
src/code/test2.py
src/code/test0.py

Summary

Function	Description
`fnmatch.fnmatch(filename, pattern)`	Tests whether the filename matches the pattern and returns `True` or `False`
`glob.glob()`	Returns a list of filenames that match a pattern
`glob.iglob()`	Returns an iterator of filenames that match a pattern
`pathlib.Path.glob()`	Finds patterns in path names and returns a generator object
`pathlib.Path.rglob()`	Finds patterns in path names recursively and returns a generator object

Reference

Serialization

Sun, 01 May 2022 00:00:00 +0000

JSON

Wed, 09 Dec 2020 00:00:00 +0000

TL;DR

Python and JSON conversion

JSON (JavaScript Object Notation) is a popular data format used for representing structured data. It’s common to transmit and receive data between a server and web application in JSON format.

JSON in Python 3

Python has a built-in package called json, which can be used to work with JSON data.

import json

JSON to Python

You can parse a JSON string using json.loads() method. The method returns a dictionary.

For example:

import json

person = '{"name": "Bob", "languages": ["English", "Fench"]}' # JSON string
person_dict = json.loads(person) # Python dictionary

# Output: {'name': 'Bob', 'languages': ['English', 'Fench']}
print(person_dict)

# Output: ['English', 'French']
print(person_dict['languages'])

To read a file containing JSON object, you can use json.load() method

Suppose, you have a file named person.json which contains a JSON object.

{
 "name": "Bob",
 "languages": [
 "English",
 "Fench"
 ]
}

Parse this file:

import json

with open('person.json') as f:
 data = json.load(f)

# Output: {'name': 'Bob', 'languages': ['English', 'Fench']}
print(data)

Json2Python Conversion table

JSON	Python Equivalent
object	dict
array	list
string	str
number (int)	int
number (real)	float
true	True
false	False
null	None

Python to JSON

You can convert a dictionary to JSON string using json.dumps() method.

import json

person_dict = {
 'name': 'Bob',
 'age': 12,
 'children': None
}
person_json = json.dumps(person_dict)

# Output: {"name": "Bob", "age": 12, "children": null}
print(person_json)

To write JSON to a file in Python, we can use json.dump() method.

import json

person_dict = {
 "name": "Bob",
 "languages": [
 "English",
 "Fench"
 ],
 "married": True,
 "age": 32
}

with open('person.json', 'w') as json_file:
 json.dump(person_dict, json_file)

When you run the program, the person.txt file will be created. The file has following text inside it.

{"name": "Bob", "languages": ["English", "Fench"], "married": true, "age": 32}

Python2Json Conversion table

Python	JSON Equivalent
dict	object
list, tuple	array
str	string
int, float, int- & float-derived Enums	number
True	true
False	false
None	null

Formatting

Indent

Use the indent parameter to define the numbers of indents. For example:

json.dumps(person_dict, indent=4)

Order the result

Sort keys in ascending order using sort_keys=True. E.g.:

json.dumps(person_dict, indent = 4, sort_keys=True)

Reference

YAML in Python

Wed, 09 Dec 2020 00:00:00 +0000

YAML

YAML = YAML Ain’t Markup Language
Human-readable data-serialization language
It is commonly used for configuration files, but it is also used in data storage (e.g. debugging output) or transmission (e.g. document headers).
Filename extension: .yaml

Use YAML in Python

PyYAML is a YAML parser and emitter for Python.

$ pip install pyyaml

Read YAML

Let’s say we have a YAML file called person_info.yaml:

name: "Ecko"
age: 20
hobby:
 - "Basketball"
 - "gym"
 - "coding":
 language:
 - Python
 - Java
 editor:
 - vscode
 - JupyterLab
university: "KIT"
student: True

We use yaml.safe_load() to read a YAML file:

with open("test.yaml", "r") as f:
 person_info = yaml.safe_load(f)

person_info

{
 'age': 20,
 'hobby': [
 'Basketball',
 'gym',
 {'coding':
 {
 'editor': ['vscode', 'JupyterLab'],
 'language': ['Python', 'Java']
 }
 }
 ],
 'name': 'Ecko',
 'student': True,
 'university': 'KIT'
}

The YAML file will be read to a dict, and we can access keys and values. For example,

print(person_info["name"])

Ecko

Read multiple YAMLs

Multiple YAML documents are read with yaml.safe_load_all().

Example: persons.yaml

---
name: "Ecko"
age: 20
hobby:
 - "Basketball"
 - "gym"
 - "coding":
 language:
 - Python
 - Java
 editor:
 - vscode
 - JupyterLab
university: "KIT"
student: True
---
name: "Tan"
age: 21
hobby:
 - football
 - reading
university: "TUM"
student: True

We read this YAML file and print name and age of each person:

with open("persons.yaml") as stream:
 persons = yaml.safe_load_all(stream)

 for person in persons:
 name = person["name"]
 age = person["age"]
 print(f"Name: {name}, Age: {age}")

Name: Ecko, Age: 20
Name: Tan, Age: 21

Write YAML

To serialize a Python object into a YAML stream, use yaml.safe_dump().

In order to format the output YAML, you can use keyword arguments of safe_dump(). Some useful arguments are:

default_flow_style : indicates if a collection is block or flow. The possible values are None, True, False.
indent: sets the preferred indentation
explicit_start: if True, adds an explicit start using ---
explicit_end: if True, adds an explicit end using ---

For example

person_info = {
 "name": "Ecko",
 "age": 20,
 "hobby": [
 "Basketball",
 {
 "coding": {
 "language": ["Python", "Java", "C#"],
 "start_from": 2014
 }
 }
 ]
}

with open("person.yaml", "w") as f:
 yaml.safe_dump(person_info, f, default_flow_style=False, indent=2)

person.yaml will look like following:

age: 20
hobby:
- Basketball
- coding:
 language:
 - Python
 - Java
 - C#
 start_from: 2014
name: Ecko

Write multiple YAML files

Use yaml.safe_dump_all()

Reference

Python YAML tutorial: short and quick tutorial
Python的PyYAML模块详解
PyYAML Documentation: official documentation of PyYAML. However, it is out-of-date and maybe the most messy and the worst documentation I’ve ever read. 👎

Object Oriented Programming in Python and

Wed, 04 May 2022 00:00:00 +0000

OOP Basics

Wed, 04 May 2022 00:00:00 +0000

The concept of OOP in Python focuses on creating reusable code. This concept is also known as DRY (Don’t Repeat Yourself).

Class

A class is a blueprint for the object.

Example

class Parrot:

 # class attribute (same for all instances of a class)
 species = "bird"

 # instance attribute (different for every instance of a class)
 def __init__(self, name, age):
 self.name = name
 self.age = age

Object

An object (instance) is an instantiation of a class.

Example:

blu = Parrot("Blu", 10)
woo = Parrot("Woo", 15)

Access class attributes:

>>> blu.species
'bird'
>>> woo.species
'bird'

Access instance attributes:

>>> print(f"{blu.name} is {blu.age} years old.")
Blu is 10 years old.
>>> print(f"{woo.name} is {woo.age} years old.")
Woo is 15 years old.

Methods

Functions defined inside the body of a class
Define the behaviors of an object

Example

class Parrot:

 # class attribute
 species = "bird"

 # instance attribute
 def __init__(self, name, age):
 self.name = name
 self.age = age

 def sing(self, song):
 print(f"{self.name} is singing {song}.")

 def dance(self):
 print(f"{self.name} is dancing.")

>>> blu = Parrot("Blu", 10)
>>> blu.sing("'Happy'")
Blu is singing 'Happy'.

Inheritance

Inheritance enables us to define a class that takes all the functionality from a parent class and allows us to add more.

class SuperClass:

 def __init__(self):
 pass

 def super_method(self):
 pass


class SubClass(SuperClass):

 def __init__(self):
 super().__init__()
 pass

 def super_method(self):
 """Override method of SuperClass"""
 pass

 def sub_method(self):
 """Method that only belongs to SubClass"""
 pass

Example

# parent class
class Bird:

 def __init__(self):
 print("Bird is ready")

 def who_is_this(self):
 print("Bird")

 def swim(self):
 print("Swim faster")

# child class
class Penguin(Bird):

 def __init__(self):
 # call super() function
 # run the __init__() method of the parent class inside the child class.
 super().__init__()
 print("Penguin is ready")

 # overrides method
 def who_is_this(self):
 print("Penguin")

 def run(self):
 print("Run faster")

>>> peggy = Penguin()
Bird is ready
Penguin is ready
>>> peggy.who_is_this()
Penguin
>>> peggy.run()
Run faster

Encapsulation

Encapsulation: Restrict access to methods and variables to prevent data from direct modification.

In python, private attributes is denoted using underscore(s) as the prefix, i.e., _ or __. The main difference between them is:

Prefix _ is just a naming convention indicating a name is meant for internal use. It is NOT enforced by the Python interpreter (except in wildcard imports) and meant as a hint to the programmer only.
Prefix __ triggers name mangling when used in a class context (i.e., can be not directly accessed) and is enforced by the Python interpreter.

More about underscores see: Underscores in Python.

Example:

class Computer:

 def __init__(self):
 self._min_price = 800
 self.__max_price = 900

 def sell(self):
 print(f"Price: {self._min_price} ~ {self.__max_price}")

 def set_max_price(self, max_price):
 self.__max_price = max_price

>>> computer.sell()
Price: 800 ~ 900

Variable with __ prefix can not be directly accessed or modified:

>>> computer.__max_price
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-20-2eb906d81ddf> in <module>()
----> 1 computer.__max_price

AttributeError: 'Computer' object has no attribute '__max_price'

>>> computer.__max_price = 1000
>>> computer.sell()
Price: 800 ~ 900

We have to use setters and getters:

>>> computer.get_max_price()
900

>>> computer.set_max_price(1000)
computer.sell()

In contrast, variable with _ can be directly accessed or modified:

>>> computer._min_price
800
>>> computer._min_price = 950
>>> computer._min_price
950

Polymorphism

Polymorphism is an ability (in OOP) to use a common interface for multiple forms (data types).

Example:

class Parrot:

 def fly(self):
 print("Parrot can fly")

 def swim(self):
 print("Parrot can't swim")

class Penguin:

 def fly(self):
 print("Penguin can't fly")

 def swim(self):
 print("Penguin can swim")

# common interface
def flying_test(bird):
 bird.fly()

#instantiate objects
blu = Parrot()
peggy = Penguin()

# passing the object
flying_test(blu)
flying_test(peggy)

Output

Parrot can fly
Penguin can't fly

Reference

Operator Overloading

Wed, 04 May 2022 00:00:00 +0000

In Python, methods with __ (double underscore) prefix and suffix (e.g., __init__()) are special methods.They are called magic methods or dunder methods (dunder for “double underscore”). They can help override functionality for built-in functions for custom classes.

Essentially, each built-in function or operator has a special method corresponding to it. For example, there’s __len__(), corresponding to len(), and __add__(), corresponding to the + operator.

By default, most of the built-ins and operators will not work with objects of your classes. You must add the corresponding special methods in your class definition to make your object compatible with built-ins and operators. When you do this, the behavior of the function or operator associated with it changes according to that defined in the method.

The Internals of Operations

Every class in Python defines its own behavior for built-in functions and methods. Under the hood, when you pass an instance of some class to a built-in function or use an operator on the instance, it is actually equivalent to calling a special method with relevant arguments.

If there is a built-in function, func(), and the corresponding special method for the function is __func__(), Python interprets a call to the function as obj.__func__(), where obj is the object.
In the case of operators, if you have an operator opr and the corresponding special method for it is __opr__(), Python interprets something like obj1 <opr> obj2 as obj1.__opr__(obj2).

For example

When you’re calling len() on an object, Python handles the call as obj.__len__().
```
>>> a = 'Real Python'
>>> len(a)
11
>>> a.__len__()
11
```
When you use the [] operator on an iterable to obtain the value at an index, Python handles it as itr.__getitem__(index), where itr is the iterable object and index is the index you want to obtain.
```
b = ['Real', 'Python']
>>> b[0]
'Real'
>>> b.__getitem__(0)
'Real'
```

You can see these special methods using the built-in dir() function.

Overloading Built-in Functions

To overload the built-in functions, you only need to define the corresponding special method in your class.

In the following, we’ll demonstrate the overloading with some common built-in functions.

`len()`: Gives a Length to Your Objects

To change the behavior of len(), you need to define the __len__() special method in your class. Whenever you pass an object of your class to len(), your custom definition of __len__() will be used to obtain the result.

Example

class Order:

 def __init__(self, cart, customer):
 self.cart = list(cart)
 self.customer = customer

 def __len__(self):
 return len(self.cart)

>>> order = Order(["apple", "banana"], "Ben")
>>> len(order)
2

Use len() to directly obtain the length of the cart is more pythonic and more intuitive than calling something like order.get_cart_len().

`str()`: Prints Your Objects Prettily

The str() built-in is used to obtain a user-friendly string representation of the object which can be read by a normal user rather than the programmer.
__str__() is the method that is used by Python when you call print() on your object.
__str__() must return a str object

Example

class Order:

 def __init__(self, cart, customer):
 self.cart = list(cart)
 self.customer = customer

 def __len__(self):
 return len(self.cart)

 def __str__(self):
 return f"{self.customer} has {self.__len__()} products in the cart."

>>> order = Order(["apple", "banana"], "Ben")
>>> print(order)
Ben has 2 products in the cart.

`repr()`: Represents Your Objects

The repr() built-in is used to obtain the parsable string representation of an object.
- An object is parsable means that Python should be able to recreate the object from the representation when repr is used in conjunction with functions like eval().
repr() is also the method Python uses to display the object in a REPL session.

Example

class Vector:

 def __init__(self, x_comp, y_comp):
 self.x_comp = x_comp
 self.y_comp = y_comp

 def __repr__(self):
 return f"Vector({self.x_comp}, {self.y_comp})"

>>> vector = Vector(3, 4)
>>> repr(vector)
'Vector(3, 4)'

>>> new_vector = eval(repr(vector))
>>> new_vector # Looking at object; __repr__ used
Vector(3, 4)

Note:

In cases where the __str__() method is not defined, Python uses the __repr__() method to print the object, as well as to represent the object when str() is called on it.

# In the vector example above, we do not defined __str__() method
>>> print(new_vector)
Vector(3, 4) # vector object is printed using __repr__()

If both the methods are missing, it defaults to <__main__.Vector ...>.
__repr__() is the only method that is used to display the object in an interactive session. Absence of it in the class yields <__main__.Vector ...>.
Many of the popular libraries ignore this distinction and use the two methods interchangeably 🤪.

For more about __repr__() and __str__(), check: Python String Conversion 101: Why Every Class Needs a “repr”.

`bool()`: Makes Your Objects Truthy or Falsey

The bool() built-in can be used to obtain the truth value of an object.
The behavior defined here will determine the truth value of an instance in all contexts that require obtaining a truth value such as in if statements.

Example

class Order:

 def __init__(self, cart, customer):
 self.cart = list(cart)
 self.customer = customer

 def __len__(self):
 return len(self.cart)

 def __str__(self):
 return f"{self.customer} has {self.__len__()} products in the cart."

 def __bool__(self):
 return len(self.cart) > 0

>>> order = Order(["apple", "banana"], "Ben")
>>> bool(order)
True
>>> order_2 = Order([], "Amy")
>>> bool(order)
False

Overloading Built-in Operators

Changing the behavior of operators is just as simple as changing the behavior of functions: You define their corresponding special methods in your class, and the operators work according to the behavior defined in these methods.

Usually, these special methods need to accept another argument in the definition other than self, generally referred to by the name other.

`+`: Makes Your Objects Capable of Being Added

The special method corresponding to the + operator is the __add__() method.
It is recommended that __add__() returns a new instance of the class instead of modifying the calling instance itself.

Actually this behaviour is quiet often in Python:

>>> a = "Hello"
>>> b = "World"
>>> a + b
`Hello World`
>>> a # remains unchanged
'Hello '

>>> list_1 = [1, 2]
>>> list_2 = [3, 4]
>>> list_1 + list_2
[1, 2, 3, 4]
>>> list_1 # remains unchanged
[1, 2]

Example (using the Order class above):

class Order:

 def __init__(self, cart, customer):
 self.cart = list(cart)
 self.customer = customer

 def __len__(self):
 return len(self.cart)

 def __str__(self):
 return f"{self.customer} has {self.__len__()} products in the cart."

 def __bool__(self):
 return len(self.cart) > 0

 def __add__(self, other):
 new_cart = self.cart.copy()
 new_cart + other if isinstance(other, list) else new_cart.append(other)
 return Order(new_cart, self.customer)

>>> order = Order(["apple", "banana"], "Ben")
>>> (order + "pear").cart # New Order instance
['apple', 'banana', 'pear']
>>> order.cart # Original instance unchanged
['apple', 'banana']

>>> order = order + 'mango' # Changing the original instance
>>> order.cart
['apple', 'banana', 'mango']

Similarly, you have the __sub__(), __mul__(), and other special methods which define the behavior of -, *, and so on. These methods should return a new instance of the class as well.

Shortcuts: the `+=` operator

The += operator stands as a shortcut to the expression obj1 = obj1 + obj2
Corresponding special method: __iadd()__
The __iadd__() method should make changes directly to the self argument and return the result, which may or may not be self.

Roughly, any += use on two objects is equivalent to:
```
result = obj1 + obj2
obj1 = resul
```

Example:

class Order:

 def __init__(self, cart, customer):
 self.cart = list(cart)
 self.customer = customer

 def __len__(self):
 return len(self.cart)

 def __str__(self):
 return f"{self.customer} has {self.__len__()} products in the cart."

 def __bool__(self):
 return len(self.cart) > 0

 def __add__(self, other):
 new_cart = self.cart.copy()
 new_cart + other if isinstance(other, list) else new_cart.append(other)
 return Order(new_cart, self.customer)

 def __iadd__(self, other):
 self.cart + other if isinstance(other, list) else self.cart.append(other)
 return self

>>> order = Order(["apple", "banana"], "Ben")
>>> order += "mange"
>>> order.cart
['apple', 'banana', 'mange']

Always make sure that you’re returning something in your implementation of __iadd__() and that it is the result of the operation and not anything else.

Similar to __iadd__(), you have __isub__(), __imul__(), __idiv__() and other special methods which define the behavior of -=, *=, /=, and others alike.

`[]`: Indexes and Slices Your Objects

The [] operator is called the indexing operator and is used in various contexts in Python such as g

etting the value at an index in sequences,
getting the value associated with a key in dictionaries,
obtaining a part of a sequence through slicing.

You can change its behavior using the __getitem__() special method.

Example:

class Order:

 def __init__(self, cart, customer):
 self.cart = list(cart)
 self.customer = customer

 def __len__(self):
 return len(self.cart)

 def __str__(self):
 return f"{self.customer} has {self.__len__()} products in the cart."

 def __bool__(self):
 return len(self.cart) > 0

 def __add__(self, other):
 new_cart = self.cart.copy()
 new_cart + other if isinstance(other, list) else new_cart.append(other)
 return Order(new_cart, self.customer)

 def __iadd__(self, other):
 self.cart + other if isinstance(other, list) else self.cart.append(other)
 return self

 def __getitem__(self, key):
 return self.cart[key]

>>> order = Order(["apple", "banana"], "Ben")
>>> order[1]
'banana'

Reverse Operators: Makes Your Classes Mathematically Correct

While defining the __add__(), __sub__(), __mul__(), and similar special methods allows you to use the operators when your class instance is the left-hand side operand, the operator will NOT work if the class instance is the right-hand side operand.

If your class represents a mathematical entity like a vector, a coordinate, or a complex number, applying the operators should work in both the cases since it is a valid mathematical operation. To help you make your classes mathematically correct, Python provides you with reverse special methods such as __radd__(), __rsub__(), __rmul__(), and so on.

These handle calls such as x + obj, x - obj, and x * obj, where x is not an instance of the concerned class.
These reverse special methods should return a new instance of class with the changes of the operation rather than modifying the calling instance itself.

Resource

Section 3.3, Special Method Names of the Data Model section in the Python documentation
Fluent Python by Luciano Ramalho
Python Tricks: The Book

Reference

Operator and Function Overloading in Custom Python Classes

Object Comparison: == vs is

Sun, 04 Dec 2022 00:00:00 +0000

The == operator compares by checking for equality (the objects referred to by the variables are equal, i.e. have the same contents)
The is operator comparies identities (whether two variables are in fact pointing to the same identical object)

Example

>>> a = [1, 2, 3]
>>> b = a

>>> a
[1, 2, 3]
>>> b
[1, 2, 3]

>>> a == b
True

>>> a is b
True

a and b point to the same list object, as we assigned them earlier.

>>> c = list(a) # create an identical copy of the list a

>>> c
[1, 2, 3]

>>> a == c # should be true, as a and c have the same content
True

>>> a is c
False

String Conversion

Sun, 04 Dec 2022 00:00:00 +0000

TL;DR

Control to-string conversion in your own classes using the __str__ and __repr__ “dunder” methods
The result of __str__ should be readable. The result of __repr__ should be unambiguous.
Always add a __repr__ to your classes. The default implemen- tation for __str__ just calls __repr__.

When you define a custom class in Python and then try to print one of its instances to the console (or inspect it in an interpreter session), by default you will get a string containing the class name and the id of the object instance (which is the object’s memory address in CPython.) → This default “to string” conversion behavior is basic and lacks detail.

Example

class Car:
 def __init__(self, color, mileage):
 self.color = color
 self.mileage = mileage

>>> my_car = Car('red', 37281)
>>> print(my_car)
<__console__.Car object at 0x109b73da0>
>>> my_car
<__console__.Car object at 0x109b73da0>

nstead of building your own to-string conversion machinery, you’ll be better off adding the __str__ and __repr__ “dunder” methods to your class. They are the Pythonic way to control how objects are converted to strings in different situations

class Car:
 def __init__(self, color, mileage):
 self.color = color
 self.mileage = mileage

 def __str__(self):
 # return f"a {self.color} car"
 return f'a {self.color} car'

>>> my_car = Car('red', 37281)
>>> print(my_car)
'a red car'
>>> my_car
<__console__.Car object at 0x109ca24e0>

`str` vs. `repr`

When print() an object, or involving string, the __str__ method will be called.
inspecting an object in a Python inter- preter session simply prints the result of the object’s __repr__.

Example:

class Car:
 def __init__(self, color, mileage):
 self.color = color
 self.mileage = mileage

 def __str__(self):
 # return f"a {self.color} car"
 return "__str__ for Car"

 def __repr__(self):
 return "__repr__ for Car"

>>> my_car = Car('red', 37281)

>>> print(my_car)
__str__ for Car

>>> '{}'.format(my_car)
'__str__ for Car'

>>> my_car
__repr__ for Car

Interestingly, containers like lists and dicts always use the result of __repr__ to represent the objects they contain. Even if you call str on the container itself:

>>> str([my_car])
'[__repr__ for Car]'

To express your code’s intent more clearly, it’s best to use the built-in str() and repr() functions. Using them is preferable over calling the object’s __str__ or __repr__ directly, as it looks nicer and gives the same result:

>>> str(my_car)
'__str__ for Car'

>>> repr(my_car)
'__repr__ for Car'

Difference of usage of __str__ and __repr__:

The result of the date object’s __str__ function should primarily be readable.
- It’s meant to return a concise textual representation for human consumption—something you’d feel comfortable displaying to a user.
With __repr__, the idea is that its result should be, above all, unambiguous.
- The resulting string is intended more as a debugging aid for developers.
  
  → It needs to be as explicit as possible about what this object is.
- Rule of thumb: make __repr__ strings unambiguous and helpful for developers 💪

Example:

>>> import datetime
>>> today = datetime.date.today()

>>> str(today)
'2022-12-04'

>>> repr(today)
'datetime.date(2022, 12, 4)'

Why Every Class Needs a `repr`

If you don’t add a __str__ method, Python falls back on the result of __repr__ when looking for __str__. Therefore, it is recommended to always add at least a __repr__ method to your classes. This will guarantee a useful string conversion result in almost all cases, with a minimum of implementation work 👏.

Example:

class Car:
 def __init__(self, color, mileage):
 self.color = color
 self.mileage = mileage

 def __str__(self):
 return f"a {self.color} car"

 def __repr__(self):
 return f"{self.__class__.__name__}({self.color!r}, {self.mileage!r})"

Note: using the !r conversion flag makes sure the output string uses repr(self.color) and repr(self.mileage) instead of str(self.color) and str(self.mileage)

Define Your Own Exception Classes

Sun, 04 Dec 2022 00:00:00 +0000

TL;DR

Defining your own exception types will state your code’s intent more clearly and make it easier to debug.
Derive your custom exceptions from Python’s built-in Exception class or from more specific exception classes like ValueError or KeyError.
You can use inheritance to define logically grouped exception hierarchies.

Defining your own error types can be of great value:

You’ll make potential error cases stand out clearly → your functions and modules will become more maintainable.
You can also use custom error types to provide additional debugging information.

→ All of this will improve your Python code and make it easier to under- stand, easier to debug, and more maintainable.

Example: Let’s say you wanted to validate an input string representing a person’s name in your application.

def validate(name):
 if len(name) < 10:
 raise ValueError

However, there’s a downside to using a “high-level” generic exception class like ValueError. When a name fails to validate, it’ll look like this in the debug stack trace:

>>> validate('joe')
Traceback (most recent call last):
 File "<input>", line 1, in <module>
 validate('joe')
 File "<input>", line 3, in validate
 raise ValueError
ValueError

This stack trace isn’t really all that helpful. To be able to fix the problem, your teammate almost certainly has to look up the implementation of validate(). This could be time consuming.

An improvement for this is to introduce a custom exception type to represent a failed name validation:

class NameToShortError(ValueError):
 pass

def validate(name):
 if len(name) < 10:
 raise NameToShortError

The custom NameTooShortError exception is “self-documenting”, which results in a much nicer stack track when the validation fails:

>>> validate('jane')
Traceback (most recent call last):
 File "<input>", line 1, in <module>
 validate('jane')
 File "<input>", line 3, in validate
 raise NameTooShortError(name)
NameTooShortError: jane

Custom exception classes make it much easier to understand what’s going on when things go wrong (and eventually they always do). By spending just 30 seconds on defining a simple exception class, this code snippet became much more communicative already.

Good Practice

It’s good practice to create a custom exception base class for the module and then derive all of your other exceptions from it.

Declare a base class that all of our concrete errors will inherit from. E.g.
```
class BaseValidationError(ValueError):
 pass
```

All of our “real” error classes can be derived from the base error class. This gives a nice and clean exception hierarchy with little extra effort.

E.g.

class NameTooShortError(BaseValidationError):
 pass

class NameTooLongError(BaseValidationError):
 pass

class NameTooCuteError(BaseValidationError):
 pass

Of course you can take this idea further and logically group your exceptions into fine grained sub-hierarchies. But be careful—it’s easy to introduce unnecessary complexity by going overboard with this.

In conclusion, defining custom exception classes makes it easier for your users to adopt an it’s easier to ask for forgiveness than permission (EAFP) coding style that’s considered more Pythonic.

Object Cloning

Sun, 11 Dec 2022 00:00:00 +0000

TL;DR

Making a shallow copy of an object won’t clone child objects. → The copy is not fully independent of the original.
A deep copy of an object will recursively clone child objects.The clone is fully independent of the original, but creating a deep copy is slower.
You can deep or shallow copy arbitrary objects with the standard copy module.

Shallow vs. Deep Copy

For compound objects, there’s an important difference between shallow and deep copying:

Shallow copy
- Constructing a new collection object and then populating it with references to the child objects found in the original → only one level deep
- The copying process does NOT recurse and therefore won’t create copies of the child objects themselves.
Deep copy
- Makes the copying process recursive
  - first constructing a new collection object
  - then recursively populating it with copies of the child objects found in the original
  → Copying an object this way walks the whole object tree to create a fully independent clone of the original object and all of its children.

Shallow Copy

Python’s built-in mutable collections like lists, dicts, and sets can be shallowly copied by calling their factory functions on an existing collection:

new_list = list(original_list)
new_dict = dict(original_dict)
new_set = set(original_set)

Example

we’ll create a new nested list and then shallowly copy it with the list() factory function:

>>> xs = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
>>> ys = list(xs) # Make a shallow copy

Now ys will be a new and independent object with the same contents as xs:

>>> xs
[[1, 2, 3], [4, 5, 6], [7, 8, 9]]

>>> ys
[[1, 2, 3], [4, 5, 6], [7, 8, 9]]

To further verify that ys is really independent from the original, we add a new sublist ot the original xs and check to make sure this modification didn’t affect the copy ys:

>>> xs.append(['new'])
>>> xs
[[1, 2, 3], [4, 5, 6], [7, 8, 9], ['new']]

>>> ys
[[1, 2, 3], [4, 5, 6], [7, 8, 9]]

However, because we only created a shallow copy of the original list xs, ys still contains references to the original child objects stored in xs. These children were NOT copied. They were merely referenced again in the copied list.

>>> xs[1][0] = 'X'
>>> xs
[[1, 2, 3], ['X', 5, 6], [7, 8, 9], ['new']]

>>> ys
[[1, 2, 3], ['X', 5, 6], [7, 8, 9]]

The diagram below shows how this works:

Use The`copy` Module

The copy module in the Python standard library provides a simple interface for creating shallow and deep copies of arbitrary Python objects.

copy.deepcopy() creates deep copy
copy.copy() creates shallow copy

For built-in collections it’s considered more Pythonic to simply use the list, dict, and set factory functions to create shallow copies.

Abstract Base Class (ABC)

Sun, 11 Dec 2022 00:00:00 +0000

TL;DR

Abstract Base Classes (ABCs) ensure that derived classes implement particular methods from the base class at instantiation time
Using Python’sabc module help avoid bugs and make class hierarchies easier to maintain.

In OOP, inheritance is very common. To make the code of inheritance relationship as maintainable and programmer-friendly as possible, we want to make sure that

instantiating the base class is impossible
forgetting to implement interface methods in one of the sub-classes raises an error as early as possible.

To enforce that a derived class implements a number of meth- ods from the base class, something like this Python idiom is typically used:

class Base:
 def foo(self):
 raise NotImplementedError()

 def bar(self):
 raise NotImplementedError()


class Concrete(Base):
 def foo(self):
 return "foo() called"

 # We forgot to override bar()...

Calling methods on an instance of Base correctly raises NotImplementedError exceptions:

>>> b = Base()
>>> b.foo()
NotImplementedError

Furthermore, instantiating and using Concrete works as expected. If we call an unimplemented method like bar() on it, this also raises an exception:

>>> c = Concrete()

>>> c.foo()
'foo() called'

>>> c.bar()
NotImplementedError

This implementation is decent. However, it has two downsides

instantiate Base just fine without getting an error (which is not expected, as the base class should be uninstantiatable)
provide incomplete subclasses—instantiating Concrete will not raise an error until we call the missing method bar()

We can solve these issues with Python’s abc module. Here’s an updated implemen- tation using an Abstract Base Class defined with the abc module:

from abc import ABCMeta, abstractmethod

class Base(metaclass=ABCMeta):
 @abstractmethod
 def foo(self):
 pass

 @abstractmethod
 def bar(self):
 pass

class Concrete(Base):
 def foo(self):
 pass

 # We forget to declare bar()

This still behaves as expected and creates the correct class hierarchy:

>>> issubclass(Concrete, Base)
True

Benefit of applying the abc module:

Instantiating base class will raise an exception

>>> b = Base()
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [19], line 1
----> 1 b = Base()

TypeError: Can't instantiate abstract class Base with abstract methods bar, foo

Subclasses of Base raise a TypeError at instantiation time whenever we forget to im- plement any abstract methods. The raised exception also tells us which method or methods we’re missing, makes it more difficult to write invalid subclasses.

>>> c = Concrete()
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [10], line 1
----> 1 c = Concrete()

TypeError: Can't instantiate abstract class Concrete with abstract method bar

These advantages often make the class hierarchies more robust and more readily maintainable. Using ABCs states the programmer’s intent clearly and thus makes the code more communicative. 👍

Class vs Instance Variable

Fri, 16 Dec 2022 00:00:00 +0000

TL;DR

Class variables are for data shared by all instances of a class. They belong to a class, not a specific instance and are shared among all instances of a class.
Instance variables are for data that is unique to each instance. They belong to individual object instances and are not shared among the other instances of a class. Each instance variable gets a unique backing store specific to the instance.
Class variables can be incautiously “shadowed” by instance variables of the same name, which could introduceds bugs and odd behaviour.
Be careful and double-check your scoping when dealing with shared state on a class. Automated tests and peer reviews help greatly with that.

Class Variable vs. Instance Variable

Two kinds of data attributes on Python objects

Class variables
- Declared inside the class definition (but outside of any instance methods)
- Not tied to any particular instance of a class
- Class variables store their contents on the class itself, and all objects created from a particular class share access to the same set of class variables.
  - I.e.modifying a class variable affects all object instances at the same time.
Instance variables
- Always tied to a particular object instance
- Contents of an instance vari- able are completely independent from one object instance to the next.
  - I.e. modifying an instance variable only affects one object instance at a time.

Example

class Dog:
 num_legs = 4 # class variabel

 def __init__(self, name):
 self.name = name # instance variable

>>> jack = Dog('Jack')
>>> jill = Dog('Jill')
>>> jack.name, jill.name
('Jack', 'Jill')

We can access the num_legs class variable either directly on each Dog instance or on the class itself :

>>> jack.num_legs, jill.num_legs
(4, 4)

>>> Dog.num_legs
4

If we try to access an instance variable through the class, it’ll fail with an AttributeError.

>>> Dog.name
AttributeError:
"type object 'Dog' has no attribute 'name'"

The reason is that instance variables are specific to each object instance and are created when the __init__ constructor runs—they do NOT even exist on the class itself.

Now let’s say we want jack to have 6 legs. If we directly modify the num_legs class variable on the Dogclass:

>>> Dog.num_legs = 6

>>> jack.num_legs, jill.num_legs
(6, 6)

As we can see, modifying a class variable on the class namespace affects all instances of the class.

Let’s roll back the change to the class variable and instead try to give an extra pair o’ legs specifically to jackonly:

>>> Dog.num_legs = 4
>>> jack.num_legs = 6

>>> jack.num_legs, jill.num_legs, Dog.num_legs
(6, 4, 4)

It seems ok, and we got the result we wanted. But a pitfalls occurs: we introduced a num_legs instance variable to the jack instance. And now the new num_legs instance variable “shadows” the class variable of the same name, overriding and hiding it when we access the object instance scope.

>>> jack.num_legs, jack.__class__.num_legs
(6, 4)

Instance, Class, and Static Methods

Fri, 16 Dec 2022 00:00:00 +0000

TL:DR

Instance methods need a class instance and can access the instance through self.
Class methods don’t need a class instance. They can’t accessthe instance (self) but they have access to the class itself via cls.
Static methods don’t have access to cls or self. They work like regular functions but belong to the class’ namespace.
Static and class methods communicate and (to a certain degree) enforce developer intent about class design. This can have definite maintenance benefits.

Instance vs. Class vs. Static Methods

class MyClass:

 # instance method
 def method(self):
 return "Instance method called", self

 # class method
 @classmethod
 def classmethod(cls):
 return "Class method called", cls

 # static method
 @staticmethod
 def staticmethod():
 return "Static method called"

Instance method (method(self))
- Takes one parameter, self, which points to an instance of MyClass when the method is called
  - Through the self parameter, instance methods can freely access attributes and other methods on the same object.
- instance methods can also access the class itself through the self.__class__ attribute. I.e., instance methods can also modify class state.
Class method (classmethod(cls))
- Marked with a @classmethod decorator
- Takes a cls that points to the class when the method is called
- Since the class method only has access to this cls argument, it can’t modify object instance state. But it can still modify class state that applies across all instances of the class.
Static method (staticmethod)
- Does not take a self or cls parameter
- Can NOT modify object state or class state

Example

Instance method:

>>> obj = MyClass()
>>> obj.method()
('Instance method called', <__main__.MyClass at 0x7f7fd8365eb0>)

When the method is called, Python replaces the self argument with the instance object, obj.

Instance methods can also access the class itself through the self.__class__ attribute. $\rightarrow$ Instance methods can freely modify state on the object instance and on the class itself.

Class method:

>>> obj.classmethod()
('Class method called', __main__.MyClass)

The result shows us that it doesn’t have access to the <MyClass instance> object,but only to the <class MyClass> object, representing the class itself (everything in Python is an object, even classes themselves).

Static method:

>>> obj.staticmethod()
'static method called'

Static methods can neither access the object instance state nor the class state. They work like regular functions but belong to the class’ (and every instance’s) namespace.

Now let’s call these method on the class itself:

>>> MyClass.classmethod()
('Class method called', __main__.MyClass)

>>> MyClass.staticmethod()
'Static method called'

>>> MyClass.method()
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [8], line 1
----> 1 MyClass.method()

TypeError: method() missing 1 required positional argument: 'self'

Attempting to call the instance method method() failed with a TypeError.

This is to be expected. As this time we didn’t create an object instance and tried calling an instance function directly on the class blueprint itself. This means there is no way for Python to populate the self argument and therefore the call fails with a TypeError exception.

When to Use `@classmethod`

Let’s say we have a Pizza class:

class Pizza:
 def __init__(self, ingredients):
 self.ingredients = ingredients

 def __repr__(self):
 return f"pizza ({self.ingredients!r})"

and want to creat various pizza objects:

Pizza(['mozzarella', 'tomatoes'])
Pizza(['mozzarella', 'tomatoes', 'ham', 'mushrooms'])
Pizza(['mozzarella'] * 4)

A nice and clean way to do that is by using class methods as factory functions for the different kinds of pizzas we can create:

class Pizza:
 def __init__(self, ingredients):
 self.ingredients = ingredients

 def __repr__(self):
 return f"pizza ({self.ingredients!r})"

 # Use classmethod as factory function to create different kinds of pizza

 @classmethod
 def margherita(cls):
 return cls(['mozzarella', 'tomatoes'])

 @classmethod
 def prosciutto(cls):
 return cls(['mozzarella', 'tomatoes', 'ham'])

Now we use the cls argument in the margherita and prosciutto factory methods instead of calling the Pizza constructor directly. Under the hood, these factory methods all use the same __init__ constructor internally and simply provide a shortcut for remembering all of the various ingredients.

This is a trick called Don’t Repeat Yourself (DRY) principle. The advantage is that if we decide to rename this class at some point, we won’t have to remember to update the constructor name in all of the factory functions.

Another way to look at this use of class methods is to realize that they allow you to define alternative constructors for your classes.

Python only allows one __init__ method per class.
Using class methods makes it possible to add as many alternative constructors as necessary. $\rightarrow$ This can make the interface for your classes self-documenting (to a certain degree) and simplify their usage.

When To Use `@staticmethod`

Static methods can’t access class or instance state because they don’t take a cls or self argument
- Using static methods is a great signal to show that particular methods are independent from everything else around them
- This restriction is also enforced by the Python runtime. It allows you to communicate clearly about parts of your class architecture so that new development work is naturally guided to happen within these boundaries. In practice, they often help avoid accidental modifications that go against the original design.
using static methods and class methods are ways to communicate developer intent while enforcing that intent enough to avoid most “slip of the mind” mistakes and bugs that would break the design.
Static methods also have benefits when it comes to writing test code.
- As static methods are completely independent from the rest of the class, we do NOT have to worry about setting up a complete class instance before we can test the method in a unit test. We can just fire away like we would if we were testing a regular function. $\rightarrow$ this makes future maintenance easier and provides a link between object-oriented and procedural programming styles.

Property

Sun, 12 Feb 2023 00:00:00 +0000

TL;DR

Use @property decorator for getters and setter in OOP in a more pythonic way.

Python programming provides us with a built-in @property decorator which makes usage of getter and setters much easier in Object-Oriented Programming. We will learn it step by step with an example.

The Temperature Example

Let us assume that we decide to make a class that stores the temperature in degrees Celsius. And, it would also implement a method to convert the temperature into degrees Fahrenheit.

The first naive solution is as follows:

class Celsius:
 def __init__(self, temperature=0):
 self.temperature = temperature

 def to_fahrenheit(self):
 return (self.temperature * 1.8) + 32

>>> human = Celsius()
>>> human.temperature = 37

>>> human.temperature
37

>>> human.to_fahrenheit()
98.60000000000001

whenever we assign or retrieve any object attribute like temperature as shown above, Python searches it in the object’s built-in __dict__ dictionary attribute as

print(human.__dict__)
# Output: {'temperature': 37}

Therefore, human.temperature internally becomes human.__dict__['temperature'].

Suppose we want to extend the usability of the Celsius class defined above. We know that the temperature of any object cannot reach below -273.15 degrees Celsius. An obvious solution to the above restriction will be to hide the attribute temperature (make it private) and define new getter and setter methods to manipulate it:

class Celsius:
 def __init__(self, temperature=0):
 self.set_temperature(temperature)

 def to_fahrenheit(self):
 return (self.get_temperature() * 1.8) + 32

 # getter method
 def get_temperature(self):
 return self._temperature

 # setter method
 def set_temperature(self, value):
 if value < -273.15:
 raise ValueError("Temperature below -273.15 is not possible.")
 self._temperature = value

>>> human = Celsius()
>>> print(human.get_temperature())
37

>>> print(human.to_fahrenheit())
98.60000000000001

>>> human.set_temperature(-300)
Traceback (most recent call last):
 File "<string>", line 30, in <module>
 File "<string>", line 16, in set_temperature
ValueError: Temperature below -273.15 is not possible.

However, the bigger problem with the above update is that all the programs that implemented our previous class have to modify their code from obj.temperature to obj.get_temperature() and all expressions like obj.temperature = val to obj.set_temperature(val). This refactoring can cause problems while dealing with hundreds of thousands of lines of codes!!!

→ All in all, our new update was not backwards compatible.

The `Property` class

A pythonic way to deal with the above problem is to use the property class.

We can update our code:

class Celsius:
 def __init__(self, temperature=0):
 self.temperature = temperature

 def to_fahrenheit(self):
 return (self.temperature * 1.8) + 32

 # getter
 def get_temperature(self):
 print("Getting value...")
 return self._temperature

 # setter
 def set_temperature(self, value):
 print("Setting value...")
 if value < -273.15:
 raise ValueError("Temperature below -273.15 is not possible")
 self._temperature = value

 # creating a property object
 temperature = property(get_temperature, set_temperature)

The last line of the code makes a property object temperature. Property attaches some code (get_temperature and set_temperature) to the member attribute accesses (temperature).

>>> human = Celsius(37)
Setting value...

>>> print(human.temperature)
Getting value...
37

>>> print(human.to_fahrenheit())
Getting value...
98.60000000000001

>>> human.temperature = -300
Setting value...
Traceback (most recent call last):
 File "<string>", line 31, in <module>
 File "<string>", line 18, in set_temperature
ValueError: Temperature below -273 is not possible

Any code that retrieves the value of temperature (including self.temperature) will automatically call get_temperature() instead of a dictionary (dict) look-up. Similarly, any code that assigns a value to temperature will automatically call set_temperature().

By using property, we can see that NO modification is required in the implementation of the value constraint. Thus, our implementation is backward compatible. 👏

The actual temperature value is stored in the private _temperature variable. The temperature attribute is a property object which provides an interface to this private variable.

More details

In Python, property() is a built-in function that creates and returns a property object. The syntax of this function is:

property(fget=None, fset=None, fdel=None, doc=None)

Here,

fget is function to get value of the attribute
fset is function to set value of the attribute
fdel is function to delete the attribute
doc is a string (like a comment)

A property object has three methods, getter(), setter(), and deleter() to specify fget, fset and fdel at a later point. This means, the line:

temperature = property(get_temperature,set_temperature)

can be broken down as:

# make empty property
temperature = property()

# assign fget
temperature = temperature.getter(get_temperature)

# assign fset
temperature = temperature.setter(set_temperature)

The `@property` Decorator

A more python (and also the more recommended) way is to use the @property decorator. We can even not define the names get_temperature and set_temperature as they are unnecessary and pollute the class namespace. For this, we reuse the temperature name while defining our getter and setter functions.

class Celsius:
 def __init__(self, temperature=0):
 self.temperature = temperature

 def to_fahrenheit(self):
 return (self.temperature * 1.8) + 32

 @property
 def temperature(self):
 print("Getting value...")
 return self._temperature

 @temperature.setter
 def temperature(self, value):
 print("Setting value...")
 if value < -273.15:
 raise ValueError("Temperature below -273 is not possible")
 self._temperature = value

Reference

Python @property decorator

Linked List

Thu, 15 Apr 2021 00:00:00 +0000

class ListNode:
 def __init__(self, val=0, next=None):
 self.val = val
 self.next = next

Draw It out!

To solve the linked list problems, the most important thing is to draw the linked list and the procedure out. The figure can help you to think and code.

2 Main Types of Problems

Modification of links
Concatenation of linked lists

To Be Noticed

Loops

When modifying links, it is easy to create loop uncarefully. To prevent this problem, draw the list out! With the help of figure, we can immediately notice and avoid loops.

Corner Cases

Common corner cases are

Linked list is empty
Linked list contains only one node

Techniques

Dummy Head

Add a dummy_head before head

dummy_head = ListNode(val=-1, next=head)

dummy_head.next is always the first nodes after all operations
Advantage of adding a dummy_head is that we can treat head as a normal node, which can help us to handle the corner cases easily (e.g. deleting head node)
Leetcode problems
- 25

Fast and Slow pointers

Since linked list does not support random indexing as array, we have to access nodes starting from head
We can use fast and slow pointers to simulate the random access, e.g.
- If we want to access the middle node of the linked list
  - fast and slow pointers start from head.
  - At each step, slow moves one step forward, fast moves two steps forward
  - When fast reaches the end, slow reaches the middle
- If we want to access the n-th node from the end
  - slow starts from head, fast starts from the n-th node
  - At each step, slow and fast move one step forward
  - When fast reaches the end, slow reaches n-th node from the end
Leetcode problems
- 141. Linked List Cycle
- 142. Linked List Cycle II

Recursion

Linked list has the nature of recursion. If we master the idea of recursion, the solution will be suprisingly clean and succinct
To apply recursion, we just need to consider three questions
- Base case: Unter which situation / When should the recursion be ended?
  - In linked list, base cases are empty list or list containing only one node
- Return value: What should be returned to the previous level?
- Goal of current level: What should be done in the current level?
Reference: 三道题套路解决递归问题
Pre-order traversal vs. post-orderrecursion
- In pre-order traversal, we image that the previous nodes are already processed (we don’t care how they are processed)
- In post-order traversal, we image that the nodes behind are already processed (we don’t care how they are processed)
- Example: Swap nodes in pairs
Example: 24. Swap Nodes in Pairs
Leetcode problems
- 24. Swap Nodes in Pairs
- 206. Reverse Linked List

Sort Nodes Alphabetically When Modifying Links or Concatenating Nodes

When modifying links or concatenating nodes, it’s easy to get lost or create loops uncarefully. A small trick to avoid these problems is to

Draw out how the list looks like before and after modification
Mark the order of nodes alphabetically

Example

We want to move node 3 to the front of node 2. We draw the “before and after” out:

Then we mark the order of nodes alphabetically

Basics | Haobin Tan

Getting Started

Source

Operators

Identity and Membership

Built-in Types

Complex Numbers

String Type

None Type

Boolean

Built-In Data Structures

Defining and Using Functions

*args and **kwargs

Iterators

enumerate

zip

map and filter

Iterators as function arguments

Specialized Iterators: itertools

List Comprehensions

Generators

List comprehensions use square brackets, while generator expressions use parentheses

A list is a collection of values, while a generator is a recipe for producing values

A list can be iterated multiple times; a generator expression is single-use

Generator Functions: Using yield

Example: Prime Number Generator

args and kwargs

Passing multiple arguments to a function

Using the Python kwargs Variable in Function Definitions

Ordering Arguments in a Function

Forwarding Optional or Keyword Arguments

Unpacking With the Asterisk Operators: * & **

*

Other convenient uses of the unpacking operator

E.g. 1

E.g. 2

For strings

**

zip

Use zip() in python

Loop Over Multiple Iterables

Traverse lists in parallel

Traverse dictionaries in parallel

Unzip a sequence

Sorting in parallel

Calculating in pairs

Building Dictionaries

Modules and Packages

Python modules

The Module Search Path

The import statement

import <module_name>

from <module_name> import <name(s)>

from <module_name> import <name> as <alt_name>[, <name> as <alt_name> …]

import <module_name> as <alt_name>

The dir() function

Executing a Module as a Script

Reloading a module

Python packages

Package Initialization

Importing * from a package

Subpckages

Reference

Underscores in Python

TL;DR

Single Leading Underscore: _var

Single Trailing Underscore: var_

Double Leading Underscore: __var

Double Leading and Trailing Underscore: __var__

Single Underscore _

Ignoring Values

Use in Looping

Use in Interprerter

Reference

Basic Input and Output

Reading Input From the Keyboard

Writing Output to the Console

Example

Reference

String

`*args` and `**kwargs`

`enumerate`

`zip`

`map` and `filter`

Specialized Iterators: `itertools`

Generator Functions: Using `yield`

`*`

`**`

Use `zip()` in python

The `import` statement

`import <module_name>`

`from <module_name> import <name(s)>`

`from <module_name> import <name> as <alt_name>[, <name> as <alt_name> …]`

`import <module_name> as <alt_name>`

The `dir()` function

Importing `*` from a package

Single Leading Underscore: `_var`

Single Trailing Underscore: `var_`

Double Leading Underscore: `__var`

Double Leading and Trailing Underscore: `var`

Single Underscore `_`

The `+` Operator

The `*` Operator

The `in` Operator

`ord(c)`

`chr(n)`

`len(s)`

`str(obj)`

`s.capitalize()`

`s.lower()`

`s.swapcase()`

`s.title()`

`s.upper()`

`s.count(<sub>[, <start>[, <end>]])`

`s.endswith(<suffix>[, <start>[, <end>]])`

`s.find(<sub>[, <start>[, <end>]])`

`s.index(<sub>[, <start>[, <end>]])`

`s.rfind(<sub>[, <start>[, <end>]])`

`s.rindex(<sub>[, <start>[, <end>]])`

`s.startswith(<prefix>[, <start>[, <end>]])`

`s.isalnum()`

`s.isalpha()`

`s.isdigit()`

`s.isidentifier()`

`s.islower()`

`s.isprintable()`

`s.isspace()`

`s.istitle()`

`s.isupper()`

`s.center(<width>[, <fill>])`

`s.expandtabs(tabsize=8)`

`s.ljust(<width>[, <fill>])`

`s.lstrip([<chars>])`

`s.replace(<old>, <new>[, <count>])`

`s.rjust(<width>[, <fill>])`

`s.rstrip([<chars>])`

`s.strip([<chars>])`

`s.zfill(<width>)`

`s.join(<iterable>)`

`s.partition(<sep>)`

`s.rsplit(sep=None, maxsplit=-1)`

`s.split(sep=None, maxsplit=-1)`

`s.splitlines([<keepends>])`

Use built-in `re` module