Algo | Haobin Tan

Big O Notation

Sun, 04 Apr 2021 00:00:00 +0000

Definition

In general, a better algorithm means a FASTER algorithm. In other words, it costs less runtime or it has lower time complexity.

Big O notation is one of the most fundamental tools to analyze the cost of an algorithm. It is about finding an asymptotic upper bound of the time complexity of an algorithm. I.e., Big O notation is about the worst-case scenario.

Basically, it answers the question: “How the runtime of an algorithm grows as the input grow?”

Here we use number of operations to evaluate the runtime of an algorithm, Instead of second/microsecond.

Rule of Thumbs

Different steps get added

Example

def func():
 step1() # O(a)
 step2() # O(b)

Overall complexity is $O(a + b)$

Constants do NOT matter

Constant are inconsequential, when input size $n$ is getting massively huge.

Example

$O(2n) = O(n)$
$O(500) = O(1)$

Different inputs $\Rightarrow$ different variables

Example

def intersection_size(list_A, list_B):
 # Assumes that len(list_A) is lenA, len(len_listB) is lenB
 count = 0
 for a in list_A:
 for b in list_B:
 if a == b:
 count += 1

 return count

Overall complexity is $O(\text{lenA} \times \text{lenB})$

Smaller terms do NOT matter

Smaller terms are non-dominant, when input size $n$ is getting massively huge.

Example

$O(n + 100) = O(n)$
$O(n^2 + 200n) = O(n^2)$

Shorthands

Arithmetric operations are constant
Variable assignment is constant
Accessing elements in an array (by index) or object (by key) is constant
In a loop:
$$ \text{overall complexity} = (\text{length of loop}) \times (\text{complexity inside of the loop}) $$

Summary of Complexity Classes

Reference

Big O Notation

Complete Beginner’s Guide to Big O Notation

Binary Search

Sun, 04 Apr 2021 00:00:00 +0000

Binary search only works when your list is in sorted order.

Code

def binary_search(list, item):
 # keep track of low and high
 low = 0
 high = len(list) - 1

 while low <= high: # when we haven't narrowed down to one element
 mid = (low + high) // 2 # index of the middle element
 guess = list[mid] # value of the middle element

 if guess == item:
 return mid
 elif guess > item:
 # Guess is too high,
 # we narrow our search on the "left" part
 high = mid - 1
 else:
 # Guess is too low,
 # we narrow our search on the "right" part
 low = mid + 1

 # If we have anrrowed down to one element but still can not find the item
 # => Item is not in the list 
 return -1

Complexity

Time complexity: $O(\log n)$

Binary Vs Linear Search

Average Case

Binary Search Worst Case

Binary Search Best Case

Recursion

Tue, 06 Apr 2021 00:00:00 +0000

Every recursive function has two parts

base case: the function does not call itself again, so it will not go into an infinite loop
recursive case: the function calls itself

Sometimes recursion could be also re-written using loops. In fact, loops are sometimes better for performance. But recursion is used when it makes the solution clearer.

Examples

Count down

def countdown(i):
 print(i)
 if i <= 0: # base case
 return
 else: # recursive case
 countdown(i-1)

countdown(5)

Factorial

factorial(3) is defined as $3! = 3 \cdot 2 \cdot 1 = 3$

def factorial(x):
 # base case
 if x == 1:
 return 1 # 1! = 1
 else:
 return x * factorial(x-1)

factorial(3)

factorial(5)

Divide and Conquer

Divide and conquer (D&C) is a well-known recursive technique for solving problems. To solve a problem using D&C, there’re 2 steps:

Figure out the base case. This should be the simplest possible case.
Divide or decrease the problem until it becomes the base case.

Sum of Array

Let’s calculate the sum of an array using D&C.

Figure out the base case.

Base case is that the array is empty. In this situation, sum of this array is 0.

Decrease the problem until it becomes the base case.

sum = a[0] + a[1] + a[2]
 = a[0] + (a[1] + a[2])
 = a[0] + (a[1] + (a[2] + 0))

Example:

As we can see, in the 2nd and 3rd step, we’re passing a smaller (sub)array into the sum function. That is, we’re decreasing the size of our problem!

When we have figured out the two steps of D&C, the implementation is pretty simple:

def sum(arr):
 if not arr: # arr is empty
 return 0
 else:
 return arr[0] + sum(arr[1:])

When you’re writing a recursive function involving an array, the base case is often an empty array or an array with one element. If you’re stuck, try that first.

Count Number of Elements

def length(arr):
 if not arr:
 # base case: length of empty array is 0
 return 0

 # recursive case: 1 + length of the remaining subarray
 return 1 + length(arr[1:])

Find Maximum

def max(arr):
 if len(arr) == 2:
 return arr[0] if arr[0] > arr[1] else arr[1]
 return arr[0] if arr[0] > max(arr[1:]) else max(arr[1:])

Recursive Binary Search

def binary_search(arr, low, high, target):
 if low <= high:
 mid = (low + high) // 2
 if arr[mid] > target:
 # If element is smaller than mid, 
 # then it can only be present in left subarray
 return binary_search(arr, low, mid-1, target)
 elif arr[mid] == target:
 return mid
 else:
 # If element is greater than mid, 
 # then it can only be present in right subarray
 return binary_search(arr, mid+1, high, target)

 else: # Element is not present in the array
 return -1

Array and Linked List

Mon, 05 Apr 2021 00:00:00 +0000

TL;DR

	Array	Linked List
Store in memory	Elements are stored in contiguous locations in memory.	Elements can be stored anywhere in memory
Size	Fixed, must be specified at the time of initialization	Changeable, can grow/shrink by insertion/deletion
Access element	Random access $O(1)$	Sequence access $O(n)$
Insert element	$O(n)$	$O(1)$
Delete element	$O(n)$	$O(1)$

Storing In Memory

Array store elements in contiguous memory locations/blocks.

By Linked list, elements (also called nodes) can be stored anywhere in memory. Each node stores its next node’s address (link/pointer to next node).

Operations

Access

Array

Accessing an element in Array is simple. As array is essentially a block of continuous memory, we can directly access an element using index.

In other words, accessing an element in the array is independent of the size of the array. Hence, time complexity is $O(1)$.

Linked List

However, accessing a node in Linked list is not that easy. Since nodes are scattered out in memory, we have to traverse the linked list from the head node until we reach the target node.

The worst case is that our target node is at the tail of the linked list. In this case we have to traverse the whole linked list. Time complexity is thus O(n).

Insert

Array

Inserting an element in the Array is a pain.

For example, we want to insert 11 in the 3rd position of array [1, 3, 5, 7, 9]. We need to

allocate a new memory block of suitable size (in our case, 6)，
put new element 11 in the 3rd position, and copy other elements to the right place.

Linked list

Key of insertion is to maintain the order/sequence of nodes using links.

Insert in the middle

Let’s say we want to insert a new_node between left_node and right_node.

Before insertion:

left_node --> right_node

After insertion, the linked should look like:

left_node --> new_node --> right_node

Keeping this in mind, we have the idea of insertion:

new_node.next = left_node.next
left_node.next = new_node

Insert at head

By inserting a new node at the head of linked list, we need to specify the new head after the insertion.

Assume that we want to insert new_node before linked_list.head:

new_node.next = linked_list.head
linked_list.head = new_node

Insert at tail

By inserting a new node at the tail of linked list, we need to specify that the new node is the tail.

Assume that we want to insert new_node after tail_node.

Before insertion:

tail_node --> None

After insertion:

tail_node --> new_node --> None

Thus, the code of insertion should be:

# traverse linked list until reaching the tail node
tail_node.next = new_node
new_node.next = None

Delete

Array

Deleting of element from an array is similar to insertion. We need to allocate a new memory of suitable size, and copy remaining elements to the right place.

Linked list

Similar to insertion, we just need to maintain the order/sequence of nodes using links.

Insert a node in the middle

Let’s say we want to delete del_node between left_node and right_node.

Before deletion:

left_node --> del_node --> right_node

After deletion:

left_node --> right_node

Therefore, we just need to do the followings

left_node.next = right_node

Example:

Delete head node

Let’s say we want to delete linked_list.head. After deletion, we need to announce that the new head is linked_list.head.next.

linked_list.head = linked_list.head.next

Delete tail node

After deleting the tail node, the second last node becomes the new tail node.

Before deletion:

left_node --> tail_node --> None

After deletion:

left_node --> None

Therefore, we just need to do the followings

left_node.next = None

Array vs. Linked List

	Array	Linked List
Access element	Fast	Slow
Insert element	Slow	Fast
Delete element	Slow	Fast

By frequent retrieval/accessing, use Array
By frequent insertion/deletion, use Linked List

LinkedList in Python

Implement our own Linked List

To implement the linked list, we firstly need to implment a class for node, element of the list:

class Node:

 def __init__(self, data):
 self.data = data
 self.next = None

 def __repr__(self):
 return self.data

Then we implment the linked list:

class LinkedList:

 def __init__(self, data_list=None):
 self.head = None
 if data_list is not None:
 node = Node(data=data_list.pop(0))
 self.head = node
 for element in data_list:
 node.next = Node(data=element)
 node = node.next

 def __repr__(self):
 node = self.head
 nodes = list()
 while node is not None:
 nodes.append(node.data)
 node = node.next
 nodes.append("None")
 return " -> ".join(nodes)

 def __iter__(self):
 node = self.head
 while node is not None:
 yield node
 node = node.next

 def add_first(self, new_node):
 """
 Insert new_node at head
 """
 new_node.next = self.head
 self.head = new_node

 def add_last(self, new_node):
 """
 Insert new_node at tail
 """

 # If linked list is empty,
 # new node will be the only node in linked list,
 # i.e. head and tail node are the same
 if self.head is None:
 self.head = new_node
 return

 # If linked list is not empty:
 # 1. we need to traverse the whole list until we reach the current last node
 # 2. we add the new node as the next node of the current last node
 for current_node in self:
 pass
 current_node.next = new_node

 def add_after(self, target_node_data, new_node):
 """
 Insert new_node after the node whose data is target_node_data
 """
 if self.head is None:
 raise Exception("Linked list is empty.")

 for node in self:
 if node.data == target_node_data:
 new_node.next = node.next
 node.next = new_node
 return

 raise Exception(f"Node with data '{target_node_data}' not found.")

 def remove_node(self, target_node_data):
 """
 Remove node whose data is target_node_data
 """

 # If linked list is empty, then raise an exception
 if self.head is None:
 raise Exception("Linked list is empty.")

 # If the node to be removed is the current head,
 # then we want the next node in the list to become the new head
 if self.head.data == target_node_data:
 self.head = self.head.next
 return

 # If list is not empty and node to be removed is not the current head,
 # we traverse the list looking for the node to be removed.
 # If we find it, then we need to update its previous node to point to its next node
 previous_node = self.head
 for node in self:
 if node.data == target_node_data:
 previous_node.next = node.next
 return

 previous_node = node

 # If we traverse the whole list without finding the node to be removed,
 # then raise an exception
 raise Exception(f"Node with data '{target_node_data}' not found.")

Traverse

llist = LinkedList(["a", "b", "c", "d", "e"])
llist

a -> b -> c -> d -> e -> None

for node in llist:
 print(node)

a
b
c
d
e

Insert

llist = LinkedList()
llist

None

Insert at head:

llist.add_first(Node("a"))
llist

a -> None

Insert at tail:

llist.add_last(Node("b"))
llist

a -> b -> None

llist.add_last(Node("d"))
llist

a -> b -> d -> None

Insert in the middle:

llist.add_after("b", Node("c"))
llist

a -> b -> c -> d -> None

Remove

llist = LinkedList(["a", "b", "c", "d", "e"])
llist

a -> b -> c -> d -> e -> None

Remove head:

llist.remove_node("a")
llist

b -> c -> d -> e -> None

Remove tail:

llist.remove_node("e")
llist

b -> c -> d -> None

Remove node in the middle:

llist.remove_node("c")
llist

b -> d -> None

Double-ended Queue (Deque)

collections.deque uses an implementation of a linked list in which you can access, insert, or remove elements from the beginning or end of a list with constant 𝑂(1) performance.

Append/Remove element from the right side: append() / pop()
Append/Remove element from the left side: appendleft() / popleft()

Implment Queue using `deque`

For a queue, we use a First-In/First-Out (FIFO) approach. I.e., the first element inserted in the list is the first one to be retrieved.

queue = deque()
queue

deque([])

queue.append("Mary")
queue.append("John")
queue.append("Susan")
queue

deque(['Mary', 'John', 'Susan'])

Retrieve: (Order should be Mary --> John --> Susan)

queue.popleft()

'Mary'

queue.popleft()

'John'

queue.popleft()

'Susan'

Implment Stack using `deque`

For a stack, we use a Last-In/Fist-Out (LIFO) approach, meaning that the last element inserted in the list is the first to be retrieved.

queue = deque()
queue

deque([])

queue.append("Mary")
queue.append("John")
queue.append("Susan")
queue

deque(['Mary', 'John', 'Susan'])

Retrieve: (Order should be Susan --> John --> Mary)

queue.pop()

'Susan'

queue.pop()

'John'

queue.pop()

'Mary'

Reference

Linked Lists in Python: An Introduction

Hash Table

Thu, 08 Apr 2021 00:00:00 +0000

Hash Table

The Hash table data structure stores elements in key-value pairs where

Key - unique integer that is used for indexing the values
Value - data that are associated with keys.

Hash table is also called hash map, map, dictionary, and associative arrays.

Python has built-in hash tables, they’re called dictionary. E.g. we model a dictionary, where key and value correspond to fruit and price, respectively.

catalog = dict() # we use fruit name as key, and price as value
catalog["apple"] = 1.5
catalog["watermelon"] = 3.2
catalog["banana"] = 3

print(catalog)

{'apple': 1.5, 'watermelon': 3.5, 'milk': 3}

Hash Functions (Hashing)

In a hash table, a new index is processed using the key. And the value corresponding to that key is stored in this index. This process is called hashing.

Let $k$ be a key and $h(x)$ be a hash function. $h(k)$ will give us a new index in the table to store the value linked with $k$.

Requirements of a hash function:

Consistent. Every time we input the same key, we should get the same index.
Different keys should be mapped to different indexes.

Example

In the example above, let’s say for apple:

hash_function("apple") = 4

Therefore, the price of apple, 1.5, will be stored in table[4].

When we want to retrieve the price of apple (catalog.get("apple")), the hash function will tell us that its price is stored in the index 4 in table. So table[4] will be returned.

Hash Colliison

When the hash function generates the same index for multiple keys, there will be a conflict (what value to be stored in that index). This is called a hash collision.

We can resolve the hash collision using one of the following techniques.

Collision resolution by chaining
Open Addressing

Collision resolution by chaining

In chaining, if a hash function produces the same index for multiple elements, these elements are stored in the same index by using a doubly-linked list.

If j is the slot for multiple elements, it contains a pointer to the head of the list of elements.

Open Addressing

Linear Probing

In linear probing, collision is resolved by checking the next slot.

$$ h(k, i) = (h^{\prime}(k) + i) \mod m $$

where

$i = \{0 ,1, ...\}$
$h^{\prime}(k)$ is a new hash function.

Quadratic Probing

It works similar to linear probing but the spacing between the slots is increased (greater than one) by using the following relation.

$$ h(k, i) = (h^{\prime}(k) + c\_1 i + c\_2 i^2) \mod m $$

where

$i = \{0 ,1, ...\}$
$c\_1$ and $c\_2$ are positive auxiliary constants

Double Hashing

If a collision occurs after applying a hash function h(k), then another hash function is applied for finding the next slot.

$$ h(k, i) = (h\_1(k) + ih\_2(k)) \mod m $$

Avoid Hash Collision

To avoid hash collisions, we need

a low load factor
a good hash function

Load Factor

Load factor of a hash table is defined as

$$ \text{load factor} = \frac{\text{number of items in hash table}}{\text{total number of slots}} $$

For example

Having a load factor greater than 1 means you have more items than slots in your array. Once the load factor starts to grow, you need to add more slots to your hash table. This is called resizing. With a lower load factor, you’ll have fewer collisions, and your table will perform better.

Rule of thumb

Resize when load factor is greater than 0.7
Make an array that is twice the size.

Good Hash Function

A good hash function distributes values in the array evenly. It can reduce the number of collisions.

We have different methods to find a good hash function (Let $k$ be a key and m be the size of the hash table):

Division method
$$ h(k) = k \mod m $$
Multiplication Method
$$ h(k) = \lfloor m(kA \mod 1)\rfloor $$
- $\lfloor \rfloor$ gives the floor value
- $A \in (0, 1)$. $kA \mod 1$ gives the fractional part of $kA$. A suggested optimal choice for $A$ is $\frac{\sqrt{5} - 1}{2}$

Performance

Best case

The hash function maps keys evenly all over the hash table. In this case, hash table takes $O(1)$ for everything.

Worst case

The hash function maps all keys to the SAME index. In this case, hash table takes $O(n)$ for everything.

Comparison with Arrays and Linked Lists

	Hash table (average)	Hash table (worst)	Array	Linked list
Search	$O(1)$	$O(n)$	$O(1)$	$O(n)$
Insert	$O(1)$	$O(n)$	$O(n)$	$O(1)$
Delete	$O(1)$	$O(n)$	$O(n)$	$O(1)$

Reference

Selection Sort

Tue, 06 Apr 2021 00:00:00 +0000

TL;DR

Key idea: repeatedly selecting the smallest remaining item
Time complexity: $O(n^2)$

Idea

The idea of selection sort is pretty simple:

First, find the smallest item in the array and exchange it with the first entry
Then, find the next smallest item and exchange it with the sec- ond entry.
Continue in this way until the entire array is sorted.

This method is called selection sort because it works by repeatedly selecting the smallest remaining item.

Pseudocode

repeat (len(arr) - 1) times:
 set the first unsorted element as the current_minimum
 for each of the unsorted elements:
 if element < current_minimum:
 set element as new minimum
 swap minimum with first unsorted position

Complexity Analysis

Selection sort uses 􏰐$\sim N^{2}/2$ compares and $N$ exchanges to sort an array of length $N$.

Therefore, time complexity is $O(n^2)$.

Python Implementation

In-place Implementation

def selection_sort_in_place(arr):
 for i in range(len(arr)):
 print(f"i = {i}")
 print(f"Sorted part: {arr[:i]}, unsorted part: {arr[i:]}\n")

 # Set the first unsorted element as current minimum
 min_idx = i

 # Check the remaining part:
 # Is there an element which is smaller than the current minimum?
 # If yes, then set this element as the new minimum
 for j in range(i+1, len(arr)):
 if arr[j] < arr[min_idx]:
 min_idx = j

 # If the first unsorted element is not the minimum,
 # then swap it with the minimum
 if i != min_idx:
 arr[i], arr[min_idx] = arr[min_idx], arr[i]

 return arr

arr = [37, 10, 14, 29, 13]
print(selection_sort_in_place(arr))

i = 0
Sorted part: [], unsorted part: [37, 10, 14, 29, 13]

i = 1
Sorted part: [10], unsorted part: [37, 14, 29, 13]

i = 2
Sorted part: [10, 13], unsorted part: [14, 29, 37]

i = 3
Sorted part: [10, 13, 14], unsorted part: [29, 37]

i = 4
Sorted part: [10, 13, 14, 29], unsorted part: [37]

[10, 13, 14, 29, 37]

Non In-place Implementation

def find_smallest(arr):
 """
 Find the index of the smallest value in the array
 """
 smallest_val = arr[0] # stores the smallest value
 smallest_index = 0 # stores the index of the smallest value

 for i in range(1, len(arr)):
 if arr[i] < smallest_val:
 smallest_val = arr[i]
 smallest_index = i
 print(f"smallest val: {smallest_val}, smallest index: {smallest_index}")

 return smallest_index


def selection_sort(arr):
 new_arr = []
 for i in range(len(arr)):
 # Finds the smallest element in the array,
 # and adds it to the new array
 print(f"arr: {arr}")
 smallest = find_smallest(arr)
 new_arr.append(arr.pop(smallest))
 print(f"new_arr: {new_arr} \n")

 return new_arr

arr = [37, 10, 14, 29, 13]
print(f"sorted arr: {selection_sort(arr)}")

arr: [37, 10, 14, 29, 13]
smallest val: 10, smallest index: 1
new_arr: [10]

arr: [37, 14, 29, 13]
smallest val: 13, smallest index: 3
new_arr: [10, 13]

arr: [37, 14, 29]
smallest val: 14, smallest index: 1
new_arr: [10, 13, 14]

arr: [37, 29]
smallest val: 29, smallest index: 1
new_arr: [10, 13, 14, 29]

arr: [37]
smallest val: 37, smallest index: 0
new_arr: [10, 13, 14, 29, 37]

sorted arr: [10, 13, 14, 29, 37]

Quick Sort

Tue, 06 Apr 2021 00:00:00 +0000

TL;DR

Divide and Conquer (D&C)
Partition the array with the selected pivot: [elements <= pivot, pivot, elements > pivot]. Recursively apply quick sort on left and right sub-arrays.
Average time complexity: $O(n \log n)$

💡 Idea

Pick a pivot
Partition the array into two sub-arrays
- Elements less thant the pivot go to the left sub-array
- Elements greater thant the pivot go to the right sub-array
Call quick sort recursively on the two sub-arrays
Concat left sub-array, pivot, and right sub-array

Implementation

Quick sort can be implemented using Divide and Conquer (D&C).

Base case will be
- Empty array
- Array with just one element
We can just return those arrays as is – there’s nothing to sort.
Recursive case

Decrease the array to two sub-arrays, apply quick sort on them, until they are decreased to base case.

Non In-place Implementation

In this version, we choose the first element as pivot.

def quick_sort(array):
 # base case
 if len(array) < 2:
 return array
 else:
 pivot = array[0] # choose the first element as pivot

 # Sub-array of all the elements less than the pivot
 less = [i for i in array[1:] if i <= pivot]

 # Sub-array of all the elements greater than the pivot
 greater = [i for i in array[1:] if i > pivot]

 return quick_sort(less) + [pivot] + quick_sort(greater)

In-place Implementation

To implement in-place quick sort, we won’t create any new array. Instead, we just swap the elements.

Here we will choose the last element as our pivot. In order to partition the array, we shift the elements which are less than pivot to the left, and shift the elements which are greater than pivot to the right.

def swap(arr, i, j):
 """
 Swap arr[i] and arr[j]
 """
 arr[i], arr[j] = arr[j], arr[i]

def partition(arr, start, end):
 """
 Partition the array with pivot.
 After partitioning, the array should look like
 [elements <= pivot, pivot, elements > pivot]
 """
 pivot = arr[end]
 pivot_index = start

 for i in range(start, end):
 print(f"i={i}, arr: {arr}, pivot_index: {pivot_index}")
 if arr[i] < pivot:
 swap(arr, i, pivot_index)
 pivot_index += 1
 swap(arr, pivot_index, end)

 return pivot_index

arr = [7, 2, 1, 6, 8, 5, 3, 4]
partition(arr, 0, len(arr)-1)
arr

i=0, arr: [7, 2, 1, 6, 8, 5, 3, 4], pivot_index: 0
i=1, arr: [7, 2, 1, 6, 8, 5, 3, 4], pivot_index: 0
i=2, arr: [2, 7, 1, 6, 8, 5, 3, 4], pivot_index: 1
i=3, arr: [2, 1, 7, 6, 8, 5, 3, 4], pivot_index: 2
i=4, arr: [2, 1, 7, 6, 8, 5, 3, 4], pivot_index: 2
i=5, arr: [2, 1, 7, 6, 8, 5, 3, 4], pivot_index: 2
i=6, arr: [2, 1, 7, 6, 8, 5, 3, 4], pivot_index: 2
[2, 1, 3, 4, 8, 5, 7, 6]

With the implementation of partition(), in-place quick sort is simple:

def quick_sort_inplace(arr, start, end):
 # If start >= end, that means array is empty or contains only one element, which is the base case.
 # We do NOT need to sort the array, just keep it as it is.

 # If start < end, the array contains more than 1 element, which is the recursive case.
 if start < end:

 # Partition the array with pivot.
 pivot_index = partition(arr, start, end)

 # Recursively apply quick sort on the left sub-array ([elements <= pivot]) 
 quick_sort_inplace(arr, start, pivot_index-1)

 # Recursively apply quick sort on the right sub-array ([elements > pivot]) 
 quick_sort_inplace(arr, pivot_index+1, end)

arr = [7, 2, 1, 6, 8, 5, 3, 4]
quick_sort_inplace(arr, 0, len(arr)-1)
arr

[1, 2, 3, 4, 5, 6, 7, 8]

Complexity Analysis

Time Complexity

Quick sort is unique because its performance depends on the pivot choice.

Worst case

It occurs when the pivot element picked is either the greatest or the smallest element. This condition leads to the case in which the pivot element lies in an extreme end of the sorted array. One sub-array is always empty and another sub-array contains n - 1 elements. Thus, quicksort is called only on this sub-array.

Notice that we’re NOT splitting the array into two halves. Instead, one of the sub-arrays is always empty. So the call stack is really long. In this case, the stack size is $O(n)$.
Best case

The array is already sorted, and we always pick the middle element as the pivot.

In this case, the stack size is $O(\log n)$

On every level of the call stack, we “touch” $O(n)$ elements. So each level stack takes $O(n)$ time to complete.

In our best case, there’re $O(\log n)$ levels. Each level takes $O(n)$ time. Therefore, time of the entire algorithm is

$$ O(n) \cdot O(\log n) = O(n \log n). $$

In the worst case, there are $O(n)$ levels, so the algorithm will take $O(n) \cdot O(n) = O(n^2)$ time.

Actually, the best case is also the average case. If we always choose a random element in the array as the pivot, quicksort will complete in $O(n \log n)$ time on average.

Space Complexity

The space complexity for quicksort is $O(\log n)$

Reference

Quicksort algorithm 👍

Merge Sort

Tue, 06 Apr 2021 00:00:00 +0000

TL;DR

Merge sort use Divide and Conquer strategy
- Repeatedly divides the array into two subarrays until subarrays of size 1
- Merge two sorted array to achieve a bigger sorted array
Time complexity is $O(n \log n)$
Space complexity is $O(n)$ (non-inplace)

How Merge Sort Works?

Divide and Conquer Strategy

Merge Sort is one of the most popular sorting algorithms that is based on the principle of Divide and Conquer.

Using the Divide and Conquer technique, we divide a problem into subproblems. When the solution to each subproblem is ready, we “combine” the results from the subproblems to solve the main problem.

Suppose we had to sort an array arr. A subproblem would be to sort a sub-section of this array starting at index p and ending at index r, denoted as arr[p..r].

Divide

If q is the half-way point between p and r, then we can split the subarray arr[p..r] into two arrays arr[p..q] and arr[q+1, r].
Conquer

In the conquer step, we try to sort both the subarrays arr[p..q] and arr[q+1, r]. If we haven’t yet reached the base case, we again divide both these subarrays and try to sort them.
Combine

When the When the conquer step reaches the base step and we get two sorted subarrays arr[p..q] and arr[q+1, r] for array arr[p..r], we combine the results by creating a sorted array arr[p..r] from two sorted subarrays arr[p..q] and arr[q+1, r]

The MergeSort Algorithm

Repeatedly divides the array into two halves until we reach a stage where we try to perform MergeSort on a subarray of size 1 (the base case)
Combine the sorted arrays into larger sorted arrays until the whole array is merged.

Implementation

Merge: merge two sorted arrays to a larger sorted array

def merge(left_arr, right_arr, arr):
 """
 Merge left_arr and right_arr to arr.
 left_arr and right_arr are already sorted.
 After merging, arr should be sorted.
 """
 left_len = len(left_arr)
 right_len = len(right_arr)

 i = 0 # Mark the position of current smallest element in left_arr
 j = 0 # Mark the position of current smallest element in right_arr
 k = 0 # Mark the position to be filled in arr

 while (i < left_len and j < right_len): # left_arr and right_arr are not exhausted
 # We compare the current smallest elements of left_arr and right_arr,
 # fill arr with the smaller one
 if left_arr[i] <= right_arr[j]:
 arr[k] = left_arr[i]
 i += 1
 else:
 arr[k] = right_arr[j]
 j += 1

 k += 1

 if i > left_len - 1:
 # left_arr is exhausted.
 # Thus, we fill up the rest of arr with right_arr
 arr[k:] = right_arr[j:]
 if j > right_len - 1: \
 # right_arr is exhausted
 # Thus, we fill up the rest of arr with left_arr
 arr[k:] = left_arr[i:]

 return arr

arr = [2, 4, 1, 6, 8, 5, 3, 7]
left_arr = [1, 2, 4, 6]
right_arr = [3, 5, 7, 8]
merge(left_arr, right_arr, arr)

[1, 2, 3, 4, 5, 6, 7, 8]

Merge sort

def merge_sort(arr):
 print(arr)

 # Base case
 if len(arr) < 2:
 return arr

 # Divide
 mid = len(arr) // 2
 left_arr = arr[:mid]
 right_arr = arr[mid:]

 # Conquer
 merge_sort(left_arr)
 merge_sort(right_arr)

 # Combine
 merge(left_arr, right_arr, arr)
 print(f"Merge {left_arr} and {right_arr} --> {arr}\n")

 return arr

arr = [2, 4, 1, 6, 8, 5, 3, 7]
print(merge_sort(arr))

[2, 4, 1, 6, 8, 5, 3, 7]
[2, 4, 1, 6]
[2, 4]
[2]
[4]
Merge [2] and [4] --> [2, 4]

[1, 6]
[1]
[6]
Merge [1] and [6] --> [1, 6]

Merge [2, 4] and [1, 6] --> [1, 2, 4, 6]

[8, 5, 3, 7]
[8, 5]
[8]
[5]
Merge [8] and [5] --> [5, 8]

[3, 7]
[3]
[7]
Merge [3] and [7] --> [3, 7]

Merge [5, 8] and [3, 7] --> [3, 5, 7, 8]

Merge [1, 2, 4, 6] and [3, 5, 7, 8] --> [1, 2, 3, 4, 5, 6, 7, 8]

[1, 2, 3, 4, 5, 6, 7, 8]

Complexity Analysis

Time Complexity

Best/Average/Worst Case Complexity: $O(n \log n)$

Space Complexity

The space complexity of merge sort is $O(n)$.

Reference

Mergesort algorithm 👍

Merge Sort Algorithm 👍

Linked List

Thu, 15 Apr 2021 00:00:00 +0000

class ListNode:
 def __init__(self, val=0, next=None):
 self.val = val
 self.next = next

Draw It out!

To solve the linked list problems, the most important thing is to draw the linked list and the procedure out. The figure can help you to think and code.

2 Main Types of Problems

Modification of links
Concatenation of linked lists

To Be Noticed

Loops

When modifying links, it is easy to create loop uncarefully. To prevent this problem, draw the list out! With the help of figure, we can immediately notice and avoid loops.

Corner Cases

Common corner cases are

Linked list is empty
Linked list contains only one node

Techniques

Dummy Head

Add a dummy_head before head

dummy_head = ListNode(val=-1, next=head)

dummy_head.next is always the first nodes after all operations
Advantage of adding a dummy_head is that we can treat head as a normal node, which can help us to handle the corner cases easily (e.g. deleting head node)
Leetcode problems
- 25

Fast and Slow pointers

Since linked list does not support random indexing as array, we have to access nodes starting from head
We can use fast and slow pointers to simulate the random access, e.g.
- If we want to access the middle node of the linked list
  - fast and slow pointers start from head.
  - At each step, slow moves one step forward, fast moves two steps forward
  - When fast reaches the end, slow reaches the middle
- If we want to access the n-th node from the end
  - slow starts from head, fast starts from the n-th node
  - At each step, slow and fast move one step forward
  - When fast reaches the end, slow reaches n-th node from the end
Leetcode problems
- 141. Linked List Cycle
- 142. Linked List Cycle II

Recursion

Linked list has the nature of recursion. If we master the idea of recursion, the solution will be suprisingly clean and succinct
To apply recursion, we just need to consider three questions
- Base case: Unter which situation / When should the recursion be ended?
  - In linked list, base cases are empty list or list containing only one node
- Return value: What should be returned to the previous level?
- Goal of current level: What should be done in the current level?
Reference: 三道题套路解决递归问题
Pre-order traversal vs. post-orderrecursion
- In pre-order traversal, we image that the previous nodes are already processed (we don’t care how they are processed)
- In post-order traversal, we image that the nodes behind are already processed (we don’t care how they are processed)
- Example: Swap nodes in pairs
Example: 24. Swap Nodes in Pairs
Leetcode problems
- 24. Swap Nodes in Pairs
- 206. Reverse Linked List

Sort Nodes Alphabetically When Modifying Links or Concatenating Nodes

When modifying links or concatenating nodes, it’s easy to get lost or create loops uncarefully. A small trick to avoid these problems is to

Draw out how the list looks like before and after modification
Mark the order of nodes alphabetically

Example

We want to move node 3 to the front of node 2. We draw the “before and after” out:

Then we mark the order of nodes alphabetically

Algo | Haobin Tan

Big O Notation

Definition

Rule of Thumbs

Different steps get added

Constants do NOT matter

Different inputs $\Rightarrow$ different variables

Smaller terms do NOT matter

Shorthands

Summary of Complexity Classes

Reference

Big O Notation

Complete Beginner’s Guide to Big O Notation

Binary Search

Code

Complexity

Binary Vs Linear Search

Average Case

Binary Search Worst Case

Binary Search Best Case

Recursion

Examples

Count down

Factorial

Divide and Conquer

Sum of Array

Count Number of Elements

Find Maximum

Recursive Binary Search

Array and Linked List

TL;DR

Storing In Memory

Operations

Access

Array

Linked List

Insert

Array

Linked list

Delete

Array

Linked list

Array vs. Linked List

LinkedList in Python

Implement our own Linked List

Traverse

Insert

Remove

Double-ended Queue (Deque)

Implment Queue using deque

Implment Stack using deque

Reference

Hash Table

Hash Table

Hash Functions (Hashing)

Example

Hash Colliison

Collision resolution by chaining

Open Addressing

Linear Probing

Quadratic Probing

Double Hashing

Avoid Hash Collision

Load Factor

Good Hash Function

Performance

Comparison with Arrays and Linked Lists

Reference

Selection Sort

TL;DR

Idea

Pseudocode

Complexity Analysis

Python Implementation

In-place Implementation

Non In-place Implementation

Quick Sort

TL;DR

💡 Idea

Implementation

Implment Queue using `deque`

Implment Stack using `deque`