Orders of Growth

Orders of Growth

We have seen more than one algorithm that performed the same task. How do we choose which one is best?

We could just time the two algorithms but there are some weaknesses with this approach. What input do we choose? The algorithms might perform faster or slower depending on the input. What if one algorithm is faster on one input and the other faster on another?

Instead, computer scientists use an asymptotic approach in which we look at which algorithm is more rapidly taking longer as the problem size increases. To get a feel for this approach we will try out two algorithms for sorting a deck of cards. I will give you index cards with numbers written on them and you will sort them with each of the algorithms for decks of size 4, 8, 16, and 32 cards and time your results. You will complete this task in Activity 3.

Let's take a close look at the two sorting algorithms.

The first algorithm is Selection Sort:

Selection Sort

You will use three positions for stacks of cards: source stack, destination stack, and the discard stack.

Initially you should put all the cards, face down, on the source stack, with the other two positions empty. Now do the following steps repeatedly:

Take the top card off the source stack and put it face-up on the destination stack.
If that makes the source stack empty, you are done. The destination stack is in numerical order.
Otherwise, do the following steps repeatedly until the source stack is empty:
1. Take the card off the source stack and compare it with the top of the destination stack.
2. If the source card has a larger number,
  1. Take the card on the top of the destination stack and put it face down on the discard stack.
  2. Put the card you took from the source stack face up on the destination stack.
3. Otherwise, put the card from the source stack face down on the discard stack.
Slide the discard stack over into the source position, and start again with step 1.

The second algorithm is Merge Sort:

Merge Sort

Lay out the cards face down in a long row. We will consider these to be the initial source "stacks" of cards, even though there is only one card per stack. The merge sorts works by progressively merging pairs of stacks so that there are fewer stacks but each is larger; at the end, there will be a single large stack of cards.

Repeat the following steps until there is a single stack of cards:

Merge the first two face-down stacks of cards using Merge algorithm given below.
As long as there are a least two face-down stacks, repeat the merging with the next two stacks.
Flip each face-up stack over.

Merging

You will have the two sorted stacks of cards to merge side by side, face down. You will be producing the result stack above the other two, face up.

Take the top card off of each source stack -- one in your left hand and one in your right hand. Now do the following repeatedly, until all the cards are on the destination stack:

Compare the two cards you are holding.
Place the one with the larger number on it onto the destination stack, face-up.
With the hand you just emptied, pick up the next card from the corresponding source stack and go back to step 1. If there is no next card in the empty hand's stack because that stack is empty, put the other card you are holding on the destination stack face-up and continue flipping the rest of the cards over onto the destination stack.

When you time your trials, you should see that the growth rate of one of your sorts is much greater than the other. We should be able to determine the rate of growth without actually timing the sorts.

We will consider the number of "steps" it takes for each of the algorithms when sorting n cards.

In Selection Sort we will consider a pass to be one time through steps 1 through 4.

Number of cards handled on pass 1 : all n cards handled once or twice
Number of cards handled on pass 2 : n-1 cards handled once or twice
Number of cards handled on pass 3 : n-2 cards handled once or twice
...
Number of cards handled on pass n : 1 card handled

So the number of cards handled is 1 + 2 + 3 + ...+ n. How big is that sum? Well, it is easy to see that the sum is less than n², because each number in the sum is no bigger than n so the sum is no bigger than n + n + n + ... + n = n². We can also see that the sum is at least n²/4, because there are n/2 numbers which are larger than n/2. So the sum is somewhere between n²/4 and n², both multiples of n². From this we can determine that the growth rate will be roughly quadratic (a parabola) and we will refer to this growth rate as "big theta of n squared" or Q(n²). This means that with all but a finite number of exceptions, the time will lie between two multiples of n².
Note: Sometimes instead of big theta we use big-Oh which means that the growth rate is bounded above by a multiple. So this sum would also be O(n²).

Now let's consider Merge Sort.

On pass 1 : all n cards handled to merge n stacks to n/2 stacks
On pass 2 : all n cards handled to merge n/2 stacks to n/4 stacks
On pass 3 : all n cards handled to merge n/4 stacks to n/8 stacks
...
On last pass : all n cards handled to merge 2 stacks to 1 stack

How many passes will we need? Another way to answer this is to go in the opposite direction and ask how many times does 1 need to double to reach n? Answer --- log₂ n. Therefore the order of growth for merge sort is Q(n log n). This growth rate is faster than linear but slower than quadratic. This should confirm the results from the time trials.

A Haskell example

We will look at three different algorithms for computing bⁿ : a linear recursive algorithm, a tail recursive algorithm, and a third algorithm using stronger recursion.

Linear Recursion:

power :: Int -> Int -> Int
power b n = if n == 0
then 1
else (power b (n - 1)) * b

Both the time and space (memory use) is proportional to the depth of the recursion (number of winding steps), which is n. Therefore the time and space complexities are both O(n).

Tail Recursion:

powerTail :: Int -> Int -> Int
powerTail b n = pTail n 1 where
    pTail n result =
        if n == 0
            then result
            else pTail (n-1) (result * b)

The time is proportional to the depth of the recursion, which is n. Therefore the time is O(n). The space complexity, however, will always be constant regardless of the values of the input. Therefore the space complexity is O(1).

Strong Recursion:

For this version we will observe that if n is even then bⁿ= (b^n/2)²

power2 :: Int -> Int -> Int
power2 b n = if n == 0 then 1
    else if even n
    then square (power2 b (n `div` 2))
    else (power2 b (n - 1)) * b

The time is proportional to the depth of the recursion. Just as in Merge Sort, if n is a power of 2, it will take log n steps to get to the base case. So, the depth of the recursion is roughly log n. Therefore the time and space complexities are both O(log n).