{"id":"dsa-complexity-sets-and-dicts","slug":"sets-and-dicts","title":"Sets and dicts for speed","text":"The last step removed hidden loops by remembering something as you go. Very often, what you need to remember is *which values you have seen*, and then everything depends on how fast you can ask \"have I seen this one?\"\n\n### `in` on a list and on a set\n\n`x in values` on a list compares `x` with the items one by one, from the start: O(n). A **set** keeps its values in a table, each at a place worked out from the value's **hash**, a number Python computes from the value. To answer `x in seen`, Python computes the hash of `x` and looks in that one place, so the answer takes O(1) on average, however big the set is. A dict finds its keys in the same way. (The worst case, when many values land in the same place, is O(n), and it's rare. The module on hash tables shows how the table works.)\n\n```pycon\n>>> numbers = [4, 8, 15, 16, 23, 42]\n>>> 23 in numbers\nTrue\n>>> lookup = set(numbers)\n>>> 23 in lookup\nTrue\n>>> 7 in lookup\nFalse\n```\n\nThe answers are the same; the cost isn't. Building `set(numbers)` costs O(n) time once, and the set takes O(n) extra memory: you **trade memory for speed**. In a loop over n values, `in` on a list makes the loop O(n²); on a set, O(n).\n\nOne trap: `if x in set(numbers):` inside the loop builds a new set in every round, which is O(n) again. Build the set once, before the loop.\n\n### Duplicates and the first repeated value\n\n`set(values)` drops repeats, so a list has a duplicate when its set is smaller than the list. To find *which* value repeats first, keep a set of the values seen so far:\n\n```python\ndef has_duplicate(values):\n    return len(set(values)) != len(values)\n\n\ndef first_repeat(values):\n    seen = set()\n    for value in values:\n        if value in seen:\n            return value\n        seen.add(value)\n    return None\n\n\nprint(has_duplicate([3, 1, 4, 1, 5]))\nprint(has_duplicate([2, 7, 1, 8]))\nprint(first_repeat([5, 3, 9, 3, 5]))\nprint(first_repeat([1, 2, 3]))\n```\n\n```output\nTrue\nFalse\n3\nNone\n```\n\nEach round does one lookup and one `add`, both O(1) on average, so `first_repeat` is O(n) time and O(n) memory. With `seen = []` and `append`, it gives the same answers in O(n²).\n\n### Counting with Counter\n\nHow often does each value appear? You counted with a dict in the dictionaries module. **`collections.Counter`** does that loop for you, in one O(n) pass. It's a dict from each value to its count, a value it hasn't seen counts as 0, and `most_common(k)` gives the k most common values with their counts, the most common first:\n\n```pycon\n>>> from collections import Counter\n>>> votes = Counter([\"tea\", \"coffee\", \"tea\", \"juice\", \"tea\", \"coffee\"])\n>>> votes[\"tea\"]\n3\n>>> votes[\"water\"]\n0\n>>> votes.most_common(2)\n[('tea', 3), ('coffee', 2)]\n```\n\n\"The k most common values\" is a classic interview question. Counting is O(n) on average. Then `most_common(k)` picks the top k of the m different values in O(m log k) (the module on sorting shows how), and without a k it sorts all of them, O(m log m).\n\n### Two values that add up to a target\n\nIs there a pair of values that adds up to `target`? Comparing every pair is O(n²). Turn the question around: for each value `x`, the partner it needs is `target - x`. If that partner is among the values seen so far, there's a pair:\n\n```python\ndef has_pair(values, target):\n    seen = set()\n    for x in values:\n        if target - x in seen:\n            return True\n        seen.add(x)\n    return False\n\n\nprint(has_pair([8, 3, 11, 5], 16))\nprint(has_pair([8, 3, 11, 5], 10))\nprint(has_pair([5], 10))\n```\n\n```output\nTrue\nFalse\nFalse\n```\n\n11 and 5 make 16, and no two values make 10. The order of the two lines in the loop matters: the check comes before the `add`, so a value never pairs with itself, and the lone 5 doesn't count as 5 + 5.\n\nWhen you need *where* the partner is, not just whether it exists, use a dict instead of a set: map each value seen to its index, `positions[x] = i`, and `positions[target - x]` gives the partner's index. That's the classic interview question \"two sum\".\n\nTo have each index together with its value, loop with **`enumerate`**, in place of `for i in range(len(values))` and `values[i]`:\n\n```pycon\n>>> for i, x in enumerate([8, 3, 11]):\n...     print(i, x)\n...\n0 8\n1 3\n2 11\n```\n","task":"A shop sent out some orders, and `missing` lists the ones that never arrived, in the order they were sent. Fill in the blank so that each `not in` check is O(1): the hidden test has 200,000 orders. The call `print(missing([101, 102, 103, 104, 105], [104, 101, 105]))` should print:\n\n```\n[102, 103]\n```\n","kind":"fill","seed":"def missing(sent, received):\n    received_set = [[set(received)]]\n    result = []\n    for order in sent:\n        if order not in received_set:\n            result.append(order)\n    return result\n\n\nif __name__ == \"__main__\":\n    print(missing([101, 102, 103, 104, 105], [104, 101, 105]))\n","solution":"def missing(sent, received):\n    received_set = set(received)\n    result = []\n    for order in sent:\n        if order not in received_set:\n            result.append(order)\n    return result\n\n\nif __name__ == \"__main__\":\n    print(missing([101, 102, 103, 104, 105], [104, 101, 105]))\n","hints":["`order not in received` would look through the whole list of received orders for every order sent: O(n²).","Turn the list into a set once, before the loop, so each lookup is O(1) on average: `set(received)`."],"tests":[{"name":"print(missing([101, 102, 103, 104, 105], [104, 101, 105]))","stdin":"","call":"print(missing([101, 102, 103, 104, 105], [104, 101, 105]))\n","expect":"[102, 103]","hidden":false},{"name":"missing([1, 2, 3], [])","stdin":"","call":"missing([1, 2, 3], [])\n","expect":"[1, 2, 3]","hidden":true,"returns":true},{"name":"missing([7, 8], [8, 7, 9])","stdin":"","call":"missing([7, 8], [8, 7, 9])\n","expect":"[]","hidden":true,"returns":true},{"name":"len(missing(list(range(200_000)), list(range(0, 400_000, 2))))","stdin":"","call":"len(missing(list(range(200_000)), list(range(0, 400_000, 2))))\n","expect":"100000","hidden":true,"big":true,"returns":true}],"require":[],"forbid":[],"more":[{"task":"Write `two_sum(numbers, target)`: return a tuple `(i, j)` with the indexes of two values that add up to `target`, with `i < j`, or `None` if no two values do. A value can't pair with itself. In the tests, the values are all different, and at most one pair adds up to the target. It must be O(n): the hidden test has 200,000 values.\n\nFor example, `two_sum([7, 2, 13, 11], 24)` returns (13 + 11):\n\n```\n(2, 3)\n```\n","kind":"code","seed":"def two_sum(numbers, target):\n    pass\n\n\nif __name__ == \"__main__\":\n    print(two_sum([7, 2, 13, 11], 24))\n","solution":"def two_sum(numbers, target):\n    positions = {}\n    for j, x in enumerate(numbers):\n        if target - x in positions:\n            return (positions[target - x], j)\n        positions[x] = j\n    return None\n\n\nif __name__ == \"__main__\":\n    print(two_sum([7, 2, 13, 11], 24))\n","hints":["Use a dict that maps each value seen so far to its index. For each value `x` at index `j`, its partner is `target - x`.","`for j, x in enumerate(numbers):` gives each index with its value. If `target - x` is a key of the dict, its index comes first: return `(positions[target - x], j)`. Otherwise add `x` to the dict.","Check before you add `x`, so a value doesn't pair with itself: `two_sum([5, 1, 4], 10)` must give `None`."],"tests":[{"name":"two_sum([7, 2, 13, 11], 24)","stdin":"","call":"two_sum([7, 2, 13, 11], 24)\n","expect":"(2, 3)","hidden":false,"returns":true},{"name":"two_sum([3, 9], 12)","stdin":"","call":"two_sum([3, 9], 12)\n","expect":"(0, 1)","hidden":true,"returns":true},{"name":"print(two_sum([4, 6, 1], 20))","stdin":"","call":"print(two_sum([4, 6, 1], 20))\n","expect":"None","hidden":true},{"name":"two_sum([-5, 10, 0, 8], 3)","stdin":"","call":"two_sum([-5, 10, 0, 8], 3)\n","expect":"(0, 3)","hidden":true,"returns":true},{"name":"print(two_sum([5, 1, 4], 10))","stdin":"","call":"print(two_sum([5, 1, 4], 10))\n","expect":"None","hidden":true},{"name":"print(two_sum([], 1))","stdin":"","call":"print(two_sum([], 1))\n","expect":"None","hidden":true},{"name":"two_sum([2 * i for i in range(200_000)] + [7], 400_005)","stdin":"","call":"two_sum([2 * i for i in range(200_000)] + [7], 400_005)\n","expect":"(199999, 200000)","hidden":true,"big":true,"returns":true}],"require":[],"forbid":[]},{"task":"Write `longest_streak(values)`: the length of the longest run of consecutive whole numbers that are all in the list, such as 4, 5, 6, 7. The values can be in any order and can repeat. An empty list gives 0. It must be O(n): the hidden test has 200,000 values that form one long run, and counting up from every value of it is O(n²).\n\nFor example, `longest_streak([10, 4, 20, 1, 3, 2, 11])` returns (1, 2, 3 and 4):\n\n```\n4\n```\n","kind":"code","seed":"def longest_streak(values):\n    pass\n\n\nif __name__ == \"__main__\":\n    print(longest_streak([10, 4, 20, 1, 3, 2, 11]))\n","solution":"def longest_streak(values):\n    present = set(values)\n    best = 0\n    for value in present:\n        if value - 1 not in present:\n            length = 1\n            while value + length in present:\n                length += 1\n            best = max(best, length)\n    return best\n\n\nif __name__ == \"__main__\":\n    print(longest_streak([10, 4, 20, 1, 3, 2, 11]))\n","hints":["Put the values in a set, so that \"is `value + 1` in the list?\" is an O(1) question.","From a value, count upward while `value + length` is in the set. But counting from every value of a run of 200,000 numbers is O(n²) again.","Only start counting at the first value of a run: a value whose `value - 1` is not in the set. Then every value is counted at most twice in all."],"tests":[{"name":"longest_streak([10, 4, 20, 1, 3, 2, 11])","stdin":"","call":"longest_streak([10, 4, 20, 1, 3, 2, 11])\n","expect":"4","hidden":false,"returns":true},{"name":"longest_streak([])","stdin":"","call":"longest_streak([])\n","expect":"0","hidden":true,"returns":true},{"name":"longest_streak([5, 5, 5])","stdin":"","call":"longest_streak([5, 5, 5])\n","expect":"1","hidden":true,"returns":true},{"name":"longest_streak([-2, 0, -1, 7, 1])","stdin":"","call":"longest_streak([-2, 0, -1, 7, 1])\n","expect":"4","hidden":true,"returns":true},{"name":"longest_streak([9, 1, 5])","stdin":"","call":"longest_streak([9, 1, 5])\n","expect":"1","hidden":true,"returns":true},{"name":"longest_streak([6, 8, 7, 7, 9, 2, 3])","stdin":"","call":"longest_streak([6, 8, 7, 7, 9, 2, 3])\n","expect":"4","hidden":true,"returns":true},{"name":"longest_streak(list(range(200_000, 0, -1)))","stdin":"","call":"longest_streak(list(range(200_000, 0, -1)))\n","expect":"200000","hidden":true,"big":true,"returns":true}],"require":[],"forbid":[]}]}