{"id":"dsa-hash-rehashing","slug":"rehashing","title":"Load factor and rehashing","text":"The table of the last step keeps the number of buckets it started with. Put 100,000 pairs into 8 buckets and each chain holds 12,500 of them: every `get` is a linear search again, and adding the 100,000 keys takes more than half a billion comparisons.\n\n### The load factor\n\nThe **load factor** is the number of pairs divided by the number of buckets, `count / len(buckets)`: the average length of a chain. A hash table keeps it small by **growing**. When the load factor passes a limit, often 0.75, the table makes twice as many buckets and moves every pair into them. That's called **rehashing**.\n\nEvery pair has to move, because a key's bucket depends on the number of buckets. Four keys that share bucket 1 of 4 split up when there are 8:\n\n```pycon\n>>> keys = [5, 9, 13, 17]\n>>> [key % 4 for key in keys]\n[1, 1, 1, 1]\n>>> [key % 8 for key in keys]\n[5, 1, 5, 1]\n```\n\nSo growing takes three moves:\n\n1. Keep the old list of buckets in a variable.\n2. Make a new list with twice as many empty buckets.\n3. Go through every pair of every old bucket and append it to its bucket in the new list, worked out with the new number of buckets.\n\nA common mistake is to append empty buckets to the old list and stop there. The table gets bigger, but the old pairs stay where the old size put them: key 13 sits in bucket 1, while `get(13)` now looks in bucket 5 and returns `None`, though the key is still in the table.\n\n### Why adding stays O(1)\n\nA rebuild moves every pair, so the add that triggers it costs O(n). But rebuilds get rarer as the table grows: after one, the table has room for as many new pairs as it already holds. This program counts the pairs moved while n keys go into a table of 8 buckets that grows when the load factor passes 0.75, once by doubling and once by adding 8 buckets each time:\n\n```python\ndef moves_while_adding(n, double):\n    buckets = 8\n    moves = 0\n    for count in range(1, n + 1):\n        if count / buckets > 0.75:\n            moves += count\n            if double:\n                buckets *= 2\n            else:\n                buckets += 8\n    return moves\n\n\nfor n in [1000, 10_000, 100_000]:\n    print(n, moves_while_adding(n, True), moves_while_adding(n, False))\n```\n\n```output\n1000 1538 83332\n10000 12293 8333332\n100000 196617 833333332\n```\n\nWith doubling, the moves never get much past twice the number of keys. The rebuilds move 7, 13, 25, 49 pairs and so on, each about twice the one before, so all of them together come to about twice the last one, and the last one is at most n. So n adds cost O(n) in all: **O(1) amortized** per add. Now and then one add is slow, but averaged over all of them each add is O(1), for the same reason that `list.append` is. Adding a fixed 8 buckets instead rebuilds every few adds, and the moves grow like n²: over 800 million for 100,000 keys.\n\n### Python's dict: open addressing\n\nPython's dict and set grow the same way, but they don't keep chains. All the keys sit in one list of **slots**. When a key's slot is taken, the table tries another slot, and another, until it finds the key or an empty slot. That's **open addressing**. The simplest kind, **linear probing**, tries the next slot, and wraps around to the start after the last one:\n\n```python\nslots = [None] * 8\nfor key in [3, 11, 19, 6]:\n    i = key % 8\n    while slots[i] is not None:\n        i = (i + 1) % 8\n    slots[i] = key\nprint(slots)\n```\n\n```output\n[None, None, None, 3, 11, 19, 6, None]\n```\n\n3, 11 and 19 all start at slot 3, so 11 moves on to slot 4 and 19 to slot 5. A search for 19 starts at slot 3 and steps along until it finds it. A search for 27 steps from slot 3 over 3, 11, 19 and 6 to the empty slot 7: an empty slot means the key isn't there. So the table must never fill up, or a search for a missing key would never end, and long runs of taken slots make every search slow. Python's dict grows when about two-thirds of its slots are taken, and instead of the next slot it jumps around in a scrambled order, which keeps the runs short.\n","task":"Make `HashMap` grow. `put` already counts the new pairs in `self.count`. After it adds a new pair, when the load factor `self.count / len(self.buckets)` is more than 0.75, it should call `_grow`, which doubles the number of buckets and moves every pair to its new bucket. The table starts with 8 buckets, so the 7th pair makes it grow to 16. The hidden test puts 50,000 pairs in the table, which is only fast when the table doubles each time. This program:\n\n```python\ntable = HashMap()\nfor key in range(6):\n    table.put(key, key * 10)\nprint(len(table.buckets))\ntable.put(6, 60)\nprint(len(table.buckets), table.get(3), table.get(6))\n```\n\nprints:\n\n```\n8\n16 30 60\n```\n","kind":"code","seed":"class HashMap:\n    def __init__(self):\n        self.buckets = [[] for i in range(8)]\n        self.count = 0\n\n    def _index(self, key):\n        return hash(key) % len(self.buckets)\n\n    def put(self, key, value):\n        bucket = self.buckets[self._index(key)]\n        for pair in bucket:\n            if pair[0] == key:\n                pair[1] = value\n                return\n        bucket.append([key, value])\n        self.count += 1\n        # grow here when the load factor is more than 0.75\n\n    def get(self, key):\n        for pair in self.buckets[self._index(key)]:\n            if pair[0] == key:\n                return pair[1]\n        return None\n\n    def _grow(self):\n        pass\n\n\nif __name__ == \"__main__\":\n    table = HashMap()\n    for key in range(6):\n        table.put(key, key * 10)\n    print(len(table.buckets))\n    table.put(6, 60)\n    print(len(table.buckets), table.get(3), table.get(6))\n","solution":"class HashMap:\n    def __init__(self):\n        self.buckets = [[] for i in range(8)]\n        self.count = 0\n\n    def _index(self, key):\n        return hash(key) % len(self.buckets)\n\n    def put(self, key, value):\n        bucket = self.buckets[self._index(key)]\n        for pair in bucket:\n            if pair[0] == key:\n                pair[1] = value\n                return\n        bucket.append([key, value])\n        self.count += 1\n        if self.count / len(self.buckets) > 0.75:\n            self._grow()\n\n    def get(self, key):\n        for pair in self.buckets[self._index(key)]:\n            if pair[0] == key:\n                return pair[1]\n        return None\n\n    def _grow(self):\n        old = self.buckets\n        self.buckets = [[] for i in range(len(old) * 2)]\n        for bucket in old:\n            for pair in bucket:\n                self.buckets[self._index(pair[0])].append(pair)\n\n\nif __name__ == \"__main__\":\n    table = HashMap()\n    for key in range(6):\n        table.put(key, key * 10)\n    print(len(table.buckets))\n    table.put(6, 60)\n    print(len(table.buckets), table.get(3), table.get(6))\n","hints":["In `put`, after `self.count += 1`, check `self.count / len(self.buckets) > 0.75`, and call `self._grow()` when it's true.","In `_grow`, keep the old buckets in a variable, `old = self.buckets`, and set `self.buckets` to a list of twice as many empty buckets.","Then go through every bucket of `old` and every pair in it, and append the pair to `self.buckets[self._index(pair[0])]`: `_index` uses the new number of buckets now. Growing by a fixed number of buckets instead of doubling gives the right answers, but too slowly for the big test."],"tests":[{"name":"Test 1","stdin":"","call":"table = HashMap()\nfor key in range(6):\n    table.put(key, key * 10)\nprint(len(table.buckets))\ntable.put(6, 60)\nprint(len(table.buckets), table.get(3), table.get(6))\n","expect":"8\n16 30 60","hidden":false},{"name":"Test 2","stdin":"","call":"t = HashMap()\nkeys = [5, 13, 21, 29, 37, 45, 53]\nfor key in keys:\n    t.put(key, key + 1)\nprint([t.get(key) for key in keys], t.get(61), len(t.buckets))\n","expect":"[6, 14, 22, 30, 38, 46, 54] None 16","hidden":true},{"name":"Test 3","stdin":"","call":"t = HashMap()\nfor i in range(10):\n    t.put(\"x\", i)\nprint(t.get(\"x\"), t.count, len(t.buckets))\n","expect":"9 1 8","hidden":true},{"name":"Test 4","stdin":"","call":"t = HashMap()\nwords = [\"ant\", \"bee\", \"cat\", \"dog\", \"eel\", \"fox\", \"gnu\", \"hen\"]\nfor word in words:\n    t.put(word, len(word) * 2)\nt.put(\"cat\", 0)\nprint([t.get(word) for word in words], t.get(\"owl\"), t.count)\n","expect":"[6, 6, 0, 6, 6, 6, 6, 6] None 8","hidden":true},{"name":"Test 5","stdin":"","call":"t = HashMap()\nfor i in range(50_000):\n    t.put(i * 7, i)\nprint(t.get(7 * 49_999), t.get(70), t.get(5), t.count)\n","expect":"49999 10 None 50000","hidden":true,"big":true}],"require":[],"forbid":[{"pattern":"[{}]|\\bset\\s*\\(|\\bdict\\s*\\(","message":"Grow your own buckets, without Python's `set` or `dict`."}],"more":[{"task":"What does this program print? It follows a table that starts with 4 buckets and doubles when its load factor is more than 0.75. Type each line it prints.\n","kind":"predict","seed":"buckets = 4\ncount = 0\nfor key in range(10):\n    count += 1\n    if count / buckets > 0.75:\n        buckets *= 2\n        print(\"grow at\", count, \"to\", buckets)\nprint(buckets, count / buckets)\n","solution":"buckets = 4\ncount = 0\nfor key in range(10):\n    count += 1\n    if count / buckets > 0.75:\n        buckets *= 2\n        print(\"grow at\", count, \"to\", buckets)\nprint(buckets, count / buckets)\n","hints":["Work out `count / buckets` after each key. The table grows only when the result is more than 0.75: exactly 0.75 isn't.","With 4 buckets, the 4th key gives 1.0, so the table grows to 8. With 8 buckets, which count is the first to give more than 0.75?","After the last key, `count` is 10. Divide it by the final number of buckets."],"tests":[{"name":"Output","stdin":"","expect":"grow at 4 to 8\ngrow at 7 to 16\n16 0.625","hidden":false}],"require":[],"forbid":[],"lines":["grow at 4 to 8","grow at 7 to 16","16 0.625"]},{"task":"Write a class `OpenSet`: a set that keeps its keys with open addressing and linear probing, as the last example of the lesson does. The keys sit in one list, `self.slots`, where `None` marks an empty slot (a key is never `None`), and `self.count` is the number of keys. The seed starts with 4 slots and a count of 0.\n\n- The search for a key starts at slot `hash(key) % len(self.slots)` and moves one slot to the right at a time, back to slot 0 after the last one, until it reaches the key or an empty slot.\n- `add(key)` returns `False` when the search reaches the key. Otherwise it stores the key in the empty slot the search reached, adds one to `self.count`, and returns `True`.\n- `contains(key)` returns whether the key is in the set.\n- Like Python's dict, the table grows when more than two-thirds of its slots are taken. After `add` stores a new key, if `self.count * 3 > len(self.slots) * 2`, the table makes twice as many empty slots and stores every key again: a key's first slot depends on the number of slots.\n\nThe hidden test adds 100,000 keys and looks up 100,000: `key in self.slots` would search the whole list every time.\n\nFor example, this program:\n\n```python\ns = OpenSet()\nprint(s.add(7), s.add(15), s.add(7))\nprint(s.slots)\nprint(s.contains(15), s.contains(23))\n```\n\nprints (7 and 15 both start at slot 3, the last one, so 15 goes around to slot 0; a search for 23 starts at slot 3, goes on to slot 0 and stops at the empty slot 1):\n\n```\nTrue True False\n[15, None, None, 7]\nTrue False\n```\n","kind":"code","seed":"class OpenSet:\n    def __init__(self):\n        self.slots = [None] * 4\n        self.count = 0\n\n    def add(self, key):\n        pass\n\n    def contains(self, key):\n        pass\n\n\nif __name__ == \"__main__\":\n    s = OpenSet()\n    print(s.add(7), s.add(15), s.add(7))\n    print(s.slots)\n    print(s.contains(15), s.contains(23))\n","solution":"class OpenSet:\n    def __init__(self):\n        self.slots = [None] * 4\n        self.count = 0\n\n    def _find(self, key):\n        i = hash(key) % len(self.slots)\n        while self.slots[i] is not None and self.slots[i] != key:\n            i = (i + 1) % len(self.slots)\n        return i\n\n    def add(self, key):\n        i = self._find(key)\n        if self.slots[i] == key:\n            return False\n        self.slots[i] = key\n        self.count += 1\n        if self.count * 3 > len(self.slots) * 2:\n            old = self.slots\n            self.slots = [None] * (len(old) * 2)\n            for k in old:\n                if k is not None:\n                    self.slots[self._find(k)] = k\n        return True\n\n    def contains(self, key):\n        return self.slots[self._find(key)] == key\n\n\nif __name__ == \"__main__\":\n    s = OpenSet()\n    print(s.add(7), s.add(15), s.add(7))\n    print(s.slots)\n    print(s.contains(15), s.contains(23))\n","hints":["Write a helper `_find(key)` that returns the slot holding the key, or the empty slot where the search stopped: start at `hash(key) % len(self.slots)`, and while that slot isn't `None` and isn't the key, move on with `i = (i + 1) % len(self.slots)`.","`add`: if the slot `_find` returns holds the key, return `False`. Otherwise store the key there and add one to `self.count`. `contains`: the slot `_find` returns holds either the key or `None`.","When `self.count * 3 > len(self.slots) * 2`, keep the old slots in a variable, make `[None] * (len(old) * 2)`, and store each old key that isn't `None` in the slot `_find` gives it in the new list."],"tests":[{"name":"Test 1","stdin":"","call":"s = OpenSet()\nprint(s.add(7), s.add(15), s.add(7))\nprint(s.slots)\nprint(s.contains(15), s.contains(23))\n","expect":"True True False\n[15, None, None, 7]\nTrue False","hidden":false},{"name":"Test 2","stdin":"","call":"s = OpenSet()\nprint(s.add(-5), s.add(3), s.add(-5), s.contains(3), s.contains(7))\nprint(s.slots)\n","expect":"True True False True False\n[3, None, None, -5]","hidden":true},{"name":"Test 3","stdin":"","call":"s = OpenSet()\nprint(s.add(7), s.add(15), s.add(15), s.count)\nprint(s.slots)\nprint(s.contains(15), s.contains(23))\n","expect":"True True False 2\n[15, None, None, 7]\nTrue False","hidden":true},{"name":"Test 4","stdin":"","call":"s = OpenSet()\nfor key in [0, 4, 1]:\n    s.add(key)\nprint(s.slots)\nprint(s.contains(4), s.contains(8), s.count)\n","expect":"[0, 1, None, None, 4, None, None, None]\nTrue False 3","hidden":true},{"name":"Test 5","stdin":"","call":"s = OpenSet()\nsizes = []\nfor key in range(12):\n    s.add(key * 5)\n    sizes.append(len(s.slots))\nprint(sizes)\nprint(all(s.contains(key * 5) for key in range(12)), s.contains(7))\n","expect":"[4, 4, 8, 8, 8, 16, 16, 16, 16, 16, 32, 32]\nTrue False","hidden":true},{"name":"Test 6","stdin":"","call":"s = OpenSet()\nfor word in [\"oak\", \"elm\", \"oak\", \"ash\", \"fir\", \"yew\", \"elm\"]:\n    s.add(word)\nprint(s.count, s.contains(\"ash\"), s.contains(\"pine\"), s.add(\"yew\"))\n","expect":"5 True False False","hidden":true},{"name":"Test 7","stdin":"","call":"s = OpenSet()\nadded = sum(1 for i in range(100_000) if s.add(i * 7))\nprint(added, sum(1 for i in range(100_000) if s.contains(i * 3)), len(s.slots))\n","expect":"100000 14286 262144","hidden":true,"big":true}],"require":[],"forbid":[{"pattern":"[{}]|\\bset\\s*\\(|\\bdict\\s*\\(","message":"Keep the keys in your own slots, without Python's `set` or `dict`."}]}]}