Gigi Sayfan
16 likes
Read Time: 12 min
PythonProgramming Fundamentals
Python is a fantastic programming language. It is also known for being pretty slow, due mostly to its enormous flexibility and dynamic features. For many applications and domains, it is not a problem due to their requirements and various optimization techniques. It is less known that Python object graphs (nested dictionaries of lists and tuples and primitive types) take a significant amount of memory. This can be a much more severe limiting factor due to its effects on caching, virtual memory, multi-tenancy with other programs, and in general exhausting the available memory, which is a scarce and expensive resource.
It turns out that it is not difficult to figure out how much memory is actually consumed. In this article, I'll walk you through the intricacies of a Python object's memory management and show how to measure the consumed memory accurately.
In this article, I focus solely on CPython—the primary implementation of the Python programming language. The experiments and conclusions here don't apply to other Python implementations like IronPython, Jython, and PyPy.
Depending on the Python version, the numbers are sometimes a little different (especially for strings, which are always Unicode), but the concepts are the same. In my case, am using Python 3.10.
As of 1st January 2020, Python 2 is no longer supported, and you should have already upgraded to Python 3.
Hands-On Exploration of Python Memory Usage
First, let's explore a little bit and get a concrete sense of the actual memory usage of Python objects.
The sys.getsizeof() Built-in Function
The standard library's sys module provides the getsizeof() function. That function accepts an object (and optional default), calls the object's sizeof() method, and returns the result, so you can make your objects inspectable as well.
Measuring the Memory of Python Objects
Let's start with some numeric types:
1 | import sys |
2 | |
3 | sys.getsizeof(5) |
4 | 28 |
Interesting. An integer takes 28 bytes.
1 | sys.getsizeof(5.3) |
2 | 24 |
Hmm… a float takes 24 bytes.
1 | from decimal import Decimal |
2 | sys.getsizeof(Decimal(5.3)) |
3 | 104 |
Wow. 104 bytes! This really makes you think about whether you want to represent a large number of real numbers as floats or Decimals.
Let's move on to strings and collections:
1 | sys.getsizeof('') |
2 | 49 |
3 | sys.getsizeof('1') |
4 | 50 |
5 | sys.getsizeof('12') |
6 | 51 |
7 | sys.getsizeof('123') |
8 | 52 |
9 | sys.getsizeof('1234') |
10 | 53 |
OK. An empty string takes 49 bytes, and each additional character adds another byte. That says a lot about the tradeoffs of keeping multiple short strings where you'll pay the 49 bytes overhead for each one vs. a single long string where you pay the overhead only once.
The bytes object has an overhead of only 33 bytes.
1 | sys.getsizeof(bytes()) |
2 | 33 |
Lets look at lists.
1 | sys.getsizeof([]) |
2 | 56 |
3 | sys.getsizeof([1]) |
4 | 64 |
5 | sys.getsizeof([1, 2]) |
6 | 72 |
7 | sys.getsizeof([1, 2,3]) |
8 | 80 |
9 | sys.getsizeof([1, 2, 3, 4]) |
10 | 88 |
11 | |
12 | sys.getsizeof(['a long longlong string']) |
13 | 64 |
What's going on? An empty list takes 56 bytes, but each additional int adds just 8 bytes, where the size of an int is 28 bytes. A list that contains a long string takes just 64 bytes.
The answer is simple. The list doesn't contain the int objects themselves. It just contains an 8-byte (on 64-bit versions of CPython) pointer to the actual int object. What that means is that the getsizeof() function doesn't return the actual memory of the list and all the objects it contains, but only the memory of the list and the pointers to its objects. In the next section I'll introduce the deep\_getsizeof() function, which addresses this issue.
1 | sys.getsizeof(()) |
2 | 40 |
3 | sys.getsizeof((1,)) |
4 | 48 |
5 | sys.getsizeof((1,2,)) |
6 | 56 |
7 | sys.getsizeof((1,2,3,)) |
8 | 64 |
9 | sys.getsizeof((1, 2, 3, 4)) |
10 | 72 |
11 | sys.getsizeof(('a long longlong string',)) |
12 | 48 |
The story is similar for tuples. The overhead of an empty tuple is 40 bytes vs. the 56 of a list. Again, this 16 bytes difference per sequence is low-hanging fruit if you have a data structure with a lot of small, immutable sequences.
1 | sys.getsizeof(set()) |
2 | 216 |
3 | sys.getsizeof(set([1)) |
4 | 216 |
5 | sys.getsizeof(set([1, 2, 3, 4])) |
6 | 216 |
7 | |
8 | sys.getsizeof({}) |
9 | 64 |
10 | sys.getsizeof(dict(a=1)) |
11 | 232 |
12 | sys.getsizeof(dict(a=1, b=2, c=3)) |
13 | 232 |
Sets and dictionaries ostensibly don't grow at all when you add items, but note the enormous overhead.
The bottom line is that Python objects have a huge fixed overhead. If your data structure is composed of a large number of collection objects like strings, lists and dictionaries that contain a small number of items each, you pay a heavy toll.
The deep\_getsizeof() Function
Now that I've scared you half to death and also demonstrated that sys.getsizeof() can only tell you how much memory a primitive object takes, let's take a look at a more adequate solution. The deep\_getsizeof() function drills down recursively and calculates the actual memory usage of a Python object graph.
1 | from collections.abc import Mapping, Container |
2 | from sys import getsizeof |
3 | |
4 | def deep\_getsizeof(o, ids): |
5 | """Find the memory footprint of a Python object |
6 | |
7 | This is a recursive function that drills down a Python object graph |
8 | like a dictionary holding nested dictionaries with lists of lists |
9 | and tuples and sets. |
10 | |
11 | The sys.getsizeof function does a shallow size of only. It counts each |
12 | object inside a container as pointer only regardless of how big it |
13 | really is. |
14 | |
15 | :param o: the object |
16 | :param ids: |
17 | :return: |
18 | """ |
19 | d = deep\_getsizeof |
20 | if id(o) in ids: |
21 | return 0 |
22 | |
23 | r = getsizeof(o) |
24 | ids.add(id(o)) |
25 | |
26 | if isinstance(o, str) or isinstance(0, str): |
27 | return r |
28 | |
29 | if isinstance(o, Mapping): |
30 | return r + sum(d(k, ids) + d(v, ids) for k, v in o.iteritems()) |
31 | |
32 | if isinstance(o, Container): |
33 | return r + sum(d(x, ids) for x in o) |
34 | |
35 | return r |
There are several interesting aspects to this function. It takes into account objects that are referenced multiple times and counts them only once by keeping track of object ids. The other interesting feature of the implementation is that it takes full advantage of the collections module's abstract base classes. That allows the function very concisely to handle any collection that implements either the Mapping or Container base classes instead of dealing directly with myriad collection types like: string, Unicode, bytes, list, tuple, dict, frozendict, OrderedDict, set, frozenset, etc.
Let's see it in action:
1 | x = '1234567' |
2 | deep\_getsizeof(x, set()) |
3 | 56 |
A string of length 7 takes 56 bytes (49 overhead + 7 bytes for each character).
1 | deep\_getsizeof([], set()) |
2 | 56 |
An empty list takes 56 bytes (just overhead).
1 | deep\_getsizeof([x], set()) |
2 | 120 |
A list that contains the string "x" takes 124 bytes (56 + 8 + 56).
1 | deep\_getsizeof([x, x, x, x, x], set()) |
2 | 152 |
A list that contains the string "x" five times takes 156 bytes (56 + 5\*8 + 56).
The last example shows that deep\_getsizeof() counts references to the same object (the x string) just once, but each reference's pointer is counted.
Treats or Tricks
It turns out that CPython has several tricks up its sleeve, so the numbers you get from deep\_getsizeof() don't fully represent the memory usage of a Python program.
Reference Counting
Python manages memory using reference counting semantics. Once an object is not referenced anymore, its memory is deallocated. But as long as there is a reference, the object will not be deallocated. Things like cyclical references can bite you pretty hard.
Small Objects
CPython manages small objects (less than 256 bytes) in special pools on 8-byte boundaries. There are pools for 1-8 bytes, 9-16 bytes, and all the way to 249-256 bytes. When an object of size 10 is allocated, it is allocated from the 16-byte pool for objects 9-16 bytes in size. So, even though it contains only 10 bytes of data, it will cost 16 bytes of memory. If you allocate 1,000,000 objects of size 10, you actually use 16,000,000 bytes and not 10,000,000 bytes as you may assume. This 60% extra overhead is obviously not trivial.
Integers
CPython keeps a global list of all the integers in the range -5 to 256. This optimization strategy makes sense because small integers pop up all over the place, and given that each integer takes 28 bytes, it saves a lot of memory for a typical program.
It also means that CPython pre-allocates 266 * 28 = 7448 bytes for all these integers, even if you don't use most of them. You can verify it by using the id() function that gives the pointer to the actual object. If you call id(x) for any x in the range -5 to 256, you will get the same result every time (for the same integer). But if you try it for integers outside this range, each one will be different (a new object is created on the fly every time).
Here are a few examples within the range:
1 | id(-3) |
2 | 9788832 |
3 | |
4 | id(-3) |
5 | 9788832 |
6 | |
7 | id(-3) |
8 | 9788832 |
9 | |
10 | id(201) |
11 | 9795360 |
12 | |
13 | id(201) |
14 | 9795360 |
15 | |
16 | id(201) |
17 | 9795360 |
Here are some examples outside the range:
1 | id(257) |
2 | 140276939034224 |
3 | |
4 | id(301) |
5 | 140276963839696 |
6 | |
7 | id(301) |
8 | 140276963839696 |
9 | |
10 | id(-6) |
11 | 140276963839696 |
12 | |
13 | id(-6) |
14 | 140276963839696 |
Python Memory vs. System Memory
CPython is kind of possessive. In many cases, when memory objects in your program are not referenced anymore, they are not returned to the system (e.g. the small objects). This is good for your program if you allocate and deallocate many objects that belong to the same 8-byte pool because Python doesn't have to bother the system, which is relatively expensive. But it's not so great if your program normally uses X bytes and under some temporary condition it uses 100 times as much (e.g. parsing and processing a big configuration file only when it starts).
Now, that 100X memory may be trapped uselessly in your program, never to be used again and denying the system from allocating it to other programs. The irony is that if you use the processing module to run multiple instances of your program, you'll severely limit the number of instances you can run on a given machine.
Memory Profiler
To gauge and measure the actual memory usage of your program, you can use the memory\_profiler module. I played with it a little bit and I'm not sure I trust the results. Using it is very simple. You decorate a function (could be the main function) with an @profiler decorator, and when the program exits, the memory profiler prints to standard output a handy report that shows the total and changes in memory for every line. Here is a sample program I ran under the profiler:
1 | from memory\_profiler import profile |
2 | |
3 | @profile |
4 | def main(): |
5 | a = [] |
6 | b = [] |
7 | c = [] |
8 | for i in range(100000): |
9 | a.append(5) |
10 | for i in range(100000): |
11 | b.append(300) |
12 | for i in range(100000): |
13 | c.append('123456789012345678901234567890') |
14 | del a |
15 | del b |
16 | del c |
17 | |
18 | print('Done!') |
19 | |
20 | if __name__ == '__main__': |
21 | main() |
Here is the output:
1 | Filename: python_obj.py |
2 | |
3 | Line # Mem usage Increment Occurrences Line Contents |
4 | ============================================================= |
5 | 3 17.3 MiB 17.3 MiB 1 @profile |
6 | 4 def main(): |
7 | 5 17.3 MiB 0.0 MiB 1 a = [] |
8 | 6 17.3 MiB 0.0 MiB 1 b = [] |
9 | 7 17.3 MiB 0.0 MiB 1 c = [] |
10 | 8 18.0 MiB 0.0 MiB 100001 for i in range(100000): |
11 | 9 18.0 MiB 0.8 MiB 100000 a.append(5) |
12 | 10 18.7 MiB 0.0 MiB 100001 for i in range(100000): |
13 | 11 18.7 MiB 0.7 MiB 100000 b.append(300) |
14 | 12 19.5 MiB 0.0 MiB 100001 for i in range(100000): |
15 | 13 19.5 MiB 0.8 MiB 100000 c.append('123456789012345678901234567890') |
16 | 14 18.9 MiB -0.6 MiB 1 del a |
17 | 15 18.2 MiB -0.8 MiB 1 del b |
18 | 16 17.4 MiB -0.8 MiB 1 del c |
19 | 17 |
20 | 18 17.4 MiB 0.0 MiB 1 print('Done!') |
As you can see, there is 17.3 MB of memory overhead. The reason the memory doesn't increase when adding integers both inside and outside the [-5, 256] range and also when adding the string is that a single object is used in all cases. It's not clear why the first loop of range(100000) on line 9 adds 0.8MB while the second on line 11 adds just 0.7MB and the third loop on line 13 adds 0.8MB. Finally, when deleting the a, b and c lists, -0.6MB is released for a, -0.8MB is released for b, and -0.8MB is released for c.
How To Trace Memory Leaks in Your Python application with tracemalloc
tracemalloc is a Python module that acts as a debug tool to trace memory blocks allocated by Python. Once tracemalloc is enabled, you can obtain the following information :
- identify where the object was allocated
- give statistics on allocated memory
- detect memory leaks by comparing snapshots
Consider the example below:
1 | import tracemalloc |
2 | |
3 | tracemalloc.start() |
4 | |
5 | a = [] |
6 | b = [] |
7 | c = [] |
8 | for i in range(100000): |
9 | a.append(5) |
10 | for i in range(100000): |
11 | b.append(300) |
12 | for i in range(100000): |
13 | c.append('123456789012345678901234567890') |
14 | # del a |
15 | # del b |
16 | # del c |
17 | |
18 | |
19 | snapshot = tracemalloc.take_snapshot() |
20 | for stat in snapshot.statistics('lineno'): |
21 | print(stat) |
22 | print(stat.traceback.format()) |
23 |
Explanation
tracemalloc.start()—starts the tracing of memorytracemalloc.take_snapshot()—takes a memory snapshot and returns theSnapshotobjectSnapshot.statistics()—sorts records of tracing and returns the number and size of objects from the traceback.linenoindicates that sorting will be done according to the line number in the file.
When you run the code, the output will be:
1 | [' File "python_obj.py", line 13', " c.append('123456789012345678901234567890')"] |
2 | python_obj.py:11: size=782 KiB, count=1, average=782 KiB |
3 | [' File "python_obj.py", line 11', ' b.append(300)'] |
4 | python_obj.py:9: size=782 KiB, count=1, average=782 KiB |
5 | [' File "python_obj.py", line 9', ' a.append(5)'] |
6 | python_obj.py:5: size=576 B, count=1, average=576 B |
7 | [' File "python_obj.py", line 5', ' a = []'] |
8 | python_obj.py:12: size=28 B, count=1, average=28 B |
9 | [' File "python_obj.py", line 12', ' for i in range(100000):'] |
Conclusion
CPython uses a lot of memory for its objects. It also uses various tricks and optimizations for memory management. By keeping track of your object's memory usage and being aware of the memory management model, you can significantly reduce the memory footprint of your program.
This post has been updated with contributions fromEsther Vaati. Esther is a software developer and writer for Envato Tuts+.
Gigi Sayfan
Principal Software Architect at Helix
Gigi Sayfan is a principal software architect at Helix — a bioinformatics and genomics start-up. Gigi has been developing software professionally for more than 20 years in domains as diverse as instant messaging, morphing, chip fabrication process control, embedded multimedia applications for game consoles, brain-inspired machine learning, custom browser development, web services for 3D distributed game platforms, IoT sensors and virtual reality.He has written production code in many programming languages such as Go, Python, C, C++, C#, Java, Delphi, JavaScript, and even Cobol and PowerBuilder for operating systems such as Windows (3.11 through 7), Linux, Mac OSX, Lynx (embedded), and Sony PlayStation. His technical expertise includes databases, low-level networking, distributed systems, unorthodox user interfaces, and the general software development lifecycle.
FAQs
How do I check the memory consumption of an object in Python? ›
In this article, we are going to see how to find out how much memory is being used by an object in Python. For this, we will use sys. getsizeof() function can be done to find the storage size of a particular object that occupies some space in the memory. This function returns the size of the object in bytes.
How to trace memory usage in Python? ›To trace most memory blocks allocated by Python, the module should be started as early as possible by setting the PYTHONTRACEMALLOC environment variable to 1 , or by using -X tracemalloc command line option. The tracemalloc. start() function can be called at runtime to start tracing Python memory allocations.
How much memory does Python use? ›Python has a pymalloc allocator optimized for small objects (smaller or equal to 512 bytes) with a short lifetime. It uses memory mappings called “arenas” with a fixed size of 256 KiB.
How do I find the memory address of a Python object? ›We can get an address using the id() function. id() function gives the address of the particular object.
How do I check memory utilization? ›- Press Ctrl + Shift + Esc to launch Task Manager. Or, right-click the Taskbar and select Task Manager.
- Select the Performance tab to see current RAM usage displayed in the Memory box, and total RAM capacity listed under Physical Memory.
Check Computer Memory Usage Easily
To open up Resource Monitor, press Windows Key + R and type resmon into the search box. Resource Monitor will tell you exactly how much RAM is being used, what is using it, and allow you to sort the list of apps using it by several different categories.
To check the memory usage of a DataFrame in Pandas we can use the info(~) method or memory_usage(~) method. The info(~) method shows the memory usage of the whole DataFrame, while the memory_usage(~) method shows memory usage by each column of the DataFrame.
What is memory profiler in Python? ›memory-profiler is an open-source Python module that offers us the functions to track memory consumption of a program while its execution, and we can even monitor memory consumption with line-by-line analysis in the program and functions of the program.
Does Python have memory management? ›Memory management in Python involves the management of a private heap. A private heap is a portion of memory that is exclusive to the Python process. All Python objects and data structures are stored in the private heap. The operating system cannot allocate this piece of memory to another process.
Why is Python consuming so much memory? ›Those numbers can easily fit in a 64-bit integer, so one would hope Python would store those million integers in no more than ~8MB: a million 8-byte objects. In fact, Python uses more like 35MB of RAM to store these numbers. Why? Because Python integers are objects, and objects have a lot of memory overhead.
How Python objects are stored in memory? ›
Memory allocation in Python
The function calls and the references are stored in the stack memory whereas all the value objects are stored in the heap memory.
Storing integers or floats in Python has a huge overhead in memory. Learn why, and how NumPy makes things better. Objects in Python have large memory overhead. Learn why, and what do about it: avoiding dicts, fewer objects, and more.
What is the memory address of an object? ›During program execution, each object (such as a variable or an array) is located somewhere in an area of memory. The location of an object in the memory is called its address.
How do we calculate the memory address? ›Step 1: calculate the length of the address in bits (n bits) Step 2: calculate the number of memory locations 2^n(bits) Step 3: take the number of memory locations and multiply it by the Byte size of the memory cells.
What is using my memory? ›In the full Task Manager window, navigate to the “Processes” tab. You'll see a list of every application and background task running on your machine. Collectively, those programs are called “processes.” To sort the processes by which one is using the most memory, click the “Memory” column header.
How do I find memory leaks in Python code? ›- Get and store the number of objects, tracked ( created and alive) by Collector. ...
- Call the function that calls the request. ...
- Print the response status code, so that we can confirm that the object is created.
- Then return the function.
Here's is how you can check your PC's system resource usage with Task Manager. Press CTRL + Shift + Esc to open Task Manager. Click the Performance tab. This tab displays your system's RAM, CPU, GPU, and disk usage, along with network info.
How much memory does a Dataframe use? ›The long answer is the size limit for pandas DataFrames is 100 gigabytes (GB) of memory instead of a set number of cells.
How do I check my allocated virtual memory? ›Click Start > Settings > Control Panel. Double-click the System icon. In the System Properties dialog box, click the Advanced tab and click Performance Options. In the Performance Options dialog, under Virtual memory, click Change.
How to check memory size in Pandas? ›We can use Pandas info() function to find the total memory usage of a dataframe. Pandas info() function is mainly used for information about each of the columns, their data types, and how many values are not null for each variable. Pandas info() fnction also gives us the memory usage at the end of its report.
What happens if Python runs out of memory? ›
Crashing is just one symptom of running out of memory. Your process might instead just run very slowly, your computer or VM might freeze, or your process might get silently killed. Sometimes if you're lucky you might even get a nice traceback, but then again, you might not.
How do I check memory usage in Jupyter? ›The jupyter-resource-usage extension is part of the default installation, and tells you how much memory your user is using right now, and what the memory limit for your user is. It is shown in the top right corner of the notebook interface.
How do I check memory usage in PyCharm? ›PyCharm can show you the amount of used memory in the status bar. Use it to judge how much memory to allocate. Right-click the status bar and select Memory Indicator.
Is Python a stack or heap? ›Memory Allocation in Python
The methods/method calls and the references are stored in stack memory and all the values objects are stored in a private heap.
- Static memory. The stack data structure provides static memory allocation, meaning the variables are in the stack memory. ...
- Dynamic memory.
All objects and instance variables are stored in the heap memory. When a variable is created in Python, it is stored in a private heap which will then allow for allocation and deallocation.
What slows down Python code? ›In summary: code is slowed down by the compilation and interpretation that occurs during runtime. Compare this to a statically typed, compiled language which runs just the CPU instructions once compilated. It's actually possible to extend Python with compiled modules that are written in C.
How much memory is allocated for a variable in Python? ›Memory allocation for variables is completely independent of what type of object they refer to. Whether you assign x = [] , x = 5 , or x = SomeCrazyHugeThing() , the value doesn't affect the memory allocated for x .
Which memory are objects stored? ›The object values are stored in heap memory. An object reference on the stack is only an address that refers to the place in heap memory where that object is kept.
Where is object memory stored? ›In Java, all objects are dynamically allocated on Heap. This is different from C++ where objects can be allocated memory either on Stack or on Heap. In JAVA , when we allocate the object using new(), the object is allocated on Heap, otherwise on Stack if not global or static.
Where does the object get stored in memory? ›
The heap is used for storing objects in memory.
How do I calculate memory size in bytes? ›Bytes - A byte is simply 8 bits of memory or storage. This is the smallest amount of memory that standard computer processors can manipulate in a single operation. If you determine the number of bits of memory that are required, and divide by 8, you will get the number of bytes of memory that are required.
How the size of memory is specified? ›The size of a memory unit is specified in terms of the number of storage locations available; 1 K is 210 = 1024 locations and thus a 4 K memory has 4096 locations. The term erasable and programmable ROM (EPROM) is used for ROMs that can be programmed and their contents altered.
Is address space the same as memory? ›An address space is a range of valid addresses in memory that are available for a program or process. That is, it is the memory that a program or process can access. The memory can be either physical or virtual and is used for executing instructions and storing data.
How do I check memory occupied by DataFrame in python? ›To check the memory usage of a DataFrame in Pandas we can use the info(~) method or memory_usage(~) method. The info(~) method shows the memory usage of the whole DataFrame, while the memory_usage(~) method shows memory usage by each column of the DataFrame.
How to check memory occupied by list in python? ›getsizeof() function includes the marginal space usage, which includes the garbage collection overhead for the object. Meaning it returns the total space occupied by the object in addition to the garbage collection overhead for the spaces being used.
How much memory does a DataFrame use? ›The long answer is the size limit for pandas DataFrames is 100 gigabytes (GB) of memory instead of a set number of cells.
What is DF info () in Python? ›Pandas DataFrame info() Method
The info() method prints information about the DataFrame. The information contains the number of columns, column labels, column data types, memory usage, range index, and the number of cells in each column (non-null values). Note: the info() method actually prints the info.
Computer storage and memory is often measured in megabytes (MB) and gigabytes (GB). A medium-sized novel contains about 1 MB of information. 1 MB is 1,024 kilobytes, or 1,048,576 (1024x1024) bytes, not one million bytes. Similarly, one 1 GB is 1,024 MB, or 1,073,741,824 (1024x1024x1024) bytes.
Are Python objects stored in RAM? ›No, they are in a different memory called “Heap Memory” (also called the Heap). To store objects, we need memory with dynamic memory allocation (i.e., size of memory and objects can change). Python interpreter actively allocates and deallocates the memory on the Heap (what C/C++ programmers should do manually!!!