Strange Python memory usage with Scapy

I managed to find the actual cause to the memory consumption. It was scapy itself. Scapy by default is set to store all packets it captures. But you can disable it.

Disable:

sniff(iface=interface, prn=sniffmgmt, store=0)

Enable:

sniff(iface=interface, prn=sniffmgmt, store=1)

Thanks to BitBucket Ticket

As you can see profiler output suggests you use less memory by the end, so this is not representative of your situation.

Some directions to dig deeper: 1) add_time (why is it increasing memory usage?) 2) db_add (why is it decreasing memory usage? caching? closing/opening db connection? what happens in case of failure?) 3) populate_observed_list (is return value safe for garbage collection? may be there are some packets for which certain exception occurs?)

Also, what happens if you sniff more packets than your code is able to process do to performance?

I would profile these 3 functions and analyze possible exceptions/failures.

Very hard to say anything without the code, assuming it's not a leak in SQLAlchemy or scapy rather than in your code (seems unlikely).

You seem to have an idea of where the leak might happen, do some memory profiling to see if you were right.

Once your python process eats enough memory, you will probably get a MemoryError exception.

Thanks for the guidance everyone. I think I managed to resolve the increasing memory consumption.

A: Code logic plays a very big role in memory consumption as I have learnt. If you look at the memory_profiler output in my initial question, I moved lines 131-133 into the IF statement at line 136. This seems to not increase the memory so frequently. I now need to refine that populate_observedlist() a bit more to not waste so much memory.