std::Queue leaking memory

std::Queue leaking memory

Post by Andrew Cha » Fri, 20 Apr 2007 01:38:34


From several worker threads I add a pointer to an instance of a class to a
std::queue. Another thread copies the contents of the pointer from the front
of the queue and then deletes the object pointed to. Over the course of a
few hours this application gradually consumes more and more memory. It
appears that the object is not deleted. Any idea why this might happen?

Could it be something to do with the fact that the object I am trying to
delete contains STL string objects and the program considers these to still
has references to them?

Here is a smaple of the thread that adds to the queue:
CMergeObject * pMO = new CMergeObject(m_sFileNameTemp, m_sFileNamePair,
m_sFileNameFinal, sm_eFileFormatFinal, m_pInfoOutputSink);
::EnterCriticalSection(&CMergeObject::sm_csMergeQueue);
g_qMO.push(pMO);
::LeaveCriticalSection(&CMergeObject::sm_csMergeQueue);

Here is an example of the removal from the queue and deletion of the object:
::EnterCriticalSection(&CMergeObject::sm_csMergeQueue);
oMO = * g_qMO.front();
delete g_qMO.front(); // delete the Merge Object
g_qMO.pop(); // pop the queue element from the queue
::LeaveCriticalSection(&CMergeObject::sm_csMergeQueue);

If you need more information please let me know.

Thanks.
 
 
 

std::Queue leaking memory

Post by aao » Fri, 20 Apr 2007 02:29:02

I believe std::queue caches memory for optimization. So if at any point of
the time your processor thread lagging behind the worker thread(s), the
queue grows but never releases the memory, in case it needs that memory
later. Tyr std::list instead.

 
 
 

std::Queue leaking memory

Post by Tom Widmer » Fri, 20 Apr 2007 02:56:08


Why are you using pointers? Given that:
oMO = * g_qMO.front();

copies the object anyway (and hence they are copyable), you might as
well stick with a queue<CMergeObject>. Then you don't need the delete.
You can use swap based idioms if copying is expensive.

std:queue, by default, uses std::deque under the hood. In VC7.1 and
later, IIRC, std::deque does not release segments back to the OS when
you make pop calls, so at any one time, it will consume the memory of
the maximum number of items that have ever been on the queue.

std::string obviously cleans up its own memory when you destroy it, so
that shouldn't be the problem.

Are you sure your leaks are due to this code? Have you used any kind of
leak detector?

Tom
 
 
 

std::Queue leaking memory

Post by Andrew Cha » Fri, 20 Apr 2007 05:15:53

Thanks, but I don't think this is the explanation. My app. is using more and
more memory over time. The memory used does not peak.

- A
 
 
 

std::Queue leaking memory

Post by Ulrich Eck » Fri, 20 Apr 2007 17:54:13


All containers of the C++ standard do that (well, at least in any
implementation I know, I'm not sure it is required), so switching to
std::list wouldn't change a thing.

Anyway, what is missing is a complete example for reproduction. As far as
the shown code is concerned it is just buggy in the sense that it isn't
exception-safe, otherwise it should normally work. There are way too many
chances for f***up in the code that wasn't shown though.

Uli
 
 
 

std::Queue leaking memory

Post by Jason » Thu, 26 Apr 2007 21:56:38

On Apr 18, 1:56 pm, "Tom Widmer [VC++ MVP]" < XXXX@XXXXX.COM >


"swap-based idioms" ? could you elaborate on that? I'm just curious.
 
 
 

std::Queue leaking memory

Post by Jason » Fri, 27 Apr 2007 00:14:24

On Apr 18, 1:56 pm, "Tom Widmer [VC++ MVP]" < XXXX@XXXXX.COM >


Just curious -- what do you mean by "swap-based idioms"?
 
 
 

std::Queue leaking memory

Post by Ulrich Eck » Fri, 27 Apr 2007 17:17:07


You can implement a swap operation (like the one that several containers in
the standard lib have) that swaps two values with little computational
overhead. This can be used in places where you would normally make a copy
and then discard the original.

Uli
 
 
 

std::Queue leaking memory

Post by Tom Widmer » Sat, 28 Apr 2007 01:59:33


You can use these for containers of a particular value type when:

a) Default constructing the value type is cheap
b) Copying some objects of the value type is expensive
c) Copying of your objects has been identified as a bottleneck

Basically, instead of calling push_back or insert to add an
expensive-to-copy element, you instead push_back or insert a default
constructed element, and then swap your expensive-to-copy element with
the one now in the container. e.g.

T expensiveElement(...);

//instead of:
cont.push_back(expensiveElement);

//you can do:
cont.push_back(T());
cont.back().swap(expensiveElement);

bearing in mind that expensiveElement will now just be a default
constructed element.

Tom
 
 
 

std::Queue leaking memory

Post by Andrew Cha » Sun, 29 Apr 2007 08:22:02

Thanks. You were right. The leaks were elsewhere!

- A