Memory usage changes

[Maccara]

Hi,

I am wondering if I'm imagining things, but since there recently has been memory allocation changes I'm asking to make sure.

I "think" UE used to release used memory when groups were unloaded, but now it doesn't seem to anymore (v2.4.5, but noticed this a couple of versions ago).

Now it is constantly using over 1GB (mem usage in taskmanager, but VM Size is small though). And I distinctly remember UE releasing pretty much all memory in the past (at least when it was minimized & all groups closed).

This is also reflected in "Performance" tab available mem count (yes, I'm also aware of cache allocation separately).

I just noticed that UE DOES release memory for a short while when minimized, but will reclaim the memory for itself again in ab 10 seconds.

I didn't take notice of this earlier but yesterday I was wondering why my virtual machines were running slow I noticed major part of memory was allocated for UE.

Can anyone confirm?

I'm pretty sure something's changed, as otherwise there wouldn't be an option to unload groups, if it doesn't make any difference.

[Archibald]

I'm not seeing anything like that in 2.4.5.

As a matter of fact, with the x264 group loaded with 220 days of retention (140 million+ headers), and another smaller group ("only" 8 million headers) and also downloading a good sized queue of articles, UE is still using less memory than Yahoo Messenger. (58,244K vs 58,136K)

That's pretty embarrassing for Yahoo.

[Maccara]

Thanks for the confirmation that it indeed should use less memory.

Edit: Maybe you missed my point? I'm not interested in the amount of memory used when group is loaded - it could be 16GB for all I care. All I'm interested in is that the memory is released after the group is unloaded - otherwise, why there even should be an "unload" function? Although the memory usage in your example is still much lower than mine when loaded.

I wonder what is happening on my system. I load all groups (several binary groups w 120d retention) and mem usage of UE is then around 1GB. When I unload them, that memory is never released.

I'm running on XP X64, AMD X2 4400+, 4GB memory.

It used to be ok (UE released mem) and I tested for fun with standard malloc&free that mem is released ok in general in this system.

Alex, is it safe to install an older UE on top of new one (i.e. have there been any config changes etc which would screw up something if I tried with v231 for example).
(edit: with this I'm trying to verify if indeed behavior has changed in some certain version for this OS and what version that might be)

Any other ideas?

Edit: "Allocations" shows "Database 30.4 MB ( 31929341 bytes ) allocated / 933 MB reserved", but it never releases that reserved amount even if groups are unloaded.

[KilJaden]

Edit: "Allocations" shows "Database 30.4 MB ( 31929341 bytes ) allocated / 933 MB reserved", but it never releases that reserved amount even if groups are unloaded.

I noticed the same issues on my system . I have several groups setup to auto update headers every hour . The groups are loaded the headers processed , the group is unloaded but when i look at memory allocation UE takes the same amount of memory as when the groups where loaded.

This causes UE to allocate around 1 GB even though it only needs at that moment 5-10 MB , and UE doesn't release the memory when the group is unloaded.

[alex]

to revert to the old behaviour - in edit menu->properties->general, db deswapping, empty the value of the field (delete the number) and press ok.

the option ensures UE is not subjectted to swapping, i took it from search service code, although in search service case allocated memory is normally about the same as reserved memory.

just you may notice with other applications, if they allocated a lot of memory, you minimze them, after some time it may take a long time to bring them back to life because of disk thrashing on the system swap file. in v2.4+ it is possible to load several times more headers comparing to prior versions with memory allocations in the same order as for the search server and i saw this problem there.

e.g. with unpack memory is no such problem since it can be easily discarded between unpack tasks, with headers it is more complicated, i need to think if something more can be realistically done here, e.g. triggered by significant allocated/reserved discrepancy.

in short there are two choices, use whatever better fits you.

i see i forgot to mention the option in the v2.4 release notes, i added it now.

[Maccara]

in short there are two choices, use whatever better fits you.

Thank you very much - that solved the "problem" (I can understand why the new option might be desired too).

For my usage pattern, the old option is more suitable.

[alex]

yes, probably by default the option should be not engaged and left only for users who need extreme usage pattern, i'll consider changing the default in the next version.

the code which manages allocations is not completely dumb and coming out of swapped state was somewhat addressed already from the beginning, but if you fill couple of gigabytes and then swap them out it may take some time.

with the option enabled newsgroups may load much faster as well after a period of inactivity.

[Maccara]

the code which manages allocations is not completely dumb and coming out of swapped state was somewhat addressed already from the beginning, but if you fill couple of gigabytes and then swap them out it may take some time.

Yes, I was kind of hoping you were not just doing the usual dumb page locking to prevent swapping. (generally, a very bad idea in multitasking environment and should be reserved for device drivers which really need it; OS will usually have a better chance of deciding when to page applications)

By observing the behavior of the app, I'd suggest some tweaking still would be needed, as how it was implemented now it was very detrimental for OS performance when heavy multitasking was happening.

Definitely better to let OS choose the paging strategy as a default option, and only allow the option for users who really need it.

with the option enabled newsgroups may load much faster as well after a period of inactivity.

Absolutely. However, a moot point in my case, as it is a CPU bound operation in my case anyway.

And for normal usage, I have enough memory that OS rarely decides to page out the whole UE context so it doesn't affect me much, but with the new option it effectively prevented efficient multitasking when I needed it and I was running near the memory limits of this system - effectively inactive UE was using memory unnecessarily while it was not doing anything with it whereas other apps would have had plenty of use of the ~1GB UE was currenty "hoarding".

[alex]

tested it further, coming out of swapped out state appears to be responsive enough (checked it with around 1GB allocated), so i replaced the option with unchecked checkbox in v2.4.7, it is not the indexing server typical load yet.

[KilJaden]

tested it further, coming out of swapped out state appears to be responsive enough (checked it with around 1GB allocated), so i replaced the option with unchecked checkbox in v2.4.7, it is not the indexing server typical load yet.

Everything is back to normal now .

As for the new option introduced I'm one of those people who loves the automation in UE so it's a very big bonus to me .

And if you have time you might wana make UE IPv6 compatible because there are some quite good free ivp6 out there , if not for main at least for fill ups. I'm just tired of using netsh command

Any way keep up the good work Alex.

[alex]

maybe after v2.5 i'll check it out (ipv6), for v2.5 i have one more change (at least) in mind. i need something which won't break older systems compatibility.

[francky013]

Hi,

I have a problem with UE 2.4x & 2.5.
After many hours, UE take all my ram (4 go), all my swap (8 Go) and make my pc crash. I think UE make RAM reservation (when get new headers ?? I set on auto get headers every 2 hours) and not free them : nothing in task manager, UE is allways about 80 M of Ram.
The check box option is unchecked.
No error log on UE.

No problem when UE is off or with UE 2.31.

I'am on x64 vista.

Any idea ?

[alex]

there are no changes between v2.3.1 and v2.5 with default settings which could cause any such difference in behaviour.

if you are sure about v2.3.1 try to delete and readd UE in firewall definitions, maybe firewall driver causes that.

UE doesn't make any new RAM reservations, just v2.5 can utilize more memory, after the point when v2.3.1 would quit or run out of memory, it is rather compiling with large address aware compiler option and increasing some constant which wouldn't change anything in program behaviour, so the source code is practically the same.

if UE takes RAM it would be seen it takes RAM. try to check other processes in the windows task manager, maybe you see one which takes the RAM if you observe swapping.

also UE shouldn't be able to crash the system, at worst if, hypothetically, you would see high UE process memory usage which would result in eventual swapping the system would be not responsive, but if you see small memory usage there cannot be swapping caused by the UE process.

if it allocates memory you'll immediately see it in memory usage, if it reserves memory without committing it - you cannot see it and it doesn't require swapping, it is rather an instruction not to touch certain range of virtual addresses which has nothing to do with allocating physical memory.

in short try to check for interference, what do you mean by system crashing? blue screen?

[francky013]

yes blue screen when I want to do tomething on the pc.
I will try to check for interference

[alex]

blue screen you should be able to make it show the driver on which it stops, maybe you see the name (UE doesn't have any drivers).

i don't install much so didn't experience blue screens for years, maybe this is relevant:

http://www.aoaforums.com/forum/aoa-faq/ ... rrors.html

if you have the driver name look for it on google, should point to something you have or had installed on your computer.