Severe slowdown

Pheonix · Post by **Pheonix** » Mon Sep 15, 2003 9:23 pm

I posted a message about this yesterday but can't find that thread now.

Anyway, a few people have noticed that NewsPro gets very slow once it's downloaded a few thousand headers. Here's my experience today.

Last night I set it to get headers for three newsgroups (in a folder), and to download messages for two other groups. Newspro is looking at all the various Newsfeeds servers, and is set to detach servers if they don't have the groups. So, five groups were active.

Today, when I checked the computer, it was slow. It took minutes to get the screen to redraw anything when the screen blanker quit. This was just after 12 noon. I started keeping track of time at 12:20, when I finally got the file menu to show up and clicked on Save Database & restart. After 20 minutes, I did it again. The program finally exited at 1:09 pm, approximately one hour from the time I first told it to restart. During this time, I was also waiting for my mail client to start, which it didn't do until NewsPro quit. Also during this time, the hard drive light was on constantly.

Is there any way to get NewsPro to not bring the system to a halt like this?

I'm currently downloading message using the Download with priority command, so I can try out attachments before saving them. I tried having them saven when the message was downloaded, but there's no easy way to get rid of attachments I don't want after that (a Delete Attachments command, under Delete Header and Delete Body would be greatly apprieciated since it's a major hassle to have to search through various folders manually for them later.

Givver · Post by **Givver** » Thu Sep 18, 2003 7:57 pm

I am experiencing exactly the same problem. It was very fast when I started using it. Now I have several newsgroups with thousands of messages each and when I run NewsPro my entire Computer slows to a crawl. Windows XP Task Manager says NewsPro is using 150Meg of RAM and is only using 0-10 % CPU time. I have 256 Meg of ram with nothing else running and Task Manager says I have 30Meg Free Physical Memory.

I have been trying to download new headers in one group for the last hour and it is going at 15-20 KB/sec when I first started using NewPro It was downloading at 380 KB/sec.

Is there anyway to make this better?

Thanks
Givver

Pheonix · Post by **Pheonix** » Fri Sep 19, 2003 12:38 am

In my case, the download speed seems to be fast, at least according to NewsPro's task manager (usually around 500KB and up), CPU usage is minimal, as is RAM usage. It's the HD access that's killing things. Even when I'm finally able to tell NP to exit, it takes several minutes of disk activity before it finally goes away. Then I can restart and it works fine again, for about an hour, before the slowdown occurs once more.

blackdog56 · Post by **blackdog56** » Fri Sep 19, 2003 1:31 am

This seems to be a common issue with Windows XP and all newsreaders when processing that many headers. The issue seems to be Windows XP method of swapping out to disk even though there is RAM remaining. The answers seem to be .. getting more ram and processing fewer headers, or perhaps switching to Windows 2000. This is a topic in almost every newsreader forum that I have browsed in.

Pheonix · Post by **Pheonix** » Fri Sep 19, 2003 2:12 am

I am using windows 2000. It's more about how windows handles accessing directories with many files. In the database folder, I have over 34,000 files, apparently one for each header, plus a few large ones that I can't figure out.

I used to use agent, and that only created two files for each newsgroup. Slowdowns like this were never a problem, but it doesn't support multiple servers, which is why I switched.

NoNo · Post by **NoNo** » Fri Sep 19, 2003 10:04 am

Hi!
I agree with blackdog56.
U might consider to get a file cache manager, that could help a little.
If u have enough RAM and u use "minimize disk accesses on start", NEVER minimize NewsPro.
If u do , u can still invoke F10.
U really need to optimize the number of Headers u retrieve!
I remember having Newspro eating like 720MB, now it's all about 150-200MB.

Phoenix :
that's rather strange that u got 34,000 files in ur data folder...
afaik each Newsgroups leads to 6 files, even posted messages & newsgroups list +search & so on...:
1 with the plain name newsgroup and others end by $Author, $Date, $Lines, $Number, $Subject
Hdr.dat contains the Headers & Del.dat the deleted ones.
Other details u need to ask to Alex !

Unless u have 34,000 messages read with bodies.
In this case it's a normal event as each messages should be present as a .npr file.
In this case u really need to do something ! cleaning ?

bruce73 · Post by **bruce73** » Fri Sep 19, 2003 2:29 pm

NEVER minimize NewsPro.

Why do you suggest this?

NoNo · Post by **NoNo** » Fri Sep 19, 2003 7:17 pm

Mmmh I'm no expert to explain that and got poor english but :

In standard usage I have allways open :
- Pending&failed tasks.
- a virtual folder with an initial filter.
- a virtual folder that is a sub-folder of the first one.

With the option "minimize disk accesses on start", I figured out that provided u have enough ram, all headers & necessary data will be hold on memory, well it's what we wanted !

I also noticed that in the task manager when I minimize NewsPro,"Mem Usage" for it drop from 160,288K to a value like under 30,000K.
It doesn't mean a problem becoz If I recall NewsPro, I do a filter extract, no noticeable disk activity occur and "Mem Usage" rise up like 60,000K.

But given time , if I re-minimize it and go to sleep , I wake up , restore it,and even a simple scrolling on a window make noticeable disk activity.
On the contrary if I hit F10 and just go to sleep , I wake up , all seems normal as I left it.
I suppose Windows "paged" a lot of the mem used by NewsPro but I'm still no expert , just a user.

Of Course that can be only me

so here's the details:
ASUS P2B-DS, PIII 650Mhz x2, 1GB RAM
XPPro SP1 + O&OCleverCache + O&ODriveLed (no Screen Saver, just a "turn off monitor" active)
NewsPro 3.71 Database is on 60GB IBM HDD that runs on an additional Promise IDE card. (%systemroot% & PageFile are not on this HDD)
NewsPro folder is like 914 MB with 2,897 files with database backup active
1st virtual folder I'm using contains 596045+21088 headers over 12+7 newsgroups.
2nd virtual folder is 596045 headers over 12 newsgroups.
18 News Servers are hiding behind those newsgroups and thx to Alex only 2 Servers are used for headers.

2 days of testing, in fact 2 nights

Think of it , I should have said "hit F10, don't minimize it & go to sleep!"

bruce73 · Post by **bruce73** » Fri Sep 19, 2003 11:24 pm

Think of it , I should have said "hit F10, don't minimize it & go to sleep!"

That was going to be my question: is it hitting F10 that prevents the lag and disk activity, or the fact that you don't minimize?

I don't know enough about Windows either to say whether or not when you minimize a window over a long period of time, it will eventually page out what's in RAM, as SOP. That would certainly make sense, if it does. So when you get back and maximize it in the morning, Windows must access the HD to place it all back in memory, hence the disk thrashing.

NoNo · Post by **NoNo** » Sat Sep 20, 2003 12:54 am

Well I will say both.
F10 should be invoke if u previously minimized.
Still I think u need to test it urself.

bruce73 · Post by **bruce73** » Sat Sep 20, 2003 1:02 am

OK, fair enough.

panasync · Post by **panasync** » Tue Sep 23, 2003 11:49 am

I'm not so sure it's the way XP handles it's caching. I've seen the same slowdowns with 2k-sp4

blackdog56 · Post by **blackdog56** » Tue Sep 23, 2003 6:59 pm

I am not sure either, but I never try to manage a ton of headers. I scan only a few newsgroups regularly and XPAT into the big groups for specific things when I want them. I do scan other multiserver newsreader, that can combine groups into folders forums, and they hmm murphys law.. if you have the capability to exceed your memory and MHrz, you will find out a way how to do it.

Pheonix · Post by **Pheonix** » Wed Sep 24, 2003 9:46 am

If NewsPro could dould delete attachments on a per message basis, then I'd be downloading & saving attachments automatically. Then I could go through & see what I want to keep & use it to delete the rest. However, since it doesn't do that, I'm forced to either just download & save only the attachments I want after checking them, or save everything, and then manually hunt through different folders later using windows explorer and get rid of files I don't want, and rename files I do want. Anyone that's downloaded any attachments off newsgroups will know that about half the time attachments are not named very descriptively, or even accurately.

I used to use Agent for newsgroups, and I set it up to automatically save attachments. Later, I could just double-click the message to run the attachment from wherever it was saved, and if I wanted it, I just delete the message. The attachment would be untouched. If I didn't want it, I just select Delete Attachment, and the file would be gone. I didn't have to hunt through multiple folders for it.

Since I save files to different folders for different newsgroups, I rarely ever run into a situation where disc activity is a problem. I've had folders filled with nearly 100,000 files before, and they do take a long time to list, but they've never brought the system to a halt like the database folder does with just 30,000 files.

Again, this is with Windows 2000, not XP, with a dual Celeron600 and an ultraATA 100 Western Digital HD (200GB).

NoNo · Post by **NoNo** » Wed Sep 24, 2003 11:08 pm

Hi!

Again, this is with Windows 2000, not XP, with a dual Celeron600 and an ultraATA 100 Western Digital HD (200GB).

Well I don't think Microsoft have change much how XP handle memory as virtual memory.
They quite introduce new official bugs that didn't exist under 2k (& still unpatched in their last SP), numerical order with bugs & inverse behaviour for special characters w/o giving the user the opportunity to change it easily...
Scan of multimedia files for retrieving info is annoying & so on...
But Still I do believe that Virtual Memory management under XP is quite the same , files are unecessary or too much cached(file caching) , memory of proggies will hit the swap file at a time & so on...

I remember having Newspro eating like 720MB, now it's all about 150-200MB.

Well, I did this when I was still under 2k, but afairemember this occured as well:

But given time , if I re-minimize it and go to sleep , I wake up , restore it,and even a simple scrolling on a window make noticeable disk activity.

The thing is I'm carelless of disk activity as I significantly reduce memory usage.
Now as I said , I'm not an expert, I tried to describe noticeable events on my config that I was waiting for some1 to compare, but well seems it's just me !:P
Discussing how NewsPro handle messages & attachements is another matter & every proggies has its "pros & cons"... because we're all differents heh !