C++/VB - Why is network reading slow?

Asked By Peter Olcott
13-Jan-10 07:52 PM
I can copy a file to and from my Samba network in 20
seconds, yet it takes 80 seconds to compute its MD5 value.
The delay is not related to the MD5 processing time because
it can compute this in 12 seconds from the local drive.

What is going on here?
Windows XP
(1)
Windows 7
(1)
Vista
(1)
VB
(1)
GetFileSizeEx
(1)
CacheBYTE
(1)
SetFilePointer
(1)
GetTickCount
(1)
  Hector Santos replied to Peter Olcott
13-Jan-10 09:49 PM
Probably in how you are reading the file to hash it.

Are you doing block I/O or streaming (byte) I/O?

To me, that might explain the different timings your describe.

--
HLS
  Hector Santos replied to Hector Santos
13-Jan-10 09:57 PM
Just to note, the baseline time you are shooting for <= 32 seconds:

20 to read, 12 to process.

However, your 12 seconds includes local file I/O time, so whatever
that is, your ideal network/processing time should be:

20 + 12 - localIOtime <= 32

--
HLS
  Peter Olcott replied to Hector Santos
13-Jan-10 10:09 PM
I am doing block I/O, and it is very fast on the local drive
and much slower on the 1.0 gb LAN, yet file copies to and
from the LAN are still fast.

(1) File copy to and from the LAN is faster than local drive
copies, 20 seconds for LAN, 25 seconds for local.
(2) Block I/O is fast on the local drive, 12 seconds for 632
MB.
(3) Block I/O is slow on the LAN, 80 seconds for 632 MB.
I also tried changing the block size from 4K to 1500 bytes
and 9000 bytes (consistent with Ethernet frame size), this
did not help.
  Peter Olcott replied to Hector Santos
13-Jan-10 10:18 PM
It takes 12 seconds to read and process the file locally the
first time, it only takes 4 seconds to process the same file
again. Both the network and the local machine have
equivalent processors and disk drives. The LAN has a 1.0 gb
connection.
  Hector Santos replied to Peter Olcott
14-Jan-10 12:09 AM
By File Copy, you mean DOS copy command or the CopyFile() API?

To me, the above appears to be consistent with a caching issue that
your code is not enabling when the file is first open. The "File Copy"
is doing it, but you are not. Probably showing how you are opening the
file will help, i.e. the CreateFile() function or fopen().

Another thing is maybe to check google

Search: SAMBA Slow File Copy

There are some interesting hits, in particular if you are using Vista,
this MS Samba hotfix for Vista,

http://support.microsoft.com/kb/931770

There was another 2007 thread that the original poster said turning
off indexing improved the speed.

--
HLS
  Peter Olcott replied to Hector Santos
14-Jan-10 12:55 AM
I am using the DOS command prompt's copy command.  This is
fast.


The problem is the contradiction formed by the fact that
reading and writng the file is fast, while reading and not
wrting this same file is slow.
I am currently using fopen() and fread();  I am using
Windows XP.
  Tom Serface replied to Peter Olcott
14-Jan-10 02:28 AM
Not sure if this helps or not, but I have also noticed that network access
speeds up if you map a drive letter to the share rather than using a UNC.
I have seen as much as 3x speed up.  I am not sure why.

Tom
  Hector Santos replied to Peter Olcott
14-Jan-10 03:10 AM
True, if the DOS copy command is fast,then I believe the code you are
using is not optimal.  The DOS Copy is using the same CreateFile() API
which fopen() also finally uses in the RTL.  So you should be able to
match the same performance of the DOS Copy command.

Have you tried using setvbuf to set a buffer cache?

Here is a small test code that opens a 50 meg file:

// File: V:\wc7beta\testbufsize.cpp
// Compile with:  cl testbufsize.cpp


void main(char argc, char *argv[])
{
char _cache[1024*16] = {0};  // 16K cache
BYTE buf[1024*1]     = {0};  // 1K buffer

FILE *fv = fopen("largefile.dat","rb");
if (fv) {
int res = setvbuf(fv, _cache, _IOFBF, sizeof(_cache));
DWORD nTotal = 0;
DWORD nDisks = 0;
DWORD nLoops = 0;
DWORD nStart = GetTickCount();
while (!feof(fv)) {
nLoops++;
memset(&buf,sizeof(buf),0);
int nRead = fread(buf,1,sizeof(buf),fv);
nTotal +=nRead;
if (nRead > 0 && !fv->_cnt) nDisks++;
}
fclose(fv);
printf("Time: %d | Size: %d | Reads: %d | Disks: %d\n",
GetTickCount()-nStart,
nTotal,
nLoops,
nDisks);
}
}

What this basically shows is the number of disk hits it makes
by checking the fv->_cnt value. It shows that as long as the cache
size is larger than the read buffer size, you get the same number of
disk hits.  I also spit out the milliseconds.  Subsequent runs, of
course, is faster since the OS API CreateFile() is used by the RTL in
buffer mode.

Also do you know what protocol you have Samba using?


--
HLS
  Hector Santos replied to Tom Serface
14-Jan-10 03:34 AM
Thats a good point. That was differently something I had to do under
NT and always keep that in mind with other OSes. But after running out
of drive letters and using UNC in certain new scenarios, I have not
seen a slow down.  I figured MS addressed this old problem.  I seem to
recall this was explained once back in the day.

--
HLS
  Hans-J. Ude replied to Peter Olcott
14-Jan-10 04:11 AM
You can use a network sniffer like Wireshark to see what is going on.
It gives you detailed information, includes filters, timestamps, etc.

Hans
  Peter Olcott replied to Tom Serface
14-Jan-10 09:15 AM
I am already doing that.
  Peter Olcott replied to Hector Santos
14-Jan-10 10:01 AM
I am guessing that the code above will work with a file of
any size?
If that is the case, then you solved my problem.
The only Samba protocol that I am aware of is smb.
  Hector Santos replied to Peter Olcott
14-Jan-10 12:14 PM
Should be fine as long at you keep in 32 bit chucks.

For the benefit of the future mail archive researchers coming across
the same issue, I am going to assume the solution was using rtl
setvbuf() to set a user-defined cache buffer of reasonable size.

--
HLS
  Joseph M. Newcomer replied to Peter Olcott
14-Jan-10 02:27 PM
See below...

****
Use of fopen/fread would certainly indicate that you are not doing this in anything like
an optimal fashion.

If you want to read a file that is under 100MB, it is usually best just to allocate a
buffer the size of the file, CreateFile, do a single ReadFile, do your computation, do a
WriteFile, and you are done.  You are comparing two completely unrelated concepts:
fopen/fread and a copy command; what you did not ask was "what is the fastest way to read a
file"; instead, you observe that two completely different technologies have different
performance.  You did not actually state this in your original question; you just used a
generic concept of "copy".  Details matter!

Note that fread, called thousands of times, is amazingly slow in comparison to a single
ReadFile.

By failing to supply all the critical information, you essentially asked "Why is it that I
can get from city A to city B in 20 minutes, but my friend takes two hours?" and neglected
to mention you took the high-speed train while your friend went by bicycle.
joe
****
Joseph M. Newcomer [MVP]
email: newcomer@flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
  Joseph M. Newcomer replied to Peter Olcott
14-Jan-10 02:46 PM
See below...

****
Reading a 50MB file, why such an incredibly tiny buffer?
****
****
The memset is silly.  Wastes time, accomplishes nothing.  You are setting a buffer to 0
right before completely overwriting it!  This is like writing
int a;

a = 0;   // make sure a is 0 before assigning b
a = b;
****
****
If I were reading a small 50MB file, I would do

void tmain(int argc, _TCHAR * argv[])
{
HANDLE h = CreateFile(_T("largefile.dat"), GENERIC_READ, 0, NULL, OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL, NULL);

LARGE_INTEGER size;

GetFileSizeEx(h, &size);

// This code assumes file is < 4.2GB!
LPVOID p = VirtualAlloc(NULL, (SIZE_T)size.LowPart, MEM_COMMIT, PAGE_READWRITE);
DWORD bytesRead;
ReadFile(h, p, size.LowPart, &bytesRead, NULL);
... process data
  Stephen Myers replied to Joseph M. Newcomer
14-Jan-10 03:24 PM
Just to verify my (admittedly limited) understanding...

I assume that the code posted will fail for files greater than 2GB or so
with a 32 bit OS due to available address space.

Steve
  Joseph M. Newcomer replied to Stephen Myers
14-Jan-10 04:37 PM
Yes, but the file size was given as 50MB.
joe
  Joseph M. Newcomer replied to Joseph M. Newcomer
14-Jan-10 05:22 PM
By the way, did anyone really notice that ReadFile and WriteFile in Win64 cannot read or
write more than 4.2GB?  Seems really, really strange the length and bytes read did not
become DWORD_PTR values...
joe
  Peter Olcott replied to Hector Santos
14-Jan-10 05:32 PM
char _cache[1024*64] = {0};  // 64K cache
BYTE buf[1024*4]     = {0};  //    4K buffer

These buffer sizes match the DOS copy speed, and provide the
best performance.
  Peter Olcott replied to Joseph M. Newcomer
14-Jan-10 05:40 PM
I tested this and increasing the buffer size beyond 64K and
4K respectively does not measurably increase speed. Also in
my case I must process files of arbitrary sizes to compute
their MD5.
  Hector Santos replied to Stephen Myers
14-Jan-10 09:30 PM
I should probably test it, but it should work because the code is
streaming and not doing any seeking.

remove the DWORD nTotal counter or change it to _int64 if he wants to
collect that bit of information.

But as a streaming reader, I do not expect an issue.

The problems, as we seen it in our 32 bit product when it comes to a
file that has gone beyond 2 gigs, is when there is SEEK beyond a 32
bit range, like a file index returning a DWORD position.

This is something that is currently pushing our agenda as more and
more customers are coming across a 2 gig+ file size need. We tested
the Extended seeking function that offer 64 bit quads positions so I
know that works.  What we are debating is whether we should add 64 bit
FILE I/0 (which again I know works) or just move to pure 64 bit.  The
latter is a big revamping investment and not quite ready for that.

So off hand, just starting at the beginning of the file and reading
32bit chucks, should be fine.  I do not see a technical reason why that
that will not work.

--
HLS
  Hector Santos replied to Joseph M. Newcomer
14-Jan-10 09:51 PM
Its amazing how "big" is relative these days. My first computer had a
$1,500 Micropolis 10 meg drive! :)

BTW, it (the code posted) should not be an issue when just streaming
in a file of any size.

The problem begins when seeking a file.

When seeking is required, we know by using the 64 bit WIN32 file I/O
functions that it works for large +2gig files.

In one part of our server product file handing, a ISAM database with
four key index files have DWORD position indices.  Issues occur as the
database grows and a index goes beyond a 32 bit value.  A documented
limitation but a limitations that is not outdated.

A simple solution in the works was to use the 64 bit extended file I/O
functions which offer QUAD (double DWORD) positions.  Its about to be
implemented in new major revision of our server.

For backward single source compatibility, I produced a header and
wrapper functions.  Here is the *.h and *.cpp files:

//------------------------------------------------------------
// File Name : ss64lib.h
//------------------------------------------------------------






TINT ssFileSeek(HANDLE hf, TINT distance,
WORD MoveMethod = FILE_BEGIN);
TINT ssFileEnd(HANDLE hf);
TINT ssFilePos(HANDLE hf);
BOOL ssFileRewind(HANDLE hf);
BOOL ssFileLock(HANDLE hf, TINT Offset, TINT nBytes);
BOOL ssFileUnlock(HANDLE hf, TINT Offset, TINT nBytes);
TFILESIZE ssFileSize(HANDLE hf);


//------------------------------------------------------------
// File Name : ss64lib.cpp
//------------------------------------------------------------


TINT ssFileSeek(HANDLE hf, TINT dist, WORD method)
{
LARGE_INTEGER li;
li.QuadPart = dist;
li.LowPart = SetFilePointer (hf, li.LowPart, &li.HighPart, method);
if (li.LowPart == 0xFFFFFFFF && GetLastError() != NO_ERROR) {
li.QuadPart = -1;
}
return li.QuadPart;
}

BOOL ssFileRewind(HANDLE hf)
{
return ssFileSeek(hf,0,FILE_BEGIN) == 0;
}

TINT ssFilePos(HANDLE hf)
{
return ssFileSeek(hf,0,FILE_CURRENT);
}

TINT ssFileEnd(HANDLE hf)
{
return ssFileSeek(hf,0,FILE_END);
}

BOOL ssFileLock(HANDLE hf, TINT Offset, TINT nBytes)
{
LARGE_INTEGER fp, nb;
  Hector Santos replied to Peter Olcott
14-Jan-10 10:06 PM
Ok, thanks for the feedback.

The reason I asked what protocol was Samba was setup to use is because
it can use sockets and when it comes to sockets, we generally use as a
rule of thumb, a telecommunications "Bucket Brigade" concept:

Receive HIGH,  Write LOW

This helps to minimize contention (pressure) issues.  Receivers know
what they can handle for reception but not what the remote has for
transmission.  So overwhelming it with large buffer writes can (and
generally will) increase flow control issues.

A "smooth" bucket brigade is when the right buffer (packets) sizes are
used and the brigade is working in harmony, e.g.  a full bucket is
used, there are no spill overs and no bucket handler that is saying:


For you, you were receiving, so your receiver optimization was
possible.  When writing, it might be a different story.

--
HLS
  Alexander Grigoriev replied to Joseph M. Newcomer
14-Jan-10 10:43 PM
Historically, I/O sizes in the kernel drivers has been stored in ULONG. And
Length in IO_STACK_LOCATION is ULONG, too. That would be a bit too much
hassle to convert everything to SIZE_T...
  Hector Santos replied to Joseph M. Newcomer
14-Jan-10 11:21 PM
Hmmm, both are limited to what the backend protocol driver is using
for packet size.

I would never recommend using a single ReadFile read with a large
block such as a 100mb. Now if the file was local as a memory map, that
would be a different story and even when we know the OS itself (by
default) actually using memory maps when opening files, a large single
read like 100mb IMO definitely adds design pressures.  What if there
is a failure or some condition he wants to detect during the very
large block read?

IOW, he now needs to make it asynchronous anyway!

Again, I can see your design point if the file was LOCAL and knowing
that Windows itself actually opens file in page mode.  But not over a
network. I would naturally suspect engineering issues there.



I did not have a problem myself. It was obvious what the issue was when
stating a DOS file copy was faster than his code - although it did
take a few mail tags.

--
HLS
  Hector Santos replied to Alexander Grigoriev
14-Jan-10 11:55 PM
To add to this, the other issue with the misconception of using ULONG
or DWORD, is that even if your own application is "Unsigned Ready"
with the hope to handing the positive range representations, i.e, 0 to
4 MEG with support an extended a file size of an ideal 4 meg size in a
32 bit memory space, the problem I have seen is interfacing "other" API
functions, libraries and/or WIN32 API itself where its still working
and sometimes naturally so in an +/- signed integer world.

In short, for example, once upon a time when we documented our
requirements and limits, it use to say among its list:

o Up to 4 GigaByte file sizes

That was based on our persistent and consistent usage of DWORD within
our own code.  It was a theoretical limit, but empirically  the limit
was 2 GIG because of the various external interfaces. So today, we use
the 2 gig limit in our support docs and do not bother taking about what
we theoretically can handle.

My Point?

As we move into the 64 bit world, the positive range would still be a
theory and not one to rely on in new 64 bit designs.  I guess that
will design on how an application interfaces with the outside world.

Of course, one might might suggest

and no one should have problems. So design with INT64 in
mind"

Two things come to mind:

- We thought that was the case when moving to 32 bit,
- Quoting the late George Carlin:



As as side issue.

One of my beefs with Microsoft and .NET, and I have to review this
issue again, but under .NET, natural LONG type is 64 bit!  I have to
review this again to see if it was all the .NET languages or one or
more, C# C++/.NET and/or VB, I seem to recall it was not consistent
and I thought that was a mistake to create confusion by using a long
time established LONG type keyword to be 64 bit still within a 32 bit
compiled world.  I can certainly understand the thinking of the design
team but it definitely was absent of established C/C++ 32 bit
engineering considerations.

--
HLS
  Tom Serface replied to Joseph M. Newcomer
15-Jan-10 01:38 AM
Joe,

My experience has been that if the buffer gets too large it starts to slow
down the operation.  In my case, I have to read all sizes of files and I have
found the optimal buffer to be around 16K (I think that is what OP was
using).

I use the SDK functions CreateFile, ReadFile, WriteFile, rather than MFC and
my copy routine is around 2-3x what the DOS copy command does (or CopyFile).
I had to write my own for a special purpose though because I have to glue
files back together that span multiple volumes, but I was happy to get
better performance than I was getting with CopyFile previously.

I think there is a trade-off somewhere, but I am not entirely sure where.  I
just did trial and error with different scenarios until I got it the best I
could.

Tom
  Tom Serface replied to Joseph M. Newcomer
15-Jan-10 01:40 AM
I think you need to be really careful not to use up all the real memory as
well so that you do not start swapping to disk.  That is a killer that you
do not even see coming, although 50MB should not be a problem on most modern
computers.

Tom
  Tom Serface replied to Hector Santos
15-Jan-10 01:42 AM
I still had the problem on XP and Vista.  I have not tried testing it on Win7
yet, but I should do that since managing mapped drives is a holy nightmare
especially if you are using services that cannot access them.

Tom
  Tom Serface replied to Peter Olcott
15-Jan-10 01:44 AM
Well, as someone pointed out, it may not be that big of a difference these
days.  I know that Microsoft is not a big fan of supporting mapped drives
any longer so they may not be putting a lot of effort into improving it.  It
was worth mentioning.

Tom
  Tom Serface replied to Peter Olcott
15-Jan-10 01:46 AM
That's what my test showed me as well.   To be fair, I am often copying
files from optical media (CD, DVD, Blu-ray) so my buffer (16K) works well in
that environment.

Tom
  Joseph M. Newcomer replied to Tom Serface
15-Jan-10 06:56 PM
It seems to be largely an artifact of delay-to-processing.  I have regularly read in massive
files and run all over them, and it is vastly easier than trying to do some of the
computations, particularly if you have to cross buffer boundaries.  I usually get away
with this because we are looking at files under 100MB, that is, small files.  In some
cases, little tiny files that do not exceed 10MB.  But for serious files, in the TB range,
I have typically used file mappings unless there is a motivation to allow concurrent access.
joe
  Joseph M. Newcomer replied to Hector Santos
15-Jan-10 07:03 PM
See below...

****
In the cases I deal with, either file works, or it does not.  If there is an error
anywhere, for any reason, the integrity of the file is suspect, and it needs to be
reconsidered.
****
****
Any time performance matters, asynchrony is good.  But in the case of file systems, they
are almost always synchronous reads even when asynchronous is requested (there is a KB
article on this, KB156932, so it takes serious effort to defeat this)
****
****
It would have been easier to answer if the correct data had been given in the original
question, which stated


Note that at no point does it suggest a DOS Copy command is being used; it could have been
copy/paste in Windows Explorer.  It could have been CopyFile.
joe
****
Joseph M. Newcomer [MVP]
email: newcomer@flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
  Hector Santos replied to Joseph M. Newcomer
15-Jan-10 11:37 PM
Hi Joe,

I do not wish to spend much time on this, but my only point was that
there is major difference when going over the network or wire using a
single read 'huge" buffer call.  You got greater reliability,
performance and scalability issues to address.

--
HLS
Create New Account
help
Find and replace with vbscript C++ / VB Hi everybody I am pretty new to vbscript and right now I am sitting with 8 For Each strFile In strLocalFolder.Files strFileName = strLocalFolder & " \ " & strFile.Name If Left(strFile.Name, 7) = "PHFN007" Then Set objFile = objFSO.OpenTextFile(strFileName, ForReading) 'msgbox "anything" Do Until objFile.AtEndOfStream strLine 1 For Each strFile In strLocalFolder.Files strFileName = strLocalFolder & " \ " & strFile.Name If Left(strFile.Name, 7) = "PHFN007" Then Set objFile = objFSO.OpenTextFile(strFileName, ForReading) Do Until objFile.AtEndOfStream strLine = objFile.readLine objFile.WriteLine(strHeader) objFile.Write(strNewText) objFile.WriteLine(strFooter) objFile.Close msgbox "done" VBScript Discussions Windows XP (1) Visual Studio (1) Windows 7 (1) Office (1) Adobe (1) Linux (1) Vista (1) GetTickCount (1) {code removed} Here is how I would try this, though for very
HTA populating list box with subfolders of a folder C++ / VB Can I be helped? I am trying to explore to a folder and list the onclick = "GetDate(me)"> onclick = "GetDate(me)"> classid = "clsid:8E27C92B-1264-101C-8A2F-040224009C02"> VBScript Discussions Windows 7 (1) Vista (1) Date (1) XP (1) CreateObject (1) Application (1) Registry (1) GetDate (1) It looks like you pretty much what I wanting to do. I get error on what you returned. executed on an XP computer, do you have any suggestions? Thanks, Cisco Yes and no, I am finding scripts what I wanting to do. I get error on what you returned. executed on an XP computer, do you have any suggestions? I do not know without knowing the error. I which script you are having trouble with! It works fine for me. I am on XP. But I am using IE6. In most cases with XP+ those webpage utilities have to
port a MFC program to run with a new operating system (David Webber+others)? C++ / VB I am on my way to programming a financial software. Was just wondering, that how if it works there? Is it how difficult to convert a version which worked on Windows 98 to work on Win 7? I am not speaking about converting 32 bit program to 64, . . . just that the old MFC Discussions Joseph M. Joseph M. Newcomer (1) Dave David Webber Mozart Music Software (1) Windows XP (1) Visual Studio (1) PowerPoint (1) Joseph M. Newcomer (1) Windows 7 (1) Oracle (1) how much work it takes to compile older version to work on if it works there? Is it how difficult to convert a version which worked on Windows 98 to work on Win 7? < All versions of Mozart from v1 (released in 1994
Windows XP MFC program on Windows 7 issue C++ / VB I have a Windows XP application written in MFC. Can it runs on Windows 7 (32-bit)? Can Windows XP device driver work on Windows 7 (32-bits)? My
unicode and string C++ / VB Hi all, I was recently studying unicode and internationalization and some questions come to my Visual (1) Correct (1) I am going through the same kind of thing. In a Windows environment all the literals are 'UTF-16', but more accurately it is the subset of code point per element, because this is not the case in either Unix variants nor Windows. It is not the case because none are using Unicode encoding that allows that (UTF whatever representation you get from the system, to e.g. ICU. Finaly, cspisz is wrong, Windows does know UTF-16 (what he describes is UCS2, and that has been abandoned in windows about a decade ago). As for "best strategy", I would go for "play with the ever, forget to convert that to UTF-16 when using your stuff to interact with Windows. That would typically mean using MultiByteToWideChar with CP_UTF8 to get your wstring. Goran. - - [ See http a typical modern Linux `char` means UTF-8 and `wchar_t` means UTF-32, . . . while in Windows `char` definitiely means Windows ANSI (which is a locale specific encoding, defined by GetACP API function) and `wchar_t` definitely