Implementing fast file transfer with Socket.SendFile
When writing network applications, we usually have a need to implement file transfer between two hosts. For eg, imaging an FTP client, where the client is downloading or uploading a file from an FTP server. Similarly, you could be uploading an image file (probably a photo) as an attachment to a Blog or a website like Facebook or Flickr.Usually, file transfer is implemented as a Read/Write pattern, where you read from the source stream and write into the destination stream. Here the source stream is the stream constructed from the Socket, and the target stream is the file, or vice versa if the file is being transferred to a destination server.
The simple Read/Write pattern for file transfer is implemented as follows.
In the .NET framework, there is a better way to do file uploads, which is exposed through Socket.SendFile method. This method exposes the underlying Winsock API TransmitFile. This API is much more powerful and faster in terms of performance.
In order to check the performance difference, I wrote an application that compares the difference in performance between using the Read/Write pattern and the Socket.SendFile method.
Here is the test program:
A couple of things to note about this implementation:
1) It uses Message framing to frame file transfers, since it uses the same socket for multiple file transfers. I have used the techniques in Serializing data from .NET to Java to do this. Even though there is no Java app that is involved here, the techniques are the same.
2) The server just drains the incoming stream. It does not save the incoming data to a file. Since we are just interested in benchmarking the performance between the two Send implementations, we should be ok here.
3) The program, which is basically a perf harness, uses a Strategy pattern to change the SendFile method used. That way everything else remains the same, and it just changes the SendFile method to get performance numbers.
Perf Comparison
The following graph shows the performance with the simple Read/Write pattern for file transfer.
The following chart shows the performance when Socket.SendFile is used.
As you can see, there is a huge difference between the two, specially for 1M file size. With Socket.SendFile, it takes max 129ms for upload, whereas without this API, it takes 1000ms for upload. For smaller file sizes, there is not that much of a difference.
There is a huge variance in timings for the SendFile() method for 1M file size, but I havent been able to figure out the reason for that yet. Anyway, the fact that Socket.SendFile() is faster should not be impacted by that.
Hi. I got this error.
ReplyDeleteError 1 A namespace cannot directly contain members such as fields or methods D:\PortableSpeedometer\socket_sendfile\Program.cs 220 1 socket_sendfile
This error happens at line 220. Im building a tcp based bandwidth estimation by the way.
After removing/commenting line 220, i got this error
ReplyDeleteAn attempt was made to access a socket in a way forbidden by its access permissions at line 194.
thanks.
There a problem with your measurement.
ReplyDelete1. you should start the measurement on EXACTLY the process that you want to measure. Do not include the time to openfile, readfile, seeking etc.
2. calling multiple read will definitely cause overhead. you should increase the buffer to what you expect the buffer to allocate.
this is the new results
0 1024 0.0396
1 1024 0.0321
2 1024 0.0312
3 1024 0.0307
4 1024 0.0312
5 1024 0.0303
6 1024 0.0307
7 1024 0.0307
8 1024 0.0303
9 1024 0.0303
0 4096 0.111
1 4096 0.0237
2 4096 0.0499
3 4096 0.0433
4 4096 0.0466
5 4096 0.0443
6 4096 0.0471
7 4096 0.0466
8 4096 0.0447
9 4096 0.0461
0 8192 0.1147
1 8192 0.0685
2 8192 0.0802
3 8192 0.0685
4 8192 0.069
5 8192 0.076
6 8192 0.0783
7 8192 0.0681
8 8192 0.0667
9 8192 0.0685
0 16385 0.1651
1 16385 0.1423
2 16385 0.1185
3 16385 0.1166
4 16385 0.2062
5 16385 0.118
6 16385 0.1143
7 16385 0.1157
8 16385 0.0382
9 16385 0.1273
0 65536 0.3774
1 65536 0.3247
2 65536 0.2962
3 65536 0.286
4 65536 0.3102
5 65536 0.2939
6 65536 0.2911
7 65536 0.299
8 65536 0.293
9 65536 0.3004
0 1048576 4.2447
1 1048576 4.2545
2 1048576 4.4803
3 1048576 0.4712
4 1048576 0.5599
5 1048576 0.3028
6 1048576 0.3
7 1048576 0.705
8 1048576 0.3476
9 1048576 0.3046
0 1024 3.6661
1 1024 0.2407
2 1024 0.1871
3 1024 0.1801
4 1024 0.1983
5 1024 0.2081
6 1024 0.1833
7 1024 0.1908
8 1024 0.1847
9 1024 0.1927
0 4096 2.075
1 4096 0.265
2 4096 0.2309
3 4096 0.202
4 4096 0.1931
5 4096 0.1917
6 4096 0.1894
7 4096 0.1941
8 4096 0.1922
9 4096 0.1707
0 8192 2.7016
1 8192 0.2855
2 8192 0.223
3 8192 0.2197
4 8192 0.2123
5 8192 0.2277
6 8192 0.2146
7 8192 0.8249
8 8192 0.3364
9 8192 0.2538
0 16385 5.624
1 16385 0.3611
2 16385 0.2706
3 16385 0.2697
4 16385 0.2641
5 16385 0.2645
6 16385 0.2916
7 16385 0.4222
8 16385 0.3359
9 16385 0.2711
0 65536 6.8428
1 65536 0.6089
2 65536 0.5207
3 65536 0.5291
4 65536 0.5431
5 65536 0.5478
6 65536 0.5156
7 65536 0.5146
8 65536 1.1147
9 65536 0.6257
0 1048576 12.0945
1 1048576 5.2148
2 1048576 4.5559
3 1048576 4.6684
4 1048576 4.876
5 1048576 4.604
6 1048576 4.6913
7 1048576 4.5261
8 1048576 5.1495
9 1048576 4.6157
i initially thought that this could improve my network benchmarking system. but it doesnt. you should update your page or it will mislead other people.
i cannot send you the modified code. i dont have your mail. comment is limited to 4096 characters only. my mail is bukan dot ijam //at// gmail dote com.
ReplyDeleteI did an end to end comparison because TransmitFile (Winsock API) also reads the file and send it over the network. So, excluding the time to read the file will not yield an apples to apples comparison.
ReplyDeleteWhen I set the buffer size to 1 MB in SendFile1 it is about the same speed as SendFile2, thuough with less consistency. Doing that and using a 100 MB file to test, SendFile1 is slightly faster than SendFile2.
ReplyDelete