Sunday, October 7, 2007

Is bfile faster than old erlang file?

Steve Vinoski is using klacke’s bfile module in his Wide Finder Project work, but I don't know why bfile should be faster than erlang OTP file. Well, then I measured. I tried Steve's read test and my test on my old home desktop (model name : AMD Athlon(tm) processor, stepping : 2, cpu MHz : 1199.805, cache size : 256 KB).
-export([start/1, start/2]).

scan_file(F, Readsize, Total) ->
   Rd = file:read(F, Readsize),
   case Rd of
       {ok, Bin} -> scan_file(F, Readsize, size(Bin)+Total);
       eof -> Total
scan_file(F, Readsize) -> scan_file(F, Readsize, 0).

start(File, Readsize) ->
   {ok, F} = file:open(File, [raw, binary, read]),
   T = scan_file(F, Readsize),
   io:format("read ~p bytes~n", [T]),
start(File) ->
   start(File, 512*1024).
And there are results here:
2> timer:tc(readold,start,["o1M.ap"]).
read 200995500 bytes
3> timer:tc(readold,start,["o1M.ap"]).
read 200995500 bytes
4> c(readold).
5> timer:tc(readold,start,["o1M.ap"]).
read 200995500 bytes
6> timer:tc(read,start,["o1M.ap"]).  
read 200995500 bytes
7> timer:tc(read,start,["o1M.ap"]).
read 200995500 bytes
8> timer:tc(read,start,["o1M.ap"]).
read 200995500 bytes
9> timer:tc(readold,start,["o1M.ap"]).
read 200995500 bytes
In generally, I don't know why erlang's file should be slower. I don't know why bfile is 45% slower than file on my old home desktop, but why should be faster anywhere? I tested it on Linux, may be bfile using BSD file implementation is faster on Darwin aka BSD clone? The file implementation is fast enough on my Erlang/OTP R11B-5.

1 comment:

Caoyuan Deng said...

Hi pichi,

I agree with your conclusion about the file reading. I think what bfile try to resolve is the bad file:get_line performance. bfile:get_line should be much faster than file:get_line.

In summary, the total problem is still the Erlang's processing performance for large Binary or List