ppe memcpy

c94wjpn · Dec 7, 2007

What is the fastest way to get the ppe to copy a big (1MB) block from one memory location to another? This does not involve spu memories. I'm using memcpy at the moment and it appears to manage 700Mb/sec which is phenomenally slow I think.

I mean, a rough estimate is that it should manage to r/w a register every 2 clock cycles. If a reg is 4 bytes, then it should manage 6Gb / sec. so I've lost a factor of 10 somewhere.

Can I use dma hardware to move memory around the ppe's memory.?

Any ideas? If you don't have anything helpful to say, please don't reply.

patsu · Dec 7, 2007

Have you checked these out ?

"PPE RAM Latency" discussion on IBM Cell forum
http://www-128.ibm.com/developerworks/forums/thread.jspa?threadID=163046
(Can't you push [all] the work to SPU ?)

libfreevec (http://www.freevec.org)
"libfreevec is a free (LGPL) library with hand-optimized replacement routines for GLIBC, such as memcpy(), strlen(), etc. These routines have been written specifically to take advantage of the AltiVec unit (a.k.a Velocity Engine or VMX), and will only work on processors that include this unit. This means they will not work on older processors, such as 603, 604, 750 (G3) or the POWER family of CPUs."

c94wjpn · Dec 11, 2007

patsu said:
Have you checked these out ?

"PPE RAM Latency" discussion on IBM Cell forum
http://www-128.ibm.com/developerworks/forums/thread.jspa?threadID=163046

this has confused me completely. According to my ibm docs, the latency of stqd (store) is 6, and the same for an lqd (load). The discussion seems to be talking about RAM latencies of 10x that, and suggests that the RAM is on a slower clock than the cell. What memory caches does the Cell have and how do I control them?

Could someone clarify the situation with a few meaningful sentences in English?

ADEX · Dec 11, 2007

It'll depending on how you're trying to move the memory.
If you're reading a long or a long long per loop iteration it'll not work very fast.

It's best to use vectors and use them all in one go.
Declare 32 vector variables then setup a loop.

per iteration:
Use a pointer to fill them all.
Use another pointer to empty them all.
You can use a second thread to do the same thing with another part of the memory block.

That should be a lot faster but be careful of unaligned memory at the beginning and end.

It'll be easier and probably faster on an SPE though.

Also, look at the LibFreeVec lib.

BTW check how your timing works, I find compilers tend to mess with this no end.

seebs · Dec 12, 2007

I'm a bit unclear on how the SPE would be doing it; just DMA to and from local buffers?

You'd think that it'd be possible to trick the DMA engine into copying from memory to memory.

Laurent06 · Dec 14, 2007

I managed to get about 4 GB/s memcpy using only the PPE.

The trick is to manually unroll ld/st 64 (or was it 128? I need to turn on my PS3

) bit instructions and inserting cache preload instructions.

popper · Dec 15, 2007

c94wjpn said:
What is the fastest way to get the ppe to copy a big (1MB) block from one memory location to another? This does not involve spu memories. I'm using memcpy at the moment and it appears to manage 700Mb/sec which is phenomenally slow I think.

I mean, a rough estimate is that it should manage to r/w a register every 2 clock cycles. If a reg is 4 bytes, then it should manage 6Gb / sec. so I've lost a factor of 10 somewhere.

Can I use dma hardware to move memory around the ppe's memory.?

Any ideas? If you don't have anything helpful to say, please don't reply.

(thats not very nice)

go and ask and comment on this thread, gunnar and markos etc can help readers, as can you and many readers here perhaps.

http://www.powerdeveloper.org/forums/viewtopic.php?t=1426&postdays=0&postorder=asc&start=0
Gunnar PPC linux patch

and dont forget to contribute your code, comments, test results etc on this thread too
http://www.powerdeveloper.org/forums/viewtopic.php?t=1453

markos said:

FWIW, AltiVec could do even more! Here are the results of the latest memcpy in freevec:

Code:

$ ./bench_memcpy -s -v -t 1 -l 1000000 -M 20000
Test: memcpy
Will do scalar (glibc) tests
Will do vector (AltiVec/VMX) tests
Number of threads       : 1
Number of loops         : 1000000
Minimum buffer size     : 10
Maximum buffer size     : 20000

Scalar memcpy()

title = memcpy_scalar, minsize = 10, maxsize = 20000
process = 1/1, size = 10, arraysize = 629145
1/1                   10 bytes            0.11 sec       86.70 MB/s     0/0
process = 1/1, size = 20, arraysize = 314572
1/1                   20 bytes            0.12 sec      158.95 MB/s     0/0
process = 1/1, size = 40, arraysize = 157286
1/1                   40 bytes            0.14 sec      272.48 MB/s     0/0
process = 1/1, size = 80, arraysize = 78643
1/1                   80 bytes            0.17 sec      448.79 MB/s     0/0
process = 1/1, size = 160, arraysize = 39321
1/1                  160 bytes            0.23 sec      663.43 MB/s     0/0
process = 1/1, size = 320, arraysize = 19660
1/1                  320 bytes            0.35 sec      871.93 MB/s     0/0
process = 1/1, size = 640, arraysize = 9830
1/1                  640 bytes            0.59 sec      1034.49 MB/s    0/0
process = 1/1, size = 1280, arraysize = 4915
1/1                 1280 bytes            1.08 sec      1130.28 MB/s    0/0
process = 1/1, size = 2560, arraysize = 2457
1/1                 2560 bytes            2.04 sec      1196.77 MB/s    0/0
process = 1/1, size = 5120, arraysize = 1228
1/1                 5120 bytes            3.98 sec      1226.84 MB/s    0/0
process = 1/1, size = 10240, arraysize = 614
1/1                10240 bytes            7.84 sec      1245.62 MB/s    0/0

AltiVec memcpy()

title = memcpy_vector, minsize = 10, maxsize = 20000
process = 1/1, size = 10, arraysize = 629145
1/1                   10 bytes            0.07 sec      136.24 MB/s     0/0
process = 1/1, size = 20, arraysize = 314572
1/1                   20 bytes            0.11 sec      173.40 MB/s     0/0
process = 1/1, size = 40, arraysize = 157286
1/1                   40 bytes            0.12 sec      317.89 MB/s     0/0
process = 1/1, size = 80, arraysize = 78643
1/1                   80 bytes            0.11 sec      693.58 MB/s     0/0
process = 1/1, size = 160, arraysize = 39321
1/1                  160 bytes            0.12 sec      1271.57 MB/s    0/0
process = 1/1, size = 320, arraysize = 19660
1/1                  320 bytes            0.14 sec      2179.83 MB/s    0/0
process = 1/1, size = 640, arraysize = 9830
1/1                  640 bytes            0.20 sec      3051.76 MB/s    0/0
process = 1/1, size = 1280, arraysize = 4915
1/1                 1280 bytes            0.33 sec      3699.10 MB/s    0/0
process = 1/1, size = 2560, arraysize = 2457
1/1                 2560 bytes            0.56 sec      4359.65 MB/s    0/0
process = 1/1, size = 5120, arraysize = 1228
1/1                 5120 bytes            1.05 sec      4650.30 MB/s    0/0
process = 1/1, size = 10240, arraysize = 614
1/1                10240 bytes            2.02 sec      4834.47 MB/s    0/0

The code of the new library will RSN be released as I'm working on the final 1.0 version. I had to fix all the bugs so that it can be used at least as an LD_PRELOAD replacement until it gets into glibc proper.

popper · Dec 15, 2007

Laurent06 said:
I managed to get about 4 GB/s memcpy using only the PPE.

The trick is to manually unroll ld/st 64 (or was it 128? I need to turn on my PS3 ) bit instructions and inserting cache preload instructions.

perhaps you could go and post a bit of working code and what you did, what aspect of copying, test results graphs etc on the http://www.powerdeveloper.org/forums/viewtopic.php?t=1426&postdays=0&postorder=asc&start=15 thread.

it all helps increase the speed of takeup if people contribute their findings.

c94wjpn · Dec 17, 2007

ok thanks for the advice folks. The impression I get is that the memcpy I am using is not designed for this hardware and I need to write my own function or use libfreevec.

popper · Dec 17, 2007

its designed or more like ported and so functions as required on PPC, but it needs people to put the time in and improve its speed for everyones benefit as per the links you can follow and help out and receave help if your so inclined, its your choice.

sure libFreeVec will help you, as will talking with Gunnar and Markos if your after understanding the best code you might use in your own code functions, and in the process you might have some fun and help improve the general PPC linux speed for all users in time.

just read the thread to get the idea of how they do it and ask questions,markos will probably have you test the pre 1.0 or you might prod him to get stuck in and add the latest ideas coming to light (if you have an idea to improve it, tell it)and increasing speed even more.

peoples code snippets and benchmarking for better PPC/PS3 performance is always welcome there and here too when people wake on and post interesting ideas/thoughts etc.

however, its been very quiet around here/there of late, is everyone hibernating for the winter

its a shame, but it seems that other than the Amiga PPC lads no one seems interested in tuning/patching the current PPC/PS3 linux and apps for potentially far better speed, its going to take a long time at this rate if we cant find new people that enjoy the challenge and have some fun with it.... perhaps this cristmas will be different and new people can find some time to help out or just hang out and teach and drop some advanced working code snippets ...