Hai,
2011/10/7 Raja Subramanian <rajasuperman at gmail.com>
Is there any way to store some data in SWAP (but not in RAM) and get the?
data from the SWAP partition whenever we need it in Python or C language
yup. It may be slower to access the swap area rather than RAM, but it isdisk.
much faster than access the data of files which are stored in the hard
Use mmap to map a file directly into your processes address space. When
you reference the memory address, the kernel will read/write data from the
file using the same mechanisms it uses to swap. You will bypass all file
IO buffering and have direct access to the files. This is how all DB
engines
handle IO. mmap() vs read() will give you a 3 x speedup for IO bound jobs.
Advanced Programming in the Unix Environment by Stevens has a good
example of file copy implemented using mmap.
Thanks for above suggestions.
Mechanisms to speed up scientific code:
1. Make fewer passes over your data. Ideally you want to read the
data once, complete all processing and never have to read it again.
Do not make several passes over data doing a little processing each
time.
2. Use efficient IO like mmap.
3. Use the correct algorithms suitable for your data size and work.
4. Profile your code to understand which step takes the most time.
Then start fine tuning.
5. Run a 64bit OS, you'll sometimes need to use more RAM if you
want faster performance. It's a trade-off.
6. Code the hot spots in C if you need more speed.