[Ilugc] Can we access the SWAP SPACE in Python
- From: rajasuperman@xxxxxxxxx (Raja Subramanian)
- Date: Fri, 7 Oct 2011 18:09:32 +0530
On Fri, Oct 7, 2011 at 5:16 PM, Arulalan T <arulalant at gmail.com> wrote:
Is there any way to store the data (temporarily) in the SWAP SPACE instead
of RAM ?
No, it's best to let the kernel handle memory management. But read below.
Is there any way to store some data in SWAP (but not in RAM) and get the
data from the SWAP partition whenever we need it in Python or C language ?
yup. It may be slower to access the swap area rather than RAM, but it is
much faster than access the data of files which are stored in the hard disk.
Use mmap to map a file directly into your processes address space. When
you reference the memory address, the kernel will read/write data from the
file using the same mechanisms it uses to swap. You will bypass all file
IO buffering and have direct access to the files. This is how all DB engines
handle IO. mmap() vs read() will give you a 3 x speedup for IO bound jobs.
Advanced Programming in the Unix Environment by Stevens has a good
example of file copy implemented using mmap.
It takes 2 days to complete this job and eats 3GB RAM memory.
Mechanisms to speed up scientific code:
1. Make fewer passes over your data. Ideally you want to read the
data once, complete all processing and never have to read it again.
Do not make several passes over data doing a little processing each
time.
2. Use efficient IO like mmap.
3. Use the correct algorithms suitable for your data size and work.
4. Profile your code to understand which step takes the most time.
Then start fine tuning.
5. Run a 64bit OS, you'll sometimes need to use more RAM if you
want faster performance. It's a trade-off.
6. Code the hot spots in C if you need more speed.
One of the reasons is ' 2 x 360 x 720 ( = 518400 ) times needs to access
python file object to retrieve the data of corresponding latitude,
longitude'.
OS, python libs will cache IO. So you will not hit disk each time you
read the data.
- Raja
Other related posts: