Take the 2-minute tour ×

I have a long-running process which writes a lot of stuff in a file. The result should be everything or nothing, so I'm writing to a temporary file and rename it to the real name at the end. Currently, my code is like this:

filename = 'whatever'
tmpname = 'whatever' + str(time.time())

with open(tmpname, 'wb') as fp:
    fp.write(more stuff)

if os.path.exists(filename):
os.rename(tmpname, filename)

I'm not happy with that for several reasons:

  • it doesn't clean up properly if an exception occurs
  • it ignores concurrency issues
  • it isn't reusable (I need this in different places in my program)

Any suggestions how to improve my code? Is there a library that can help me out?

share|improve this question
To make it reusable, you can turn it into a context manager. –  Janne Karila Aug 17 '12 at 13:48

4 Answers 4

up vote 5 down vote accepted

You can use Python's tempfile module to give you a temporary file name. It can create a temporary file in a thread safe manner rather than making one up using time.time() which may return the same name if used in multiple threads at the same time.

As suggested in a comment to your question, this can be coupled with the use of a context manager. You can get some ideas of how to implement what you want to do by looking at Python tempfile.py sources.

The following code snippet may do what you want. It uses some of the internals of the objects returned from tempfile.

  • Creation of temporary files is thread safe.
  • Renaming of files upon successful completion is atomic, at least on Linux. There isn't a separate check between os.path.exists() and the os.rename() which could introduce a race condition. For an atomic rename on Linux the source and destinations must be on the same file system which is why this code places the temporary file in the same directory as the destination file.
  • The RenamedTemporaryFile class should behave like a NamedTemporaryFile for most purposes except when it is closed using the context manager, the file is renamed.


import tempfile
import os

class RenamedTemporaryFile(object):
    A temporary file object which will be renamed to the specified
    path on exit.
    def __init__(self, final_path, **kwargs):
        tmpfile_dir = kwargs.pop('dir', None)

        # Put temporary file in the same directory as the location for the
        # final file so that an atomic move into place can occur.

        if tmpfile_dir is None:
            tmpfile_dir = os.path.dirname(final_path)

        self.tmpfile = tempfile.NamedTemporaryFile(dir=tmpfile_dir, **kwargs)
        self.final_path = final_path

    def __getattr__(self, attr):
        Delegate attribute access to the underlying temporary file object.
        return getattr(self.tmpfile, attr)

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type is None:
            self.tmpfile.delete = False
            result = self.tmpfile.__exit__(exc_type, exc_val, exc_tb)
            os.rename(self.tmpfile.name, self.final_path)
            result = self.tmpfile.__exit__(exc_type, exc_val, exc_tb)

        return result

You can then use it like this:

with RenamedTemporaryFile('whatever') as f:

During writing, the contents go to a temporary file, on exit the file is renamed. This code will probably need some tweaks but the general idea should help you get started.

share|improve this answer
+1 for NamedTemporaryFile and the context manager. Note: you might loose data without os.fsync() call. See my answer –  J.F. Sebastian Aug 17 '12 at 20:37
@J.F.Sebastian Using os.fsync() may get you closer to a fault tolerant system at the expense of performance, but fsync() still doesn't guarantee that data is on disk, especially for storage with write caches. I'd probably choose to use/not use fsync() depending on the application. blogs.gnome.org/alexl/2009/03/16/ext4-vs-fsync-my-take has some interesting info on this. –  Austin Phillips Aug 18 '12 at 0:24
Philips: the OP specifically asked about fault tolerant, not the fastest way. POSIX allows int fsync(int) { return 0; } but most implementations do something useful. From your link: "Some filesystems may order the metadata writes such that the rename is written to disk, but the contents of the new file are not yet on disk. If we crash at this point this is detected on mount and the file is truncated to 0 bytes. Calling fsync() guarantees that this does not happen. [ext4]". –  J.F. Sebastian Aug 18 '12 at 3:03
Though fsync() might not be necessary for some kernels in the rename() case. –  J.F. Sebastian Aug 18 '12 at 3:04
Thanks, this helps a lot! J.F. is right, I don't care about performance much in this case, so one extra function call is fine. –  georg Aug 18 '12 at 8:52

You could use the lockfile module to lock the file while you are writing to it. Any subsequent attempt to lock it will block until the lock from the previous process/thread has been released.

from lockfile import FileLock
with FileLock(filename):
    #open your file here....

This way, you circumvent your concurrency issues and do not have to clean up any leftover file if an exception occurs.

share|improve this answer
+1 the lockfile link. Note: for threads in the same process threading.Lock could be used (or multiprocessing.Lock for processing created via multiprocessing module). –  J.F. Sebastian Aug 17 '12 at 20:42
Thanks, that module appears to be really useful. –  georg Aug 18 '12 at 8:53

The with construct is useful for cleaning up on exit, but not for the commit/rollback system you want. A try/except/else block can be used for that.

You also should use a standard way for creating the temporary file name, for example with the tempfile module.

And remember to fsync before rename

Below is the full modified code:

import time, os, tempfile

def begin_file(filepath):
    (filedir, filename) = os.path.split(filepath)
    tmpfilepath = tempfile.mktemp(prefix=filename+'_', dir=filedir)
    return open(os.path.join(filedir, tmpfilepath), 'wb') 

def commit_file(f):
    tmppath = f.name
    (filedir, tmpname) = os.path.split(tmppath)
    origpath = os.path.join(filedir,tmpname.split('_')[0])


    if os.path.exists(origpath):
    os.rename(tmppath, origpath)

def rollback_file(f):
    tmppath = f.name

fp = begin_file('whatever')
share|improve this answer
+1 for the fsync link. Note: you might need to f.flush() before os.fsync(f). The code might be simplified. –  J.F. Sebastian Aug 17 '12 at 20:40

To write all or nothing to a file reliably:

import os
from contextlib import contextmanager
from tempfile   import NamedTemporaryFile

if not hasattr(os, 'replace'):
    os.replace = os.rename #NOTE: it won't work for existing files on Windows

def FaultTolerantFile(name):
    dirpath, filename = os.path.split(name)
    # use the same dir for os.rename() to work
    with NamedTemporaryFile(dir=dirpath, prefix=filename, suffix='.tmp') as f:
        yield f
        f.flush()   # libc -> OS
        os.fsync(f) # OS -> disc (note: on OSX it is not enough)
        f.delete = False # don't delete tmp file if `replace()` fails
        os.replace(f.name, name)

See also Is rename() without fsync() safe? (mentioned by @Mihai Stan)


with FaultTolerantFile('very_important_file') as file:
    file.write('either all ')
    file.write('or nothing is written')

To implement missing os.replace() you could call MoveFileExW(src, dst, MOVEFILE_REPLACE_EXISTING) (via win32file or ctypes modules) on Windows.

In case of multiple threads you could call queue.put(data) from different threads and write to file in a dedicated thread:

 for data in iter(queue.get, None):

queue.put(None) breaks the loop.

As an alternative you could use locks (threading, multiprocessing, filelock) to synchronize access:

def write(self, data):
    with self.lock:
share|improve this answer
Thanks, very useful stuff. Some new concepts to learn (trying to wrap my head around how @contextmanager works). –  georg Aug 18 '12 at 8:55
@thg435: the docstring might explain it –  J.F. Sebastian Aug 18 '12 at 18:35

Your Answer


