Caching async method results

Classical approaches and quotations

From msdn, by Stephen Cleary

Asynchronous code is often used to initialize a resource that’s then cached and shared. There isn’t a built-in type for this, but Stephen Toub developed an AsyncLazy that acts like a merge of Task and Lazy. The original type is described on his blog, and an updated version is available in my AsyncEx library.

public class AsyncLazy<T> : Lazy<Task<T>>
{
    public AsyncLazy(Func<T> valueFactory) :
        base(() => Task.Factory.StartNew(valueFactory)) { }
    public AsyncLazy(Func<Task<T>> taskFactory) :
        base(() => Task.Factory.StartNew(() => taskFactory()).Unwrap()) { }
}

Context

Let’s say in our program we have one of these AsyncLazy instances:

static string LoadString() { … }
static AsyncLazy<string> m_data = new AsyncLazy<string>(LoadString);

Usage

Thus, we can write an asynchronous method that does:

string data = await m_data.Value;

The Lazy would be appropriate, but unfortunately it seems to lack the input parameter to index the result. The same issue was solved here where it is explained how to cache the results from a long-running, resource-intensive method, in case it is not async

Optimization

Let me suggest a couple of marginal optimization options, based on the following concerns.

often with locks, when you access them they’re uncontended, and in such cases you really want acquiring and releasing the lock to be as low-overhead as possible; in other words, accessing uncontended locks should involve a fast path

Since they’re just performance optimization tricks, I will leave them commented in the code so that you can measure their effects in your specific situation before.

  1. You need to test TryGetValue again after awaiting because another parallel process could have added that value in the meantime
  2. You don’t need to keep the lock while you’re awaiting

This balance of overhead vs cache misses was already pointed out in a previous answer

Obviously, there’s overhead keeping SemaphoreSlim objects around to prevent cache misses so it may not be worth it depending on the use case. But if guaranteeing no cache misses is important than this accomplishes that.

Main solution: the cache management

Regarding the cache expiration, I would suggest to add the creation DateTime to the value of the Dictionary (i.e. the time when the value is returned from GetSomethingTheLongWayAsync) and consequently discard the cached value after a fixed time span.

Find a draft below

    private static readonly ConcurrentDictionary<object, SemaphoreSlim> _keyLocks = new ConcurrentDictionary<object, SemaphoreSlim>();
    private static readonly ConcurrentDictionary<object, Tuple<string, DateTime>> _cache = new ConcurrentDictionary<object, Tuple<string, DateTime>>();

    private static bool IsExpiredDelete(Tuple<string, DateTime> value, string key)
    {
        bool _is_exp = (DateTime.Now - value.Item2).TotalMinutes > Expiration;
        if (_is_exp)
        {
            _cache.TryRemove(key, out value);
        }
        return _is_exp;
    }
    public async Task<string> GetSomethingAsync(string key)
    {
        Tuple<string, DateTime> cached;
        // get the semaphore specific to this key
        var keyLock = _keyLocks.GetOrAdd(key, x => new SemaphoreSlim(1));
        await keyLock.WaitAsync();
        try
        {
            // try to get value from cache
            if (!_cache.TryGetValue(key, out cached) || IsExpiredDelete(cached,key))
            {
                //possible performance optimization: measure it before uncommenting
                //keyLock.Release();
                string value = await GetSomethingTheLongWayAsync(key);
                DateTime creation = DateTime.Now;
                // in case of performance optimization
                // get the semaphore specific to this key
                //keyLock = _keyLocks.GetOrAdd(key, x => new SemaphoreSlim(1));
                //await keyLock.WaitAsync();
                bool notFound;
                if (notFound = !_cache.TryGetValue(key, out cached) || IsExpiredDelete(cached, key))
                {
                    cached = new Tuple<string, DateTime>(value, creation);
                    _cache.TryAdd(key, cached);
                }
                else
                {
                    if (!notFound && cached.Item2 < creation)
                    {
                        cached = new Tuple<string, DateTime>(value, creation);
                    _cache.TryAdd(key, cached);
                    }
                }
            }
        }
        finally
        {
            keyLock.Release();
        }
        return cached?.Item1;
    }

Please, adapt the above code to your specific needs.

Making it more generic

Finally you may want to generalize it a little bit.

By the way, notice that the Dictionary are not static since one could cache two different methods with the same signature.

public class Cached<FromT, ToT>
{
    private Func<FromT, Task<ToT>> GetSomethingTheLongWayAsync;
    public Cached (Func<FromT, Task<ToT>> _GetSomethingTheLongWayAsync, int expiration_min ) {
        GetSomethingTheLongWayAsync = _GetSomethingTheLongWayAsync;
        Expiration = expiration_min;
}

    int Expiration = 1;

    private ConcurrentDictionary<FromT, SemaphoreSlim> _keyLocks = new ConcurrentDictionary<FromT, SemaphoreSlim>();
    private ConcurrentDictionary<FromT, Tuple<ToT, DateTime>> _cache = new ConcurrentDictionary<FromT, Tuple<ToT, DateTime>>();

    private bool IsExpiredDelete(Tuple<ToT, DateTime> value, FromT key)
    {
        bool _is_exp = (DateTime.Now - value.Item2).TotalMinutes > Expiration;
        if (_is_exp)
        {
            _cache.TryRemove(key, out value);
        }
        return _is_exp;
    }
    public async Task<ToT> GetSomethingAsync(FromT key)
    {
        Tuple<ToT, DateTime> cached;
        // get the semaphore specific to this key
        var keyLock = _keyLocks.GetOrAdd(key, x => new SemaphoreSlim(1));
        await keyLock.WaitAsync();
        try
        {
            // try to get value from cache
            if (!_cache.TryGetValue(key, out cached) || IsExpiredDelete(cached, key))
            {
                //possible performance optimization: measure it before uncommenting
                //keyLock.Release();
                ToT value = await GetSomethingTheLongWayAsync(key);
                DateTime creation = DateTime.Now;
                // in case of performance optimization
                // get the semaphore specific to this key
                //keyLock = _keyLocks.GetOrAdd(key, x => new SemaphoreSlim(1));
                //await keyLock.WaitAsync();
                bool notFound;
                if (notFound = !_cache.TryGetValue(key, out cached) || IsExpiredDelete(cached, key))
                {
                    cached = new Tuple<ToT, DateTime>(value, creation);
                    _cache.TryAdd(key, cached);
                }
                else
                {
                    if (!notFound && cached.Item2 < creation)
                    {
                        cached = new Tuple<ToT, DateTime>(value, creation);
                        _cache.TryAdd(key, cached);
                    }
                }
            }
        }
        finally
        {
            keyLock.Release();
        }
        return cached.Item1;
    }

}

For a generic FromT an IEqualityComparer is needed for the Dictionary

Usage/Demo

    private static async Task<string> GetSomethingTheLongWayAsync(int key)
    {
        await Task.Delay(15000);
        Console.WriteLine("Long way for: " + key);
        return key.ToString();
    }

    static void Main(string[] args)
    {
        Test().Wait();
    }

    private static async Task Test()
    {
        int key;
        string val;
        key = 1;
        var cache = new Cached<int, string>(GetSomethingTheLongWayAsync, 1);
        Console.WriteLine("getting " + key);
        val = await cache.GetSomethingAsync(key);
        Console.WriteLine("getting " + key + " resulted in " + val);

        Console.WriteLine("getting " + key);
        val = await cache.GetSomethingAsync(key);
        Console.WriteLine("getting " + key + " resulted in " + val);

        await Task.Delay(65000);

        Console.WriteLine("getting " + key);
        val = await cache.GetSomethingAsync(key);
        Console.WriteLine("getting " + key + " resulted in " + val);
        Console.ReadKey();
    }

Sophisticated alternatives

There are also more advanced possibilities like the overload of GetOrAdd that takes a delegate and Lazy objects to ensure that a generator function is called only once (instead of semaphores and locks).

   public class AsyncCache<FromT, ToT>
    {
        private Func<FromT, Task<ToT>> GetSomethingTheLongWayAsync;
        public AsyncCache(Func<FromT, Task<ToT>> _GetSomethingTheLongWayAsync, int expiration_min)
        {
            GetSomethingTheLongWayAsync = _GetSomethingTheLongWayAsync;
            Expiration = expiration_min;
        }

        int Expiration;

        private ConcurrentDictionary<FromT, Tuple<Lazy<Task<ToT>>, DateTime>> _cache =
            new ConcurrentDictionary<FromT, Tuple<Lazy<Task<ToT>>, DateTime>>();

        private bool IsExpiredDelete(Tuple<Lazy<Task<ToT>>, DateTime> value, FromT key)
        {
            bool _is_exp = (DateTime.Now - value.Item2).TotalMinutes > Expiration;
            if (_is_exp)
            {
                _cache.TryRemove(key, out value);
            }
            return _is_exp;
        }
        public async Task<ToT> GetSomethingAsync(FromT key)
        {
            var res = _cache.AddOrUpdate(key,
                t =>  new Tuple<Lazy<Task<ToT>>, DateTime>(new Lazy<Task<ToT>>(
                      () => GetSomethingTheLongWayAsync(key)
                    )
                , DateTime.Now) ,
                (k,t) =>
                {
                    if (IsExpiredDelete(t, k))
                    {
                        return new Tuple<Lazy<Task<ToT>>, DateTime>(new Lazy<Task<ToT>>(
                      () => GetSomethingTheLongWayAsync(k)
                    ), DateTime.Now);
                    }
                    return t;
                }

                );
            return await res.Item1.Value;
        }

Same usage, just replace AsyncCache instead of Cached.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s