One of the easiest cloud design pattern that one can try out is the Retry Pattern. I wanted to show how to use an Retry Pattern using Polly in C# as a example. So what does the Retry Pattern achieves?

Problem Statement – What is the issue the pattern solves?

When building applications you always have some sort of outside/external service including another MicroService that you have to consume or call. Sometimes there could be momentary loss of network connectivity, or a temporary unavailability, or timeouts that occur when that service is busy. You may be calling a database or a restful service that may be busy and fail but if you try back again it will pass. These types of faults are usually self-correcting, and most of the time require some type of delay in calling it again, which will have a success response.

Retry Pattern

• Enable an application to handle transient failures
• When the applications tries to connect to a service or network resource
• By transparently retrying a failed operation
• Improves the stability of your application

Typical Application

Below is a typical application diagram, where you a service or web app.




TypicalApplication

But when the connection to the service fails we usually get an error on our application.




Typical-Application-Network-Failure

When to use Retry Pattern

• Use retry for only transient failure that is more than likely to resolve themselves quickly
• Match the retry policies with the application
• Otherwise use the circuit break pattern

When not to use Retry Pattern

• Don’t cause a chain reaction to all components
• For internal exceptions caused by business logic
• Log all retry attempts to the service

Sample Code

Below is a sample dotnet core Console Application that shows the usage using Polly. The code tries to call https://httpbin.org/status/200,408 with a POST which gives us a status of 200 or 408 randomly. First, lets create our code and add the package Polly into it.

Shell

$mkdir RetryPattern $cd RetryPattern #create console app $dotnet new console The template "Console Application" was created successfully. Processing post-creation actions... Running 'dotnet restore' on C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj... Restoring packages for C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj... Generating MSBuild file C:\Users\Taswar\Documents\GitHub\RetryPattern\obj\RetryPattern.csproj.nuget.g.props. Generating MSBuild file C:\Users\Taswar\Documents\GitHub\RetryPattern\obj\RetryPattern.csproj.nuget.g.targets. Restore completed in 1.18 sec for C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj. Restore succeeded. #add the polly package $dotnet add package Polly Writing C:\Users\Taswar\AppData\Local\Temp\tmp4FCB.tmp info : Adding PackageReference for package 'Polly' into project 'C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj'. log : Restoring packages for C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj... info : GET https://www.nuget.org/api/v2/FindPackagesById()?id='Polly'&semVerLevel=2.0.0 info : OK https://www.nuget.org/api/v2/FindPackagesById()?id='Polly'&semVerLevel=2.0.0 2341ms info : GET https://www.nuget.org/api/v2/package/Polly/6.1.2 info : OK https://www.nuget.org/api/v2/package/Polly/6.1.2 2254ms log : Installing Polly 6.1.2. info : Package 'Polly' is compatible with all the specified frameworks in project 'C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj'. info : PackageReference for package 'Polly' version '6.1.2' added to file 'C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj'. info : Committing restore... info : Writing lock file to disk. Path: C:\Users\Taswar\Documents\GitHub\RetryPattern\obj\project.assets.json log : Restore completed in 7.77 sec for C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj. #launch vscode $code .

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

$mkdir RetryPattern $cd RetryPattern #create console app $dotnet new console The template "Console Application" was created successfully.   Processing post-creation actions... Running 'dotnet restore' on C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj...   Restoring packages for C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj...   Generating MSBuild file C:\Users\Taswar\Documents\GitHub\RetryPattern\obj\RetryPattern.csproj.nuget.g.props.   Generating MSBuild file C:\Users\Taswar\Documents\GitHub\RetryPattern\obj\RetryPattern.csproj.nuget.g.targets.   Restore completed in 1.18 sec for C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj.   Restore succeeded.   #add the polly package $dotnet add package Polly   Writing C:\Users\Taswar\AppData\Local\Temp\tmp4FCB.tmp info : Adding PackageReference for package 'Polly' into project 'C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj'. log  : Restoring packages for C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj... info :   GET https://www.nuget.org/api/v2/FindPackagesById()?id='Polly'&semVerLevel=2.0.0 info :   OK https://www.nuget.org/api/v2/FindPackagesById()?id='Polly'&semVerLevel=2.0.0 2341ms info :   GET https://www.nuget.org/api/v2/package/Polly/6.1.2 info :   OK https://www.nuget.org/api/v2/package/Polly/6.1.2 2254ms log  : Installing Polly 6.1.2. info : Package 'Polly' is compatible with all the specified frameworks in project 'C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj'. info : PackageReference for package 'Polly' version '6.1.2' added to file 'C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj'. info : Committing restore... info : Writing lock file to disk. Path: C:\Users\Taswar\Documents\GitHub\RetryPattern\obj\project.assets.json log  : Restore completed in 7.77 sec for C:\Users\Taswar\Documents\GitHub\RetryPattern\RetryPattern.csproj.   #launch vscode $code .

Without Polly

We will write a sample application that will call the the web service without polly to get 408 errors.


I am not using a logger but you should be using a logger when you do retry pattern.
C#

using System; using System.Net.Http; using System.Threading.Tasks; namespace RetryPattern { class Program { static async Task Main(string[] args) { //create the http client var httpClient = new HttpClient(); //call the httpbin service var response = await httpClient.PostAsync("https://httpbin.org/status/200,408", null); if (response.IsSuccessStatusCode) Console.WriteLine("Response was successful with 200"); else Console.WriteLine($"Response failed. Status code {response.StatusCode}"); } } }

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

using System; using System.Net.Http; using System.Threading.Tasks;   namespace RetryPattern {     class Program     {         static async Task Main(string[] args)         {             //create the http client             var httpClient = new HttpClient();               //call the httpbin service             var response = await httpClient.PostAsync("https://httpbin.org/status/200,408", null);                      if (response.IsSuccessStatusCode)                 Console.WriteLine("Response was successful with 200");             else                 Console.WriteLine($"Response failed. Status code {response.StatusCode}");         }     } }

If you have issues with 7.0 C# change your csproj to include <LangVersion>7.1</LangVersion>

After couple of runs you will see it response back with 408 RequestTimeout

Shell

> dotnet run Response was successful with 200 > dotnet run Response was successful with 200 > dotnet run Response failed. Status code RequestTimeout

1 2 3 4 5 6 7 8 9

> dotnet run Response was successful with 200     > dotnet run Response was successful with 200   > dotnet run Response failed. Status code RequestTimeout

Using Polly

Now we will introduce Polly into our code with an incremental delay of 1 second to 3 seconds and lastly 9 seconds.

C#

using System; using System.Net.Http; using System.Threading.Tasks; using Polly; namespace RetryPattern { class Program { static async Task Main(string[] args) { //create the http client var httpClient = new HttpClient(); //call the httpbin service with Polly var response = await Policy .HandleResult<HttpResponseMessage>(message => !message.IsSuccessStatusCode) .WaitAndRetryAsync(new[] { TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3), TimeSpan.FromSeconds(9) }, (result, timeSpan, retryCount, context) => { Console.WriteLine($"Request failed with {result.Result.StatusCode}. Retry count = {retryCount}. Waiting {timeSpan} before next retry. "); }) .ExecuteAsync(() => httpClient.PostAsync("https://httpbin.org/status/200,408", null)); if (response.IsSuccessStatusCode) Console.WriteLine("Response was successful with 200"); else Console.WriteLine($"Response failed. Status code {response.StatusCode}"); } } }

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

using System; using System.Net.Http; using System.Threading.Tasks; using Polly;   namespace RetryPattern {     class Program     {         static async Task Main(string[] args)         {             //create the http client             var httpClient = new HttpClient();               //call the httpbin service with Polly             var response = await Policy                             .HandleResult<HttpResponseMessage>(message => !message.IsSuccessStatusCode)                             .WaitAndRetryAsync(new[]                             {                                 TimeSpan.FromSeconds(1),                                 TimeSpan.FromSeconds(3),                                 TimeSpan.FromSeconds(9)                             }, (result, timeSpan, retryCount, context) => {                                 Console.WriteLine($"Request failed with {result.Result.StatusCode}. Retry count = {retryCount}. Waiting {timeSpan} before next retry. ");                             })                             .ExecuteAsync(() => httpClient.PostAsync("https://httpbin.org/status/200,408", null));                        if (response.IsSuccessStatusCode)                 Console.WriteLine("Response was successful with 200");             else                 Console.WriteLine($"Response failed. Status code {response.StatusCode}");         }     } }

Output

Below you will see three runs of the application with sample output.

Shell

#First Run > dotnet run Request failed with RequestTimeout. Retry count = 1. Waiting 00:00:01 before next retry. Request failed with RequestTimeout. Retry count = 2. Waiting 00:00:03 before next retry. Response was successful with 200 #second run > dotnet run Request failed with RequestTimeout. Retry count = 1. Waiting 00:00:01 before next retry. Request failed with RequestTimeout. Retry count = 2. Waiting 00:00:03 before next retry. Request failed with RequestTimeout. Retry count = 3. Waiting 00:00:09 before next retry. Response failed. Status code RequestTimeout #third run > dotnet run Request failed with RequestTimeout. Retry count = 1. Waiting 00:00:01 before next retry. Response was successful with 200

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

#First Run > dotnet run Request failed with RequestTimeout. Retry count = 1. Waiting 00:00:01 before next retry. Request failed with RequestTimeout. Retry count = 2. Waiting 00:00:03 before next retry. Response was successful with 200   #second run > dotnet run Request failed with RequestTimeout. Retry count = 1. Waiting 00:00:01 before next retry. Request failed with RequestTimeout. Retry count = 2. Waiting 00:00:03 before next retry. Request failed with RequestTimeout. Retry count = 3. Waiting 00:00:09 before next retry. Response failed. Status code RequestTimeout   #third run > dotnet run Request failed with RequestTimeout. Retry count = 1. Waiting 00:00:01 before next retry. Response was successful with 200

Summary

As you can see Retry Pattern is quite useful for transient and self correcting failure, not to mention it is quite simple to implement in C# with the help of Polly. If you are looking for Java solutions you can look at Hysterix or even roll your own.