Cloud Design Patterns – Talk at Ottawa .NET Community

Taswar Bhatti November 10, 2017 No Comments

This week I gave a talk on Cloud Design Patterns at the Ottawa .NET Community. I wanted to share the sides here and will most likely write on articles on the topic using real world examples. Samples in C# and node.js, in AWS and Azure.

For the talk I went through, what Cloud Design Patterns are and mainly focused on the patterns below without using any platform specifications. (i.e cloud agnostics)

The External Configuration Pattern
The Cache Aside Pattern
The Federated Identity Pattern
The Valet Key Pattern
The Gatekeeper Pattern
The Circuit Breaker Pattern

For each of them I also went through when you should use the pattern and when not to use it, I also provided Cloud Solutions Offering that one can use to implement the pattern.

Enjoy the slides 🙂

Cloud Design Patterns from Taswar Bhatti, MVP

The Transcript:

Agenda

What are Patterns?
The External Configuration Pattern
The Cache Aside Pattern
The Federated Identity Pattern
The Valet Key Pattern
The Gatekeeper Pattern
The Circuit Breaker Pattern
The Retry Pattern
The Strangler Pattern

What are Patterns?

General reusable solution to a recurring problem
A template on how to solve a problem
Best practices
Patterns allow developers communicate with each other in well known and understand names for software interactions.

External Configuration Pattern
- Helps move configuration information out of the application deployment
- This pattern can provide for easier management and control of configuration data
- For sharing configuration data across applications and other application instances
Typical Application
Storing Configuration in file
Multiple application
Problems

Configuration becomes part of deployment
Multiple applications share the same configuration
Hard to have access control over the configuration

External Configuration Pattern
When to use the pattern

When you have shared configuration, multiple application
You want to manage configuration centrally by DevOps
Provide audit for each configuration

When not to use

When you only have a single application there is no need to use this pattern it will make things more complex

Cloud Solution Offerings

Azure Key Vault
Vault by Hashicorp
AWS KMS
Keywhiz

Cache Aside Pattern

Load data on demand into a cache from datastore
Helps improve performance
Helps in maintain consistency between data held in the cache and data in the underlying data store.

Typical Application
Cache Aside Pattern
When to use the pattern

Resource demand is unpredictable.
This pattern enables applications to load data on demand
It makes no assumptions about which data an application will require in advance

When not to use

Don’t use it for data that changes very often
Things to consider

Sometimes data can be changed from outside process
Have an expiry for the data in cache
When update of data, invalidate the cache before updating the data in database
Pre populate the data if possible

Cloud Offerings

Redis (Azure and AWS)
Memcache
Hazelcast
Elastic Cache (AWS)

Federated Identity Pattern

Delegate authentication to an external identity provider.
Simplify development, minimize the requirement for user administration
Improve the user experience of the application
Centralized providing MFA for user authentication

Typical Application
Problem

Complex development and maintenance (Duplicated code)
MFA is not an easy thing
User administration is a pain with access control
Hard to keep system secure
No single sign on (SSO) everyone needs to login again to different systems

Federated Identity Pattern
When to use

When you have multiple applications and want to provide SSO for applications
Federated identity with multiple partners
Federated identity in SAAS application

When not to use it

You already have a single application and have custom code
that allows you to login

Things to consider

The identity Server needs to be highly available
Single point of failure, must have HA
RBAC, identity server usually does not have authorization information
Claims and scope within the security auth token

Cloud Offerings

Azure AD
Gemalto STA and SAS
Amazon IAM
GCP Cloud IAM

Valet Key Pattern

Use a token that provides clients with restricted direct access to a specific resource
Provide offload data transfer from the application
Minimize cost and maximize scalability and performance

Typical Application Client App Storage
Problem
Valet Key Pattern
Client App Generate Token Limited Time And Scope Storage
When to use it

The application has limited resources
To minimize operational cost
Many interaction with external resources (upload, download)
When the data is stored in a remote data store or a different datacenter

When not to use it

When you need to transform the data before upload or download

Cloud Offerings

Azure Blob Storage
Amazon S3
GCP Cloud Storage

Gatekeeper Pattern

Using a dedicated host instance that acts as a broker between clients and services
Protect applications and services
Validates and sanitizes requests, and passes requests and data between them
Provide an additional layer of security, and limit the attack surface of the system

Typical Application
Problem
Gatekeeper Pattern
When to use it
- Sensitive information (Health care, Authentication)
- Distributed System where perform request validation separately
When not to use

• Performance vs security

Things to consider
- WAF should not hold any keys or sensitive information
- Use a secure communication channel
- Auto scale
- Endpoint IP address (when scaling application does the WAF know the new applications)
Circuit Breaker Pattern

To handle faults that might take a variable amount of time to recover
When connecting to a remote service or resource

Typical Application
Problem
When to use it

To prevent an application from trying to invoke a remote service or access a shared resource if this operation is highly likely to fail
Better user experience

When not to use

Handling access to local private resources in an application, such as in-memory data structure
Creates an overhead
Not a substitute for handling exceptions in the business logic of your applications

Libraries

• Polly (http://www.thepollyproject.org/)
• Netflix (Hystrix) https://github.com/Netflix/Hystrix/wiki

Retry pattern

Enable an application to handle transient failures
When the applications tries to connect to a service or network resource
By transparently retrying a failed operation

Typical Application has Network Failure
Retry Pattern

• Retry after 2, 5 or 10 seconds

When to use it

Use retry for only transient failure that is more than likely to resolve themselves quicky
Match the retry policies with the application
Otherwise use the circuit break pattern

When not to use it

Don’t cause a chain reaction to all components
For internal exceptions caused by business logic
Log all retry attempts to the service

Libraries

Roll your own code
Polly (http://www.thepollyproject.org/)
Netflix (Hystrix) https://github.com/Netflix/Hystrix/wiki

Strangler Pattern

Incrementally migrate a legacy system
Gradually replacing specific pieces of functionality with new applications and services
Features from the legacy system are replaced by new system features eventually
Strangling the old system and allowing you to decommission it

Monolith Application
When to use

Gradually migrating a back-end application to a new architecture

When not to use

When requests to the back-end system cannot be intercepted
For smaller systems where the complexity of wholesale replacement is low

Considerations

Handle services and data stores that are potentially used by both new and legacy systems.
Make sure both can access these resources side-by-side
When migration is complete, the strangler façade will either go away or evolve into an adaptor for legacy clients
Make sure the façade doesn't become a single point of failure or a performance bottleneck.

.NET, AWS, Azure, Cloud, Design

cloud design pattern

Redis for .NET Developers – Redis Sentinel

Taswar Bhatti October 3, 2017 No Comments

If you are serious in running Redis, you will want to run it under HA mode (High Availability). So far for learning purpose you can run a single instance of redis under your docker environment quite easily, but what if you need to run it in production? This is where Redis Sentinel comes in to play, let’s see what the official document of Redis Sentinel have to say.

Redis Sentinel

Redis Sentinel provides high availability for Redis. In practical terms this means that using Sentinel you can create a Redis deployment that resists without human intervention to certain kind of failures.

The main thing that Redis Sentinel provides is when there is a failure on master node, it will automatically choose a salve and promote it to a master. How does it do that, it basically periodically check the health and live of a Redis instance, it will also notify clients and slaves about the new master. The protocol used is gossip protocol with leader election algorithms. Sentinel also acts as a central source of authority for client discovery, clients are connected to Sentinel for the address of the master node.

Things that Sentinel doesn’t do are mainly:

Manage client connections
Store configurations changes to disk

Usually you would want to run your Sentinel on different servers than your Redis Server, for the very simple fact that you don’t want your monitoring software on the same server right? Can you imagine a sql server with sql monitoring on the same machine to tell you if its alive? What if the sql server machine goes down, then both goes down? One can potentially run Sentinel on client nodes and its best to not to run it on master nodes.

Here is a typical way of setting up Sentinel.

redis-sentinel

Unfortunately Redis StackExchange does not support Sentinel there are pull request that have the functionality (https://github.com/StackExchange/StackExchange.Redis/pull/692) but have not been merged into Redis StackExchange yet as we speak.

What are the alternative? You can roll your own RedisClient and have Sentinel support or manage it yourself in your code, or use Service Stack is another option.
If you are interested in rolling your own you may want to go through this code by PaulB from Stack Overflow provided
https://stackoverflow.com/questions/25385075/redis-failover-with-stackexchange-sentinel-from-c-sharp

The concepts remain the same as in ask sentinel for who the master is and then do the connection.

Note: The code is not mine its from the link above.

namespace Smartodds.Framework.Redis
{
    public class RedisClient : IDisposable
    {
        public RedisClient(RedisEnvironmentElement environment, Int32 databaseId)
        {
            m_ConnectTimeout = environment.ConnectTimeout;
            m_Timeout = environment.Timeout;
            m_DatabaseId = databaseId;
            m_ReconnectTime = environment.ReconnectTime;
            m_CheckSubscriptionsTime = environment.CheckSubscriptions;
            if (environment.TestWrite == true)
            {
                m_CheckWriteTime = environment.TestWriteTime;
            }

environment.Password.ToCharArray().ForEach((c) => m_Password.AppendChar(c));

foreach (var server in environment.Servers)
            {
                if (server.Type == ServerType.Redis)
                {
                    // will be ignored if sentinel servers are used
                    m_RedisServers.Add(new RedisConnection { Address = server.Host, Port = server.Port });
                }
                else
                {
                    m_SentinelServers.Add(new RedisConnection { Address = server.Host, Port = server.Port });
                }
            }
        }

public bool IsSentinel { get { return m_SentinelServers.Count > 0; } }

public IDatabase Database { get { return _Redis.GetDatabase(m_DatabaseId); } }

private ConnectionMultiplexer _Redis
        {
            get
            {
                if (m_Connecting == true)
                {
                    throw new RedisConnectionNotReadyException();
                }

ConnectionMultiplexer redis = m_Redis;
                if (redis == null)
                {
                    throw new RedisConnectionNotReadyException();
                }

return redis;
            }
        }

private ConnectionMultiplexer _Sentinel
        {
            get
            {
                if (m_Connecting == true)
                {
                    throw new RedisConnectionNotReadyException("Sentinel connection not ready");
                }

ConnectionMultiplexer sentinel = m_Sentinel;
                if (sentinel == null)
                {
                    throw new RedisConnectionNotReadyException("Sentinel connection not ready");
                }

return sentinel;
            }
        }

public void RegisterSubscription(string channel, Action<RedisChannel, RedisValue> handler, Int32 maxNoReceiveSeconds)
        {
            m_Subscriptions.Add(channel, new RedisSubscription
            {
                Channel = channel,
                Handler = handler,
                MaxNoReceiveSeconds = maxNoReceiveSeconds,
                LastUsed = DateTime.UtcNow,
            });
        }

public void Connect()
        {
            _Connect(true);
        }

private void _Connect(object state)
        {
            bool throwException = (bool)state;

// if a reconnect is already being attempted, don't hang around waiting
            if (Monitor.TryEnter(m_ConnectionLocker) == false)
            {
                return;
            }

// we took the lock, notify everything we are connecting
            m_Connecting = true;

try
            {
                Stopwatch sw = Stopwatch.StartNew();
                LoggerQueue.Debug(">>>>>> REDIS CONNECTING... >>>>>>");

// if this is a reconnect, make absolutely sure everything is cleaned up first
                _KillTimers();
                _KillRedisClient();

if (this.IsSentinel == true && m_Sentinel == null)
                {
                    LoggerQueue.Debug(">>>>>> CONNECTING TO SENTINEL >>>>>> - " + sw.Elapsed);

// we'll be getting the redis servers from sentinel
                    ConfigurationOptions sentinelConnection = _CreateRedisConfiguration(CommandMap.Sentinel, null, m_SentinelServers);
                    m_Sentinel = ConnectionMultiplexer.Connect(sentinelConnection);
                    LoggerQueue.Debug(">>>>>> CONNECTED TO SENTINEL >>>>>> - " + sw.Elapsed);

_OutputConfigurationFromSentinel();

// get all the redis servers from sentinel and ignore any set by caller
                    m_RedisServers.Clear();
                    m_RedisServers.AddRange(_GetAllRedisServersFromSentinel());

if (m_RedisServers.Count == 0)
                    {
                        throw new RedisException("Sentinel found no redis servers");
                    }
                }

LoggerQueue.Debug(">>>>>> CONNECTING TO REDIS >>>>>> - " + sw.Elapsed);

// try to connect to all redis servers
                ConfigurationOptions connection = _CreateRedisConfiguration(CommandMap.Default, _SecureStringToString(m_Password), m_RedisServers);
                m_Redis = ConnectionMultiplexer.Connect(connection);
                LoggerQueue.Debug(">>>>>> CONNECTED TO REDIS >>>>>> - " + sw.Elapsed);

// register subscription channels
                m_Subscriptions.ForEach(s =>
                {
                    m_Redis.GetSubscriber().Subscribe(s.Key, (channel, value) => _SubscriptionHandler(channel, value));
                    s.Value.LastUsed = DateTime.UtcNow;
                });

if (this.IsSentinel == true)
                {
                    // check subscriptions have been sending messages
                    if (m_Subscriptions.Count > 0)
                    {
                        m_CheckSubscriptionsTimer = new Timer(_CheckSubscriptions, null, 30000, m_CheckSubscriptionsTime);
                    }

if (m_CheckWriteTime != null)
                    {
                        // check that we can write to redis
                        m_CheckWriteTimer = new Timer(_CheckWrite, null, 32000, m_CheckWriteTime.Value);
                    }

// monitor for connection status change to any redis servers
                    m_Redis.ConnectionFailed += _ConnectionFailure;
                    m_Redis.ConnectionRestored += _ConnectionRestored;
                }

LoggerQueue.Debug(string.Format(">>>>>> ALL REDIS CONNECTED ({0}) >>>>>>", sw.Elapsed));
            }
            catch (Exception ex)
            {
                LoggerQueue.Error(">>>>>> REDIS CONNECT FAILURE >>>>>>", ex);

if (throwException == true)
                {
                    throw;
                }
                else
                {
                    // internal reconnect, the reconnect has failed so might as well clean everything and try again
                    _KillTimers();
                    _KillRedisClient();

// faster than usual reconnect if failure
                    _ReconnectTimer(1000);
                }
            }
            finally
            {
                // finished connection attempt, notify everything and remove lock
                m_Connecting = false;
                Monitor.Exit(m_ConnectionLocker);
            }
        }

private ConfigurationOptions _CreateRedisConfiguration(CommandMap commandMap, string password, List<RedisConnection> connections)
        {
            ConfigurationOptions connection = new ConfigurationOptions
            {
                CommandMap = commandMap,
                AbortOnConnectFail = true,
                AllowAdmin = true,
                ConnectTimeout = m_ConnectTimeout,
                SyncTimeout = m_Timeout,
                ServiceName = "master",
                TieBreaker = string.Empty,
                Password = password,
            };

connections.ForEach(s =>
            {
                connection.EndPoints.Add(s.Address, s.Port);
            });

return connection;
        }

private void _OutputConfigurationFromSentinel()
        {
            m_SentinelServers.ForEach(s =>
            {
                try
                {
                    IServer server = m_Sentinel.GetServer(s.Address, s.Port);
                    if (server.IsConnected == true)
                    {
                        try
                        {
                            IPEndPoint master = server.SentinelGetMasterAddressByName("master") as IPEndPoint;
                            var slaves = server.SentinelSlaves("master");

StringBuilder sb = new StringBuilder();
                            sb.Append(">>>>>> _OutputConfigurationFromSentinel Server ");
                            sb.Append(s.Address);
                            sb.Append(" thinks that master is ");
                            sb.Append(master);
                            sb.Append(" and slaves are ");

foreach (var slave in slaves)
                            {
                                string name = slave.Where(i => i.Key == "name").Single().Value;
                                bool up = slave.Where(i => i.Key == "flags").Single().Value.Contains("disconnected") == false;

sb.Append(name);
                                sb.Append("(");
                                sb.Append(up == true ? "connected" : "down");
                                sb.Append(") ");
                            }
                            sb.Append(">>>>>>");
                            LoggerQueue.Debug(sb.ToString());
                        }
                        catch (Exception ex)
                        {
                            LoggerQueue.Error(string.Format(">>>>>> _OutputConfigurationFromSentinel Could not get configuration from sentinel server ({0}) >>>>>>", s.Address), ex);
                        }
                    }
                    else
                    {
                        LoggerQueue.Error(string.Format(">>>>>> _OutputConfigurationFromSentinel Sentinel server {0} was not connected", s.Address));
                    }
                }
                catch (Exception ex)
                {
                    LoggerQueue.Error(string.Format(">>>>>> _OutputConfigurationFromSentinel Could not get IServer from sentinel ({0}) >>>>>>", s.Address), ex);
                }
            });
        }

private RedisConnection[] _GetAllRedisServersFromSentinel()
        {
            // ask each sentinel server for its configuration
            List<RedisConnection> redisServers = new List<RedisConnection>();
            m_SentinelServers.ForEach(s =>
            {
                try
                {
                    IServer server = m_Sentinel.GetServer(s.Address, s.Port);
                    if (server.IsConnected == true)
                    {
                        try
                        {
                            // store master in list
                            IPEndPoint master = server.SentinelGetMasterAddressByName("master") as IPEndPoint;
                            redisServers.Add(new RedisConnection { Address = master.Address.ToString(), Port = master.Port });

var slaves = server.SentinelSlaves("master");
                            foreach (var slave in slaves)
                            {
                                string address = slave.Where(i => i.Key == "ip").Single().Value;
                                string port = slave.Where(i => i.Key == "port").Single().Value;

redisServers.Add(new RedisConnection { Address = address, Port = Convert.ToInt32(port) });
                            }
                        }
                        catch (Exception ex)
                        {
                            LoggerQueue.Error(string.Format(">>>>>> _GetAllRedisServersFromSentinel Could not get redis servers from sentinel server ({0}) >>>>>>", s.Address), ex);
                        }
                    }
                    else
                    {
                        LoggerQueue.Error(string.Format(">>>>>> _GetAllRedisServersFromSentinel Sentinel server {0} was not connected", s.Address));
                    }
                }
                catch (Exception ex)
                {
                    LoggerQueue.Error(string.Format(">>>>>> _GetAllRedisServersFromSentinel Could not get IServer from sentinel ({0}) >>>>>>", s.Address), ex);
                }
            });

return redisServers.Distinct().ToArray();
        }

private IServer _GetRedisMasterFromSentinel()
        {
            // ask each sentinel server for its configuration
            foreach (RedisConnection sentinel in m_SentinelServers)
            {
                IServer sentinelServer = _Sentinel.GetServer(sentinel.Address, sentinel.Port);
                if (sentinelServer.IsConnected == true)
                {
                    try
                    {
                        IPEndPoint master = sentinelServer.SentinelGetMasterAddressByName("master") as IPEndPoint;
                        return _Redis.GetServer(master);
                    }
                    catch (Exception ex)
                    {
                        LoggerQueue.Error(string.Format(">>>>>> Could not get redis master from sentinel server ({0}) >>>>>>", sentinel.Address), ex);
                    }
                }
            }

throw new InvalidOperationException("No sentinel server available to get master");
        }

private void _ReconnectTimer(Nullable<Int32> reconnectMilliseconds)
        {
            try
            {
                lock (m_ReconnectLocker)
                {
                    if (m_ReconnectTimer != null)
                    {
                        m_ReconnectTimer.Dispose();
                        m_ReconnectTimer = null;
                    }

// since a reconnect will definately occur we can stop the check timers for now until reconnect succeeds (where they are recreated)
                    _KillTimers();

LoggerQueue.Warn(">>>>>> REDIS STARTING RECONNECT TIMER >>>>>>");

m_ReconnectTimer = new Timer(_Connect, false, reconnectMilliseconds.GetValueOrDefault(m_ReconnectTime), Timeout.Infinite);
                }
            }
            catch (Exception ex)
            {
                LoggerQueue.Error("Error during _ReconnectTimer", ex);
            }
        }

private void _CheckSubscriptions(object state)
        {
            if (Monitor.TryEnter(m_ConnectionLocker, TimeSpan.FromSeconds(1)) == false)
            {
                return;
            }

try
            {
                DateTime now = DateTime.UtcNow;
                foreach (RedisSubscription subscription in m_Subscriptions.Values)
                {
                    if ((now - subscription.LastUsed) > TimeSpan.FromSeconds(subscription.MaxNoReceiveSeconds))
                    {
                        try
                        {
                            EndPoint endpoint = m_Redis.GetSubscriber().IdentifyEndpoint(subscription.Channel);
                            EndPoint subscribedEndpoint = m_Redis.GetSubscriber().SubscribedEndpoint(subscription.Channel);

LoggerQueue.Warn(string.Format(">>>>>> REDIS Channel '{0}' has not been used for longer than {1}s, IsConnected: {2}, IsConnectedChannel: {3}, EndPoint: {4}, SubscribedEndPoint: {5}, reconnecting...", subscription.Channel, subscription.MaxNoReceiveSeconds, m_Redis.GetSubscriber().IsConnected(), m_Redis.GetSubscriber().IsConnected(subscription.Channel), endpoint != null ? endpoint.ToString() : "null", subscribedEndpoint != null ? subscribedEndpoint.ToString() : "null"));
                        }
                        catch (Exception ex)
                        {
                            LoggerQueue.Error(string.Format(">>>>>> REDIS Error logging out details of Channel '{0}' reconnect", subscription.Channel), ex);
                        }

_ReconnectTimer(null);
                        return;
                    }
                }
            }
            catch (Exception ex)
            {
                LoggerQueue.Error(">>>>>> REDIS Exception ERROR during _CheckSubscriptions", ex);
            }
            finally
            {
                Monitor.Exit(m_ConnectionLocker);
            }
        }

private void _CheckWrite(object state)
        {
            if (Monitor.TryEnter(m_ConnectionLocker, TimeSpan.FromSeconds(1)) == false)
            {
                return;
            }

try
            {
                this.Database.HashSet(Environment.MachineName + "SmartoddsWriteCheck", m_CheckWriteGuid.ToString(), DateTime.UtcNow.Ticks);
            }
            catch (RedisConnectionNotReadyException)
            {
                LoggerQueue.Warn(">>>>>> REDIS RedisConnectionNotReadyException ERROR DURING _CheckWrite");
            }
            catch (RedisServerException ex)
            {
                LoggerQueue.Warn(">>>>>> REDIS RedisServerException ERROR DURING _CheckWrite, reconnecting... - " + ex.Message);
                _ReconnectTimer(null);
            }
            catch (RedisConnectionException ex)
            {
                LoggerQueue.Warn(">>>>>> REDIS RedisConnectionException ERROR DURING _CheckWrite, reconnecting... - " + ex.Message);
                _ReconnectTimer(null);
            }
            catch (TimeoutException ex)
            {
                LoggerQueue.Warn(">>>>>> REDIS TimeoutException ERROR DURING _CheckWrite - " + ex.Message);
            }
            catch (Exception ex)
            {
                LoggerQueue.Error(">>>>>> REDIS Exception ERROR during _CheckWrite", ex);
            }
            finally
            {
                Monitor.Exit(m_ConnectionLocker);
            }
        }

private void _ConnectionFailure(object sender, ConnectionFailedEventArgs e)
        {
            LoggerQueue.Warn(string.Format(">>>>>> REDIS CONNECTION FAILURE, {0}, {1}, {2} >>>>>>", e.ConnectionType, e.EndPoint.ToString(), e.FailureType));
        }

private void _ConnectionRestored(object sender, ConnectionFailedEventArgs e)
        {
            LoggerQueue.Warn(string.Format(">>>>>> REDIS CONNECTION RESTORED, {0}, {1}, {2} >>>>>>", e.ConnectionType, e.EndPoint.ToString(), e.FailureType));
        }

private void _SubscriptionHandler(string channel, RedisValue value)
        {
            // get handler lookup
            RedisSubscription subscription = null;
            if (m_Subscriptions.TryGetValue(channel, out subscription) == false || subscription == null)
            {
                return;
            }

// update last used
            subscription.LastUsed = DateTime.UtcNow;

// call handler
            subscription.Handler(channel, value);
        }

public Int64 Publish(string channel, RedisValue message)
        {
            try
            {
                return _Redis.GetSubscriber().Publish(channel, message);
            }
            catch (RedisConnectionNotReadyException)
            {
                LoggerQueue.Error("REDIS RedisConnectionNotReadyException ERROR DURING Publish");
                throw;
            }
            catch (RedisServerException ex)
            {
                LoggerQueue.Error("REDIS RedisServerException ERROR DURING Publish - " + ex.Message);
                throw;
            }
            catch (RedisConnectionException ex)
            {
                LoggerQueue.Error("REDIS RedisConnectionException ERROR DURING Publish - " + ex.Message);
                throw;
            }
            catch (TimeoutException ex)
            {
                LoggerQueue.Error("REDIS TimeoutException ERROR DURING Publish - " + ex.Message);
                throw;
            }
            catch (Exception ex)
            {
                LoggerQueue.Error("REDIS Exception ERROR DURING Publish", ex);
                throw;
            }
        }

public bool LockTake(RedisKey key, RedisValue value, TimeSpan expiry)
        {
            return _Execute(() => this.Database.LockTake(key, value, expiry));
        }

public bool LockExtend(RedisKey key, RedisValue value, TimeSpan extension)
        {
            return _Execute(() => this.Database.LockExtend(key, value, extension));
        }

public bool LockRelease(RedisKey key, RedisValue value)
        {
            return _Execute(() => this.Database.LockRelease(key, value));
        }

private void _Execute(Action action)
        {
            try
            {
                action.Invoke();
            }
            catch (RedisServerException ex)
            {
                LoggerQueue.Error("REDIS RedisServerException ERROR DURING _Execute - " + ex.Message);
                throw;
            }
            catch (RedisConnectionException ex)
            {
                LoggerQueue.Error("REDIS RedisConnectionException ERROR DURING _Execute - " + ex.Message);
                throw;
            }
            catch (TimeoutException ex)
            {
                LoggerQueue.Error("REDIS TimeoutException ERROR DURING _Execute - " + ex.Message);
                throw;
            }
            catch (Exception ex)
            {
                LoggerQueue.Error("REDIS Exception ERROR DURING _Execute", ex);
                throw;
            }
        }

private TResult _Execute<TResult>(Func<TResult> function)
        {
            try
            {
                return function.Invoke();
            }
            catch (RedisServerException ex)
            {
                LoggerQueue.Error("REDIS RedisServerException ERROR DURING _Execute - " + ex.Message);
                throw;
            }
            catch (RedisConnectionException ex)
            {
                LoggerQueue.Error("REDIS RedisConnectionException ERROR DURING _Execute - " + ex.Message);
                throw;
            }
            catch (TimeoutException ex)
            {
                LoggerQueue.Error("REDIS TimeoutException ERROR DURING _Execute - " + ex.Message);
                throw;
            }
            catch (Exception ex)
            {
                LoggerQueue.Error("REDIS ERROR DURING _Execute", ex);
                throw;
            }
        }

public string[] GetAllKeys(string pattern)
        {
            if (m_Sentinel != null)
            {
                return _GetAnyRedisSlaveFromSentinel().Keys(m_DatabaseId, pattern).Select(k => (string)k).ToArray();
            }
            else
            {
                return _Redis.GetServer(_Redis.GetEndPoints().First()).Keys(m_DatabaseId, pattern).Select(k => (string)k).ToArray();
            }
        }

private void _KillSentinelClient()
        {
            try
            {
                if (m_Sentinel != null)
                {
                    LoggerQueue.Debug(">>>>>> KILLING SENTINEL CONNECTION >>>>>>");

ConnectionMultiplexer sentinel = m_Sentinel;
                    m_Sentinel = null;

sentinel.Close(false);
                    sentinel.Dispose();
                }
            }
            catch (Exception ex)
            {
                LoggerQueue.Error(">>>>>> Error during _KillSentinelClient", ex);
            }
        }

private void _KillRedisClient()
        {
            try
            {
                if (m_Redis != null)
                {
                    Stopwatch sw = Stopwatch.StartNew();
                    LoggerQueue.Debug(">>>>>> KILLING REDIS CONNECTION >>>>>>");

ConnectionMultiplexer redis = m_Redis;
                    m_Redis = null;

if (this.IsSentinel == true)
                    {
                        redis.ConnectionFailed -= _ConnectionFailure;
                        redis.ConnectionRestored -= _ConnectionRestored;
                    }

redis.Close(false);
                    redis.Dispose();

LoggerQueue.Debug(">>>>>> KILLED REDIS CONNECTION >>>>>> " + sw.Elapsed);
                }
            }
            catch (Exception ex)
            {
                LoggerQueue.Error(">>>>>> Error during _KillRedisClient", ex);
            }
        }

private void _KillClients()
        {
            lock (m_ConnectionLocker)
            {
                _KillSentinelClient();
                _KillRedisClient();
            }
        }

private void _KillTimers()
        {
            if (m_CheckSubscriptionsTimer != null)
            {
                m_CheckSubscriptionsTimer.Dispose();
                m_CheckSubscriptionsTimer = null;
            }
            if (m_CheckWriteTimer != null)
            {
                m_CheckWriteTimer.Dispose();
                m_CheckWriteTimer = null;
            }
        }

public void Dispose()
        {
            _KillClients();
            _KillTimers();
        }
    }
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

namespace Smartodds.Framework.Redis

{

public class RedisClient : IDisposable

{

public RedisClient(RedisEnvironmentElement environment, Int32 databaseId)

{

m_ConnectTimeout = environment.ConnectTimeout;

m_Timeout = environment.Timeout;

m_DatabaseId = databaseId;

m_ReconnectTime = environment.ReconnectTime;

m_CheckSubscriptionsTime = environment.CheckSubscriptions;

if (environment.TestWrite == true)

{

m_CheckWriteTime = environment.TestWriteTime;

}

environment.Password.ToCharArray().ForEach((c) => m_Password.AppendChar(c));

foreach (var server in environment.Servers)

{

if (server.Type == ServerType.Redis)

{

// will be ignored if sentinel servers are used

m_RedisServers.Add(new RedisConnection { Address = server.Host, Port = server.Port });

}

else

{

m_SentinelServers.Add(new RedisConnection { Address = server.Host, Port = server.Port });

}

public bool IsSentinel { get { return m_SentinelServers.Count > 0; } }

public IDatabase Database { get { return _Redis.GetDatabase(m_DatabaseId); } }

private ConnectionMultiplexer _Redis

{

get

{

if (m_Connecting == true)

{

throw new RedisConnectionNotReadyException();

}

ConnectionMultiplexer redis = m_Redis;

if (redis == null)

{

throw new RedisConnectionNotReadyException();

}

return redis;

}

private ConnectionMultiplexer _Sentinel

{

get

{

if (m_Connecting == true)

{

throw new RedisConnectionNotReadyException("Sentinel connection not ready");

}

ConnectionMultiplexer sentinel = m_Sentinel;

if (sentinel == null)

{

throw new RedisConnectionNotReadyException("Sentinel connection not ready");

}

return sentinel;

}

public void RegisterSubscription(string channel, Action<RedisChannel, RedisValue> handler, Int32 maxNoReceiveSeconds)

{

m_Subscriptions.Add(channel, new RedisSubscription

{

Channel = channel,

Handler = handler,

MaxNoReceiveSeconds = maxNoReceiveSeconds,

LastUsed = DateTime.UtcNow,

});

}

public void Connect()

{

_Connect(true);

}

private void _Connect(object state)

{

bool throwException = (bool)state;

// if a reconnect is already being attempted, don't hang around waiting

if (Monitor.TryEnter(m_ConnectionLocker) == false)

{

return;

}

// we took the lock, notify everything we are connecting

m_Connecting = true;

try

{

Stopwatch sw = Stopwatch.StartNew();

LoggerQueue.Debug(">>>>>> REDIS CONNECTING... >>>>>>");

// if this is a reconnect, make absolutely sure everything is cleaned up first

_KillTimers();

_KillRedisClient();

if (this.IsSentinel == true && m_Sentinel == null)

{

LoggerQueue.Debug(">>>>>> CONNECTING TO SENTINEL >>>>>> - " + sw.Elapsed);

// we'll be getting the redis servers from sentinel

ConfigurationOptions sentinelConnection = _CreateRedisConfiguration(CommandMap.Sentinel, null, m_SentinelServers);

m_Sentinel = ConnectionMultiplexer.Connect(sentinelConnection);

LoggerQueue.Debug(">>>>>> CONNECTED TO SENTINEL >>>>>> - " + sw.Elapsed);

_OutputConfigurationFromSentinel();

// get all the redis servers from sentinel and ignore any set by caller

m_RedisServers.Clear();

m_RedisServers.AddRange(_GetAllRedisServersFromSentinel());

if (m_RedisServers.Count == 0)

{

throw new RedisException("Sentinel found no redis servers");

}

LoggerQueue.Debug(">>>>>> CONNECTING TO REDIS >>>>>> - " + sw.Elapsed);

// try to connect to all redis servers

ConfigurationOptions connection = _CreateRedisConfiguration(CommandMap.Default, _SecureStringToString(m_Password), m_RedisServers);

m_Redis = ConnectionMultiplexer.Connect(connection);

LoggerQueue.Debug(">>>>>> CONNECTED TO REDIS >>>>>> - " + sw.Elapsed);

// register subscription channels

m_Subscriptions.ForEach(s =>

{

m_Redis.GetSubscriber().Subscribe(s.Key, (channel, value) => _SubscriptionHandler(channel, value));

s.Value.LastUsed = DateTime.UtcNow;

});

if (this.IsSentinel == true)

{

// check subscriptions have been sending messages

if (m_Subscriptions.Count > 0)

{

m_CheckSubscriptionsTimer = new Timer(_CheckSubscriptions, null, 30000, m_CheckSubscriptionsTime);

}

if (m_CheckWriteTime != null)

{

// check that we can write to redis

m_CheckWriteTimer = new Timer(_CheckWrite, null, 32000, m_CheckWriteTime.Value);

}

// monitor for connection status change to any redis servers

m_Redis.ConnectionFailed += _ConnectionFailure;

m_Redis.ConnectionRestored += _ConnectionRestored;

}

LoggerQueue.Debug(string.Format(">>>>>> ALL REDIS CONNECTED ({0}) >>>>>>", sw.Elapsed));

}

catch (Exception ex)

{

LoggerQueue.Error(">>>>>> REDIS CONNECT FAILURE >>>>>>", ex);

if (throwException == true)

{

throw;

}

else

{

// internal reconnect, the reconnect has failed so might as well clean everything and try again

_KillTimers();

_KillRedisClient();

// faster than usual reconnect if failure

_ReconnectTimer(1000);

}

finally

{

// finished connection attempt, notify everything and remove lock

m_Connecting = false;

Monitor.Exit(m_ConnectionLocker);

}

private ConfigurationOptions _CreateRedisConfiguration(CommandMap commandMap, string password, List<RedisConnection> connections)

{

ConfigurationOptions connection = new ConfigurationOptions

{

CommandMap = commandMap,

AbortOnConnectFail = true,

AllowAdmin = true,

ConnectTimeout = m_ConnectTimeout,

SyncTimeout = m_Timeout,

ServiceName = "master",

TieBreaker = string.Empty,

Password = password,

};

connections.ForEach(s =>

{

connection.EndPoints.Add(s.Address, s.Port);

});

return connection;

}

private void _OutputConfigurationFromSentinel()

{

m_SentinelServers.ForEach(s =>

{

try

{

IServer server = m_Sentinel.GetServer(s.Address, s.Port);

if (server.IsConnected == true)

{

try

{

IPEndPoint master = server.SentinelGetMasterAddressByName("master") as IPEndPoint;

var slaves = server.SentinelSlaves("master");

StringBuilder sb = new StringBuilder();

sb.Append(">>>>>> _OutputConfigurationFromSentinel Server ");

sb.Append(s.Address);

sb.Append(" thinks that master is ");

sb.Append(master);

sb.Append(" and slaves are ");

foreach (var slave in slaves)

{

string name = slave.Where(i => i.Key == "name").Single().Value;

bool up = slave.Where(i => i.Key == "flags").Single().Value.Contains("disconnected") == false;

sb.Append(name);

sb.Append("(");

sb.Append(up == true ? "connected" : "down");

sb.Append(") ");

}

sb.Append(">>>>>>");

LoggerQueue.Debug(sb.ToString());

}

catch (Exception ex)

{

LoggerQueue.Error(string.Format(">>>>>> _OutputConfigurationFromSentinel Could not get configuration from sentinel server ({0}) >>>>>>", s.Address), ex);

}

else

{

LoggerQueue.Error(string.Format(">>>>>> _OutputConfigurationFromSentinel Sentinel server {0} was not connected", s.Address));

}

catch (Exception ex)

{

LoggerQueue.Error(string.Format(">>>>>> _OutputConfigurationFromSentinel Could not get IServer from sentinel ({0}) >>>>>>", s.Address), ex);

}

});

}

private RedisConnection[] _GetAllRedisServersFromSentinel()

{

// ask each sentinel server for its configuration

List<RedisConnection> redisServers = new List<RedisConnection>();

m_SentinelServers.ForEach(s =>

{

try

{

IServer server = m_Sentinel.GetServer(s.Address, s.Port);

if (server.IsConnected == true)

{

try

{

// store master in list

IPEndPoint master = server.SentinelGetMasterAddressByName("master") as IPEndPoint;

redisServers.Add(new RedisConnection { Address = master.Address.ToString(), Port = master.Port });

var slaves = server.SentinelSlaves("master");

foreach (var slave in slaves)

{

string address = slave.Where(i => i.Key == "ip").Single().Value;

string port = slave.Where(i => i.Key == "port").Single().Value;

redisServers.Add(new RedisConnection { Address = address, Port = Convert.ToInt32(port) });

}

catch (Exception ex)

{

LoggerQueue.Error(string.Format(">>>>>> _GetAllRedisServersFromSentinel Could not get redis servers from sentinel server ({0}) >>>>>>", s.Address), ex);

}

else

{

LoggerQueue.Error(string.Format(">>>>>> _GetAllRedisServersFromSentinel Sentinel server {0} was not connected", s.Address));

}

catch (Exception ex)

{

LoggerQueue.Error(string.Format(">>>>>> _GetAllRedisServersFromSentinel Could not get IServer from sentinel ({0}) >>>>>>", s.Address), ex);

}

});

return redisServers.Distinct().ToArray();

}

private IServer _GetRedisMasterFromSentinel()

{

// ask each sentinel server for its configuration

foreach (RedisConnection sentinel in m_SentinelServers)

{

IServer sentinelServer = _Sentinel.GetServer(sentinel.Address, sentinel.Port);

if (sentinelServer.IsConnected == true)

{

try

{

IPEndPoint master = sentinelServer.SentinelGetMasterAddressByName("master") as IPEndPoint;

return _Redis.GetServer(master);

}

catch (Exception ex)

{

LoggerQueue.Error(string.Format(">>>>>> Could not get redis master from sentinel server ({0}) >>>>>>", sentinel.Address), ex);

}

throw new InvalidOperationException("No sentinel server available to get master");

}

private void _ReconnectTimer(Nullable<Int32> reconnectMilliseconds)

{

try

{

lock (m_ReconnectLocker)

{

if (m_ReconnectTimer != null)

{

m_ReconnectTimer.Dispose();

m_ReconnectTimer = null;

}

// since a reconnect will definately occur we can stop the check timers for now until reconnect succeeds (where they are recreated)

_KillTimers();

LoggerQueue.Warn(">>>>>> REDIS STARTING RECONNECT TIMER >>>>>>");

m_ReconnectTimer = new Timer(_Connect, false, reconnectMilliseconds.GetValueOrDefault(m_ReconnectTime), Timeout.Infinite);

}

catch (Exception ex)

{

LoggerQueue.Error("Error during _ReconnectTimer", ex);

}

private void _CheckSubscriptions(object state)

{

if (Monitor.TryEnter(m_ConnectionLocker, TimeSpan.FromSeconds(1)) == false)

{

return;

}

try

{

DateTime now = DateTime.UtcNow;

foreach (RedisSubscription subscription in m_Subscriptions.Values)

{

if ((now - subscription.LastUsed) > TimeSpan.FromSeconds(subscription.MaxNoReceiveSeconds))

{

try

{

EndPoint endpoint = m_Redis.GetSubscriber().IdentifyEndpoint(subscription.Channel);

EndPoint subscribedEndpoint = m_Redis.GetSubscriber().SubscribedEndpoint(subscription.Channel);

}

catch (Exception ex)

{

LoggerQueue.Error(string.Format(">>>>>> REDIS Error logging out details of Channel '{0}' reconnect", subscription.Channel), ex);

}

_ReconnectTimer(null);

return;

}

catch (Exception ex)

{

LoggerQueue.Error(">>>>>> REDIS Exception ERROR during _CheckSubscriptions", ex);

}

finally

{

Monitor.Exit(m_ConnectionLocker);

}

private void _CheckWrite(object state)

{

if (Monitor.TryEnter(m_ConnectionLocker, TimeSpan.FromSeconds(1)) == false)

{

return;

}

try

{

this.Database.HashSet(Environment.MachineName + "SmartoddsWriteCheck", m_CheckWriteGuid.ToString(), DateTime.UtcNow.Ticks);

}

catch (RedisConnectionNotReadyException)

{

LoggerQueue.Warn(">>>>>> REDIS RedisConnectionNotReadyException ERROR DURING _CheckWrite");

}

catch (RedisServerException ex)

{

LoggerQueue.Warn(">>>>>> REDIS RedisServerException ERROR DURING _CheckWrite, reconnecting... - " + ex.Message);

_ReconnectTimer(null);

}

catch (RedisConnectionException ex)

{

LoggerQueue.Warn(">>>>>> REDIS RedisConnectionException ERROR DURING _CheckWrite, reconnecting... - " + ex.Message);

_ReconnectTimer(null);

}

catch (TimeoutException ex)

{

LoggerQueue.Warn(">>>>>> REDIS TimeoutException ERROR DURING _CheckWrite - " + ex.Message);

}

catch (Exception ex)

{

LoggerQueue.Error(">>>>>> REDIS Exception ERROR during _CheckWrite", ex);

}

finally

{

Monitor.Exit(m_ConnectionLocker);

}

private void _ConnectionFailure(object sender, ConnectionFailedEventArgs e)

{

LoggerQueue.Warn(string.Format(">>>>>> REDIS CONNECTION FAILURE, {0}, {1}, {2} >>>>>>", e.ConnectionType, e.EndPoint.ToString(), e.FailureType));

}

private void _ConnectionRestored(object sender, ConnectionFailedEventArgs e)

{

LoggerQueue.Warn(string.Format(">>>>>> REDIS CONNECTION RESTORED, {0}, {1}, {2} >>>>>>", e.ConnectionType, e.EndPoint.ToString(), e.FailureType));

}

private void _SubscriptionHandler(string channel, RedisValue value)

{

// get handler lookup

RedisSubscription subscription = null;

if (m_Subscriptions.TryGetValue(channel, out subscription) == false || subscription == null)

{

return;

}

// update last used

subscription.LastUsed = DateTime.UtcNow;

// call handler

subscription.Handler(channel, value);

}

public Int64 Publish(string channel, RedisValue message)

{

try

{

return _Redis.GetSubscriber().Publish(channel, message);

}

catch (RedisConnectionNotReadyException)

{

LoggerQueue.Error("REDIS RedisConnectionNotReadyException ERROR DURING Publish");

throw;

}

catch (RedisServerException ex)

{

LoggerQueue.Error("REDIS RedisServerException ERROR DURING Publish - " + ex.Message);

throw;

}

catch (RedisConnectionException ex)

{

LoggerQueue.Error("REDIS RedisConnectionException ERROR DURING Publish - " + ex.Message);

throw;

}

catch (TimeoutException ex)

{

LoggerQueue.Error("REDIS TimeoutException ERROR DURING Publish - " + ex.Message);

throw;

}

catch (Exception ex)

{

LoggerQueue.Error("REDIS Exception ERROR DURING Publish", ex);

throw;

}

public bool LockTake(RedisKey key, RedisValue value, TimeSpan expiry)

{

return _Execute(() => this.Database.LockTake(key, value, expiry));

}

public bool LockExtend(RedisKey key, RedisValue value, TimeSpan extension)

{

return _Execute(() => this.Database.LockExtend(key, value, extension));

}

public bool LockRelease(RedisKey key, RedisValue value)

{

return _Execute(() => this.Database.LockRelease(key, value));

}

private void _Execute(Action action)

{

try

{

action.Invoke();

}

catch (RedisServerException ex)

{

LoggerQueue.Error("REDIS RedisServerException ERROR DURING _Execute - " + ex.Message);

throw;

}

catch (RedisConnectionException ex)

{

LoggerQueue.Error("REDIS RedisConnectionException ERROR DURING _Execute - " + ex.Message);

throw;

}

catch (TimeoutException ex)

{

LoggerQueue.Error("REDIS TimeoutException ERROR DURING _Execute - " + ex.Message);

throw;

}

catch (Exception ex)

{

LoggerQueue.Error("REDIS Exception ERROR DURING _Execute", ex);

throw;

}

private TResult _Execute<TResult>(Func<TResult> function)

{

try

{

return function.Invoke();

}

catch (RedisServerException ex)

{

LoggerQueue.Error("REDIS RedisServerException ERROR DURING _Execute - " + ex.Message);

throw;

}

catch (RedisConnectionException ex)

{

LoggerQueue.Error("REDIS RedisConnectionException ERROR DURING _Execute - " + ex.Message);

throw;

}

catch (TimeoutException ex)

{

LoggerQueue.Error("REDIS TimeoutException ERROR DURING _Execute - " + ex.Message);

throw;

}

catch (Exception ex)

{

LoggerQueue.Error("REDIS ERROR DURING _Execute", ex);

throw;

}

public string[] GetAllKeys(string pattern)

{

if (m_Sentinel != null)

{

return _GetAnyRedisSlaveFromSentinel().Keys(m_DatabaseId, pattern).Select(k => (string)k).ToArray();

}

else

{

return _Redis.GetServer(_Redis.GetEndPoints().First()).Keys(m_DatabaseId, pattern).Select(k => (string)k).ToArray();

}

private void _KillSentinelClient()

{

try

{

if (m_Sentinel != null)

{

LoggerQueue.Debug(">>>>>> KILLING SENTINEL CONNECTION >>>>>>");

ConnectionMultiplexer sentinel = m_Sentinel;

m_Sentinel = null;

sentinel.Close(false);

sentinel.Dispose();

}

catch (Exception ex)

{

LoggerQueue.Error(">>>>>> Error during _KillSentinelClient", ex);

}

private void _KillRedisClient()

{

try

{

if (m_Redis != null)

{

Stopwatch sw = Stopwatch.StartNew();

LoggerQueue.Debug(">>>>>> KILLING REDIS CONNECTION >>>>>>");

ConnectionMultiplexer redis = m_Redis;

m_Redis = null;

if (this.IsSentinel == true)

{

redis.ConnectionFailed -= _ConnectionFailure;

redis.ConnectionRestored -= _ConnectionRestored;

}

redis.Close(false);

redis.Dispose();

LoggerQueue.Debug(">>>>>> KILLED REDIS CONNECTION >>>>>> " + sw.Elapsed);

}

catch (Exception ex)

{

LoggerQueue.Error(">>>>>> Error during _KillRedisClient", ex);

}

private void _KillClients()

{

lock (m_ConnectionLocker)

{

_KillSentinelClient();

_KillRedisClient();

}

private void _KillTimers()

{

if (m_CheckSubscriptionsTimer != null)

{

m_CheckSubscriptionsTimer.Dispose();

m_CheckSubscriptionsTimer = null;

}

if (m_CheckWriteTimer != null)

{

m_CheckWriteTimer.Dispose();

m_CheckWriteTimer = null;

}

public void Dispose()

{

_KillClients();

_KillTimers();

}

In the next post we will go through redis cluster, I wanted to talk about sentinel first such that readers tend to mix them up and by first explaining sentinel we can talk about clusters which are different concepts.

Redis

redis, redis sentinel, sentinel, StackExchange.Redis

Redis for .NET Developers – Redis GeoSpatial

Taswar Bhatti September 9, 2017 No Comments

Redis GeoSpatial data sets are actually just SortedSets in Redis, there is no secret about it. Basically it provides an easy way to store geo spatial data like longitude/latitude coordinates into Redis.Lets look at some of the commands that Redis provides for Geo Spatial data.

Redis Geo Datatype – Operations

GEOADD: Adds or updates one or more members to a Geo Set O(log (N)) where N is the number of elements in the sorted set.
GEODIST: Return the distance between two members in the geo spatial index represented by the sorted set O(log(N)).
GEOHASH: Gets valid Geohash strings representing the position of one or more elements from the Geo Sets O(log(N)), where N is the number of elements in the sorted set.
GEOPOS: Return the longitude,latitude of all the specified members of the geo spatial sorted set at key O(log(N)), where N is the number of elements in the sorted set.
GEORADIUS: Return the members of a sorted set populated with geo spatial information using GEOADD, which are within the borders of the area specified with the center location and the maximum distance from the center (the radius) O(N+log(M))
GEORADIUSBYMEMBER: Same as GEORADIUS with the only difference that instead of taking, as the center of the area to query, it takes a longitude and latitude value O(N+log(M))

I wanted to use some open data to showcase the usage of Redis GeoSpatial data in Redis. I chose to use Basketball courts in Ottawa since my son plays a bit of basketball.
Here is a map of basketball courts in Ottawa.

The map is also located at https://gist.github.com/6992f8a5b28e7e944ae241e86b8b0066

Ottawa Basketball Courts

C# code using Redis Geo Set Datatype

class Program
    {
        static void Main(string[] args)
        {
             var redis = RedisStore.RedisCache;

var list = new List<GeoEntry>();

//BasetBallExtractor is just a class that calls Ottawa OpenData
            foreach (var item in BasketBallExtractor.Extract())
            {
                var longtitue = Convert.ToDouble(item.Item1);
                var latitude = Convert.ToDouble(item.Item2);
                var name = item.Item3;
                list.Add(new GeoEntry(longtitue, latitude, name));
            }
            
            var key = "locations";

//clear out the data
            redis.KeyDelete(key, CommandFlags.FireAndForget);

//add all the geoentry
            redis.GeoAdd(key, list.ToArray());

//find the distance between parks
            var val = redis.GeoDistance(key, "Gesner Court Park", "Walter Baker Park", GeoUnit.Kilometers);
            
            Console.WriteLine($"Distance from Gesner Court to Walter Baker Park is {val} km");
            //ouput - Distance from Gesner Court to Walter Baker Park is 0.4377 km
            
            val = redis.GeoDistance(key, "Brewer Park", "Walter Baker Park", GeoUnit.Kilometers);

Console.WriteLine($"Distance from Brewer Park to Walter Baker Park is {val} km");
            //output - Distance from Brewer Park to Walter Baker Park is 19.6528 km

//get the position of Walter Baker Park
            var pos = redis.GeoPosition(key, "Walter Baker Park");

Console.WriteLine($"Walter Baker Park is located at: Lat {pos.Value.Latitude}, Long {pos.Value.Longitude} ");
            //Walter Baker Park is located at: Lat 45.2947688017077, Long - 75.902011692524

//assume we are at walter baker and the place is closed
            //we want to find out what is close by within 5 km for us to play basketball and find us the top 10 closes place
            var results = redis.GeoRadius(key, "Walter Baker Park", 5, GeoUnit.Kilometers, 10, Order.Ascending, GeoRadiusOptions.WithDistance);

foreach (var geoRadiusResult in results)
            {
                Console.WriteLine($"- {geoRadiusResult.Member},  Distance: {geoRadiusResult.Distance}");
            }

/*
             * - Walter Baker Park,  Distance: 0
             * - Gesner Court Park,  Distance: 0.4377
             * - Stonegate Park,  Distance: 0.9454
             * - Fringewood Park and Community Centre,  Distance: 1.8288
             * - Clarence Maheral Park,  Distance: 1.9128
             * - Cattail Creek Park,  Distance: 2.0176
             * - Amberway Park,  Distance: 2.4079
             * - Ed Hollyer Park,  Distance: 2.5855
             * - Bryanston Gate Park,  Distance: 2.7199
             * - Jim Malone Park,  Distance: 2.9452
             */

//lets say we are a Ottawa Parliment Hill and want to play basketball where can we go
            var parliamentHillLatitude = 45.424807;
            var parliamentHillLongtitude = -75.699234;
            
            //find me all the locations nearby
            results = redis.GeoRadius(key, parliamentHillLongtitude, parliamentHillLatitude, 3, GeoUnit.Kilometers, -1, Order.Ascending, GeoRadiusOptions.WithCoordinates);
            foreach (var geoRadiusResult in results)
            {
                Console.WriteLine($"- {geoRadiusResult.Member}, Position {geoRadiusResult.Position}");
            }

/*
             * - Cathcart Park, Position -75.6929334998131 45.4346575277704
             * - St. Luke's Park, Position -75.6867161393166 45.4152390289687
             * - Bordeleau Park, Position -75.6928798556328 45.4373215197089
             * - New Edinburgh Park, Position -75.6871077418327 45.4377904431234
             * - Dalhousie Community Centre, Position -75.7093647122383 45.4104889615161
             * - Sandy Hill Park and Community Centre, Position -75.6765183806419 45.4219332275505
             * - Chaudière Park, Position -75.7137742638588 45.409835003457
             * - Plouffe Park, Position -75.7150831818581 45.4074777127788
             * - Dutchie's Hole Park, Position -75.6693086028099 45.4209015960387
             * - Riverain Park, Position -75.6697806715965 45.4313877374749
             * - Springhurst Park, Position -75.6728920340538 45.4127347244633
             * - Sylvia Holden Park, Position -75.6818452477455 45.4029101452497
             * - Lindenlea Park, Position -75.6763252615929 45.4449763776101
             * - Ev Tremblay Park, Position -75.7115051150322 45.3998811534643
             * - Laroche Park, Position -75.7291594147682 45.408689309493
             */
               
            //we can get back the hash of the geo location
            //The command returns 11 characters Geohash strings, so no precision is loss compared to the Redis internal 52 bit representation
            var hash = redis.GeoHash(key, "Walter Baker Park");

//we can now use http://geohash.org/<geohash-string>
            Console.WriteLine($"Walter Baker Park geo hash is {hash}");
            //output Walter Baker Park geo hash is f2418v9v4u0
            //this will also work http://geohash.org/f2418v9v4u0

Console.ReadKey();
        }
    }

100

101

102

103

104

105

class Program

{

static void Main(string[] args)

{

var redis = RedisStore.RedisCache;

var list = new List<GeoEntry>();

//BasetBallExtractor is just a class that calls Ottawa OpenData

foreach (var item in BasketBallExtractor.Extract())

{

var longtitue = Convert.ToDouble(item.Item1);

var latitude = Convert.ToDouble(item.Item2);

var name = item.Item3;

list.Add(new GeoEntry(longtitue, latitude, name));

}

var key = "locations";

//clear out the data

redis.KeyDelete(key, CommandFlags.FireAndForget);

//add all the geoentry

redis.GeoAdd(key, list.ToArray());

//find the distance between parks

var val = redis.GeoDistance(key, "Gesner Court Park", "Walter Baker Park", GeoUnit.Kilometers);

Console.WriteLine($"Distance from Gesner Court to Walter Baker Park is {val} km");

//ouput - Distance from Gesner Court to Walter Baker Park is 0.4377 km

val = redis.GeoDistance(key, "Brewer Park", "Walter Baker Park", GeoUnit.Kilometers);

Console.WriteLine($"Distance from Brewer Park to Walter Baker Park is {val} km");

//output - Distance from Brewer Park to Walter Baker Park is 19.6528 km

//get the position of Walter Baker Park

var pos = redis.GeoPosition(key, "Walter Baker Park");

Console.WriteLine($"Walter Baker Park is located at: Lat {pos.Value.Latitude}, Long {pos.Value.Longitude} ");

//Walter Baker Park is located at: Lat 45.2947688017077, Long - 75.902011692524

//assume we are at walter baker and the place is closed

//we want to find out what is close by within 5 km for us to play basketball and find us the top 10 closes place

var results = redis.GeoRadius(key, "Walter Baker Park", 5, GeoUnit.Kilometers, 10, Order.Ascending, GeoRadiusOptions.WithDistance);

foreach (var geoRadiusResult in results)

{

Console.WriteLine($"- {geoRadiusResult.Member}, Distance: {geoRadiusResult.Distance}");

}

* - Walter Baker Park, Distance: 0

* - Gesner Court Park, Distance: 0.4377

* - Stonegate Park, Distance: 0.9454

* - Fringewood Park and Community Centre, Distance: 1.8288

* - Clarence Maheral Park, Distance: 1.9128

* - Cattail Creek Park, Distance: 2.0176

* - Amberway Park, Distance: 2.4079

* - Ed Hollyer Park, Distance: 2.5855

* - Bryanston Gate Park, Distance: 2.7199

* - Jim Malone Park, Distance: 2.9452

//lets say we are a Ottawa Parliment Hill and want to play basketball where can we go

var parliamentHillLatitude = 45.424807;

var parliamentHillLongtitude = -75.699234;

//find me all the locations nearby

results = redis.GeoRadius(key, parliamentHillLongtitude, parliamentHillLatitude, 3, GeoUnit.Kilometers, -1, Order.Ascending, GeoRadiusOptions.WithCoordinates);

foreach (var geoRadiusResult in results)

{

Console.WriteLine($"- {geoRadiusResult.Member}, Position {geoRadiusResult.Position}");

}

* - Cathcart Park, Position -75.6929334998131 45.4346575277704

* - St. Luke's Park, Position -75.6867161393166 45.4152390289687

* - Bordeleau Park, Position -75.6928798556328 45.4373215197089

* - New Edinburgh Park, Position -75.6871077418327 45.4377904431234

* - Dalhousie Community Centre, Position -75.7093647122383 45.4104889615161

* - Sandy Hill Park and Community Centre, Position -75.6765183806419 45.4219332275505

* - Chaudière Park, Position -75.7137742638588 45.409835003457

* - Plouffe Park, Position -75.7150831818581 45.4074777127788

* - Dutchie's Hole Park, Position -75.6693086028099 45.4209015960387

* - Riverain Park, Position -75.6697806715965 45.4313877374749

* - Springhurst Park, Position -75.6728920340538 45.4127347244633

* - Sylvia Holden Park, Position -75.6818452477455 45.4029101452497

* - Lindenlea Park, Position -75.6763252615929 45.4449763776101

* - Ev Tremblay Park, Position -75.7115051150322 45.3998811534643

* - Laroche Park, Position -75.7291594147682 45.408689309493

//we can get back the hash of the geo location

//The command returns 11 characters Geohash strings, so no precision is loss compared to the Redis internal 52 bit representation

var hash = redis.GeoHash(key, "Walter Baker Park");

//we can now use http://geohash.org/<geohash-string>

Console.WriteLine($"Walter Baker Park geo hash is {hash}");

//output Walter Baker Park geo hash is f2418v9v4u0

//this will also work http://geohash.org/f2418v9v4u0

Console.ReadKey();

}

So this covers the basic usage of Redish GeoSpatial Datatype, in the next blog post I will cover using Sentinel which provides high availability for Redis.

For the code please visit
https://github.com/taswar/RedisForNetDevelopers/tree/master/13.RedisGeo/RediusGeo

For previous Redis topics

C#, Redis

redis, redis geo, redis geospatial, StackExchange.Redis

Redis for .NET Developer – Lua Scripting

Taswar Bhatti August 24, 2017 No Comments

Redis Lua Scripting

Redis provides a way to extend its functionality on the server side by providing support for Lua Scripting. If you are coming from a relational database world, you already know that you can use Stored Procedures to extend functionality of your relational database. Now, you may also know that some people do frown upon using stored procedures, I think one could also think of using scripting in Redis sort of belongs in the same category. Nevertheless its still good to know what you can do with Redis and Lua.

If you want to learn more Lua try this site http://tylerneylon.com/a/learn-lua/

In order to call Lua Script from Redis.StackExchange library one can use the LuaScript class or IServer.ScriptLoad(Async), IServer.ScriptExists(Async), IServer.ScriptFlush(Async), IDatabase.ScriptEvaluate, and IDatabaseAsync.ScriptEvaluateAsync methods.

Lets try to do something with the redis console first by using the redis-cli

$redis-cli

127.0.0.1:6379> SCRIPT LOAD "return 'hello redis'"

"69dd69fc0ba1e25d8e2972008b6baee8eccf7da6"

127.0.0.1:6379> evalsha 69dd69fc0ba1e25d8e2972008b6baee8eccf7da6 0

"hello redis"

As you can see we first load the script and we get back a SHA1 Hash from it. Redis basically stores the information in one of its mapping table and we can reuse the sha1 hash to call the script by using the EVALSHA command, which in this case gave us back “hello redis”

Remember: When Redis is running your Lua script, it will not run anything else because Redis is single threaded

C# code using Redis Lua Script

class Program
    {
        static void Main(string[] args)
        {

const string script = @"
local function MGETSUM(keys)
    local sum = 0
    for _,key in ipairs(keys) do
        local val = redis.call('GET', key) or 0
        sum = sum + tonumber(val)
    end
    return sum
end
return MGETSUM(KEYS)";

var redis = RedisStore.RedisCache;

var server = RedisStore.Server;

var test1 = "test1";
            var test2 = "test2";
            var test3 = "test3";

//delete the keys
            redis.KeyDelete(test1, CommandFlags.FireAndForget);
            redis.KeyDelete(test2, CommandFlags.FireAndForget);
            redis.KeyDelete(test3, CommandFlags.FireAndForget);

redis.StringSet(test1, 1);
            redis.StringSet(test2, 2);
            redis.StringSet(test3, 3);

RedisKey[] keys = { test1, test2, test3 };

byte[] hash = server.ScriptLoad(script);

//get back the sha1 key
            string hexHash = string.Concat(hash.Select(x => x.ToString("X2")));

var result = redis.ScriptEvaluate(hexHash, keys);
          
            Console.WriteLine(result); //result will be 6

Console.ReadKey();
        }

class Program

{

static void Main(string[] args)

{

const string script = @"

local function MGETSUM(keys)

local sum = 0

for _,key in ipairs(keys) do

local val = redis.call('GET', key) or 0

sum = sum + tonumber(val)

end

return sum

end

return MGETSUM(KEYS)";

var redis = RedisStore.RedisCache;

var server = RedisStore.Server;

var test1 = "test1";

var test2 = "test2";

var test3 = "test3";

//delete the keys

redis.KeyDelete(test1, CommandFlags.FireAndForget);

redis.KeyDelete(test2, CommandFlags.FireAndForget);

redis.KeyDelete(test3, CommandFlags.FireAndForget);

redis.StringSet(test1, 1);

redis.StringSet(test2, 2);

redis.StringSet(test3, 3);

RedisKey[] keys = { test1, test2, test3 };

byte[] hash = server.ScriptLoad(script);

//get back the sha1 key

string hexHash = string.Concat(hash.Select(x => x.ToString("X2")));

var result = redis.ScriptEvaluate(hexHash, keys);

Console.WriteLine(result); //result will be 6

Console.ReadKey();

}

You may realize that every time we call the server we need to load the script and then execute on it, there is also another way that StackExchange Redis allows us to to avoid the overhead of transmitting the script text for every call. One can convert a LuaScript into a LoadedLuaScript like the code below:

//create another script to store

var script2 = @"

local name=@name

local age=@age

return name..' '..age

var prepared = LuaScript.Prepare(script2);

//we now need to cache this loaded script somewhere

var loaded = prepared.Load(server);

//such that we do not need to uplaod the script every time

var val = loaded.Evaluate(redis, new { name="taswar", age=5});

//Output: taswar 5

Console.WriteLine(val);

val = loaded.Evaluate(redis, new { name = "peter", age = 25 });

//Output: peter 25

Console.WriteLine(val);

You would cache the loaded value somewhere in your application, usually its best to load the scripts when you start your application

So this covers the basic usage of Redis LuaScript, in the next blog post I will cover how to use Geo spatial data in Redis.

For the code please visit
https://github.com/taswar/RedisForNetDevelopers/tree/master/12.RedisLua/RedisLua

For previous Redis topics

.NET, Redis

lua, redis, StackExchange.Redis

OAuth and OpenId Connect Demystified – DevTeach Montreal 2017 Talk

Taswar Bhatti July 14, 2017 No Comments

Wanted to share my DevTeach Montreal 2017 talk where I talked about OAuth and OpenId Connect. The types of OAuth Grants, how to consume them, the flows in OAuth and what OpenId Connect comes into play, what does it solve.

Hope you like the presentation and if you are interested in more security topics, ping me and let me know what would you be interested in.

Devteach 2017 OAuth and Open id connect demystified from Taswar Bhatti, MVP

Transcript

1. OAUTH2 & OPENID CONNECT DEMYSTIFIED Taswar Bhatti (Microsoft MVP) GEMALTO @taswarbhatti http://taswar.zeytinsoft.co m taswar@gmail.com
2. WHO AM I?? – 4 years Microsoft MVP – 17 years in software – Author of Instant Automapper (Packt) – Currently working at as System Architect at Enterprise Security Space (Gemalto) – You may not have heard of Gemalto but 1/3 of the world population uses Gemalto but they just dont know that
3. WHAT WE WILL COVER TODAY? OAuth 2.0 OAuth flows OpenID JWT (JavaScript Web Token) some says “jot” OpenID Connect Demo (Keycloak IDP)
4. WHAT IS OAUTH? An open protocol to allow secure authorization in a simple and standard method from web, mobile and desktop applications.
5. OAUTH HISTORY OAuth started circa 2007 2010 – RFC 5849 defines OAuth 1.0 2010 – OAuth 2.0 work begins in IETF Working deployments of various drafts & versions at Google, Microsoft, Facebook, Github , Twitter, Flickr, Dropbox … Mid 2012 – Lead author and editor resigned & withdraws his name from all specs (DRAMA……) October 2012 – RFC 6749, RFC 6750
6. THE GOOD OAuth 2.0 is easier to implement than OAuth 1.0 Wide spread and continue growing Shorted lived token Encapsulated Token OAuth2 makes it HTTP/JSON friendly to request and transmit tokens Takes “multiple client” architectures into account Clients can have varying trust levels
7. OAUTH 2.0 – Transport Security : Using HTTPS and TLS – Ease : Usable (no digital certs to verify) – Flexible : Mobile, Web SPA apps, etc – Decoupled: Resource server and authorization server – Bearer Token : Easy for integration; Id Token also known as keys 9/24/2017 7
8. SO I CAN USE MY PASSWORD??? 9/24/2017 8
9. OAUTH IS LIKE A VALET KEY – Provides another domain delegated access to your application server resources 9/24/2017 9
10. OAUTH ROLES 9/24/2017 10 User Application API
11. OAUTH ROLES 9/24/2017 11 User Application API
12. OAUTH MISCONCEPTION Ohh this is easy!! When I login to Spotify with Twitter, it grabs by username and password from Twitter…. Wrong !!!!!!!!!!!!!! 9/24/2017 12 Developer
13. OAUTH IS NOT FOR 9/24/2017 13 – Traditional Access Control – Not for authentication – Not for Federation – OAuth should be used for delegation
14. BEARER TOKEN GET /somedata HTTP/1.1 Host: someserver.com Authorization: Bearer a3b4c55cf The access token can be JWT format – A security token with the property that any party in possession of the token (a “bearer”) can use the token in any way that any other party in possession of it can 9/24/2017 14
15. OAUTH TERMINOLOGY – Client or Consumer Application : Is typically a web based or mobile application that wants to access User’s Protected Resources – Resource Server or the Resource Provider: Is a web site or web service API where the User keeps his/her protected data – Authorized Server : The server issuing access tokens to the client after successfully authenticating the resources and obtaining authorization – User or the Resource Owner : Is a member of the Resource Provider, wanting to share certain resources with a third party – Client Credentials : Are the consumer keys and consumer secret used to authenticated the Client – Tokens : are the access token generated by server after request from client
16. OAUTH TOKEN TYPES – Access Token : Used to directly access protected resources on behalf of a user or service – Refresh Token : When given to an authorization server, it will give you a new access token – Authorization Code Token : Use only in the authorization code grant type for access token or refresh token 9/24/2017 16
17. HIGH LEVEL FLOW OF OAUTH 2 – An app registers him/herself on an oauth service provider (lets say twitter) – S/he gets an app key/secret for each app that s/he registers – When users login they are redirected to the service provider to provide the credentials – If user approves then a token is issued to the app for a limited time – Finally the client uses the token to access the resource
18. OAUTH USAGE In OAuth [authorization] You are in BigPhotoPrintingCorp.net account and you need to access your images from AwesomeImage.com site BigPhotoPrintingCorp.net site will redirect you to AwesomeImage.com site You enter you credential to AwesomeImage.com site and authenticated your self. This is like in openId AwesomeImage.com site will ask if you want to give permission to access only photos of AwesomeImage.com site you select yes AwesomeImage.com site will redirect back to BigPhotoPrintingCorp.net site BigPhotoPrintingCorp.net can access AwesomeImage.com site
19. GRANTING THE WRONG PERMISSIONS 9/24/2017 19
20. 4 TYPES OF OAUTH FLOW Authorization Code Grant : for apps running on a web server, long lived tokens Implicit Grant : For browser-based or mobile apps, during user is logged in, short lived tokens Resource Owner Credentials Grant : For logging in with a username and password, trusted application Client credentials Grant : for application access machine to machine
21. AUTHORIZATION CODE FOR APPS RUNNING ON A WEB SERVER This is the most common type of application you have when dealing with OAuth servers. Web apps are run on a server where the source code of the application is not available to the public. This case your site will REDIRECT you to particular authorization server. If webserver making multiple request it can use STATE parameter for map callback response with request One of the most complicated one in OAuth
22. YOU HAVE SEEN THIS BEFORE 9/24/2017 22
23. IMPLICIT FOR BROWSER-BASED OR MOBILE APPS Browser-based apps run entirely in the browser after getting source code from a web server. Since the entire source code is available to the public, they cannot maintain the confidentiality of their client secret, so the secret is not used in this case One will make api calls with the token that is assign to it For mobile apps also cannot maintain the confidentiality of their client secret. Because of this, mobile apps must also use an OAuth flow that does not require a client secret. With this concept token is exposed to local operating system. So there are no refresh tokens.
24. PASSWORD FOR LOGGING IN WITH A USERNAME AND PASSWORD OAuth 2 also provides a “password” grant type which can be used to exchange a username and password for an access token directly. This obviously requires the application to collect the user’s password. As a result users may hesitate to use this service unless this app comes from the auth service provider. Only used in highly trusted application, your social media Facebook app, rather than 3rd party apps (Batman Fancy Facebook app)
25. MEET THE ACTORS IN OUR OAUTH 9/24/2017 25 Resource Owner Or User Application Authorization Server Resource Server Or API
26. CLIENT CREDENTIALS FOR APPLICATION ACCESS There are scenarios that applications may wish to get statistics about the users of the app. In this case, applications need a way to get an access token for their own account, outside the context of any specific user. OAuth provides the client credentials grant type for this purpose. This is machine to machine communication sort of concept
27. 9/24/2017 27
28. TOKEN – CLIENT CREDENTIAL GRANT $ curl –XPOST https://api.mysite.com/oauth/token -d ‘grant_type=client_credentials’ -d ‘client_id=TestClient’ -d ‘client_secret=TestSecret’ 9/24/2017 28
29. TOKEN – CLIENT CREDENTIAL GRANT Response from Authorization Server { “access_token”:”03807cb390319329bdf6c777d4dfae9c0d3b3c35″, “expires_in”:3600, token_type”:”bearer”, “scope”:null } 9/24/2017 29
30. PASSWORD GRANT TYPE 9/24/2017 30
31. PASSWORD GRANT $ curl –XPOST https://api.mysite.com/oauth/token -d ‘client_id=TestClient’ -d ‘client_secret=TestSecret’ -d ‘grant_type=password’ -d ‘username=batman’ -d ‘password=nananananananannaBatman’ 9/24/2017 31
32. SCOPES AKA PERMISSIONS – Roles, Authority where you want to give access control to who can do what with it – The name of permissions – User scopes – Client/Applications Scopes – Token contains intersection 9/24/2017 32
33. SCOPES 9/24/2017 33 CarKey.Ignite
34. SCOPES 9/24/2017 34 CarKey.OpenTrunk CarKey.Ignite
35. SCOPES IN TOKEN Response from Authorization Server { “access_token”:”03807cb390319329bdf6c777d4dfae9c0d3b3c35″, “expires_in”:3600, token_type”:”bearer”, “scope”: “CarKey.Ignite” } 9/24/2017 35
36. AUTHORIZATION CODE GRANT TYPE 9/24/2017 36
37. AUTHORIZATION GRANT $ https://fancy.mysite.com/oidc #Reaching out to application, are you logged in? 302 HTTP Redirect https://api.mysite.com/authorize?response_type=code&client_id=Te stClient&redirect_uri=https://fancy.mysite.com/oidc 9/24/2017 37
38. AUTHORIZATION CODE GRANT GET /oauth/authorize #Login to the app SUCCESS you get back a code HTTP 302 redirect back to redirect_uri https://fancy.mysite.com/oidc?code=SplxlOBeZQQYbYS6WxSbIA&stat e=xyz 9/24/2017 38
39. AUTHORIZATION CODE GETTING THE TOKEN $ curl –XPOST https://api.mysite.com/oauth/token -d ‘client_id=TestClient’ -d ‘client_secret=TestSecret’ -d ‘grant_type=authorization_code’ -d ‘code=SplxlOBeZQQYbYS6WxSbIA’ 9/24/2017 39
40. ACCESS TOKEN Response from Authorization Server { “access_token”:”03807cb390319329bdf6c777d4dfae9c0d3b3c35″, “expires_in”:3600,“ token_type”:”bearer”, “scope”: “CarKey.Ignite” } 9/24/2017 40
41. RESOURCE SERVER CHECK TOKEN – If it is a Jwt token you can verify the key who signed it – Endpoint to check the token returning the scopes to verify if valid token 9/24/2017 41
42. IMPLICT GRANT TYPE – Used for clients that can easily be impersonated like phone or mobile application – 3rd party application – A simplified Authorization Code Grant with eliminating the code step – Access token is given directly to the app – No Refresh Token are given, Access token are short lived – Requires Resource Owner to invoke for new Access Token 9/24/2017 42
43. IMPLICIT GRANT FLOW
44. OPENID Sharing a single Identity with different consumers Decentralized OpenID is a form of Single Sign On (SSO) OpenID is a URL http://myname.myopenid.com
45. WHAT CAN YOU DO? One can claim and prove they own the openid Use it for authentication At a high level its like Microsoft Passport It’s a form of authentication, if you have a system you still will need to populate your fields (e.g firstname, email, etc) OpenId does not provide you with those information
46. OPENID USAGE In OpenId [authentication] You want to access your account on bigcorp.net bigcorp.net is asking your openId You entered your username for openId bigcorp.net will redirect you to the your openid providers site User give password to openId provider and authenticate him/her self openId provider will redirect user back to bigcorp.net site bigcorp.net will grant you to access your account
47. OPENID CONNECT We have talked about OAuth and OpenId and there is also OpenId Connect It’s the new SSO authentication for the internet OpenId Connect build on top of OAuth2 since sometimes you may just need authentication Remember OAuth2 is for authorization OpenID Connect provides Implict flows and Authorization code flow
48. OPENID CONNECT FLOW – SIMPLE 9/24/2017 48
49. OPENID CONNECT TOKEN { “sub” : “alice”, “user_name” : “Taswar” “iss” : “https://openid.c2id.com”, “aud” : “client-12345”, “auth_time” : 123456789, “iat” : 1311280970, “exp” : 1311281970, “email” : Taswar@gmail.com, “phone_number”: 123-4567 } 9/24/2017 49
50. OPENID CONNECT – HYBRID 9/24/2017 50
51. JWT (JAVA WEB TOKEN) JSON Web Token (JWT) is a compact URL-safe means of representing claims to be transferred between two parties. The claims in a JWT are encoded as a JavaScript Object Notation (JSON) object that is used as the payload of a JSON Web Signature (JWS) structure or as the plaintext of a JSON Web Encryption (JWE) structure, enabling the claims to be digitally signed or MACed and/or encrypted.
52. JWT CONT JWT Token looks like this eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjEzODY4OTkxMzEsI mlzcyI6ImppcmE6MTU0ODk1OTUiLCJxc2giOiI4MDYzZmY0Y2ExZTQx ZGY3YmM5MGM4YWI2ZDBmNjIwN2Q0OTFjZjZkYWQ3YzY2ZWE3OTdi NDYxNGI3MTkyMmU5IiwiaWF0IjoxMzg2ODk4OTUxfQ.uKqU9dTB6gK wG6jQCuXYAiMNdfNRw98Hw_IWuA5MaMo Ok great…………. Once you understand the format, it’s actually pretty simple: .. [header].[payload].[signature] 53. JWT CONT In other words: You create a header object, with the JSON format. Then you encode it as a base64 You create a claims object, with the JSON format. Then you encode it in base64 You create a signature for the URI. Then you encode it in base64 You concatenate the three items, with the “.” separator
54. BENEFITS JSON Web Tokens work across different programming languages: JWTs work in .NET, Python, Node.js, Java, PHP, Ruby, Go, JavaScript, and Haskell. So you can see that these can be used in many different scenarios. JWTs are self-contained: They will carry all the information necessary within itself. This means that a JWT will be able to transmit basic information about itself, a payload (usually user information), and a signature. JWTs can be passed around easily: Since JWTs are self-contained, they are perfectly used inside an HTTP header when authenticating an API. You can also pass it through the URL.
55. HEADER The header carries 2 parts (JWT and the hashing algorithm like below) { “typ”: “JWT”. “algo”, “HS256” } Then base64 encode it
56. PAYLOAD & CLAIMS The payload will carry the bulk of our JWT, also called the JWT Claims. This is where we will put the information that we want to transmit and other information about our token. There are multiple claims that we can provide. This includes registered claim names, public claim names, and private claim names. { “iss”: “taswar.zeytinsoft.com”, “exp”: 1300819380, “name”: “Taswar Bhatti”, “admin”: true }
57. SIGNATURE The third and final part of our JSON Web Token is going to be the signature. This signature is made up of a hash of the following components: the header the payload Secret The secret is the signature held by the server. This is the way that our server will be able to verify existing tokens and sign new ones. var encodedString = base64UrlEncode(header) + “.” + base64UrlEncode(payload); HMACSHA256(encodedString, ‘secret’);
58. ENDPOINTS OF OPENID CONNECT Authorization Endpoint (Regular OAuth) Identity Endpoint (username/pass, hardware token, biometrics) UserInfo Endpoint (name, birthday, picture, etc) Optionals (Session Endpoint, WebFinger etc)
59. OPENID CONNECT TOKENS The OpenID Connect server provides client applications with two key tokens: ID token – asserts the users identity in a signed and verifiable way. Access token – provides access to the user’s details at the UserInfo endpoint and other protected web APIs.
60. DEMO
61. THANK YOU Questions? Contact: Taswar@gmail.com Blog: http://Taswar.zeytinsoft.com Twitter: @taswarbhatti And a special thanks to Lego Batman !

OAuth2, Public Speaking, Security

devteach, oauth2, openid, openid connect, security

Elastic Stack talk – DevTeach Montreal 2017

Taswar Bhatti July 13, 2017 No Comments

Wanted to share my DevTeach talk slides on Elastic Search. Where I went into introducing the Elastic Stack. Consisting of Elastic Search, Logstash and Kibana. I also went into the constraints that we had and the design approaches that we took.

Hope you enjoy and expect more ElasticSearch blogs this year 🙂

Devteach 2017 Store 2 million of audit a day into elasticsearch from Taswar Bhatti, MVP

Transcript

1. STORE 2 MILLION OF AUDIT LOGS A DAY INTO ELASTICSEARCH Taswar Bhatti (Microsoft MVP) GEMALTO @taswarbhatti http://taswar.zeytinsoft.co m taswar@gmail.com
2. WHO AM I? – 4 years Microsoft MVP – 17 years in software industry – Currently working as System Architect in Enterprise Security Space (Gemalto) – You may not have heard of Gemalto but 1/3 of the world population uses Gemalto they just dont know it – Gemalto has stacks build in many environnent .NET, Java, Node, Lua, Python, mobile (Android, IOS), ebanking etc 9/22/2017 2
3. AGENDA – Problem we had and wanted to solve with Elastic Stack – Intro to Elastic Stack (Ecosystem) – Logstash – Kibana – Beats – Elastic Search flows designs that we have considered – Future plans of using Elastic Search 9/22/2017 3
4. QUESTION & POLL – How many of you are using Elastic or some other logging solution? – How do you normally log? Where do you log? – Do you log in Relational Database? 9/22/2017 4
5. HOW DO YOU TROUBLESHOOT OR FIND YOUR BUGS – Typically in a distributed environment one has to go through the logs to find out where the issue is – Could be multiple systems that you have to go through which machine/server generated the log or monitoring multiple logs – Even monitor firewall logs to find traffic routing through which data center – Chuck Norris never troubleshoot; the trouble kills themselves when they see him coming 9/22/2017 5
6. 9/22/2017 6
7. OUR PROBLEM – We had distributed systems (microservices) that would generate many different types of logs, in different data centers – We also had authentication audit logs that had to be secure and stored for 1 year – We generate around 2 millions records of audit logs a day, 4TB with replications – We need to generate reports out of our data for customers – We were still using Monolith Solution in some core parts of the application – Growing pains of a successful application – We want to use a centralized scalable logging system for all our9/22/2017 7
8. FINDING BUGS THROUGH LOGS 9/22/2017 8
9. A LITTLE HISTORY OF ELASTICSEARCH – Shay Banon created Compass in 2004 – Released Elastic Search 1.0 in 2010 – ElasticSearch the company was formed in 2012 – Shay wife is still waiting for her receipe app 9/22/2017 9
10. 9/22/2017 10
11. ELASTIC STACK 9/22/2017 11
12. ELASTICSEARCH – Written in Java backed by Lucene – Schema free, REST & JSON based document store – Search Engine – Distributed, Horizontally Scalable – No database storage, storage is Lucene – Apache 2.0 License 9/22/2017 12
13. COMPANIES USING ELASTIC STACK 9/22/2017 13
14. ELASTICSEARCH INDICES – Elastic organizes document in indices – Lucene writes and maintains the index files – ElasticSearch writes and maintains metadata on top of Lucene – Example: field mappings, index settings and other cluster metadata 9/22/2017 14
15. DATABASE VS ELASTIC 9/22/2017 15
16. ELASTIC CONCEPTS – Cluster : A cluster is a collection of one or more nodes (servers) – Node : A node is a single server that is part of your cluster, stores your data, and participates in the cluster’s indexing and search capabilities – Index : An index is a collection of documents that have somewhat similar characteristics. (e.g Product, Customer, etc) – Type : Within an index, you can define one or more types. A type is a logical category/partition of your index. – Document : A document is a basic unit of information that can be indexed – Shard/Replica: Index divided into multiple pieces called shards, replicas are copy of your shards9/22/2017 16
17. ELASTIC NODES – Master Node : which controls the cluster – Data Node : Data nodes hold data and perform data related operations such as CRUD, search, and aggregations. – Ingest Node : Ingest nodes are able to apply an ingest pipeline to a document in order to transform and enrich the document before indexing – Coordinating Node : only route requests, handle the search reduce phase, and distribute bulk indexing. 9/22/2017 17
18. SAMPLE JSON DOCUMENT HTTP CALL JSON DOCUMENT 9/22/2017 18
19. ELASTICSEARCH CLUSTER 9/22/2017 19
20. TYPICAL CLUSTER SHARD & REPLICA 9/22/2017 20
21. SHARD SEARCH AND INDEX 9/22/2017 21
22. DEMO OF ELASTICSEARCH 9/22/2017 22
23. LOGSTASH – Ruby application runs under JRuby on the JVM – Collects, parse, enrich data – Horizontally scalable – Apache 2.0 License – Large amount of public plugins written by Community https://github.com/logstash- plugins 9/22/2017 23
24. TYPICAL USAGE OF LOGSTASH 9/22/2017 24
25. 9/22/2017 25
26. LOGSTASH INPUT 9/22/2017 26
27. LOGSTASH FILTER 9/22/2017 27
28. LOGSTASH OUTPUT 9/22/2017 28
29. DEMO LOGSTASH 9/22/2017 29
30. BEATS 9/22/2017 30
31. BEATS – Lightweight shippers written in Golang (Non JVM shops can use them) – They follow unix philosophy; do one specific thing, and do it well – Filebeat : Logfile (think of it tail –f on steroids) – Metricbeat : CPU, Memory (like top), redis, mongodb usage – Packetbeat : Wireshark uses libpcap, monitoring packet http etc – Winlogbeat : Windows event logs to elastic – Dockbeat : Monitoring docker – Large community lots of other beats offered as opensource 9/22/2017 31
32. 9/22/2017 32
33. FILEBEAT 9/22/2017 33
34. X-PACK – Elastic commercial offering (This is one of the ways they make money) – X-Pack is an Elastic Stack extension that bundles – Security (https to elastic, password to access Kibana) – Alerting – Monitoring – Reporting – Graph capabilities – Machine Learning 9/22/2017 34
35. 9/22/2017 35
36. KIBANA – Visual Application for Elastic Search (JS, Angular, D3) – Powerful frontend for dashboard for visualizing index information from elastic search – Historical data to form charts, graphs etc – Realtime search for index information 9/22/2017 36
37. 9/22/2017 37
38. DEMO KIBANA 9/22/2017 38
39. DESIGNS WE WENT THROUGH – We started with simple design to measure throughput – One instance of logstash and one instance of ElasticSearch with filebeat 9/22/2017 39
40. DOTNET CORE APP – We used a dotnetcore application to generate logs – Serilog to generate into json format and stored on file – Filebeat was installed on the linux machine to ship the logs to logstash 9/22/2017 40
41. PERFORMANCE ELASTIC – 250 logs item per second for 30 minutes 9/22/2017 41
42. OVERVIEW 9/22/2017 42
43. LOGSTASH 9/22/2017 43
44. ELASTIC SEARCH RUN TWO – 1000 logs per second, run for 30 minutes 9/22/2017 44
45. PERFORMANCE 9/22/2017 45
46. OTHER DESIGNS 9/22/2017 46
47. WHAT WE ARE GOING WITH FOR NOW, UNTIL….. 9/22/2017 47
48. CONSIDERATIONS OF DATA – Index by day make sense in some cases – In other you may want to index by size rather (Black Friday more traffic than other days) when Shards are not balance ElasticSearch doesn’t like that – Don’t index everything, if you are not going to search on specific fields mark them as text 9/22/2017 48
49. FUTURE CONSIDERATIONS – Investigate into Elastic Search Machine learning – ElasticSearch with Kafka for cross data center replication 9/22/2017 49
50. THANK YOU & OPEN TO QUESTIONS – Questions??? – Contact: Taswar@gmail.com – Blog: http://Taswar.zeytinsoft.com – Twitter: @taswarbhatti – LinkedIn (find me and add me)

ElasticSearch, Public Speaking

devteach, elastic search, kibana, logstash

Introduction to Microsoft Bot Framework

Taswar Bhatti June 13, 2017 No Comments

In May 2017 I did a talk in Ottawa .NET User Group on Introduction to Microsoft Bot Framework, it was a interesting turnout and lots of conversation on what a bot can do for a business and how to use them.

Below you will find the slides for my talk on Microsoft Bot Framework. The sample demo code can be find on github https://github.com/Microsoft/BotBuilder-Samples, where we demo searching of Real Estate, image search, etc.

Ottawa IT Meetup Community: https://www.meetup.com/ottawaitcommunity/events/235920172/

An introduction to Microsoft Bot Framework from Taswar Bhatti, MVP

If you are interested in more on bot framework and like to see more articles on it, please let me know 🙂

Transcript

1. Introduction to Bot Framework Taswar Bhatti @taswarbhatti http://taswar.zeytinsoft.com
2. What is a Bot? • In this context Microsoft Bot Framework, bots are computer assistants or think of it as just a web api (ASP.NET Web App) • Bots appear in your favorite messaging app including Facebook Messenger and Skype • You use bots for booking flights, hotels, ordering takeaway and much, much more • Purchase Request Api to build commerce enabled bot
3. Not, not this Bot!
4. Bot Framework Overview • The Microsoft Bot Framework consists of: • Bot Builder SDK(.NET, Node.js, REST Api) • Bot Framework Emulator for Windows, Linux and Mac • Bot Connector • Developer Portal (http://dev.botframework.com) • Bot Directory
5. High Level view of Bots
6. Bot Connector
7. Deep Vision of Bot
8. An Echo Bot Code • [BotAuthentication] • public class MessagesController : ApiController{ • public async Task Post([FromBody]Activity activity) { • if (activity.Type == ActivityTypes.Message) { • ConnectorClient connector = new ConnectorClient(new Uri(activity.ServiceUrl)); • // return our reply to the user • Activity reply = activity.CreateReply($”Echo {activity.Text}”); • await connector.Conversations.ReplyToActivityAsync(reply); • } else { • HandleSystemMessage(activity); • } • var response = Request.CreateResponse(HttpStatusCode.OK); • return response; • } • }
9. Bot Emulator • The Bot Famework Emulator is a desktop application that allows bot developers to test and debug their bots on localhost • It is supported on Windows, Linux and Mac
10. Bot Emulator Screenshot
11. Demo Bot Framework
12. Bot Builder Dialog and Sessions • Bot builder uses dialogs to manage a bot conversation between a user • Session Objects are passed to the dialog handlers • Session Objects are the primary mechnics between bots and users
13. Types of Dialogs • Prompts – Text, Confirm, Number, Time, Choice, etc • Waterfall dialogs – collection and use sequence of steps • User Data – Store data of user, conversation (private data) Payload max is 32k • Intent Dialog – listen to keywords or phrases using NLP (e.g LUIS) • Attachments and Cards – For Rich Objects (media or images)
14. Demo Search Image and Card • Using cognitive vision api
15. Guideline for bots • Don’t send a bot to do a webpages’s job • Bots are just apps • Don’t abuse Natural Language Processing • Sometimes buttons do everything you need • Its all a matter of User Experience
16. Resources • Microsoft Bot Framework Resources (http://aka.ms/botresources) • LUIS (https://www.luis.ai) • Azure Trail (https://azure.Microsoft.com/services/bot-services) • Books (Building Bots with Microsoft Bot Framework PACKT) • Books (Programming the Microsoft Bot Framework MSPress) • An introduction to Microsoft Bot Framework
17. Questions • Questions?????

.NET

bot, bot framework, botframework, ms bot

Xamarin Dev Days

Taswar Bhatti May 15, 2017 No Comments

Last week I did a presentation on Xamarin Dev Days here in Ottawa, the presentation was on Introduction to Xamarin, unfortunately my demo did not work, the demo gods were not with me that day 🙁
The demo code uses Microsoft Cognitive Service Bing Search to search for images to display on a mobile phone. Mainly I was doing a demo on the Android platform. The code base is on Github
https://github.com/xamarin/dev-days-labs/tree/master/Demos/app-imagesearch-cogs

The slides for my presentation are located here.

Dev days 1 Introduction to Xamarin Taswar Bhatti from Taswar Bhatti, MVP

I also did another demo for Carleton University Business Program on Xamarin and if you wish to see more Xamarin related article please let me know 🙂

.NET, Android, Xamarin

.net, android, vs 2017, Xamarin

Redis for .NET Developer – Redis Transactions

Taswar Bhatti May 1, 2017 No Comments

Redis Transactions

Redis provides a way to do Transactions, but the transactions are somewhat different than what you know from a SQL Relational Database perspective. In SQL Relation Database world you can executed partially and then rolled back the entire transaction. While Redis uses the MULTI and EXEC commands where every command passed as part of a transaction is executed one after another until they have all completed and only after they have completed then other clients may execute their commands. You can think of it more like a queue that stores all your commands and when you call EXEC it loops through the queue and execute the task on the queue one after the other while being locked in the loop i,e no other thread could come in.

Below is an example of a Client queuing up the calls to Redis using MULTI in Redis client library to make a call to Redis Server to process the commands in an EXEC.

redis-transaction

Note: You see Client2 also making a call to Redis but the command is only executed after the process of Client commands.

One important thing to note is when using Redis Transaction one cannot make decisions inside the block, but you can use WATCH to prevent a key to be changed in a transaction. E.g We can WATCH the key “A” in our sample and when we try to use it in MULTI and EXEC it will fail to change A. A will be discarded as shown below, when A was set as 4, even inside of the block we were not able to make the change the EXEC was in some way ROLLBACKED i.e DISCARD

127.0.0.1:6379> WATCH A

127.0.0.1:6379> INCR A

(integer) 4

127.0.0.1:6379> GET A

"4"

127.0.0.1:6379> MULTI

127.0.0.1:6379> INCR A

QUEUED

127.0.0.1:6379> EXEC

(nil)

127.0.0.1:6379>

127.0.0.1:6379> GET A

"4"

In Redis StackExchange library uses Conditions to mimic Watch since Redis StackExchange uses Multiplex. Commands can come in different stages but conditions can be used to detect changes.

C# code using Redis Transactions

class Program
    {
        static void Main(string[] args)
        {
            var redis = RedisStore.RedisCache;

RedisKey alphaKey = "alphaKey";
            RedisKey betaKey = "betaKey";
            RedisKey gammaKey = "gammaKey";

redis.KeyDelete(alphaKey, CommandFlags.FireAndForget);
            redis.KeyDelete(betaKey, CommandFlags.FireAndForget);
            redis.KeyDelete(gammaKey, CommandFlags.FireAndForget);

var trans = redis.CreateTransaction();

var incr = trans.StringSetAsync(alphaKey, "abc");
            var exec = trans.ExecuteAsync();

var result = redis.Wait(exec);
            var alphaValue = redis.StringGet(alphaKey);

Console.WriteLine($"Alpha key is {alphaValue} and result is {result}");
            
            //using conditions to watch keys
            var condition = trans.AddCondition(Condition.KeyNotExists(gammaKey));
            var keyIncrement = trans.StringIncrementAsync(gammaKey);
            exec = trans.ExecuteAsync();
            result = redis.Wait(exec);

var gammaValue = redis.StringGet(gammaKey);

Console.WriteLine($"Gamma key is {gammaValue} and result is {result}");

//fail condition
            condition = trans.AddCondition(Condition.KeyNotExists(gammaKey));
            keyIncrement = trans.StringIncrementAsync(gammaKey);
            exec = trans.ExecuteAsync();

//resultis false 
            result = redis.Wait(exec);

gammaValue = redis.StringGet(gammaKey);

//value is still 1 and result is false
            Console.WriteLine($"Gamma key is {gammaValue} and result is {result}");

Console.ReadKey();
        }
    }

class Program

{

static void Main(string[] args)

{

var redis = RedisStore.RedisCache;

RedisKey alphaKey = "alphaKey";

RedisKey betaKey = "betaKey";

RedisKey gammaKey = "gammaKey";

redis.KeyDelete(alphaKey, CommandFlags.FireAndForget);

redis.KeyDelete(betaKey, CommandFlags.FireAndForget);

redis.KeyDelete(gammaKey, CommandFlags.FireAndForget);

var trans = redis.CreateTransaction();

var incr = trans.StringSetAsync(alphaKey, "abc");

var exec = trans.ExecuteAsync();

var result = redis.Wait(exec);

var alphaValue = redis.StringGet(alphaKey);

Console.WriteLine($"Alpha key is {alphaValue} and result is {result}");

//using conditions to watch keys

var condition = trans.AddCondition(Condition.KeyNotExists(gammaKey));

var keyIncrement = trans.StringIncrementAsync(gammaKey);

exec = trans.ExecuteAsync();

result = redis.Wait(exec);

var gammaValue = redis.StringGet(gammaKey);

Console.WriteLine($"Gamma key is {gammaValue} and result is {result}");

//fail condition

condition = trans.AddCondition(Condition.KeyNotExists(gammaKey));

keyIncrement = trans.StringIncrementAsync(gammaKey);

exec = trans.ExecuteAsync();

//resultis false

result = redis.Wait(exec);

gammaValue = redis.StringGet(gammaKey);

//value is still 1 and result is false

Console.WriteLine($"Gamma key is {gammaValue} and result is {result}");

Console.ReadKey();

}

So this covers the basic usage of Redis Transactions, in the next blog post I will cover how to use Lua in Redis.

For the code please visit
https://github.com/taswar/RedisForNetDevelopers/tree/master/11.RedisTransaction

For previous Redis topics

C#, Redis

C#, redis, StackExchange.Redis

Redis for .NET Developers – Redis Pipeline Batching

Taswar Bhatti April 25, 2017 1 Comment

Redis provides a way to use Pipeline Batching to send messages to Redis but first we must understand that Redis uses tcp request response protocol, some may know it as client server model. If you are coming from a web http world, you will have no issue understanding client server, where the browser acts like a client and the web server running (IIS, Nginx or Apache) as the server.

Below is an example of a Client calling the Redis library to make a call to Redis Server to store a value of 10.

In the above diagram as you can see there is a large amount of time where the client is just waiting for a response, the round trip time can have a significant impact if the latency between the network is long.

Redis Pipelining

Redis provides a mechanics called pipelining, which allows a client to send multiple messages to Redis Server without waiting for a reply on each message. Something like the diagram below. Note: Something to consider, Redis does stored these messages into queue such that it can reply to the client and as a result it is using memory on the server.
redis-pipeline

In .NET we are lucky that we have TPL and asycn/await built into our languages, so the design choice that StackExchange.Redis has done is to use the framework itself to handle such situations. One can simply use the Wait() keyword or ContinueWith() for completion of task. There is also another case, which is the CommandFlags.FireAndForget that you may have seen in my previous examples, the FireAndForget allows us to continue immediately in our code since we do not care about the response we get back from Redis Server.

C# code using Redis Pipeliine (TPL)

class Program

{

static void Main(string[] args)

{

var redis = RedisStore.RedisCache;

RedisKey alphaKey = "alphaKey";

RedisKey betaKey = "betaKey";

redis.KeyDelete(alphaKey, CommandFlags.FireAndForget);

redis.KeyDelete(betaKey, CommandFlags.FireAndForget);

var writeTask = redis.StringSetAsync(alphaKey, "abc");

var writeBetaTask = redis.StringSetAsync(betaKey, "beta");

var readTask = redis.StringGetAsync(alphaKey);

redis.Wait(writeTask);

var readValue = redis.Wait(readTask);

Console.WriteLine($"Redis Task wait and read {readValue}");

writeBetaTask.Wait();

readValue = redis.StringGet(betaKey);

Console.WriteLine($"Task wait and read {readValue}");

Console.ReadKey();

}

Redis Batching

StackExchange.Redis also provides a way to send batch request to Redis. What it allows us to do is to send a block of operations to the server together. The reason for this is that it will help reduce packet fragmentation when the connection to redis is slow. The downside is it will take a longer time to get the first operation complete to process, but overall it can improve the time to get all the operation processed. Below is an example of using Batching.

redis-batch

C# code using Redis batching

class Program
    {
        static void Main(string[] args)
        {
            var redis = RedisStore.RedisCache;          
            
            RedisKey alphaKey = "alphaKey";
            RedisKey betaKey = "betaKey";

//Batching
            var list = new List<Task<bool>>();
            var keys = new List<RedisKey> {alphaKey, betaKey};
            IBatch batch = redis.CreateBatch();

//add the delete into batch
            batch.KeyDeleteAsync(alphaKey);

foreach (var key in keys)
            {
                var task = batch.StringSetAsync(key, "123");
                list.Add(task);
            }

batch.Execute();

Task.WhenAll(list.ToArray());

readTask = redis.StringGetAsync(alphaKey);
            readValue = redis.Wait(readTask);
            Console.WriteLine($"Alpha read value {readValue}");

readTask = redis.StringGetAsync(betaKey);
            readValue = redis.Wait(readTask);
            Console.WriteLine($"Beta read value {readValue}");

Console.ReadKey();
        }
    }

class Program

{

static void Main(string[] args)

{

var redis = RedisStore.RedisCache;

RedisKey alphaKey = "alphaKey";

RedisKey betaKey = "betaKey";

//Batching

var list = new List<Task<bool>>();

var keys = new List<RedisKey> {alphaKey, betaKey};

IBatch batch = redis.CreateBatch();

//add the delete into batch

batch.KeyDeleteAsync(alphaKey);

foreach (var key in keys)

{

var task = batch.StringSetAsync(key, "123");

list.Add(task);

}

batch.Execute();

Task.WhenAll(list.ToArray());

readTask = redis.StringGetAsync(alphaKey);

readValue = redis.Wait(readTask);

Console.WriteLine($"Alpha read value {readValue}");

readTask = redis.StringGetAsync(betaKey);

readValue = redis.Wait(readTask);

Console.WriteLine($"Beta read value {readValue}");

Console.ReadKey();

}

So this covers the basic usage of Redis Pipeline and Batching, the recommendation for StackExchange.Redis is to use the normal async/await of TPL to handle all the pipeline for you, since StackExchange.Redis does it best under the cover to handle multiplex connections. In the next blog post I will cover how to use Transactions in Redis.

For the code please visit
https://github.com/taswar/RedisForNetDevelopers/tree/master/10.RedisPipeline

For previous Redis topics

.NET, Redis

.net, redis, StackExchange.Redis

Redis for .NET Developers – Redis Sentinel

Redis for .NET Developers – Redis GeoSpatial

Redis Geo Datatype – Operations

C# code using Redis Geo Set Datatype

Redis for .NET Developer – Lua Scripting

Redis Lua Scripting

C# code using Redis Lua Script

OAuth and OpenId Connect Demystified – DevTeach Montreal 2017 Talk

Transcript

Elastic Stack talk – DevTeach Montreal 2017

Transcript

Introduction to Microsoft Bot Framework

Transcript

Xamarin Dev Days

Redis for .NET Developer – Redis Transactions

Redis Transactions

C# code using Redis Transactions

Redis for .NET Developers – Redis Pipeline Batching

Redis Pipelining

C# code using Redis Pipeliine (TPL)

Redis Batching

C# code using Redis batching

Opinions are my own and not the views of my employer

LinkedIn

My Book

Archives