Taswar Bhatti
The synonyms of software simplicity
Redis

If you are serious in running Redis, you will want to run it under HA mode (High Availability). So far for learning purpose you can run a single instance of redis under your docker environment quite easily, but what if you need to run it in production? This is where Redis Sentinel comes in to play, let’s see what the official document of Redis Sentinel have to say.

Redis Sentinel
Redis Sentinel provides high availability for Redis. In practical terms this means that using Sentinel you can create a Redis deployment that resists without human intervention to certain kind of failures.

The main thing that Redis Sentinel provides is when there is a failure on master node, it will automatically choose a salve and promote it to a master. How does it do that, it basically periodically check the health and live of a Redis instance, it will also notify clients and slaves about the new master. The protocol used is gossip protocol with leader election algorithms. Sentinel also acts as a central source of authority for client discovery, clients are connected to Sentinel for the address of the master node.

Things that Sentinel doesn’t do are mainly:

  1. Manage client connections
  2. Store configurations changes to disk

Usually you would want to run your Sentinel on different servers than your Redis Server, for the very simple fact that you don’t want your monitoring software on the same server right? Can you imagine a sql server with sql monitoring on the same machine to tell you if its alive? What if the sql server machine goes down, then both goes down? One can potentially run Sentinel on client nodes and its best to not to run it on master nodes.

Here is a typical way of setting up Sentinel.

redis-sentinel

redis-sentinel

Unfortunately Redis StackExchange does not support Sentinel there are pull request that have the functionality (https://github.com/StackExchange/StackExchange.Redis/pull/692) but have not been merged into Redis StackExchange yet as we speak.

What are the alternative? You can roll your own RedisClient and have Sentinel support or manage it yourself in your code, or use Service Stack is another option.
If you are interested in rolling your own you may want to go through this code by PaulB from Stack Overflow provided
https://stackoverflow.com/questions/25385075/redis-failover-with-stackexchange-sentinel-from-c-sharp

The concepts remain the same as in ask sentinel for who the master is and then do the connection.

Note: The code is not mine its from the link above.

In the next post we will go through redis cluster, I wanted to talk about sentinel first such that readers tend to mix them up and by first explaining sentinel we can talk about clusters which are different concepts.

Redis

Redis GeoSpatial data sets are actually just SortedSets in Redis, there is no secret about it. Basically it provides an easy way to store geo spatial data like longitude/latitude coordinates into Redis.Lets look at some of the commands that Redis provides for Geo Spatial data.

Redis Geo Datatype – Operations

  • GEOADD: Adds or updates one or more members to a Geo Set O(log (N)) where N is the number of elements in the sorted set.
  • GEODIST: Return the distance between two members in the geo spatial index represented by the sorted set O(log(N)).
  • GEOHASH: Gets valid Geohash strings representing the position of one or more elements from the Geo Sets O(log(N)), where N is the number of elements in the sorted set.
  • GEOPOS: Return the longitude,latitude of all the specified members of the geo spatial sorted set at key O(log(N)), where N is the number of elements in the sorted set.
  • GEORADIUS: Return the members of a sorted set populated with geo spatial information using GEOADD, which are within the borders of the area specified with the center location and the maximum distance from the center (the radius) O(N+log(M))
  • GEORADIUSBYMEMBER: Same as GEORADIUS with the only difference that instead of taking, as the center of the area to query, it takes a longitude and latitude value O(N+log(M))

I wanted to use some open data to showcase the usage of Redis GeoSpatial data in Redis. I chose to use Basketball courts in Ottawa since my son plays a bit of basketball.
Here is a map of basketball courts in Ottawa.

ottawa-basketball-court

Ottawa Basketball Courts

C# code using Redis Geo Set Datatype

So this covers the basic usage of Redish GeoSpatial Datatype, in the next blog post I will cover using Sentinel which provides high availability for Redis.

For the code please visit
https://github.com/taswar/RedisForNetDevelopers/tree/master/13.RedisGeo/RediusGeo

For previous Redis topics

  1. Intro to Redis for .NET Developers
  2. Redis for .NET Developer – Connecting with C#
  3. Redis for .NET Developer – String Datatype
  4. Redis for .NET Developer – String Datatype part 2
  5. Redis for .NET Developer – Hash Datatype
  6. Redis for .NET Developer – List Datatype
  7. Redis for .NET Developer – Redis Sets Datatype
  8. Redis for .NET Developer – Redis Sorted Sets Datatype
  9. Redis for .NET Developer – Redis Hyperloglog
  10. Redis for .NET Developer – Redis Pub Sub
  11. Redis for .NET Developers – Redis Pipeline Batching
  12. Redis for .NET Developers – Redis Transactions
  13. Redis for .NET Developers – Lua Scripting
  14. Redis for .NET Developers – Redis running in Docker
  15. Redis for .NET Developers – Redis running in Azure
  16. Redis for .NET Developers – Redis running in AWS ElastiCache
Redis

Redis Lua Scripting

Redis provides a way to extend its functionality on the server side by providing support for Lua Scripting. If you are coming from a relational database world, you already know that you can use Stored Procedures to extend functionality of your relational database. Now, you may also know that some people do frown upon using stored procedures, I think one could also think of using scripting in Redis sort of belongs in the same category. Nevertheless its still good to know what you can do with Redis and Lua.

If you want to learn more Lua try this site http://tylerneylon.com/a/learn-lua/

In order to call Lua Script from Redis.StackExchange library one can use the LuaScript class or IServer.ScriptLoad(Async), IServer.ScriptExists(Async), IServer.ScriptFlush(Async), IDatabase.ScriptEvaluate, and IDatabaseAsync.ScriptEvaluateAsync methods.

Lets try to do something with the redis console first by using the redis-cli

As you can see we first load the script and we get back a SHA1 Hash from it. Redis basically stores the information in one of its mapping table and we can reuse the sha1 hash to call the script by using the EVALSHA command, which in this case gave us back “hello redis”

Remember: When Redis is running your Lua script, it will not run anything else because Redis is single threaded

C# code using Redis Lua Script

You may realize that every time we call the server we need to load the script and then execute on it, there is also another way that StackExchange Redis allows us to to avoid the overhead of transmitting the script text for every call. One can convert a LuaScript into a LoadedLuaScript like the code below:

You would cache the loaded value somewhere in your application, usually its best to load the scripts when you start your application

So this covers the basic usage of Redis LuaScript, in the next blog post I will cover how to use Geo spatial data in Redis.

For the code please visit
https://github.com/taswar/RedisForNetDevelopers/tree/master/12.RedisLua/RedisLua

For previous Redis topics

  1. Intro to Redis for .NET Developers
  2. Redis for .NET Developer – Connecting with C#
  3. Redis for .NET Developer – String Datatype
  4. Redis for .NET Developer – String Datatype part 2
  5. Redis for .NET Developer – Hash Datatype
  6. Redis for .NET Developer – List Datatype
  7. Redis for .NET Developer – Redis Sets Datatype
  8. Redis for .NET Developer – Redis Sorted Sets Datatype
  9. Redis for .NET Developer – Redis Hyperloglog
  10. Redis for .NET Developer – Redis Pub Sub
  11. Redis for .NET Developers – Redis Pipeline Batching
  12. Redis for .NET Developers – Redis Transactions
oauth and openid_

Wanted to share my DevTeach Montreal 2017 talk where I talked about OAuth and OpenId Connect. The types of OAuth Grants, how to consume them, the flows in OAuth and what OpenId Connect comes into play, what does it solve.

Hope you like the presentation and if you are interested in more security topics, ping me and let me know what would you be interested in.

elastic search

Wanted to share my DevTeach talk slides on Elastic Search. Where I went into introducing the Elastic Stack. Consisting of Elastic Search, Logstash and Kibana. I also went into the constraints that we had and the design approaches that we took.

Hope you enjoy and expect more ElasticSearch blogs this year 🙂

Taswar Bhatti Talk on MS Bot Framework

In May 2017 I did a talk in Ottawa .NET User Group on Introduction to Microsoft Bot Framework, it was a interesting turnout and lots of conversation on what a bot can do for a business and how to use them.

Below you will find the slides for my talk on Microsoft Bot Framework. The sample demo code can be find on github https://github.com/Microsoft/BotBuilder-Samples, where we demo searching of Real Estate, image search, etc.

Ottawa IT Meetup Community: https://www.meetup.com/ottawaitcommunity/events/235920172/

If you are interested in more on bot framework and like to see more articles on it, please let me know 🙂

Taswar Bhatti Xamarin Dev Days

Last week I did a presentation on Xamarin Dev Days here in Ottawa, the presentation was on Introduction to Xamarin, unfortunately my demo did not work, the demo gods were not with me that day 🙁
The demo code uses Microsoft Cognitive Service Bing Search to search for images to display on a mobile phone. Mainly I was doing a demo on the Android platform. The code base is on Github
https://github.com/xamarin/dev-days-labs/tree/master/Demos/app-imagesearch-cogs

The slides for my presentation are located here.

I also did another demo for Carleton University Business Program on Xamarin and if you wish to see more Xamarin related article please let me know 🙂

Redis

Redis Transactions

Redis provides a way to do Transactions, but the transactions are somewhat different than what you know from a SQL Relational Database perspective. In SQL Relation Database world you can executed partially and then rolled back the entire transaction. While Redis uses the MULTI and EXEC commands where every command passed as part of a transaction is executed one after another until they have all completed and only after they have completed then other clients may execute their commands. You can think of it more like a queue that stores all your commands and when you call EXEC it loops through the queue and execute the task on the queue one after the other while being locked in the loop i,e no other thread could come in.

Below is an example of a Client queuing up the calls to Redis using MULTI in Redis client library to make a call to Redis Server to process the commands in an EXEC.

redis-transaction

redis-transaction


Note: You see Client2 also making a call to Redis but the command is only executed after the process of Client commands.

One important thing to note is when using Redis Transaction one cannot make decisions inside the block, but you can use WATCH to prevent a key to be changed in a transaction. E.g We can WATCH the key “A” in our sample and when we try to use it in MULTI and EXEC it will fail to change A. A will be discarded as shown below, when A was set as 4, even inside of the block we were not able to make the change the EXEC was in some way ROLLBACKED i.e DISCARD

In Redis StackExchange library uses Conditions to mimic Watch since Redis StackExchange uses Multiplex. Commands can come in different stages but conditions can be used to detect changes.

C# code using Redis Transactions

So this covers the basic usage of Redis Transactions, in the next blog post I will cover how to use Lua in Redis.

For the code please visit
https://github.com/taswar/RedisForNetDevelopers/tree/master/11.RedisTransaction

For previous Redis topics

  1. Intro to Redis for .NET Developers
  2. Redis for .NET Developer – Connecting with C#
  3. Redis for .NET Developer – String Datatype
  4. Redis for .NET Developer – String Datatype part 2
  5. Redis for .NET Developer – Hash Datatype
  6. Redis for .NET Developer – List Datatype
  7. Redis for .NET Developer – Redis Sets Datatype
  8. Redis for .NET Developer – Redis Sorted Sets Datatype
  9. Redis for .NET Developer – Redis Hyperloglog
  10. Redis for .NET Developer – Redis Pub Sub
  11. Redis for .NET Developers – Redis Pipeline Batching
Redis

Redis provides a way to use Pipeline Batching to send messages to Redis but first we must understand that Redis uses tcp request response protocol, some may know it as client server model. If you are coming from a web http world, you will have no issue understanding client server, where the browser acts like a client and the web server running (IIS, Nginx or Apache) as the server.

Below is an example of a Client calling the Redis library to make a call to Redis Server to store a value of 10.


In the above diagram as you can see there is a large amount of time where the client is just waiting for a response, the round trip time can have a significant impact if the latency between the network is long.

Redis Pipelining

Redis provides a mechanics called pipelining, which allows a client to send multiple messages to Redis Server without waiting for a reply on each message. Something like the diagram below. Note: Something to consider, Redis does stored these messages into queue such that it can reply to the client and as a result it is using memory on the server.
redis-pipeline

In .NET we are lucky that we have TPL and asycn/await built into our languages, so the design choice that StackExchange.Redis has done is to use the framework itself to handle such situations. One can simply use the Wait() keyword or ContinueWith() for completion of task. There is also another case, which is the CommandFlags.FireAndForget that you may have seen in my previous examples, the FireAndForget allows us to continue immediately in our code since we do not care about the response we get back from Redis Server.

C# code using Redis Pipeliine (TPL)

Redis Batching

StackExchange.Redis also provides a way to send batch request to Redis. What it allows us to do is to send a block of operations to the server together. The reason for this is that it will help reduce packet fragmentation when the connection to redis is slow. The downside is it will take a longer time to get the first operation complete to process, but overall it can improve the time to get all the operation processed. Below is an example of using Batching.

redis-batch

C# code using Redis batching

So this covers the basic usage of Redis Pipeline and Batching, the recommendation for StackExchange.Redis is to use the normal async/await of TPL to handle all the pipeline for you, since StackExchange.Redis does it best under the cover to handle multiplex connections. In the next blog post I will cover how to use Transactions in Redis.

For the code please visit
https://github.com/taswar/RedisForNetDevelopers/tree/master/10.RedisPipeline

For previous Redis topics

  1. Intro to Redis for .NET Developers
  2. Redis for .NET Developer – Connecting with C#
  3. Redis for .NET Developer – String Datatype
  4. Redis for .NET Developer – String Datatype part 2
  5. Redis for .NET Developer – Hash Datatype
  6. Redis for .NET Developer – List Datatype
  7. Redis for .NET Developer – Redis Sets Datatype
  8. Redis for .NET Developer – Redis Sorted Sets Datatype
  9. Redis for .NET Developer – Redis Hyperloglog
  10. Redis for .NET Developer – Redis Pub Sub
Redis

Redis Pub Sub, somehow may sound weird that it provides such a functionality since when you think of Redis the first thing that comes to mind is of a cache key value store. But indeed Redis does allow us to use the messaging paradigm by using channels to publish messages and for subscribers to listen for notification of the message. Redis Pub Sub allows multiple subscribers to listen to one or more channels, and to only receive messages that are of interest. It decouples the subscriber from the publisher since the subscriber has no knowledge of whom the publishers are and vice versa there could be multiple publisher not knowing whom the subscribers are.

Sample Diagram of Redis Pub Sub

RedisPubSub

Redis Pub Sub

There are definitely certain restrictions of using Redis Pub Sub as a Messaging System, it will not be like RabbitMQ, Kafka or Azure MessageBus etc. Those message bus are able to store the message for durability or even replay of an old message for consumption. Redis uses a listener model where there are no listeners (subscribers) it will not receive those messages. But if you wish to have a simple pub sub without the heavy tools then Redis does quite a good job at it.

Redis Pub Sub – Operations

  • PUBLISH channels message: Posts a message to the given channel O(N+M)
  • SUBSCRIBE [channel]: Subscribe to a given channel for message, O(N) where N is the number of channels to subscribe to
  • PSUBSCRIBE [channel]: Subscribes the client to the given patterns, O(N) where N is the number of patterns the client is already subscribed to.
  • PUBSUB CHANNELS pattern: Currently active channels, O(N)
  • PUBSUB NUMSUB channel: Number of subscribers to the channels provided, O(N)
  • PUBSUB NUMPAT: Number of subscriptions to all the patterns O(N)
  • PUNSUBSCRIBE: Unsubscribes the client from a pattern, O(N+M)
  • UNSUBSCRIBE: Unsubscribes the client from a channel, O(N) where N is the number of clients already subscribed to a channel.

C# code using Redis PubSub

So this covers the basic usage of Redis PubSub, in the next blog post I will cover how to use Pipelines in Redis.

For the code please visit
https://github.com/taswar/RedisForNetDevelopers/tree/master/9.RedisPubSub

For previous Redis topics

  1. Intro to Redis for .NET Developers
  2. Redis for .NET Developer – Connecting with C#
  3. Redis for .NET Developer – String Datatype
  4. Redis for .NET Developer – String Datatype part 2
  5. Redis for .NET Developer – Hash Datatype
  6. Redis for .NET Developer – List Datatype
  7. Redis for .NET Developer – Redis Sets Datatype
  8. Redis for .NET Developer – Redis Sorted Sets Datatype
  9. Redis for .NET Developer – Redis Hyperloglog
UA-4524639-2