Taswar Bhatti
The synonyms of software simplicity

In November I went to Prague my first time and it was amazing being hosted by Update Conference there.
Below you will find my presentation on 8 Cloud Design Patterns that I did for Update Conference In Prague, I wanted to give a special thanks for @tomasherceg and his team for creating such an awesome conference.
Enjoy the sides!.

Transcript

1. 8 Cloud Design Patterns you ought to know Taswar Bhatti System/Solutions Architect at Gemalto (Canada) Microsoft MVP
2. Ponder • For every 25 percent increase in problem complexity, there is a 100 percent increase in solution complexity. • There is seldom one best design solution to a software problem. • If cars were like software, they would crash twice a day for no reason, and when you called for service, they’d tell you to reinstall the engine.
3. Who am I • Taswar Bhatti – Microsoft MVP since 2014 • Global Solutions Architect/System Architect at Gemalto • In Software Industry since 2000 • I know Kung Fu (Languages)
4. What I am not
5. Agenda • What are Patterns? • The External Configuration Pattern • The Cache Aside Pattern • The Federated Identity Pattern • The Valet Key Pattern • The Gatekeeper Pattern • The Circuit Breaker Pattern • The Retry Pattern • The Strangler Pattern • Demo • Questions
6. Bad Design
7. Bad Design
8. Bad Design
9. Bad Design
10. Bad Design
11. Feature doesn’t make sense????
12. Anger
13. Bad day at work
14. Happy family
15. Next day at work
16. Ship it
17. Customer Feature didn’t make sense
18. Bad Design?
19. Itunes when I use it
20. What are Patterns? • General reusable solution to a recurring problem • A template on how to solve a problem • Best practices • Patterns allow developers communicate with each other in well known and understand names for software interactions.
21. External Configuration Pattern
22. External Configuration Pattern • Helps move configuration information out of the application deployment • This pattern can provide for easier management and control of configuration data • For sharing configuration data across applications and other application instances
23. Typical Application
24. Storing Configuration in file
25. Multiple application
26. Problems • Configuration becomes part of deployment • Multiple applications share the same configuration • Hard to have access control over the configuration
27. External Configuration Pattern
28. When to use the pattern • When you have shared configuration, multiple application • You want to manage configuration centrally by DevOps • Provide audit for each configuration
29. When not to use • When you only have a single application there is no need to use this pattern it will make things more complex
30. Cloud Solution Offerings • Azure Key Vault • Vault by Hashicorp • AWS KMS • Keywhiz
31. Demo KeyVault
32. Cache Aside Pattern
33. Cache Aside Pattern • Load data on demand into a cache from datastore • Helps improve performance • Helps in maintain consistency between data held in the cache and data in the underlying data store.
34. Typical Application
35. Cache Aside Pattern
36. When to use the pattern • Resource demand is unpredictable. • This pattern enables applications to load data on demand • It makes no assumptions about which data an application will require in advance
37. When not to use • Don’t use it for data that changes very often
38. Things to consider • Sometimes data can be changed from outside process • Have an expiry for the data in cache • When update of data, invalidate the cache before updating the data in database • Pre populate the data if possible
39. Cloud Offerings • Redis (Azure and AWS) • Memcache • Hazelcast • Elastic Cache (AWS)
40. Federated Identity Pattern
41. Federated Identity Pattern • Delegate authentication to an external identity provider. • Simplify development, minimize the requirement for user administration • Improve the user experience of the application • Centralized providing MFA for user authentication
42. Typical Application
43. Problem
44. Problem • Complex development and maintenance (Duplicated code) • MFA is not an easy thing • User administration is a pain with access control • Hard to keep system secure • No single sign on (SSO) everyone needs to login again to different systems
45. Federated Identity Pattern
46. When to use • When you have multiple applications and want to provide SSO for applications • Federated identity with multiple partners • Federated identity in SAAS application
47. When not to use it • You already have a single application and have custom code that allows you to login
48. Things to consider • The identity Server needs to be highly available • Single point of failure, must have HA • RBAC, identity server usually does not have authorization information • Claims and scope within the security auth token
49. Cloud Offerings • Azure AD • Gemalto STA and SAS • Amazon IAM • GCP Cloud IAM
50. Valet Key Pattern
51. Valet Key Pattern • Use a token that provides clients with restricted direct access to a specific resource • Provide offload data transfer from the application • Minimize cost and maximize scalability and performance
52. Typical Application Client App Storage
53. Problem Client App Storage Client Client Client Client
54. Valet Key Pattern Client App Generate Token Limited Time And Scope Storage
55. When to use it • The application has limited resources • To minimize operational cost • Many interaction with external resources (upload, download) • When the data is stored in a remote data store or a different datacenter
56. When not to use it • When you need to transform the data before upload or download
57. Cloud Offerings • Azure Blob Storage • Amazon S3 • GCP Cloud Storage
58. Gatekeeper Pattern
59. Gatekeeper Pattern • Using a dedicated host instance that acts as a broker between clients and services • Protect applications and services • Validates and sanitizes requests, and passes requests and data between them • Provide an additional layer of security, and limit the attack surface of the system
60. Typical Application
61. Problem
62. Gatekeeper Pattern
63. When to use it • Sensitive information (Health care, Authentication) • Distributed System where perform request validation separately
64. When not to use • Performance vs security
65. Things to consider • WAF should not hold any keys or sensitive information • Use a secure communication channel • Auto scale • Endpoint IP address (when scaling application does the WAF know the new applications)
66. Circuit Breaker Pattern
67. Circuit Breaker Pattern • To handle faults that might take a variable amount of time to recover • When connecting to a remote service or resource
68. Typical Application
69. Problem
70. Client Circuit Breaker Api Closed State Timeout Closed State Open State Half Open State After X Retry Closed State
71. Circuit Breaker
72. When to use it • To prevent an application from trying to invoke a remote service or access a shared resource if this operation is highly likely to fail • Better user experience
73. When not to use • Handling access to local private resources in an application, such as in-memory data structure • Creates an overhead • Not a substitute for handling exceptions in the business logic of your applications
74. Libraries • Polly (http://www.thepollyproject.org/) • Netflix (Hystrix) https://github.com/Netflix/Hystrix/wiki
75. Retry pattern
76. Retry Pattern • Enable an application to handle transient failures • When the applications tries to connect to a service or network resource • By transparently retrying a failed operation
77. Typical Application Network Failure
78. Retry Pattern • Retry after 2, 5 or 10 seconds
79. When to use it • Use retry for only transient failure that is more than likely to resolve themselves quickly • Match the retry policies with the application • Otherwise use the circuit break pattern
80. When not to use it • Don’t cause a chain reaction to all components • For internal exceptions caused by business logic • Log all retry attempts to the service
81. Libraries • Roll your own code • Polly (http://www.thepollyproject.org/) • Netflix (Hystrix) https://github.com/Netflix/Hystrix/wiki
82. Demo Retry
83. Strangler Pattern
84. Strangler Pattern • Incrementally migrate a legacy system • Gradually replacing specific pieces of functionality with new applications and services • Features from the legacy system are replaced by new system features eventually • Strangling the old system and allowing you to decommission it
85. Monolith Application
86. Strangler Pattern
87. When to use • Gradually migrating a back-end application to a new architecture
88. When not to use • When requests to the back-end system cannot be intercepted • For smaller systems where the complexity of wholesale replacement is low
89. Considerations • Handle services and data stores that are potentially used by both new and legacy systems. • Make sure both can access these resources side-by-side • When migration is complete, the strangler façade will either go away or evolve into an adaptor for legacy clients • Make sure the façade doesn’t become a single point of failure or a performance bottleneck.
90. Questions? Taswar Bhatti System Solutions Architect (Gemalto) Microsoft MVP http://taswar.zeytinsoft.com @taswarbhatti
91. Credits • For the background • www.Vecteezy.com

elastic search introduction

In September I was invited to speak at the Ottawa Elastic Search Meetup Group at Kinaxis. It was great seeing new faces at the meetup, I spoke on Elastic Search Introduction so that new user group members can go through the entire stack of ElasticSearch, logstash and kibana. Hope to speak there some other time soon.

Here are my slides for anyone interested.

Transcript

1. A Gentle Intro to ElasticSearch Taswar Bhatti System/Solutions Architect (Ottawa) GEMALTO
2. Who amI?  System/Solution Architect at Gemalto Ottawa (Microsoft MVP)  I am somewhat of a language geek; I speak a few languages  Kind of like Neo (I KNOW KUNG FU) for languages 2 – Merhaba – नमस्ते – 你好 – ‫ہیلو‬ – Comment ca va? – ਸਤ ਸਰੀ ਅਕਾਲ
3. 9/14/2018 3 Reuters Top 100: Gemalto rated top Global Tech Leaders https://www.thomsonreuters.com/en/products-services/technology/top-100.html
4. Agenda  Problem we had and wanted to solve with Elastic Stack  Intro to Elastic Stack (Ecosystem)  Logstash  Kibana  Beats  Elastic Search flows designs that we have considered  Future plans of using Elastic Search 4
5. How doyouTroubleshootorfindyourbugs?  Typically in a distributed environment one has to go through the logs to find out where the issue is  Could be multiple systems that you have to go through which machine/server generated the log or monitoring multiple logs  Even monitor firewall logs to find traffic routing through which data center  Chuck Norris never troubleshoot; the trouble kills themselves when they see him coming 9/14/2018 5
6. 9/14/2018 6
7. OurProblem  We had distributed systems (microservices) that would generate many different types of logs, in different data centers  We also had authentication audit logs that had to be secure and stored for 1 year  We generate around 2 millions records of audit logs a day, 4TB with replications  We need to generate reports out of our data for customers  We were still using Monolith Solution in some core parts of the application  Growing pains of a successful application  We want to use a centralized scalable logging system for all our logs 9/14/2018 7
8. Findingbugsthroughlogs 9/14/2018 8
9. Alittlehistoryof ElasticSearch  Shay Banon created Compass in 2004  Released Elastic Search 1.0 in 2010  ElasticSearch the company was formed in 2012  Shay wife is still waiting for her receipe app 9/14/2018 9
10. 9/14/2018 10
11. ElasticStack 9/14/2018 11
12. ElasticSearch  Written in Java backed by Lucene  Schema free, REST & JSON based document store  Search Engine  Distributed, Horizontally Scalable  No database storage, storage is Lucene  Apache 2.0 License 9/14/2018 12
13. CompaniesusingElasticStack 9/14/2018 13
14. ElasticSearchindices  Elastic organizes document in indices  Lucene writes and maintains the index files  ElasticSearch writes and maintains metadata on top of Lucene  Example: field mappings, index settings and other cluster metadata 9/14/2018 14
15. Databasevs ElasticSearch 9/14/2018 15
16. ElasticConcepts  Cluster : A cluster is a collection of one or more nodes (servers)  Node : A node is a single server that is part of your cluster, stores your data, and participates in the cluster’s indexing and search capabilities  Index : An index is a collection of documents that have somewhat similar characteristics. (e.g Product, Customer, etc)  Type : Within an index, you can define one or more types. A type is a logical category/partition of your index.  Document : A document is a basic unit of information that can be indexed  Shard/Replica: Index divided into multiple pieces called shards, replicas are copy of your shards 9/14/2018 16
17. Elasticnodes  Master Node : which controls the cluster  Data Node : Data nodes hold data and perform data related operations such as CRUD, search, and aggregations.  Ingest Node : Ingest nodes are able to apply an ingest pipeline to a document in order to transform and enrich the document before indexing  Coordinating Node : only route requests, handle the search reduce phase, and distribute bulk indexing. 9/14/2018 17
18. 9/14/2018 18
19. ElasticsearchCLUSTER 9/14/2018 19
20. TYPICALCLUSTERSHARD&REPLICA 9/14/2018 20
21. Shardsearchandindex 9/14/2018 21
22. DemoofElasticSearch 9/14/2018 22
23. LOGSTASH  Ruby application runs under JRuby on the JVM  Collects, parse, enrich data  Horizontally scalable  Apache 2.0 License  Large amount of public plugins written by Community  https://github.com/logstash-plugins 9/14/2018 23
24. Typicalusageof Logstash 9/14/2018 24
25. 9/14/2018 25
26. Logstashinput 9/14/2018 26
27. Logstashfilter 9/14/2018 27
28. Logstashoutput 9/14/2018 28
29. DEMOLogstash 9/14/2018 29
30. Beats 9/14/2018 30
31. Beats  Lightweight shippers written in Golang (Non JVM shops can use them)  They follow unix philosophy; do one specific thing, and do it well  Filebeat : Logfile (think of it tail –f on steroids)  Metricbeat : CPU, Memory (like top), redis, mongodb usage  Packetbeat : Wireshark uses libpcap, monitoring packet http etc  Winlogbeat : Windows event logs to elastic  Dockbeat : Monitoring docker  Large community lots of other beats offered as opensource 9/14/2018 31
32. 9/14/2018 32
33. FILEBEAT 9/14/2018 33
34. X-Pack  Elastic commercial offering (This is one of the ways they make money)  X-Pack is an Elastic Stack extension that bundles  Security (https to elastic, password to access Kibana)  Alerting  Monitoring  Reporting  Graph capabilities  Machine Learning 9/14/2018 34
35. 9/14/2018 35
36. Kibana  Visual Application for Elastic Search (JS, Angular, D3)  Powerful frontend for dashboard for visualizing index information from elastic search  Historical data to form charts, graphs etc  Realtime search for index information 9/14/2018 36
37. 9/14/2018 37
38. DEMOKIBANA 9/14/2018 38
39. Designswewentthrough  We started with simple design to measure throughput  One instance of logstash and one instance of ElasticSearch with filebeat 9/14/2018 39
40. DotnetCoreapp  We used a dotnetcore application to generate logs  Serilog to generate into json format and stored on file  Filebeat was installed on the linux machine to ship the logs to logstash 9/14/2018 40
41. Performanceelastic  250 logs item per second for 30 minutes 9/14/2018 41
42. overview 9/14/2018 42
43. logstash 9/14/2018 43
44. Elasticsearchruntwo  1000 logs per second, run for 30 minutes 9/14/2018 44
45. performance 9/14/2018 45
46. Otherdesigns 9/14/2018 46
47. Otherdesignsusingredis 9/14/2018 47
48. Usingfilebeat 9/14/2018 48
49. Filebeatwithoutrelay 9/14/2018 49
50. Log4j 9/14/2018 50
51. Log4jdirect 9/14/2018 51
52. Whatwearegoingwithfornow,until….. 9/14/2018 52
53. Considerationsofdata  Index by day make sense in some cases  In other you may want to index by size rather (Black Friday more traffic than other days) when Shards are not balance ElasticSearch doesn’t like that  Don’t index everything, if you are not going to search on specific fields mark them as text 9/14/2018 53
54. FutureConsiderations  Investigate into Elastic Search Machine learning  ElasticSearch with Kafka for cross data center replication  Logstash Centralizex Pipeline for SEIM intergations 9/14/2018 54
55. Thankyou& Opento questions  – Questions???  – Contact: Taswar.bhatti@gemalto.com  – LinkedIn (find me and add me) 9/14/2018 55

Csharp_Pkcs11_Interop_Encryption_Decryption

I wanted to blog about using C# with pkcs11 on SafeNet ProtectServer HSM for your encryption need. The library I intent to use is the Pkcs11 Interop library on GitHub.
To being with we need to understand what an HSM is? In wikipedia we find this definition.

A hardware security module (HSM) is a physical computing device that safeguards and manages digital keys for strong authentication and provides cryptoprocessing. These modules traditionally come in the form of a plug-in card or an external device that attaches directly to a computer or network server.

So how does an HSM really look like? Its just like a Pizza box that you see for other servers like below the Gemalto ProtectServer HSM.

protectserverhsm

protectserverhsm

Getting Started

In order to use the Gemalto ProtectServer HSM you first need to download the Driver. Unfortunately it is not distributed publicly so you need to have an account to download the driver and software related to it. Trust me I work for Gemalto and had to create a customer account in order to download it.
Assuming that you have the driver and have installed you should have most of your software located at C:\Program Files\SafeNet\Protect Toolkit 5
In our application we will be using a Network HSM, in order to use an Network HSM you need to execute the command of SetMode.cmd

One will have to choose the Network Mode, remember to enter the ip address of the HSM in your registry key. ET_HSM_NETCLIENT_SERVERLIST, needs to have the ip or ip address of the machines you plan to connect.

One can then use KMU (a java program) or ctbrowse.exe (a native windows application) to view the HSM.

Tools

Here is the Java Program to use to create keys and manage HSM etc

kmu

kmu

Another application that one can use to create keys and manage the HSM.

ctbrowse

ctbrowse

Now lets use the tool to create the key we want, I will use the KMU tool for this and create an AES key labelled as demokey, note there are attributes for the key and I have generated it into a Slot that I plan to use. HSM are divided into Slots that one can use.

generatekey_aes

generatekey_aes

C# PKCS11 on SafeNet ProtectServer HSM

Now we can finally get into the code we will use the interop library one can install using nuget.

PM> Install-Package Pkcs11Interop

Summary

In the above example we have encrypted hello world with our demo key and also decrypted it using C# with the Interop library. The key never gets into your code since the HSM is the one which encrypts it.

codeaholics-hongkong

I had the opportunity to speak in my native home Hong Kong for the first time when I was visiting for holidays on Cloud Design Patterns. It was a lot of fun and thanks to Codeaholics for holding the meetup group in such great form in Hong Kong.

For people who wish to see my slides.

Transcript:

  1. Ponder • For every 25 percent increase in problem complexity, there is a 100 percent increase in solution complexity. • There is seldom one best design solution to a software problem. • If cars were like software, they would crash twice a day for no reason, and when you called for service, they’d tell you to reinstall the engine.
  2. Who am I • Taswar Bhatti – Microsoft MVP since 2014 • Global Solutions Architect/System Architect at Gemalto • In Software Industry since 2000 • Native Hong Kong, but live in Canada now • I know Kung Fu (Languages)
  3. What I am not
  4. Agenda • What are Patterns? • The External Configuration Pattern • The Cache Aside Pattern • The Federated Identity Pattern • The Valet Key Pattern • The Gatekeeper Pattern • The Circuit Breaker Pattern • The Retry Pattern • The Strangler Pattern • Questions
  5. Bad Design
  6. Bad Design
  7. Bad Design
  8. Bad Design?
  9. Itunes when I use it
  10. What are Patterns? • General reusable solution to a recurring problem • A template on how to solve a problem • Best practices • Patterns allow developers communicate with each other in well known and understand names for software interactions.
  11. External Configuration Pattern
  12. External Configuration Pattern • Helps move configuration information out of the application deployment • This pattern can provide for easier management and control of configuration data • For sharing configuration data across applications and other application instances
  13. Typical Application
  14. Storing Configuration in file
  15. Multiple application
  16. Problems • Configuration becomes part of deployment • Multiple applications share the same configuration • Hard to have access control over the configuration
  17. External Configuration Pattern
  18. When to use the pattern • When you have shared configuration, multiple application • You want to manage configuration centrally by DevOps • Provide audit for each configuration
  19. 20. When not to use • When you only have a single application there is no need to use this pattern it will make things more complex
  20. Cloud Solution Offerings • Azure Key Vault • Vault by Hashicorp • AWS KMS • Keywhiz
  21. Cache Aside Pattern
  22. Cache Aside Pattern • Load data on demand into a cache from datastore • Helps improve performance • Helps in maintain consistency between data held in the cache and data in the underlying data store.
  23. Typical Application
  24. Cache Aside Pattern
  25. When to use the pattern • Resource demand is unpredictable. • This pattern enables applications to load data on demand • It makes no assumptions about which data an application will require in advance
  26. When not to use • Don’t use it for data that changes very often
  27. Things to consider • Sometimes data can be changed from outside process • Have an expiry for the data in cache • When update of data, invalidate the cache before updating the data in database • Pre populate the data if possible
  28. Cloud Offerings • Redis (Azure and AWS) • Memcache • Hazelcast • Elastic Cache (AWS)
  29. Federated Identity Pattern
  30. Federated Identity Pattern • Delegate authentication to an external identity provider. • Simplify development, minimize the requirement for user administration • Improve the user experience of the application • Centralized providing MFA for user authentication
  31. Typical Application
  32. Problem
  33. Problem • Complex development and maintenance (Duplicated code) • MFA is not an easy thing • User administration is a pain with access control • Hard to keep system secure • No single sign on (SSO) everyone needs to login again to different systems
  34. Federated Identity Pattern
  35. When to use • When you have multiple applications and want to provide SSO for applications • Federated identity with multiple partners • Federated identity in SAAS application
  36. When not to use it • You already have a single application and have custom code that allows you to login
  37. Things to consider • The identity Server needs to be highly available • Single point of failure, must have HA • RBAC, identity server usually does not have authorization information • Claims and scope within the security auth token
  38. Cloud Offerings • Azure AD • Gemalto STA and SAS • Amazon IAM • GCP Cloud IAM
  39. Valet Key Pattern
  40. Valet Key Pattern • Use a token that provides clients with restricted direct access to a specific resource • Provide offload data transfer from the application • Minimize cost and maximize scalability and performance
  41. Typical Application Client App Storage
  42. Problem Client App Storage Client Client Client Client
  43. Valet Key Pattern Client App Generate Token Limited Time And Scope Storage
  44. When to use it • The application has limited resources • To minimize operational cost • Many interaction with external resources (upload, download) • When the data is stored in a remote data store or a different datacenter
  45. When not to use it • When you need to transform the data before upload or download
  46. Cloud Offerings • Azure Blob Storage • Amazon S3 • GCP Cloud Storage
  47. Gatekeeper Pattern
  48. Gatekeeper Pattern • Using a dedicated host instance that acts as a broker between clients and services • Protect applications and services • Validates and sanitizes requests, and passes requests and data between them • Provide an additional layer of security, and limit the attack surface of the system
  49. Typical Application
  50. Problem
  51. Gatekeeper Pattern
  52. When to use it • Sensitive information (Health care, Authentication) • Distributed System where perform request validation separately
  53. When not to use • Performance vs security
  54. Things to consider • WAF should not hold any keys or sensitive information • Use a secure communication channel • Auto scale • Endpoint IP address (when scaling application does the WAF know the new applications)
  55. Circuit Breaker Pattern
  56. Circuit Breaker Pattern • To handle faults that might take a variable amount of time to recover • When connecting to a remote service or resource
  57. Typical Application
  58. Problem
  59. Client Circuit Breaker Api Closed State Timeout Closed State Open State Half Open State After X Retry Closed State
  60. Circuit Breaker
  61. When to use it • To prevent an application from trying to invoke a remote service or access a shared resource if this operation is highly likely to fail • Better user experience
  62. When not to use • Handling access to local private resources in an application, such as in-memory data structure • Creates an overhead • Not a substitute for handling exceptions in the business logic of your applications
  63. Libraries • Polly (http://www.thepollyproject.org/) • Netflix (Hystrix) https://github.com/Netflix/Hystrix/wiki
  64. Retry pattern
  65. Retry Pattern • Enable an application to handle transient failures • When the applications tries to connect to a service or network resource • By transparently retrying a failed operation
  66. Typical Application Network Failure
  67. Retry Pattern • Retry after 2, 5 or 10 seconds
  68. When to use it • Use retry for only transient failure that is more than likely to resolve themselves quicky • Match the retry policies with the application • Otherwise use the circuit break pattern
  69. When not to use it • Don’t cause a chain reaction to all components • For internal exceptions caused by business logic • Log all retry attempts to the service
  70. Libraries • Roll your own code • Polly (http://www.thepollyproject.org/) • Netflix (Hystrix) https://github.com/Netflix/Hystrix/wiki
  71. Strangler Pattern
  72. Strangler Pattern • Incrementally migrate a legacy system • Gradually replacing specific pieces of functionality with new applications and services • Features from the legacy system are replaced by new system features eventually • Strangling the old system and allowing you to decommission it
  73. Monolith Application
  74. Strangler Pattern
  75. When to use • Gradually migrating a back-end application to a new architecture
  76. When not to use • When requests to the back-end system cannot be intercepted • For smaller systems where the complexity of wholesale replacement is low
  77. Considerations • Handle services and data stores that are potentially used by both new and legacy systems. • Make sure both can access these resources side-by-side • When migration is complete, the strangler façade will either go away or evolve into an adaptor for legacy clients • Make sure the façade doesn’t become a single point of failure or a performance bottleneck.
  78. Questions? Taswar Bhatti System Solutions Architect (Gemalto) Microsoft MVP http://taswar.zeytinsoft.com @taswarbhatti
csharp-dpapi-registry

This would be the second part of the blog post where we used powershell to store some secure data into our registry and have used DPAPI to encrypt the data. I wanted to cover how I would read the data back from the registry in my C# application. An example of C# using DPAPI to read data from Registry.
Feel free to read the blog post on Powershell using DPAPI to store secure data in Registry

I am using .NET Framework 4.6 rather than dotnet core, there is no DPAPI in dotnet core

C# using DPAPI to read data from Registry

I will be using a Console application with .NET Framework 4.6 just to show how I would read the registry and use System.Security.Cryptography to decrypt the data.

If we run this application we will see the data that we have stored in the registry.

Retry-Pattern-using-Polly-in-Csharp

One of the easiest cloud design pattern that one can try out is the Retry Pattern. I wanted to show how to use an Retry Pattern using Polly in C# as a example. So what does the Retry Pattern achieves?

Problem Statement – What is the issue the pattern solves?

When building applications you always have some sort of outside/external service including another MicroService that you have to consume or call. Sometimes there could be momentary loss of network connectivity, or a temporary unavailability, or timeouts that occur when that service is busy. You may be calling a database or a restful service that may be busy and fail but if you try back again it will pass. These types of faults are usually self-correcting, and most of the time require some type of delay in calling it again, which will have a success response.

Retry Pattern

  • Enable an application to handle transient failures
  • When the applications tries to connect to a service or network resource
  • By transparently retrying a failed operation
  • Improves the stability of your application
  • Typical Application

    Below is a typical application diagram, where you a service or web app.

    TypicalApplication

    TypicalApplication

    But when the connection to the service fails we usually get an error on our application.

    TypicalApplication-Network-Failure

    Typical-Application-Network-Failure

    When to use Retry Pattern

    • Use retry for only transient failure that is more than likely to resolve themselves quickly
    • Match the retry policies with the application
    • Otherwise use the circuit break pattern

    When not to use Retry Pattern

    • Don’t cause a chain reaction to all components
    • For internal exceptions caused by business logic
    • Log all retry attempts to the service

    Sample Code

    Below is a sample dotnet core Console Application that shows the usage using Polly. The code tries to call https://httpbin.org/status/200,408 with a POST which gives us a status of 200 or 408 randomly. First, lets create our code and add the package Polly into it.

    Without Polly

    We will write a sample application that will call the the web service without polly to get 408 errors.

    I am not using a logger but you should be using a logger when you do retry pattern.

    If you have issues with 7.0 C# change your csproj to include <LangVersion>7.1</LangVersion>

    After couple of runs you will see it response back with 408 RequestTimeout

    Using Polly

    Now we will introduce Polly into our code with an incremental delay of 1 second to 3 seconds and lastly 9 seconds.

    Output

    Below you will see three runs of the application with sample output.

    Summary

    As you can see Retry Pattern is quite useful for transient and self correcting failure, not to mention it is quite simple to implement in C# with the help of Polly. If you are looking for Java solutions you can look at Hysterix or even roll your own.

X509_certficiate_serial_with_Csharp_or_Python

In this blog post I wanted to show how one can use C# or Python to view the serial numbers of a X509 certificate. The serial number can be used to identify the certificate that one plans to use in their C# application, lets say for mutual authentication to another service.

Why use X509 Certificates

  1. Client X.509 certificate identity adds an additional level of asymmetrical cryptography to the standard SSL/TLS channel.
  2. Long security keys (2046 bits)
  3. One can revoke certificates by using Certificate Revocation List (CRL).
  4. Easy to use in application to application
Mutual Authentication

Mutual Authentication

View X509 Certificate Serial Number using C#

If you are using dotnetcore first create a console app using the command below

The command above will create a console app called certsharp and then you can use vscode to load the application.
Below is the sample code that allows one to see the serial number in C#, dotnetcore.

One can the simply run the command below to view the serial, and example show below.

Now lets say you need the serial to find the cert to communicate with another service you can use the code below to find the cert by serial number, and create a rest client to your api like below.

View X509 Certificate Serial Number using Python

For Python I usually create a virtual env before I start coding in python such that I have a separate environment for all my library needs.
To create the virtual env one will need virtualenv installed, you can look at my other Python example of how to install virtualenv and virtualenv-wrapper for windows.

To create a virtual env type the following into your command prompt inside of vscode.

The above command has create the virtual env but not enabled it yet, in order to enable we need to activate it.

Now we can install the package that we need for getting more certificate information, we will use pip and the package is pyopenssl.

Now we can code it up to view the required X509 certificate serial number.

When you run the script it will output something like this

Summary

Above are two examples of getting x509 certificate information specifically serial number, there are additional information that one can view, just check out the api documentation.
One additional thing I found is in python it does not show me the leading 00 of a certificate but C# is able to show that.

Powershell-dpapi

I wanted to store some data into registry but how do I store it such that it is secure for my application to read later on. Well there is a way using Powershell to store data into Registry using DPAPI which uses the machine key to store the information. As you know the only way to store data securely is by using encryption and DPAPI provides us with that.

DPAPI only works in windows, if you are on linux environment there is no DPAPI

What is DPAPI?

From Wikipedia we get this definition

DPAPI (Data Protection Application Programming Interface) is a simple cryptographic application programming interface available as a built-in component in Windows 2000 and later versions of Microsoft Windows operating systems. In theory the Data Protection API can enable symmetric encryption of any kind of data; in practice, its primary use in the Windows operating system is to perform symmetric encryption of asymmetric private keys, using a user or system secret as a significant contribution of entropy.

Now that we know it uses the machine key we can write some code to store some value into our registry using powershell. We will be writing into the Wow64 registry since we will most likely use a 64bit application later on. If you need to write to 32bit also just remove the wow64node.

Powershell using DPAPI to store secure data in Registry

What we will do is use SecureString to read the data and convert the secure string into a hash, then we will base64 encode it and store it to the registry, one could just leave it as it and not base64 it. We use the New-ItemProperty to create the value into the registry.

powershell-execute-command

powershell-execute-command

So here it is how to store data into your registry using DPAPI. In my next post I will show how to read back the value using C#.

nodejs-vscode-docker-vault-logo

To continue on with our previous blog post, I will introduce Hashicorp Vault as a key management to manage our secrets for our Nodejs weather application.

Installing Vault

I will use docker to pull the docker image from dockerhub.

docker-install-vault

docker-install-vault

One can also download Vault for their OS at (https://www.vaultproject.io/downloads.html)

Now that I have vault image pulled, I will create a docker compose file for Vault to use mysql as a back-end store. I can also run Vault in dev mode but if I enable dev mode then Vault runs entirely in-memory and starts unsealed with a single unseal key. I wanted to show more of a real life scenario of starting Vault.

First thing I will create couple of directory and files that I will store some configuration into.

Now that we have a directory we can create a docker-compose.yml file inside of the myvault directory. I assume you already have mysql image, if not you call pull the image from dockerhub.

Before we fire up vault, here is the content of the config.hcl file, located right in the config folder we just created. This will configure Vault with the storage options and listen on port 8200

Running Vault

Use the docker compose file and run the following command to bring up Vault

If you mess up you can always run $docker-compose rm to remove the created containers.

Testing our installation

Now that we have Vault configured and running, we now need to initialize Vault. Vault uses Shamir Secret Sharing Technique for initializing the key to use for Vault. Which takes multiple keys and combines into one single master key.

shamir-secret-sharing-vault-unseal

shamir-secret-sharing-vault-unseal

Unseal Vault

We will use the operator init method to call Vault to initialize.

As we can see it showed us some of the keys we need to use to unseal the Vault. So let unseal it so that we can use it. We will need to unseal the vault with the command 3 times using different keys each time for it, I have skipped the first one.

Now we can use the Vault to write data to, first we need to auth with root token which was given to us when we started the Vault with the keys. My test envirnoment key was “5mEKu64nAk1PA5luVQHRmGLM”. Lets try to auth/login and also write and read something to Vault to store.

We can now also create a sample.json config file that a new Microservice can consume I have placed it in config folder since it was already mounted (note the file is in 1 line)

We can load the sample json data into Vault by using this command.

I am using the weatherapp as a example above

We will most probably also create a policy file so that we can limit the access to this secret, this will be inside the policies directory

In order to load the policy into Vault we will use the command to load the policy in

Now policy is in, lets test out Vault to read some of they values we just added.

Above we have just used the root token to read but at least we know we can get the data

Wrap Token

Vault has a nice feature called wrap token, where you can give a limited amount of time to a token which one can use to access Vault. Lets try to create one for us.

We can use the wrap token above “lYO2AoJ95QEDnZgUbNxoWWsw ” to read the data now, and it is only valid for 60 seconds.

If we try to use another token of if the token is already used we will get an error like below.

App Roles and Secrets

Now that we have learned something about Wrap tokens of how to create and use them, I wanted to switch gear and talk about App Roles and Secrets. Vault provides app roles for you application to login to the system. This is definitely not the best security option out there since its just a like a basic authentication with username and password, but when we use wrap token with it we can mitigate some of the security concerns.

There are better options out there using kubernetes to authenticate for you app etc I will try to cover those later in another blog post

Lets create us an approle for our weather app, and get the roleid that we can use for our application.

Sample Application

Now that we have the roleid we now need a secret in order for our application to login to Vault and get its secrets. Rather than just giving the secret to our application we will give it a wrap token to get the secret since we can time limit the amount of time the token is allowed to live, thus mitigating the risk of exposing the secret. The sample code below is using the env variable and the wrap token to get its secret.

I am using node-vault, you can install node-vault by > npm install node-vault

In the above code we can see that we have hard coded the roleid into our code, but we mitigated the risk of someone stealing our docker env variable by having a limited time to live for the wrap token to authenticate and get our secrets from Vault. The command to generate the wrap token is below.

When we run the code we would do something like, with the third parameter being the wrap token.

If we try to reuse the wrap token we will get an error.

Summary

We have covered using roleid and secret using Vault to authenticate and get our application secrets. There is definitely a down side to this, but we mitigated the risk by having a short time for the wrap token to live. There is also the option of having the wrap token mounted as a volume for your application and pick it up from the volume that was mounted. I will cover those topics in a later blog post.

    Advantages

  • Even with docker inspect we are only are able to see the token which is last with a TTL and is a guid token the attack surface would be lower.
  • A very simple pattern to follow, all config are stored with application Name (e.g secret/appName/config, etc)
    Disadvantages

  • The microservice would require some Restful call to Vault to consume data, may have dependency on vault library to consume it
  • DevOps or some scripts would still be required to put secrets in the correct place
  • Failure of Vault what happens? Service does not start

Source code at
https://github.com/taswar/weatherappwithvault

Yesterday I gave a talk on using ForwardJS conference here in Ottawa on Hashicorp Vault – Using Vault for your Nodejs Secrets.

You can find the slides here on slideshare.

Transcript:

  1. Using vault for your NodeJS Secrets Taswar Bhatti – Solutions Architect Gemalto
  2. Secrets •
  3. About me • Taswar Bhatti (Microsoft MVP) • @taswarbhatti • http://taswar.zeytinsoft.com • Gemalto (System Architect)
  4. So what are secrets? • Secrets grants you AuthN or AuthZ to a system • Examples • Username & Passwords • Database credentials • API Token • TLS Certs
  5. Secret Sprawl • Secrets ends up in • Source Code • Version Control Systems (Github, Gitlab, Bitbucket etc) • Configuration Management (Chef, Puppet, Ansible etc)
  6. Issues • How do we know who has access to those secrets • When was the last time they accessed it? • What if we want to change/rotate the secrets
  7. Desire secrets • Encryption in rest and transit • Only decrypted in memory • Access control • Rotation & Revocation
  8. Secret Management – Vault • Centralized Secret Management • Encrypted at rest and transit • Lease and Renewal • ACL • Audit Trail • Multiple Client Auth Method (Ldap,Github, approle) • Dynamic Secrets • Encryption as a Service
  9. Dynamic Secrets • Allows one to lease a secret for a period of time e.g 2 hrs • Generates on demand and unique for each user/consumption • Audit trail
  10. Secure Secrets • AES 256 with GCM encryption • TLS 1.2 for clients • No HSM is required
  11. Unsealing the Vault • Vault requires encryption keys to encrypt data • Shamir Secret Key Sharing • Master key is split into multiple keys
  12. Shamir Secret Sharing
  13. Unseal • Unseal Key 1: QZdnKsOyGXaWoB2viLBBWLlIpU+tQrQy49D+Mq24/V0B • Unseal Key 2: 1pxViFucRZDJ+kpXAeefepdmLwU6QpsFZwseOIPqaPAC • Unseal Key 3: bw+yIvxrXR5k8VoLqS5NGW4bjuZym2usm/PvCAaMh8UD • Unseal Key 4: o40xl6lcQo8+DgTQ0QJxkw0BgS5n6XHNtWOgBbt7LKYE • Unseal Key 5: Gh7WPQ6rWgGTBRSMecuj8PR8IM0vMIFkSZtRNT4dw5MF • Initial Root Token: 5b781ff4-eee8-d6a1-ea42-88428a7e8815 • Vault initialized with 5 keys and a key threshold of 3. Please • securely distribute the above keys. When the Vault is re-sealed, • restarted, or stopped, you must provide at least 3 of these keys • to unseal it again. • Vault does not store the master key. Without at least 3 keys, • your Vault will remain permanently sealed.
  14. How to unseal • vault unseal -address=${VAULT_ADDR} QZdnKsOyGXaWoB2viLBBWLlIpU+tQrQy49D+Mq24/V0B • vault unseal -address=${VAULT_ADDR} bw+yIvxrXR5k8VoLqS5NGW4bjuZym2usm/PvCAaMh8UD • vault unseal -address=${VAULT_ADDR} Gh7WPQ6rWgGTBRSMecuj8PR8IM0vMIFkSZtRNT4dw5MF
  15. Writing Secrets • vault write -address=${VAULT_ADDR} secret/hello value=world • vault read -address=${VAULT_ADDR} secret/hello • Key Value • — —– • refresh_interval 768h0m0s • Value world
  16. Policy on secrets • We can assign application roles to the policy path “secret/web/*” { policy = “read” } • vault policy write -address=${VAULT_ADDR} web-policy ${DIR}/web-policy.hcl
  17. Reading secrets based on policy • vault read -address=${VAULT_ADDR} secret/web/web-apps • vault read -address=${VAULT_ADDR} secret/hello • Error reading secret/hello: Error making API request. • URL: GET http://127.0.0.1:8200/v1/secret/hello • Code: 403. Errors: • * permission denied
  18. Demo Using Vault
  19. Demo Docker Environment VAR • Issues with env variables
  20. Mount Temp File System into App • docker run –v /hostsecerts:/secerts …. • To mitigate reading from Env • Store your wrap token in the filesystem to use with vault • Have limit time on wrap token
  21. Wrap Token for App Secrets • Limit time token • Used to unwrap some secrets • vault read -wrap-ttl=60s -address=http://127.0.0.1:8200 secret/weatherapp/config • Key Value • — —– • wrapping_token: 35093b2a-60d4-224d-5f16-b802c82de1e7 • wrapping_token_ttl: 1m0s • wrapping_token_creation_time: 2017-09-06 09:29:03.4892595 +0000 UTC • wrapping_token_creation_path: secret/weatherapp/config
  22. App Roles • Allows machines or apps to authenticate with Vault • Using a role_id and secret_id as credentials • Assign polices to the app • Once logged in you get back a token to get secrets
  23. Demo App Using Node
  24. Kubernetes with Vault • Read Service Account JWT • App Sends Jwt and Role Name to Vault • Vault checks the signature of Jwt • Sends to TokenReviewer API • Vault sends back valid token for app
  25. Thankyou • Contact me (taswar.bhatti@gemalto.com) • @taswarbhatti
UA-4524639-2