Taswar Bhatti
The synonyms of software simplicity
8 Cloud Design Patterns Update Conference

Here is the youtube recording for my 8 Cloud Design Pattern you ought to know talk at update conference in Prague Nov 2018. Hope you like it.

Javascript

Thanks to the organizers of ForwardJS Ottawa to let me speak on Cloud Design Patterns using Nodejs, I had a great time at the conference for anyone who is looking for my sides there are located right below.
If anyone wants to view the demo code I showed please look at this blog post on using Promise in Nodejs for Retry Pattern.

retry-pattern-nodejs-with-promise

One of the easiest cloud design pattern that one can try out is the Retry Pattern. I wanted to show how to use an Retry Pattern in Node.js using Promise as a example. So what does the Retry Pattern achieves?

Problem Statement – What is the issue the pattern solves?

When building applications you always have some sort of outside/external service including another MicroService that you have to consume or call. Sometimes there could be momentary loss of network connectivity, or a temporary unavailability, or timeouts that occur when that service is busy. You may be calling a database or a restful service that may be busy and fail but if you try back again it will pass. These types of faults are usually self-correcting, and most of the time require some type of delay in calling it again, which will have a success response.

Retry Pattern

  • Enable an application to handle transient failures
  • When the applications tries to connect to a service or network resource
  • By transparently retrying a failed operation
  • Improves the stability of your application
  • Typical Application

    Below is a typical application diagram, where you a service or web app.

    TypicalApplication

    TypicalApplication

    But when the connection to the service fails we usually get an error on our application.

    TypicalApplication-Network-Failure

    Typical-Application-Network-Failure

    When to use Retry Pattern

    • Use retry for only transient failure that is more than likely to resolve themselves quickly
    • Match the retry policies with the application
    • Otherwise use the circuit break pattern

    When not to use Retry Pattern

    • Don’t cause a chain reaction to all components
    • For internal exceptions caused by business logic
    • Log all retry attempts to the service

    Sample Code

    Below is a sample in node.js that shows the usage using Promise in Node.js. The code tries to call https://httpbin.org/status/200,408 with a POST which gives us a status of 200 or 408 randomly. First, lets create our code and add the package fetch into it.

    Without Promise

    We will write a sample application that will call the the web service without retry to get 408 errors.

    I am just using a console logger but you should be using a proper logger when you do retry pattern.

    After couple of runs you will see it response back with 408 RequestTimeout

    Using Retry with Promise

    Now we will introduce the retry pattern with using Promise into our code with an incremental delay of 1 second to 3 seconds and lastly 9 seconds.

    Output

    Below you will see three runs of the application with sample output.

    Summary

    As you can see Retry Pattern is quite useful for transient and self correcting failure, not to mention it is quite simple to implement in NodeJS with the help of Promise.

update_conference_taswar_bhatti

Here is the youtube version of my presentation that I did in Prague at Update Conference. You can now watch my entire presentation. Thanks to Update Conference for doing such a fantastic job.

Enjoy.

DevTeach Mntreal Speaker

I will be speaking at forwardjs Ottawa on April 10th and 11th 2019. I will be doing 8 Cloud Design Pattern you ought to know but more specific to using Node.js. There should be a video for it months later that I hope to share.

If you wish to watch my last year talk on Using Vault for your Nodejs Secrets

To purchase tickets for the session check out https://ti.to/forwardjs/forwardjs-ottawa-2019

Microsoft Ignite Tour - Taswar Bhatti

On January 11 2019, I had the opportunity to speak at Toronto Microsoft Ignite Tour – 8 Cloud Design Patterns you ought to know. It was a great seeing that over 700+ people registered to my talk, it was pretty much full house session over 500+ attended. Unfortunately Microsoft did not record any session in Toronto. I have had people ask me of a recording, although I do remember a gentleman who had his camera and was recording all my slides. (If someone knows him please feel free to ask him to get in touch with me, I would also love a copy of it also).

Attendees Review

One benefit of speaking at Ignite is they have a good evaluation system where users can give feedback.
So far the feedback has been very good. Here are the result of the evaluation score.

Ignite - Taswar Bhatti Review

Ignite – Taswar Bhatti Review

Attendee Comments

Some comments that I received are below. I actually wanted to speak slower but knowing the fact that I have 8 topics to go through and 90 slides in total with 2 demos, would rather like to cover all topics than missing any. In fact I covered 9 patterns with a bonus pattern,

Ignite - Taswar Bhatti Review Comments

Ignite – Taswar Bhatti Review Comments

Attendee Suggested Improvements

Its sometimes hard to satisfy all people in a talk, with over 500 people there will be someone who doesn’t like your talk. You cannot make everyone happy but I think I did make it clear sometimes patterns are like “Duh” moments where it is just common sense. Overall I still think people learned something new and enjoyed the talk. Below are some of the improvements.

Ignite - Taswar Bhatti Review Improvement

Ignite – Taswar Bhatti Review Improvement

Slides 8 Cloud Design Patterns you ought to know

Last but not least here are the slides from the talk. Enjoy…..

update_conference_managing_cloud_secrets

In November I did a presentation at the Update Conference 2018 on Managing your secrets in a cloud environment using Azure Key Vault and Hashicorp Vault. Comparing both products and demo on using Key Vault to store your keys and secrets. It was excellent at Prague and once again thanks Update Conference inviting me to speak there.

Transcript

1. Managing your Secrets in a Cloud Environment Taswar Bhatti System/Solutions Architect at Gemalto (Canada) Microsoft MVP
2. Is your personal data important?
3. Who am I • Taswar Bhatti – Microsoft MVP since 2014 • Global Solutions Architect/System Architect at Gemalto • In Software Industry since 2000 • I know Kung Fu (Languages)
4. Good old days robbery
5. Today’s Robbery
6. Data breach……
7. Consequences
8. System with no Trust
9. Salesman
10. Data breach??
11. Delivery
12. Agenda • Intro • What are we trying to solve with KeyVault? • What is Azure Key Vault • Using Azure Key Vault with your application • Managed Service Identity • Demo • HashiCorp Vault • Best practices • Questions
13. So what are secrets? • Secrets grants you AuthN or AuthZ to a system • Examples • Username & Passwords • Database credentials • API Token • TLS Certs
14. Typical Application
15. Storing Configuration in file
16. Multiple application
17. Secret Sprawl • Secrets ends up in • Source Code • Version Control Systems (Github, Gitlab, Bitbucket etc) • Configuration Management (Chef, Puppet, Ansible etc)
18. Problems • Configuration becomes part of deployment • Multiple applications share the same configuration • Hard to have access control over the configuration
19. Issues • How do we know who has access to those secrets • When was the last time they accessed it? • What if we want to change/rotate the secrets
20. Desire secrets • Encryption in rest and transit • Only decrypted in memory • Access control • Rotation & Revocation
21. What is Azure Key Vault? • Secrets Management – Azure Key Vault can be used to Securely store and tightly control access to tokens, passwords, certificates, API keys, and other secrets. • Key Management – Azure Key Vault can also be used as a Key Management solution. Azure Key Vault makes it easy to create and control the encryption keys used to encrypt your data. • Certificate Management – Azure Key Vault is also a service that lets you easily provision, manage, and deploy public and private Secure Sockets Layer/Transport Layer Security (SSL/TLS) certificates for use with Azure and your internal connected resources. • Store secrets backed by Hardware Security Modules – The secrets and keys can be protected either by software or FIPS 140-2 Level 2 validates HSMs.
22. Gemalto Luna HSM (New)
23. PKCS11 Interop • Managed .NET wrapper for unmanaged PKCS#11 libraries • https://pkcs11interop.net/
24. Typical Application • In web.config
25. With Key Vault
26. Azure Key Vault • Register your app with Active Directory • Associated credential, and using that credential to get a token • Retrieve your secrets from Key Vault • PROBLEM SOLVED
27. Adding it back to web.config •
28. Code that looks like this ClientCredential clientCred = new ClientCredential( WebConfigurationManager.AppSettings[“ClientId”], WebConfigurationManager.AppSettings[“ClientSecret”]);
29. But???? • Confused?? • Isn’t that still in web.config?
30. Security doesn’t have to be like this
31. Managed Service Identity (MSI) • MSI gives your code an automatically managed identity for authenticating to Azure services, so that you can keep credentials out of your code • You create an identity for your application in Azure Active Directory using Managed Service Identity
32. Benefits • No need to authenticate to Azure Key Vault to get secrets • No client id and client secret is needed in the code • Easier to configure comparing to Azure Key Vault • You can authenticate to any service that supports Azure AD authentication
33. Demo
34. HSBC Hong Kong PayMe Hack
35. HashiCorp Vault • Centralized Secret Management • Encrypted at rest and transit • Lease and Renewal • ACL • Audit Trail • Multiple Client Auth Method (Ldap,Github, approle) • Dynamic Secrets • Encryption as a Service
36. Secure Secrets • AES 256 with GCM encryption • TLS 1.2 for clients • No HSM is required • One could also integrate with Azure Key Vault
37. Unsealing the Vault • Vault requires encryption keys to encrypt data • Shamir Secret Key Sharing • Master key is split into multiple keys
38. Shamir Secret Sharing
39. Unseal • Unseal Key 1: QZdnKsOyGXaWoB2viLBBWLlIpU+tQrQy49D+Mq24/V0B • Unseal Key 2: 1pxViFucRZDJ+kpXAeefepdmLwU6QpsFZwseOIPqaPAC • Unseal Key 3: bw+yIvxrXR5k8VoLqS5NGW4bjuZym2usm/PvCAaMh8UD • Unseal Key 4: o40xl6lcQo8+DgTQ0QJxkw0BgS5n6XHNtWOgBbt7LKYE • Unseal Key 5: Gh7WPQ6rWgGTBRSMecuj8PR8IM0vMIFkSZtRNT4dw5MF • Initial Root Token: 5b781ff4-eee8-d6a1-ea42-88428a7e8815 • Vault initialized with 5 keys and a key threshold of 3. Please • securely distribute the above keys. When the Vault is re-sealed, • restarted, or stopped, you must provide at least 3 of these keys • to unseal it again. • Vault does not store the master key. Without at least 3 keys, • your Vault will remain permanently sealed.
40. How to unseal • vault unseal -address=${VAULT_ADDR} QZdnKsOyGXaWoB2viLBBWLlIpU+tQrQy49D+Mq24/V0B • vault unseal -address=${VAULT_ADDR} bw+yIvxrXR5k8VoLqS5NGW4bjuZym2usm/PvCAaMh8UD • vault unseal -address=${VAULT_ADDR} Gh7WPQ6rWgGTBRSMecuj8PR8IM0vMIFkSZtRNT4dw5MF
41. Writing Secrets • vault write -address=${VAULT_ADDR} secret/hello value=world • vault read -address=${VAULT_ADDR} secret/hello • Key Value • — —– • refresh_interval 768h0m0s • Value world
42. Policy on secrets • We can assign application roles to the policy path “secret/web/*” { policy = “read” } • vault policy write -address=${VAULT_ADDR} web-policy ${DIR}/web-policy.hcl
43. Reading secrets based on policy • vault read -address=${VAULT_ADDR} secret/web/web-apps • vault read -address=${VAULT_ADDR} secret/hello • Error reading secret/hello: Error making API request. • URL: GET http://127.0.0.1:8200/v1/secret/hello • Code: 403. Errors: • * permission denied
44. Docker and Secrets • Docker does not have good integration with secrets • If you use env variables, it will show in docker inspect
45. Mount Temp File System into App • docker run –v /hostsecerts:/secerts …. • To mitigate reading from Env • Store your wrap token in the filesystem to use with vault • Have limit time on wrap token
46. Wrap Token for App Secrets • Limit time token • Used to unwrap some secrets • vault read -wrap-ttl=60s -address=http://127.0.0.1:8200 secret/weatherapp/config • Key Value • — —– • wrapping_token: 35093b2a-60d4-224d-5f16-b802c82de1e7 • wrapping_token_ttl: 1m0s • wrapping_token_creation_time: 2017-09-06 09:29:03.4892595 +0000 UTC • wrapping_token_creation_path: secret/weatherapp/config
47. Kubernetes with Vault • Read Service Account JWT • App Sends Jwt and Role Name to Vault • Vault checks the signature of Jwt • Sends to TokenReviewer API • Vault sends back valid token for app
48. Token Reviewer in K8s
49. Best Practices or Patterns • Cache Aside Encryption Key • Tag version of encryption
50. Cache Aside Encryption Key • Use Key Vault to Encrypt your Generated AES Key • For all encryption of your data you can use the AES Key rather than going back and Key Vault to encrypt • Allows you to penny pinch KeyVault
51. Tag Version of Encryption Level • Each Row of your database is tagged with the encryption version • This allows you when you rotate keys or change encryption level for example moving to a new Encryption Key to eventual encryption of data that gets updated or new.
52. New and Updated Data
53. Advantages • You do not have to go through all the records to re-encrypt them • Eventual Encryption of all data to new encryption • Mitigates the risk of all data or updating all records
54. Questions? • taswar@gmail.com • @taswarbhatti • http://taswar.zeytinsoft.com
55. Credits • For the background • www.Vecteezy.com

In November I went to Prague my first time and it was amazing being hosted by Update Conference there.
Below you will find my presentation on 8 Cloud Design Patterns that I did for Update Conference In Prague, I wanted to give a special thanks for @tomasherceg and his team for creating such an awesome conference.
Enjoy the sides!.

Transcript

1. 8 Cloud Design Patterns you ought to know Taswar Bhatti System/Solutions Architect at Gemalto (Canada) Microsoft MVP
2. Ponder • For every 25 percent increase in problem complexity, there is a 100 percent increase in solution complexity. • There is seldom one best design solution to a software problem. • If cars were like software, they would crash twice a day for no reason, and when you called for service, they’d tell you to reinstall the engine.
3. Who am I • Taswar Bhatti – Microsoft MVP since 2014 • Global Solutions Architect/System Architect at Gemalto • In Software Industry since 2000 • I know Kung Fu (Languages)
4. What I am not
5. Agenda • What are Patterns? • The External Configuration Pattern • The Cache Aside Pattern • The Federated Identity Pattern • The Valet Key Pattern • The Gatekeeper Pattern • The Circuit Breaker Pattern • The Retry Pattern • The Strangler Pattern • Demo • Questions
6. Bad Design
7. Bad Design
8. Bad Design
9. Bad Design
10. Bad Design
11. Feature doesn’t make sense????
12. Anger
13. Bad day at work
14. Happy family
15. Next day at work
16. Ship it
17. Customer Feature didn’t make sense
18. Bad Design?
19. Itunes when I use it
20. What are Patterns? • General reusable solution to a recurring problem • A template on how to solve a problem • Best practices • Patterns allow developers communicate with each other in well known and understand names for software interactions.
21. External Configuration Pattern
22. External Configuration Pattern • Helps move configuration information out of the application deployment • This pattern can provide for easier management and control of configuration data • For sharing configuration data across applications and other application instances
23. Typical Application
24. Storing Configuration in file
25. Multiple application
26. Problems • Configuration becomes part of deployment • Multiple applications share the same configuration • Hard to have access control over the configuration
27. External Configuration Pattern
28. When to use the pattern • When you have shared configuration, multiple application • You want to manage configuration centrally by DevOps • Provide audit for each configuration
29. When not to use • When you only have a single application there is no need to use this pattern it will make things more complex
30. Cloud Solution Offerings • Azure Key Vault • Vault by Hashicorp • AWS KMS • Keywhiz
31. Demo KeyVault
32. Cache Aside Pattern
33. Cache Aside Pattern • Load data on demand into a cache from datastore • Helps improve performance • Helps in maintain consistency between data held in the cache and data in the underlying data store.
34. Typical Application
35. Cache Aside Pattern
36. When to use the pattern • Resource demand is unpredictable. • This pattern enables applications to load data on demand • It makes no assumptions about which data an application will require in advance
37. When not to use • Don’t use it for data that changes very often
38. Things to consider • Sometimes data can be changed from outside process • Have an expiry for the data in cache • When update of data, invalidate the cache before updating the data in database • Pre populate the data if possible
39. Cloud Offerings • Redis (Azure and AWS) • Memcache • Hazelcast • Elastic Cache (AWS)
40. Federated Identity Pattern
41. Federated Identity Pattern • Delegate authentication to an external identity provider. • Simplify development, minimize the requirement for user administration • Improve the user experience of the application • Centralized providing MFA for user authentication
42. Typical Application
43. Problem
44. Problem • Complex development and maintenance (Duplicated code) • MFA is not an easy thing • User administration is a pain with access control • Hard to keep system secure • No single sign on (SSO) everyone needs to login again to different systems
45. Federated Identity Pattern
46. When to use • When you have multiple applications and want to provide SSO for applications • Federated identity with multiple partners • Federated identity in SAAS application
47. When not to use it • You already have a single application and have custom code that allows you to login
48. Things to consider • The identity Server needs to be highly available • Single point of failure, must have HA • RBAC, identity server usually does not have authorization information • Claims and scope within the security auth token
49. Cloud Offerings • Azure AD • Gemalto STA and SAS • Amazon IAM • GCP Cloud IAM
50. Valet Key Pattern
51. Valet Key Pattern • Use a token that provides clients with restricted direct access to a specific resource • Provide offload data transfer from the application • Minimize cost and maximize scalability and performance
52. Typical Application Client App Storage
53. Problem Client App Storage Client Client Client Client
54. Valet Key Pattern Client App Generate Token Limited Time And Scope Storage
55. When to use it • The application has limited resources • To minimize operational cost • Many interaction with external resources (upload, download) • When the data is stored in a remote data store or a different datacenter
56. When not to use it • When you need to transform the data before upload or download
57. Cloud Offerings • Azure Blob Storage • Amazon S3 • GCP Cloud Storage
58. Gatekeeper Pattern
59. Gatekeeper Pattern • Using a dedicated host instance that acts as a broker between clients and services • Protect applications and services • Validates and sanitizes requests, and passes requests and data between them • Provide an additional layer of security, and limit the attack surface of the system
60. Typical Application
61. Problem
62. Gatekeeper Pattern
63. When to use it • Sensitive information (Health care, Authentication) • Distributed System where perform request validation separately
64. When not to use • Performance vs security
65. Things to consider • WAF should not hold any keys or sensitive information • Use a secure communication channel • Auto scale • Endpoint IP address (when scaling application does the WAF know the new applications)
66. Circuit Breaker Pattern
67. Circuit Breaker Pattern • To handle faults that might take a variable amount of time to recover • When connecting to a remote service or resource
68. Typical Application
69. Problem
70. Client Circuit Breaker Api Closed State Timeout Closed State Open State Half Open State After X Retry Closed State
71. Circuit Breaker
72. When to use it • To prevent an application from trying to invoke a remote service or access a shared resource if this operation is highly likely to fail • Better user experience
73. When not to use • Handling access to local private resources in an application, such as in-memory data structure • Creates an overhead • Not a substitute for handling exceptions in the business logic of your applications
74. Libraries • Polly (http://www.thepollyproject.org/) • Netflix (Hystrix) https://github.com/Netflix/Hystrix/wiki
75. Retry pattern
76. Retry Pattern • Enable an application to handle transient failures • When the applications tries to connect to a service or network resource • By transparently retrying a failed operation
77. Typical Application Network Failure
78. Retry Pattern • Retry after 2, 5 or 10 seconds
79. When to use it • Use retry for only transient failure that is more than likely to resolve themselves quickly • Match the retry policies with the application • Otherwise use the circuit break pattern
80. When not to use it • Don’t cause a chain reaction to all components • For internal exceptions caused by business logic • Log all retry attempts to the service
81. Libraries • Roll your own code • Polly (http://www.thepollyproject.org/) • Netflix (Hystrix) https://github.com/Netflix/Hystrix/wiki
82. Demo Retry
83. Strangler Pattern
84. Strangler Pattern • Incrementally migrate a legacy system • Gradually replacing specific pieces of functionality with new applications and services • Features from the legacy system are replaced by new system features eventually • Strangling the old system and allowing you to decommission it
85. Monolith Application
86. Strangler Pattern
87. When to use • Gradually migrating a back-end application to a new architecture
88. When not to use • When requests to the back-end system cannot be intercepted • For smaller systems where the complexity of wholesale replacement is low
89. Considerations • Handle services and data stores that are potentially used by both new and legacy systems. • Make sure both can access these resources side-by-side • When migration is complete, the strangler façade will either go away or evolve into an adaptor for legacy clients • Make sure the façade doesn’t become a single point of failure or a performance bottleneck.
90. Questions? Taswar Bhatti System Solutions Architect (Gemalto) Microsoft MVP http://taswar.zeytinsoft.com @taswarbhatti
91. Credits • For the background • www.Vecteezy.com

elastic search introduction

In September I was invited to speak at the Ottawa Elastic Search Meetup Group at Kinaxis. It was great seeing new faces at the meetup, I spoke on Elastic Search Introduction so that new user group members can go through the entire stack of ElasticSearch, logstash and kibana. Hope to speak there some other time soon.

Here are my slides for anyone interested.

Transcript

1. A Gentle Intro to ElasticSearch Taswar Bhatti System/Solutions Architect (Ottawa) GEMALTO
2. Who amI?  System/Solution Architect at Gemalto Ottawa (Microsoft MVP)  I am somewhat of a language geek; I speak a few languages  Kind of like Neo (I KNOW KUNG FU) for languages 2 – Merhaba – नमस्ते – 你好 – ‫ہیلو‬ – Comment ca va? – ਸਤ ਸਰੀ ਅਕਾਲ
3. 9/14/2018 3 Reuters Top 100: Gemalto rated top Global Tech Leaders https://www.thomsonreuters.com/en/products-services/technology/top-100.html
4. Agenda  Problem we had and wanted to solve with Elastic Stack  Intro to Elastic Stack (Ecosystem)  Logstash  Kibana  Beats  Elastic Search flows designs that we have considered  Future plans of using Elastic Search 4
5. How doyouTroubleshootorfindyourbugs?  Typically in a distributed environment one has to go through the logs to find out where the issue is  Could be multiple systems that you have to go through which machine/server generated the log or monitoring multiple logs  Even monitor firewall logs to find traffic routing through which data center  Chuck Norris never troubleshoot; the trouble kills themselves when they see him coming 9/14/2018 5
6. 9/14/2018 6
7. OurProblem  We had distributed systems (microservices) that would generate many different types of logs, in different data centers  We also had authentication audit logs that had to be secure and stored for 1 year  We generate around 2 millions records of audit logs a day, 4TB with replications  We need to generate reports out of our data for customers  We were still using Monolith Solution in some core parts of the application  Growing pains of a successful application  We want to use a centralized scalable logging system for all our logs 9/14/2018 7
8. Findingbugsthroughlogs 9/14/2018 8
9. Alittlehistoryof ElasticSearch  Shay Banon created Compass in 2004  Released Elastic Search 1.0 in 2010  ElasticSearch the company was formed in 2012  Shay wife is still waiting for her receipe app 9/14/2018 9
10. 9/14/2018 10
11. ElasticStack 9/14/2018 11
12. ElasticSearch  Written in Java backed by Lucene  Schema free, REST & JSON based document store  Search Engine  Distributed, Horizontally Scalable  No database storage, storage is Lucene  Apache 2.0 License 9/14/2018 12
13. CompaniesusingElasticStack 9/14/2018 13
14. ElasticSearchindices  Elastic organizes document in indices  Lucene writes and maintains the index files  ElasticSearch writes and maintains metadata on top of Lucene  Example: field mappings, index settings and other cluster metadata 9/14/2018 14
15. Databasevs ElasticSearch 9/14/2018 15
16. ElasticConcepts  Cluster : A cluster is a collection of one or more nodes (servers)  Node : A node is a single server that is part of your cluster, stores your data, and participates in the cluster’s indexing and search capabilities  Index : An index is a collection of documents that have somewhat similar characteristics. (e.g Product, Customer, etc)  Type : Within an index, you can define one or more types. A type is a logical category/partition of your index.  Document : A document is a basic unit of information that can be indexed  Shard/Replica: Index divided into multiple pieces called shards, replicas are copy of your shards 9/14/2018 16
17. Elasticnodes  Master Node : which controls the cluster  Data Node : Data nodes hold data and perform data related operations such as CRUD, search, and aggregations.  Ingest Node : Ingest nodes are able to apply an ingest pipeline to a document in order to transform and enrich the document before indexing  Coordinating Node : only route requests, handle the search reduce phase, and distribute bulk indexing. 9/14/2018 17
18. 9/14/2018 18
19. ElasticsearchCLUSTER 9/14/2018 19
20. TYPICALCLUSTERSHARD&REPLICA 9/14/2018 20
21. Shardsearchandindex 9/14/2018 21
22. DemoofElasticSearch 9/14/2018 22
23. LOGSTASH  Ruby application runs under JRuby on the JVM  Collects, parse, enrich data  Horizontally scalable  Apache 2.0 License  Large amount of public plugins written by Community  https://github.com/logstash-plugins 9/14/2018 23
24. Typicalusageof Logstash 9/14/2018 24
25. 9/14/2018 25
26. Logstashinput 9/14/2018 26
27. Logstashfilter 9/14/2018 27
28. Logstashoutput 9/14/2018 28
29. DEMOLogstash 9/14/2018 29
30. Beats 9/14/2018 30
31. Beats  Lightweight shippers written in Golang (Non JVM shops can use them)  They follow unix philosophy; do one specific thing, and do it well  Filebeat : Logfile (think of it tail –f on steroids)  Metricbeat : CPU, Memory (like top), redis, mongodb usage  Packetbeat : Wireshark uses libpcap, monitoring packet http etc  Winlogbeat : Windows event logs to elastic  Dockbeat : Monitoring docker  Large community lots of other beats offered as opensource 9/14/2018 31
32. 9/14/2018 32
33. FILEBEAT 9/14/2018 33
34. X-Pack  Elastic commercial offering (This is one of the ways they make money)  X-Pack is an Elastic Stack extension that bundles  Security (https to elastic, password to access Kibana)  Alerting  Monitoring  Reporting  Graph capabilities  Machine Learning 9/14/2018 34
35. 9/14/2018 35
36. Kibana  Visual Application for Elastic Search (JS, Angular, D3)  Powerful frontend for dashboard for visualizing index information from elastic search  Historical data to form charts, graphs etc  Realtime search for index information 9/14/2018 36
37. 9/14/2018 37
38. DEMOKIBANA 9/14/2018 38
39. Designswewentthrough  We started with simple design to measure throughput  One instance of logstash and one instance of ElasticSearch with filebeat 9/14/2018 39
40. DotnetCoreapp  We used a dotnetcore application to generate logs  Serilog to generate into json format and stored on file  Filebeat was installed on the linux machine to ship the logs to logstash 9/14/2018 40
41. Performanceelastic  250 logs item per second for 30 minutes 9/14/2018 41
42. overview 9/14/2018 42
43. logstash 9/14/2018 43
44. Elasticsearchruntwo  1000 logs per second, run for 30 minutes 9/14/2018 44
45. performance 9/14/2018 45
46. Otherdesigns 9/14/2018 46
47. Otherdesignsusingredis 9/14/2018 47
48. Usingfilebeat 9/14/2018 48
49. Filebeatwithoutrelay 9/14/2018 49
50. Log4j 9/14/2018 50
51. Log4jdirect 9/14/2018 51
52. Whatwearegoingwithfornow,until….. 9/14/2018 52
53. Considerationsofdata  Index by day make sense in some cases  In other you may want to index by size rather (Black Friday more traffic than other days) when Shards are not balance ElasticSearch doesn’t like that  Don’t index everything, if you are not going to search on specific fields mark them as text 9/14/2018 53
54. FutureConsiderations  Investigate into Elastic Search Machine learning  ElasticSearch with Kafka for cross data center replication  Logstash Centralizex Pipeline for SEIM intergations 9/14/2018 54
55. Thankyou& Opento questions  – Questions???  – Contact: Taswar.bhatti@gemalto.com  – LinkedIn (find me and add me) 9/14/2018 55

Csharp_Pkcs11_Interop_Encryption_Decryption

I wanted to blog about using C# with pkcs11 on SafeNet ProtectServer HSM for your encryption need. The library I intent to use is the Pkcs11 Interop library on GitHub.
To being with we need to understand what an HSM is? In wikipedia we find this definition.

A hardware security module (HSM) is a physical computing device that safeguards and manages digital keys for strong authentication and provides cryptoprocessing. These modules traditionally come in the form of a plug-in card or an external device that attaches directly to a computer or network server.

So how does an HSM really look like? Its just like a Pizza box that you see for other servers like below the Gemalto ProtectServer HSM.

protectserverhsm

protectserverhsm

Getting Started

In order to use the Gemalto ProtectServer HSM you first need to download the Driver. Unfortunately it is not distributed publicly so you need to have an account to download the driver and software related to it. Trust me I work for Gemalto and had to create a customer account in order to download it.
Assuming that you have the driver and have installed you should have most of your software located at C:\Program Files\SafeNet\Protect Toolkit 5
In our application we will be using a Network HSM, in order to use an Network HSM you need to execute the command of SetMode.cmd

One will have to choose the Network Mode, remember to enter the ip address of the HSM in your registry key. ET_HSM_NETCLIENT_SERVERLIST, needs to have the ip or ip address of the machines you plan to connect.

One can then use KMU (a java program) or ctbrowse.exe (a native windows application) to view the HSM.

Tools

Here is the Java Program to use to create keys and manage HSM etc

kmu

kmu

Another application that one can use to create keys and manage the HSM.

ctbrowse

ctbrowse

Now lets use the tool to create the key we want, I will use the KMU tool for this and create an AES key labelled as demokey, note there are attributes for the key and I have generated it into a Slot that I plan to use. HSM are divided into Slots that one can use.

generatekey_aes

generatekey_aes

C# PKCS11 on SafeNet ProtectServer HSM

Now we can finally get into the code we will use the interop library one can install using nuget.

PM> Install-Package Pkcs11Interop

Summary

In the above example we have encrypted hello world with our demo key and also decrypted it using C# with the Interop library. The key never gets into your code since the HSM is the one which encrypts it.

UA-4524639-2