Taswar Bhatti
The synonyms of software simplicity

I had the opportunity to speak in my native home Hong Kong for the first time when I was visiting for holidays on Cloud Design Patterns. It was a lot of fun and thanks to Codeaholics for holding the meetup group in such great form in Hong Kong.

For people who wish to see my slides.

Transcript:

  1. Ponder • For every 25 percent increase in problem complexity, there is a 100 percent increase in solution complexity. • There is seldom one best design solution to a software problem. • If cars were like software, they would crash twice a day for no reason, and when you called for service, they’d tell you to reinstall the engine.
  2. Who am I • Taswar Bhatti – Microsoft MVP since 2014 • Global Solutions Architect/System Architect at Gemalto • In Software Industry since 2000 • Native Hong Kong, but live in Canada now • I know Kung Fu (Languages)
  3. What I am not
  4. Agenda • What are Patterns? • The External Configuration Pattern • The Cache Aside Pattern • The Federated Identity Pattern • The Valet Key Pattern • The Gatekeeper Pattern • The Circuit Breaker Pattern • The Retry Pattern • The Strangler Pattern • Questions
  5. Bad Design
  6. Bad Design
  7. Bad Design
  8. Bad Design?
  9. Itunes when I use it
  10. What are Patterns? • General reusable solution to a recurring problem • A template on how to solve a problem • Best practices • Patterns allow developers communicate with each other in well known and understand names for software interactions.
  11. External Configuration Pattern
  12. External Configuration Pattern • Helps move configuration information out of the application deployment • This pattern can provide for easier management and control of configuration data • For sharing configuration data across applications and other application instances
  13. Typical Application
  14. Storing Configuration in file
  15. Multiple application
  16. Problems • Configuration becomes part of deployment • Multiple applications share the same configuration • Hard to have access control over the configuration
  17. External Configuration Pattern
  18. When to use the pattern • When you have shared configuration, multiple application • You want to manage configuration centrally by DevOps • Provide audit for each configuration
  19. 20. When not to use • When you only have a single application there is no need to use this pattern it will make things more complex
  20. Cloud Solution Offerings • Azure Key Vault • Vault by Hashicorp • AWS KMS • Keywhiz
  21. Cache Aside Pattern
  22. Cache Aside Pattern • Load data on demand into a cache from datastore • Helps improve performance • Helps in maintain consistency between data held in the cache and data in the underlying data store.
  23. Typical Application
  24. Cache Aside Pattern
  25. When to use the pattern • Resource demand is unpredictable. • This pattern enables applications to load data on demand • It makes no assumptions about which data an application will require in advance
  26. When not to use • Don’t use it for data that changes very often
  27. Things to consider • Sometimes data can be changed from outside process • Have an expiry for the data in cache • When update of data, invalidate the cache before updating the data in database • Pre populate the data if possible
  28. Cloud Offerings • Redis (Azure and AWS) • Memcache • Hazelcast • Elastic Cache (AWS)
  29. Federated Identity Pattern
  30. Federated Identity Pattern • Delegate authentication to an external identity provider. • Simplify development, minimize the requirement for user administration • Improve the user experience of the application • Centralized providing MFA for user authentication
  31. Typical Application
  32. Problem
  33. Problem • Complex development and maintenance (Duplicated code) • MFA is not an easy thing • User administration is a pain with access control • Hard to keep system secure • No single sign on (SSO) everyone needs to login again to different systems
  34. Federated Identity Pattern
  35. When to use • When you have multiple applications and want to provide SSO for applications • Federated identity with multiple partners • Federated identity in SAAS application
  36. When not to use it • You already have a single application and have custom code that allows you to login
  37. Things to consider • The identity Server needs to be highly available • Single point of failure, must have HA • RBAC, identity server usually does not have authorization information • Claims and scope within the security auth token
  38. Cloud Offerings • Azure AD • Gemalto STA and SAS • Amazon IAM • GCP Cloud IAM
  39. Valet Key Pattern
  40. Valet Key Pattern • Use a token that provides clients with restricted direct access to a specific resource • Provide offload data transfer from the application • Minimize cost and maximize scalability and performance
  41. Typical Application Client App Storage
  42. Problem Client App Storage Client Client Client Client
  43. Valet Key Pattern Client App Generate Token Limited Time And Scope Storage
  44. When to use it • The application has limited resources • To minimize operational cost • Many interaction with external resources (upload, download) • When the data is stored in a remote data store or a different datacenter
  45. When not to use it • When you need to transform the data before upload or download
  46. Cloud Offerings • Azure Blob Storage • Amazon S3 • GCP Cloud Storage
  47. Gatekeeper Pattern
  48. Gatekeeper Pattern • Using a dedicated host instance that acts as a broker between clients and services • Protect applications and services • Validates and sanitizes requests, and passes requests and data between them • Provide an additional layer of security, and limit the attack surface of the system
  49. Typical Application
  50. Problem
  51. Gatekeeper Pattern
  52. When to use it • Sensitive information (Health care, Authentication) • Distributed System where perform request validation separately
  53. When not to use • Performance vs security
  54. Things to consider • WAF should not hold any keys or sensitive information • Use a secure communication channel • Auto scale • Endpoint IP address (when scaling application does the WAF know the new applications)
  55. Circuit Breaker Pattern
  56. Circuit Breaker Pattern • To handle faults that might take a variable amount of time to recover • When connecting to a remote service or resource
  57. Typical Application
  58. Problem
  59. Client Circuit Breaker Api Closed State Timeout Closed State Open State Half Open State After X Retry Closed State
  60. Circuit Breaker
  61. When to use it • To prevent an application from trying to invoke a remote service or access a shared resource if this operation is highly likely to fail • Better user experience
  62. When not to use • Handling access to local private resources in an application, such as in-memory data structure • Creates an overhead • Not a substitute for handling exceptions in the business logic of your applications
  63. Libraries • Polly (http://www.thepollyproject.org/) • Netflix (Hystrix) https://github.com/Netflix/Hystrix/wiki
  64. Retry pattern
  65. Retry Pattern • Enable an application to handle transient failures • When the applications tries to connect to a service or network resource • By transparently retrying a failed operation
  66. Typical Application Network Failure
  67. Retry Pattern • Retry after 2, 5 or 10 seconds
  68. When to use it • Use retry for only transient failure that is more than likely to resolve themselves quicky • Match the retry policies with the application • Otherwise use the circuit break pattern
  69. When not to use it • Don’t cause a chain reaction to all components • For internal exceptions caused by business logic • Log all retry attempts to the service
  70. Libraries • Roll your own code • Polly (http://www.thepollyproject.org/) • Netflix (Hystrix) https://github.com/Netflix/Hystrix/wiki
  71. Strangler Pattern
  72. Strangler Pattern • Incrementally migrate a legacy system • Gradually replacing specific pieces of functionality with new applications and services • Features from the legacy system are replaced by new system features eventually • Strangling the old system and allowing you to decommission it
  73. Monolith Application
  74. Strangler Pattern
  75. When to use • Gradually migrating a back-end application to a new architecture
  76. When not to use • When requests to the back-end system cannot be intercepted • For smaller systems where the complexity of wholesale replacement is low
  77. Considerations • Handle services and data stores that are potentially used by both new and legacy systems. • Make sure both can access these resources side-by-side • When migration is complete, the strangler façade will either go away or evolve into an adaptor for legacy clients • Make sure the façade doesn’t become a single point of failure or a performance bottleneck.
  78. Questions? Taswar Bhatti System Solutions Architect (Gemalto) Microsoft MVP http://taswar.zeytinsoft.com @taswarbhatti

This would be the second part of the blog post where we used powershell to store some secure data into our registry and have used DPAPI to encrypt the data. I wanted to cover how I would read the data back from the registry in my C# application. An example of C# using DPAPI to read data from Registry.
Feel free to read the blog post on Powershell using DPAPI to store secure data in Registry

I am using .NET Framework 4.6 rather than dotnet core, there is no DPAPI in dotnet core

C# using DPAPI to read data from Registry

I will be using a Console application with .NET Framework 4.6 just to show how I would read the registry and use System.Security.Cryptography to decrypt the data.

If we run this application we will see the data that we have stored in the registry.

One of the easiest cloud design pattern that one can try out is the Retry Pattern. I wanted to show how to use an Retry Pattern using Polly in C# as a example. So what does the Retry Pattern achieves?

Problem Statement – What is the issue the pattern solves?

When building applications you always have some sort of outside/external service including another MicroService that you have to consume or call. Sometimes there could be momentary loss of network connectivity, or a temporary unavailability, or timeouts that occur when that service is busy. You may be calling a database or a restful service that may be busy and fail but if you try back again it will pass. These types of faults are usually self-correcting, and most of the time require some type of delay in calling it again, which will have a success response.

Retry Pattern

  • Enable an application to handle transient failures
  • When the applications tries to connect to a service or network resource
  • By transparently retrying a failed operation
  • Improves the stability of your application
  • Typical Application

    Below is a typical application diagram, where you a service or web app.

    TypicalApplication

    But when the connection to the service fails we usually get an error on our application.

    Typical-Application-Network-Failure

    When to use Retry Pattern

    • Use retry for only transient failure that is more than likely to resolve themselves quickly
    • Match the retry policies with the application
    • Otherwise use the circuit break pattern

    When not to use Retry Pattern

    • Don’t cause a chain reaction to all components
    • For internal exceptions caused by business logic
    • Log all retry attempts to the service

    Sample Code

    Below is a sample dotnet core Console Application that shows the usage using Polly. The code tries to call https://httpbin.org/status/200,408 with a POST which gives us a status of 200 or 408 randomly. First, lets create our code and add the package Polly into it.

    Without Polly

    We will write a sample application that will call the the web service without polly to get 408 errors.

    I am not using a logger but you should be using a logger when you do retry pattern.

    If you have issues with 7.0 C# change your csproj to include <LangVersion>7.1</LangVersion>

    After couple of runs you will see it response back with 408 RequestTimeout

    Using Polly

    Now we will introduce Polly into our code with an incremental delay of 1 second to 3 seconds and lastly 9 seconds.

    Output

    Below you will see three runs of the application with sample output.

    Summary

    As you can see Retry Pattern is quite useful for transient and self correcting failure, not to mention it is quite simple to implement in C# with the help of Polly. If you are looking for Java solutions you can look at Hysterix or even roll your own.

In this blog post I wanted to show how one can use C# or Python to view the serial numbers of a X509 certificate. The serial number can be used to identify the certificate that one plans to use in their C# application, lets say for mutual authentication to another service.

Why use X509 Certificates

  1. Client X.509 certificate identity adds an additional level of asymmetrical cryptography to the standard SSL/TLS channel.
  2. Long security keys (2046 bits)
  3. One can revoke certificates by using Certificate Revocation List (CRL).
  4. Easy to use in application to application

Mutual Authentication

View X509 Certificate Serial Number using C#

If you are using dotnetcore first create a console app using the command below

The command above will create a console app called certsharp and then you can use vscode to load the application.
Below is the sample code that allows one to see the serial number in C#, dotnetcore.

One can the simply run the command below to view the serial, and example show below.

Now lets say you need the serial to find the cert to communicate with another service you can use the code below to find the cert by serial number, and create a rest client to your api like below.

View X509 Certificate Serial Number using Python

For Python I usually create a virtual env before I start coding in python such that I have a separate environment for all my library needs.
To create the virtual env one will need virtualenv installed, you can look at my other Python example of how to install virtualenv and virtualenv-wrapper for windows.

To create a virtual env type the following into your command prompt inside of vscode.

The above command has create the virtual env but not enabled it yet, in order to enable we need to activate it.

Now we can install the package that we need for getting more certificate information, we will use pip and the package is pyopenssl.

Now we can code it up to view the required X509 certificate serial number.

When you run the script it will output something like this

Summary

Above are two examples of getting x509 certificate information specifically serial number, there are additional information that one can view, just check out the api documentation.
One additional thing I found is in python it does not show me the leading 00 of a certificate but C# is able to show that.

I wanted to store some data into registry but how do I store it such that it is secure for my application to read later on. Well there is a way using Powershell to store data into Registry using DPAPI which uses the machine key to store the information. As you know the only way to store data securely is by using encryption and DPAPI provides us with that.

DPAPI only works in windows, if you are on linux environment there is no DPAPI

What is DPAPI?

From Wikipedia we get this definition

DPAPI (Data Protection Application Programming Interface) is a simple cryptographic application programming interface available as a built-in component in Windows 2000 and later versions of Microsoft Windows operating systems. In theory the Data Protection API can enable symmetric encryption of any kind of data; in practice, its primary use in the Windows operating system is to perform symmetric encryption of asymmetric private keys, using a user or system secret as a significant contribution of entropy.

Now that we know it uses the machine key we can write some code to store some value into our registry using powershell. We will be writing into the Wow64 registry since we will most likely use a 64bit application later on. If you need to write to 32bit also just remove the wow64node.

Powershell using DPAPI to store secure data in Registry

What we will do is use SecureString to read the data and convert the secure string into a hash, then we will base64 encode it and store it to the registry, one could just leave it as it and not base64 it. We use the New-ItemProperty to create the value into the registry.

powershell-execute-command

So here it is how to store data into your registry using DPAPI. In my next post I will show how to read back the value using C#.

To continue on with our previous blog post, I will introduce Hashicorp Vault as a key management to manage our secrets for our Nodejs weather application.

Installing Vault

I will use docker to pull the docker image from dockerhub.

docker-install-vault

One can also download Vault for their OS at (https://www.vaultproject.io/downloads.html)

Now that I have vault image pulled, I will create a docker compose file for Vault to use mysql as a back-end store. I can also run Vault in dev mode but if I enable dev mode then Vault runs entirely in-memory and starts unsealed with a single unseal key. I wanted to show more of a real life scenario of starting Vault.

First thing I will create couple of directory and files that I will store some configuration into.

Now that we have a directory we can create a docker-compose.yml file inside of the myvault directory. I assume you already have mysql image, if not you call pull the image from dockerhub.

Before we fire up vault, here is the content of the config.hcl file, located right in the config folder we just created. This will configure Vault with the storage options and listen on port 8200

Running Vault

Use the docker compose file and run the following command to bring up Vault

If you mess up you can always run $docker-compose rm to remove the created containers.

Testing our installation

Now that we have Vault configured and running, we now need to initialize Vault. Vault uses Shamir Secret Sharing Technique for initializing the key to use for Vault. Which takes multiple keys and combines into one single master key.

shamir-secret-sharing-vault-unseal

Unseal Vault

We will use the operator init method to call Vault to initialize.

As we can see it showed us some of the keys we need to use to unseal the Vault. So let unseal it so that we can use it. We will need to unseal the vault with the command 3 times using different keys each time for it, I have skipped the first one.

Now we can use the Vault to write data to, first we need to auth with root token which was given to us when we started the Vault with the keys. My test envirnoment key was “5mEKu64nAk1PA5luVQHRmGLM”. Lets try to auth/login and also write and read something to Vault to store.

We can now also create a sample.json config file that a new Microservice can consume I have placed it in config folder since it was already mounted (note the file is in 1 line)

We can load the sample json data into Vault by using this command.

I am using the weatherapp as a example above

We will most probably also create a policy file so that we can limit the access to this secret, this will be inside the policies directory

In order to load the policy into Vault we will use the command to load the policy in

Now policy is in, lets test out Vault to read some of they values we just added.

Above we have just used the root token to read but at least we know we can get the data

Wrap Token

Vault has a nice feature called wrap token, where you can give a limited amount of time to a token which one can use to access Vault. Lets try to create one for us.

We can use the wrap token above “lYO2AoJ95QEDnZgUbNxoWWsw ” to read the data now, and it is only valid for 60 seconds.

If we try to use another token of if the token is already used we will get an error like below.

App Roles and Secrets

Now that we have learned something about Wrap tokens of how to create and use them, I wanted to switch gear and talk about App Roles and Secrets. Vault provides app roles for you application to login to the system. This is definitely not the best security option out there since its just a like a basic authentication with username and password, but when we use wrap token with it we can mitigate some of the security concerns.

There are better options out there using kubernetes to authenticate for you app etc I will try to cover those later in another blog post

Lets create us an approle for our weather app, and get the roleid that we can use for our application.

Sample Application

Now that we have the roleid we now need a secret in order for our application to login to Vault and get its secrets. Rather than just giving the secret to our application we will give it a wrap token to get the secret since we can time limit the amount of time the token is allowed to live, thus mitigating the risk of exposing the secret. The sample code below is using the env variable and the wrap token to get its secret.

I am using node-vault, you can install node-vault by > npm install node-vault

In the above code we can see that we have hard coded the roleid into our code, but we mitigated the risk of someone stealing our docker env variable by having a limited time to live for the wrap token to authenticate and get our secrets from Vault. The command to generate the wrap token is below.

When we run the code we would do something like, with the third parameter being the wrap token.

If we try to reuse the wrap token we will get an error.

Summary

We have covered using roleid and secret using Vault to authenticate and get our application secrets. There is definitely a down side to this, but we mitigated the risk by having a short time for the wrap token to live. There is also the option of having the wrap token mounted as a volume for your application and pick it up from the volume that was mounted. I will cover those topics in a later blog post.

    Advantages

  • Even with docker inspect we are only are able to see the token which is last with a TTL and is a guid token the attack surface would be lower.
  • A very simple pattern to follow, all config are stored with application Name (e.g secret/appName/config, etc)
    Disadvantages

  • The microservice would require some Restful call to Vault to consume data, may have dependency on vault library to consume it
  • DevOps or some scripts would still be required to put secrets in the correct place
  • Failure of Vault what happens? Service does not start

Source code at
https://github.com/taswar/weatherappwithvault

Yesterday I gave a talk on using ForwardJS conference here in Ottawa on Hashicorp Vault – Using Vault for your Nodejs Secrets.

You can find the slides here on slideshare.

Transcript:

  1. Using vault for your NodeJS Secrets Taswar Bhatti – Solutions Architect Gemalto
  2. Secrets •
  3. About me • Taswar Bhatti (Microsoft MVP) • @taswarbhatti • http://taswar.zeytinsoft.com • Gemalto (System Architect)
  4. So what are secrets? • Secrets grants you AuthN or AuthZ to a system • Examples • Username & Passwords • Database credentials • API Token • TLS Certs
  5. Secret Sprawl • Secrets ends up in • Source Code • Version Control Systems (Github, Gitlab, Bitbucket etc) • Configuration Management (Chef, Puppet, Ansible etc)
  6. Issues • How do we know who has access to those secrets • When was the last time they accessed it? • What if we want to change/rotate the secrets
  7. Desire secrets • Encryption in rest and transit • Only decrypted in memory • Access control • Rotation & Revocation
  8. Secret Management – Vault • Centralized Secret Management • Encrypted at rest and transit • Lease and Renewal • ACL • Audit Trail • Multiple Client Auth Method (Ldap,Github, approle) • Dynamic Secrets • Encryption as a Service
  9. Dynamic Secrets • Allows one to lease a secret for a period of time e.g 2 hrs • Generates on demand and unique for each user/consumption • Audit trail
  10. Secure Secrets • AES 256 with GCM encryption • TLS 1.2 for clients • No HSM is required
  11. Unsealing the Vault • Vault requires encryption keys to encrypt data • Shamir Secret Key Sharing • Master key is split into multiple keys
  12. Shamir Secret Sharing
  13. Unseal • Unseal Key 1: QZdnKsOyGXaWoB2viLBBWLlIpU+tQrQy49D+Mq24/V0B • Unseal Key 2: 1pxViFucRZDJ+kpXAeefepdmLwU6QpsFZwseOIPqaPAC • Unseal Key 3: bw+yIvxrXR5k8VoLqS5NGW4bjuZym2usm/PvCAaMh8UD • Unseal Key 4: o40xl6lcQo8+DgTQ0QJxkw0BgS5n6XHNtWOgBbt7LKYE • Unseal Key 5: Gh7WPQ6rWgGTBRSMecuj8PR8IM0vMIFkSZtRNT4dw5MF • Initial Root Token: 5b781ff4-eee8-d6a1-ea42-88428a7e8815 • Vault initialized with 5 keys and a key threshold of 3. Please • securely distribute the above keys. When the Vault is re-sealed, • restarted, or stopped, you must provide at least 3 of these keys • to unseal it again. • Vault does not store the master key. Without at least 3 keys, • your Vault will remain permanently sealed.
  14. How to unseal • vault unseal -address=${VAULT_ADDR} QZdnKsOyGXaWoB2viLBBWLlIpU+tQrQy49D+Mq24/V0B • vault unseal -address=${VAULT_ADDR} bw+yIvxrXR5k8VoLqS5NGW4bjuZym2usm/PvCAaMh8UD • vault unseal -address=${VAULT_ADDR} Gh7WPQ6rWgGTBRSMecuj8PR8IM0vMIFkSZtRNT4dw5MF
  15. Writing Secrets • vault write -address=${VAULT_ADDR} secret/hello value=world • vault read -address=${VAULT_ADDR} secret/hello • Key Value • — —– • refresh_interval 768h0m0s • Value world
  16. Policy on secrets • We can assign application roles to the policy path “secret/web/*” { policy = “read” } • vault policy write -address=${VAULT_ADDR} web-policy ${DIR}/web-policy.hcl
  17. Reading secrets based on policy • vault read -address=${VAULT_ADDR} secret/web/web-apps • vault read -address=${VAULT_ADDR} secret/hello • Error reading secret/hello: Error making API request. • URL: GET http://127.0.0.1:8200/v1/secret/hello • Code: 403. Errors: • * permission denied
  18. Demo Using Vault
  19. Demo Docker Environment VAR • Issues with env variables
  20. Mount Temp File System into App • docker run –v /hostsecerts:/secerts …. • To mitigate reading from Env • Store your wrap token in the filesystem to use with vault • Have limit time on wrap token
  21. Wrap Token for App Secrets • Limit time token • Used to unwrap some secrets • vault read -wrap-ttl=60s -address=http://127.0.0.1:8200 secret/weatherapp/config • Key Value • — —– • wrapping_token: 35093b2a-60d4-224d-5f16-b802c82de1e7 • wrapping_token_ttl: 1m0s • wrapping_token_creation_time: 2017-09-06 09:29:03.4892595 +0000 UTC • wrapping_token_creation_path: secret/weatherapp/config
  22. App Roles • Allows machines or apps to authenticate with Vault • Using a role_id and secret_id as credentials • Assign polices to the app • Once logged in you get back a token to get secrets
  23. Demo App Using Node
  24. Kubernetes with Vault • Read Service Account JWT • App Sends Jwt and Role Name to Vault • Vault checks the signature of Jwt • Sends to TokenReviewer API • Vault sends back valid token for app
  25. Thankyou • Contact me (taswar.bhatti@gemalto.com) • @taswarbhatti

In this post, we will continue our previous nodejs weather application and introduce Docker for NodeJS application using VSCode. We will build our application and host it in docker by adding a Dockerfile. We will modify our code to take the APIKEY from an environment variable.

Create Dockerfile

Let’s launch our editor VSCode again with our solution and add the .dockerignore file first, since there are things that we do not want in our Docker image, kind of like gitignore.

Next add a Dockerfile with vscode so that we can build our image, we will use the carbon-alpine image which is v8 of nodejs, the reason we choose alpine is because its small and has minimal size and is secure.

Carbon is the latest LTS (long term support) version of node

Above we will create a working directory and copy our package*.json then we run npm to install our packages. We also expose the default port 3000 and we run our application with npm start.

Note: using –only=production will run npm in production (will not download devDependencies)

Modify our source code

Before we build our image, we need to modify our app.js such that the APIKEY is no longer in the source code. Let’s make it so that it takes it from an environment variable. In doing so we will need to pass our APIKEY in our environment when we start to run our docker image, more on this later.

Build our NodeJS Image

Now that we modified app.js we can now go back to our docker file and build our image. Hit Ctrl+` in your vscode to launch terminal.

Once finished building you can see your docker image by this command

Run the image

Since we modified app.js we now need to inject the APIKEY into our docker image, we can do so with this command in vscode terminal using powershell. You need to use the proper APIKEY that you have got from openweather. We are also exposing the inside port 3000 to our local machine port of 8080

Now we should be able to view our application on http://localhost:8080

weatherapp-8080

Security Issue

So you may think that now that we pass our APIKEY through the environment we are all good, but there is still a security issue here. If someone hacks into your machine that runs docker they are still able to see your secrets. Say What???

Say What?

Lets try to inspect our docker container by using the following commands, first lets find out the container id that docker is running under

Now we can inspect the container by using this command, we don’t have to type the entire id, the first 3 characters would do the trick.

If we look at the config section of the inspect we will see our APIKEY is in plain text

Now you may say someone needs to have root access to view this information but maybe there is a better way we can mitigate this by maybe using a Token. This is where a secret key management comes in to help in storing your secrets. In the next blog post we will go over what Vault is and how it helps us in storing secrets and exposing a token for our docker container to consume without us exposing the APIKEY.

The source code of this can be found at https://github.com/taswar/nodejs-weather-app

In this post I wanted to go through how do we create a Nodejs application with vscode. The application is yet another weather node app, it is an easy example and loosely based on this project (https://codeburst.io/build-a-weather-website-in-30-minutes-with-node-js-express-openweather-a317f904897b), the difference is that we will build upon it to a series of how do we store secrets in node.js application. In the first part we will start by creating a weather app that would require an APIKEY to access an external API.

So let’s get started, first thing first, we need to get us an apikey
1. Head over to https://openweathermap.org/api and set yourself up with an api key, its free.

Let’s now create a node.js express application, let’s open up a cmd or powershell prompt and install express generator

We will now create the weather application, also we will be using ejs for our view and adding the .gitignore file by using the –git option.

In order to run our application we will need to run npm install and afterwards run vscode with it to open our project up with our favorite IDE VsCode.

We should now see the view of vs code.

vscode-nodejs

Let’s open up our index.ejs located in the views folder and modify the html file there to something like this.

We also need to add a style2.css so that our html page looks a bit better. Add a new file in your public/stylesheets folder. The css is taken by the article at (https://codeburst.io/build-a-weather-website-in-30-minutes-with-node-js-express-openweather-a317f904897b)

Now that we have views in place, lets try to tackle our routes. Open up index.js in routes folder.

Last we need to modify our app.js and add our APIKEY to it such that our route/index.js can access it. Add these 2 lines above your 404 code in app.js

Now we can run our application in vscode. Hit Ctrl+` and get your terminal up and running. Type npm start, and you should be able to hit http://localhost:3000 to view your site.

vscode-node-run-app

Now you can test out your weather by just typing in the city.

vscode-node-run-app-ottawa

One can view the source code at https://github.com/taswar/nodejs-weather-app

In our next post we will be dockerizing our solution so that we can run the app on docker and we will try to tackle the APIKEY issue which is embedded right now in our source code.

Yesterday I gave a talk on using Elastic Search for .NET Developers here in Ottawa. The slides used were from mostly from my presentation at DevTeach Montreal last year.

You can find the slides here on slideshare.

Transcript:

1. STORE 2 MILLION OF AUDIT LOGS A DAY INTO ELASTICSEARCH Taswar Bhatti (Microsoft MVP) GEMALTO @taswarbhatti http://taswar.zeytinsoft.co m taswar@gmail.com
2. WHO AM I? – 4 years Microsoft MVP – 17 years in software industry – Currently working as System Architect in Enterprise Security Space (Gemalto) – You may not have heard of Gemalto but 1/3 of the world population uses Gemalto they just dont know it – Gemalto has stacks build in many environnent .NET, Java, Node, Lua, Python, mobile (Android, IOS), ebanking etc
3. AGENDA – Problem we had and wanted to solve with Elastic Stack – Intro to Elastic Stack (Ecosystem) – Logstash – Kibana – Beats – Elastic Search flows designs that we have considered – Future plans of using Elastic Search
4. QUESTION & POLL – How many of you are using Elastic or some other logging solution? – How do you normally log? Where do you log? – Do you log in Relational Database?
5. HOW DO YOU TROUBLESHOOT OR FIND YOUR BUGS – Typically in a distributed environment one has to go through the logs to find out where the issue is – Could be multiple systems that you have to go through which machine/server generated the log or monitoring multiple logs – Even monitor firewall logs to find traffic routing through which data center – Chuck Norris never troubleshoot; the trouble kills themselves when they see him coming
6. Image
7. OUR PROBLEM – We had distributed systems (microservices) that would generate many different types of logs, in different data centers – We also had authentication audit logs that had to be secure and stored for 1 year – We generate around 2 millions records of audit logs a day, 4TB with replications – We need to generate reports out of our data for customers – We were still using Monolith Solution in some core parts of the application – Growing pains of a successful application – We want to use a centralized scalable logging system for all our
8. FINDING BUGS THROUGH LOGS
9. A LITTLE HISTORY OF ELASTICSEARCH – Shay Banon created Compass in 2004 – Released Elastic Search 1.0 in 2010 – ElasticSearch the company was formed in 2012 – Shay wife is still waiting for her recipe app
10. Image
11. ELASTIC STACK
12. ELASTICSEARCH – Written in Java backed by Lucene – Schema free, REST & JSON based document store – Search Engine – Distributed, Horizontally Scalable – No database storage, storage is Lucene – Apache 2.0 License
13. COMPANIES USING ELASTIC STACK
14. ELASTICSEARCH INDICES – Elastic organizes document in indices – Lucene writes and maintains the index files – ElasticSearch writes and maintains metadata on top of Lucene – Example: field mappings, index settings and other cluster metadata
15. DATABASE VS ELASTIC
16. ELASTIC CONCEPTS – Cluster : A cluster is a collection of one or more nodes (servers) – Node : A node is a single server that is part of your cluster, stores your data, and participates in the cluster’s indexing and search capabilities – Index : An index is a collection of documents that have somewhat similar characteristics. (e.g Product, Customer, etc) – Type : Within an index, you can define one or more types. A type is a logical category/partition of your index. – Document : A document is a basic unit of information that can be indexed – Shard/Replica: Index divided into multiple pieces called shards, replicas are copy of your shards
17. ELASTIC NODES – Master Node : which controls the cluster – Data Node : Data nodes hold data and perform data related operations such as CRUD, search, and aggregations. – Ingest Node : Ingest nodes are able to apply an ingest pipeline to a document in order to transform and enrich the document before indexing – Coordinating Node : only route requests, handle the search reduce phase, and distribute bulk indexing.
18. SAMPLE JSON DOCUMENT HTTP CALL JSON DOCUMENT
19. ELASTICSEARCH CLUSTER
20. TYPICAL CLUSTER SHARD & REPLICA
21. SHARD SEARCH AND INDEX
22. DEMO OF ELASTICSEARCH
23. LOGSTASH – Ruby application runs under JRuby on the JVM – Collects, parse, enrich data – Horizontally scalable – Apache 2.0 License – Large amount of public plugins written by Community https://github.com/logstash- plugins
24. TYPICAL USAGE OF LOGSTASH
25. Image
26. LOGSTASH INPUT
27. LOGSTASH FILTER
28. LOGSTASH OUTPUT
29. DEMO LOGSTASH
30. BEATS
31. BEATS – Lightweight shippers written in Golang (Non JVM shops can use them) – They follow unix philosophy; do one specific thing, and do it well – Filebeat : Logfile (think of it tail –f on steroids) – Metricbeat : CPU, Memory (like top), redis, mongodb usage – Packetbeat : Wireshark uses libpcap, monitoring packet http etc – Winlogbeat : Windows event logs to elastic – Dockbeat : Monitoring docker – Large community lots of other beats offered as opensource
32. Image
33. FILEBEAT
34. X-PACK – Elastic commercial offering (This is one of the ways they make money) – X-Pack is an Elastic Stack extension that bundles – Security (https to elastic, password to access Kibana) – Alerting – Monitoring – Reporting – Graph capabilities – Machine Learning
35. Image
36. KIBANA – Visual Application for Elastic Search (JS, Angular, D3) – Powerful frontend for dashboard for visualizing index information from elastic search – Historical data to form charts, graphs etc – Realtime search for index information
37. Image
38. DEMO KIBANA
39. DESIGNS WE WENT THROUGH – We started with simple design to measure throughput – One instance of logstash and one instance of ElasticSearch with filebeat 9/22/2017 39
40. DOTNET CORE APP – We used a dotnetcore application to generate logs – Serilog to generate into json format and stored on file – Filebeat was installed on the linux machine to ship the logs to logstash
41. PERFORMANCE ELASTIC – 250 logs item per second for 30 minutes
42. OVERVIEW
43. LOGSTASH
44. ELASTIC SEARCH RUN TWO – 1000 logs per second, run for 30 minutes
45. PERFORMANCE
46. OTHER DESIGNS
47. WHAT WE ARE GOING WITH FOR NOW, UNTIL…..
48. CONSIDERATIONS OF DATA – Index by day make sense in some cases – In other you may want to index by size rather (Black Friday more traffic than other days) when Shards are not balance ElasticSearch doesn’t like that – Don’t index everything, if you are not going to search on specific fields mark them as text
49. FUTURE CONSIDERATIONS – Investigate into Elastic Search Machine learning – ElasticSearch with Kafka for cross data center replication
50. THANK YOU & OPEN TO QUESTIONS – Questions??? – Contact: Taswar@gmail.com – Blog: http://Taswar.zeytinsoft.com – Twitter: @taswarbhatti – LinkedIn (find me and add me)

UA-4524639-2