Saturday, May 21, 2022

Database DevOps Toools

 For versioning and organizing your database changes there are vwaios devops tools available in market. 

The most known ones are 

  • Liquibase
  • Flyway
  • Mastereo

However Liquibase currently wins over all the avalable options primarily becuase of the following support:

  • Works with both relational & NoSQL database types
  • Provides a snapshot of the current state of your database
  • Compares the state of two databases (to detect potential malicious code)
  • Allows rollbacks to undo a database change
  • Targets (cherry picks) a change or set of changes to roll back
  • Validates database code against predefined rules set by DBAs on-demand or in automation
  • Provides a dashboard view into each database change in your pipeline
  • Measures & provides database DevOps metrics for your pipeline
  • Works with stored logic
  • Defining changesSQL, XML, JSON, YAML
  • Java-based migrations
  • Dry runs
  • Preconditions
  • Selective deployments Many Files
  • Manage the order of changes Hard
Liquibase run the script in its local db and validate both forward and rollback scripts. The local sandbox can run postgres in doackerized environment, 
Liquibase stores the run details in DATABASECHANGELOG table to know the script id has been executed or not.
mvn clean install will run the scripts locally if the the below pluggin is added to the pom.xml:
<build> <pluginManagement> <plugins> <plugin> <groupId>org.liquibase</groupId> <artifactId>liquibase-maven-plugin</artifactId> <version>3.8.0</version> <configuration> <propertyFile>liquibase.properties</propertyFile> </configuration> <dependencies> <dependency> <groupId>postgresql</groupId> <artifactId>postgresql</artifactId> <version>9.1-901-1.jdbc4</version> </dependency> </dependencies> </plugin> </plugins> </pluginManagement>

More :https://docs.liquibase.com/tools-integrations/maven/workflows/
creating-liquibase-projects-with-maven-postgresql.html

Thursday, May 12, 2022

Open-source Library Rule Book

Opens-source Library : When to use What ?
  • Chatbot - Apache Tensor
  • Avoid DB Load: ehcache, Elastic Search
  • Intelligence, running queries from large data, Long running queries: Snowflake
  • Key value based storage: MongoDB, DynamoDB, Postrgresql
  • Storing Josn based data: MongoDB, DynamoDB, Postrgresql
  • Parallel work in pipeline,  splitting work by various actors : Akka
  • Server side templating/html generation: Velocity, Thymeleaf
  • Model based Front end/ pre-build library : Reactjs
  • Download/upload file via SFTP : SSHJ library /ssh/sftp client
  • XML marshalling: JAXB
  • Big data alalytics marshalling: [Python + Spark API] as a parser/pre-processor + [Hadoop] for storage