Showing posts with label development. Show all posts
Showing posts with label development. Show all posts

Wednesday, October 22, 2014

What qualifies a Project to be a Product?

Disclaimer:
This post is based upon a piece of dream (not fiction). Any relation to any enterprise grade product is purely coincidental.

---

Now, this is not a flame post against Proprietary Software. I'm a FOSS supporter but with understanding that for some Business... 
* paid 24x7 support is a lot more critical than quality
* need to put trust in a product where the Vendor is bound by agreement to help
which is perfectly fine.
Depending on a business requirement and policies, there need to be different solutions provided. Some community supported FOSS and then many large corporations (backed for years) Enterprise Software.

Again, no this post is not on license of software.
This post is on some core values of a software (FOSS or Proprietary) that make it eligible to be used at a worthy scale in a Organization that is depending on it.
These also stand true for any user of that Software Project, but are more crucial for users depending their economy on it.
And are ethically right of the Corporations which are paying big bucks for a piece of code sold to them on high brand and big promises.

Here on I use "Products" for all paid-for (some Enterprise grade) Projects that I got to work on since college and experience the truth underneath.
---

So... What qualifies a Project to be a Product?

  • 2 old-skool fundamental mantras: loose coupling; high cohesionThere are "Products" with bunch of modules that have separate responsibilities. Correct approach. But when you start to build around them, sometimes they are not that independent in containing self responsibilities.
    That's worse than not having your project modular at all, at least that way you promote users to treat it as a black-box.
    "Product" need to be well modular-ized emitting the age old beautiful development practices of keeping your modules loosely coupled, strongly cohesive.
  • isolated from client specific details
    Anything and everything that depicts details about client specific implementation need to be managed as a configuration provided explicitly. There shouldn't be any necessity of find-replace any hard-coded "config text" in source or set-up files for the "Product". These details shan't be required to be packaged with custom traditional set-up of "Product".
  • generic out-of-the-box setup... tested and isolated
    The "Product" installer binaries shall be self-contained. They can be O.S. distribution specific, which is fine.
    They should be unaffected of O.S. level restrictions that may be there or not. Say if SELinux is enabled, your setup shall be able to initiate changes required for the "Product" to function.
    Any dependency shall be either bundled in the installer package, or shall utilize the targeted package manager to lay it down for itself.
    The installer shall be tested completely in the mode it is supposed to be used. For example, if it sets up the machine remotely then it shall be capable of handling all tasks remotely by itself and not depend on user to place some files for it first or in-between.
  • no mesh between services required for initiation
    The "Product" might have distributed architecture support, might have modular components collaborating.
    Now these distributed components need to be aware of each other to be able to collaborate. The components shall be robust enough to handle unavailable dependency components and regain activity once available.
    This awareness can be maintained on a component dedicated to dynamic configuration query and update, where all component instances export details about self and gather information on others. Environment preparation mechanism can populate the initial dynamic information which on requirement can be updated in collaboration-enabling component and gathered by others.
    Every component can persist information required for it within itself as well, but that shouldn't be dependent on the other components. If component-F collaborates with component-A and requires component-A to mark some activities for it even to start itself. Bad design.
  • don't promote deprecated technology in newer component
    Some "Product" have big lifetime, depending on their vastness &/or critical nature. This might lead to existence of certain obsolete technologies (like SOAP in yr2014) usage in newer components.
    You shouldn't start rebuilding entire perfectly working "Product" for that, yes. But neither shall you use that as an excuse for holding back entire new development around it. It will cripple the "Product" much faster of surviving in newer circumstances if it can't use the power required for them.
    Build a mid-layer contract API and seclude your new work over your legacy "Product". Then build new features over that mid-layer API.
    This will help you avoid "Product" becoming a bloatware just because new features can't be used inherently from the design itself. It will also help contain the sanity of perfectly working "Product" from some (library, etc.) changes required for new features. And will let you use more advance current best practices for all the newer work, not cripple it.
    If not "Product" in entirety; at least every individual component in itself shall follow unified design/development/interface/platform strategy... it shouldn't have bifurcation of ideals, if necessary a new component shall be carved out and plugged-in instead of corrupting the existing piece.
  • different style of code, different degree of documentation
    If your code-base is not very huge, beautifully written clean and modular code can survive without documentation.
    If your code is not clean (don't judge it yourself, ask someone expert in language/framework but unaware of logic to guess) then have freaking documentation all over it.
    If your code-base is "really" huge, even if your code is mostly clean... have a basic module level documentation at least.
    So anyone using it knows how to handle them and build upon or around them. Anyone visiting the code-base many years after (if you think your "Product" is capable) with lot improved language features is still able to make sense of what was carved with stone on cave walls.
  • building upon/around it shouldn't involve hack-y ways
    In small home-brewed solutions, unpack-replace-repack is still arguably accepted solution.
    In a freaking "Product", it shall expose an elegant yet secure API interface to extend features. If it allows overriding of existing features, even they shall be plugged in via that interface causing the built-in feature to be subdued.
    It's not effortless but is a sane and secure method. The "Product" need this embedded in it's design.
  • for all kinds of data-set involved, should have a proper data-modification strategy
    Hand modification seem harmless at development stage, even at testing sometimes. In production if any kind of data modification done during setup or updates doesn't have a migration/rollback strategy around it, it's a danger zone signal.
    Like proper DB migration scripts over (and not plain diffs of) current version of scripts. I might seem obvious as all other points, but there are bunch of Product missing one or other variations of it.
  • take care of sensitive data involved even during setup of "Product"
    If your Product in any manner forces display or storage of sensitive data (usernames, passwords, machine names, network details, yada yada yada), danger zone again.
  • have at least some level of sanity tests for all logic flows
    Even in this age of software development, if importance of software testing (acceptance, regression, security) need to be advocated to you, it's disaster.
    To state obvious... any change can be tracked properly, all technologists have some level of idea and trust on what's where doing what, people picking it much later can build over it without breaking anything.


and some things, but that for some other time.....

---

LONG QUESTION:
Could a piece of software sold by "so called tech moghul" Organizations be so crappy that the only purpose it fills is vendor lock-in for years of licensing and not even the creators know it's buggy to its core?
SHORT ANSWER:
Yes! So chose with care.

Monday, February 3, 2014

golang ~ get local changes into GOPATH without pushing them upstream

To get your local Golang repo's sym-linked at your GOPATH and local changes available...

goenv_link(){
  if [ $# -ne 2 ]; then
    echo "Links up current dir to it's go-get location in GOPATH"
    echo "SYNTAX: goenv_linkme  "
    return 1
  fi
  _REPO_DIR=$1
  _REPO_URL=$2

  _TMP_PWD=$PWD
  cd $_REPO_DIR

  if [ -d "${GOPATH}/src/${_REPO_URL}" ]; then
    echo "$_REPO_URL already exists at GOPATH $GOPATH"
    go get "${_REPO_URL}"
    return 1
  fi
  _REPO_BASEDIR=$(dirname "${GOPATH}/src/${_REPO_URL}")
  if [ ! -d "${_REPO_BASEDIR}" ]; then
    mkdir -p "${_REPO_BASEDIR}/src"
  fi

  ln -sf "${PWD}" "${GOPATH}/src/${_REPO_URL}"
  go get "${_REPO_URL}"

  cd $_TMP_PWD
}

alias goenv_linkme="goenv_link $PWD"

---


Every now and then working on my favorite new programming language Golang, I have inter-dependent changes among different packages. To confirm their as-required working state, I'd like the GOPATH to provide the compiled object with local-changes included.

The utility I've been using to push local package changes to GOPATH provided object is following "goenv_alpha" bash function as a shell-profile provided utility.

Say, I've a golang project "github.com/abhishekkr/goshare" which utilizes "github.com/abhishekkr/goshare/httpd", "github.com/abhishekkr/goshare/zeromq" and few more.

If I make some local changes at "{PROJECTS}/goshare" and "{PROJECTS}/goshare/httpd". To push those into GOPATH provided package for testing, following commands using below function "goenv_alpha" shell-util would do the job...

$ goenv_alpha "{PROJECTS}/goshare" "github.com/abhishekkr/goshare"
$ goenv_alpha "{PR..}/goshare/httpd" "github.com/abhishekkr/goshare/httpd
"

These commands will ask you to make a backup file for current existing version of package resource from GOPATH, you can give any name... which will be asked while restoring or you can leave it empty to avoid creating a backup file.

~

goenv_alpha(){   _TMP_PWD=$PWD   if [ $# -ne 2 ]; then echo "Provide Alpha changes usable as any other go package."     echo "Just the import path changes to 'alpha/'"     echo "SYNTAX: goenv_alpha "     return 1   fi _REPO_DIR=$1   _REPO_URL=$2   cd $_REPO_DIR   _PKG_PARENT_NAME=$(dirname $PWD)   _PKG_NAME=$(basename $PWD)
  _PKG_NAME_IN_REPO=$(basename $_REPO_URL)   if [ $_PKG_NAME_IN_REPO != $_PKG_NAME ]; then echo "Path for creating alpha doesn't match the import 'url' for it."     return 1   fi   `go build -work . 2> /tmp/$_PKG_NAME`   _BUILD_PATH=`cat /tmp/$_PKG_NAME | sed 's/WORK=//'`   if [ ! -d $_BUILD_PATH ]; then echo "An error occured while building, it's recorded at /tmp/$_PKG_NAME"     return 1   fi rm -f /tmp/$_PKG_NAME   _CURRENT_OBJECT_PATH="${GOPATH}/pkg/${GOOS}_${GOARCH}"   _CURRENT_OBJECT="${_CURRENT_OBJECT_PATH}/${_REPO_URL}.a"   _NEW_OBJECT="${_BUILD_PATH}/_${_PKG_PARENT_NAME}/${_PKG_NAME}.a"   echo "Do you wanna backup current object? If yes enter a filename for it: "   read GO_ALPHA_BACKUP   if [ ! -z $GO_ALPHA_BACKUP ]; then mv $_CURRENT_OBJECT "${_CURRENT_OBJECT_PATH}/${_REPO_URL}/${GO_ALPHA_BACKUP}.backup"   fi mv $_NEW_OBJECT $_CURRENT_OBJECT   cd $_TMP_PWD   echo "\nAlpha changes have been updated at ${_CURRENT_OBJECT}." }

~

You can undo the pushing of local changes inclusive package resource if you have created a backup file for earlier existing file.

Following commands utilizes the below provided shell-util function "goenv_alpha_undo".

$ goenv_alpha_undo "{PROJECTS}/goshare" "github.com/abhishekkr/goshare"
$ goenv_alpha_undo "{PR..}/goshare/httpd" "github.com/abhishekkr/goshare/httpd"

This will list you the names of backup files present if any, then you can provide the name of your chosen backup file and restore to that package state.

~
goenv_alpha_undo(){
  _TMP_PWD=$PWD
  if [ $# -ne 2 ]; then
    echo "Provide Alpha changes usable as any other go package."
    echo "Just the import path changes to 'alpha/'"
    echo "SYNTAX: goenv_alpha  "
    return 1
  fi _REPO_DIR=$1
  _REPO_URL=$2   cd $_REPO_DIR   _PKG_PARENT_NAME=$(dirname $PWD)   _PKG_NAME=$(basename $PWD)   _PKG_NAME_IN_REPO=$(basename $_REPO_URL)   if [ $_PKG_NAME_IN_REPO != $_PKG_NAME ]; then echo "Path for creating alpha doesn't match the import 'url' for it."     return 1   fi _CURRENT_OBJECT_PATH="${GOPATH}/pkg/${GOOS}_${GOARCH}"   _CURRENT_OBJECT="${_CURRENT_OBJECT_PATH}/${_REPO_URL}.a"   _BACKUP_OBJECT="${_BUILD_PATH}/_${_PKG_PARENT_NAME}/${_PKG_NAME}.a"   echo "Available package files are:"   ls -1 $_CURRENT_OBJECT_PATH/$_REPO_URL | grep $_PKG_NAME | grep -v grep   echo "Enter your backup filename for it: "   read GO_ALPHA_BACKUP   if [ -z $GO_ALPHA_BACKUP ]; then echo "\nNo Backup file was entered." ; return 1   fi mv "${_CURRENT_OBJECT_PATH}/${_REPO_URL}/${GO_ALPHA_BACKUP}" $_CURRENT_OBJECT   cd $_TMP_PWD   echo "\nAlpha changes have been reverted with the provided backup file." }
~

The full [WIP] shell-profile for golang utilities is at:
https://github.com/abhishekkr/tux-svc-mux/blob/master/shell_profile/a.golang.sh

Wednesday, April 24, 2013

Beginner's Guide to OpenStack : Basics of Nova [Part 2]

parts of Beginner's Guide to OpenStack to read before this ~

[Part.2 Basics of Nova] Beginner's Guide to OpenStack

# Nova?
It's the main fabric controller for IaaS providing Cloud Computing Service by OpenStack. Took its first baby steps in NASA. Contributed to OpenSource and became most important component of OpenStack.
It built of multiple components performing different tasks turning End User's API request into a virtual machine service. All these components run in a non-blocking message based architecture, and can be run off from same or different locations with just access to same message queue service.

---

# Components?

Nova stores states of virtual machines in a central database. It's optimal for small deployments. Nova is moving towards multiple data stores with aggregation for high scale requirements.








  • Nova API : supports OpenStack Compute API, Amazon's EC2 API and powerful Admin API (for privileged users). It's used to initiate most of orchestration activities and policies (like Quota). It gets communicated over HTTP, converts the requests to commands further contacting other components via Message Broker and HTTP for ObjectStore. It's a WSGI application which routes and authenticates requests.
  • Nova Compute : worker daemon taking orders from its Message Broker and perform virtual machine create/delete tasks using Hypervisor's API. It also updates status of its tasks in Database.
  • Nova Scheduler : decides which Nova Compute Host to allot for virtual machine request.
  • Network Manager : worker daemon picking network related tasks from its Message Broker and performing those. OpenStack's Quantum now with Grizzly release can be opted instead of nova-network. Tasks like maintaining IP Forwarding, Network Bridges and VLANs get covered.
  • Volume Manager : handles attach/detach of persistent block storage volumes to virtual machines (similar to Amazon's EBS). This functionality has been extracted to OpenStack's Cinder. It's an ISCSI solution utilizing Logical Volume Manager. Network Manager doesn't interfere in Cinder's tasks but need to be setup for Cinder to be used.
  • Authorization Manager : interfaces authorized APIs usage for Users, Projects and Roles. It communicates with OpenStack's KeyStone for details.
  • WebUI : OpenStack's Horizon communicates with Nova API for Dashboard interfacing.
  • Message Broker : All components of Nova communicate with each other in a non-blocking callback-oriented manner using AMQP protocol well supported by RabbitMQ, Apache QPid. There is also emerging support for ZeroMQ integration as Message Queue. It's like central task list shared and updated by all Nova components.
  • ObjectStore : It's a simple file-based storage (like Amazon's S3) for images. This can be replaced with OpenStack's Glance.
  • Database : used to gather build times, run states of virtual machines. It has details around instance types available, networks available (if nova-network), and projects. Any database supported by SQLAlchemy can be used. It's central information hub for all Nova components.


---

# API Style
Interface is mostly RESTful. Routes (python re-implementation of Rails route system) packages maps URIs to action methods on controller classes.
Each HTTP Request to Compute requires specific authentication credentials required. Multiple authentication schemes can be allowed for a Compute node, provider determines the one to be used.

---

# Threading Model
Uses Green Thread implementation by design using eventlet and greenlet libraries. This results into single process thread for O.S. with it's blocking I/O issues. Though single reduces race conditions to great extent, to eliminate them further in suspicious scenarios use decorator @lockutils.synchronized('lock_name') over methods to be protected from it.
If any action is long-running, it should have methods with desired process-state location triggering eventlet context switch. Placing something like following code-piece will switch context to waiting threads, if any. And will continue on current thread without any delay if there is no other thread in wait.
from eventlet import greenthread
greenthread.sleep(0)
MySQL query uses drivers blocking main process thread. In Diablo release a thread pool was implemented but removed because of trade-off for advantages over bugs.

---

# Filtering Scheduler
In short it's the mechanism used by 'nova-scheduler' to choose the worthy nova-compute host for new required virtual machine to be spawned upon. It prepares a dictionary of unfiltered hosts and weigh their costing for creating required virtual machine(s) request. Then it chooses the least costly host.
Hosts are weighted based on the configuration options for virtual machines.
It's a better practice for customer to ask for large count of required instances together as each request computes weight.

---

# Message Queue Usage
Nova components use RPC to communicate each other via Message Broker using PubSub. Nova implements rpc.call (request/response, API acts as consumer) and rpc.cast (one way, API acts as publisher).
Nova API and Scheduler uses message queue as Invoker, whereas Network and Compute act as workers. Invoker pattern sends messages via rpc.call or rpc.cast. Worker pattern receives messages from queue and respond back to rpc.call with appropriate response.
Nova uses Kombu library when interfacing with RabbitMQ.

---

# Hooks
Enable developers to extend Nova capabilities by adding named hooks to Nova code as decorator that will lazily load plug-in code matching hook name (using setuptools entrypoints, it's an extension mechanism). The hook's class definition should have pre and post method.
Don't use hooks when stability is a factor, internal APIs may change.

---

# Dev Bootstrap
To get started with contributing... read this (OpenStack Wiki on HowToContribute) in detail.

To get rolling with Nova wheels, system will need to have libvirt and one of the hypervisors (xen/kvm preferred for linux hosts) present.
$ git clone git://github.com/openstack/nova.git
$ cd nova
$ python ./tools/install_venv.py
this will prepare your copy of nova codebase with virtualenv required, now any command you wanna run on this in context of required codebase
$ ./tools/with_venv.sh

---

# Run My Tests
to run the nose tests and pep8 checker, when you are done with virtualenv setup (or that will be initiated first here)... inside 'nova' codebase
$ ./run_tests.sh

---

# Terminology

  • Server: Virtual Machines created inside Compute System, required Flavor & Image detail.
  • Flavor: Represents unique hardware configurations with disk space, memory and CPU time priority
  • Image: System Image File used to create/rebuild a Server
  • Reboot: Soft Server Reboot sends a graceful shutdown signal. Hard Reboot does power reset.
  • Rebuild: Removes all data on Server and replaces it with specified image. Server's IP Address and ID remains same.
  • Resize: Converts existing server to a different flavor. All resize need to be explicitly confirmed, only then the original server is removed. After 24hrs. delay, there is an automated confirmation.


Wednesday, June 6, 2012

Get Set Go Lang ~ part#1

Get Set GO Lang
part# 1
_________________________

What Is Go Lang?
(in case you just came here while curious web surfing)


Go is an OpenSource programming platform developed by Google (and contributors) to be expressive and efficient at the same point.
It's distributed under BSD-style License
It's a concurrency favoring, statically typed, compiler-based language. Though it declares to be giving ease like dynamically typed interpreted code.
_________________________

On your mark, Get Set GO
(getting started with the quick boost usage)

To directly start playing with Go Lang, visit http://play.golang.org/,
where you can directly type/paste in your go-lang code in an online editor and run to get output.

Just a small ++HelluvaWorld code-piece
package main 
import ("fmt"
        "time"
        "os"
        "math" )
func main() {
  fmt.Println("Today is ", time.Now().Weekday())
  fmt.Println("env as ", os.Environ())


  fmt.Println("A Pi on Ceil looks like ",
               math.Ceil(math.Pi), 
      " and a Pie on Floor looks like", 
      math.Floor(math.Pi))
}
[] Installing it for local & full-flown development practice http://golang.org/doc/install would guide you getting 'go' working on your Linux, FreeBSD, OSX & Win platforms.
_________________________

Rewind before the Start Line and take your First Leap
(first useful step to starting use of Go Lang)

[] quickie at variables and constants, a look at GO's declaration style
// var used to declare variable with type at end
var a, b, c int
// direct initialization doesn't require providing type
var x, y, z = 1, true, "yes"
// constants just require a 'const' keyword
const newconst = 10
func tellvar() {
a, b, c = newconst + 1, newconst + 2, newconst + 3
// inside a function, even := construct
// could be used to assign and not use 'var'
clang, java, ruby := "dRitchie", "jGosling", "Matz"
fmt.Println(a, b, c, x, y, z, clang, ruby, java)
}
now, you also know '//' is to comment as in C/C++ and more.

[] mobilizing functions
just an emulation of 'math' libraries 'pow' method (also a look at using for loop)
package main 
import "fmt" 
func pow(x int, y int) int {
  a := 1
  for i := 0; i < y; i++ {
    a = a * x
  }
  return a
}
 
func main() {
  fmt.Println( "2 to the power of 5 is ", pow(2, 5) )
}

[] some function parameters style, the pow above is same as
func pow(x, y int) int { ... }

[] function returning multiple values
dream come true of how a function can return any number of values (also use of if condition)
func plusminus(a, b int) (int, int) { 
     if a > b {return a+b, a-b} 
     return a+b, b-a
}
  or could be like
func plusminus(a, b int) (plus, minus int) { 
     plus = a + b 
    if a > b {   
                   minus = a - b  
     } else {    
       minus = b - a  
     }   
     return
   }

usage:
plus, minus := plusminus(1, 2)
fmt.Println("Plus: ", plus,
            "\nMinus: ", minus)

[] more to go..... in more to come


_________________________

Shops to Go
(other fine links to Go, until next part of this tutorial comes)
_________________________

Wednesday, December 14, 2011

Basics of Powershell ~ empowering Windows Config/Admin Automation

Basics  of  MS Windows Powershell


Introduction
A dotNet framework based scripting language to automate the configuration/administration of Microsoft Windows machine.
Powershell is loaded with several cmdlets (special command-lets) acting as a built-in shell utilities to perform different tasks on Windows machine for performing administrative tasks.


Getting Started
Powershell's cmdlets act upon and return objects as a result of any action taken.
These can be used in combination with traditional Windows services like Registry, Net and more.

To try your hands over powershell, access it at 'Start Menu' > 'Accessories' > 'Windows Powershell';
there you'll get mainly a shell from 'Powershell', and an interactive IDE-like shell 'Powershell ISE'.

Using powershell, cmdlets are the main power-source of this Powershell which are discussed briefly below.....

to get a quick /Hello World/ feel of Powershell, you could try on few next steps
* emulating a dos command 'echo' used to print message at console, 
   command: 
      Write-Host "[get-help write-host] would tell about cmdlet write-host"
* emulating a dos command 'cd' used to change current directory, 
   command: Set-Location C:\Temp
* emulating a dos command 'mkdir' used to create directory, 
   command: New-Item -name "ztemp" -type directory -Force
* emulating a dos command 'dir' used to list items in current directory, 
   command: Get-ChildItem -path C:\Temp
* even the windows 'dir' command can be executed and played with inside powershell.
.....could get a list of cmdlets available in Powershell2.0 ~ http://ss64.com/ps/


[] cmdlets
Command-lets have a specific name format of /'verb'-'noun'/ such as 'Get-ChildItem', 'Get-Help', etc.
To know more about any 'cmdlet' (like using man in linux shell), use
command: Get-Help <_cmdlet_> -detailed

[] A Glimpse of system level SuperUser stuff, as possible on a linux shell
  [+] to get a listing of all System Services,
          command:  Get-Service
  [+] to just get a listing of 'Spooler' named system service, telling its 'name','status' and 'display name'
          command:  Get-Service | Where {$_.name -eq 'Spooler'}
  [+] to get current state of 'Spooler' named system service, whether its running/stopped/paused
          command:  Get-Service | Where {$_.name -eq 'Spooler'} | %{ $_.status }
  [+] to have a Powershell script, check for a system service..... start it if stopped
  $svc_name = "aspnet_state"
  $svc_status = Get-Service | Where {$_.name -eq $svc_name} | %{ $_.status }
  if (-not $svc_status) {
     Write-Host "Error: $svc_name not found"
  }
  elseif ($svc_status -eq "running"){
     Write-Host "status ok $svc_name"
  } else {
     Start-Service $svc_name
  }
  [+] Now you can save the script above as any file say 'start_aspnet.ps1', but to execute it as an external script you would need the local system's Execution Policies to be unrestricted for the script.
       It's not advisable to have it un-restricted all the times, so you could pass on the specific modes along-with the script you desire to be run in un-restricted mode. As below.....
          cmd_prompt:> powershell -executionpolicy unrestricted -file ".\start_aspnet.ps1"
     


helpful links:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa973757(v=vs.85).aspx
http://blogs.msdn.com/b/powershell/
http://thepowershellguy.com/blogs/posh/default.aspx
http://powershell.com/cs/
http://stackoverflow.com/questions/496234/what-tutorial-do-you-recommend-for-learning-powershell