Tuesday, May 20, 2014

Welcoming Microsoft world onto my blog...

For most of my IT life, I have been thoroughly disconnected with the Microsoft world of enterprise products. One of the reasons could be the fact that I was introduced into the IT through the corridors of Java & then Unix.. Subsequently more things came on the way, but I never got around to diving deep in the MS world.

Sometime back, I did try and explored SQL server 2008 R2 and associated ETL/Analytics pieces, but it didnt excite me as much.

Now, I have an opportunity to learn and share the MS wisdom through a dear friend Sachin Dubey.  He's going to be putting up content on to this blog. He would be contributing content geared towards Microsoft products, e.g. SQL Server, MS-BI, Dot-Net etc.  With significant experience working around MS BI space, Sachin is a self built knowledge repository. I hope we all learn with him.

Sunday, March 16, 2014

shuklaam vardhanam vishnu


I had to prepare my son for his shloka competition, but it was so hard to find it..
Finally, I managed to find the right sanskrit lyrics, hopefully, they are correct.

शुक्लां वर्धनम् विष्णुं  शशिवर्णं चतुर्भुजं ।
प्रसन्नवदनम् ध्यायेत् सर्वविघ्नोपशान्तये ॥ 
in roman,

Shuklaam Vardhanam Vishnum Shashivarnam chaturbhujam
prasannavadanam dhyaayet sarvavighnopashaantaye

Hope it helps

Saturday, February 8, 2014

Programmatic interaction with Facebook data

The last time I interacted with Facebook using a program (2011), it was fairly straightforward, and perhaps it was too simple, it was abused.  As a result, in my recent experiments, I come to find out that, they have made it so complicated (my feeling, no offence to Facebook here), that its quite hard, if not impossible to do genuine development work using Facebook API, let alone achieve something sinister.

I am a self-anointed tech junky, and therefore no one should be surprised that i took up this challenge couple of hours back to build a brand new routine to get data in/out from/to facebook using a popular and open source Data integration tool, called Pentaho 

There were two triggers, one, I had not done some core debugging based research using something new. Second, a friend of mine is building this ecommerce site, www.freemall.in for which the marketing efforts would need a simpler interface to push details to the site, instead of some human going up there and doing all.  So, this programming interface becomes the first step of the future plugin for his ecommerce venture.

As I already indicated, programming with facebook's current API is a relatively hard task, at least for couple of iterations, before you get the hang of it.  So, I started with the tutorial at pentaho's site - http://www.pentahoevalcenter.com/data-integration/advanced-data-integration/facebook/ which turns out to be based on the old Facebook API and prompted me to look elsewhere for the right way of working.

Then comes the big thing - the oAuth authentication protocol employed by Facebook, get a token of this type, then of this type. go to this site, achieve this, copy this and take it there.. Its all so much that I had to go back to google and see if there is something simpler.. and yes, god bless google, there was someone like me who had gone to all the trouble and then of documenting it... God bless his soul for this - 
http://stackoverflow.com/questions/12168452/long-lasting-fb-access-token-for-server-to-pull-fb-page-info

After this, I went ahead with the pentaho tutorial, which did work out nice and fine for me.

I was able to post couple of messages to my test page using the application that i built for this test case.

hope this collection of links at a single place helps others.



Tuesday, February 4, 2014

National Pension System (NPS) - India

The National Pension System (NPS) in India has been pushed for some time now. Government has been trying to increase the scope of the scheme, and hardly been successful at that.

One of the primary reasons has been the low cost structure for the Pension fund managers. While a regular mutual fund charges around 1.5% for the fund management effort, NPS fund managers were charging somewhere between 0.0009% to 0.25%.  At such low costs, it would have made little sense for anybody to be in business.

However, some of the fund managers persisted and are in relatively better standing today.  Some of them who did not continue (e.g. IDFC) continue to observe from a distance.

Over the last few years, returns from the NPS fund managers have been surpassing the general market returns, mostly in all categories.

Also, recently, govt regulations allowed hiking the fund management charges to 0.25%  in addition to allowing partial withdrawals from the NPS corpus.  Both of these measures have flamed the interest in NPS.  By and large, people are now taking notice of the possibility of considering NPS as a possible investment vehicle. 

Slowly, but surely, many private sector companies are also adopting the NPS route for their employees.  Beyond the long term investment option, NPS offers an instant benefit in terms of tax benefit above and beyond 80C.  10% of your Basic Salary + DA is the cap for Tax benefit under NPS. 

Therefore effectively, who falls in the 30% tax bracket, stands to shave of a substantial amount from his taxable income.  Furthermore, the returns on this additional investment are not as measly as the EPF (approx 8-9%), but much more controlled (you can chose ur fund options) and market linked.

All these measures are slowly resurrecting NPS as a possible investment vehicle.

Sunday, February 2, 2014

Proud - Cloudera Certified Developer for Apache Hadoop

I have not been a believer of certifications, that should be clear from the fact that even though I have been working with Informatica since 2002, I never tried their certification till 2011. And, same about Oracle, whose exam I have not yet attempted/planned/thought over. I used to think that the knowledge level will prevail anyway, whether or not an authority stamps on it.

However, I believe, I am changing, to a certain extent so to say.  It seems that I have come to accept the certifications' worth, and therefore, after the training provided by Cloudera, I picked up the opportunity and went through with the rigor of examination.

Fortunately, I came through.  And, as much as I try not to showcase it, its a great feeling.  Somehow, the knowledge is vindicated, that yes, this guy knows something about hadoop and you better listen to him, :) . Funny that one has to put a badge out there to be heard.

Well, all said and done, the certification is done, courtesy the employer, who sponsored the training and of course the examination coupon that came along.  I would like to thank our trainer from cloudera, Amandeep Khurana for his depth and breadth around all things hadoop.  As much as I knew about hadoop before going in the training, those 2.5 days added precious layers to my knowledge.  Thanks Amandeep.

Thursday, January 30, 2014

Removing ports from an existing mapping - Do's and Dont's

Recently, a colleague called in for an issue that he was facing with an Informatica mapping. Let me recreate that situation here -

There is a mapping that get n ports from a source and loads m ports to a target. Standard stuff, nothing special or fishy around there. There comes a change request that says, such and such x number of columns are not required in the target, and since there would be a sure performance penalty for carrying through extra data (however small), the mapping should be changed to remove those ports from the pipeline.

What this gentleman did was to remove the connections for those ports from the source qualifier onwards. That saved him from changing and re-importing the source definition.

However, the mapping execution failed, complaining about a certain error situation. Thats when I got the call :)

whether I could discuss and fix it is something else, but what was the reason of the error - ?  

The very fact that if you are bringing in certain ports in the source qualifier, you HAVE to take them forward.  Thats a rule from informatica's side. Which means, if you are pulling up n ports from the source, you HAVE to expose all those n ports going away from the source qualifier. Or, putting it differently, in whichever way you create a set of ports in your source qualifier, all of them have to be consumed by some transformation object. Not a single one can be left unconnected, on the input side or the output side. Simple.

That was the reason, and the solution was fairly simple, to reduce his effort, either remove the ports from source qualifier as well, or just carry them forward to one transformation. and then drop them onwards.

Having said that, the very reason for which this whole change was initiated, the performance gain that would come through by not having the extra ports being carried forward, would come only when you remove the un-wanted ports from the source itself.

Thursday, January 23, 2014

Moving an ecommerce site to Amazon Web Services

This is an interesting one.. There is an e-commerce startup, run by a dear friend.  One fine day we were chatting around and he mentioned challenges with his business. Challenges included performance, scaling and cost issues about his hosting services provider.

My instant reaction was, why dont you move to aws.. and it clicked just like that.. We decided that we'd start with the free tier, with his mysql server on amazon RDS and rest of app server functionality on the micro instance. All of which is within purview of the free tier.  Using Cloudfront we'd localize the static content, to add to the performance of content delivery. Not now, but eventually, we'd start using ELB - the load balancer for distributing load across the instances we would have.

With something like aws, planning all that is really really that straightforward.  You just have to be a bit more technically oriented to think in terms of these things and you are set for good.

We are already thinking about configuring autoscaling for the services, so that running on the micro instance doesnt become a bottleneck for the customers. With that we'd hope to achieve dynamic automatic scaling up and down the infrastructure.

We have already taken the first steps, and I am helping his venture migrate to AWS, one of my dream jobs, to be able to consult around migration / integration of Traditional systems to cloud based systems.