Sunday, March 13, 2011

convert Informatica Session Logs to text/xml format for out of tool readability

There was a situation recently when the infa repository was not able to point us to the old session logs.
However, the file system still had those files.

But the session log files are in binary format by default. If you didnt ask for backward compatible session log files, you'd get a binary format of session log file on file system. This is done to allow better importability of the session log files for infa support guys.

So, I had a few log files in binary format, and needed to analyze them.
Informatica provides a subcommand for infacmd to achieve that conversion.

The convertLogFile subcommand takes three parameters. Syntax is as follows -


convertLogFile <-InputFile|-in> input_file_name
                 [<-Format|-fm> format_TEXT_XML]
                 [<-OutputFile|-lo> output_file_name]


so, when you launch this, you can specify the input file to be converted, the format as TEXT or XML and the output file that you'd want as a result of conversion.

An example call would look like this -  (expecting server on unix)

infacmd.sh convertLogFile -in /path/to/binary/format/sesslog/file -fm TEXT -lo /my/home/text/format/sesslog/file



Monday, February 21, 2011

Just found out about this amazing thing...

A research initiative at Stanford University, Data Wrangler.. Wonderfully helping for analysts.

Try a demo video here -




And read more about it on http://vis.stanford.edu/wrangler

Wrangler Demo Video from Stanford Visualization Group on Vimeo.

Saturday, February 19, 2011

Exadata - is it really worth the hype

Well, I am not going to try to answer that, rather, more on the question side...
Recently one of my projects moved to exadata device. There was so much talk around that, the queries and db processes need not be looked into, exadata will take care of them already.

However, two things happened.. first, there was a technical talk on the device's configuration. The device turned out to be a mammoth piece of hardware. In nutshell, its a 8 node cluster, each node having 4 cpu's.  Each CPU has around 2-4 GB of RAM. Then there is this high speed secondary storage which can hold a lot of cache.

The nodes are interconnected using a special switch which can transfer data faster than Gigabit networks.

With such hardware configuration, any software can claim the kind of performance gain they claim. Not to undermine the performance gains, I just want to say that the hype around the out of the world performance gains, is actually the result of better hardware, not really revolutionary software.

I, personally was expecting something of that type from Oracle, since they lack in that area. Except Teradata, there is almost no player who delivers that kind of Data Warehouse architecture and performance. and I was hoping that Oracle would do something around there and bring out something.

And, the second thing, one of the processes tried to load data to an exadata instance using informatica. Initially we left things at default so that exadata can tune it itself and we should not force anything.  However, there too, exadata failed big time and couldnt put in any perf gain. At the end, all the tuning had to be done by us only.

So, the other claim of exadata regarding intelligence to pick up processes and fix them on its own also went down for us.

Though I agree that its rather new for its own evolution, i believe oracle marketing should be doing a better job.:)

The wavy Tricolor



The wavy Tricolor, originally uploaded by s_raghu20.
This was captured at the Aero India 2011, the bi-annual air show held in Bangalore. It was my first ever attempt to watch and shoot these metal birds.

I have realized that the sharpness of some of the shots is less than ideal, perhaps a function of their speeds and movement. Also a function of my reflexes and my equipment.

Wednesday, December 1, 2010

Rejecting records in Fact table loads - Informatica

In some development environments, you dont have all the required dimension data and as a result, your fact loading mapping's test runs go for a toss. The mapping wont be able to load anything (reject everything) since some or other foreign key would be missing for each row.

In other words, only those records would be loaded for which ALL the foreign key constraints would be satisfied. However, in Production environment, this would almost never happen. Or even if its the case, we'd actually want those rows to be rejected.

This can work out to be a serious impediment to development/unit testing. It prevents the developer from seeing whether or not his his mapping is behaving appropriately for the happy flow functionality.

One way of achieving this can be to work using a Mapping Variable indicating the Environment. The developer can run using a value like 'D' or something, indicating a different environment than Production, with 'P' (production) being the default value.

Now in the mapping, just about where you decide to reject a record based on different type of conditions, you could put an AND condition involving this mapping variable, e.g. ....AND $$MAPP_ENV = 'P'

Now, the expression would return true only in the Production environment, and therefore would work as expected. In Dev though, this expression would return false and would not reject that row.  

Now, to be able to satisfy the db constraints for the fact table so that the row is actually inserted, you'd need to use some placeholder convention. One of the approaches can be to use an outlier value as the foreign key value.  For Example, for customer id, keep a -1 in dimension table, meaning "Undefined".  And, in all such dev cases, send -1 to the fact. 

It would serve both the purposes, tell your fact table that there is something diff about that row, and still inserting a row in there, so that the testing for the rest of the columns is not stopped because of one foreign key missing out.

Wednesday, November 17, 2010

Setting a useful command prompt in Unix

I just came across a unix system at my workplace that had a static prompt set. Basically, the prompt was just the shell executable's name and version, more like 

bash-3.2 $

Well, this kind of prompt has many drawbacks, some of them i'd list here - 
1. You never know (just like that) where you are in the file system. When you are dealing with multi-directory situations, you might want to stop typing pwd to figure out ur current location.

2. You never know by what user you are logged in (again, just by looking at the prompt). You'd have to run a whoami to figure that out.

3. More importantly, if you are dealing with multiple systems, this one's the most killer.  You never know to what system you are logged in right now. you'd have to issue a hostname command.

Well, there might be, and for sure there are many other, consequences of having such a cryptic command prompt.  And therefore, my favourite, to have a command prompt, that displays at least these three things, always, dynamically...

Something like, 

raghav@deskubuntu:/homes/raghav/rails $ 

wherein, I am always aware of the three things mentioned earlier. This is very very useful when you are dealing with multiple systems and you have multiple users who are configured to run different types of processes.  For example, an oracle user who is supposed to be owner of oracle processes, and an informatica user which is supposed to own everything linked to informatica, and then a connect direct user which owns the CD processes, which receives files coming in from some other system.

With this kind of system, and your own user id to log in to the system, you'd better be careful which processes you are looking at / launching and by what user.  Its really very very important.

When and if you are dealing with a multiple system scenario, like dev / test / acceptance / production, you'd be better advised to use something like this only.

the magic command to do that is by setting appropriate flags and text in a environment variable called PS1.

Just set PS1 to your .profile or .bashrc (depending on your environment) file and you are set .

The example prompt that I mentioned can be achieved by saying - 

export PS1="\\u@\\h:\\w \\$ "

There are many more possibilities that go with special meanings for PS1 variable. Read some of them here - 

    * \$ : if the effective UID is 0, a #, otherwise a $
    * \[ : begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt
    * \\ : a backslash
    * \] : end a sequence of non-printing characters
    * \a : an ASCII bell character (07)
    * \@ : the current time in 12-hour am/pm format
    * \A : the current time in 24-hour HH:MM format
    * \d : the date in "Weekday Month Date" format (e.g., "Tue May 26")
    * \D{format} : the format is passed to strftime(3) and the result is inserted into the prompt string; an empty format results in a locale-specific time representation. The braces are required
    * \e : an ASCII escape character (033)
    * \H : the hostname
    * \h : the hostname up to the first '.'
    * \j : the number of jobs currently managed by the shell
    * \l : the basename of the shell’s terminal device name
    * \n : newline
    * \nnn : the character corresponding to the octal number nnn
    * \r : carriage return
    * \T : the current time in 12-hour HH:MM:SS format
    * \t : the current time in 24-hour HH:MM:SS format
    * \u : the username of the current user
    * \s : the name of the shell, the basename of $0 (the portion following the final slash)
    * \V : the release of bash, version + patch level (e.g., 2.00.0)
    * \v : the version of bash (e.g., 2.00)
    * \W : the basename of the current working directory
    * \w : the current working directory