Bulk load data from file to oracle tables

I was trying to analyze the performance test results of a customer facing e-commerce application. Number of concurrent users were close to 700 and the load testing tool (HP performance center) picked these users from a range of 100K records stored in flat files. My objective was to find out the number of orders placed by each of these users and confirm that they are uniformly distributed.

To begin with, I tried opening the flat files (there were many of them, each script had its own flat file for storing user ids) and noted down the range of user ids for comparing the orders. I soon realized that the user ids stored in flat file were not ordered and my range logic is not going to work. That's when I thought about loading these data into a database table so that analysis is much simpler.

I tried creating INSERT scripts for each record using a macro and tried running the script for the first 65 K records. The tool I was using threw out of memory error and I needed a better solution.

Oracle SQL* Loader
I googled a bit and found that Oracle SQL*Loader can do this easily, here is one blog which explains how - http://www.thegeekstuff.com/2012/06/oracle-sqlldr/

Key part of this is the creation of a control file where the input file, target table and other config parameters are specified.

My control file looks like below - save this as sqlldr-tusertest.ctl

load data
infile '/home/kiran/userids.txt'
into table T_USERTEST
fields terminated by "|"
Now for loading this using SQL*Loader, execute below command

$sqlldr <dbuserid>/<dbpassword>@<dbhost>:<port>/<sid> control=/home/kiran/sqlldr-tusertest.ctl log=/home/kiran/sqlldr.log

Profiling WebSphere Commerce 7.x using VisualVM

Enable JMX client access to local JVM

1. Navigate to Servers -> Server Types -> WebSphere application servers -> server1 -> Process definition -> Java Virtual Machine and append the following line to Generic JVM Argument

-Djavax.management.builder.initial= -Dcom.sun.management.jmxremote
2. Uncomment the following three lines in file /SDP/runtimes/base_v7/java/jre/lib/management/management.properties

com.sun.management.jmxremote.port=3333
com.sun.management.jmxremote.authenticate=false
com.sun.management.jmxremote.ssl=false
com.sun.management.jmxremote.local.only=false

3. Restart WebSphere Application Server

Connect Visual VM to WAS

4. Run visualvm.exe from bin directory and add JMX connection to WebSphere Application Server. Enter localhost:3333 in the connection text box, assuming the server is running on the same machine at port 3333. 3333 is the JMX port configured in management.properties (See step 2)

Deploying Rational Application Developer installation to multiple desktops

My team uses a specific version of Rational Application Developer, 7.5.5.4 with a specific version of WAS test server, 7.0.0.19.  One challenge that I face every time I upgrade RAD and the test server is applying the upgrade on all my team members workstations. Installing the fixes on each machine takes a lot of time. Sometimes the installation fails halfway, so you have to start from scratch again.


If you are part of a large organization often executing large projects using Rational Application Developer, you would have already faced this issue. The severity of the problem multiplies when you are using products like WebSphere Commerce Developer which requires a set of additional fix packs/feature packs to be installed on top of the base Rational Application Developer.

One solution that I have tried successfully over the years is copying the installed directories of RAD to all the machines. Here are the steps :-

1. Copy RAD installation directories. For e.g., if you have installed RAD 7.5x under D:\IBM\SDP and D:\IBM\SDPShared, copy these folders to the target machine. Make sure that you maintain the same directory structure on the target machine.
2. Copy Installation Manager under the same path on the target machine.
3. Copy C:\ProgramData\IBM\Installation Manager to the target machine.

Note: Please make sure that you have purchased enough number of RAD licenses for development. This approach should not be used for pirating RAD.

Permission error while starting Rational Application Developer (RAD) on Windows 7

Below error is shown in the logs while starting Rational Application Developer 7.5x on a Windows 7 machine. If you are starting RAD through a shortcut, right click and select "Run As Administrator" - this will resolve the issue.

ADMU0116I: Tool information is being logged in file D:\Apps\IBM\WCD7\WCDE_E~1\wasprofile\logs\server1\startServer.log java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:599) at com.ibm.ws.bootstrap.WSLauncher.main(WSLauncher.java:260) Caused by: com.ibm.websphere.management.exception.AdminException: ADMU7707E: Failed while trying to determine the Windows Service name for server: server1; probable error executing WASService.exe: com.ibm.websphere.management.exception.AdminException: ADMU7709E: Unexpected exception while processing server: server1; exception = java.io.IOException: Cannot run program "D:\Apps\IBM\RAD7\SDP\runtimes\base_v7\bin\WASService.exe": CreateProcess error=740, The requested operation requires elevation. at com.ibm.ws.management.tools.WindowsService.getWindowsServiceNameForServer(WindowsService.java:221) at com.ibm.ws.management.tools.WindowsService.(WindowsService.java:154) at com.ibm.ws.management.tools.WindowsService.getWindowsServiceInstance(WindowsService.java:139) at com.ibm.ws.management.tools.WsServerController.getWindowsServiceIfAvailable(WsServerController.java:161) at com.ibm.ws.management.tools.WsServerLauncher.getWindowsServiceIfAvailable(WsServerLauncher.java:157) at com.ibm.ws.management.tools.WsServerController.executeUtilityOnWindows(WsServerController.java:129) at com.ibm.ws.management.tools.WsServerLauncher.main(WsServerLauncher.java:138) ... 5 more

Installing MySQL on Ubuntu without apt-get

If you are trying to install MySQL on a box that does not have internet connection, installation will be challenging.

This post explains how MySQL can be installed and configured by downloading individual .deb files.

1. Identify your Ubuntu distribution name/version and architecture 

For checking your Ubuntu distribution name,

root@localhost:$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.04
DISTRIB_CODENAME=lucid
DISTRIB_DESCRIPTION="Ubuntu 10.04.3 LTS"

For checking the architecture,use "uname -m" or arch command

root@localhost:/etc/mysql$ uname -m
x86_64

2. Download below packages from packages.ubuntu.com (links below correspond to amd64 architecture and lucid distribution, you will have to change this according to your configuration)

http://packages.ubuntu.com/lucid/all/libnet-daemon-perl/download
http://packages.ubuntu.com/lucid/all/libplrpc-perl/download
http://packages.ubuntu.com/lucid/amd64/libdbi-perl/download
http://packages.ubuntu.com/lucid/amd64/mysql-common/download
http://packages.ubuntu.com/lucid/amd64/libmysqlclient16/download
http://packages.ubuntu.com/lucid/amd64/libdbd-mysql-perl/download
http://packages.ubuntu.com/lucid-updates/amd64/mysql-client-core-5.1/download
http://packages.ubuntu.com/lucid-updates/amd64/mysql-client-5.1/download
http://packages.ubuntu.com/lucid-updates/amd64/mysql-server-core-5.1/download
http://packages.ubuntu.com/lucid-updates/amd64/mysql-server-5.1/download
http://packages.ubuntu.com/lucid/all/libhtml-template-perl/download
http://packages.ubuntu.com/lucid-updates/all/mysql-server/download


3. Install downloaded packages in the same order

sudo dpkg -i libnet-daemon-perl_0.43-1_all.deb
sudo dpkg -i libplrpc-perl_0.2020-2_all.deb
sudo dpkg -i libdbi-perl_1.609-1build1_amd64.deb
sudo dpkg -i mysql-common_5.1.41-3ubuntu12.10_all.deb
sudo dpkg -i libmysqlclient16_5.1.41-3ubuntu12.10_amd64.deb
sudo dpkg -i libdbd-mysql-perl_4.012-1ubuntu1_amd64.deb
sudo dpkg -i mysql-client-core-5.1_5.1.41-3ubuntu12.10_amd64.deb
sudo dpkg -i mysql-client-5.1_5.1.41-3ubuntu12.10_amd64.deb
sudo dpkg -i mysql-server-core-5.1_5.1.41-3ubuntu12.10_amd64.deb
sudo dpkg -i mysql-server_5.1.41-3ubuntu12.10_all.deb
sudo dpkg -i libhtml-template-perl_2.9-1_all.deb
sudo dpkg -i mysql-server-5.1_5.1.41-3ubuntu12.10_amd64.deb

During mysql-server-5.1 installation, you will be asked to provide the root user name and password. Make sure you remember what you provide here, without the root id and password you will not be able to connect to MySQL server.

To check whether MySQL is installed correctly, run the command

sudo netstat -tap | grep mysql

Reference 1:- http://askubuntu.com/questions/29533/install-mysql-without-apt-get

Syntax highlighting on blogger

I always wanted a 'wiki' like functionality on blogger so that my technical posts are more readable (Highlighting commands, posting source code with line numbers etc..)

Some good options are :-

1. Using JEdit and Code2Html plugin - http://thinktibits.blogspot.com/2011/04/best-syntax-highlighter-for-bloggerpart.html

2. Alex Gorbatchev's Syntax Highlighter - http://alexgorbatchev.com/SyntaxHighlighter. Here is an example of how it is used on blogger - http://abhisanoujam.blogspot.com/2008/12/blogger-syntax-highlighting.html

3. Using google code prettify - http://code.google.com/p/google-code-prettify/. See the README file for implementation details. Here is a sample blog that uses google code preffify - http://blog.js-development.com/2008/01/finally-syntax-highlighting-on-my-blog.html

If you just need a simple highlighting mechanism, try editing the design template as explained below.

1. Open the template html (Template -> Edit HTML)

2. Look out for </b:skin> and add below piece of code as shown :-

code {background:whitesmoke; font-family: Arial; display:block; border:1px solid #999999; padding:10px;}

]]></b:skin>

3. Enclose the text that you want to highlight in <code></code> block

<code>
Your text here...
</code>

Rpm installation - where can I find the installed files?

I recently installed ganglia to monitor the performance of my hadoop cluster running on fedora 14. Ganglia installation mainly consists of gmetad, gmond and the php web front end (ganglia-web). I installed these packages using the software update manager that comes with Fedora 14. After the installation was completed, I was trying to copy the gangila web files to my apache web root (/var/www/html) and I just could not locate the php web front end files!!

After a bit of googling I found out the command that I was searching for - rpm -ql <package name>

rpm -ql ganglia-web

SQL1092N "USER" does not have the authority to perform the requested command

The error "SQL1092N <USER> does not have the authority to perform the requested command" usually comes when a domain user tries to perform a database operation that requires administrative privileges.

To resolve this, perform below steps (Windows OS):-

1. Execute "db2set DB2_GRP_LOOKUP=LOCAL" from db2 command editor
2. Create a new group called DB2ADMNS (Right click My Computer -> Manage -> Computer Management ->  System Tools -> Local Users and Groups -> Groups
3. Add the domain user to the DB2ADMNS group
4. Execute "db2 update dbm cfg using sysadm_group DB2ADMNS" from db2 command editor
5. Execute "db2stop"
6. Execute "db2start"