Out Of Process Memory Error When Upgrading From Oracle 9.2.0.8 To 11.2.0.2

I got the following error when upgrading from Oracle 9.2.0.8 to Oracle 11.2.0.2 version using catupgrd.sql script.

=================================================================
CREATE OR REPLACE PACKAGE BODY kupm$mcp wrapped
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ERROR:
ORA-03114: not connected to ORACLE

SP2-1519: Unable to write to the error log table sys.registry$error
ORA-04030: out of process memory when trying to allocate 8392728 bytes (pga
heap,redo read buffer)
ORA-04030: out of process memory when trying to allocate 8392728 bytes (pga
heap,redo read buffer)
ORA-04030: out of process memory when trying to allocate 8168 bytes
(callheap,kcbtmal allocation)
Process ID: 81984
Session ID: 1 Serial number: 5

==========================================================

My machine environment is:

UNIX AIX 5.3 level 12
Oracle 11.2.0.2
RAM size 2GB
ulimit settings:
> ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) unlimited
coredump(blocks) 2
nofiles(descriptors) unlimited

Currently i tried to increase SGA & PGA to possible extent, but still getting this error? Is there any change required in ulimit, RAM etc?