RAC & Failsafe :: Why CPU Utilization Is High (99%) On One Server In 2 Node At Every Time
May 28, 2013why CPU utilization is high(99%) on one server in 2 node RAC at every time. On the other server it is very low.
View 9 Replieswhy CPU utilization is high(99%) on one server in 2 node RAC at every time. On the other server it is very low.
View 9 Replies I would like get more details about single node RAC.. What is single Node RAC?..
1). just configure only one server instead of two node... same RAC installation.. cluster installation and shared disk configuration in single server itself.. in future we can add node to this ...
2). in 11gR2 having new feature Single Node RAC.. Multiple instances running on same node and configure using Cluster...
is there any advantage using single node rac(other than online upgrade to full RAC).
The below query is utilizing more than 17 Gb temp space. But still it is getting failed out due to insufficient temp space. is there any way to rewrite this query to reduce the temp utilization?
SELECT T12.FRGHT_AMT_CURCY_CD,T23.LAST_UPD,T11.PAR_OU_ID,T9.MAIN_PH_NUM,T23.DISCNT_PERCENT,T23.X_ERROR_NUM,T18.ADDR,T14.X_ECO_B_END_1141,
T14.X_ECO_A_END_1141,T9.X_ECO_VALIDATION_FLG,T23.X_ECO_ERR_DESCR,T14.ASSET_NUM,T20.NAME,T23.X_ECO_REASON2,T14.X_ECO_B_END_ID,
T14.ASSET_NUM,T14.X_ECO_B_END_IWPC,T23.X_AE_CON_PH_NUM,T23.SHIP_ADDR_ID,T19.NAME,T23.X_BE_CON_LST_NAME,T23.CREATED_BY,T23.X_ECO_LOCATION,T8.LOC,
T3.MODIFICATION_NUM,T10.INTEGRATION_ID,T23.INTEGRATION_ID,T23.X_MESSAGE,T9.PR_ADDR_ID,T12.ACCNT_ID,T23.X_BEARERNO,T23.X_SUB_STATUS_CD,
[code]....
In a 3-node RAC setup; one node is showing high CPU utilization around 40~50%. The CPU utilization was less than 20% 10 days back but from 9th oldest day it jumped and consistently shows the double figure. I ran AWR reports on all three nodes and found one node with high CPU utilization and shows below tops events-
EVENT WAITS TIME(S) AVG WAIT(MS) %TOTAL CALL TIME WAIT CLASS
CPU
time 5,802 34.9
RFS
ping 15 5,118 33,671 30.8 Other
Log file sequential
read 234,831 5,036 21 30.3 System I/O
Sql*Net
more data from
client 24,1711,08745 6.5 Network
Db file sequential
read130,939 4533 2.7 User I/O
Findings:-
On AWR report(file attached) for node= sipd207; we can see that "RFS PING" wait event takes 30% of the waits and "log file sequential read" wait event takes 30% of the waits that occurs in database.
1)Are these symptoms of undersized log buffer?
2)I feel Network wait can be reduced by tweaking SDU & TDU values based on MDU.
I had one of my RAC nodes go down due to a disk failure. I have a 3 node cluster running 10.2.0.4 on Dell 610's running Windows Server 2008. I have been running AWR reports this afternoon and am seeing CPU time as my top timed event. Here is the exerpt from the report I am looking at:
Cache Sizes
~~~~~~~~~~~ Begin End
---------- ----------
Buffer Cache: 20,032M 20,032M Std Block Size: 8K
Shared Pool Size: 12,688M 12,688M Log Buffer: 6,336K
[code]...
I wanted to ask if there was anything that I could be doing to alleviate the workload on the 2 remaining nodes right now? As far as I understand it there is no way to stop users from hitting the database and without my 3rd node to load balance the CPU will continue to be pegged until the end of the day as the users are logging off.
Can a server (node) have more than one databases in RAC environment?
View 2 Replies View RelatedWe have oracle database 10g(10.2.0.4) running on AIX 5.1. we have installed 2-node RAC on another server AIX 6.1 . Now we want to migrate our single node database to new 2-node rac environment. how we can move/migrate/cluster-aware our single node database to 2-node rac?
View 3 Replies View RelatedI have 2 node 11gR1 running on Unix, I want to add 3 more RAC node to it. What are all the steps involved and should I first add 3rd node and then the 4th node or I should add both at the same time? is there any difference between these two methods?
View 1 Replies View RelatedI need to change Virtual IP for one of my RAC node as it is conflicting with another server's IP.
View 1 Replies View RelatedI am having an issue when trying to run a backup from node 207...Initially When trying to run the backup I got this error.
scrbdefrmr207:orarer 54> brbackup -u / -c force -t online -m all -p initRER_online.sap
BR0051I BRBACKUP 7.00 (40)
BR0055I Start of database backup: begnlswy.anf 2011-08-11 21.29.08
BR0484I BRBACKUP log file: /oracle/RER/sapbackup/begnlswy.anf
BR0477I Oracle pfile /oracle/RER/102_64/dbs/initRER001.ora created from spfile /oracle/RER/102_64/dbs/spfileRER001.ora
BR0280I BRBACKUP time stamp: 2011-08-11 21.29.13
BR1351I Configuration of database services:
RER001 PREF: RER001 AVAIL:
RER002 PREF: RER002 AVAIL:
DVEBMGS50_scrbdefrmr207 PREF: RER001 AVAIL: RER002
DVEBMGS50_scrbdefrmr208 PREF: RER002 AVAIL: RER001
[code]....
I have many other rac servers and sids setup this way and they all work,,I can login to sqlplus and do any changes .. no issues going into sqlplus in node 207 and node 208..I have checked the directory i.e mesg and teh files are there ..
scrbdefrmr207:orarer 56> pwd
/oracle/RER/102_64/sqlplus/mesg
scrbdefrmr207:orarer 57> ls
cpyus.msb cpyus.msg sp1us.msb sp1us.msg sp2us.msb sp2us.msg
I think the message is decieving as it complains about oracle home and sp1<lang>.msb not found ,, when i tis there.I have also checked my env variables .. they all there i.e
scrbdefrmr207:orarer 58> env |grep ORA
dbms_type=ORA
ORACLE_SID=RER001
ORACLE_HOME=/oracle/RER/102_64
ORACLE_BASE=/oracle
scrbdefrmr207:orarer 59> env | grep LD_
LD_LIBRARY_PATH=/usr/sap/RER/SYS/exe/run:/oracle/RER/102_64/lib
I an run archives log backups perfectly.. So its only when its trying to connect into the db.
How to find out which node is master node in oracle 9i RAC database.
View 7 Replies View RelatedIn case of RAC the node which is up first is master node.
In case of 4 nodes RAC master node itself is down then how to identify which node is master node out of serving nodes
I have 2 node 11gR2 RAC running on AIX 6.1, after I shutdown the database and restart crs using crsctl start crs, ASM instance comes back up but not the database and I had to start the database using srvctl, isn't that the database should come up when I start the crs?
View 4 Replies View RelatedI've plan to use "CPU Time Per User Call" metrics.
The thresholds are:
Warning: 8000
Critical: 10000
But this alarm raise every minute.
I think it's too low.But which is the correct value to identify performance problems?
What could be the reason for Such Errors.?OCR Integrity was Successful on one node and on the cloned node it was failed.
View 8 Replies View RelatedI have a production database with 2 node Physical RAC ENV on two separate machines , and we are going to enable audit trail parameter to DB values to capture all failed login attempts on a database.
we have done this on our testing ENV but dont know the procedure to deploy this steps on 2 node RAC ENV . I have done the below steps on our standalone testing ENV
1.Log in as SYS and connect with the SYSDBA privilege.
oUser Name: SYS
oPassword: Enter your password.
oConnect As: SYSDBA
2.To enable audit on DB use below command
SQL>alter system set audit_trail=DB scope=SPFILE;
3.Then restart the database.
4.After successful startup use the below command to capture log
SQL> audit create session whenever not successful;
steps to enable audit_trail to capture failed login attempts on two node RAC ENV.
here we have san server and rac database two node database.i want to know how can i create directory on rac database. i know how to create directory on single database but i do not know how can i create directory on rac database.
View 3 Replies View Relatedi am configuring asm on clusterware when i create instance on node1 and node 2 a message screen appear " Cant start asm instance on node 2)When i start asm instance manually on node 2 i found the below error in alert log file.
ORA-27508 IPC error sending a message
ORA-27300 OS system dependent operation:send msg failed with status: 101
ORA-27301 OS failure message: Network is unreachable
ORA-27302 failure occurred at: sskgxpsnd1
during installtion of Oracle RAC 11g r1 on Linux using VMware server, all the pre cluster installation was successful, and root.sh on node 1 was successful too, but whenever I run the root.sh script on the second node I get error message "Failure at final check of oracle CRS stack. 10"
View 5 Replies View RelatedI have installed Oracle 10g RAC crs and asm on CentOS release 5.4. When I am rebooting the nodes and starting crs manually All the services on both nodes starting successfully.
---------------------------------------------------------
[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
[code].....
But After a Few time/min few services goes down automatically and its show something like
---------------------------------------------------------
[root@rac1 ~]# crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE UNKNOWN rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
[code].....
I have few doubts in rac environment
1.In 2 node rac while adding datafile to tablespace if you forget to mention '+'then what will happen whether it is going to be create or it throws an error if it creates where exactly located and other node users how to work on that tablespace .what all steps to perform that datafile is useful for all node users.
2. In Rac environment how to check how many sessions connected to particular node.
3)In Rac any node is evicted due to network failure then after we rebuild the network .Is there any steps to do manually to access the failure node after rebuilding the network or it will automatically available in cluster group which service is perform this activity.
4.While configuring cluster ware you choose voting disk and ocr disk location and which redundancy you will choose suppose if you go for normal redundancy how many disks you can select for each file either one or two?
I have implemented Rac One node with two machine(dbtest01,dbtest02) in 11.2.0.3 Redhat 5.3 one for online another one for offline. i have created TAF also.
dbtest01 is online and create one query
DECLARE
i number(36):=1;
begin
while(i<10000000)
loop
dbms_output.put_line(i);
i:=i+1;
end loop;
end;
i executed above query for test and i made relocate database using srvctl utility to dbtest02 machine.
session is relocated successfully with new session id...above query getting stop in 25000 itself. i want to make it continue that process..also i tried to find Omotion utility in all the path.. couldn't find it... where will it be.. i want to relocate all the session without losing work.
from application i am getting disconnect after relocation.. application server need to restart to get work again...
We have 5 node RAC database. I want to user 2 node for OLTP and 3 for DSS system. Guide now to create service for the requirement.
View 10 Replies View Relatedwe encounter a problem with a 2-nodes RAC with 11gR2 SE : Database RAC is correctly running but we encounter problems using RAC TNS entry, obtaining the message ORA-01033.We can access to the DB using a TNS entry using each one of IP addresses but when we use the SCAN address, it fails. 2 weeks ago RAC2 node could connect perfectly with the SCAN address but RAC1 (used as a client) obtained error message almost 30% of tries.
We made a reboot of RAC1 node and now SCAN address works correctly from RAC1 and RAC2 (used as a client) fails systematically with SCAN address but works perfectly with direct access to the database.On both nodes, SCAN address is perfectly resolved by DNS server, cluster seems correct, listeners seems correct...We have opened an SR on Metalink 4 weeks ago but we don't progress.
I am configuring two node RAC on oracle virtual box. Did its mandatory to configure DNS on both nodes before Rac installation? Whats the purpose of DNS in RAC. And if its mandatory did i have to configure DNS on each node separately?
View 1 Replies View RelatedI have the following problem on a 10.2.0.4 Oracle on AIX 1.6...
I have 2 RAC nodes as the node # 2 goes down, the VIP migrated correctly to the node # 1, but the listener # 1 do not see the change, and just resolve to the local ip.
The same does not apply in the reverse case, when it drops the node N 1, the listener attends to the fallen VIP node and migrated.The configuration is correct and identical in both nodes, according to the documentation, but I can not find a reasonable answer.
IP node # 1.
scm1or01 192.68.60.11
scm1or01-VIP 192.68.60.13
IP node # 2
scm1or02 192.168.60.12
scm1or01-VIP 192.168.60.14
Below I write as I detect the problem: when scm1or02 is down:
scm1or01:/cots/oracle/app/oracle/product/10.2.0/db_1/network/admin> lsnrctl status
LSNRCTL for IBM/AIX RISC System/6000: Version 10.2.0.4.0 - Production on 11-APR-2012 19:56:46
Copyright © 1991, 2007, Oracle. All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
[code]...
Also the listener.ora in scm1or01 starts how LISTENER, but the crsstat says:
scm1or01:/cots/oracle/app/oracle/product/10.2.0/db_1/network/admin>crsstat listener_scm1oro1 ONLINE
I do not understand the reason, I've rebuilt the configuration of the listener, and netmgr netca tools, but the problem persists.
I am facing a very strange issue with one of our Oracle query. The query is usually completes in a minute or two. Even the execution plan of the query is good and it works perfect most of the times, as expected. The query fetches about 1000-2000 records each day.
But on a given day, the query takes about 30-40 mins to execute completely. Upon checking the load on DB server, there are no other processes running which can impact the run time of this query. Moreover, the record counts fetched are almost same as compared to other days. There is no pattern observed as that this phenomenon occurs. it all happens once in a while.
Configuration is Oracle 10g with RAC environment on LINUX
I have two node RAC database running on linux machine (Oracle10G SE), our requirnment is to install new oracle binaries for Oracle10G EE on diffrent home location.after install Oracle10G EE binaries on new oracle home , i want to point our existing database to new Oracle10G EE home to adotp a features of Oracle10G EE features.
But at the same time i dont want to deinstall our Oracle10G SE binaries.So it is possible to rollback our process to point our database back to Oracle10G SE from Oracle10G EE in case anything going wrong.
We have to configure Dataguard for our 24x7 critical banking 2 Node RAC database(10.2.0.4). Before proceeding with the configuration we have to make sure what steps we should follow to have minimum or no downtime.
1) Document covering DG setup in RAC environment.
2) We have to perform switchover as well so need its steps too. Either the normal switchover steps would be used or have to stop/start rac services as well.
when running the root.sh on second node the following error occured
adding daemons to inittab
expecting the crs daemons to be up within 600 seconds
failure at final check of oracle CRS stack. 10.