27 March, 2017

12cR1 RAC Posts -- 8d : Registering the two databases in DataGuard Broker

Continuing my series on setting up a non-RAC, FileSystem Standby for a RAC, ASM Database....

On the Standby:

[oracle@oem132 ~]$ . oraenv
ORACLE_SID = [oracle] ? STBY
The Oracle base has been set to /u01/app/oracle
[oracle@oem132 ~]$ lsnrctl start

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 27-MAR-2017 22:24:54

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /u01/app/oracle/product/12.1.0/dbhome_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production
System parameter file is /u01/app/oracle/product/12.1.0/dbhome_1/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/oem132/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oem132.racattack)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oem132.racattack)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                27-MAR-2017 22:24:58
Uptime                    0 days 0 hr. 0 min. 2 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/12.1.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/oem132/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oem132.racattack)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Services Summary...
Service "STBY" has 1 instance(s).
  Instance "STBY", status UNKNOWN, has 2 handler(s) for this service...
The command completed successfully
[oracle@oem132 ~]$ sqlplus '/ as sysdba'

SQL*Plus: Release 12.1.0.2.0 Production on Mon Mar 27 22:25:07 2017

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount
ORACLE instance started.

Total System Global Area  838860800 bytes
Fixed Size                  2929936 bytes
Variable Size             230689520 bytes
Database Buffers          599785472 bytes
Redo Buffers                5455872 bytes
SQL> alter database mount standby database;

Database altered.

SQL>
SQL> show parameter dg_broker

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
dg_broker_config_file1               string      /u01/app/oracle/product/12.1.0
                                                 /dbhome_1/dbs/dr1STBY.dat
dg_broker_config_file2               string      /u01/app/oracle/product/12.1.0
                                                 /dbhome_1/dbs/dr2STBY.dat
dg_broker_start                      boolean     FALSE
SQL>
SQL> alter system set dg_broker_Start=TRUE;

System altered.

SQL>


On the first instance of the Primary :

SQL> show parameter dg_broker

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
dg_broker_config_file1               string      /u01/app/oracle/product/12.1.0
                                                 /dbhome_1/dbs/dr1RAC.dat
dg_broker_config_file2               string      /u01/app/oracle/product/12.1.0
                                                 /dbhome_1/dbs/dr2RAC.dat
dg_broker_start                      boolean     FALSE
SQL> alter system set dg_broker_config_file1='+DATA/RAC/dgbroker1.dat' sid='*' scope=BOTH;

System altered.

SQL> alter system set dg_broker_config_file2='+FRA/RAC/dgbroker2.dat' sid='*' scope=BOTH;

System altered.

SQL> alter system set dg_broker_start=TRUE;

System altered.

SQL>

[oracle@collabn1 ~]$ tnsping RAC1

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 27-MAR-2017 22:37:41

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = collabn1)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RAC.racattack)))
OK (0 msec)
[oracle@collabn1 ~]$ sqlplus sys/racattack@STBY

SQL*Plus: Release 12.1.0.2.0 Production on Mon Mar 27 22:37:51 2017

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

ERROR:
ORA-01033: ORACLE initialization or shutdown in progress
Process ID: 0
Session ID: 0 Serial number: 0


Enter user-name: sys/racattack@STBY as SYSDBA

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
[oracle@collabn1 ~]$ tnsping RAC1

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 27-MAR-2017 22:38:07

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = collabn1)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RAC.racattack)))
OK (0 msec)
[oracle@collabn1 ~]$ tnsping STBY

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 27-MAR-2017 22:38:10

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = oem132)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = STBY)))
OK (0 msec)
[oracle@collabn1 ~]$ dgmgrl
DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production

Copyright (c) 2000, 2013, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/racattack@RAC1
Connected as SYSDBA.
DGMGRL> create configuration RAC as primary database is RAC connect identifier is RAC1;
Configuration "rac" created with primary database "rac"
DGMGRL> add database stby as connect identifier is stby maintained as physical;
Database "stby" added
DGMGRL> enable configuration;
Enabled.
DGMGRL> show configuration;

Configuration - rac

  Protection Mode: MaxPerformance
  Members:
  rac  - Primary database
    stby - Physical standby database
      Error: ORA-16664: unable to receive the result from a database

Fast-Start Failover: DISABLED

Configuration Status:
ERROR   (status updated 7 seconds ago)

DGMGRL>


Errors in the STBY DataGuard Broker Logfile drcSTBY.log :

03/27/2017 22:45:12
Warning: Property 'StandbyFileManagement' has inconsistent values:METADATA='MANUAL', SPFILE='', DATABASE='MANUAL'
Warning: Property 'ArchiveLagTarget' has inconsistent values:METADATA='0', SPFILE='', DATABASE='0'
Warning: Property 'LogArchiveMaxProcesses' has inconsistent values:METADATA='4', SPFILE='', DATABASE='4'
Warning: Property 'LogArchiveMinSucceedDest' has inconsistent values:METADATA='1', SPFILE='', DATABASE='1'
SPFILE is missing value for property 'LogArchiveTrace' with sid='STBY'
Warning: Property 'LogArchiveTrace' has inconsistent values:METADATA='0', SPFILE='(missing)', DATABASE='0'
SPFILE is missing value for property 'LogArchiveFormat' with sid='STBY'
Warning: Property 'LogArchiveFormat' has inconsistent values:METADATA='%t_%s_%r.dbf', SPFILE='(missing)', DATABASE='%t_%s_%r.dbf'
Failed to send message to site rac. Error code is ORA-16501.

Errors in the RAC1 DataGuard Broker Logfile drcRAC1.log :

Site stby returned ORA-16664.
Data Guard Broker Status Summary:
  Type                        Name                             Severity  Status
  Configuration               rac                               Warning  ORA-16607
  Primary Database            rac                               Success  ORA-00000
  Physical Standby Database   stby                                Error  ORA-16664


Actions taken on the Standby :

SQL> show parameter standby

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
standby_archive_dest                 string      ?/dbs/arch
standby_file_management              string      MANUAL
SQL> alter system set standby_file_management='AUTO' scope=BOTH;

System altered.

SQL> alter system set archive_lag_target=0 scope=BOTH;

System altered.

SQL> alte system set log_archive_max_processes=4 scope=BOTH;
SP2-0734: unknown command beginning "alte syste..." - rest of line ignored.
SQL> alter system set log_archive_max_processes=4 scope=BOTH;

System altered.

SQL> alter system set log_archive_min_succeed_dest=1 scope=BOTH;

System altered.

SQL> alter system set log_archive_trace=0 scope=BOTH;

System altered.

SQL> alter system set log_archive_format='%t_%s_%r.dbf' scope=BOTH;
alter system set log_archive_format='%t_%s_%r.dbf' scope=BOTH
                 *
ERROR at line 1:
ORA-02095: specified initialization parameter cannot be modified


SQL>


I also had to edit the TNSNAMES.ORA on the STBY server for the correct *VIP* entry (not the physical hostname) for the RAC1 instance.

RAC1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.78.61)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = RAC.racattack)
    )
  )

RAC1_DGB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.78.61)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = RAC_DGMGRL)
    )
  )


Thereafter, it worked.  Note : You sometimes have to wait a while to enter the SHOW CONFIGURATION, the first result may show an error.  So I entered it twice.

DGMGRL> disable configuration;
Disabled.
DGMGRL> enable configuration;
Enabled.
DGMGRL> show configuration;

Configuration - rac

  Protection Mode: MaxPerformance
  Members:
  rac  - Primary database
    stby - Physical standby database
      Error: ORA-16664: unable to receive the result from a database

Fast-Start Failover: DISABLED

Configuration Status:
ERROR   (status updated 125 seconds ago)

DGMGRL> show configuration;

Configuration - rac

  Protection Mode: MaxPerformance
  Members:
  rac  - Primary database
    stby - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 14 seconds ago)

DGMGRL>


These are the drcSTBY.log entries when the TNSNAMES.ORA for RAC1 was incorrect on the Standby Server :

Failed to connect to remote database rac. Error is ORA-12541
Failed to send message to site rac. Error code is ORA-12541.


This is the drcSTBY.log entry on success :

03/27/2017 23:00:51
Apply Instance for Database stby set to STBY


And the entry in drcRAC1.log on success :

03/27/2017 23:00:10
DISABLE CONFIGURATION
03/27/2017 23:00:46
ENABLE CONFIGURATION
03/27/2017 23:00:57
EDIT DATABASE stby SET PROPERTY ActualApplyInstance = STBY
03/27/2017 23:01:00
Apply Instance for database stby is STBY


Setting Maximum Performance mode (command issued on the *Primary*) :

SQL> alter database set standby to maximize performance;

Database altered.

SQL>

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MaxPerformance;
Succeeded.
DGMGRL> show configuration;

Configuration - rac

  Protection Mode: MaxPerformance
  Members:
  rac  - Primary database
    stby - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 34 seconds ago)

DGMGRL>
DGMGRL> show configuration verbose

Configuration - rac

  Protection Mode: MaxPerformance
  Members:
  rac  - Primary database
    stby - Physical standby database

  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    TraceLevel                      = 'USER'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    ObserverReconnect               = '0'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'
    ObserverOverride                = 'FALSE'
    ExternalDestination1            = ''
    ExternalDestination2            = ''
    PrimaryLostWriteAction          = 'CONTINUE'

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL>


Will be testing Redo Transmission and Apply later.
.
.
.

13 March, 2017

1.5million PageViews

My Oracle Blog now has had 1.5million PageViews.


The 1million PageViews mark was hit in March 2015.
.
.
.

12cR1 RAC Posts -- 8c : Ignorable "Errors" during the DUPLICATE

In yesterday's post, I had shown a DUPLICATE DATABASE from RAC-ASM to SingleInstance-FileSystem.

During the course of the DUPLICATE DATABASE run, the Standby alert log seemingly reported errors.  I chose to ignore the "errors" as I know the DUPLICATE was running successfully.

For your reference, here are some of the good messages :

Sun Mar 12 23:17:55 2017
ALTER SYSTEM SET control_files='/u01/app/oracle/oradata/STBY/controlfile/o1_mf_ddbs0p3w_.ctl','/u01/app/oracle/fast_recovery_area/STBY/controlfile/o1_mf_ddbs0p40_.ctl' COMMENT='Set by RMAN' SCOPE=SPFILE;
Sun Mar 12 23:17:59 2017
ALTER SYSTEM SET control_files='/u01/app/oracle/oradata/STBY/controlfile/o1_mf_ddbs0p3w_.ctl','/u01/app/oracle/fast_recovery_area/STBY/controlfile/o1_mf_ddbs0p40_.ctl' COMMENT='Set by RMAN' SCOPE=SPFILE;

Sun Mar 12 23:21:41 2017
Switch of datafile 1 complete to datafile copy
  checkpoint is 3315193
Switch of datafile 3 complete to datafile copy
  checkpoint is 3315211
Switch of datafile 4 complete to datafile copy
  checkpoint is 3315270
Switch of datafile 5 complete to datafile copy
  checkpoint is 1754712
Switch of datafile 6 complete to datafile copy
  checkpoint is 3315280
Switch of datafile 7 complete to datafile copy
  checkpoint is 1754712
Switch of datafile 8 complete to datafile copy
  checkpoint is 3315276
Switch of datafile 9 complete to datafile copy
  checkpoint is 2978862
Switch of datafile 10 complete to datafile copy
  checkpoint is 2978862
Switch of datafile 11 complete to datafile copy
  checkpoint is 2978862


And here are the errors (about the online redo logs) continuously being reported.  These are messages which I ignored because the Standby has no online redo logs yet.

Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_1.257.931825281'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/STBY/onlinelog/group_1.283.931825279'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_1.257.931825281'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/STBY/onlinelog/group_1.283.931825279'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_2.258.931825287'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/STBY/onlinelog/group_2.284.931825283'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_2.258.931825287'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/STBY/onlinelog/group_2.284.931825283'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 3 of thread 2
ORA-00312: online log 3 thread 2: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_3.259.931826417'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 3 thread 2: '/u01/app/oracle/oradata/STBY/onlinelog/group_3.290.931826413'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 3 of thread 2
ORA-00312: online log 3 thread 2: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_3.259.931826417'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 3 thread 2: '/u01/app/oracle/oradata/STBY/onlinelog/group_3.290.931826413'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 4 of thread 2
ORA-00312: online log 4 thread 2: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_4.260.931826421'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 4 thread 2: '/u01/app/oracle/oradata/STBY/onlinelog/group_4.291.931826417'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 4 of thread 2
ORA-00312: online log 4 thread 2: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_4.260.931826421'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 4 thread 2: '/u01/app/oracle/oradata/STBY/onlinelog/group_4.291.931826417'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 5 of thread 0
ORA-00312: online log 5 thread 0: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_5.303.937936343'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 5 thread 0: '/u01/app/oracle/oradata/STBY/onlinelog/group_5.292.937936339'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 5 of thread 0
ORA-00312: online log 5 thread 0: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_5.303.937936343'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 5 thread 0: '/u01/app/oracle/oradata/STBY/onlinelog/group_5.292.937936339'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 6 of thread 0
ORA-00312: online log 6 thread 0: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_6.304.937936363'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 6 thread 0: '/u01/app/oracle/oradata/STBY/onlinelog/group_6.298.937936361'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 6 of thread 0
ORA-00312: online log 6 thread 0: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_6.304.937936363'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 6 thread 0: '/u01/app/oracle/oradata/STBY/onlinelog/group_6.298.937936361'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 7 of thread 0
ORA-00312: online log 7 thread 0: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_7.305.937936377'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 7 thread 0: '/u01/app/oracle/oradata/STBY/onlinelog/group_7.299.937936375'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 7 of thread 0
ORA-00312: online log 7 thread 0: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_7.305.937936377'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 7 thread 0: '/u01/app/oracle/oradata/STBY/onlinelog/group_7.299.937936375'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 8 of thread 0
ORA-00312: online log 8 thread 0: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_8.306.937936389'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 8 thread 0: '/u01/app/oracle/oradata/STBY/onlinelog/group_8.300.937936389'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 8 of thread 0
ORA-00312: online log 8 thread 0: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_8.306.937936389'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 8 thread 0: '/u01/app/oracle/oradata/STBY/onlinelog/group_8.300.937936389'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 9 of thread 0
ORA-00312: online log 9 thread 0: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_9.307.937936405'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 9 thread 0: '/u01/app/oracle/oradata/STBY/onlinelog/group_9.301.937936403'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Sun Mar 12 23:18:22 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_lgwr_16416.trc:
ORA-00313: open failed for members of log group 9 of thread 0
ORA-00312: online log 9 thread 0: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_9.307.937936405'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 9 thread 0: '/u01/app/oracle/oradata/STBY/onlinelog/group_9.301.937936403'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

alter database clear logfile group 1
Clearing online log 1 of thread 1 sequence number 41
Sun Mar 12 23:21:42 2017
Errors in file /u01/app/oracle/diag/rdbms/stby/STBY/trace/STBY_ora_16474.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/u01/app/oracle/fast_recovery_area/STBY/onlinelog/group_1.257.931825281'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3



It is also interesting to see how CONTROL_FILES, DB_FILE_NAME_CONVERT and LOG_FILE_NAME_CONVERT are reported in the alert log :

  
  control_files            = "/u01/app/oracle/oradata/STBY/controlfile/o1_mf_ddbs0p3w_.ctl"
  control_files            = "/u01/app/oracle/fast_recovery_area/STBY/controlfile/o1_mf_ddbs0p40_.ctl"
  db_file_name_convert     = "+DATA/RAC"
  db_file_name_convert     = "/u01/app/oracle/oradata/STBY"
  db_file_name_convert     = "+FRA/RAC"
  db_file_name_convert     = "/u01/app/oracle/fast_recovery_area/STBY"
  log_file_name_convert    = "+DATA/RAC"
  log_file_name_convert    = "/u01/app/oracle/oradata/STBY"
  log_file_name_convert    = "+FRA/RAC"
  log_file_name_convert    = "/u01/app/oracle/fast_recovery_area/STBY"

Each component of the entry in the init parameter file is reported on a separate line in the alert log.
.
.
.

12 March, 2017

12cR1 RAC Posts -- 8b : DUPLICATE DATABASE FOR STANDBY

Continuing the previous post, where I have done the pre-setup...
(A reminder : This is a SingleInstance Standby on FileSystem for a RAC Database with a PDB on ASM)


On the SingleInstance Node (for the Standby database)

Precreate the required folders for the database (and redo and control) files (I later realised that the "onlinelog" folders for the SEED and PDB are not required)

[oracle@oem132 ~]$ cd /u01/app/oracle/oradata
[oracle@oem132 oradata]$ mkdir STBY
[oracle@oem132 oradata]$ cd STBY
[oracle@oem132 STBY]$ mkdir datafile
[oracle@oem132 STBY]$ mkdir onlinelog
[oracle@oem132 STBY]$ mkdir 44bbc69ce8f552aee053334ea8c07365
[oracle@oem132 STBY]$ mkdir fd9ac20f64d244d7e043b6a9e80a2f2f
[oracle@oem132 STBY]$ cd 44*
[oracle@oem132 44bbc69ce8f552aee053334ea8c07365]$ mkdir datafile tempfile
[oracle@oem132 44bbc69ce8f552aee053334ea8c07365]$ cd ../fd*
[oracle@oem132 fd9ac20f64d244d7e043b6a9e80a2f2f]$ mkdir datafile tempfile
[oracle@oem132 fd9ac20f64d244d7e043b6a9e80a2f2f]$
[oracle@oem132 STBY]$
[oracle@oem132 STBY]$ cd /u01/app/oracle/fast_recovery_area
[oracle@oem132 fast_recovery_area]$ mkdir STBY
[oracle@oem132 fast_recovery_area]$ cd STBY
[oracle@oem132 STBY]$ mkdir onlinelog
[oracle@oem132 STBY]$ mkdir 44bbc69ce8f552aee053334ea8c07365
[oracle@oem132 STBY]$ mkdir fd9ac20f64d244d7e043b6a9e80a2f2f
[oracle@oem132 STBY]$ cd 44*
[oracle@oem132 44bbc69ce8f552aee053334ea8c07365]$ mkdir onlinelog
[oracle@oem132 44bbc69ce8f552aee053334ea8c07365]$ cd ../fd*
[oracle@oem132 fd9ac20f64d244d7e043b6a9e80a2f2f]$ mkdir onlinelog
[oracle@oem132 fd9ac20f64d244d7e043b6a9e80a2f2f]$
[oracle@oem132 STBY]$

Note that the names of the two folders for the SEED and the PDB are in *lower case*.

Next start the listener for STBY.

[oracle@oem132 STBY]$ cd
[oracle@oem132 ~]$ tail -1 /etc/oratab
STBY:/u01/app/oracle/product/12.1.0/dbhome_1:N
[oracle@oem132 ~]$ . oraenv
ORACLE_SID = [oracle] ? STBY
The Oracle base has been set to /u01/app/oracle
[oracle@oem132 ~]$ ls -l $ORACLE_HOME/dbs/*STBY*
-rw-r--r-- 1 oracle oinstall  508 Mar  6 22:11 /u01/app/oracle/product/12.1.0/dbhome_1/dbs/initSTBY.ora
-rw-r----- 1 oracle oinstall 7680 Mar  6 22:13 /u01/app/oracle/product/12.1.0/dbhome_1/dbs/orapwSTBY
[oracle@oem132 ~]$ sqlplus '/ as sysdba'

SQL*Plus: Release 12.1.0.2.0 Production on Sun Mar 12 23:03:43 2017

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount
ORACLE instance started.

Total System Global Area  838860800 bytes
Fixed Size                  2929936 bytes
Variable Size             230689520 bytes
Database Buffers          599785472 bytes
Redo Buffers                5455872 bytes
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
[oracle@oem132 ~]$ lsnrctl start

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 12-MAR-2017 23:04:28

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /u01/app/oracle/product/12.1.0/dbhome_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production
System parameter file is /u01/app/oracle/product/12.1.0/dbhome_1/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/oem132/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oem132.racattack)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oem132.racattack)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                12-MAR-2017 23:04:28
Uptime                    0 days 0 hr. 0 min. 12 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/12.1.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/oem132/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oem132.racattack)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Services Summary...
Service "STBY" has 1 instance(s).
  Instance "STBY", status UNKNOWN, has 2 handler(s) for this service...
The command completed successfully
[oracle@oem132 ~]$

I am now ready to issue the DUPLICATE DATABASE command from the RAC Primary.

[oracle@collabn1 ~]$ . oraenv
ORACLE_SID = [RAC1] ? RAC1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@collabn1 ~]$ rman

Recovery Manager: Release 12.1.0.2.0 - Production on Sun Mar 12 23:17:27 2017

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

RMAN> connect target sys/racattack

connected to target database: RAC (DBID=2519807290)

RMAN> connect auxiliary sys/racattack@STBY

connected to auxiliary database: RAC (not mounted)

RMAN> DUPLICATE TARGET DATABASE FOR STANDBY FROM ACTIVE DATABASE ;

Starting Duplicate Db at 12-MAR-17
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=12 device type=DISK

contents of Memory Script:
{
   backup as copy reuse
   targetfile  '+DATA/RAC/PASSWORD/pwdrac.277.931824933' auxiliary format
 '/u01/app/oracle/product/12.1.0/dbhome_1/dbs/orapwSTBY'   ;
}
executing Memory Script

Starting backup at 12-MAR-17
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=273 instance=RAC1 device type=DISK
Finished backup at 12-MAR-17

contents of Memory Script:
{
   sql clone "alter system set  control_files =
  ''/u01/app/oracle/oradata/STBY/controlfile/o1_mf_ddbs0p3w_.ctl'', ''/u01/app/oracle/fast_recovery_area/STBY/controlfile/o1_mf_ddbs0p40_.ctl'' comment=
 ''Set by RMAN'' scope=spfile";
   backup as copy current controlfile for standby auxiliary format  '/u01/app/oracle/oradata/STBY/controlfile/o1_mf_ddbs0p3w_.ctl';
   restore clone primary controlfile to  '/u01/app/oracle/fast_recovery_area/STBY/controlfile/o1_mf_ddbs0p40_.ctl' from
 '/u01/app/oracle/oradata/STBY/controlfile/o1_mf_ddbs0p3w_.ctl';
   sql clone "alter system set  control_files =
  ''/u01/app/oracle/oradata/STBY/controlfile/o1_mf_ddbs0p3w_.ctl'', ''/u01/app/oracle/fast_recovery_area/STBY/controlfile/o1_mf_ddbs0p40_.ctl'' comment=
 ''Set by RMAN'' scope=spfile";
   shutdown clone immediate;
   startup clone nomount;
}
executing Memory Script

sql statement: alter system set  control_files =   ''/u01/app/oracle/oradata/STBY/controlfile/o1_mf_ddbs0p3w_.ctl'', ''/u01/app/oracle/fast_recovery_area/STBY/controlfile/o1_mf_ddbs0p40_.ctl'' comment= ''Set by RMAN'' scope=spfile

Starting backup at 12-MAR-17
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
copying standby control file
output file name=/u01/app/oracle/oradata/STBY/controlfile/o1_mf_ddbs0p3w_.ctl tag=TAG20170312T231755
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at 12-MAR-17

Starting restore at 12-MAR-17
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: copied control file copy
Finished restore at 12-MAR-17

sql statement: alter system set  control_files =   ''/u01/app/oracle/oradata/STBY/controlfile/o1_mf_ddbs0p3w_.ctl'', ''/u01/app/oracle/fast_recovery_area/STBY/controlfile/o1_mf_ddbs0p40_.ctl'' comment= ''Set by RMAN'' scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area     838860800 bytes

Fixed Size                     2929936 bytes
Variable Size                230689520 bytes
Database Buffers             599785472 bytes
Redo Buffers                   5455872 bytes

contents of Memory Script:
{
   sql clone 'alter database mount standby database';
}
executing Memory Script

sql statement: alter database mount standby database

contents of Memory Script:
{
   set newname for tempfile  1 to
 "/u01/app/oracle/oradata/STBY/tempfile/temp.285.931825311";
   set newname for tempfile  2 to
 "/u01/app/oracle/oradata/STBY/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/pdbseed_temp012016-12-29_12-23-03-am.dbf";
   set newname for tempfile  3 to
 "/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/tempfile/temp.295.931827153";
   switch clone tempfile all;
   set newname for datafile  1 to
 "/u01/app/oracle/oradata/STBY/datafile/system.279.931825083";
   set newname for datafile  3 to
 "/u01/app/oracle/oradata/STBY/datafile/sysaux.278.931825019";
   set newname for datafile  4 to
 "/u01/app/oracle/oradata/STBY/datafile/undotbs1.281.931825149";
   set newname for datafile  5 to
 "/u01/app/oracle/oradata/STBY/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/system.287.931825323";
   set newname for datafile  6 to
 "/u01/app/oracle/oradata/STBY/datafile/users.280.931825149";
   set newname for datafile  7 to
 "/u01/app/oracle/oradata/STBY/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/sysaux.286.931825323";
   set newname for datafile  8 to
 "/u01/app/oracle/oradata/STBY/datafile/undotbs2.289.931826143";
   set newname for datafile  9 to
 "/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/system.293.931827089";
   set newname for datafile  10 to
 "/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/sysaux.294.931827089";
   set newname for datafile  11 to
 "/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/users.296.931827189";
   backup as copy reuse
   datafile  1 auxiliary format
 "/u01/app/oracle/oradata/STBY/datafile/system.279.931825083"   datafile
 3 auxiliary format
 "/u01/app/oracle/oradata/STBY/datafile/sysaux.278.931825019"   datafile
 4 auxiliary format
 "/u01/app/oracle/oradata/STBY/datafile/undotbs1.281.931825149"   datafile
 5 auxiliary format
 "/u01/app/oracle/oradata/STBY/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/system.287.931825323"   datafile
 6 auxiliary format
 "/u01/app/oracle/oradata/STBY/datafile/users.280.931825149"   datafile
 7 auxiliary format
 "/u01/app/oracle/oradata/STBY/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/sysaux.286.931825323"   datafile
 8 auxiliary format
 "/u01/app/oracle/oradata/STBY/datafile/undotbs2.289.931826143"   datafile
 9 auxiliary format
 "/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/system.293.931827089"   datafile
 10 auxiliary format
 "/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/sysaux.294.931827089"   datafile
 11 auxiliary format
 "/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/users.296.931827189"   ;
   sql 'alter system archive log current';
}
executing Memory Script

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

renamed tempfile 1 to /u01/app/oracle/oradata/STBY/tempfile/temp.285.931825311 in control file
renamed tempfile 2 to /u01/app/oracle/oradata/STBY/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/pdbseed_temp012016-12-29_12-23-03-am.dbf in control file
renamed tempfile 3 to /u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/tempfile/temp.295.931827153 in control file

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting backup at 12-MAR-17
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=+DATA/RAC/DATAFILE/system.279.931825083
output file name=/u01/app/oracle/oradata/STBY/datafile/system.279.931825083 tag=TAG20170312T231824
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:45
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=+DATA/RAC/DATAFILE/sysaux.278.931825019
output file name=/u01/app/oracle/oradata/STBY/datafile/sysaux.278.931825019 tag=TAG20170312T231824
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:35
channel ORA_DISK_1: starting datafile copy
input datafile file number=00007 name=+DATA/RAC/FD9AC20F64D244D7E043B6A9E80A2F2F/DATAFILE/sysaux.286.931825323
output file name=/u01/app/oracle/oradata/STBY/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/sysaux.286.931825323 tag=TAG20170312T231824
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:35
channel ORA_DISK_1: starting datafile copy
input datafile file number=00010 name=+DATA/RAC/44BBC69CE8F552AEE053334EA8C07365/DATAFILE/sysaux.294.931827089
output file name=/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/sysaux.294.931827089 tag=TAG20170312T231824
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting datafile copy
input datafile file number=00009 name=+DATA/RAC/44BBC69CE8F552AEE053334EA8C07365/DATAFILE/system.293.931827089
output file name=/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/system.293.931827089 tag=TAG20170312T231824
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=+DATA/RAC/FD9AC20F64D244D7E043B6A9E80A2F2F/DATAFILE/system.287.931825323
output file name=/u01/app/oracle/oradata/STBY/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/system.287.931825323 tag=TAG20170312T231824
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=+DATA/RAC/DATAFILE/undotbs1.281.931825149
output file name=/u01/app/oracle/oradata/STBY/datafile/undotbs1.281.931825149 tag=TAG20170312T231824
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00008 name=+DATA/RAC/DATAFILE/undotbs2.289.931826143
output file name=/u01/app/oracle/oradata/STBY/datafile/undotbs2.289.931826143 tag=TAG20170312T231824
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting datafile copy
input datafile file number=00006 name=+DATA/RAC/DATAFILE/users.280.931825149
output file name=/u01/app/oracle/oradata/STBY/datafile/users.280.931825149 tag=TAG20170312T231824
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting datafile copy
input datafile file number=00011 name=+DATA/RAC/44BBC69CE8F552AEE053334EA8C07365/DATAFILE/users.296.931827189
output file name=/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/users.296.931827189 tag=TAG20170312T231824
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at 12-MAR-17

sql statement: alter system archive log current

contents of Memory Script:
{
   switch clone datafile all;
}
executing Memory Script

datafile 1 switched to datafile copy
input datafile copy RECID=3 STAMP=938474501 file name=/u01/app/oracle/oradata/STBY/datafile/system.279.931825083
datafile 3 switched to datafile copy
input datafile copy RECID=4 STAMP=938474501 file name=/u01/app/oracle/oradata/STBY/datafile/sysaux.278.931825019
datafile 4 switched to datafile copy
input datafile copy RECID=5 STAMP=938474501 file name=/u01/app/oracle/oradata/STBY/datafile/undotbs1.281.931825149
datafile 5 switched to datafile copy
input datafile copy RECID=6 STAMP=938474501 file name=/u01/app/oracle/oradata/STBY/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/system.287.931825323
datafile 6 switched to datafile copy
input datafile copy RECID=7 STAMP=938474501 file name=/u01/app/oracle/oradata/STBY/datafile/users.280.931825149
datafile 7 switched to datafile copy
input datafile copy RECID=8 STAMP=938474501 file name=/u01/app/oracle/oradata/STBY/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/sysaux.286.931825323
datafile 8 switched to datafile copy
input datafile copy RECID=9 STAMP=938474501 file name=/u01/app/oracle/oradata/STBY/datafile/undotbs2.289.931826143
datafile 9 switched to datafile copy
input datafile copy RECID=10 STAMP=938474501 file name=/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/system.293.931827089
datafile 10 switched to datafile copy
input datafile copy RECID=11 STAMP=938474501 file name=/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/sysaux.294.931827089
datafile 11 switched to datafile copy
input datafile copy RECID=12 STAMP=938474501 file name=/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/users.296.931827189
Finished Duplicate Db at 12-MAR-17

RMAN>

The DUPLICATE ... FOR STANDBY has completed.  Let me just run two verification checks.

On the Standby :

[oracle@oem132 ~]$ sqlplus '/ as sysdba'

SQL*Plus: Release 12.1.0.2.0 Production on Sun Mar 12 23:24:05 2017

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> select database_role from v$database;

DATABASE_ROLE
----------------
PHYSICAL STANDBY

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
MOUNTED

SQL>
SQL> select file#, name from v$datafile order by 1;

     FILE#
----------
NAME
--------------------------------------------------------------------------------
         1
/u01/app/oracle/oradata/STBY/datafile/system.279.931825083

         3
/u01/app/oracle/oradata/STBY/datafile/sysaux.278.931825019

         4
/u01/app/oracle/oradata/STBY/datafile/undotbs1.281.931825149

         5
/u01/app/oracle/oradata/STBY/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/system.28
7.931825323

         6
/u01/app/oracle/oradata/STBY/datafile/users.280.931825149

         7
/u01/app/oracle/oradata/STBY/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/sysaux.28
6.931825323

         8
/u01/app/oracle/oradata/STBY/datafile/undotbs2.289.931826143

         9
/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/system.29
3.931827089

        10
/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/sysaux.29
4.931827089

        11
/u01/app/oracle/oradata/STBY/44bbc69ce8f552aee053334ea8c07365/datafile/users.296
.931827189


10 rows selected.

SQL>
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
[oracle@oem132 ~]$ cd $ORACLE_HOME/dbs
[oracle@oem132 dbs]$ ls -ltr *STBY*
-rw-r--r-- 1 oracle oinstall   508 Mar  6 22:11 initSTBY.ora
-rw-r----- 1 oracle oinstall    24 Mar 12 23:07 lkSTBY
-rw-r----- 1 oracle oinstall  7680 Mar 12 23:17 orapwSTBY
-rw-rw---- 1 oracle oinstall  1544 Mar 12 23:18 hc_STBY.dat
-rw-r----- 1 oracle oinstall 17920 Mar 12 23:18 spfileSTBY.ora
[oracle@oem132 dbs]$

On the Primary :

RMAN> exit


Recovery Manager complete.
[oracle@collabn1 ~]$ sqlplus '/ as sysdba'

SQL*Plus: Release 12.1.0.2.0 Production on Sun Mar 12 23:26:17 2017

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SQL> select database_role from v$database;

DATABASE_ROLE
----------------
PRIMARY

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE

SQL> set pages600
SQL> select file#, name from v$datafile order by 1;

     FILE#
----------
NAME
--------------------------------------------------------------------------------
         1
+DATA/RAC/DATAFILE/system.279.931825083

         3
+DATA/RAC/DATAFILE/sysaux.278.931825019

         4
+DATA/RAC/DATAFILE/undotbs1.281.931825149

         5
+DATA/RAC/FD9AC20F64D244D7E043B6A9E80A2F2F/DATAFILE/system.287.931825323

         6
+DATA/RAC/DATAFILE/users.280.931825149

         7
+DATA/RAC/FD9AC20F64D244D7E043B6A9E80A2F2F/DATAFILE/sysaux.286.931825323

         8
+DATA/RAC/DATAFILE/undotbs2.289.931826143

         9
+DATA/RAC/44BBC69CE8F552AEE053334EA8C07365/DATAFILE/system.293.931827089

        10
+DATA/RAC/44BBC69CE8F552AEE053334EA8C07365/DATAFILE/sysaux.294.931827089

        11
+DATA/RAC/44BBC69CE8F552AEE053334EA8C07365/DATAFILE/users.296.931827189


10 rows selected.

SQL>

All datafiles (CDB, SEED and 1 PDB) are replicated.  Note how the folder names for the SEED and PDB are in lower-case on the Standby.  Since the source database is RAC, it has two Undo Tablespaces.  (This is 12.1 so I do not have the 12.2 feature of Local Undo in PDBs)

The next few posts will be on the DataGuard Broker and Redo Shipping and Applly.
(Should I also do a SWITCHOVER ?)
.
.
.


08 March, 2017

12cR1 RAC Posts -- 8a : Setting up SingleInstance DG Standby for RAC

This is the first of a small series of subposts on setting up SingleInstance DataGuard Standby for my 12cR1 RAC environment.

Primary Database : 12cR1 MultiTenant RAC : 2 nodes, database on ASM
Standby Database : 12cR1 MultiTenant SingleInstance : 1 node, database on FileSystem

The "difficulties" are :
a.  The Primary Database is MultiTenant with 1 PDB besides the SEED
b.  The Primary Database is on ASM but the Standby is to be built on FileSystem
c.  The Primary is a 2-node Cluster while the Standby is a SingleInstance installatin

This post is on the initial setup.

RAC Nodes
collabn1.racattack   192.168.78.51
collabn2.racattack   192.168.78.52

SingleInstance Node 
oem132.racattack     192.168.78.100


Database Changes :
SQL> alter database force logging;

Database altered.

SQL> alter database flashback on;

Database altered.

SQL>
SQL> alter database add standby logfile size 50M;

Database altered.

SQL> alter database add standby logfile size 50M;

Database altered.

SQL> alter database add standby logfile size 50M;

Database altered.

SQL> alter database add standby logfile size 50M;

Database altered.

SQL> alter database add standby logfile size 50M;

Database altered.

SQL>
SQL>select  group# , status, type, member from v$logfile order by 1;


Listener on Standby Server :
LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = oem132.racattack)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
  )

SID_LIST_LISTENER=
(SID_LIST =
  (SID_DESC =
    (ORACLE_HOME = /u01/app/oracle/product/12.1.0/dbhome_1)
    (SID_NAME = STBY)
   )
  (SID_DESC =
    (GLOBAL_NAME = STBY_DGMGRL)
    (ORACLE_HOME = /u01/app/oracle/product/12.1.0/dbhome_1)
    (SID_NAME = STBY)
   )
)


tnsnames.ora on first RAC Node :
STBY =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = oem132)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = STBY)
    )
  )

STBY_DGB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = oem132)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = STBY_DGMGRL)
    )
  )


STBY Instance Parameter File :
compatible=12.1.0.2.0
db_name=RAC
db_unique_name=STBY
enable_pluggable_database=TRUE
sga_target=800M
db_file_name_convert='+DATA/RAC','/u01/app/oracle/oradata/STBY','+FRA/RAC','/u01/app/oracle/fast_recovery_area/STBY'
log_file_name_convert='+DATA/RAC','/u01/app/oracle/oradata/STBY','+FRA/RAC','/u01/app/oracle/fast_recovery_area/STBY'
db_create_file_dest='/u01/app/oracle/oradata'
db_recovery_file_dest='/u01/app/oracle/fast_recovery_area'
db_recovery_file_dest_size=16G
remote_login_passwordfile=EXCLUSIVE


Copying the Password File :
ASMCMD> pwcopy +DATA/RAC/PASSWORD/pwdrac.277.931824933 /var/tmp/RACPasswordFile
copying +DATA/RAC/PASSWORD/pwdrac.277.931824933 -> /var/tmp/RACPasswordFile
ASMCMD> exit
scp /var/tmp/RACPasswordFile oracle@oem132:/u01/app/oracle/product/12.1.0/dbhome_1/dbs/orapwSTBY


to be continued in post 8b on the filesystem directories for the STBY instance and the execution of the DUPLICATE DATABASE command.

06 March, 2017

12cR1 RAC Posts -- 7 : OCR Commands

[Yes, I know that 12.2 is now available for download but it will be some time before I have a running 12.2 RAC environment]

Some OCR / OLR Commands :

The OCR is the Cluster Registry.  We also have an OLR that is the Local Registry which is created on a local filesystem.

We can check the consistency of the Registry with ocrcheck.  Note the difference between using oracle (or grid) and using root to run the check.  oracle can't check the OLR and can't do a logical consistency check of the OCR -- both require to be run as root.

[root@collabn1 ~]# su - oracle
[oracle@collabn1 ~]$ . oraenv
ORACLE_SID = [oracle] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[oracle@collabn1 ~]$ ocrcheck -local
PROTL-602: Failed to retrieve data from the local registry
PROCL-26: Error while accessing the physical storage Operating System error [Permission denied] [13]
[oracle@collabn1 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       1676
         Available space (kbytes) :     407892
         ID                       :  827167720
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

[oracle@collabn1 ~]$ su
Password:
[root@collabn1 oracle]# ocrcheck -local
Status of Oracle Local Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       1036
         Available space (kbytes) :     408532
         ID                       : 1014277103
         Device/File Name         : /u01/app/12.1.0/grid/cdata/collabn1.olr
                                    Device/File integrity check succeeded

         Local registry integrity check succeeded

         Logical corruption check succeeded

[root@collabn1 oracle]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       1676
         Available space (kbytes) :     407892
         ID                       :  827167720
         Device/File Name         :   +OCRVOTE
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@collabn1 oracle]#


Oracle automates backups of the OCR (but not the OLR !).  Below, the -showbackuploc shows the location of backups.

[root@collabn1 oracle]# ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [/u01/app/12.1.0/grid/cdata/]
[root@collabn1 oracle]# ls -lt /u01/app/12.1.0/grid/cdata
total 1272
-rw-------. 1 root   oinstall 503484416 Mar  6 17:03 collabn1.olr
drwxrwxr-x. 2 oracle oinstall      4096 Jan 16 14:12 collabn-cluster
drwxr-xr-x. 2 oracle oinstall      4096 Dec 19 15:06 collabn1
drwxr-xr-x. 2 oracle oinstall      4096 Dec 19 14:37 localhost
[root@collabn1 oracle]# ls -lt /u01/app/12.1.0/grid/cdata/collabn1
total 820
-rw-r--r--. 1 root root 839680 Dec 19 15:06 backup_20161219_150615.olr
[root@collabn1 oracle]# ocrconfig -showbackup

collabn1     2017/01/16 14:09:40     /u01/app/12.1.0/grid/cdata/collabn-cluster/backup00.ocr     0

collabn1     2017/01/16 14:09:40     /u01/app/12.1.0/grid/cdata/collabn-cluster/day.ocr     0

collabn1     2017/01/16 14:09:40     /u01/app/12.1.0/grid/cdata/collabn-cluster/week.ocr     0

collabn2     2016/12/19 15:47:24     /u01/app/12.1.0/grid/cdata/collabn-cluster/backup_20161219_154724.ocr     0   

collabn2     2016/12/19 15:47:16     /u01/app/12.1.0/grid/cdata/collabn-cluster/backup_20161219_154716.ocr     0   
[root@collabn1 oracle]#


All recent (4-Hourly, Daily, Weekly) of the OCR are on the "master" node -- collabn1 -- which comes up first in my cluster.    The 19-Dec backups (of the OCR and OLR) are when I started setting up the Cluster.  Note that there are no subsequent (automated) OLR backups.
Note : There are no 4-Hourly/Daily/Weekly backups since 16-Jan because I haven't had my cluster running for long enough for those backups to kick in.

[root@collabn1 oracle]# ocrconfig -local -manualbackup

collabn1     2017/03/06 17:11:29     /u01/app/12.1.0/grid/cdata/collabn1/backup_20170306_171129.olr     0

collabn1     2016/12/19 15:06:15     /u01/app/12.1.0/grid/cdata/collabn1/backup_20161219_150615.olr     0
[root@collabn1 oracle]# ocrconfig -manualbackup

collabn1     2017/03/06 17:12:21     /u01/app/12.1.0/grid/cdata/collabn-cluster/backup_20170306_171221.ocr     0   

collabn2     2016/12/19 15:47:24     /u01/app/12.1.0/grid/cdata/collabn-cluster/backup_20161219_154724.ocr     0   

collabn2     2016/12/19 15:47:16     /u01/app/12.1.0/grid/cdata/collabn-cluster/backup_20161219_154716.ocr     0   
[root@collabn1 oracle]#


I can run manual backups (the -local is for the OLR) as shown above.

It is important to include these backups in the backup strategy for the filesystem(s) that hold the Grid Infrastructure and RDBMS installations (binaries, configuration files, trace files etc).
.
.
.