Oracle blog

An interesting 5 day EMC SAN Storage Implementation course

2008-07-05T12:37:00.000-07:00

When I reached the office this morning, I have been informed by my manager that my name has been suggested from the DBA team to attend the SAN Storage Implementation 5 day course and I must rush to the training center right away. It was a bit suprise news for me and I initally thought its not my cup of tea, but, still wanted to attend.

Today, we have just completed day 1 class in which all the basic stuff about SAN storage (specially EMC) has been discussed and I must say it is really worthy attending this training. It did really help me to clear some of the basic concepts which I was thinking about SAN Storage.

Now, I can talk about, what is a SAN Block Box (what does it consists of), what is a VLAN/VSAN, Fiber Channel, type of ports on a switch, how volume manager works, what is a LUN, port zoaning. prot channels, what is multipathing & it works, and etc. stuff. I am not sure how long I am going to keep this terminology in my memory, but, surely, this course going to give me bit of confidence to configure our RAC enviornment optimally.

I will be posting about the interesting stuff that learned in the class in coming days.

A final word about our storage. Our bank has brought EMC DMX4 SAN storage.

Happy reading,

Jaffar

Database migration across platforms and conversion of single instance to RAC

2008-07-02T05:42:00.000-07:00

Working in a huge RAC environment, each passing day is a learning day for me. After setting up 4 node RAC on HP UX Itanium 2 Severs, the subsequent task/challenge is to migrate single instance databases (around 16) from AIX OS to HP OS and plus converting single instance to RAC.

I have started this task on a low confidence note, I would be lying if I say I was very much confident. However, as thing progress, my confidence level went high and at the end I have successfully migrated a database (single instance) from AIX OS to HP OS and then converted the single instance database to RAC database.

To give you a rough idea what I have done:

Converted database from AIX to HP using RMAN.
On HP, then, converted this databases from single-instance to RAC database. There are few ways using which you can achieve this. However, I have opted the DBCA option to execute the task.

Next step is to migrate and convert the rest of the databases using the same method.

Happy reading,

Jaffar

Oracle doesn't allow replacing ocr disk or adding the mirror copy if the new disk size is less than the existing ocr disk size

2008-06-29T08:19:00.000-07:00

At least my self now decided not to believe in the ML Notes (not always though), its better testing what they have said. I was trying to replace an existing ocr location and also trying to add mirror copy of ocr. During RAC setup, we have used 1gb disks for OCR and voting disk (with single copy of each).
In order to multiplex the OCR and Voting disk we have requsted our UNXI team to gives us the four raw disks with 500mb each (oracle recommends 300MB for OCR/Voting Disk, may be less) and whil trying to replace/mirror the ocr to the new raw disks, I have got the following errors:

for OCR Mirror copy: PROT-22:Storage is too small
for replace OCR : PROT-16: Internal Error

As usaul, we search in the ML and found couple of notes for the same errors (317628.1 & 444757.1). Well, the ML Note : 317628.1 said the following:

" fails with "PROT-22: Storage too small" error. The problem is due to an Oracle bug where this operation requires the OCR mirror partition/file tobe larger than the original by up to 128MB. The bug has been fixed in the 10.2.0.2 patchset."

The note says that this bug has been fixed in 10.2.0.2 patch, we are on 10.2.0.3 patch and the bug still persists.

Happy reading,

Jaffar

A stupid and silly issue took hours to resolve

2008-06-28T12:17:00.000-07:00

This morning we were trying to create a RAC database using the dbca utility on HP UX Itanium II servers and a silly/stupid error keep coming for a long time and we got strucked for good amount of time.

The error message which was coming 'Unable to copy the '/etc/oratab' to '/tmp/oratab.' and when ignore option was choosen from the error screen, the subsequent message was 'Encountered file error when copying listeners from home=$ORACLE_HOME/asm".

We certainly knew that the issue is either with the permission or copying the said file and our immediate step was checking the permisson and ownership on the oratab file. Everything was just fine and then we try to create a file in the /tmp directory using touch command from the Oracle user and it worked without any hassles.

We were able to create the RAC database successfully on a single node and when tried to add instance on the other node, the same issue encountered again and again.

We finally thought of running the dbca in tracing mode and when the same problem arises and when we had a look at the trace file, we couldn't stop wondering about the silly/stupid thing which Oracle was complaining.

The trace file reveals that it is looking for 'scp' executable in '/usr/local/bin' location, whereas, the 'scp' excutable was in '/usr/bin' directory. We simply created a link for 'scp' in the '/usr/local/bin' and dbca run smoothly and we were able to complete the RAC database creation.

Past three months, its been really a great working expeirence on RAC and I must say, we have faced hell a lot of issues while trying to implement RAC on HP Virtual Machines. Definately, its a great learning phase.

Guess what's next? MOVING 8 DEVELOPMENT DATABASE TO RAC from AIX to HP. Umm.. a good and taugh task ahead.

I will keep posting my experience and the issues that have faced while setting up our 16 node RAC with 8 databases.

Happy reading,

Jaffar

Happy Birthday To ME! I am 34 years old now!

2008-06-02T23:24:00.000-07:00

I was just wondering how days, months and a year past my life . It looks like I recently blogged about my previous birthday and now I am blogging about my another birthday.

I am not one of those good people who do set some goals every year to achieve. I am bit lazy doing so. However, this time I do set couple of goals, like, achieving the OCM status and a very careful planning about financial savings(which I never think seriously before).

It was really a very fruitful last year where I did achieve something good in my life, like, given my ever first international Oracle seminars in Sweden & Denmark, spending more time working on RAC environments and brought a good house.

Unlike last year, no French Corner Cacke this time. No plans of celebration. Its like a normal day for me.

Thanks for passing by my blog.

Happy reading,

Jaffar

16 node RAC with 12 databases

2008-05-28T10:14:00.000-07:00

Umm.. its been really a quite long time that I did update my blog. becauseI have left my previous company (bank) after over 4 fruitful years. I have joined a new upcoming bank where I have got an opportunity to setup a 16 node RAC on HPUX (development) with 12 databases plus grid computing setup.

Though the soft lunch of the bank would be in coming few months , we are too busy setting up the RAC and Grid environment. In process, we have just started with a setup of 8 node RAC for the development environment this evening and will be ending up adding 8 more nodes on HP-UX Superdom (Itanimum 2) servers using EMC storage.

It is going to be the RAC setup all the way, i.e., starting with development, testing, pre-production and finally to production environments. I am not sure whether any coompany has such sort setup here in Saudi Arabia and this is going to be very challenging to our DBA's team as well to the OS administrators.

The next challenging which lays after the RAC setup is moving/migrating stand alone databases from AIX OS to the RAC setup on HPUX os. Which required database conversion and then adding to RAC.

I am sure that it gonna be a great fun handling such setup. There is plan to fully utilize the Computing technology, most probably looking at the workload management, TAF, FAN and etc. Its is not yet clear that all the databases (probably 12) will have scalability across all the nodes or something else.

I will keep writing in my blog about the developments, issues we faced and other technical issues during this juroney.

Till then, have good time.

Jaffar

First ever Saudi Oracle User Group

2008-04-07T10:08:00.000-07:00

Its been really quite a long wait for all Oracle techies (who are working in Saudi Arabia) to have a local Oracle User Group: that is, a Saudi Oracle User Group. Well, cherished dream became finally true on March 11 when Oracle had an formal meeting about starting an official Saudi Oracle User Group and explained their plans to run the group.

As per the press news, Dr. Majid Altuwaijri, CIO, National Guard Health Affairs will be the president of Saudi Oracle User Group.

Following is the excerpt from the press release:
=================
The autonomous group comprises 14 most distinguished CIO's from the various industry verticals in the Kingdom.

Apart from addressing member problems, the group will provide networking and collaboration opportunities to Oracle users in the Kingdom, enabling them to have a better understanding on specialist areas of Oracle software. The Saudi Oracle User Group will enable Oracle customers from around the Kingdom to share their firsthand experiences in using Oracle technology and solutions, and how to best leverage their IT investments.

'Though independent, the Saudi Oracle User Group will work closely with Oracle to offer constructive feedback and assist different customers with their crisis,' said Dr. Majid Altuwaijri, CIO, National Guard Health Affairs, and President of the Saudi Oracle User Group. 'The feedback will help Oracle determine long term customer requirements, improvise on product and solution offerings to help customers in the Kingdom have a global presence with groups around the world.'

Moving forward, the group will meet members to discuss the Charter to establish a set of milestones to be achieved in the short and long term. Once the Charter is defined, members will meet to define online objectives. The website will be crucial in sharing information with member and will boost memberships. It will also act as a medium of managing and publishing the group's presence. The group will also attend meetings at international and local Oracle User Groups, participate in IT exhibitions and third party forums.

Over the next 12 months, the group will meet on a quarterly basis to form special interest groups across industries. This initiative will allow groups to be more focused and interactive as it speaks the same language for a set of customers.
=========

I am very curious about joining this group and would really like to involve actively.

Jaffar

Oracle 10g Admin I - 5 day course.

2008-03-29T04:25:00.000-07:00

Now-a-day I am busy delivering Oracle University 5 days courses at various places (oracle onsite and customer site). Last week I have delivered a 5 day course on 'Oracle 10g: programming with PL/SQL' and starting with today, I am teaching 'Oracle 10g: Admin I' here at Oracle faclities in Riyadh, Saudi Arabia. I always felt that teaching is a tough job, however, I am loving it.

Happy Reading,

Jaffar

Fun in teaching 'Oracle Programming with PL/SQL'

2008-03-22T08:39:00.000-07:00

I am currently teaching a 5 day Oracle training course 'Oracle 10g: Programming with PL/SQL' and its been pleasure refreshing the basic concepts once again.

There were 12 atendees on the day 1 for this class, all of them have ample of programming experience in VB and .NET technologies. Though, not many serious questions have been asked on day 1, but, I loved teaching the Programming concepts.

Today day was the first day and still 4 days to go.

Happy Reading,

Jaffar

100,000 HITS!!!

2008-03-14T23:10:00.000-07:00

This morning my blog got 100,000 hits and its glad to see the magical number (100,000) despite the fact that my blog is not so much familiar like other Oracle Gurus blog.

Regards,

Jaffar

Now my articles appears on SSWUG.org web site - WITHOUT MY PERMISSION!!

2008-03-12T05:56:00.000-07:00

If I remember correctly, sometime back couple of people complained that their articles appeared / published on SSWUG.ORG web site without their knowledge and permissions.

Today, I was shocked and surprised to see couple of my articles (which were written on blog) appeared on this site. I don't remember that they have ever taken my permissions to publish at their site.

Following is the link of few articles published on their web site:

Simulating 11g Snapshot Standby Database feature on Oracle 10g?
http://www.sswug.org/see/37089

Change behavior of GROUP BY clause in 10g
http://www.sswug.org/see/34897

I don't mind publishing my articles on their web site if they would have taken my permission.

Happy Reading,

Jaffar

Ad - Looking for some consulting work (specially in Saudi Arabia)

2008-03-12T02:21:00.000-07:00

Currently I am looking for any Oracle database consultancy work or any training offer here in Saudi Arabia (even out side Saudi Arabia).

Following are my specialized ares for consultancy work:

Database installation/configuration
RAC Setup
Database Migration
Data Guard/Physical Standby Setup (Disaster Recovery)
Performance Tuning
RMAN Policies and Setup

Following are my expertise for Oracle Technology Training:

Advanced RMAN concepts - 1 day
Automatic Storage Management (ASM) - 1 day
Data Guard - 2 days
Database Concepts

Feel free to contact me on sjaffarhussain@gmail.com for further enquirers.

Happy Reading,

Jaffar

ocrconfig 11g improvement

2008-03-11T22:35:00.000-07:00

Starting in 11g, the ocrconfig command now supports manual backup option which enables you to perform ocr file backup at any time, rather than waiting for the default automatic backup (i.e. every 4 or more hrs) interval.

Using the ocrconfig -manualbackup option, you can perform the manual (binary) backup on demand.

The -showbackup option now includes auto and manual flags that you can optionally specify to view the auto/manual existing backup information.

http://download.oracle.com/docs/cd/B28359_01/rac.111/b28255/votocr.htm#i1012456

Happy Reading,

Jaffar

Wow! I am able to change my email ID on OTN forums.

2008-03-09T03:18:00.000-07:00

It was a known fact that once you setup an email id on the OTN forums, it was not possible to change the email id again and many people were just surprised about this behaviour.

I don't know, how many of you aware that the changing of email id on OTN forums is possible now. I knew that many of you are least bothered about this, however, if anybody wants to change the email id, now you can do that easily and get rid off your legay email id.

How to change the profile information on OTN forums:

Once you log in on OTN forums, click 'Your Control Panel' option on right top corner which gives the facility to change the profile information, including your email id.

Happy Reading,

Jaffar

Complete Checklist for Manual Upgrades to 11gR1 - ML Note:429825.1

2008-03-06T08:21:00.000-08:00

I am really not sure how many of you aware of the newly published Metalink Note about step-by-step manual upgrade to 11gR1.

Metalink Note : Note:429825.1. (latest revision on 05-MAR-2008)

Jaffar

Does ASM has controlfile?

2008-03-06T06:21:00.000-08:00

SQL> select * from v$version;

BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Prod
PL/SQL Release 10.2.0.1.0 - Production
CORE 10.2.0.1.0 Production
TNS for 32-bit Windows: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 - Production

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
+asm

SQL> show parameter control_files

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_files string D:\ORACLE\PRODUCT\10.2.0\DB_1\DATABASE
\CTL1+ASM.ORA

Probably we all know that ASM consist of an Instance, doesn't have database, i.e. physical structure. Therefore, the above definitely misleads.

According to the ASM book, the control_files parameters point to a dummy (nonexistent) file. This is the default internal value for this parameter and it is a meaningless value in the ASM instance.

Happy Reading,

Jaffar

Oracle Automatic Storage Management - Book Review

2008-03-01T03:30:00.000-08:00

I have borrowed this book from my colleague to read during the week-end and at the same time, thought of writing a summary/review. When I told my idea, I knew that my family disliked this, specially during week-end.

The book started with the ‘Foreword’ page consists of ‘History of ASM’ headline which is written by Bill Bridge (an architect in Oracle over 18 yrs and the creator of ASM).

He revealed few interesting points about ASM history, like when did he get an Idea about ASM, how it can be useful with Oracle technology and the total time which took to build ASM technology.

Chapter 1 – Storage Stack Overview

This chapter starts explaining an overview about Disk drive technology, disk layouts, Disk Drive performance, different class of storage system architectures (such as SAN, NAS, DAS and SCSI) .

RAID levels also been discussed, and storage arrays concepts too.

Conclusion:

It might look interesting, but, I have skipped this chapter in order to read more about ASM concepts.

Chapter 2 – ASM Instances

This chapter starts on a note about ASM introduction, managing ASM (startup/shutdown) instances, few important ASM Instance parameters (such as instance_type, asm_diskstring & asm_diskgroups), ASM instance new background processes, V$ ASM views, ASM installation, upgrade and patching concepts.

It also talk about the significance of two ASM files, the ‘ab_.dat ‘ and the ‘hc_.dat’ that are present under $ORACLE_HOME/dbs directory.

These files are used by the RDBMS instances for env. Information and health check monitoring.

Note : Beware, these files should not be deleted.

Conclusion:

Nothing new has been offered in this chapter. This information is already available in the Oracle documents.

Chapter 3 – ASM disk

This chapter starts explaining about logical term LUN and explained about creating LUNs, LUN optimal sizes and what is the minimum disk size (4mb) and maximum (2 power 32 MB). Also, explained disks discovery(asm_disktring init parameter) and explained the meaning of disk header status (Former, Candidate, Member & Provisioned).

It also talks about the disk preparation/configuration on different OS, such as, Solaris, AIX and Windows and how to use the asmtool & asmtoolg utilities.

Conclusion:

Nothing new offered in this chapter as well.

Chapter 4 – Diskgroups and Failure Groups.

This chapter started explaining about ASM diskgroup concepts and why ASM diskgroups are differ from typical LVM (logical volume manager). Also discussed ASM diskgroups redundancy types (normal, high and external) and their practice.

In the next segment of this chapter, it talk about Creating Diskgroups, default ASM Disk Names & how to change default disk naming convention and Diskgroup Numbers. An interesting point was made, i.e., “you are not restricted to having two failure groups for normal redundancy and three for high redundancy”.

Couple of new feature of v11g ASM, ASM Preferred Read, Diskgroups Attributes and Fast Disk Resync have been discussed.

In the last segment of this chapter, it explained I/O concepts and I/O failures. The chapter finished explaining about the Space Management Views and important columns that need to consult for space management.

Conclusion:

This chapter gave some good inputs about I/O errors, 11g new features, ASM extent management , understanding of redundancy level and etc.

Chapter 5 – ASM Space Allocation and Rebalance

I particularly liked this chapter, it presented deeper insights about ASM Space allocation, operations, such as ASM AU (Allocation Units), ASM Extents distribution evenly among existing disks, how ASM Strips works, and difference between coarse and fine-grained striping types and which type of striping is good for what type of database files.

It also discussed when to change the default values of _asm_ausize (deault 1m) & _asm_stripesize (128k/1M) and how to set/change values for VLDs databases.

I also like the way author explained about ASM Rebalance operation concepts and the impact of setting various values to the ASM_POWER_LIMIT init parameters.

I really like the point “It is commonly believed that adding and dropping a disk against the same diskgroup required two separate rebalance activities. In fact, disks can be dropped and then added and a single rebalance command can be evenly distribute the I/O”. It shows how to avoid two rebalancing operations on this occasion.

A 11g new feature Fast Rebalancing which overhead of ASM-to-ASM and ASM-toRDBMS messaging also been discussed.

Conclusion:

I really enjoyed reading this chapter which gave me great insight of ASM extent mechanism, rebalancing and striping internals. It also provides few very useful scripts.

Chapter 6 – ASMLIB Concepts and Overview:

Since I have not worked much on Linux, I have quickly reviewed this chapter about ASMLIB API which is currently available on Linux OS.

This chapter explains the benefits of ASMLIB API, and the following functionality of ASMLIB:

Device Discovery
I/O processing
Performance and reliability

Explained ASMLIB configuring, Managing disk in ASMLIB, Upgrading ASMLIB, Migrating to ASMLIB and troubleshooting ASMLIB.

Chapter 7 – ASM Files, Alias and Templates

This chapter explains about the default ASM Filenames, creating tablespaces, ASM Alias and Templates.

Conclusion:

Nothing new offer in this chapter.

Chapter 8 – ASM Command-Line

This chapter explained about the ASMCMD (unix based) utility and its usage. It starts explaining existing means to manage and access ASM and its components.

I really liked the following 11g new features:

Bad Block Remapping in ASMCMD. Starting with 11g, when ASM redundancy is used, it is possible to recover bad sot by restoring the data from their mirror copy using the ‘remap’ command.

Note : ‘remap’ don’t fix logical errors or corruptions detected by the RDBMS.

ASMCMD Copy:

Starting with 11g, moving datafiles in and out of ASM made easy with “copy” command introduced in ASM.

Conclusion:

11g couple of new features about ASMCMD utility worth to know.

Chapter 9 – Managing Databases in ASM

This chapter started on a note on how to use ASM while creating databases and covered some good points about the internal functionality of CSS (Cluster Synchronize Services) daemon/process, how RDBMS interacts with ASM, important background process (role of RBAL, ASMB & 00nn b/g), few vital parameters that influence ASM performance (like, processes + add 16, large_pool add additional 60k and shared_pool size additional memory to store extent maps and concluded with good examples of converting ASM no non-ASM and vice-versa.

Conclusion:

Explained well about CSS internal behavior, how RDBMS communicates with ASM instance, converting instances from ASM-to-Non ASM and vice versa.

Chapter 10 – ASM Persistent Data Structures

Stars explaining about MEATADATA and where does ASM stores the metadata, and different types of metadata. ASM has two main classes of metadata, physical and virtual metadata.

Physical metadata which located at fixed locations on disk contains the following:

Disk Header
Allocation Table(AT)
Free Space Table (FST)
Partnership Status Table (PST)

Virtual Metadata stored in ASM files and contains the following:

File Directory
Disk Directory
Active Change Directory (ACD)
Continuing Operations Directory (COD)
Template Directory
Alias Directory
Attribute Directory
Staleness Directory
Staleness Registry

Chapter 11 – ASM Operations

This chapter is one of those chapters I liked in this book. It really explains how ASM works internally on the following condition:

First time when RDBMS instance tries to access an ASM file.
Operations on ASM Files
File creation
File open
File input/output (I/O)
File close
File delete

It also talk about the variable sized extents and Striping types (such as Coarse and fine-grind), Resync, Rebalancing and ASM Instance Recovery & Crash Recovery concepts.

Conclusion:

This is one of interesting chapters present in this book which offers an internal mechanism of ASM functionality on the above mentioned operations.

On APPENDIX

Appendix A: explained about ASM and Storage Array Configurations.

Appendix B : Troubleshooting – In page, it discussed about ASM startup issues, Diskgroup mounting issues, Space related Issues and a walkthrough of an ASM alert log.

Appendix C: Scripts and Tips - Automatically Starting up and Shutting down ASM instance in Non-RAC env.

Migrating Individual Non-ASM datafiles to ASM.

Overall thoughts:

This book is recommended for someone who serious wants to study about ASM technology and its in & outs.

Having a copy of this book is better than scanning 100th of Oracle documents.

This book is summarized and explained all ASM contents to a great extent.

For me, I liked 4 chapters in this books and might need another read.

Happy Reading

Jaffar

Why should we apply 10.2.0.4 patch set?

2008-02-25T08:41:00.000-08:00

At least for us, it is not really necessary to apply every latest Oracle patch set on our databases unless we really need them. After knowing about the release of 10.2.0.4 patch set from Laurent and Mr.Doug's blogs, I thought of reading a metalink note that describes the list of bugs/issues fixed by this patch set. (ML Note : 401436.1).

While reading the note, following bug fixes really caught my attention:

- SYSTEM RESET of an spfile parameter can corrupt the SPFILE
- /*append parallel*/ can corrupt an index

-deadlock hang on "cursor: pin S wait on X" (typically from DBMS_STATS)

I have come across of many DBAs complaing about this behavior. Few links:

http://momendba.blogspot.com/2007/06/cursor-pin-s-wait-on-x-wait-event-in.html
http://www.dbaportal.eu/?q=node/49/print

I am really interested in the following bugs fix:

+ Higher CPU / Higher "cache buffer chains" latch gets / Higher "consistent gets" after truncate/Rebuild.

Jonathan Lewis and others have talked about this bug at their respective blogs

http://jonathanlewis.wordpress.com/2008/02/09/index-rebuild-10g/

Few other bugs fixed in this patch set:
+ Wrong results with bind variables/CURSOR_SHARING
5079978 High US enqueue contention in RAC
5387030 Automatic tuning of undo_retention causes unusual extra space allocation
5439554 "buffer busy wait" timeouts with automatic undo management and in memory undo
5442919 Expired extents not being reused (ORA-30036)
5512921 Instance crash caused by SMON OERI[kcblus_1] / dump

It been really a long list of bugs fixed in this patch set. From my personal experience prospective, I would say, first study the bugs fixed in the patch set before applying on a production database. If all resolving majority of your issues.

Happy reading,

Jaffar

1Z0-048 Oracle Database 10g: Real Applications Clusters Administrator Certified Expert - CLEARED

2008-01-24T01:03:00.000-08:00

Well, this morning I have passed the 1ZO-048 RAC expert exam.

Let me reveal about my preparations before my mail box gets filled by hundreds of mails asking me to provide documents and etc stuff. For me, Oracle RAC documentation, Oracle university (RAC course) material and of course the extensive production practice knowledge of RAC certainly helped me a lot to prepare and pass this exam. I am sorry, I can't offer anything more than this.

After the exam, a small thought comes into my mind. Since this exam comes under EXPERT category, I thought, it would have been more better doing a day practical lab exam than simply doing the multiple choice questioning exam.

Have nice time,

Jaffar

Odd issues while preparing Physical Standby

2007-12-20T11:25:00.000-08:00

After an uphill task of upgrade/migration of 2.5TB sized data warehouse Oracle database from 9i to 10g and then from one OS to another (cross platform), the subsequent challenge is building a DR (Standby Database) for this database.

The DR (Disaster Recovery) site is around 3 km away from the HO. Since the size of the database is huge, we thought of moving tapes (physically) or duplicating the backup tapes from HO to the DR site, but, none of the ideas was materialized due to few technical/ un-technical difficulties. Fortunately, the speed of the lease line (leaser link) between the two sites is good enough and we decided trigger the duplicate database (standby) command from HO.

Ufff.. It took 3 days to finish the restore. However, the subsequent recovery on the standby database failed by the following problem:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 12/19/2007 05:16:32
RMAN-03015: error occurred in stored script Memory Script
RMAN-06136: ORACLE error from auxiliary database: ORA-19625: error identifying file /datafile1.dbf
ORA-27041: unable to open file
IBM AIX RISC System/6000 Error: 22: Invalid argument
Additional information: 2

Surprisingly, the datafile, datafile1.dbf (filename is changed here) which Oracle complaints was exists on the mount point. Then, thought of starting the MRP (recover managed standby database disconnect from session), unfortunately, the MRP terminated by the following errors:

Errors in file /dbdata/oradba/admin/OFDMP/bdump/ofdmp_dbw0_1077358.trc:
ORA-01157: cannot identify/lock data file 1479 - see DBWR trace file
ORA-01110: data file 1479: '/mountpoint/
datafile2.dbf'

Again, the complained file exists on the mount point, however, this time the problem was that the file has ‘carriage return’ before the name, (this may happen when you try to create/add a datafile from OEM). The workaround was following:

mv /mountpoint/ datafile2.dbf’ /mountpoint/datafile2.dbf

and then, change file name in the controlfile with the following command:

(make sure the standby_file_management is set to MANUAL on the standby).

alter database rename file ‘/mountpoint/ datafile2.dbf’ /mountpoint/datafile2.df’;

After the above workaround, the MRP (Media Recovery Process) was started successfully and looking for the archive gap.

When I tried to restore required archived log on the standby database from the backups (archived backups), the restored command failed with the following:

I thought of just testing the restore on production, the restore (tried t restore single archive log) was successfully.

I lately realized that the missing archives are generated after creating the standby controlfile and obviously the standby controlfile was not having the information about these archived logs. (Backups were performed with nocatalog option).

Following is the workaround:

1. Create a catalog and register production database.

2. On standby, using the catalog (rman target / catalog). Now, standby can get the info about the backups through the catalog.

3. Restoring required archived logs.

Earlier, I have done dozens of DG configuration facing different challenges. Well, everything it’s a new challenge and learning experience.

Happy reading,

Jaffar

Simulating 11g Snapshot Standby Database feature on Oracle 10g?

2007-12-10T12:11:00.000-08:00

As we all knew that the Oracle 11g improved the capabilities of standby database immensely, where a physical standby database can easily open in read-write mode, which can be ideally suitable for test and development environments. At the same time, it maintains protection by continuing to receive data from the production database, archiving it for later use.

What if you want to achieve the same on Oracle 10g? Well, I absolutely don’t have any clue about others, but, we have come across of such situation couple of days ago when our DR (Disaster Recovery Solution) team came to us with a request to test our standby database. They want the standby database in read write mode to do some real scenario tests and once the testing is done, they want the database to be back to standby mode.

We initially said, we can open the database in read only mode for their testing, but, the requirement demands the database to be in read write mode. We thought, we can break the standby database for their testing and once the testing is done, we can rebuild the standby database again. We know that this is very well possible with Oracle 11g but not with Oracle 10g. My colleague, Mr. Asif Momen, did some R&D come up with a solution where a Oracle 10g standby database can open in read write mode and can also be reverted back to standby mode.

The procedure as follows:

1. Set the following parameters on the standby database:

db_recovery_file_dest_size & db_recovery_file_dest

- Make sure the values are reflected.

2. Stop the media recovery process, if active.

3. When the standby is in MOUNT mode, Create a guaranteed restore point:

CREATE restore point before_rw guarantee flashback database;

3. Stop the log shipping on the primary database. (for safer side)

alter system archive log current;

alter system set log_archive_dest_state_2=DEFER;

4. Failover the standby database using the following command:

ALTER DATABASE ACTIVATE STANDBY DATABASE;

-Make sure the media recovery process is turned off

-Minimize the protection mode to MAXIMUM PERFORMANCE, if the mode is set other than the MAXIMUM PERFORMANCE.

5. Open the database (read write mode).

AT THIS POINT, YOU CAN USE THIS DATABASE AS NORMAL READ WRITE DATABASE.

Reverting the database back to standby mode:

Shutdown the database
Startup database in mount mode
Flash back database to restore point using the following:

FLASH BACK DATABASE TO RESTORE POINT before_rw;

Convert the database back to standby mode using the following:

ALTER DATABASE CONVERT TO PHYSICAL STANDBY;

Shutdown the standby database and remove the previously set parameters.
Start the standby database in mount state and drop the restored point.
Enable the Media Recovery on the Standby database.
Activate log shipping on the primary using the following:

ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENALE;

Since, we have done very little changes in the database, after converting to read write mode, the time which took to revert the database back to standby mode took few minutes only. Well, it is definitely need to be seen the time that take during the conversion to standby mode after huge changes in the read write database.

It worked well with us and Mr. Asif definitely deserved an appreciation

The tests have been carried out on AIX 64 bit with Oracle 10.2.0.3 release.

In the Part II, I will be posting my testing.

Happy Reading,

Jaffar

Back from vacation

2007-11-12T08:51:00.000-08:00

I just back to office after a month vacation and going through the back log/pending work. Currently, having an uphill task of upgrading an Oracle 9i data warehouse database (3 TB size) to Oracle 10g and also migrating this database from HP to AIX.

Its been nearly two months that I didn't write anything to my blog also didn't participate in any kind of database forums as I was gearing up for my every first seminar (RMAN). I have successfully delivered my one exclusive RMAN seminars in Sweden and Denmark respectively in September month. Good amount of participants were present, 20 participants in Sweden and 28 participants in Denmark. Although, the seminars went well, I felt still there was a lot room for improvement. I was bit worried about the time constraint to complete the seminar. In the modified RMAN seminar contents, now, more practical examples with 11g RMAN new features and changes are also added.

The first thing which I did was installing 11g on Windows OS in my PC. I am more interested to test RMAN changes and testing Active Standby Database(snapshot database) in 11g database. I am also busy in preparing a new one day exclusive seminar on High Availability Data Guard.

In my next blog, I am going to discuss about my experience with Active Standby Database and findings.

Thank for your time,

Jaffar

11g Database Creation with DBCA

2007-08-27T23:51:00.000-07:00

After successfully completing Oracle 11g on Red Hat Linux(v4), I thought of creating a new database using DBCA utility instead creating manually. While using DBCA utility to create a new database, I have found following three new changes:

1. If no listener is configured, DBCA utility prompts a message to configure a listener first.
2. Security Settings gives the following options:
- Keep the enhanced 11g default security settings(recommended)These settings include
enabling auditing and a new default password profile.
- Revert to pre-11g default security settings
3. Automatic maintenance task tasks
such as optimizer statistics collection and proactive advisor reports.
The default maintenance windows are 10:00 pm to 2:am on weekdays and all weekend long.

When I count records from v$system_event, surprisingly, I found only 77 records? Is it okay?

Jaffar

11g 'hot patching'

2007-08-27T00:29:00.000-07:00

One feature which caught my extreme attention while installing 11g was 'hot patching' feature.

Oracle says: ''hot patching,' which improves system availability by allowing database patches to be applied without the need to shut databases down."

For me, it is really hard to digest about upgrading (applying patches) while they are currently running. However, if this is believed to be true, then, definitely this feature going to resolve the hassles of shutting down the production databases while applying patches.

Jaffar

11g Installation

2007-08-26T23:44:00.000-07:00

Last night I was able to successfully installed Oracle 11g on Red Hat Linux(v4). The only problem I have faced during the installation stage (the prerequisite check stage) was about missing Red Hat rpms (three) and minimum memory required for installation(I have installed on a virtual machine with 700+M) .

I am very inquisitive to test RMAN enhancements and also Active Standby Database feature.

Jaffar