Update 2015/03/06: The following has been recognized as “unpublished BUG 17045279 - ASM_PREFERRED_READ DOES NOT WORK WITH FLEX ASM”, which is planned to be fixed in the next upcoming release.

Update 2017/05/20: As of 12.2, preferred reads are site-aware (extract of  Markus Michalewicz presentation available here) so that the issue described into this blog post has been addressed.

 

Introduction

As you know Oracle 11g introduced a new feature called “Asm Preferred Read”. It is very useful in extended RAC as it allows each node to define a preferred failure group, allowing nodes to access local failure groups in preference to remote ones. This is done thanks to the “asm_preferred_read_failure_groups” parameter.

Fine, but remember:

  1. This parameter has to be set at the ASM instance level (not the database instance one).
  2. This is the database instance (or its shadow processes) that is doing the IOs (not the ASM instance).

Why is it important ?   Because with Flex ASM in place, database instances are connection load balanced across the set of available ASM instances (that of course are not necessary “local” to the database instance anymore):

flex_asm1

And then you could hit what I call the “unpreferred” read behavior.

Let me explain more in depth with an example:

Suppose that you have an extended 3 nodes RAC:

  • racnode1 located in SITE A
  • racnode2 located in SITE A
  • racnode3 located in SITE B

And 2 ASM instances actives:

  • +ASM1 located in SITE A
  • +ASM3 located in SITE B
srvctl status asm
ASM is running on racnode3,racnode1

So you created 2 failgroup SITEA and SITEB and you set the asm_preferred_read_failure_groups parameter that way for the DATAP diskgroup:

SQL> alter system set asm_preferred_read_failure_groups='DATAP.SITEB' sid='+ASM3';
System altered.

SQL> alter system set asm_preferred_read_failure_groups='DATAP.SITEA' sid='+ASM1';
System altered.

So that ASM3 prefers to read from SITEB and ASM1 from SITEA (which fully makes sense from the ASM point of view).

But what if  a database instance located into SITEB (racnode3) is using ASM1 located in SITEA ?

SQL>  select I.INSTANCE_NAME,C.INSTANCE_NAME,C.DB_NAME
  2  from gv$instance I, gv$asm_client C 
  3  where C.INST_ID=I.INST_ID and C.instance_name='NOPBDT3';

INSTANCE_NAME    INSTANCE_NAME                                                    DB_NAME
---------------- ---------------------------------------------------------------- --------
+ASM1            NOPBDT3                                                          NOPBDT

As you can see the NOPBDT3 database instance is using the +ASM1 instance, while the NOPBDT3 database instance is located on racnode3:

srvctl status instance -i NOPBDT3 -d NOPBDT
Instance NOPBDT3 is running on node racnode3

Which means NOPBDT3 located into SITEB will prefer to request read IO from SITEA which is of course very bad.

Let’s check this with my asmiostat utility and Kevin Closson’s SLOB2 kit:

Let’s launch SLOB locally on NOPBDT3 only:

[oracle@racnode3 SLOB]$ ./runit.sh 3
NOTIFY: 
UPDATE_PCT == 0
RUN_TIME == 300
WORK_LOOP == 0
SCALE == 10000
WORK_UNIT == 256
ADMIN_SQLNET_SERVICE == ""
ADMIN_CONNECT_STRING == "/ as sysdba"
NON_ADMIN_CONNECT_STRING == ""
SQLNET_SERVICE_MAX == "0"

And check the IO metrics with my asmiostat utility that way (I want to see Instance, Diskgroup and Failgroup):

./real_time.pl -type=asmiostat -show=inst,dg,fg -dg=DATAP

With the following output:

flex_asm_pref_read_12c

As I am the only one to work on this Lab, you can see with no doubt that the IO metrics coming from the Instance NOPBDT3 are recorded into the ASM instance +ASM1 and clearly indicates that the read IOs have been done on SITEA.

How can we “fix” this ?

You can temporary fix this that way (connected on the +ASM1 instance):

SQL> ALTER SYSTEM RELOCATE CLIENT 'NOPBDT3:NOPBDT';
System altered.

SQL> select I.INSTANCE_NAME,C.INSTANCE_NAME,C.DB_NAME
  2  from gv$instance I, gv$asm_client C 
  3   where C.INST_ID=I.INST_ID and C.instance_name='NOPBDT3';

INSTANCE_NAME    INSTANCE_NAME                                                    DB_NAME
---------------- ---------------------------------------------------------------- --------
+ASM3            NOPBDT3                                                          NOPBDT
+ASM3            NOPBDT3                                                          NOPBDT

That way the NPPBDT3 database instance will use the +ASM3 instance and then will launch its read IO on SITEB .

But I had to bounce the NOPBDT3 database instance so that it launchs the read IO from SITEB (If not it was still using SITEA, well maybe a subject for another post)

Conclusion:

Flex ASM is a great feature but you have to be careful if you want to use it in an extended Rac with preferred read in place.  If not you may hit the “unpreferred” read behavior.

UPDATES:

unpref_12102