Ami's Den: RAC

Showing posts with label RAC. Show all posts

Thursday, 18 September 2025

RAC Services Startup Sequence

When an Oracle RAC cluster is started, the components must come up in a specific dependency order:

🧩 Step-by-Step Startup Order

Order	Component	Description	Command (if manual)
1️⃣	OHAS (Oracle High Availability Services)	Base daemon on each node. Brings up local agents and monitors.	`crsctl start has`
2️⃣	CSSD (Cluster Synchronization Services Daemon)	Handles node membership and cluster heartbeat.	Auto via OHAS
3️⃣	CRSD (Cluster Ready Services Daemon)	Manages cluster resources (like databases, services, VIPs).	Auto via OHAS
4️⃣	EVMD (Event Manager Daemon)	Manages cluster events and notifications.	Auto via OHAS
5️⃣	ASM Instances	Provides shared storage (if using ASM). Must start before any database instances.	`srvctl start asm -n <node>`
6️⃣	Listener(s)	Handles client connections. Needed before database instances start.	`srvctl start listener -n <node>`
7️⃣	Database Instances	Each RAC instance on each node. They mount and open the shared database.	`srvctl start instance -d <db> -i <inst>` or `srvctl start database -d <db>`
8️⃣	RAC Services (DB Services)	Logical services running on top of the database. Enable workload balancing, failover, etc.	`srvctl start service -d <db> -s <service>`

RAC Startup Sequence

In an Oracle RAC environment, multiple instances start up and mount one shared database. The startup is coordinated by Oracle Clusterware.

🧩 RAC Startup Steps

1. Clusterware Startup (Infrastructure Layer)

Before any database instance starts, Clusterware (CRS) must be running on all nodes.
Clusterware includes:
- Cluster Ready Services (CRS)
- Oracle Cluster Synchronization Services (CSS)
- Oracle High Availability Services (OHASD)

Commands:


crsctl start crs     # Start Clusterware stack on a node
crsctl check crs     # Verify status

2. Automatic Startup of RAC Instances

Once CRS is up, it will start:
- Oracle ASM instances (if used)
- Listener
- Database instances (configured as resources in CRS)

Commands:


srvctl start asm -n <node_name>         # Start ASM on a specific node
srvctl start listener -n <node_name>    # Start listener on a node
srvctl start database -d <db_name>      # Start all RAC instances

Alternatively, to start an instance manually (on one node):


sqlplus / as sysdba
STARTUP

⚙ RAC Instance Startup Phases

Each RAC instance goes through these standard Oracle startup stages:

Stage	What Happens
STARTUP NOMOUNT	Allocates SGA, starts background processes. Uses init.ora / spfile
STARTUP MOUNT	Reads control files from shared storage. Opens redo threads.
STARTUP OPEN	Opens datafiles and online redo logs, database available to users

At MOUNT, the instance joins the cluster group using Global Enqueue Service (GES) and Global Cache Service (GCS).
At OPEN, the database becomes fully accessible.

🔁 Typical RAC Startup Flow


[Start Clusterware] 
      ↓
[Start ASM Instances] 
      ↓
[Start Listeners] 
      ↓
[Start RAC Instances (Node1, Node2,...)]
      ↓
[Each instance does: NOMOUNT → MOUNT → OPEN]
      ↓
[Clustered Database becomes available]

📌 Useful RAC Startup/Shutdown Commands

Task	Command
Start whole RAC DB	`srvctl start database -d <db_name>`
Stop whole RAC DB	`srvctl stop database -d <db_name>`
Start one instance	`srvctl start instance -d <db_name> -i <inst_name>`
Stop one instance	`srvctl stop instance -d <db_name> -i <inst_name>`
Check database status	`srvctl status database -d <db_name>`

Components of RAC

1. Cluster Nodes (Servers)

Multiple physical or virtual servers (nodes) work together as a cluster.
Each node runs its own Oracle Instance (SGA + background processes).
All nodes access the same database files.

2. Shared Storage

All RAC nodes must see the same storage.
Contains:
- Datafiles
- Control files
- Redo log files
- Archive logs
Usually implemented using SAN / NAS / ASM (Automatic Storage Management).

3. Cluster Interconnect (Private Network)

High-speed private network between all nodes.
Used for:
- Cache Fusion (transferring data blocks between instances)
- Global resource coordination
Requires low-latency and high-bandwidth network (typically 10/25/40GbE or Infiniband).

4. Oracle Clusterware

Provides cluster management and node membership.
Ensures only active nodes access the shared storage.
Manages:
- Node heartbeat monitoring
- Automatic failover
- VIP (Virtual IP) configuration

5. Global Cache Service (GCS) & Global Enqueue Service (GES)

Ensure data consistency across instances.
Cache Fusion: If a block is modified on one node, other nodes get the updated version via interconnect (not disk I/O).
GES manages locks, GCS manages cache coherency.

6. Oracle RAC Database

One database (shared) accessed by multiple instances.
Appears as a single database to applications.
Supports load balancing and failover through services.

📊 RAC Architecture Diagram


          +-------------+           +-------------+
          |   Node 1    |           |   Node 2    |
          |-------------|           |-------------|
          | Oracle Inst |           | Oracle Inst |
          |   SGA/Proc  |           |   SGA/Proc  |
          +------|------+           +------|------+
                 |                         |
        ======= Private Interconnect =======
                 |                         |
          +---------------------------------------+
          |         Shared Storage (ASM)          |
          |  Control/Data/Redo/Archive Files       |
          +---------------------------------------+

⚙ Benefits of RAC

High Availability: Survives node failure automatically.
Scalability: Add more nodes to increase capacity.
Load Balancing: Workload distributed among nodes.
Fault Tolerance: Shared database remains available if a node fails.

Wednesday, 13 August 2014

Voting Disk and Oracle Cluster Registry

1. Voting Disk: - Manages cluster membership by way of a health check and arbitrates cluster ownership among the instances in case of network failures. RAC uses the voting disk to determine which instances are members of a cluster. The voting disk must reside on shared disk. For high availability, Oracle recommends that you have multiple voting disks. The Oracle Clusterware enables multiple voting disks.

2. OCR File :- Cluster configuration information is maintained in Oracle Cluster Registry file. OCR relies on a distributed shared-cache architecture for optimizing queries against the cluster repository. Each node in the cluster maintains an in-memory copy of OCR, along with an OCR process that accesses its OCR cache.
When OCR client application needs to update the OCR, they communicate through their local OCR process to the OCR process that is performing input/output (I/O) for writing to the repository on disk.
The OCR client applications are Oracle Universal Installer (OUI), SRVCTL, Enterprise Manger (EM), Database Configuration Assistant (DBCA), Database Upgrade Assistant(DBUA), NetCA and Virtual Internet Protocol Configuration assistant (VIPCA). OCR also maintains dependency and status information for application resources defined within CRS, specifically databases, instances, services and node applications.

Maintains cluster configuration information as well as configuration information about any cluster database within the cluster. The OCR also manages information about processes that Oracle Clusterware controls. The OCR stores configuration information in a series of key-value pairs within a directory tree structure. The OCR must reside on shared disk that is accessible by all of the nodes in your cluster. The Oracle Clusterware can multiplex the OCR and Oracle recommends that you use this feature to ensure cluster high availability.