Configuration Guide For Minimizing Disk I/O Contention#
Overview#
During operation, Altibase writes data to several kinds of files, including log files, checkpoint image files, and the data files of disk databases. A DBMS generally follows the WAL (Write Ahead Logging) protocol, which writes changes to the log file before it changes the data, and Altibase follows this approach as well. As transactions increase, writes to both the log files and the data files increase together. If these files share a single disk, disk I/O contention degrades performance.
This guide first describes where disk I/O occurs in Altibase and, based on that, presents how to distribute files across volumes. It then covers what to check for disks, file systems, and disk I/O settings.
This guide is based on Altibase 6.5 and later.
Info
For errors and improvements related to this document, please contact the technical support portal or technical support center.
- Technical support portal: http://support.altibase.com/en
- Technical support center: +82-2-2082-1114
Info
This document is provided for informational purposes only and is subject to change without prior notice. It may contain errors, and Altibase makes no express or implied warranties of any kind, including but not limited to warranties of merchantability or fitness for a particular purpose.
The development, release, and timing of any features or functionality described for Altibase products in this document are at the sole discretion of Altibase.
Altibase may hold patents, trademarks, copyrights, or other intellectual property rights related to the contents of this document.
Where Disk I/O Occurs#
In Altibase, disk I/O occurs when writing log files, during checkpoints of memory databases, during buffer flushes and buffer replacement of disk databases, and when writing to the undo tablespace. As the figure below shows, if this disk I/O is concentrated on a single disk, bottlenecks are unavoidable.
Log Files#
Altibase log files record every change made during transaction processing, that is, all information needed for data recovery, such as changes to resources and changes to data. Therefore, every change transaction writes to the log file.
Altibase first writes logs to a memory area mapped with mmap, and the log flush thread periodically saves them to the log file. The OS saves changes in the mmap-mapped memory to the file even if the process terminates abnormally. Therefore, logs are not lost unless a power failure occurs or the OS stops responding.
Altibase uses this method by default to reduce disk I/O and improve performance.
Checkpoints of Memory Databases#
A memory database loads all of its data into memory at startup and manages it in 32KB pages. When a transaction changes data, Altibase registers the page as a dirty page, and checkpoints periodically save these dirty pages to data files on disk (hereafter, checkpoint image files). Because memory is volatile, checkpoints are required to guarantee data durability. For details, see Checkpointing in the Altibase manual.
In other words, memory databases also write to disk during checkpoints, and the amount grows with the number of transactions. If the log files and the checkpoint image files share a disk, log file writes and checkpoint writes overlap and cause a bottleneck.
Buffer Flushes and Buffer Replacement of Disk Databases#
Disk databases generate disk I/O during buffer flushes and buffer replacement, which write changed pages in the buffer to disk. Therefore, if memory databases and disk databases share a disk, checkpoints of the memory database overlap with buffer flushes and buffer replacement of the disk database, and performance may degrade.
Because a disk database stores all of its data in data files on disk, Altibase keeps frequently accessed data in a memory buffer to reduce disk access. Altibase allocates the buffer at the size the user configures and manages it in 8KB pages. If the data a transaction needs is not in the buffer, Altibase loads it from the data file into the buffer.
Altibase registers pages whose data has changed in the buffer to the flush list and saves them to disk when it flushes. Also, when the buffer has no free space, Altibase uses the LRU algorithm to save rarely accessed pages to disk and then reuses that space. This process is called buffer replacement.
Undo Tablespace#
Disk databases use the in-place update method: they copy the original data to the undo tablespace and then apply the change directly to the original data area. When recovery is needed, Altibase copies the original data stored in the undo tablespace back to its original location. Therefore, every change transaction in a disk database writes to both the data file and the undo tablespace.
Volume Configuration#
Group the files described above by type, place each group on a different volume, and create each volume on a physically separate disk. Even if the system environment does not allow fine-grained separation, always place the log files on a disk physically separate from the other files.
Recommended Configuration#
This configuration separates every file type into its own volume and is the most recommended configuration.
| Classification | Volume (Mount Path) |
|---|---|
| ALTIBASE_HOME | /ALTIBASE |
| Log files | /ALTIBASE_REDO_LOG |
| Checkpoint image files | /ALTIBASE_MEMORY_DATA |
| Data files (disk database) | /ALTIBASE_DISK_DATA |
| Data files (disk indexes) | /ALTIBASE_DISK_INDEX |
| Undo tablespace data files | /ALTIBASE_DISK_UNDO |
ALTIBASE_HOME contains the binaries, headers, and libraries required for development and operation, and Altibase also writes important trace logs there during operation, so separate it into its own volume.
Separating disk databases and disk indexes minimizes disk I/O contention during operations such as data file expansion or database schema changes.
For disk considerations related to backup, see Considerations for Altibase Backup Policy.
Minimal Configuration#
This configuration separates only the log files. It is recommended when you mainly use memory databases or use only disk databases with few changes. You can also use it when the system environment provides only a few disks. In these environments, this configuration alone reduces disk I/O contention.
| Classification | Volume (Mount Path) |
|---|---|
| Log files | /ALTIBASE_REDO_LOG |
| ALTIBASE_HOME, checkpoint image files, data files | /ALTIBASE |
Configuration Separated by Workload#
This configuration is recommended when a disk database handles diverse workloads and large volumes of data.
Create separate tablespaces for complex queries and for simple queries, and place the data files of each tablespace on physically separate disks to distribute disk I/O. However, frequent buffer replacement prevents this configuration from delivering the expected benefit, so set the buffer size (BUFFER_AREA_SIZE) large enough for the frequently accessed data.
| Classification | Volume (Mount Path) |
|---|---|
| Log files | /ALTIBASE_REDO_LOG |
| ALTIBASE_HOME, checkpoint image files | /ALTIBASE |
| Data files (tablespace for complex workloads) | /ALTIBASE_DISK_COMPLEX |
| Data files (tablespace for simple workloads) | /ALTIBASE_DISK_SIMPLE |
Disk Configuration#
This section describes the basic considerations for configuring the disks on which the volumes are created.
Dedicated Storage for the Database#
Disk I/O from external sources such as the OS or applications must not affect database performance. Therefore, provide dedicated database storage that is physically separate from the disks used by the OS and applications.
RAID#
For the RAID (Redundant Array of Independent Disks) configuration of dedicated database storage, RAID 10 is commonly used because it provides both data protection and IOPS (Input/Output Operations Per Second). RAID 10 combines RAID 1 and RAID 0.
RAID 1 (mirroring) stores the same data on multiple disks, protecting against data loss from disk failure. RAID 0 (striping) splits data into blocks and distributes them across multiple disks, significantly increasing disk read and write speed.
The number of disks required depends on the RAID level, so choose an appropriate level in consultation with a storage expert, considering the database size and the number of available disks.
Disk Performance#
The required disk performance depends on system traffic, so determine it through peak traffic testing. For disk databases with frequent random I/O, SSDs with high random I/O performance are recommended.
File Systems#
This section describes the file systems Altibase supports, their characteristics, and the required settings.
Supported File Systems#
Altibase supports all major file systems that support mmap.
Depending on the file system, you may need to change mount options to use Direct I/O. A file system that does not support Direct I/O cannot open files with Direct I/O, so set LOG_IO_TYPE or DATABASE_IO_TYPE to 0, according to the type of files on that file system, to use Buffered I/O.
| OS | File System |
|---|---|
| Solaris | UFS, VxFS, ZFS |
| HP-UX | HFS, JFS, VxFS |
| AIX | JFS, VxFS |
| Windows | NTFS, FAT32 |
| Linux | Ext2, Ext3, Ext4 |
Unsupported File Systems#
Altibase does not support raw devices or file systems that do not support mmap.
Raw Storage Device#
Altibase cannot access files configured on raw devices, so you cannot create data files or log files on a raw device. A database uses raw devices to control I/O directly without going through the OS file cache; in Altibase, use a file system that supports Direct I/O instead.
File Systems That Do Not Support mmap#
On file systems that do not support mmap, such as some NFS (Network File System) and NAS (Network Attached Storage) file systems, Altibase cannot map log files into memory, so an error occurs during the database creation phase, which creates the data files and log files.
Disk I/O Optimization#
This section describes several ways to improve the disk I/O performance of storage.
File Cache#
The file cache is a system buffer managed by the operating system to reduce bottlenecks caused by the speed gap between main memory and secondary storage. Each operating system manages the file cache with its own policy, but if the file cache occupies a large portion of physical memory, the memory of other processes can be swapped out, so the file cache is directly related to swapping.
Swap lets the OS run programs larger than physical memory by moving rarely used memory pages to the swap area on disk (swap out). However, if the memory of a long-running program such as a DBMS is swapped out, the OS must read it back from disk when the program accesses it again (swap in), so response times become irregular and, in severe cases, the system may stop responding.
Therefore, to guarantee consistent transaction response times, configure the file cache and swap-related kernel parameters in advance so that the memory used by Altibase is not swapped out. For OS-specific settings, see HP-UX Setup Guide for Altibase, AIX Setup Guide for Altibase, and Linux Setup Guide for Altibase.
Direct I/O#
The OS keeps a file buffer cache in memory, which is faster than disk, and stores file blocks that were read or written there to reduce disk access. Buffered I/O on the left of the figure below works this way.
However, when an application such as a DBMS caches data itself, data moves from the disk to the file buffer cache and then again to the database buffer, which adds overhead. This is called double copying, and it consumes more CPU and memory. In this case, Direct I/O, which bypasses the OS file cache as shown on the right of the figure, reduces the CPU and memory usage of the DBMS.
When Direct I/O Is Advantageous#
Direct I/O can be advantageous when the database is larger than physical memory and the disk storage has a large buffer. For example, SAN storage often provides a large internal buffer.
Also, when bulk changes are frequent, checkpoints generate heavy disk I/O, and the cost of copying data twice, to the OS file cache and to the database buffer, can sharply increase CPU and memory usage. Direct I/O eliminates this double copying.
When Buffered I/O Is Advantageous#
In general, Buffered I/O performs better than Direct I/O on local disks. OS Buffered I/O reads several blocks at once (multi-block read) and fetches the needed disk pages in advance (prefetch), which increases disk read speed.
Direct I/O Settings#
If the OS or file system does not support Direct I/O, Altibase cannot use Direct I/O either. Some file systems require specific mount options to use Direct I/O. See the following table for the action required for each file system.
| OS | File System | Required Action |
|---|---|---|
| Solaris | UFS | None |
| VxFS | Mount with convosync=direct | |
| ZFS | Does not support Direct I/O | |
| HP-UX | HFS | None |
| JFS | None | |
| VxFS | Mount with convosync=direct | |
| AIX | JFS | Mount with -o dio |
| VxFS | Mount with convosync=direct | |
| Windows | NTFS | None |
| FAT32 | None | |
| Linux (kernel 2.4 or later) | Ext2/Ext3/Ext4 | None |
For Altibase to perform data file and log file I/O with Direct I/O, set the following properties.
DIRECT_IO_ENABLED = 1 # 0: Buffered I/O, 1: Direct I/O
DATABASE_IO_TYPE = 1 # 0: Buffered I/O, 1: Direct I/O
LOG_IO_TYPE = 1 # 0: Buffered I/O, 1: Direct I/O
Page Size#
Altibase writes data in pages. The page size is fixed and cannot be changed. Memory tablespaces use 32KB pages, and disk tablespaces use 8KB pages.
Altibase does not require the OS file system block size to match the page size. Because the page sizes are multiples of the common file system block size (4KB), you can use the default block size as is.