Responding to Failures Guide for Altibase#
Overview#
Failures caused by the system environment, user mistakes, or product defects cannot be completely prevented. This guide describes the minimum procedure that users should follow to get technical support when a failure interrupts service. To recover from failures as quickly as possible, it is recommended to become familiar with this guide in advance.
This guide does not cover root cause analysis of CPU usage spikes, memory usage growth, lock waits, or performance delays. For CPU usage spikes, see the Altibase CPU Usage Increase Analysis Guide; for memory usage growth, see the Altibase Memory Usage Increase Analysis Guide.
Info
For errors and improvements related to this document, please contact the technical support portal or technical support center.
- Technical support portal: http://support.altibase.com/en
- Technical support center: +82-2-2082-1114
Info
This document is provided for informational purposes only and is subject to change without prior notice. It may contain errors, and Altibase makes no express or implied warranties of any kind, including but not limited to warranties of merchantability or fitness for a particular purpose.
The development, release, and timing of any features or functionality described for Altibase products in this document are at the sole discretion of Altibase.
Altibase may hold patents, trademarks, copyrights, or other intellectual property rights related to the contents of this document.
Failure Definition and Classification#
Failure Definition#
Altibase classifies failures as urgent or non-urgent. The exact definition of each type should be agreed with the user, but in general, a state in which service cannot continue because of a system or Altibase problem is an urgent failure, and a state in which service continues but problems may occur is a non-urgent failure.
Altibase provides technical support for urgent failures according to the maintenance contract. Users handle non-urgent failures first, and Altibase provides guides and technical support so that users can deal with them.
Urgent Failures#
Urgent failures are divided by cause into system failures and Altibase failures. A system failure is a service interruption caused by a hardware or software problem in the system, and an Altibase failure is a service interruption caused by a defect in Altibase.
When an urgent failure occurs, users must provide the following information to Altibase. Sending this information to the technical support portal in advance helps the engineer in charge identify the cause.
| Information | How to obtain |
|---|---|
| System logs | OS-specific system logs below |
| Altibase trace logs | All files in the $ALTIBASE_HOME/trc directory (see Altibase Trace Logs) |
| Details at the time of failure | Abnormal symptoms and database operations at the time of failure |
$ALTIBASE_HOME is the installation path of Altibase. You can check it with the following command in the user account where Altibase is installed.
$ echo $ALTIBASE_HOME
The system logs for each OS are located as follows.
| OS | System log |
|---|---|
| Linux | /var/log/messages file |
| AIX | Output of the errpt -a command |
| Solaris | /var/adm/messages file |
| HP-UX | /var/adm/syslog/syslog.log file |
Procedure for Urgent Failures#
When an urgent failure occurs, follow the steps below.
Checking Altibase Status and Connection#
First, check whether Altibase is running with the following command. If it is running, one Altibase process is displayed.
$ ps -ef | grep "altibase -p boot from" | grep -v grep
If Altibase is running, check whether you can connect with iSQL.
$ is
# or
$ isql -u [db user id] -p [db user password] -s 127.0.0.1 -port [port_no]
# Example
$ isql -u sys -p manager -s 127.0.0.1 -port 20300
Requesting Technical Support#
If Altibase is not running or you cannot connect, collect the information described in Urgent Failures and submit it to the technical support portal. Also contact the Altibase technical support center (+82-2-2082-1114).
Restarting Altibase#
To restart Altibase, run the following commands.
# Forcibly stop Altibase.
$ server kill
# Start Altibase.
$ server start
Response by Failure Type#
Connection Failure#
Sometimes you cannot connect to the database even though the server and the Altibase process are working normally. The causes are resource limits of the user account, incorrect connection information, network failure, file write failure, and Altibase not responding. The first four are problems in user settings or the system environment, so users check and fix them directly. If Altibase is not responding, collect the required information and request technical support.
Resource Limits of the User Account#
If a connection attempt exceeds the number of file descriptors allowed for the user account, the connection fails and the following errors may be recorded in altibase_boot.log.
ERR-01052(errno=24) Unable to invoke open() function on [~~~]
ERR-71016(errno=24) Failed to invoke a system function, accept() Dispatcher failed callback
errno=24 is the system error (Too many open files) that means the limit on the number of files that can be open at the same time has been exceeded. Check the current value with ulimit -n, increase it, and restart Altibase. It is recommended to set the number of file descriptors to unlimited, and it must be at least 4096.
Incorrect Connection Information#
If the IP address, port number, user account, or password of the server is entered incorrectly, the connection fails. The same applies when the account is locked by the password management policy or TCP connections are restricted. In these cases, errors such as the following occur.
ERR-50032 : Client unable to establish connection.
ERR-31010 : User not found
ERR-4102E : Invalid password
ERR-31370 : The account is locked.
ERR-410E3 : The user cannot connect using TCP.
Check the connection settings again. In particular, check that you are connecting to the same port as PORT_NO in $ALTIBASE_HOME/conf/altibase.properties.
Network Failure#
If there is a problem with the network card or network settings, the connection fails. First, check for packet errors with netstat. Then check whether another machine can connect to the Altibase machine normally with ftp, telnet, and so on, and whether packet transfer performance is significantly lower than usual.
File Write Failure#
If files cannot be written because of insufficient disk space, the connection also fails, and errors such as the following may be recorded.
ERR-01052(errno=28) Unable to invoke open() function on [~~~]
ERR-01052(errno=28) Unable to invoke write() function on [~~~]
For how to check and fix this, see Insufficient Disk Space.
Altibase Not Responding#
If the Altibase process exists but connection attempts fail and already connected sessions do not respond either, suspect that the system or Altibase is hung. In this case, immediately collect the following information and send it with the system logs and Altibase trace logs to request technical support.
Run the stack command for your OS three times at 30-second intervals. Specify the ID of the Altibase process as process_id. These commands show the current state of every thread in the process, which is important information for solving the problem. For details on the commands, see the Utility Guide for each OS for Problem Analysis.
Linux#
pstack may not be installed, depending on the system.
$ /usr/bin/pstack process_id > 1.txt
$ /usr/bin/pstack process_id > 2.txt
$ /usr/bin/pstack process_id > 3.txt
AIX#
$ /usr/bin/procstack -F process_id > 1.txt
$ /usr/bin/procstack -F process_id > 2.txt
$ /usr/bin/procstack -F process_id > 3.txt
Solaris#
$ /usr/sbin/pstack -F process_id > 1.txt
$ /usr/sbin/pstack -F process_id > 2.txt
$ /usr/sbin/pstack -F process_id > 3.txt
HP-UX#
This is available only on IA systems and is not supported on PA-RISC.
$ /usr/ccs/bin/pstack process_id > 1.txt
$ /usr/ccs/bin/pstack process_id > 2.txt
$ /usr/ccs/bin/pstack process_id > 3.txt
Insufficient Physical Memory#
When physical memory runs short, an Out of memory error may be recorded in the Altibase trace logs or system logs. In this case, check the free memory of the system with the commands for your OS. Because the method differs by OS, see Step 1: Check System Memory and Swap in the Altibase Memory Usage Increase Analysis Guide. Then use the following query to check which modules use the most memory inside Altibase.
SET LINESIZE 1000
SELECT * FROM V$MEMSTAT ORDER BY MAX_TOTAL_SIZE DESC;
Memory usage can also grow because of large update operations or long-running queries. Altibase controls concurrency with MVCC, so it cannot clean up data marked for deletion until these transactions end. For the causes of memory usage growth and how to resolve them, see the Altibase Memory Usage Increase Analysis Guide. For queries that check long-running queries and the garbage collection status, see [ST05] Long-Running Queries and Memory Database GC in the Altibase Monitoring Queries Guide.
Insufficient Disk Space#
When disk space runs short, Altibase cannot write the online log files required for transaction processing, so the database may appear to hang. An errno=28 (No space left on device) error is recorded in the trace logs, but if even the trace logs cannot be written, no error message may remain. Check the disk usage with a command such as df (bdf on HP-UX) and free up space.
Caution
When freeing up disk space, do not delete the Altibase online log files (logfile). If online log files are deleted arbitrarily, the database can no longer be started or recovered.
If the online log files have suddenly increased, Altibase may be unable to delete them because of large update operations, long-running queries, or replication delay. For queries that check the log file status and long-running queries, see Log File and [ST05] Long-Running Queries in the Altibase Monitoring Queries Guide. For replication delay, see Transmission Problems.
Insufficient Tablespace#
If an error occurs because a tablespace has unexpectedly run out of space, take the emergency action below. After that, find and remove the cause of the sudden growth. Check the usage changes of the objects in each tablespace; for queries that check usage per object, see Tablespace in the Altibase Monitoring Queries Guide.
Insufficient Memory Tablespace#
When a memory tablespace runs short, the following error occurs.
[ERR-110F1 : Unable to extend the tablespace(XXXXX) because the current size of tablespace(4194304K) becomes larger than MAXSIZE(4194304K) of the tablespace.]
For a user memory tablespace, run the following commands in order to increase MAXSIZE. Use the tablespace name shown in the error message, and set MAXSIZE larger than the current MAXSIZE.
ALTER TABLESPACE [tablespace_name] ALTER AUTOEXTEND OFF;
ALTER TABLESPACE [tablespace_name] ALTER AUTOEXTEND ON MAXSIZE 1G;
If it is a system memory tablespace such as SYS_TBS_MEM_DATA or SYS_TBS_MEM_DIC, or if the total size of all memory tablespaces has already reached MEM_MAX_DB_SIZE, the tablespace cannot be extended. In this case, delete unnecessary data in the tablespace with DELETE to reuse space within the same tablespace. DELETE does not allocate new pages, so it can run even when space is insufficient. The deleted space can be reused after garbage collection completes, and running COMPACT returns the empty pages of the table to the tablespace so that other tables can use them.
-- Delete unnecessary data.
DELETE FROM [table_name] WHERE [condition];
-- Return the empty pages of the table to the tablespace.
ALTER TABLE [table_name] COMPACT;
TRUNCATE internally creates the table again, so it may fail if space cannot be allocated. If this does not solve the problem, increase MEM_MAX_DB_SIZE in $ALTIBASE_HOME/conf/altibase.properties and restart Altibase.
Insufficient Disk Tablespace#
When a disk tablespace runs short, the following error occurs.
[ERR-11123 : The tablespace does not have enough free space ( TBS Name :XXXXX ).]
Add a data file with the following command. Use the tablespace name shown in the error message. Name the data file according to your naming rules, and set SIZE appropriately by referring to the size of the other data files in the same tablespace.
ALTER TABLESPACE [tablespace_name] ADD DATAFILE 'abcd.dbf' SIZE 1G AUTOEXTEND OFF;
Replication Failure#
Replication failures are divided into transmission problems, in which the Sender or Receiver does not work normally, and data conflicts, in which data cannot be replicated because the data on the two databases differs. Altibase records replication messages in altibase_rp.log and, depending on the settings, records data conflicts separately in altibase_rp_conflict.log.
Transmission Problems#
First, check whether the Sender and Receiver are running. If they are working normally, the results of the following queries must equal the number of replication objects.
-- Number of Senders
SELECT COUNT(*) FROM V$REPSENDER;
-- Number of Receivers
SELECT COUNT(*) FROM V$REPRECEIVER;
Then check the transmission status in altibase_rp.log. The main messages are as follows.
| Situation | altibase_rp.log message |
|---|---|
| Replication started (sending server) | [Recovery Sender] Replication REP1 Start... at [6030857] |
| Replication started (receiving server) | [Receiver] Replication REP1 Started ... |
| Sender failed to connect to the peer server | ERR-61012(errno=111) [Sender] Failed to connect to the peer server |
| Receiver ended | ERR-6104b(errno=0) [Receiver] REP1 receiver is ended (by thr_exit) |
| Replication stopped normally on the peer server | RECEIVER:REPLICATION STOP MSG arrived! |
The error in which the Sender fails to connect to the peer server is caused by a network problem or a problem with the Receiver on the peer server. When such an error is recorded, check whether a user ran an intended command at that time or whether it was caused by a temporary network failure. If the status does not change after restarting replication several times, request Altibase technical support immediately.
If transmission is delayed, the data on the two servers may differ because of changes that have not been sent. In addition, the sending server must keep the online log files it has not sent, so the log files may accumulate and cause a failure on the sending server as well. Check the degree of transmission delay by querying the replication gap (REP_GAP) as follows.
SELECT REP_NAME, REP_GAP FROM V$REPGAP;
REP_NAME REP_GAP
------------------------------------------------------------------
REP1 0
1 row selected.
REP_GAP is the size from the log record currently being sent to the last log record not yet sent, in MB. Normally it changes around values close to 0; if it keeps increasing, there is a transmission problem, so check the replication status of each server and the network. In Altibase 6.5.1 and earlier, REP_GAP is calculated as the difference between the sequence number of the last written log (SN) and the sequence number of the last sent log (XSN).
Data Conflicts#
When a data conflict occurs, the following messages are recorded. Except for INSERT, each conflict records two messages.
| Conflict type | Message |
|---|---|
| Duplicate PK on INSERT | ERR-11058(errno=0) The row already exists in a unique index. |
| DELETE target data does not exist on the peer server | ERR-61036(errno=0) [Receiver] err_not found in deleteXlog()ERR-61000(errno=0) The received record is not found in the database. |
| UPDATE target data does not exist on the peer server | ERR-6103a(errno=0) [Receiver] err_not_found in updateXlog()ERR-61000(errno=0) The received record is not found in the database. |
| Before-image of UPDATE differs from the peer server | ERR-61035(errno=0) [Receiver] An update conflict encountered.ERR-61001(errno=0) A conflict has been occurred while executing the received statement. |
Altibase also records the SQL statement that caused the conflict, so you can use this log to find the conflicting data. Data conflicts occur when the source data already differs or when the replicated servers change data with the same PK without distinction. Therefore, review how the application runs based on the recorded SQL statements. For causes and solutions, see Causes and Solutions of Replication Conflicts.
Technical Support System#
Altibase provides 24x7 technical support to users with a maintenance contract. The failure handling procedure is as follows.
Altibase Trace Logs#
Altibase records trace logs in the $ALTIBASE_HOME/trc directory. The main trace log files are as follows.
| Trace log file | Description |
|---|---|
| altibase_boot.log | Records the operating status of the Altibase server and system information at startup and shutdown. |
| altibase_error.log | Records error messages that occur in the server. If Altibase terminates abnormally, the call stack of the process is also recorded. |
| altibase_dump.log | Dumps the working memory at the time Altibase terminates abnormally. Used to diagnose and debug errors. |
| altibase_sm.log | Records warning and trace messages of the storage manager module. Mainly contains checkpoint and tablespace information. |
| altibase_rp.log | Records warning and trace messages of the replication module. |
| altibase_qp.log | Records warning and trace messages of the query processing module. |
| altibase_mm.log | Records warning and trace messages of the main module. |
| altibase_dk.log | Records warning and trace messages of the database link module. |
Basic Monitoring Items#
The basic items to monitor when operating Altibase are as follows. For more detailed monitoring queries, see the Altibase Monitoring Queries Guide.
Altibase Process#
Check that the Altibase process is displayed with the following command. If it is not displayed, follow the Procedure for Urgent Failures.
$ ps -ef | grep "altibase -p boot from" | grep -v grep
Memory#
Monitor that the system keeps at least 20% free memory. For how to check it on each OS, see Step 1: Check System Memory and Swap.
Check the memory usage of Altibase with the following query and monitor that it does not increase sharply compared with usual.
SELECT SUM(MAX_TOTAL_SIZE) FROM V$MEMSTAT;
Check what percentage of MEM_MAX_DB_SIZE the memory database occupies with the following query, and monitor that it does not exceed 90%.
SELECT TRUNC((MEM_ALLOC_PAGE_COUNT * 32 * 1024) / MEM_MAX_DB_SIZE * 100.0, 2)
FROM V$DATABASE;
Disk#
Check the disk usage of the directories used by Altibase with the following command, and monitor that it does not increase sharply compared with usual.
$ df -h
Check the number of allocated pages and the total available size of each disk tablespace with the following query. This is not the actual usage, but it helps decide whether a data file needs to be added.
SELECT a.NAME,
a.ALLOCATED_PAGE_COUNT,
SUM(b.MAXSIZE)
FROM V$TABLESPACES a, V$DATAFILES b
WHERE a.ID = b.SPACEID
GROUP BY a.NAME, a.ALLOCATED_PAGE_COUNT;
Trace Logs#
Check whether error messages starting with "ERR-" are recorded in the trace logs, and take action according to their severity.
$ tail -f $ALTIBASE_HOME/trc/altibase_boot.log | grep "ERR-"
Also check that checkpoints work normally and log files are deleted. The following message must be recorded periodically in altibase_sm.log; if None keeps appearing instead of file numbers, check why the log files are not being deleted.
Remove Online Log File at LFG [0]: File[11252 ~ 11253]
Replication#
Monitor that REP_GAP in V$REPGAP does not keep increasing. For how to query it, see Transmission Problems.