2).scmd执行超时了,即一直没有返回,发生了timeout。
scsi_eh_scmd_add()在两处被调用:
1).scsi_times_out函数 => scmd超时处理函数,将timeout的scmd加入eh
2).scsi_softirp_done函数 => scmd命令结果处理函数,将结果为error的scmd加入eh
scsi_eh_scmd_add()完成以下工作:
1. Turns on scmd->eh_eflags as requested. It's 0 for error
completions and SCSI_EH_CANCEL_CMD for timeouts.
2. Links scmd->eh_entry to shost->eh_cmd_q
3. Sets SHOST_RECOVERY bit in shost->shost_state
4. Increments shost->host_failed
5. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed
scsi-high层
中间层完成命令时通过调用scmd->done()通知scsi-high层SCSI命令处理完毕。
HLD completion callback - sd:sd_rw_intr, sr:rw_intr, st:st_intr.
scsi-mid层
1). LLDD顺利完成SCSI命令 => 调用scsi-mid层提供的回调函数scsi_done()。
2). LLDD没能顺利完成SCSI命令 => scsi-mid层将该scmd time out,调用scsi_times_out()处理该命令。
LLDD层
SCSI EH如何工作
LLDD可以通过以下两种方式实现SCSI EH
- 实现EH回调函数
int (* eh_abort_handler)(struct scsi_cmnd *);
int (* eh_device_reset_handler)(struct scsi_cmnd *);
int (* eh_bus_reset_handler)(struct scsi_cmnd *);
int (* eh_host_reset_handler)(struct scsi_cmnd *);
供scsi_unjam_host调用完成SCSI EH的全部工作。
- 实现eh_strategy_handler()回调函数
transportt->eh_strategy_handler()函数完成全部的EH工作。
How SCSI EH works
LLDD's can implement SCSI EH actions in one of the following two
ways.
- Fine-grained EH callbacks
LLDD can implement fine-grained EH callbacks and let SCSI
midlayer drive error handling and call appropriate callbacks.
This will be discussed further in [2-1].
- eh_strategy_handler() callback
This is one big callback which should perform whole error
handling. As such, it should do all choirs SCSI midlayer
performs during recovery. This will be discussed in [2-2].
* scsi_error_handler - SCSI error handler thread
* @data: Host for which we are running.
* Notes:
* This is the main error handling loop. This is run as a kernel thread
* for every SCSI host and handles all error handling activity.
int scsi_error_handler(void *data)
/* 如果eh_strategy_handler在LLDD中有实现,则调用该函数处理,
否则调用通用处理函数scsi_unjam_host进行处理,scsi_unjam_host
会通过调用LLDD定义的EH回调函数们来完成工作。
if (shost->transportt->eh_strategy_handler)
shost->transportt->eh_strategy_handler(shost);
scsi_unjam_host(shost);
* scsi_host_alloc - register a scsi host adapter instance.
* @sht: pointer to scsi host template
* @privsize: extra bytes to allocate for driver
* Note:
* Allocate a new Scsi_Host and perform basic initialization.
* The host is not published to the scsi midlayer until scsi_add_host
* is called.
* Return value:
* Pointer to a new Scsi_Host
struct Scsi_Host *scsi_host_alloc(struct scsi_host_template *sht, int privsize)
shost->ehandler = kthread_run(scsi_error_handler, shost, /* scsi_error_handler在scsi_host_alloc函数中被指定 */
"scsi_eh_%d", shost->host_no);
static void scsi_unjam_host(struct Scsi_Host *shost)
unsigned long flags;
LIST_HEAD(eh_work_q);
LIST_HEAD(eh_done_q);
spin_lock_irqsave(shost->host_lock, flags);
list_splice_init(&shost->eh_cmd_q, &eh_work_q);
spin_unlock_irqrestore(shost->host_lock, flags);
SCSI_LOG_ERROR_RECOVERY(1, scsi_eh_prt_fail_stats(shost, &eh_work_q));
/* scsi_eh_get_sense函数最后调用list_empty(eh_work_q),如果为空,则返回1,否则返回0,
继续执行更高级别的EH。即如果 scsi_eh_get_sense函数能解决掉所有eh_work_q中的scmd,
则不许要下一步调用,否则,继续进行更高级别的EH。
if (!scsi_eh_get_sense(&eh_work_q, &eh_done_q))
if (!scsi_eh_abort_cmds(&eh_work_q, &eh_done_q))
scsi_eh_ready_devs(shost, &eh_work_q, &eh_done_q);
scsi_eh_flush_done_q(&eh_done_q);
对于error-completed (completed but result is error) scmd,调用scsi_eh_get_sense将它们结束
对于timed-out scmd,
a).调用scsi_eh_abort_cmds让LLDD忽略这个scmd
b).调用scsi_eh_stu 对每个device,执行一遍START_UNIT命令,
1). First call scsi_eh_get_sense
<<scsi_eh_get_sense>> /** 处理error-completed scmd,将它们连同sence data返回给scsi-high层处理 **/
该函数是为那些error-completed(complted,但是结果有问题)的scmd准备的。
如果所有scmd都是error-completed的,且成功获得了sense data,scsi_eh_finish_cmd()被调用将该scmd
移到eh_done_q,不需要进一步操作,EH完成;
否则经过scsi_eh_get_sense函数之后,eh_work_q链表仍然非空(即还有无法处理的scmd)则进行更高级别的EH。
标记为 SCSI_EH_CANCEL_CMD的scmd是因为time out而进入EH链表的,不是scsi_eh_get_sense函数处理对象。
2). If !list_empty(&eh_work_q), invoke scsi_eh_abort_cmds().
<<scsi_eh_abort_cmds>>/** 处理timed-out scmd,让底层LLDD abort(终止)这些scmd **/
该函数是为那些timed-out的scmd准备的。
调用hostt->eh_abort_handler()函数依次处理每一个scmd,如果该函数成功地让LLDD和相关的硬件忽略了这个scmd,
则成功返回SUCCESS,。
如果一个timed-out scmd被成功地终止掉了,那么这个scmd对应的device就应该处于offline或者ready状态。
根据scsi_try_to_abort_cmd函数的返回值移动scmd
a).FAST_IO_FAIL,则调用scsi_eh_finish_cmd()将该scmd 移到eh_done_q;
b).SUCCESS,则进行检查,如果scmd对应的device是offline或者ready状态,则调用scsi_eh_finish_cmd()
将它移到eh_done_q,否则不移动。
c).否则不移动scmd,它仍然保留在eh_work_q中
如果有不成功的scmd仍然留在eh_work_q链表中,则调用更高级别的EH。
3). If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs()
<<scsi_eh_ready_devs>> /** **/
void scsi_eh_ready_devs(struct Scsi_Host *shost,
struct list_head *work_q,
struct list_head *done_q)
if (!scsi_eh_stu(shost, work_q, done_q))
if (!scsi_eh_bus_device_reset(shost, work_q, done_q))
if (!scsi_eh_target_reset(shost, work_q, done_q))
if (!scsi_eh_bus_reset(shost, work_q, done_q))
if (!scsi_eh_host_reset(work_q, done_q))
scsi_eh_offline_sdevs(work_q,
done_q);
<<scsi_eh_stu>>
每个device,执行一遍START_UNIT命令,然后检查执行结果。
如果device是offline或者ready状态,则调用scsi_eh_finish_cmd()将它关联的scmd都移动到eh_done_q,否则不移动。
<<scsi_eh_test_devices>>
测试device状态:offline、online、online && ready
/**
* scsi_eh_test_devices - check if devices are responding from error recovery.
* @cmd_list: scsi commands in error recovery.
* @work_q: queue for commands which still need more error recovery
* @done_q: queue for commands which are finished
* @try_stu: boolean on if a STU command should be tried in addition to TUR.
*
* Decription:
* Tests if devices are in a working state. Commands to devices now in
* a working state are sent to the done_q while commands to devices which
* are still failing to respond are returned to the work_q for more
* processing.
**/
static int scsi_eh_test_devices(struct list_head *cmd_list,
struct list_head *work_q,
struct list_head *done_q, int try_stu)
{
struct scsi_cmnd *scmd, *next;
struct scsi_device *sdev;
int finish_cmds;
while (!list_empty(cmd_list)) {
scmd = list_entry(cmd_list->next, struct scsi_cmnd, eh_entry);
sdev = scmd->device;
/* 1.if scsi_device_online() return value is 0, means the device is offline, then the things
after || don't need to be done.
2.if scsi_device_online() return value is 1, means the device is online, then the things
after || need to be done to juge if the device is in a ready state.
3.the way to test if device is in ready state is to call function scsi_eh_tur() to
send TEST_UNIT_READY cmd to the device and see the return value.
*/
finish_cmds = !scsi_device_online(scmd->device) ||
(try_stu && !scsi_eh_try_stu(scmd) &&
!scsi_eh_tur(scmd)) ||
!scsi_eh_tur(scmd);
list_for_each_entry_safe(scmd, next, cmd_list, eh_entry)
if (scmd->device == sdev) {
if (finish_cmds)
scsi_eh_finish_cmd(scmd, done_q);
else
list_move_tail(&scmd->eh_entry, work_q);
}
}
return list_empty(work_q);
}
static void scsi_unjam_host(struct Scsi_Host *shost)
{
unsigned long flags;
LIST_HEAD(eh_work_q);
LIST_HEAD(eh_done_q);
spin_lock_irqsave(shost->host_lock, flags);
list_splice_init(&shost->eh_cmd_q, &eh_work_q); /*将eh_cmd_q中的scmd复制到eh_work_q中*/
spin_unlock_irqrestore(shost->host_lock, flags);
SCSI_LOG_ERROR_RECOVERY(1, scsi_eh_prt_fail_stats(shost, &eh_work_q));
/*eh_work_q中的scmd分为两类:
1.scmd执行完成了,但是执行结果为error的。即errored scmd。
2.scmd执行了就没有返回,知道超时发生time out,即timed-out scmd。
*/
/*1.调用scsi_eh_get_sense()从errored scmd的目标设备处获取sense信息。
2.sense信息用途:作为上层驱动判断scmd发生error的原因的依据。
3.然后将errored scmd处理掉(即加入eh_done_q并调用finish函数将scmd返回上层)。
4.eh_done_q中剩余的全部都是timed-out scmd。
*/
if (!scsi_eh_get_sense(&eh_work_q, &eh_done_q))
/*1.对eh_work_q中剩余的timed-out scmd,执行abort操作(abort,放弃执行),
2.如果abort之后,eh_work_q还不为空,即还有没处理掉的timed-out scmd,
调用scsi_eh_ready_devs开始逐步升级的reset操作,直到eh_work_q中的scmd处理光。
*/
if (!scsi_eh_abort_cmds(&eh_work_q, &eh_done_q))
scsi_eh_ready_devs(shost, &eh_work_q, &eh_done_q);
scsi_eh_flush_done_q(&eh_done_q);
}
static int scsi_eh_abort_cmds(struct list_head *work_q,
struct list_head *done_q)
{
struct scsi_cmnd *scmd, *next;
LIST_HEAD(check_list);
int rtn;
/*循环对每个scmd执行一遍abort,放弃该命令的执行*/
list_for_each_entry_safe(scmd, next, work_q, eh_entry) {
if (!(scmd->eh_eflags & SCSI_EH_CANCEL_CMD))
continue;
SCSI_LOG_ERROR_RECOVERY(3, printk("%s: aborting cmd:"
"0x%p\n", current->comm,
scmd));
/* 执行对当前scmd的abort */
rtn = scsi_try_to_abort_cmd(scmd->device->host->hostt, scmd);
/* 如果执行结果为SUCCESS,或者期望通过将IO设置为FAIL快速返回,则进入 */
if (rtn == SUCCESS || rtn == FAST_IO_FAIL) {
scmd->eh_eflags &= ~SCSI_EH_CANCEL_CMD;
if (rtn == FAST_IO_FAIL)
scsi_eh_finish_cmd(scmd, done_q);/*将返回FAST_IO_FAIL的scmd移动到done_q从而finish掉*/
else
list_move_tail(&scmd->eh_entry, &check_list); /*将返回SUCCESS的scmd加入check_list*/
} else
SCSI_LOG_ERROR_RECOVERY(3, printk("%s: aborting"
" cmd failed:"
"0x%p\n",
current->comm,
scmd));
}
/*调用scsi_eh_test_devices()函数,检查返回SUCCESS的scmd们(都保存在check_list中)的目标设备的状态。
如果一个scmd的目标设备的状态要么为offline,要么为ready,则说明对这个scmd的abort操作成功了,
可以将这个scmd移动到done_q中finish掉这个scmd了。
*/
return scsi_eh_test_devices(&check_list, work_q, done_q, 0);
}
static int scsi_eh_test_devices(struct list_head *cmd_list,
struct list_head *work_q,
struct list_head *done_q, int try_stu)
{ /*该函数的改进在于如果一个scmd的目标设备处于offline或者ready,则把check_list中的其他具有该目标设备
的scmd也移动到done_q中finish掉。而不是每个scmd都检测一遍它的目标设备,提高了效率。
即以sdev为单位处理check_list中的scmd,而不是以scmd为单位挨个执行,做很多重复工作。
*/
struct scsi_cmnd *scmd, *next;
struct scsi_device *sdev;
int finish_cmds;
while (!list_empty(cmd_list)) {
scmd = list_entry(cmd_list->next, struct scsi_cmnd, eh_entry);
sdev = scmd->device;
/* 1.if scsi_device_online return value is 0, means the device is offline, then the things
after || don't need to be done.
2.if scsi_device_online return value is 1, means the device is online, then the things
after || need to be done to juge if the device is in a ready state.
3.the way to test if device is in ready state is to call function scsi_eh_tur to
send TEST_UNIT_READY cmd to the device and see the return value.
*/
finish_cmds = !scsi_device_online(scmd->device) ||
(try_stu && !scsi_eh_try_stu(scmd) &&
!scsi_eh_tur(scmd)) ||
!scsi_eh_tur(scmd);
/*到check_list中检测目标设备是上面的sdev的scmd,将它们都处理掉,提高效率。*/
list_for_each_entry_safe(scmd, next, cmd_list, eh_entry)
if (scmd->device == sdev) {
if (finish_cmds)
scsi_eh_finish_cmd(scmd, done_q);
else
list_move_tail(&scmd->eh_entry, work_q);
}
}
return list_empty(work_q);
}
PID TTY STAT TIME COMMAND
1 ? Ss 0:01 /sbin/init
2 ? S 0:00 [kthreadd]
3 ? S 0:00 [ksoftirqd/0]
5 ? S< 0:0...
我使用过的Linux命令之pstree - 以树状图显示进程间的关系
本文链接:http://codingstandards.iteye.com/blog/842156
(转载请注明出处)
pstree命令以树状图显示进程间的关系(display a tree of processes)。ps命令可以显示当前正在运行的那些进程的信息,但是对于它们之间的关系却显示得不够清晰。...
for(;;) {
us->proto_handler(us->srb, us);
=>void usb_stor_invoke_transport(struct
scsi
_cmnd *srb, struct us_data *us)
res...
[0:0:0:0] disk ATA INTEL SSD
SC
2BX20 0150 -
[0:0:1:0] disk ATA INTEL SSD
SC
2BX20 0150 -
[0:1:0:0] disk L
SI
Logical ...
概述Linux
SCSI
子系统的分层架构:
低层:代表与
SCSI
的物理接口的实际驱动器,例如各个厂商为其特定的主机适配器(Host Bus Adapter, HBA)开发的驱动,低层驱动主要作用是发现连接到主机适配器的
scsi
设备,在内存中构建
scsi
子系统所需的数据结构,并提供消息传递接口,将
scsi
命令的接受与发送解释为主机适配器的操作。
高层: 代表各种
scsi
设备类型的驱动,如
scsi
磁盘驱
当前LINUX社区
SCSI
层错误
处理
在整个错误
处理
期间是阻塞整个
SCSI
HOST上所有的IO的,也就是即使仅某个盘的IO出错,在错误
处理
阶段会阻塞所有的盘的IO(即使其他的盘都是正常的),会造成业务的暂时停顿,这是目前
SCSI
错误
处理
存在经常让人诟病的问题。
分析完
SCSI
层的错误
处理
,个人认为可能可以优化,将基于
SCSI
HOST的错误
处理
变为基于
SCSI
DEVICE的错误
处理
(即尽可能只阻塞
SCSI
DEVICE的IO,而不是直接阻塞
SCSI
HOST的IO):
又一存储类故障,
两台DELL R610服务器.HBA : 04:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
SWITCH : Brocade 300
System : RHEL 5.6 64bit
Storage...
uptime 会显示第一行负载参数
第一行从左到右依次表示当前时间、系统已运行时间、登录用户数(终端登录显示为tty1,远程登录显示为pts/0)、平均负载(每1分钟、每5分钟、每15分钟)主要是注意第一个负载值,值越大说明服务器压力越大。一般情况下,这个值不要超过服务器的逻辑cpu数量就没有关系
cat /proc/cpuinfo 查看系统逻辑cpu
有时候打开一个应用,但是很容易造成卡顿,或者卡死,在win下直接打开任务管理器找到相对应的进程就可以杀死,但是在使用Ubuntu的话只能使用命令行进行操作Ubuntu 终端条件下查看进程: ps -e
下面是命令行的结果:
snakeson@snakeson-Inspiron-5421:~$ ps -e
PID TTY TIME CMD
1 ? 00:
1. IO 下发
流程
SCSI
IO路径上接 block 层 1.1 SQ
SCSI
SQ 注册到Block层IO钩子函数:
scsi
_old_alloc_queue()
+---> q->request_fn =
scsi
_request_fn; SQ 下发IO路径:
scsi
_request_fn()
+---> blk_peek_reque...
-c 显示CLS和PRI栏位。
c 列出进程时,显示每个进程真正的指令名称,而不包含路径,参数或常驻服务的标示。
-C<指令名称> 指定执行指令的名称,并列出该指令的进程的状况。
-d 显示所有...
ahci_init()
->pci_module_init(&ahci_pci_driver);
static struct pci_driver ahci_pci_driver = {
.name = DRV_NAME,
.id_table
= ahci_pci_tbl,
.probe
I still investigate this issue, and it now narrow down after we hard work.
Thanks my team mate.
Something we alred