Getting your driver to handle more than one I/O request at a time

Updated: May 25, 2007

Your user-mode application is sending your driver lots of I/O requests, but the driver insists on handling the requests one at a time. What's the problem?

You might think that your driver is blocking in some obscure way or that you need more threads in your application, but the solution is often much simpler: Make sure your application has opened the device for overlapped I/O. Otherwise, the I/O Manager serializes I/O requests by synchronizing through a lock in the file object before dispatching the IRP. Even if your application uses multiple threads, only one request at a time (per file handle) will get through.

Enabling overlapped I/O in an application
To enable overlapped I/O in an application, set dwFlagsAndAttributes with FILE_FLAG_OVERLAPPED when you call CreateFile to open the device. (If you skip this step you won't get overlapped I/O, even if you do everything else right.)

CreateFile returns a handle that can be used to access the device. When you call ReadFile, WriteFile, or DeviceIoControl with the file handle created by CreateFile, supply a pointer to an OVERLAPPED structure that contains a handle to an event object to signal when the operation completes. These functions return immediately for overlapped operations; the event object is signaled when the operation is completed.

Be sure to initialize the OVERLAPPED structure to zero before using it in a function call, and set only the members that are needed by the function. For example, the Offset member should be set for ReadFile and WriteFile calls, but this member should be zero for DeviceIoControl. Also, be sure to use a separate OVERLAPPED structure for each request; re-using the structure before the previous asynchronous operation has been completed can cause errors. Besides, your application will need the original structure to call HasOverlappedIoCompleted or GetOverlappedResult for the request.

After you've opened a device for overlapped I/O, can you simply omit the OVERLAPPED structure to get synchronous I/O? No, you must supply an OVERLAPPED structure for all function calls (read, write, or device control) with that handle. Passing NULL can lead to undefined behavior, even if the driver completes the request synchronously. For synchronous I/O, the OVERLAPPED structure can be declared on the stack as long as the application waits for the I/O to complete before returning.

The following code snippet shows how to open a file for writing overlapped I/O:


#include<windows.h>
#include<stdio.h>

HANDLEhFile;

hFile=CreateFile(TEXT("myfile.txt"),//filetocreate

GENERIC_WRITE,//openforwriting


0,//donotshare

NULL,//defaultsecurity

CREATE_ALWAYS,//overwriteexisting

FILE_ATTRIBUTE_NORMAL|//normalfile

FILE_FLAG_OVERLAPPED,//asynchronousI/O

NULL);//noattr.template



if(hFile==INVALID_HANDLE_VALUE)
{
printf("Couldnotopenfile(error%d)/n",GetLastError());

return0;
}

The following code snippet sets up the OVERLAPPED structure, calls ReadFile, and then checks the status of an I/O request:


OVERLAPPEDgOverlapped;

//setupoverlappedstructurefields
gOverLapped.Offset=0;
gOverLapped.OffsetHigh=0;
gOverLapped.hEvent=hEvent;

//verifythatsizeof(inBuffer>=nBytestoRead)

//attemptanasynchronousreadoperation
bResult=ReadFile(hFile,&inBuffer,nBytesToRead,&nBytesRead,
&gOverlapped);

//iftherewasaproblem,ortheasync.operation'sstillpending...

if(!bResult)
{

//dealwiththeerrorcode


switch(dwError=GetLastError())
{

caseERROR_HANDLE_EOF:
{

//wehavereachedtheendofthefile


//duringthecalltoReadFile



//codetohandlethat

}


caseERROR_IO_PENDING:
{

//asynchronousi/oisstillinprogress



//dosomethingelseforawhile

GoDoSomethingElse();


//checkontheresultsoftheasynchronousread

bResult=GetOverlappedResult(hFile,&gOverlapped,
&nBytesRead,FALSE);


//iftherewasaproblem...


if(!bResult)
{

//dealwiththeerrorcode


switch(dwError=GetLastError())
{

caseERROR_HANDLE_EOF:
{

//wehavereachedtheendof


//thefileduringasynchronous


//operation

}


//dealwithothererrorcases

}//endswitch(dwError=GetLastError())

}
}//endcase



//dealwithothererrorcases,suchasthedefault


}//endswitch(dwError=GetLastError())

}//endif

Handling overlapped I/O in your driver
From a driver's perspective, all I/O requests should be considered asynchronous. The driver does not need to check whether an I/O request actually is asynchronous, it should simply assume that it is and avoid blocking any I/O request. (The driver can block for other reasons such as acquiring a lock, but it shouldn't block just on the basis of receiving an I/O request.) If the application needs to use synchronous I/O, it simply opens the device without specifying overlapped I/O; the I/O Manager then serializes requests as described earlier, without special action on the part of the driver.

When the I/O Manager receives an I/O request from an application, the I/O Manager creates an IRP and calls your driver's dispatch routine. The role of the dispatch routine is to present the request to the driver for processing, not necessarily to handle all of the processing itself. If your driver does not complete the IRP in the dispatch routine, call IoMarkIrpPending and return STATUS_PENDING. (Remember that if you mark an IRP as pending by calling IoMarkIrpPending, the only status code you can return is STATUS_PENDING. You must not return any other status code. You can, however, mark an IRP as pending, complete it synchronously, and return STATUS_PENDING.)

For example, consider a ReadFile request sent by an application. On receiving the IRP_MJ_READ IRP from the I/O Manager, your dispatch routine might simply mark the IRP as pending, call IoStartPacket, and then return STATUS_PENDING. (The order is important here. If you put the request on the queue first, it could get picked up by another part of the driver, processed, completed and freed all before the first thread gets around to marking it pending. This would confuse the I/O system and would also crash the system when you tried to call IoMarkIrpPending on what is now a pointer to a free pool block.)

Your StartIo routine kicks off the operation by sending a command to the controller. When the controller has finished the command, your interrupt service routine (ISR) is signaled. It runs and queues a deferred procedure call (DpcForIsr). The DpcForIsr puts the appropriate status values in the IRP and calls IoCompleteRequest to actually complete the operation. The DpcForIsr then calls IoStartNextPacket, at which point your StartIo routine will be called again, in the context of the thread handling the DpcForIsr, to start another request.

Meanwhile, control returns to the application as soon as your driver's dispatch routine returns, so the application isn't blocked waiting for your driver to complete the I/O request. The status from ReadFile indicates that the request is in progress (ReadFile returns zero and GetLastError returns ERROR_IO_PENDING), so the application can continue to do other work (send more I/O requests, perhaps). After your driver calls IoCompleteRequest, the application is notified that the operation has completed through the event object specified in the OVERLAPPED structure, and it can check that structure for status information.

One last point: If you pend IRPs, you should support I/O cancellation, so the caller can cancel an I/O request if it's going to take too long. It's best to use the cancel-safe IRP queuing routines (the IoCsqXxx routines), especially if your driver performs I/O frequently, because these routines provide a framework that handles cancellation properly so that race conditions do not occur. The Cancel sample in the Windows DDK (%winddk%/src/general/cancel) shows the use of these routines.

Otherwise, you'll need to implement a Cancel routine in your driver and pass a pointer to this routine when you call IoStartPacket, so the I/O Manager will call your StartIo routine with the IRP in a cancelable state. In your StartIo routine, you'll need to check for cancellation and protect against related race conditions. This approach is not recommended for drivers that perform I/O frequently, because the I/O Manager holds the global cancel spin lock while it inserts and removes IRPs from the device queue, which can affect system performance.

See the Cancel Logic in Windows Drivers paper on WHDC for a detailed discussion of these and other IRP cancellation techniques.

What should you do?

In your application:

When calling CreateFile to open a device, set dwFlagsAndAttributes with FILE_FLAG_OVERLAPPED.
When calling ReadFile, WriteFile, or DeviceIoControl, supply a pointer to a properly initialized OVERLAPPED structure. Never omit the OVERLAPPED structure when using a handle to a device opened with FILE_FLAG_OVERLAPPED.
Never re-use an OVERLAPPED structure for subsequent requests.

In your driver:

Assume that all incoming I/O requests are asynchronous.
Unless you are completing IRPs in your dispatch routine, mark IRPs pending so the application isn't blocked waiting for the request to complete. Remember to return only STATUS_PENDING if you mark an IRP pending.
Support IRP cancellation, preferably by using the cancel-safe IRP queuing routines (IoCsqXxx), so the application can cancel an I/O request that is taking too long.

For more information:

Platform SDK: Storage
Creating and Opening Files
Synchronization and Overlapped Input and Output

WHDC
Handling IRPs: What Every Driver Writer Needs to Know
Cancel Logic in Windows Drivers
Flow of Control for Cancel-Safe IRP Queuing

Windows Driver Kit (WDK): Kernel-Mode Driver Architecture
Handling IRPs