In my last column, I demonstrated how to implement two base
classes, AsyncResultNoResult and AsyncResult
CreateFile and DeviceIoControl
In Win32®-based coding, when you want to work with a file, you open it by calling the Win32 CreateFile function and then perform operations on the file by calling functions such as WriteFile and ReadFile. Ultimately, when you’re finished working with the file, you close it by calling the CloseHandle function.
In the .NET Framework, constructing a FileStream object internally calls CreateFile. Calling FileStream’s Read and Write methods internally calls the Win32 WriteFile and ReadFile functions respectively. And Calling FileStream’s Close or Dispose methods internally calls the Win32 CloseHandle function. When working with files, most applications just need to write data to and read data from the file and so the FileStream class offers most of the operations you need.
However, Windows® actually offers many more operations that can be performed on a file. For the more common operations, Windows provides specific Win32 functions like WriteFile, ReadFile, FlushFileBuffers, and GetFileSize. However, for infrequently used operations, Win32 doesn’t supply specific functions; instead, it offers one function, DeviceIoControl, that lets an application communicate directly with the device driver (such as the NTFS disk driver) responsible for manipulating the file. Examples of infrequently used file operations include opportunistic locking (microsoft.com/msj/0100/win32/win320100.aspx), manipulating the change journal (microsoft.com/msj/0999/journal/journal.aspx), compressing disk volumes/files, creating junction points, formatting/repartitioning disks, and working with sparse files.
Furthermore, DeviceIoControl can be used by any application to communicate with any hardware device driver. Using DeviceIoControl, an application can query the computer’s battery status, query or change the LCD’s brightness, manipulate a CD or DVD changer, query and eject USB devices, and much more.
Win32 Device Communication
Let’s take a quick look at the Win32 way of having an application communicate with a device and then wrap this functionality so that it can be used from managed code.
In Win32, the way you open a device is by calling the CreateFile function. CreateFile’s first argument is a string identifying the device to open. Normally, a path name is specified for the string causing CreateFile to open a file. However, you can pass special strings to CreateFile, causing it to open devices. Figure 1 shows some potential strings and describes the kind of device the string causes CreateFile to open. Be aware that some of these devices are restricted to the system or members of the Administrators group and so attempting to open some of these devices may fail unless your application is running with the required access permissions.
Now that you know how to open a device, let’s look at the Win32 function DeviceIoControl:
BOOL DeviceIoControl(
HANDLE hDevice, // Handle returned from CreateFile
DWORD dwIoControlCode, // Operation control code
PVOID pInBuffer, // Address of input buffer
DWORD nInBufferSize, // Size, in bytes, of input buffer
PVOID pOutBuffer, // Address of output buffer
DWORD nOutBufferSize, // Size, in bytes, of output buffer
PDWORD pBytesReturned, // Gets number of bytes written to
// pOutBuffer
POVERLAPPED pOverlapped); // For asynchronous operation
When calling this function, the handle parameter refers to a file, directory, or device driver obtained by calling CreateFile. The dwIoControlCode parameter indicates the operation to perform on the device. Each operation is simply identified via this 32-bit integer code. If an operation requires arguments, then the arguments are placed in fields of a data structure and the address of the structure is passed for the pInBuffer parameter. The size of the structure is passed in the nInBufferSize parameter. Similarly, if the operation returns some data, then the output data is placed in a buffer, which must be allocated by the caller. The address of this buffer is passed in the pOutBuffer parameter. The size of this buffer is passed in the nOutBufferSize parameter. When DeviceIoControl returns, it writes the number of bytes actually written to the output buffer in the pBytesReturned parameter (which is passed by reference). Finally, when performing synchronous operations, NULL is passed for the pOverlapped parameter. But when performing an asynchronous operation, the address of an OVERLAPPED structure must be passed in. I’ll explain this is great detail later in this column.
Now that you have a sense of how to communicate with devices via Win32, let’s start writing a managed wrapper over these Win32 functions so that you can write managed code in a .NET language to communicate directly with hardware devices.
Synchronous Device I/O in Managed Code
First, let’s focus on how to perform synchronous device operations. Later, I’ll add the code necessary to perform asynchronous operations. In C#, I have defined a static DeviceIO class that is a friendly wrapper around the Win32 CreateFile and DeviceIoControl functions. Figure 2 shows the public object model for this class.
In a managed application, you can call OpenDevice, passing in a string similar to those shown in Figure 1, plus the desired access and sharing. OpenDevice will internally call CreateFile returning the handle to the device. The handle is actually returned wrapped in a SafeFileHandle object to ensure that it will ultimately be closed and to get the other benefits offered by way of all SafeHandle-derived classes. This SafeFileHandle object can be passed as the first argument to any of DeviceIO’s other static methods and this is ultimately passed as the first argument when calling the Win32 DeviceIoControl function.
The second argument to all the other methods is a DeviceControlCode, a simple structure (value type) that contains just one private instance field, an Int32, that identifies a command code you want to send to a device. I found that wrapping the Int32 command code into its own type caused the code to read better, be more type-safe, and benefit from IntelliSense® in Visual Studio®. Internally, the Int32 value contained inside a DeviceControlCode instance is passed as the second argument to the Win32 DeviceIoControl function.
Now, when examining device control codes, you’ll detect that there are a few common patterns, and I decided to offer friendly methods to make working with these common patterns convenient. All of these methods execute the operation synchronously; that is, the calling thread will not return until the operation is complete. Also, when calling any of these methods, you must specify false for the useAsync argument when calling OpenDevice.
The Control method encapsulates a pattern where the control code you’re sending to a device causes the device to perform some operation and the device has no resulting data to return. There is a Control method that takes just a SafeFileHandle and a control code and an overload that takes an additional Object argument. The additional Object argument allows you to pass some additional data to the device that it will use for an operation. When you look up control codes in the Win32 SDK documentation, the documentation will indicate what additional information (if necessary) you must pass for each control code. If you need additional data, you must define a managed type that is the equivalent of the Win32 data type, construct an instance of it, initialize its fields, and pass a reference to the instance for the inBuffer parameter.
The GetObject
The GetArray
Internally, GetObject will create an array of these elements and pass the address of the array to DeviceIoControl, which will initialize the individual array elements. When DeviceIoControl returns, it returns the number of bytes actually placed in the array. GetArray uses this value to shrink (if necessary) the array to the exact size so that the number of elements in the array is equal to the number of elements initialized. This is a big convenience as any code that uses this array (returned from GetArray) can simply query the array’s Length property to get the number of items and use this value in its loop to process the returned elements.
Figure 3 shows the managed equivalent of the Win32 DisplayBrightness structure and its helper DisplayPowerFlags enumerated type. Figure 4 shows an AdjustBrightness method that uses my static DeviceIO class to constantly adjust your LCD’s brightness up and down. I must say that this is not a very useful application in real life, but it is a simple example that shows the basic concepts.
Asynchronous Device I/O in Managed Code
Some device operations, like changing LCD brightness, are not really I/O operations and so it makes sense to call DeviceIoControl synchronously when performing these kinds of operations. However, most calls to DeviceIoControl result in I/O (hence the name of the function) and, therefore, it makes sense to execute these operations asynchronously. In this section, I’ll explain how to P/Invoke from managed code out to the Win32 DeviceIoControl function in order to perform asynchronous I/O operations to files, disks, and other hardware devices.
My static DeviceIO class offers asynchronous versions of the Control, GetObject, and GetArray methods by way of the CLR’s APM. Figure 5 shows the public methods (not shown earlier) of this class that support the APM.
As you can see, all the BeginXxx methods follow the CLR’s APM pattern; that is, they all return an IAsyncResult and the last two parameters are an AsyncCallback and an Object. The additional parameters match those of each method’s synchronous counterpart. Similarly, all the EndXxx methods accept a single parameter, an IAsyncResult, and each returns the same data type as each method’s synchronous counterpart.
In order to call any of these asynchronous methods, two things must happen. First, you must tell Windows that you want to perform operations on the device asynchronously. This is accomplished by passing the FILE_FLAG_OVERLAPPED flag (which has a numeric value of 0x40000000) to the CreateFile function. The managed equivalent of this flag is FileOptions.Asynchronous (which, of course, also has a value of 0x40000000). Second, you have to tell the device driver to post operation completion entries into the CLR’s thread pool. This is accomplished by calling ThreadPool’s static BindHandle method, which takes one parameter, a reference to a SafeHandle-derived object.
DeviceIO’s OpenDevice method performs both of these actions when you pass true for its useAsync parameter. Here is how my OpenDevice method is implemented:
public static SafeFileHandle OpenDevice(String deviceName,
FileAccess access, FileShare share, Boolean useAsync) {
SafeFileHandle device = CreateFile(deviceName, access, share,
IntPtr.Zero, FileMode.Open,
useAsync ? FileOptions.Asynchronous : FileOptions.None,
IntPtr.Zero);
if (device.IsInvalid) throw new Win32Exception();
if (useAsync) ThreadPool.BindHandle(device);
return device;
}
Now, let’s take a look at how BeginGetObject and EndGetObject work internally. The other asynchronous methods work similarly and you will be able to understand how they work easily after walking through BeginGetObject and EndGetObject. All of the BeginXxx methods are very simple; they construct an object or array if the operation returns some return value or array of elements, and then they immediately call an internal helper method, AsyncControl, which is implemented as shown in Figure 6.
When performing an operation, you can pass the address of a buffer containing additional input data to DeviceIoControl. If DeviceIoControl returns data, the memory for that data must be allocated before calling DeviceIoControl so that DeviceIoControl can initialize the data; the data is then examined when the operation completes. The problem is that the operation could theoretically take hours to complete and during this time garbage collections could occur that would move the data around. As a result, the addresses passed to DeviceIoControl would no longer refer to the desired buffers and memory corruption would occur.
You can tell the GC not to move an object around by pinning the object. My SafePinnedObject class is a helper class that pins an object in memory, preventing the GC from moving it. However, since the SafePinnedObject class is derived from SafeHandle, it comes with all the benefits you normally get with SafeHandle-derived types, including the assurance that the object will be unpinned at some point in the future. Figure 7 gives you an idea of how the SafePinnedObject class is implemented (I’ve removed some validation code to save space). For more information about SafeHandle-derived types and about pinning objects in memory, see my book CLR via C# (Microsoft Press®, 2006).
After pinning both the input and output buffers,
AsyncControl constructs a DeviceAsyncResult
DeviceIoControl’s Overlapped Parameter
Now that all this prep work is complete, AsyncControl is ready to call DeviceIoControl. It does that by calling NativeControl passing it the device handle, the control code, the input buffer, the output buffer, and a reference to an Int32 bytesReturned variable (which will basically be ignored by the method since the method will return before the operation completes). The most important argument is the last one: when performing an asynchronous operation, DeviceIoControl must be passed the address of a NativeOverlapped structure (the equivalent of a Win32 OVERLAPPED structure). AsyncControl obtains this by calling a helper method, GetNativeOverlapped, defined in my DeviceAsyncResult class. This helper method is implemented as follows:
// Create and return a NativeOverlapped structure to be passed
// to native code
public unsafe NativeOverlapped* GetNativeOverlapped() {
// Create a managed Overlapped structure that refers to our
// IAsyncResult (this)
Overlapped o = new Overlapped(0, 0, IntPtr.Zero, this);
// Pack the managed Overlapped structure into a NativeOverlapped
// structure
return o.Pack(CompletionCallback,
new Object[] { m_inBuffer.Target, m_outBuffer.Target });
}
This method constructs an instance of the System.Threading.Overlapped class and initializes it. This is a managed helper class that lets you set up and manipulate an overlapped structure. However, you cannot pass a reference to this object to native code. Instead you must first pack the Overlapped object into a NativeOverlapped object; a reference to the resulting NativeOverlapped object can then be passed to native code. When you call Pack, you pass it a delegate referring to a callback method that a thread pool thread will execute when the operation completes. In my code, this is the CompletionCallback method.
Calling the Overlapped class’s Pack method does several things. It allocates memory for a native OVERLAPPED structure from the managed heap and pins it, which ensures that the memory will not be moved should a GC occur. It then initializes the fields of the NativeOverlapped structure’s field from the fields set in the managed Overlapped object. This includes creating a normal GCHandle for the IAsyncResult object that the Overlapped object refers to ensuring that the IAsyncResult object stays alive for the duration of the operation.
Pack then pins the callback method (CompletionCallback) delegate so that the fixed address can be passed to native code, thus allowing native code to call back to managed code when this operation completes. Pack also pins any additional objects referenced by its second parameter, an Object array. In my case, the inBuffer and outBuffer objects are already pinned because I wrapped them using my SafePinnedObject class. I need to pin them myself so that I can get their address by calling GCHandle’s AddrOfPinnedObject method.
However, the Pack method pins them again, but in a special way. Normally, when an AppDomain is unloaded, the CLR automatically unpins any objects, allowing them to be collected and thus preventing a memory leak. But Pack pins the objects until the asynchronous operation completes. So if an asynchronous operation is started and then the AppDomain is unloaded, the Pack-pinned objects will not be unpinned. After the operation completes, the objects will be unpinned allowing them to be collected; this prevents memory corruption.
Pack also records which AppDomain called Pack to ensure that the thread pool thread that calls the CompletionCallback method executes in the same AppDomain. Finally, Pack captures the stack of code access security permissions so that when the callback method executes, it executes under the same security permissions that the code initiating the asynchronous operation was under. You can call the Overlapped class’s UnsafePack method if you prefer not to propagate the stack of security permissions.
Pack returns the address of the pinned NativeOverlapped structure; this address can be passed to any native function that expects the address to a Win32 OVERLAPPED structure. In my AsyncControl method, the NativeOverlapped structure’s address is passed to NativeControl (see Figure 8), which passes it to the Win32 DeviceIoControl function.
When performing an asynchronous operation, DeviceIoControl
returns immediately and, ultimately, the DeviceAsyncResult
object that implements the IAsyncResult interface is
returned from DeviceIO’s BeginObject
When the operation completes, the device driver queues an
entry in the CLR’s thread pool. You’ll recall that the
driver knows to do this because DeviceIO’s OpenDevice method
internally calls ThreadPool’s BindHandle method when the
device is opened for asynchronous access. Eventually, a
thread pool thread will extract this queued entry, adopt any
saved permission sets, jump into the right AppDomain, and
call the CompletionCallback method (see
Figure 9). Don’t forget that the CompletionCallback
method is defined inside the DeviceAsyncResult
The first thing that the CompletionCallback method does is free the nativeOverlapped object. This is extremely important as it releases the GCHandle on the IAsyncResult object, unpins all the objects passed to the Overlapped class’s Pack method, and also unpins the NativeOverlapped object itself allowing it to be GC’d. Forgetting to free the NativeOverlapped object results in leaking several objects.
The rest of CompletionCallback’s code is pretty
straightforward. It is just a matter of recording in the
AsyncResult
At some point, the application code will call DeviceIO’s
EndControl, EndGetObject, or EndGetArray method, passing in
the DeviceAsyncResult
Opportunistic Locks
An opportunistic lock gives you a simple example that demonstrates an asynchronous device operation. Figure 10 shows a static OpLockDemo class whose Main method creates a file (using the FileOptions.Asynchronous flag). The SafeFileHandle embedded inside the FileStream is then passed to BeginFilter, which internally calls the DeviceIO’s BeginControl method, passing the code that establishes a request filter opportunistic lock on the file. Now the file system device driver will watch to see if any other application attempts to open the file and, if so, the device driver will queue an entry to the CLR’s thread pool. This will in turn call the anonymous method that sets the s_earlyEnd field to 1, which indicates that another application wants to access the file. The first application will then close the file allowing the other application to continue running with access to the file.
To test all of this, start the OpLockDemo application. Then go to your desktop and attempt to delete the FilterOpLock.dat file that the application creates. This will cause the anonymous method to get called, ending the sample application.
Conclusion
Windows and its device drivers offer many features that the .NET Framework class library does not wrap. Fortunately, though, the .NET Framework does provide the appropriate mechanisms allowing you to P/Invoke to gain access to these useful features. The fact that you can perform these operations asynchronously means you can build robust, reliable, and scalable applications that take advantage of these features.
Send your questions and comments for Jeffrey to mmsync@microsoft.com.
NEW:
Explore the sample code online! - or -
Code download available at:
ConcurrentAffairs2007_06.exe (162KB) |
