Response.TransmitFile + Close Will Kill Your Application

Hello,

Nice post of the caveats of using TransmitFile() and logging…

I was wondering if you found any better option on this without having to resort to

Response.BinaryWrite or

while ((read = fs.Read(buffer, offset, chunkSize)) > 0)
{
Response.OutputStream.Write(buffer, 0, read);
Response.Flush();
}

I have been working on a httpHandler to do the same type of handling transfer and logging for different files; however I am handling large media files (anywhere between 15 to 300 MB) and so in order to not kill the performance of the web site with limited number of threads and overloading the system memory, I would like to continue using the Response.TransmitFile() calls, as this call is optimized for this scenario.

Ideally I would like to be able to log a downloadstart_event and then a download_completed event when TransmitFile() completed, however I don’t seem to be able to find an async event to catch and handle for the TransmitFile() call.

I would appreciate any insight you or anyone one else might have…

Mark -

Good tip here … sound like you’ve been working with this a lot. Good to see a recent post, since a lot of the top google hits were either old or not so performant for large files.

We’re looking to build something similar for large downloads … any suggestions for IIS tweaks or other tips? Users will be able to buy & download large media files, but I don’t want to tie up ASP.NET’s requests on downloads.

TIA,

Ben Strackany

Hello, I’m currently working on a similar project that hasn’t gone live yet but after seeing this i realized i might have the same problem.
Have you tried using the System.Web.HttpApplication.CompleteRequest() method instead of HttpResponse.Close() ?
I am doing:
TransmitFile(file.FullName);
resp.Flush();
app.CompleteRequest();
and am hoping (and now checking) that it doesn’t have the same problem as .Close() does.

Spooling up a new thread is very expensive. I would propose one of three alternatives:

1. The simpler alternative is to use a thread pool, either the system thread pool or a hand-crafted pool. Avoiding the thread startup cost will reduce the load on the server.

2. An option that is even nicer on the server is to use a single, persistent thread for logging. The first request that needs to do logging fires it up, and then from that point on, log entries are placed into a sync queue which the persistent thread pulls from. Example:

static object log_sync = new object();
static Queue<LogData> log_queue = new Queue<LogData>();
static bool log_thread_running = false;

static void post_log_entry(LogData log_entry)
{
lock (log_sync)
{
log_queue.Enqueue(log_entry);

if (log_thread_running)
Monitor.PulseAll(log_sync);
else
new Thread(log_thread_proc).Start();
}
}

static void log_thread_proc()
{
lock (log_sync)
{
if (log_thread_running)
return;

log_thread_running = true;

try
{
while (true)
{
while (log_queue.Count == 0)
Monitor.Wait(log_sync);

LogData one_item = null;
List<LogData> multiple_items = null;

if (log_queue.Count == 1)
one_item = log_queue.Dequeue();
else
{
multiple_items = new List<LogData>(log_queue);
log_queue.Clear();
}

// The following block: Exit; try/finally{Enter}
// ..is the logical inverse of a lock() block. :-)
Monitor.Exit(log_sync);

try
{
if (one_item != null)
process_log_entry(one_item);

if (multiple_items != null)
foreach (LogData item in multiple_items)
process_log_entry(item);
}
finally
{
Monitor.Enter(log_sync);
}
}
}
catch (Exception e)
{
// TODO: log this unexpected error
}
finally
{
log_thread_running = false;
}
}
}

The only problem with this approach is that if it does get backed up, eventually the worker process will be recycled at which point any log entries not yet written will be lost.

3. The most scalable approach, but the one that is the most work, is to create a separate Windows Service in charge of the logging. The ASP.NET worker thread then maintains a single, persistent connection to it and posts the log items across the boundary. The advantage of this is that if the system becomes bogged down, the logging will fill up the queue in the IPC connection and cause requests to block within the ASP.NET worker thread, thus implicitly throttling the connection rate. I would not use .NET Remoting for this, as it will spawn a separate channel for each thread. The throttling comes specifically from having only one channel. Example (client side):

static object log_sync = new object();
static BinaryFormatter log_formatter = new BinaryFormatter(); // in System.Runtime.Serialization.Formatters.Binary
static Stream log_stream;

static void post_log_entry(LogData log_entry)
{
lock (log_sync)
{
if (log_writer == null)
{
Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);

// In practice, I would let the OS pick the port number when binding in the Windows Service
// and write it to a central location that the ASP.NET process can read from.
socket.Connect(new IPEndPoint(IPAddress.Loopback, SecretPortNumber));

log_stream = new NetworkStream(socket, true);
}

log_formatter.Serialize(log_stream, log_entry);
}
}

For this to work, obviously the LogData class needs to be in a separate interface DLL that can be referenced from both the ASP.NET project and the Windows Service. You mention in the article that a friend suggested this sort of off-loading but that thread creation would be required. This example shows that thread creation would not be required. The Windows Service can operate in a single-threaded mode:

class LogService : System.ServiceProcess.ServiceBase
{
static void Main(string[] args)
{
if ((args.Length > 0)
&& string.Equals(args[0], "/console", StringComparison.InvariantCultureIgnoreCase))
{
LogService service = new LogService();

service.StartDirect();
Console.WriteLine("Press enter to stop debugging");
Console.ReadLine();
service.StopDirect();
}
else
ServiceBase.Run(new LogService());
}

LogService()
{
ServiceName = "LogService";
CanStop = true;
}

public void StartDirect()
{
OnStart(null);
}

public void StopDirect()
{
OnStop();
}

protected override void OnStart(string[] args)
{
socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);

// Again, in implementation, change this to bind to port 0 and then after the Bind call
// has succeeded, read the port number back from the LocalEndPoint property and write it
// to a place where the ASP.NET side can read it.
socket.Bind(new IPEndPoint(IPAddress.Loopback, SecretPortNumber));

socket.Listen(5);

shutdown = false;

Thread main_thread = new Thread(main_thread_proc);

main_thread.IsBackground = true;
main_thread.Start();
}

protected override void OnStop()
{
shutdown = true;
}

Socket socket;
bool shutdown;

void main_thread_proc()
{
BinaryFormatter log_formatter = new BinaryFormatter();

using (NetworkStream log_stream = new NetworkStream(socket, true))
{
while (!shutdown)
{
LogData log_entry = (LogData)log_formatter.Deserialize(stream);

process_log_entry(log_entry);
}
}
}
}

Note that the code in this post is untested. It contains all of the necessary inter-thread handling that I could see, but it doesn’t contain adequate error handling. There may also be minor typos. Use at your own risk :-)

Also, the 2048 character post limit is both annoying and buggy; I got my last block down to 2032 characters according to wc.exe, but it still insisted it was too long. I had to split the previous paragraph into a new post.

MSMQ is nice and easy to use, and I don’t think its overhead is particularly high, but as far as I know it does not offer any form of flow control. Worse, when queues start to get big, the entire system’s performance is affected (though this may not apply to your app — it’s something I’ve seen with inbound queues with more than 250,000 pending messages). One advantage of MSMQ is that it would make it relatively easy to offload the logging to a separate physical machine without losing reliability (e.g. the logging machine could be down for repairs every now and then without affecting the fidelity of the log information). It’s all trade-offs :-) As my sample shows, though, what amounts to a custom TCP protocol can be quite trivial to implement using .NET serialization.

Hi,
I have been experiencing this problem as well. Have any of you noticed that no web log entry is written when the call to close is omitted (after a call to transmitFile)?
Thanks,
David.

Mark,

Just read the initial part of your post…

Just wanted to say that we’ve been using MSMQ on http://exposureroom.com which is an extremely busy website and we’ve never had backlog issues or resource issues. In fact when we look at the queue there never anything in it while at the same time we can see that it logs 1000s of messages in a minute sometimes seconds.

We use MSMQ for more than just logging. The entire ExposureRoom system is comprised of many complimentary services and invariably they are long running tasks. These tasks are executed using MSMQ (which triggers a process sending it information via a command line and doesn’t actually do the work itself).

As regards TransmitFile, evenry thumbnail and video we serve is served using TransmitFile. Using Flus() and Close() is a bad idea.

The only Requests that we had hanging were session related (only 4 pages used sessions). So today we don’t use Sessions at all and since making that change no Requests are left hanging.

Thanks for the good write up. If I understand correctly I should have no problem doing the following:

protected void Page_Load(object sender, EventArgs e)
{
string filePath = MapPath("file.pdf");
FileInfo OutFile = new FileInfo(filePath);
Response.Clear();
Response.AddHeader("Content-Disposition", "attachment;filename=" + OutFile.Name);
Response.AddHeader("Content-Length", OutFile.Length.ToString());
Response.ContentType = "application/pdf";
Response.TransmitFile(filePath);
Response.End();
}

So I should have any problems? Or do I not want to call the Response.End()?

great stuff !
you saved my day!