Name: Anonymous 2014-02-08 16:12
Hey! It is me again.
My last posting here was about breakage with GF and that a new .NET job would be much cooler than the former C/C++ one. So the first task, employer gave me, was rewriting legacy PHP integration layer into C#.
The assignment sounded easy, because C# is a compiled language, having limited type inference. Alas after writing the initial prototype, I found that C# fails to give me a way to do efficiently something as a simple text data storage. Yes, after all the rewriting and optimization, PHP version was still faster and taking less memory than C# one, which continuously spitted System.OutOfMemoryException. Other programmers at the workplace were enraged, that I lost C#'s honor to some PHP. But the reality is that C# loses to PHP, when it comes to memory usage.
The PHP code consited of a sender.php, which repeatedly gets called by a cron job to spawn new processes, and a lock.file, which these processes lock before accessing DB. Another watchdog cron job moderates the number of these treads, killing unwanted, so that they won't eat all resources. The system can sustain around 30 of these PHP processes at any single moment. Each sender.php process fetches 10 mb of UTF8 messages from an SQL DB, converts them into x-www-form-urlencoded, compresses through libbz2 and sends the resulting stream as a POST http query to the destination. After transfer is complete, PHP gets killed, freeing all the resources.
In my C# version I copied PHP's behavior exactly: 30 threads taking messages in parallel and establishing 30 connections to destination. Yet C# version was taking 100 megabytes per thread, while PHP only 20mb. The problem unfolded as following:
1. Just like PHP, each C# thread fetched 10 mb of UTF8 messages
2. Yet unlike PHP, C# immediately converts fetched UTG8 bytes into UTF16 (string[]'s internal representation), allocating another 20mb and giving total of 30mb
3. urlencode further converts them, in some cases tripling the size due to the %DE%AD look; adding 40mb, due to the need for StringBuilder to generate the PHP's &hash_map[key0][key1]=fuck+you arrays.
4. Here inefficient bzip2 implementation kicks-in, taking 6 seconds to compress 10mb. Note that PHP's libbz2 is written in C/C++.
5. All C#'s threads are running in a single process, so nothing is freed, after o single thread finishes. This really stresses memory, compared to PHP's shared nothing architecture. And NO, garbage collector doesn't free memory, until it is too late and System.OutOfMemoryException is generated. Even explicitly forcing GC wont help.
All in one, I lost to PHP and lost my job, because team lead got really aggravated to my failure and that I had nerve to defend PHP's approach to memory management. I'm in search of a new job, my dear /prog/. Hope this time it will be at least Clojure, because it uses UTF8 to store strings and Java's GC doesn't produce System.OutOfMemoryException out of the blue.
http://www.youtube.com/watch?v=oxL_xY0Tm2w
My last posting here was about breakage with GF and that a new .NET job would be much cooler than the former C/C++ one. So the first task, employer gave me, was rewriting legacy PHP integration layer into C#.
The assignment sounded easy, because C# is a compiled language, having limited type inference. Alas after writing the initial prototype, I found that C# fails to give me a way to do efficiently something as a simple text data storage. Yes, after all the rewriting and optimization, PHP version was still faster and taking less memory than C# one, which continuously spitted System.OutOfMemoryException. Other programmers at the workplace were enraged, that I lost C#'s honor to some PHP. But the reality is that C# loses to PHP, when it comes to memory usage.
The PHP code consited of a sender.php, which repeatedly gets called by a cron job to spawn new processes, and a lock.file, which these processes lock before accessing DB. Another watchdog cron job moderates the number of these treads, killing unwanted, so that they won't eat all resources. The system can sustain around 30 of these PHP processes at any single moment. Each sender.php process fetches 10 mb of UTF8 messages from an SQL DB, converts them into x-www-form-urlencoded, compresses through libbz2 and sends the resulting stream as a POST http query to the destination. After transfer is complete, PHP gets killed, freeing all the resources.
In my C# version I copied PHP's behavior exactly: 30 threads taking messages in parallel and establishing 30 connections to destination. Yet C# version was taking 100 megabytes per thread, while PHP only 20mb. The problem unfolded as following:
1. Just like PHP, each C# thread fetched 10 mb of UTF8 messages
2. Yet unlike PHP, C# immediately converts fetched UTG8 bytes into UTF16 (string[]'s internal representation), allocating another 20mb and giving total of 30mb
3. urlencode further converts them, in some cases tripling the size due to the %DE%AD look; adding 40mb, due to the need for StringBuilder to generate the PHP's &hash_map[key0][key1]=fuck+you arrays.
4. Here inefficient bzip2 implementation kicks-in, taking 6 seconds to compress 10mb. Note that PHP's libbz2 is written in C/C++.
5. All C#'s threads are running in a single process, so nothing is freed, after o single thread finishes. This really stresses memory, compared to PHP's shared nothing architecture. And NO, garbage collector doesn't free memory, until it is too late and System.OutOfMemoryException is generated. Even explicitly forcing GC wont help.
All in one, I lost to PHP and lost my job, because team lead got really aggravated to my failure and that I had nerve to defend PHP's approach to memory management. I'm in search of a new job, my dear /prog/. Hope this time it will be at least Clojure, because it uses UTF8 to store strings and Java's GC doesn't produce System.OutOfMemoryException out of the blue.
http://www.youtube.com/watch?v=oxL_xY0Tm2w