Last night was no different. I was working on finishing up some final features before the release of version 0.1.0 of ease.js. It wasn’t a horribly productive day (code-wise), but it was still time that I had to squeeze in amongst other errands and the rest of my life. As usual, I committed my changes, found that I wasn’t in a state I wanted to push, suspended my PC and went off to bed. When I awoke in the morning, I found that I wasn’t able to do anything without filesystem errors. None of my drives were readable or writable.

Alright. It’s happened before. I’ll just reboot, run fsck and hope nothing important was corrupted. I wasn’t able to reboot properly (due to filesystem errors), so I just did a hard reset. To my relief, the PC booted back up just fine. I was able to reconnect to the network, restore my screen session on my server for IRC and re-open vim to begin hacking. All seemed well.

Normally, when I open vim after shutting down without giving it a chance to close, I’m spammed with messages regarding its swap files. While vim is running, it writes your changes to a swap file, one per open file, just in case something happens that causes vim to shut down without being able to save your changes. Next time you open that file, vim will notice that there’s already a swap file an will prompt the user, asking what should be done. Should vim recover from the swap file, delete it, or ignore it and edit the file anyway? Normally, I delete them all and start fresh, because everything is normally committed in Git.

But I noticed something odd. My shell, when in a Git project dir, will add additional information to the prompt (see git-ps1). Rather than saying “master” as it should of, it said “HEAD”. Sensing a problem, I didn’t delete the swap files. Instead, I typed ‘git status’ and was rather discomforted to find that every file was listed as modified and no previous commits were recognized.

Okay. That’s happened before too. Somehow, the HEAD was lost, likely due to corruption. Since most commands fail with an invalid HEAD, I just cloned my repository from GitHub and put the SHA-1 hash of its head into .git/HEAD. Worked like a charm. Unfortunately, `git reflog` was not displaying anything useful, so I couldn’t just check out one of my more recent commits. I then ran `git fsck --lost-found` to hopefully recover the commits I had made since I last pushed. It complained with a fatal error, stating that a commit was corrupt.

Suffice it to say that many commits were corrupt. After a while of trying every trick in the book for recovering lost commits (which I’m not going to go over here), I decided that it was a lost cause. Not having run `git gc` since the start of the project, there were files everywhere. There was no way I was going to manually sort through these to figure out my commit, and `git fsck` was useless because, every time I recovered or removed corrupt commits, another one was a problem. It was clear that this repository was a lost cause.

No developer likes to lose hours of work. As I said, I have to squeeze the time in when I can. Lost time is truly lost. I only have so much time in my day and, if I couldn’t recover that work, I may not be able to make the v0.1.0 release that I had hope to this weekend. Or at least get as close as possible to it. So, naturally, that was very disappointing.

Recovering the exact commits would have been ideal. The timestamps, original commit messages, etc are nice to have. With that lost, I at least wanted to try to recover the code. If I could do that, I could split it into a number of commits and hopefully get back to where I was before. So, I looked back to vim. Because I was unable to shut down the PC properly, none of the swap files were cleaned up. Therefore, I moved my old, corrupt repository to a temporary location, moved the new clone into the same dir as the original, and then chose the “recover” option for each of the files in my session and wrote them all to disk. That recovered most of my old files, which was very exciting.

Before I had shut down, I attempted to use :mksess to create a session to restore all my open files after reboot. However, since the disk was not writable, I was unable to do so. Therefore, I was stuck with an old session, which did not reload many of the files I had open. Some of those files didn’t even exist in the newly cloned repository. Fortunately, simply opening those non-existent files in vim would cause the swap file to be recognized. Vim would prompt, I would recover from the swap file, write it to disk and have my old file back.

Though I lost the commits, I now had all of my lost code back where it belongs. After recommitting everything, I didn’t have as many commits as I had previously, nor did I have the original messages, timestamps, etc, but at least I didn’t lose any actual work. Just a little bit of time, for which I was able to share an interesting experience. That has not been the only time that vim’s swap files have saved me from strangling myself. While it’s not good to rely on them for a backup solution, they are able to help in situations where you’re left with no other choice.

Had the entire disk been corrupted, I still would have been okay. My repository is on a separate physical disk than my home directory. Using the directory option in vim, you can set a different location for your swap files (it’s in the same directory as the file, by default). I set mine to ~/.vim-tmp by default. Had they been in the same directory, it’s likely that they may have been accidently lost while I was screwing around with Git trying to recover everything. If the entire disk did fail, I would have still had my work in the swap files, so I would have lost only the commits, not the code.

Moral of the story – store your swap files wisely. They’re an annoying pest when you don’t need them, but highly valuable in the off chance that they are needed.

http://stackoverflow.com/questions/3679233/are-there-still-ecmascript-3-implementation-differences-in-major-browsers/3679293#3679293

To clarify, this is the problem:

    var obj = {};
    obj.toString = function() {};
 
    for ( prop in obj )
    {
        // "prop" will never show 'toString' in IE
    }

I needed some way to work around this. The solution needed to be fairly elegant – that is, I don’t want to have to pass in an argument stating “yes, I set the toString property”. My workaround, though not pretty, was to test to see if toString was set:

    function hasToString( obj )
    {
        return ( obj.toString === Object.prototype.toString )
            ? false
            : true
        ;
    }

If the toString property of the object is equal to that of its prototype, then it’s safe to say that the property was not overwritten. However, there is a problem. We’re using Object.prototype for this test. We are not checking the parent. This is because IE does not have a __proto__ property like other browsers such as FF or Chromium. So we cannot test against the actual prototype.

This means that the above function will give you an incorrect result if the toString property was previously overridden.

Fortunately, in my use case, that will never be an issue. If it is an issue for you, then make sure you replace Object with whatever obj is an instance of.

• Freedom 0. the freedom to use the software for any purpose,
• Freedom 1. the freedom to change the software to suit your needs,
• Freedom 2. the freedom to share the software with your friends and neighbors, and
• Freedom 3. the freedom to share the changes you make.

The license is the child of Richard M. Stallman – the founder of the free software movement – and the Free Software Foundation – the not-for-profit organization that was born from his vision. There are multiple incarnations of the license, each with their own distinct purposes:

• GNU General Public License (GPL) – Designed to protect and enforce the aforementioned freedoms. All derivative works must also be licensed under the GPL or a GPL-compatible license.
• GNU Lesser General Public License (LGPL) – Designed to protect the four freedoms, but permits usage in proprietary applications to a limited degree. Derivative works must still be licensed under a GPL-compatible license.
• GNU Affero General Public License (AGPL) – The most noble and powerful of the GPL licenses. It is an extension of the GNU GPL, with an added clause requiring that users accessing software through a server must have access to the software’s source code (e.g. a website/web service). All derivative works must be licensed under the AGPL or a compatible license.

In addition to the above licenses, the GNU Free Documentation License (FDL) is also available for documentation and other written works.

The licenses attempt to defend a very noble cause – the free software movement. You will notice that all of the above licenses have something in common – they permit derivative works only if they are licensed under a compatible license. This ensure that the software will forever remain free software and will never be a part of proprietary software, which is not tolerated by the free software community. This is known as a copyleft.

And this is where, for some, the problem begins.

Why Copyleft?

As mentioned above, the GNU licenses are all copyleft. Copyleft is a strong alternative to other methods of making software free, such as putting it into the public domain, that provides incentive to improve upon free software. By placing software into the public domain, for example, the author allows it to be incorporated into proprietary software, or even public the exact code as proprietary software. There are no restrictions in place and the freedoms of the users remain unprotected.

Simply applying a free software license to software doesn’t solve the issue. Consider, for example, one of the most permissive software licenses – the MIT/Expat license. It is a non-copyleft license, permitting the use of the code within proprietary software so long as the license text is distributed with the resulting software. This is a problem. Should the code be included in proprietary software, it violates all of the four freedoms:

• The software could only be used under the terms permitted by the license of the proprietary software vendor,
• the software could not be altered to suit your needs, as you would not have access to the source code,
• the vendor may prevent you from sharing the software with your friends and neighbors, and
• it would be illegal to share your changes.

It is exactly this that the free software community is against and what copyleft aims to prevent. In fact, by using a non-copyleft license, the author of the software is failing to protect the freedom of his/her users by permitting derivative works to completely trash the user’s freedoms.

Why [Not] Copyleft?

There are many opponents to the GPL and copyleft. They argue that copyleft imposes unnecessary restrictions on developers by preventing them from using free software as they wish, wish some consider unethical. Denying a developer a right to use code however they please is denying a freedom. Copyleft has been deemed “viral” (by all means of the word, it is – all derivative works must also be copyleft, but the term “viral” has a highly negative connotation). Terms such as the GNU “General Public Virus” emerged shortly after its release.

It is noble to release your code into the wild with no restrictions, saying “I made this for the world and the world may use it as it pleases.” Your mind is likely in the right place. However, as mentioned above, you are failing to protect the freedoms of your users. Consider, for example, the development of the nuclear fission. Let’s say that you were the one to discover the splitting an atom releases vast amounts of energy. You think to yourself that this has incredible implications for humanity. A viable, clean energy source – nuclear power. You release this information to the world with excellent intentions – to better humanity. Your intention was totally selfless. You’re asking nothing in return for this incredible discovery.

Months later, you hear that a city was completely annihilated. They used a bomb that exploited your discovery, calling it the “atomic bomb”. Your wonderful discovery has just been used in a way unimaginable, taking advantage of all your hard work to do harm to others.

Proprietary software is by no means comparable to the death of millions of individuals (in my opinion). It is, however, comparable to taking your beautiful, hard work that you released for the better of humanity, and using it to control users. This is what you are potentially dooming users of your software to by using a non-copyleft license.

Some also argue that copyleft is no longer needed, because taking free software and using it in proprietary software has become a greatly unethical practice. That is true within many communities. But that will not stop others from taking your software and using it unjustly or attempting to circumvent licensing terms to use your software in a manner that is non-free.

Copyleft Resistance

This article was prompted by a couple of factors within a single day. The main reason was that I noticed strong resistance to the GPL within a piece of software I was looking at to determine if I needed to develop a free alternative for myself and others. I noticed the following in the licensing terms:

## LICENSE:
## 1. You may not make this GPL ever. I want my code to be freely
##    available to everyone forever. The GPL restricts freedom.
## 2. Copyright notices must remain intact and distributed with
##    the program. This includes the contributors list.
## 3. No warantee is present, whether express or implied. Use at
##    your own risk.
## 4. This license must be included with all distributions of this
##    program and no modifications to this license are allowed.
##    This implicitly makes this program GPL incompatible but
##    compatible with virtually every other OSI approved license.
## 5. You are otherwise free to distribute this program, modify it
##    and distribute those modified works.

I took the time to post a response on this person’s blog (the text below contains only minor grammatical fixes from the original and hyperlinks):

I noticed the following comment in your license:

"The GPL restricts freedom."

This seems like a contradictory statement and brings into question your
definition of “freedom”. Had you used the GPL in order to license this work, you
would have ensured that the source code will be freely available indefinitely. You would
have in fact been protecting the freedom of your users.

You impose only restrictions on your license. You impose no restrictions
whatsoever on the source code itself. This means that someone would be able take
your script and use it in their own proprietary project, obfuscating or
otherwise compiling the code, and still follow the license terms so long as they
distribute your license and do not modify it. In doing so, the freedoms of your
users have been assaulted. Should they receive a copy of this modified
software, they are not free to study and alter the source code – it is in a
state that makes doing so very difficult.

By your statement, I assume you are referring to the “viral” nature of the GNU
General Public license – that it forces any derivative works to also be free,
released under a compatible license. Some people see this as an assault on their
freedom to distribute and use the software however they please. But the GNU GPL
protects the user, not the developer – ensuring that users are not bullied and
taken advantage of. With your license, are protecting the developer (permitting
them to do whatever they wish with your code), but dooming the users of your
software.

It should also be noted that your software is stand-alone. It is not a library.
Had it been released under the terms of the GPL, you would have protected the
four essential freedoms. You also would have permitted developers to use your
software in proprietary software, because they could simply call your script and
use its output in their proprietary software.

Therefore, you have gained nothing here by disallowing the GPL, but instead have
forfeited the four essential freedoms.

When I returned to the blog to see if the author had responded, I found that my post had been deleted. The author either did not care about the freedom of his/her users or did not understand the implications of his actions or copyleft.

What the author of this software had done was develop his own Expat license or other highly permissive licenses (such as the original/modified BSD). He/she places restrictions only on the license itself – not the source code. The user was free to distribute and modify the source code in any shape or form, including proprietary.

Rather than simply using an existing permissive license, the author decided to explicitly state that the license should not be compatible with the GPL. Therefore, ironically, the author was placing more restrictions on the users than virtually any other permissive license.

"[...] I want my code to be freely available to everyone forever."

The author failed in doing so. By stating that the code cannot ever be licensed under the GPL, he/she ensures that the code can never be incorporated into any projects using the most popular free software license in the world. There are many other licenses, but the author has placed favor on proprietary software by disallowing so many free software projects the right to use his/her code.

"[...] The GPL restricts freedom."

This was the line that really got to me. It is a fascinating line, because it represents a large difference in opinion – the difference in what we consider “freedom” to be. The GPL was created to protect users’ freedoms. Why, then, does the author claim that the GPL restricts freedom? As the author did not respond, I can only speculate. Likely, it represents the concept mentioned above – that the GPL protects freedoms of the user, not the developer. The author decided that it was more important to him/her that his/her code be able to be used anywhere and specifically singled out the license designed to protect the freedom of users. Therefore, the author decided that it was ethical to allow derivative works to control its users.

Guardian or Curse

Copyleft (and consequently, the GPL) is something people tend to have strong feelings for on either side – they love it, or they hate it. I’m not one to state that someone else’s opinions are incorrect. I personally strongly support copyleft and philosophy of the GNU Project. The GPL has been the force driving the free software movement for two just over two decades. It allows individuals to develop software that respects the freedom of users – ensures that they cannot be controlled and bullied – and ensures that their software will never fall and surrender those freedoms by enforcing those freedoms with copyright.

At the same time, the GPL is an assault on freedom to developers. And that is a good thing! Large corporations use their proprietary software to control their users. For example, Amazon used DRM on the Kindle to remove copies of the book 1984 from users’ devices – something that would not have been possible with free software. DRM cannot exist within free software. Users would simply remove the offending code from the software and likely fork the project, leaving the original author in shame.

Proprietary software can also be used to spy on your personal information, with no way to disable it. In fact, devices like the iPhone/iPod/iPad cannot be modified at all. They must be “jailbroken” in order to do anything that Apple does not want you to do. It is the ultimate form of control over users. And who’s controlling the users? The developers.

The GPL is a guardian to those who believe that the user should, above all else, be free to use, alter and distribute the program as they wish. The GPL is a curse to those who wish to take advantage of their users by creating proprietary software in order to control and exploit them.

More Information

For more information on free software, GNU GPL and copyleft, please see the links throughout this article. For your convenience, many have been summarized in the list below:

• Free Software Foundation
• The GNU Operating System
• The Free Software Definition
• Copyleft
• Why Copyleft?
• Copyleft: Pragmatic Idealism
• List of Software Licenses
• Why Upgrade to GPLv3
• Categories of free and nonfree software
• The X Window System Trap
• Misinterpreting Copyright – A Series of Errors
• Copyright versus Community in the Age of Computer Networks
• Free Software Movement (Wikipedia)
• Software Freedom Law Center
• Richard M. Stallman

I also encourage you to take a look at the following related websites::

• Patent Absurdity
• End Software Patents
• LibrePlanet
• Defective By Design
• Opposing Digital Rights Management
• Windows 7 Sins
• RIAA Lawsuits – Expert Witnesses Fund
• Campaign Against the ACTA
• Hardware Devices that Support Free GNU/Linux
• High Priority Free Software Projects
• Join the Free Software Foundation

Disclaimer: No attempt was made to provide an unbiased article regarding the GNU GPL and copyleft. It can be made very apparent from the rest of this website that I am a strong supporter of free software and copyleft. This article is intended to support the GPL and copyleft. I am an associate member of the Free Software Foundation and a free software developer and user.

Modules are brought into scope with the require function. You then have access to the exports of that module. In the example below, we’re interested in the Foo class of the foobar module.

// Load the "Foo" class from the "foobar" module
var Foo = require( './foobar' ).Foo,
    obj = new Foo();
 
// we would expect the following to be true, and we are correct
console.log( obj instanceof Foo ); // true

Obviously, obj is an instance of Foo. Pretty straightforward. If you use the module elsewhere in your code, you would expect that obj is still an instance of Foo:

// ...
 
function foo( obj )
{
    return ( obj instanceof require( 'foobar' ).Foo );
}
 
console.log( foo( obj ) ); // false

Huh? That above function, foo, is returning false! That did not make sense to me. As it turns out, the problem was with how I was including the module. Early on in the project, I included the module as above – by prepending ./ to the path. However, a more convenient method is to simply add it to the include path:

// add $CWD to the include path
require.paths.unshift( '.' );

You will notice that, in foo(), we were using a different path for the module. They may resolve to the same path ultimately, but Node seems to keep track of them differently.

require.paths.unshift( '.' );
 
var Foo1 = require( './foobar' ).Foo,
    Foo2 = require( './foobar' ).Foo,
    Foo3 = require( 'foobar' ).Foo,
    obj  = new Foo1();
 
console.log( obj instanceof Foo1 ); // true
console.log( obj instanceof Foo2 ); // true
console.log( obj instanceof Foo3 ); // false

This was tested with node v0.3.1-pre. As far as I am concerned, this is a bug. Foo1, Foo2 and Foo3 are all in fact the same object, however Node will reimport the module when it is included via a relative path in require.paths. If you use an absolute path in require.paths, this is not an issue. To demonstrate this, consider the following complete example:

/**
 * foobar.js
 */
console.log( '>>> foobar module loaded' );
 
exports.Foo = function()
{
}
 
 
/**
 * test.js
 */
require.paths.unshift( '.' );
 
var paths = [
    './foobar',
    __dirname + '/foobar',
    '././foobar',
    '../module-oddity/foobar',
    'foobar',
];
 
var funcs = [],
    objs  = [],
    i     = 0;
 
// load module and instantiate objects
paths.forEach( function( path )
{
    funcs[ i ] = require( path ).Foo;
    objs[ i ]  = new funcs[ i ];
 
    console.log( "Foo%d = %s", i, path );
    i++;
});
 
// compare objects
var foo_i = obj_i = 0;
funcs.forEach( function( foo )
{
    foo_i++;
    obj_i = 0;
 
    console.log();
 
    objs.forEach( function( obj )
    {
        console.log( "Foo%d instanceof obj%d: %s",
            foo_i,
            ++obj_i,
            ( obj instanceof foo )
        );
    });
});

$ node test.js
>>> foobar module loaded
Foo0 = ./foobar
Foo1 = ./foobar
Foo2 = ././foobar
Foo3 = ../module-oddity/foobar
>>> foobar module loaded
Foo4 = foobar

Foo1 instanceof obj1: true
Foo1 instanceof obj2: true
Foo1 instanceof obj3: true
Foo1 instanceof obj4: true
Foo1 instanceof obj5: false

Foo2 instanceof obj1: true
Foo2 instanceof obj2: true
Foo2 instanceof obj3: true
Foo2 instanceof obj4: true
Foo2 instanceof obj5: false

Foo3 instanceof obj1: true
Foo3 instanceof obj2: true
Foo3 instanceof obj3: true
Foo3 instanceof obj4: true
Foo3 instanceof obj5: false

Foo4 instanceof obj1: true
Foo4 instanceof obj2: true
Foo4 instanceof obj3: true
Foo4 instanceof obj4: true
Foo4 instanceof obj5: false

Foo5 instanceof obj1: false
Foo5 instanceof obj2: false
Foo5 instanceof obj3: false
Foo5 instanceof obj4: false
Foo5 instanceof obj5: true

According to the above test, you’ll notice that every relative path is considered to be an instance of the same Function. However, the single Function loaded from a URL that must be resolved though require.paths is considered to be a different Function. Also notice that the module is re-imported. Be sure to keep your naming conventions consistent.

So, there are two solutions to this issue:

• Use absolute paths. You can avoid this issue by using require( __dirname + '/path' ) rather than require( './path' ).
• Access modules consistently. If you use ./path, always use that path when accessing the module.

Hopefully in future versions of Node, this will not be an issue (I’m looking forwarding to putting a strikethru throughout this entire post). Until then, hopefully this saves you a confusing game of cat and mouse.

A patch for this problem is available on github.

When attempting to log into the server, I was prompted for a password as if the server did not have my public key. It certainly did. Checking /var/log/auth.log, I noticed this:

# /var/log/auth.log
Nov 2 17:47:37 hostname sshd[2053]: Authentication refused: bad ownership or modes for directory /home/myuser

Somehow (likely due to user error), the permissions for /home/myuser was set to 0777!

$ ls -lah /home | grep myuser
drwxrwxrwx 119 myuser myuser 8.0K 2010-11-02 17:45 myuser

The solution was to restore the directory permissions to what they should be – r/w access for the owner only.

$ chmod 0755 ~myuser
$ ls -lah /home | grep myuser
drwxr-xr-x 119 myuser myuser 8.0K 2010-11-02 17:45 myuser

Hopefully now you’ll be able to log in with your private key.

$ ssh myhost
$ tail /var/log/auth.log
Nov  2 17:50:36 hostname sshd[2212]: Accepted publickey for myuser from 192.168.1.XXX port XXXXX ssh2

During the keynotes this morning, one of the presenters (who is a part of the GNOME foundation) stated how, in the past, she would bring a laptop running Microsoft Windows into work. It would boot up and make that distinctive noise and everyone would turn to her. They would ask why she was running a proprietary OS, and she would make excuses for it. But not everyone made such excuses. She stated that it is because of those who decided to make such sacrifices – to spend hours getting things to work using free software – that free software is where it is today. She encouraged attendees to use free software themselves. Yet, they sit there typing notes in Microsoft Word. Do the attendees not understand the values of free software? Do they even understand the values of open source? I know some of them do – there are some there that have GNU stickers on their laptops, run GNU/Linux operating systems, etc. But not enough. It is astonishing and shameful.

The conference does little to encourage free software either. Yes, there are some presenters that make the distinction between free software and open source. However, there are also sessions that are essentially advertisements for proprietary solutions! For example, a vendor may have an “open source” or “community” version of their product, which they advertise as a trial. One such phrase I heard in one of the sessions was “we encourage you to at least try the community version”, or to try the community version “first”. They would then go through a list of anti-features in the “open source” version that essentially make it useless in any performance-intensive or enterprise settings, forcing users to upgrade to their “enterprise” versions. This is an excellent example of the distinction between the free software philosophy and simply making your product “open source”. In this instance, it appears to be a sales pitch. The vendor then went on to state that they have patents pending for their software. Let’s not even get me started on that. Why would you have such a session at an open source conference?!

Clearly, while many of the attendees and presenters do recognize software freedom and understand concepts that give the meaning behind the term “open”, there are many that do not. OSCON also meets the needs of business owners. I am willing to bet that the vast majority of those using proprietary devices were there for their companies. Hell, I’m there because my employer funded it – it’s an expensive conference. However, this is also a conference I would love to go to on my own. I understand the core principles behind free software. I understand the open source movement. But if your employer simply sees open source as a benefit to business and sends you there, and you are just an employee…well, of course you’re not going to care. You’re just there to see what new and exciting technologies exist to help your employer or your own business.

I wish during the tutorial sessions the first two days that I took the time to count the number of laptops and devices in each of the areas I entered in order to give an actual figure. Now that the tutorial days are over, there are much less devices. Even still, out of a handful of 20 or so laptops, I’m lucky to see a single one running a free operating system. Even the majority of the presenters used Macs. However, some of those presenters had no idea what they were doing on a Mac, leading me to believe that OSCON supplied them! That is even worse! It left a very bad impression. For those attendees that I did see at OSCON that understood and embodied the spirit of the open source movement (or even better, the free software movement), I was very pleased. For everyone else – very disappointed. I hope that they are able to take something away from those speakers that do mention free software and do explain the values of open source.

This also brings up another thought. Lack of presence from members of the Free Software Foundation. The most I saw of them was a single booth in the Expo hall. Where are the speakers? Where are the sessions? (In their defense, I understand that there were hundreds of session proposals that were unable to make it into the conference. Maybe someone tried. I’ll ask FSF directly.) There are little tags you can put on your badges, such as “Git”, “Desperate Perl Hacker”, etc. Why is there no “Free Software” one? Why isn’t Richard Stallman himself helping to spread the word of free software at such a large conference? Now, don’t get me wrong – I completely understand him not being there. If I were Stallman I’d probably have a heart attack from what I saw there (metaphorically speaking – I almost did). But that is not the point. The point is that there are plenty of people there who have been introduced to this concept – may even be new to the concept – of “open source”. They may not understand what free software is, or why it is important. But they may be interested in it. How often do you see such a large group of software enthusiasts and developers in a single location for a conference focusing on openness in software? This would have been an excellent opportunity for the Free Software Foundation to spread its values. If anything, help combat the vendors that are lacing their “open source” products with proprietary bits. Give out CDs containing free GNU/Linux distributions. Explain our core values. Ask people to join. I was very disappointed by FSF’s presence at that conference. I’ll bring that up in the appropriate channels – I just wanted to mention it publically as well.

Overall, OSCON has been an interesting experience thus far. I’m not trying to recognize it as a free software conference – it clearly isn’t, and I never expected it to be. However, it is an excellent learning experience and there are many free software projects represented there.

Oh, and a last note that just puts the icing on the cake: After the conference, I went on a walking tour of Portland. The tour guide saw an iPhone and stated, “Isn’t this an open source conference? Aren’t iPhones kind of anti-open source?” (I quoted him to the best degree I possibly could, though those may not have been his exact words.) Yeah, that’s just great. A tour guide was able to recognize that. Yet the group of OSCON attendees that were asked that question remained silent. Swell.

First, let’s generate an array to work with. We’re going for quantity here, not size of data. Therefore, we’ll simply generate an array containing a list of integers. PHP offers a quick and easy way to do this using the array_fill() function:

// generate an array containing 100,000 elements, each with the value of 1
$data = array_fill( 0, 100000, 1 );

We want to use the array_fill() function rather than range() here because we will be summing up the values in the examples below. Range will generate incremental values, leading to integer overflows and inconsistent results across various architectures.

Note: This article assumes that the reader is running at least PHP 5.0.0. If you are running anything else, you should understand that official support for PHP 4 ended back in 2008 and you should consider upgrading immediately.

for loop

Those coming from a procedural background may be most familiar with the for loop as a means of iterating through an array. Since it is so common, let’s take a look at that first:

$time_start = microtime( true );
$total      = 0;
 
// loop through each of the array items and sum them up
for ( $i = 0; $i < count( $data ); $i++ )
{
    $total += $data[ $i ];
}
 
// calculate total running time and output the result
$time_end = ( microtime( true ) - $time_start );
printf( "Total: %d\nloop time: %fs\n",
    $total, $time_end
);

Let’s run that test and see what type of results we get:

Total: 100000
for loop time: 0.127712s

Ouch. The time that is returned from the above test will depend on the speed of your system. I do know that, on my system, that’s a lot of time to spend simply adding up elements of an array. Let’s see how we may be able to improve that a bit.

The first thing that may jump out is the use of the count method in the condition of the for clause:

for ( $i = 0; $i < count( $data ); $i++ )

It may help to understand exactly how PHP executes the above statement. The assignment portion of the loop is executed once before any iterations begin. The step/count, which is the $i++ part, is executed after each iteration. The test expression (middle) is executed before each iteration to determine if the step should be executed and if we should continue with the loop.

Alright, that’s pretty straightforward. But that is the issue. That count function is being executed for every iteration. PHP is not caching the result of that function. That gets pretty expensive considering that it would be calling the function no less than 100,000 times.

Let’s see what happens when we change it up a bit:

// ...
 
// loop through each of the array items and sum them up
for ( $i = 0, $count = count( $data ); $i < $count; $i++ )
{
    $total += $data[ $i ];
}
 
// ...

Total: 100000
for loop time: 0.046609s

Wow – that is a huge improvement! We nearly tripled the speed of the iteration simply by having the conditional check a precalculated value, rather than calling the function with each iteration. That’s pretty good, right?

No; it’s still pretty slow. We can do much better.

while loop and current()

Some people may prefer to use a while loop in order to loop through the elements of an array. One of the most common ways of doing that is to use the current() function in conjunction with next():

// loop through each of the array items and sum them up
while ( $val = current( $data ) )
{
    $total += $val;
    next( $data );
}

So how does that fair speed wise. It certainly looks a little less complicated than the for loop…that means it’s faster, right?

Total: 100000
for loop time: 0.175094s

Yikes. We took a step in the wrong direction. Let’s pretend we didn’t see this.

foreach loop

PHP 4 introduced the foreach construct which provides a much more simple means of iterating through an array. Surely if it was designed for arrays, it should be faster than for:

// loop through each of the array items and sum them up
foreach ( $data as $val )
{
    $total += $val;
}

Notice that, in the above snippet, we are also avoiding the use of a temporary variable $i to store the current index. While PHP is undoubtedly doing so internally, we’re talking about a difference between compiled and optimized C code versus interpreted, sluggish PHP code. So that alone will provide us with a bit of a benefit, however slight.

Let’s see the result:

Total: 100000
for loop time: 0.037076s

Not too bad…we managed to shave off nearly 1/4th the time of the for loop. But I’m not satisfied.

The Problem

What do all of the above have in common? They’re looping structures. Great, so what’s the issue? If you’re coming from a compiled, high-performance language like C, there is no issue. Let’s see how long it’d take to iterate through an array containing 100,000 elements in C:

#include <stdio.h>
#include <sys/time.h>
 
#define DATALEN 100000
 
int main()
{
    int            data[ DATALEN ];
    register long  i;  
    unsigned long  total = 0;
    struct timeval time_start, time_end;
    long           elapsed, uelapsed; 
    double         elapsed_total;
 
    // create our data array
    for ( i = 0; i < DATALEN; data[ i++ ] = 1 );
 
    // get current time
    gettimeofday( &time_start, NULL );  
 
    // loop through the array summing up the values
    for ( i = 0; i < DATALEN; i++ )
    {   
        total += data[ i ];
    }   
 
    // get total running time
    gettimeofday( &time_end, NULL );
    elapsed       = ( time_end.tv_sec - time_start.tv_sec );
    uelapsed      = ( time_end.tv_usec - time_start.tv_usec );
    elapsed_total = ( elapsed + ( (double)( uelapsed ) / 1000000 ) );
 
    // output total and loop time
    printf( "Total: %ld\nloop time: %fs\n",
        total, elapsed_total
    );  
}

Total: 100000
loop time: 0.000273s

Well, then. The point here is – PHP is a slow language, especially for looping. Wherever possible, you want to avoid it.

But you need to iterate through your arrays somehow! Does that mean you are stuck with PHP’s slow loops?

Use Built-In Functions

Fortunately, there’s certain functions available to you that can drastically improve the speed of array processing. In fact, PHP has a whole slew of functions devoted to processing and altering arrays.

Programmers coming form other languages such as C may prefer to write their own loops for processing data, because it is faster than a function call. This is not true at all with PHP; in fact, much of PHP’s power comes from it’s built-in functions. PHP itself and its libraries are written in C. Therefore, if you can use a built-in PHP function that will do the looping for you, that loop will be executed much more quickly.

Let’s take another look at our above example – summing up 100,000 values of an array. Is there a PHP function that could do this for us? array_sum() perhaps?

$time_start = microtime( true );
 
// sum up the values in the array
$total = array_sum( $data );
 
// calculate total running time and output the result
$time_end = ( microtime( true ) - $time_start );
printf( "Total: %d\nfor loop time: %fs\n",
    $total, $time_end
);

Above, we are making a single function call to array_sum() rather than going in a loop and manually adding everything up. Let’s see how this method does:

Total: 100000
for loop time: 0.012571s

Not bad. Using everything discussed here, we’ve cut our time down from the original 0.127712s to a mere 0.012571s. It’s still no C code, but our loop is 10 times faster than it was originally. It’s still written in PHP; we’re just using the tools we were given.

But wait – I said that PHP’s libraries were written in C. Why isn’t array_sum() just as fast as the C example provided above? Unfortunately, we’re still dealing with PHP’s variables and other internal representations here. It’s not just jumping to a position in memory like it does with an array in C. The array_sum() function still has to scan the PHP array to retrieve all the values. There’s no getting away from that.

Expense of Function Calls

One must also be mindful of what their loop is doing. Take a look at the following example. It calls a function in order to count all the values in an array that contain the number 2.

<?php
 
function check_value( $val )
{
    return ( $val === 2 )
        ? 1
        : 0
    ;
}
 
// generate an array with 1/3 values containing 1, the rest containing 2
$data = (
    array_fill( 0, 10000, 1 )
    + array_fill( 10000, 20000, 2 )
);
 
$time_start = microtime( true );
 
// count the values in the array greater than 50000
$count = 0;
foreach ( $data as $val )
{
    $count += check_value( $val );
}
 
// calculate total running time and output the result
$time_end = ( microtime( true ) - $time_start );
printf( "Count: %d\nfor loop time: %fs\n",
    $count, $time_end
);

Count: 20000
for loop time: 0.043597s

Above, we called the check_value() function 30,000 times. Functions do an excellent job at improving code clarity and reducing duplicate code. However, function and method calls are expensive. Consider what happens when we get rid of the function and do all the processing in the body of the loop:

// count the values in the array greater than 50000
$count = 0;
foreach ( $data as $val )
{
    if ( $val === 2 )
    {
        $count++;
    }
}

If we run it again using that loop instead of the previous one, we get the following result:

Count: 20000
for loop time: 0.013397s

That’s a significant speed improvement. Now, I’m not advocating that you get rid of all of your functions (though I would advocate preprocessor macros in PHP). However, consider the performance implications of excessive function or method calls where it is important. In most situations, the benefits you gain from legibility will outweigh the performance pitfalls.

It should also be noted that under many circumstances, the overhead of the function call may be insignificant. If your method takes 0.3 seconds to execute, you’re not going to care about milliseconds of overhead.

Memory Tradeoff & Apparent Processing

Let’s continue with the example from the previous section (counting the number of elements in an array that contain the value 2). PHP does not contain a function that can explicitly search an array for the given value and return the number of results. It does, however, have something very similar – array_count_values().

The array_count_values() function will loop through the array and maintain a count of each of the values. It will then return an array containing the results. Therefore, in our above example, it would return an array much like the following:

array(
    1 => 10000,  // there are 10,000 1s in our data array
    2 => 20000,  // there are 20,000 2s in our data array
);

That’s a bit more information than we want. In fact, what if we had an array containing hundreds of different values? Wouldn’t it take longer to count every element than a single one (well, no, but one might assume)? That would account for a bit more memory usage as well, even though it does give us the answer we are looking for. Which is a more elegant solution?

When attempting to optimize code, developers must often have to choose between CPU time and memory footprint. This is one of those situations due to language limitations. Let’s take a look at how much time it takes for PHP to count the values for us:

// count the values in the array greater than 50000
$count_data = array_count_values( $data );
$count      = $count_data[ 2 ];
 
// output memory consumption
printf( "Memory Consumption: %d bytes\n", memory_get_usage( true ) );

Memory Consumption: 6815744 bytes
Count: 20000
for loop time: 0.003805s

Even though we’re gathering more information, it’s still 3.5x faster than counting only a single value manually. In fact, I’m willing to bet that it even consumed less memory than our previous example, simply due to the fact that we aren’t having to deal with userland variables and function calls:

(Our example using check_value(), with memory output)

Memory Consumption: 7077888 bytes
Count: 20000
for loop time: 0.043128s

It seems that counting a single element using our check_value() method used 262144 more bytes than array_count_values()! Damn you, PHP! Yes, it’s unfortunate that considerations like this must be made, but you’d be surprised how many internal methods really do provide both strong memory and CPU benefits even though they “do more work”. Userland consumes considerable amounts of memory.

That isn’t to say that array_count_values() can’t consume more memory. Like I said previously – if it returns an array with thousands of values, it’s going to offset that memory savings.

ArrayObjects and Iterators

PHP 5 began a strong move toward Object-Oriented programming. It provides a number of powerful components that deal with iteration – specifically Traversable objects and Iterators. They provide a benefit to developers in the sense that it makes code much easier to understand and allows for powerful OO implementations, but does it provide any performance benefit?

Firstly, let’s see what happens when we loop through an ArrayObject rather than a normal array:

// generate an array containing 100,000 elements, each with the value of 1
$data = new ArrayObject( array_fill( 0, 100000, 1 ) );

Total: 100000
loop time: 0.061663s

This is compared to our foreach example with a result of 0.037076s. So we did loose a bit of time there. However, that does not mean you shouldn’t use ArrayObjects. In fact, I use them a great deal in OO scripts, because ArrayObjects coupled with Iterators provide powerful designs that aren’t quite as elegant when using procedural code. However, when performance is a requirement, I have to steer clear of them.

You will find that the Iterator object provides very little benefit over a conventional foreach loop as well. The only time I make use of an Iterator is when it contains additional logic that determines what results it should return (e.g. to iterate only over odd numbers), or if the implementation allows for an Iterator to be passed in at runtime to determine what results should be returned.

// loop through each of the array items and sum them up
$iterator = $data->getIterator();
while ( $iterator->valid() )
{
    $total += $iterator->current();
    $iterator->next();
}

Total: 100000
loop time: 0.291542s

In fact, that’s our slowest method yet. So, just be wise when weighing the costs and benefits. If performance doesn’t matter – go ahead. Use whatever implementation is best for you. That’s why you have the option. If performance is a factor, don’t even glance at those objects. Maybe future versions of PHP will enhance the functionality to the point where their speeds are equivalent to the loop structures.

Wall Of Text Summary

PHP provides a number of ways to iterate through arrays – rest assured that we haven’t nearly covered them all here. Out of all the loop structures available, foreach is the fastest for iteration. However, it comes up short when compared to built-in functions designed for array processing. Where possible, those built-in functions should be used, even where it may seem at first glance that it may be slower. They often provide a very strong performance benefit – both CPU and memory wise.

PHP 5 introduced a number of new means of iteration through the use of objects. Certain implementations may do well to improve code legibility, and may enable the developer to implement some fairly powerful object-oriented designs, but when performance is a factor, you want to steer clear of them.

Hopefully this article helped to clear up some questions. I expect not to see any more for loops with count() in the loop test expression!

Someone took the time to make their theme available under the GPL – that’s an excellent, noble thing. However, the encryption of footer data shows a complete lack of understanding of the philosophy behind Free Software and the GPL.

Freedom #1 states:

The freedom to study how the program works, and change it to make it do what you wish (freedom 1). Access to the source code is a precondition for this.

Here is an example footer from one of the themes I found (most of the content has been removed and what remains has been altered to (a) not throw anyone under a bus and (b) reduce the size of this post):

eval(gzinflate(base64_decode('gfE8f[...]T8=')));
 
/* WARNING: This file is protected by copyright law. To reverse engineer or decode this file is strictly prohibited. */
$o="ZnvJF[...]Mn=";eval(base64_decode("Xvxsbe0wO2V2YWwoYmFzZT[...]GxsfW9cnw=="));return;

Clearly, this is in violation of Freedom #1 for the following reasons:

1. The only way to gain access to the source code is to decrypt (reverse engineer) the above content. The fact that it is a very simple matter to do so is irrelevant – the source code should be clearly visible.
2. There is a comment in the code that clearly states that any attempt to reverse engineer the code is illegal. This is in direct conflict with the GPL – in fact, it’s completely incompatible. So that statement is actually negated by the license. You’re more than welcome to do so under terms of the GPL.
3. Since you do not have access to the original source code without decrypting the data, you are not free to modify it. Many try to decrypt the data and are unable to do so because they do are familiar with the PHP language or are otherwise unable to figure out how to decrypt it. If you do decrypt the above, you will find that the majority of it is HTML, with one call to wp_footer(). This means that, were it not encrypted, users would need only know HTML to safely modify it.

But those aren’t the only issues. There’s also some other considerations:

1. The encrypted footer is 7743 bytes. Decrypted, it is roughly 996 bytes. That is a huge size difference – a lot of wasted bytes to secure the footer.
2. There are a great number of function calls in order to decrypt the footer. You can see some of them above, but the eval()’d code contains additional methods to decrypt further encrypted and obfuscated code. That’s unnecessary overhead. Not to mention that eval() in itself is very slow. So, not only did the author prevent you from modifying the footer, you’re taking a performance hit because of it!

I can understand why the author would want to encrypt the footer. They probably felt that, if they’re releasing their source code free of charge to the public, they should at least be allowed to keep their copyright notice in tact. And yes, they should be able to. In fact, the copyright notice should never be removed on GPL’d code – it’s a legal requirement to keep it there. It is still the author’s work, and it is a copyrighted work, so the notice must remain in tact. But yes, people do remove it anyway, even though they shouldn’t.

But when an author takes the time to not only prevent me from modifying the source, but in doing so causes me to incur a performance hit, and on top of it licenses the theme under the GPL – I see some very large ethical issues. Especially considering that all I wanted to do was add my own link, and its placement required being able to modify the encrypted HTML.

I propose that all those who support free software boycott such themes and send a notice to the authors when they come by such themes (or send them a link to this post). Furthermore, do not let developers who try to restrict your freedom push you around. I do not propose you remove the copyright notice – in fact, I would never, ever suggest that someone do that; but I do propose that, if the theme is licensed under the GPL, you decrypt the footer regardless of any notices that state you otherwise cannot. The license clearly states you have access to the source code, and if you are not given such access, the code either needs to be relicensed, or the developer needs to comply with its terms.

As there are also many themes out there that encrypt their data and are not licensed under the GPL or other free software licenses, I’m not going to provide any methods of decrypting the data. I do not support proprietary software, but I also do not encourage license violations.

Microsoft Seeks Patent On Shaming Fat Gamers

The idea is that Microsoft wants to create a system whereby it will gather physical statistics on a user and alter the game play around those characteristics. For example, as the article mentioned, an obese player may be cut off from playing. I suppose the idea could have benefits – such as helping players get into shape if they can’t wait to get back on their consoles. But this raises some serious ethical issues. What, for example, about those players who are unable to lose weight for genetic reasons? Or what ever happened to those who play video games to escape from the real world? Hell, that’s half the fun in my opinion.

Microsoft also proposes shaping gaming experiences by using ‘psychological and demographic information such as education level, geographic location, age, sex, intelligence quotient, socioeconomic class, occupation, marital/relationship status, religious belief, political affiliation, etc.’

This to me raises some serious privacy issues. I would never wish a video game or vendor to have that much personal information on myself. In fact, I feel the name on a credit card is too much information for the vendor or any party to have. But the traits listed above? Look, placing yourself in a video game is a pretty cool idea if you consent to it. For those who do want to give up all of that personal information, good for them. But basing game play off of factors like physical appearance, sex, religion, politically affiliation, etc could turn into a discriminatory battle.

Furthermore, you know that corporations will use this information for other statistical purposes, or to sell to other companies. Would you want a company like Microsoft to know more about you than you probably do? The only way I would subscribe to such a system is if it (a) used an open protocol, (b) was subject to very strict privacy standards and policies that are heavily enforced and investigated, (c) we were able to strictly state what information we wanted to give to the vendor, (d) we were able to choose what remote server to upload such statistical data to (i.e. one we may be able to trust) and (e) if the video game provided this is an option rather than a requirement. The patent does use the word “nonvolitionally” when referencing information gathered from the user, but we’ll see how that goes. Vendors can still state, “oh, uh, you can only play this game if you give us this, this, this and this.”

…surprise, the girl you’ve been lusting over in your video game is actually a guy, and this will prove it.

The patent states:

[0009]In yet a further aspect, an apparatus is provided for interacting with a virtual gaming environment. An information source is accessed for receiving a health information nonvolitionally obtained from a user. These sources are at least two of a a physiological sensor, a networked healthcare information repository, and a healthcare smart card.

I would be highly against anyone having access to information on that type of level – as everyone else should be. This is just one example of how vendors like Microsoft may try to steal information from you. If they were able to have access to heathcare records, you wouldn’t be able to say “I only want you to see this”. No, they’d probably be able to see it all.

Again, I’m not saying it’s a horrible idea. I’ve often fantasized about placing yourself in a virtual reality, where the game actually rendered you in the virtual environment. But considering Microsoft’s track record, they are not the company to implement or control this.

I haven’t the time to figure out what Grub is trying to do in that amount of time, but I did notice a solution. I have three HDDs. In the BIOS, it was set as the third boot device. By moving it to the top, Grub quickly moved along to boot Ubuntu.