New Code Posted!

The new BuyLocal code has been posted to the Google Code site (http://code.google.com/p/buylocalopensourcerensselaer/) and has been promoted to the live site at RPI (http://www.buylocal.rpi.edu).  The new codebase is a combination of work performed by Kevin Fodness over the summer, and by a team of undergraduates (Alexander Adami, Spencer Hakim, Jeremy Walker) during the fall semester.  The codebase has been completely transitioned to an object-oriented approach, the layout is now using some elements of CSS3 to improve download and rendering speeds, and the whole site is lighter and faster than before.  Of course, all of this is possible thanks to the original codebase written by Aaron Ryden and Mike Pennisi, from which the current codebase was created.

If you have any feedback, we'd love to hear it!

New BuyLocal Developers

With the start of a new semester comes a change in BuyLocal developers. I am passing off control of development to a group in Jeff Miner's Managing IT Resources (MITR) class at RPI, consisting of Alex Adami, Jeremy Walker, and Spencer Hakim. They will be taking over development in the coming week, and will direct the project through its next phase.

Welcome aboard, guys!

Ongoing salt discussion

John, one of the guys working on the AllForLocal project, shot me an email recently regarding my decision to drop the salt out of the password hashing process. Here is his email:

Hey,

I'm e-mailing you in response to your blogger post about password salts in the Buy Local project. I tried commenting on the post, but for one reason or another it didn't seem to work (unless it's waiting on you to okay the comment before it shows up). Anyway, since it was something security related I figured it was important enough to contact you about. Here is the post:


Hey there! I just started following this blog (I'm a developer of allforlocal.com and find it interesting to see what you're up to) and I think there might be an oversight here:

It's true that to keep the salt and password hash separate would be more secure, but that isn't the point of the salt (as far as I'm aware). The real value of the salt comes in when you have an attacker who has somehow obtained the password hashes and is trying to work back to the passwords. Once they've gotten to that point they have two choices: Either brute force the hashes (very time- and computation-expensive) or perform a lookup in a hash directory/rainbow table (http://en.wikipedia.org/wiki/Rainbow_table) which can be nearly instant. It's the second attack that salts protect against, even if the attacker has the salt as well.

The reason for this is because hash directories/rainbow tables store passwords and their resulting hashes to do quick lookups, but even a small change to the password (courtesy of salts) totally changes the resulting hash, making these sorts of lookups useless. Plus there's a bit more entropy since the salts are random. The end product is that attackers _must_ use brute force and even that route is made a bit more difficult for them.

So in short: I encourage you to keep the salts! You'll be sweating a lot less if you ever do run into security troubles down the road.


-John

My reply:

John --

I worked with securing web and database applications and servers for the US Department of Labor for three years, and out of that came the following rules I use when evaluating when and how to secure applications:

1) Confidentiality - Upholding privacy of data and preventing unauthorized access;
2) Integrity - Guarding against unauthorized or malformed inserts/updates/deletes;
3) Availability - Preventing crashes and slow performance when accessing data.

The degree to which each of these categories are important depends on the sensitivity level of the application in question. An application that runs the back-end for a credit card company, say, would require higher levels of CIA than a community website such as BuyLocal.

That said, there should be a certain base layer of security in any application. The question is then one of degree, and how much of a benefit is achieved by whatever measure is taken.

When passwords are entered into the authentication system, they are passed directly to the server without being pre-hashed. In order to pre-hash with a salt, the salt value would need to be passed to the authentication system before it could hash the password, and that isn't being done. In most implementations, the login will not be protected with SSL/TLS, which means passwords transmitted in the clear over the Internet, which is a larger threat than obtaining password hashes.

Since hashes are not distributed, and are only used at authentication time, there are very limited paths to the hash data. The hash data could be obtained through a database vulnerability, or a web vulnerability allowing access to database data, such as SQL injection. A backup copy of the database could be intercepted if it were on, say, my laptop, and my laptop were stolen. The problem here is that if the database is vulnerable, the salt column in the users table is just as vulnerable as the hashed password column, and that extra step wouldn't constitute a serious hurdle for any malicious attacker. Same thing with a database backup - the salt values would be right there with the passwords. Security like this is akin to changing the port that your database server runs on or obfuscating the engine for your server-side scripts - a few seconds of scanning with any of many readily available tools gets past these 'protections,' and implementation of the protections proves to be more hassle than it is worth.

Let me give you another 'for instance' - say someone got the password hash and looked it up in a rainbow table. Say the user's password was 'abc123' and the salt is 'f0293fec'. Based on the current implementation of salt ($hashedPass = makeHash($salt . $password)), the rainbow table would return:

f0293fecabc123

If more than one hashed value was obtained, the attacker could quickly figure out that all of the salt values are 8 character hexidecimal predicates to the actual password. A simple perl script that trims off the first 8 characters is enough to turn the rainbow table results into a password list.

I'm a believer in the mantra 'do it right or don't do it at all.' If salts will be re-implemented, they will be implemented after SSL is mandated for the login, and they will be (at a minimum) stored in a separate table, accessible only by using separate credentials (to protect against SQL injection), inserted at a random position, of random length, and preferably, of random quantity - and they will certainly push the password length beyond 64 characters in all circumstances to make rainbow table lookups virtually impossible. However, I don't believe that level of security is required for an application of this type, and I'd rather spend my time working on security measures to protect against more likely vectors of attack - such as SSL/TLS for protecting the login, protecting against SQL injection, protecting against session hijacking, XSS, etc.

Switching to HTML 5

To switch to HTML 5, all you have to do is change your doctype to the following:

HTML 5 is fully backwards compatible with all browsers, so it has all of the advantages of HTML 4.01. Plus, it lets you write strict XML compatible markup without the parser throwing errors - such as being able to close tags with a trailing slash (e.g., ). HTML 5 is basically a convergence of HTML 4 and XHTML 1.1 with a whole load of new elements thrown in, which look to be very promising.

mysql & Foreign Keys

If part of your create database code looks like this:

CREATE TABLE IF NOT EXISTS `users` (
  `id` INT(5) NOT NULL AUTO_INCREMENT,
  `password` VARCHAR(128) NOT NULL,
  `salt` VARCHAR(50),
  `email` VARCHAR(320) NOT NULL,
  `zip` VARCHAR(10) NOT NULL,
  `role` INT(1) NOT NULL,
  `isactive` BOOL DEFAULT 1,
  `registeredon` DATETIME NOT NULL,
  `lastlogon` DATETIME NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=INNODB;

CREATE TABLE IF NOT EXISTS `userphones` (
  `userid` INT(5) NOT NULL,
  `phone` VARCHAR(15) NOT NULL,
  `description` VARCHAR(25) NOT NULL,
  KEY `fk_userphones_userid` (`userid`),
  CONSTRAINT `fk_userphones_userid`
  FOREIGN KEY (`userid`)
  REFERENCES `users` (`id`)
) ENGINE=INNODB;

And you try to load those statements in via a $mysqli->multi_query() command, mysql will fail with error 150, but it won't fail if you either add them sequentially using a foreach() and $mysqli->real_query or if you run them through phpmyadmin or similar.

Why?

Error 150 denotes a foreign key problem, which is triggered because the field names for the foreign key don't match up. The only way to get this error to go away on a multi_query is to make the `id` field in the `users` table `userid` instead, so the field names between tables match (which is not a requirement of a foreign key). Joy.

Escaping Data Structures

As we all know, to protect against SQL injection and snafus caused by single quotes in your data, always escape your inputs before running queries. However, what happens when your input is a complex data structure, such as an object, an array of objects, or an array of objects that in turn contain arrays of objects? Running foreach statements and escaping each individual value can result in very long functions, which accomplish basically the same thing (escape all values) but with different named variables and data types. Running mysqli::real_escape_string on the occasional single-dimension function parameter is fine, but for complicated data structures, you should farm out the work to a function.

I have written just such a function as part of my Database class, and it works like a charm:

   /**
    * Function to run real_escape_string recursively on an object or array.
    *
    * @param mixed &$source The object or array to escape.
    *
    * @access private
    * @return null
    */
   private function _escape(&$source)
   {
       foreach ($source as &$var) {
           if (is_array($var) || is_object($var)) {
               $this->_escape($var);
           } else {
               $var = $this->_mysqli->real_escape_string($var);
           }
       }
   }

To salt or not to salt?

I'm writing this as a clarification to my last post, where I defend dropping the `salt` column from the database.

Salt values are useful in hashes in two circumstances:

1) When there is a means by which a pre-hashed password can be passed directly to the system for verification (which is a flaw in many Microsoft products);
2) When a list of hashed passwords is obtained by a malicious user to be brute-force cracked offline.

Because of the way the application is designed, #1 is not a concern. SQL injection is protected against and at no point are passwords handed to the database without being hashed (which would mean a hash of the hash).

#2 is a potential concern, if a security hole were found that permitted getting a dump of the database or the `users` table. However, in this case, having a `salt` column in the users table would be like handing over the second launch key. Even embedding the `salt` column in a separate table in the same database would not present much of a hurdle for a malicious attacker.

The best way to handle `salt` values is to store them in a table in a separate database with a different username and password, so multiple sets of information would be needed to gain access to both the hashed passwords and the salt values.

However, this application doesn't need to be über secure, so instead of creating a separate database, I'm simplifying the codebase and database design by dropping the `salt` column altogether.

For more information: http://en.wikipedia.org/wiki/Salt_(cryptography)