Walled Gardens need Improvement

I came face to face with “The Man” recently, and lost hopelessly.

You see, I spent two weekends working on a relatively basic Chrome Extension that I find useful in my day job; I often have to test a Web App Firewall, and usually you need to simulate traffic from multiple different source clients. The easiest way to do this for HTTP is to use the X-Forwarded-For (XFF) header, and configure your WAF to trust it. So, I thought, If i made an extension to automatically switch the XFF header, it would take some tedium out of this process.

I spent a bit of time on the Eloquent Javascript and the Chrome Developer Documentation, and over two weekends, came up with a slightly configurable and reasonably usable extension that allows you to inject XFF header into requests to a target server you configure. I called it “x-cycle“.

Feeling proud of my accomplishment, I push this to the Chrome Store  as a Developer Tool (paid a $5 verification fee in the process).

I spend the next 12 hours eagerly searching for my extension in the Chrome Web Store, hoping that my first ever extension would show up. After frustration with not finding it, I decide to go to my Chrome Developer Dashboard and check. “At least one of your items has been removed from the store because it did not comply with our policies or terms of service.” Huh? No email, no notification, and no specification of which terms I have violated, and very little I can do about it. 

Nobody will get to use my extension, unless I distribute it manually, and Chrome already makes it near impossible to distribute an extension outside of the Chrome Store, so, I won’t get much traction there, regardless of how much I think people with my kind of job would find this tool useful.

I understand more vividly why walled gardens can be a problem. Previously, I have always preferred Android to iOS because I felt like Google didn’t do too much to dictate what kind of software I can use on my phone compared to Apple, but looking at Chrome, it seems like I never considered the fact that Chrome has become a walled Garden, and one that badly needs to improve its engagement at that.

If you take down someone’s work, you need to at least explain why, so that they can work on resolving it. I get the distinct feeling that some Reviewer just saw “X-Forwarded-For”, googled it, and saw it as a way to “mask” your IP address, and thought “Oh No! Hacking Tool!!” and took it down. This would be a real shame.

Any experienced web admin knows that you don’t trust X-Forwarded-For, unless it was set by a device that you control, thus there is limited scope for abuse of this extension, even for people who are intent on mischief. The policy for trusting XFF is that you strip whatever you received, and then inject the value that you are going to trust. Someone who tests a WAF is both in control of the client and the web-server, and can configure it to trust this header, and this is what makes this extension useful in my view.

I have submitted a support ticket to some Google Support service that took some Googling to find hoping I would get back a more actionable response whenever they do get round to it.

Until then, twiddling thumbs and wondering whether the “open” web we so proudly proclaim is not really just a walled garden with tremendous power in the hands of the people who make web browsers. It is probably little wonder then that practically all browsers are free of charge; they trade the cost for the users, which in turn amplify the power of the browser vendor. They can then hold advertisers, publishers and developers to ransom. Hmmm.

Looking for an IPv6 Tunnel?

Don’t bother trying Sixxs if you don’t intend to use your work address. The haughty folks over there might put you off IPv6 for good.

Head straight to Tunnel Broker where you can have a tunnel authorized in 5 minutes or less, with no need to read through a laundry list of Email providers you are not allowed to use.

Looking forward to beating my router into shape to provide connectivity to my LAN.

[in brief] compiling libgcrypt 1.6.3

While building this library, i ran into errors finding libgpg-error 1.11.

There is a small bug in libgcrypt’s configure script for finding this library. It tells you to set –with-libgpg-error-prefix, but it actually ultimately looks for the location of this library from the variable –with-gpg-error-prefix. You should set –with-gpg-error-prefix instead of –with-libgpg-error-prefix, if your libgpg-error is installed to a non-standard location.


Installing pycrypto on Windows 8

While trying to install pycrypto on Windows 8, because it was a requirement for “paramiko” which I needed for some SSH stuff, I kept running into failure after another. To summarise, you need to have Visual Studio 2008 installed (there is an express version available free, and there is also likely a microsoft SDK package that includes the Visual C++ 9.0 compiler). After sorting all of these out however, I still had one persistent type of problem:

warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath.
 Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File "c:\users\fanen\appdata\local\temp\pip_build_Fanen\pycrypto\setup.py", line 456, in <module>
 File "c:\Python27\lib\distutils\core.py", line 152, in setup
 File "c:\Python27\lib\distutils\dist.py", line 953, in run_commands
 File "c:\Python27\lib\distutils\dist.py", line 972, in run_command
 File "c:\Python27\lib\site-packages\setuptools\command\install.py", line 53, in run
 return _install.run(self)
 File "c:\Python27\lib\distutils\command\install.py", line 563, in run
 File "c:\Python27\lib\distutils\cmd.py", line 326, in run_command
 File "c:\Python27\lib\distutils\dist.py", line 972, in run_command
 File "c:\Python27\lib\distutils\command\build.py", line 127, in run
 File "c:\Python27\lib\distutils\cmd.py", line 326, in run_command
 File "c:\Python27\lib\distutils\dist.py", line 972, in run_command
 File "c:\users\fanen\appdata\local\temp\pip_build_Fanen\pycrypto\setup.py", line 253, in run
 File "c:\Python27\lib\distutils\command\build_ext.py", line 337, in run
 File "c:\users\fanen\appdata\local\temp\pip_build_Fanen\pycrypto\setup.py", line 150, in build_extensions
 File "c:\Python27\lib\distutils\command\build_ext.py", line 446, in build_extensions
 File "c:\Python27\lib\distutils\command\build_ext.py", line 496, in build_extension
 File "c:\Python27\lib\distutils\msvc9compiler.py", line 475, in compile
 File "c:\Python27\lib\distutils\msvc9compiler.py", line 385, in initialize
 vc_env = query_vcvarsall(VERSION, plat_spec)
 File "c:\Python27\lib\distutils\msvc9compiler.py", line 301, in query_vcvarsall
 raise ValueError(str(list(result.keys())))
 ValueError: [u'path']

It appears that the function “query_vcvarsall” fails if it doesn’t find a set of 4 environment variables (this is some dodgy coding to me :p):

  • path
  • include
  • lib
  • libpath

You may need to create these environment variables, and place in them the appropriate paths for your PC.

On my 64-bit windows 8 machine with 64-bit Python 2.7.6, I have the following:

LIB = C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\ATLMFC\LIB;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\LIB;%WindowsSdkDir%lib;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1A\Lib;C:\Python27\Lib;%LIB%
INCLUDE = C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\ATLMFC\INCLUDE;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE;%WindowsSdkDir%include;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1A\Include;C:\Python27\include;%INCLUDE%
PATH = C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN;C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\Tools;C:\WINDOWS\Microsoft.NET\Framework\v3.5;C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\VCPackages;%WindowsSdkDir%bin;C:\Python27\Scripts;C:\Python27;%PATH%
LIBPATH = C:\WINDOWS\Microsoft.NET\Framework\v3.5;C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\ATLMFC\LIB;C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\LIB;C:\Python27\Lib;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1A\Lib;%LIBPATH%

After that, I am able to use pip to install pycrypto.

Note that my variables may contain paths which are redundant, but I arrived there after a windy path of experimentation. After It worked, I said to myself: “if it ain’t broke, don’t fix it.”

The following site was very helpful in showing me alternative ways. I like what the guy did to the python utility scripts. Just hack them to do what I want. None of the cruft of general-purpose code. Just tell the bloody system I have Visual Studio 9.0 😉 http://blog.victorjabur.com/2011/06/05/compiling-python-2-7-modules-on-windows-32-and-64-using-msvc-2008-express/



Windows 8 SSD 100% disk usage

Upgraded my Windows 8.1 PC by installing a 240GB Crucial SSD, and the performance became worse than with a spinning disk.

In my instance, I was able to improve the experience simply by installing Intel Rapid Storage tools drivers (downloaded from MSI’s site for my specific motherboard). I accepted all defaults. I was using an MSI motherboard. This just might be your solution.

Openssl Hint for today

You find yourself trying to look at a certificate with Openssl (I’m using the cygwin version of openssl on windows 7):

$ openssl.exe x509 -in certificate.crt -text
unable to load certificate
2675716:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE

Openssl error messages are so cryptic, I tried looking at the code to deduce what it meant, and even that didn’t go so well.

The error message shown above doesn’t really have much to do with “trust”. It most likely means that your certificate is not in PEM format. The most common other format that certificates can be in is DER. If you wanted to view the certificate in DER format anyway, you would do this:

$ openssl.exe x509 -inform DER -in certificate.crt -text

the -inform argument allows you to specify what format the certificate you are trying to examine is in. The -text argument says “display it on screen”, and the -in argument specifies the certificate file name.

If you wanted to convert that certificate from DER to PEM, you would say:

$ openssl.exe x509 -inform DER -outform PEM -out ./PEMcert.crt -in DERcert.crt

-outform PEM says the output format should be PEM, and the -out argument specifies the filename to give the converted certificate.



Jquery .load() makes the target blank

If you run into the problem where calling .load(url) on an element makes it turn blank, you very likely have an unexpected space (” “) in your url.

You should re-examine it, it may be a trailing or leading space, which will make it difficult to see with the naked eye. Try calling the javscript .trim() on the url string.

At the time of this writing, I was using version 1.10.2.