atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title><![CDATA[DigiACTive]]></title>
  <link href="http://DigiACTive.github.io/atom.xml" rel="self"/>
  <link href="http://DigiACTive.github.io/"/>
  <updated>2014-01-19T23:09:42+11:00</updated>
  <id>http://DigiACTive.github.io/</id>
  <author>
    <name><![CDATA[DigiACTive Team]]></name>
    
  </author>
  <generator uri="http://octopress.org/">Octopress</generator>

  
  <entry>
    <title type="html"><![CDATA[Vagrant, NFS Synced Folders, Permissions, and Bindfs on CentOS 6]]></title>
    <link href="http://DigiACTive.github.io/blog/2014/01/18/fixing-permissions-on-vagrant-nfs-synced-folders/"/>
    <updated>2014-01-18T23:59:00+11:00</updated>
    <id>http://DigiACTive.github.io/blog/2014/01/18/fixing-permissions-on-vagrant-nfs-synced-folders</id>
    <content type="html"><![CDATA[<p><em>WARNING: highly technical post ahead!</em></p>

<h1>A tale of two synced folder backend implementations</h1>

<p>Here at DigiACTive, we&rsquo;ve made much use of <a href="http://www.vagrantup.com/">Vagrant</a> to help us manage and deploy a consistent development environment across all our development machines. For the uninitiated, Vagrant essentially allows developers to create a standard configuration &mdash; operating system, software packages, configuration files, and so on &mdash; which can be automagically deployed (usually using a single command!) into a variety of environments, such as a <a href="https://www.virtualbox.org/">VirtualBox</a> virtual machine. Our development team uses both Linux and Mac OS X, and our software has a large number of dependencies, so having a standard development environment has proven very useful.</p>

<p>Our Vagrant configuration uses Vagrant&rsquo;s <a href="http://docs.vagrantup.com/v2/synced-folders/">synced folders</a> to share the developer&rsquo;s working copy of our Git repository with the test server. Depending on the provider onto which the Vagrant setup is deployed, there are a number of different backend implementations used for synced folders. By default, VirtualBox deployments will use <a href="https://www.virtualbox.org/manual/ch04.html#sharedfolders">VirtualBox shared folders</a> &mdash; which, apart from being notoriously unreliable, have limited support for some important POSIX file system features, such as hard links.</p>

<p>To get around this, one can enable the alternative <a href="http://docs.vagrantup.com/v2/synced-folders/nfs.html">NFS backend</a>, which doesn&rsquo;t have these limitations &mdash; and can conveniently be enabled with a single line in the Vagrantfile. It all sounds great, right?</p>

<p><em>Not quite.</em></p>

<p>Daniel fixed up the last details of our Vagrantfile, gave it a final test, and pushed it up. Ben pulled it down on to his MacBook, ran <code>vagrant up</code>, and after a couple of hours of watching our <a href="http://www.getchef.com/chef/">Chef</a> scripts download many, many dependencies, it all worked! (Well, mostly. Anyway.)</p>

<p>Meanwhile, on my Debian box, I (Andrew) was getting very, very close to tearing my hair out. After fixing up a number of other problems to get Vagrant running properly, I tried to provision the VM &mdash; and every time I tried, it complained that the provisioning scripts didn&rsquo;t have appropriate permissions when working with our shared folders.</p>

<h1>NFS and permissions</h1>

<p>After some investigation, it became apparent what the issue was.</p>

<p>Our Chef scripts were attempting to change the owner of the configuration files they were modifying to <code>vagrant</code>, the default user set up inside the VM. However, as the synced folders are mounted using NFS, changing the ownership of a file on the remote client (the VM) means changing the ownership on the server (the host system). On the host system, <a href="http://en.wikipedia.org/wiki/Unix_security#Root_squash">root squashing</a> means that the client didn&rsquo;t have the root privileges necessary to do that.</p>

<p>However, it soon became apparent that there was another issue here:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='sh'><span class='line'><span class="o">[</span>vagrant@localhost vagrant<span class="o">]</span><span class="nv">$ </span>ls -la
</span><span class='line'>total 84
</span><span class='line'>...
</span><span class='line'>drwxr-xr-x   2 1000 1000 4096 Dec  7 05:46 scope
</span><span class='line'>drwxr-xr-x   3 1000 1000 4096 Jan 16 12:13 src
</span><span class='line'>drwxr-xr-x   4 1000 1000 4096 Nov 19 00:51 vagrant
</span></code></pre></td></tr></table></div></figure>


<p>It turns out that NFS shares UIDs between the server and clients &mdash; which is fine if all systems use the same authentication backend. This isn&rsquo;t the case in a Vagrant system, obviously. NFS does not provide a built-in way to map between different server/client users &mdash; I believe this can be accomplished with a proper directory service, but we weren&rsquo;t going to set that up just for a simple development VM.</p>

<p>This also explained why Daniel and Ben had no problems provisioning on their Macs. On my Debian host system, standard UIDs start at 1000, while in the CentOS guest system, UIDs start at 501. Coincidentally, Mac OS X also starts UIDs at 501 &mdash; so on their systems, the files were already owned by <code>vagrant</code>.</p>

<p>I searched around for a few too many hours trying to find a solution, playing around with random NFS mounting options in an effort to make it work&hellip;</p>

<h1>bindfs and vagrant-bindfs</h1>

<p>Enter <a href="http://bindfs.org/">bindfs</a>.</p>

<p><a href="http://docs.1h.com/Bind_mounts">Bind mounts</a> have existed for a while in various *nixes, allowing users to mount already-mounted filesystems to other locations.</p>

<p>bindfs takes this concept further, though. Using the wonderful powers of <a href="http://fuse.sourceforge.net/">FUSE</a>, bindfs allows you to virtually alter ownership and permission bits &mdash; which is exactly what I needed! As a FUSE filesystem, bindfs has some <a href="http://www.redbottledesign.com/blog/mirroring-files-different-places-links-bind-mounts-and-bindfs">unfortunate performance issues</a>, but hey, at least it&rsquo;d work!</p>

<p>Even better, there&rsquo;s a Vagrant plugin, <a href="https://github.com/gael-ian/vagrant-bindfs">vagrant-bindfs</a>, that allows easy configuration of bindfs mounts straight from the Vagrantfile!</p>

<p>I thought I&rsquo;d found my solution, and proceeded to try and grab the bindfs RPM for CentOS 6&hellip;</p>

<p>&hellip;which didn&rsquo;t exist. It used to be built as part of the <a href="https://fedoraproject.org/wiki/EPEL">EPEL</a> repository, but alas, no more!</p>

<h1>bindfs, vagrant-bindfs, and RPMs</h1>

<p>After much searching around for information about building RPMs, I ended up downloading the RPM spec file from the old EPEL package, updating it to bindfs 1.12.3, and compiling it myself. After learning more than I ever wanted to know about RPM building, I managed to compile a CentOS 6-compatible bindfs RPM.</p>

<p>However, there was still the small issue of installing this RPM during the Vagrant configuration process. For vagrant-bindfs to work, bindfs needs to be installed in the boot-up configuration phase, not the provisioning phase, which meant that adding it to our Chef scripts wasn&rsquo;t going to work.</p>

<p>Conveniently, vagrant-bindfs makes use of Vagrant&rsquo;s <a href="http://docs.vagrantup.com/v2/plugins/guest-capabilities.html">guest capabilities</a> framework to detect when the guest doesn&rsquo;t have bindfs installed and trigger an installation capability to download the appropriate package. The installation capability needs to be implemented separately for each type of guest, and at present only Debian is supported.</p>

<p>I implemented a quick-and-dirty hack to extend vagrant-bindfs&rsquo;s installation capability for CentOS. Because bindfs has a number of dependencies which mean we can&rsquo;t just use <code>rpm -i</code> to install it automatically, I decided to create a YUM repository (which is actually really easy!), which makes the rest really easy.</p>

<p>So, after many, many hours of trying, I finally got bindfs working, and finally managed to provision my testing VM!</p>

<p>If you&rsquo;re trying to get bindfs working with Vagrant and CentOS 6, and don&rsquo;t want to go through the same pain I went through, I give you&hellip;</p>

<h1>Downloads and Links!</h1>

<p><strong>DigiACTive provides these downloads as-is, and takes no responsibility for any issues! Use at your own risk!</strong></p>

<ul>
<li><a href="http://digiactive.com.au/digiactive-repo/centos/6/x86_64/bindfs-1.12.3-1.el6.x86_64.rpm">bindfs-1.12.3-1.el6.x86_64.rpm</a></li>
<li><a href="http://digiactive.com.au/digiactive-repo/centos/6/x86_64/bindfs-debuginfo-1.12.3-1.el6.x86_64.rpm">bindfs-debuginfo-1.12.3-1.el6.x86_64.rpm</a></li>
<li><a href="http://digiactive.com.au/digiactive-repo/centos/6/SRPMS/bindfs-1.12.3-1.el6.src.rpm">bindfs-1.12.3-1.el6.src.rpm</a></li>
<li><a href="http://digiactive.com.au/digiactive-repo/gems/vagrant-bindfs-0.2.4.digiactive2.gem">vagrant-bindfs-0.2.4.digiactive2.gem</a> (install with <code>vagrant plugin install</code>)</li>
<li><a href="https://github.com/DigiACTive/vagrant-bindfs">our version of vagrant-bindfs on GitHub</a></li>
<li><a href="http://digiactive.com.au/digiactive-repo/DigiACTive.CentOS.repo">DigiACTive YUM Repository Configuration File</a> (put in <code>/etc/yum.repos.d</code>)</li>
<li><a href="http://digiactive.com.au/digiactive-repo/RPM-GPG-KEY.asc">DigiACTive PGP Signing Key</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Setting Up Python 2.7 on CentOS 6.4 the (Really) Easy Way]]></title>
    <link href="http://DigiACTive.github.io/blog/2013/12/28/setting-up-python-2-dot-7-on-centos-6-dot-4-the-really-easy-way/"/>
    <updated>2013-12-28T19:23:30+11:00</updated>
    <id>http://DigiACTive.github.io/blog/2013/12/28/setting-up-python-2-dot-7-on-centos-6-dot-4-the-really-easy-way</id>
    <content type="html"><![CDATA[<p>Setting up Python 2.7 on CentOS 6.4 is not particularly easy: CentOS ships with 2.6 and yum seems to depend on it.</p>

<p>The traditional way to do this is to build Python from source, and there is even a <a href="https://github.com/shimizukawa/chef-python-build">Chef cookbook</a> to do this.</p>

<p>However, this is slow, and when you want to verify that your box builds correctly so that your team can start work on it, you want to minimise these sorts of delays. So I (Daniel) have been hunting around for the best way to install pre-built RPMs.</p>

<p>The closest I found was this excellent article from Red Hat: <a href="http://developerblog.redhat.com/2013/02/14/setting-up-django-and-python-2-7-on-red-hat-enterprise-6-the-easy-way/">Setting up Django and Python 2.7 on Red Hat Enterprise 6 the easy way</a>.</p>

<p>To summarise the article, RHEL (and CentOS) support &lsquo;software containers&rsquo;. They&rsquo;re sort of like virtualenvs or RVMs for packages &ndash; isolated, but more lightweight than full chroots. Python 2.7 is a supported container. Not all the steps are the same for CentOS as they are for RHEL, so here&rsquo;s an update of the article for CentOS:</p>

<ul>
<li>CentOS makes it even easier than RHEL to set up the SCLs. Instead of writing random files to your filesystem, you just need to <code>yum install centos-release-SCL</code></li>
<li>You&rsquo;re now ready to do a <code>yum install python27</code></li>
<li>As with RHEL, you can now pop a shell with Python 2.7 by simply doing <code>scl enable python27 bash</code>

<ul>
<li>If you want to run commands with more than one argument, you need to enclose them in quotes, otherwise <code>scl</code> will interpret them as other environments to enable.</li>
</ul>
</li>
</ul>


<p><strong>As long as you are happy running <code>scl enable python27 bash</code> before work, you&rsquo;re now set.</strong></p>

<h1>The gotchas</h1>

<p>While this is much, much faster than building from source, there are two, interrelated gotchas to bear in mind.</p>

<h2>Virtualenvs no longer shield you from interpreter differences</h2>

<p>I naively expected that virtualenvs would &lsquo;just work&rsquo; and would prevent me from having to think about the extra layer of &lsquo;magic&rsquo; I had just added.</p>

<p>I was wrong.</p>

<p>If you create a virtualenv inside scl, and then exit scl and try to run Python, it fails:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class='sh'><span class='line'><span class="o">[</span>vagrant@localhost tmp<span class="o">]</span><span class="nv">$ </span>scl <span class="nb">enable </span>python27 bash <span class="c"># enable the Python 2.7 environment</span>
</span><span class='line'><span class="o">[</span>vagrant@localhost tmp<span class="o">]</span><span class="nv">$ </span>virtualenv foo
</span><span class='line'><span class="o">[</span>vagrant@localhost tmp<span class="o">]</span><span class="nv">$ </span><span class="nb">exit</span> <span class="c"># leave the 2.7 environment, back to 2.6</span>
</span><span class='line'><span class="o">[</span>vagrant@localhost tmp<span class="o">]</span><span class="nv">$ </span><span class="nb">source </span>foo/bin/activate
</span><span class='line'><span class="o">(</span>foo<span class="o">)[</span>vagrant@localhost tmp<span class="o">]</span><span class="nv">$ </span>python
</span><span class='line'>python: error <span class="k">while </span>loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory
</span></code></pre></td></tr></table></div></figure>


<p>You get the same error when you try to create a virtualenv with the newly installed binary outside of scl.</p>

<p>The correct solution is to just wrap everything in scl: making <code>scl enable python27 bash</code> part of your workflow, just like <code>source env/bin/activate</code>.</p>

<h2>Not everything is easy to wrap in scl enable</h2>

<p>For times when it is impossible or very difficult to use scl enable as designed (for example when you&rsquo;re using someone else&rsquo;s chef cookbook), you can work around it. On my 64 bit system, the library it&rsquo;s looking for is in <code>/opt/rh/python27/root/usr/lib64</code>, so you just need to set an environment variable for that:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='sh'><span class='line'>env <span class="nv">LD_LIBRARY_PATH</span><span class="o">=</span>/opt/rh/python27/root/usr/lib64 /opt/rh/python27/root/usr/bin/python
</span></code></pre></td></tr></table></div></figure>


<p>If you&rsquo;re writing a Chef cookbook, you can give an <code>execute</code> block (or similar) an environment parameter:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='ruby'><span class='line'><span class="n">execute</span> <span class="s2">&quot;foo&quot;</span> <span class="k">do</span>
</span><span class='line'>  <span class="o">.</span><span class="n">.</span><span class="o">.</span>
</span><span class='line'>  <span class="n">environment</span> <span class="s2">&quot;LD_LIBRARY_PATH&quot;</span> <span class="o">=&gt;</span> <span class="s2">&quot;/opt/rh/python27/root/usr/lib64/&quot;</span>
</span><span class='line'><span class="k">end</span>
</span></code></pre></td></tr></table></div></figure>


<h1>Conclusion</h1>

<p>Using a packaged Python is a &ldquo;good thing&rdquo;. The most obvious reason is that of security: if you&rsquo;ve hardcoded a version of Python to download, build and install, and a security hole is found and a new version is released, you&rsquo;ll have to manually update all your recipes. If you&rsquo;re using distro packages, it&rsquo;ll happen more or less automatically in your regular updates.</p>

<p>The second reason distro packages are better is simply that of speed. Time is money: why spend it waiting Python to build?</p>

<p>Hopefully this makes it a bit easier for you all.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Being Part of the Open Source Software Ecosystem]]></title>
    <link href="http://DigiACTive.github.io/blog/2013/12/21/being-part-of-the-open-source-ecosystem/"/>
    <updated>2013-12-21T03:23:08+11:00</updated>
    <id>http://DigiACTive.github.io/blog/2013/12/21/being-part-of-the-open-source-ecosystem</id>
    <content type="html"><![CDATA[<p>At DigiACTive, we&rsquo;re building our solution on top of an open source software stack. Practically, we had a number of reasons for chosing open source, including price, ease of access and cross-platform compatibility. More fundamentally, open source software has played a huge role in bringing technological innovations to those who would otherwise not have had access to them, and that&rsquo;s very much in line with how we see the Digital Canberra Challenge.</p>

<p>It&rsquo;s therefore very exciting to be part of advancing the state of open source software in a couple of small ways.</p>

<h2>Improving django-storage-swift</h2>

<p>We recently submitted a set of patches to <a href="https://github.com/wecreatepixels/django-storage-swift">django-storage-swift</a>, a Python module for interfacing between <a href="http://django.org/">Django</a>, our web framework of choice, and <a href="http://docs.openstack.org/developer/swift/">OpenStack Swift</a>, our object store of choice.</p>

<p>Our main contribution is support for temporary URLs. Temporary URLs provide a way for us to grant users access to their files and their files only in a simple and transparent way. In particular, they are an excellent and simple mechanism for preventing attackers from accessing confidential information by guessing file names. We also improved the out-of-the-box experience for users of django-storage-swift, and made a couple of other minor fixes and improvements.</p>

<p>We&rsquo;ve made our changes available in <a href="https://github.com/DigiACTive/django-storage-swift/tree/digiactive">our fork of the project</a>, and contributed them back to the original project with pull requests.</p>

<h2>Updating the clamav chef cookbook</h2>

<p>We&rsquo;re using <a href="http://docs.opscode.com/chef_solo.html">chef-solo</a> with <a href="http://vagrantup.com/">Vagrant</a> to deploy repeatable environments for development. We recently tried to install <a href="https://github.com/RoboticCheese/clamav">a cookbook</a> to support <a href="http://www.clamav.net/lang/en/">clamav</a> so we can virus scan user-submitted files, but discovered that it&rsquo;s not compatible with the most recent updates to the <a href="https://github.com/opscode-cookbooks/yum#notes">yum cookbook</a>. We&rsquo;ve updated it and submitted the changes back to the original author, and they&rsquo;ve been well received and merged in.</p>

<h2>Looking forward</h2>

<p>We&rsquo;ve been making good progress building our framework and we&rsquo;re looking forward to making increasingly rapid progress towards the Digital Canberra Challenge project!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Hello World]]></title>
    <link href="http://DigiACTive.github.io/blog/2013/11/24/hello-world/"/>
    <updated>2013-11-24T16:58:20+11:00</updated>
    <id>http://DigiACTive.github.io/blog/2013/11/24/hello-world</id>
    <content type="html"><![CDATA[<p>Greetings, Internet!</p>

<p>We&rsquo;re DigiACTive, a team of students from the <a href="http://anu.edu.au">Australian National University</a> competing in the inaugural <a href="http://digitalcanberrachallenge.com.au">Digital Canberra Challenge</a>.</p>

<p>We&rsquo;re working with the <a href="http://tams.act.gov.au">ACT Territory and Municipal Services Directorate</a> and <a href="http://nicta.com.au">National ICT Australia</a> to improve the system of obtaining government permits and approvals to hold public events. At the moment, permit applications are long and complex, and it&rsquo;s easy to accidentally forget to submit important documents &mdash; making the process difficult and frustrating for event organisers. On the other side, the government&rsquo;s legacy systems make it hard to track the progress of applications, making the approval process slower than it ought to be. Overall, it&rsquo;s all a bit of a mess!</p>

<p>That&rsquo;s where we come in. Over the next few months, we&rsquo;ll be developing a proof-of-concept system to help both event organisers and government officials keep track of complex applications while making sure they&rsquo;re complying with all the appropriate regulations. Our prototype will help guide the ACT Government as it adopts new technology to replace the existing permits system.</p>

<p>It&rsquo;s early stages at the moment &mdash; we&rsquo;ll try to keep this blog updated as we go so you can have some idea what we&rsquo;re up to. In the meantime, if you&rsquo;ve got some ideas on how to improve the permits system, we&rsquo;d love to hear from you! Contact us at <a href="mailto:digiactive.canberra%20at%20gmail%20dot%20com">digiactive.canberra at gmail dot com</a>.</p>

<p>- The DigiACTive Team</p>
]]></content>
  </entry>
  
</feed>