Hello dear reader. It has been a while since I had more than 140 characters’ worth of time to write recently. This statement is not a complaint it is a fact. The reason for this situation is that I have been working on my photo processing skills of photos I have taken. I have been learning Scott Kelby’s 7-Point System for Adobe Photoshop CS3that a friend of mine gave me. This effort is in addition to going through the 40,000 and growing images that I started at the beginning of 2010.
This learning started as a diversion of the reviewing process, mainly I was looking at it as a sorbet between course, a white room in the baroque period. It started the process of reviewing what were keepers and what were electrons waiting to be recycled. I found that I was able to process more images into pictures (see previous entry about image classes). This new ability made me want to review some of the images that I culled out thinking it was not a keeper. That is where my realization of needing refinement in my work flow occurred. I could not easily find these images. Being that I have literally terabytes of data that I have cataloged using where is it but I still could not find what I was looking for (insert Bono or U2 joke here).
The flaw, I had changed file names. Some of these changes occurred during the view and culling process occurred to “help” find duplicates, or were just from not having a work flow when I took the pictures, others were me just plain doing things oddly through either lack of experience or lack of forethought. This little change made it very difficult to find the images again. As a result I have made a change and figure I should share it with my loyal reader. Now there are some variances depending on where I am in the world. So with out further exposition or commentary here is my new system.
- Go out and take pictures. Sometimes I use multiple cameras at the same location, a point and shoot, a DSLR, and even heavens no – a cellphone camera
- Return to the computer with flash cards full of images.
- Load images into Adobe Lightroom, keeping the same file names the camera assigned. However the images are grouped into directories by year – month – day. More about that later.
- Take the original JPG files and upload them to my SmugMug account in a non-public area of my site.
- Convert RAW images to JPG with no processing and upload to SmugMug at high resolution.
- Go through and start culling images in Lightroom. Do any needed processing, captioning, and keywording.
- The keeper JPG images get published to SmugMug in the appropriate location using the service built into Lightroom typically.
- The keeper RAW images are exported to SmugMug in the appropriate location using the service built into Lightroom typically.
- The RAW images are exported to DNG files.
- The DNG files are loaded onto my JungleDisk location
Through this new process I have accomplished the following relatively easily:
- Copy JPG files to an offsite location that is in the cloud using Amazon’s S3 service (SmugMug).
- Organized the images so I can find them again.
- Copy DNG files to an offsite location on the Amazon S3 service (JungleDisk).
- Not have to worry about the images being readable on the optical media I currently am using.
You are probably saying to yourself, Brad have you really made things better. From what I can tell I have. In a previous post I talked about the importance of offsite storage. I have been backing up stuff to both optical media (CD, DVD, or DVD-DL) and then key items to offsite storage. It started to become a huge pile of stuff having 400 DVDs/CD to physically store. I realized that there are lots of duplicated files in there. Why so many duplicates redundancy and peace of mind I thought. I wanted to make sure that if I lost a disk, either through damage or actually misplacing it, I would not lose an inordinate amount of data. What I ended up doing was having 400 pieces of media I had to catalog and store. Lots of duplicates yet I still had everything stored in the same physical location and then I had it backed up to the cloud. I read about Amazon’s S3 data durability and started to think about how secure my data was with "99.999999999% durability and 99.99% availability of objects over a given year". I also read a blog post and the FAQ explaining how "Amazon S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years. In addition, Amazon S3 is designed to sustain the concurrent loss of data in two facilities." I have a bigger loss from losing one CD.
I looked at the pricing, the redundancy and data durability and realized I could make my process easier and be more secure at the same time. SmugMug has unlimited storage on their service and uses Amazon. JungleDisk has two options for storage, a Rackspace hosted cloud that they can’t quantify its durability and Amazon S3. Yes, I am paying monthly rental based on spacewhich is why the really good stuff is going up there in DNG format. Everything though is going onto my SmugMug account in JPG format.
Browsing for images is now easier and possible anywhere. I log into my SmugMug account and browse through the appropriate directories and find the image I want or use Lightroom if the original is still local. Within SmugMug I can even create Smart Galleries to narrow the search. I then either download the original JPG file from SmugMug or go get DNG file from JungleDisk if it was shot RAW. But how will I correlate the information to retrieve the proper file you may ask; remember I do not change the filenames. SmugMug and Lightroom both give one the ability to display the filename.
If you are shooting with multiple cameras isn’t there a chance for duplicate file names? Yes, there is. That is how the date based directories help. They minimize the risk of having the same file name in the same directory within Lightroom. Within SmugMug I have a gallery of originals dedicated per camera so the risk of duplicates is reduced. On JungleDisk, I have a directory per camera as needed. If I find myself having duplicates on SmugMug or JungleDisk I will use the date directory approach.
Through this new process I have accomplished the task of safely storing the images, made them findable, not having to continue to purchase hard drives and optical media to keep local copies that are not are durable as the cloud copies. Is it more expensive? From my math since the only additional storage I will need to pay for is the DNG files, not really. I would already back up the keepers offsite anyway. I just changed the format and am not using JungleDisk backup service but the volume space.
The odds of all the data being effected just went way down and the places to look for it and back it up to went down. I can do this process away from home as long as I have high speed internet or patience.
And the other thing that happens is that I have moved it off a format that could become obsolete and thus unreadable. Anyone remember floppies? The 5.25 inch not the 3.25 inch ones? SCSI Drives, Zip disks, Iomega Cartridge Drives, punch cards… cuneiform and keeping the data accessible is now someone else’s problem; I just need to get to the Internet.
I also need to give credit to Don MacAskill Co-Founder, CEO & Chief Geek at SmugMug. I started thinking about data durability from his Twitter posts. Go follow him now, I will wait here.
For those that are truly curious, this post is the one I started writing on the iPad on a plane in the middle seat. I had to do lots of editting but the idea was captured.