The DSLR Filmmaker’s Handbook: Real-World Production Techniques, Second Edition (2015)

Chapter 8. Organizing and Storing Data in the Field

You are a filmmaker and shouldn’t need to concern yourself with data storage and data management, right? In the current state of production, there is a good chance that those tasks will fall to you, or it will at least be your task to find someone to do that job for you. Don’t take this task too lightly. The entire success of your production may depend on it. As productions are shrinking in overall crew size and more and more tasks are falling to fewer and fewer people, it is often easy to miss things because you will be juggling a lot. With a little forethought and planning, you will save tons of time and headaches later in post-production.

Setting Up a File System

File management has become more complicated since we left the world of film and video tapes. In those days, it was easy to refer to the capture medium and find takes or footage. On film you always could refer to the film frame numbers or reel number, and on video tape you could refer to the tape name or the time code value. Unlike the days of shooting tape or film, you cannot just label a tape or a roll of film and set it aside for later. Your capture medium consists of digital files and folders upon folders of these files. If you are not careful, you can lose, overwrite, or erase your footage in an instant.

Now that filmmakers work in the wonderful world of tapeless workflows, it is important to create an organizational system in order to locate footage easily and quickly.

If you are shooting a short film or a smaller project, file management is a lot easier. When working on a feature or long-format project, things are much more complex, and it is easy to get overwhelmed quickly. There will be dozens or hundreds of hours of footage, thousands or tens of thousands of movie files, multiple cameras, and dozens of capture cards to keep track of. To help track these details, you must have a coherent naming system for all the moving parts in this workflow.

First Things First: Planning Your Project

One step in the planning process that is often overlooked is the digital asset management of files on location. Once shooting is under way, it is hard or even impossible to change your management system without causing a lot of headaches.

Think about the details of the project you are working on, how long the shoot will be, how many cameras and cards you will be using, and what sort of audio files (if any) you will need to pair up with your video files.

Refer to Chapter 10, “Converting and Editing Your Footage,” for more information about hard drives.

Labeling Equipment

To implement a coherent naming system for all your files in your workflow, label each camera for the shoot with a number. The easiest way to do this is to put a small piece of tape directly on the camera (Figure 8-1). Shurtape white gaffer’s tape is ideal, but masking tape will work just fine. Using tape also allows you to easily and quickly relabel your cards. Just make sure the tape you are using is not thick and is applied flat. If the tape bunches or is too thick, it can get caught when putting the card into the camera.

Figure 8-1: This is camera B from a multiple-camera shoot.

Second, take all the cards you will be using during the shoot and label them with letters, as shown in Figure 8-2. This is key to being able to track what footage came from what camera, and if you have any issues with either a camera or a card, you can easily track down which one is the problem and remove it from the workflow.

Figure 8-2: Label each card you will use on your shoot.

Regardless of what camera system you are using, the end result will be a lot of video files with strange numeric names. Most of the major camera manufacturers add a three-letter prefix and a four-digit number to each file. This is great if you are using a single camera or are shooting fewer than 9,999 shots in your production.

If you are shooting with multiple cameras, there is a good chance you will end up with different files that have the same name. If you are not careful, you might overwrite or delete what you think are duplicate files by mistake.

In the days of shooting 35 mm motion picture film there was a production assistant (PA) dedicated to running exposed film back and forth to the lab. That meant someone was always in charge of making sure there wasn’t any missing film. Furthermore, if you were ever shooting in a location with no lab, the PA would hop on a plane and carry the film from the set to the lab. Nothing was left to chance.

I am finding more and more these days that people copy files onto a drive and think they are covered. I caution you not to fall into that trap. Digital files are fragile, and the files themselves and/or the devices that store those files can go bad in an instant.

Think of it as magic—now you see the file and now you don’t. I demand on all my sets that footage be backed up to a minimum of two hard drives before any card is cleared. If the media isn’t backed up to two drives, then the original capture card does not get cleared until the main storage drive on set is backed up at night back in the hotel room or studio.

If you don’t plan accordingly, you may run out of capture media if you follow this rule. I can’t tell you how many sets I have been on that spend $100,000–$500,000 on the shoot and rely on one hard drive to keep everything safe. I have given my personal drive to the person backing up footage on some of those sets where I didn’t have control and in one case saved the entire production when their drive failed. Had I not backed it up, they would have lost $350,000 in production costs that were not insured.

Even if your budget is small, treat it like you are burning money if you lose footage. Let’s assume you own your camera and you aren’t paying out any money on a short film you are shooting over the weekend. You have family and friends who will help you, and they aren’t charging you a dime. This is great—you can shoot all day Saturday and Sunday and get what you need for free. How do you think they will respond if you didn’t handle the footage properly and you have to tell them you need another weekend of their time for free because you lost footage? My guess is they won’t be very pleased. Time in many ways is worth more than money. So whether it is time or money or both you are playing with, make sure to guard your footage with your life.

If you want to do some more fancy data management starting in the camera, here are some cheats that will help you get organized from the very start. If the following sections are a bit too much for what you want to do, you can skip ahead to “Transferring Files from Capture Media to Hard Drive” and continue reading from there.

Manually Setting File Numbering

Once you’ve labeled your cards and cameras, set up the file-numbering system on your camera so that you can stay organized. This differs from labeling your camera and cards because this deals directly with the labeling of the footage on the cards and when you move the files to a hard drive. The following sections explain how to set up file numbering manually within Canon and Nikon cameras.

Canon

Many Canon cameras number their files MVI_XXXX.mov for videos and IMG_XXXX.jpg or IMG_XXXX.CR2 for images. In these models there is no way within the cameras to manually set what number you want to start the movie files at. The biggest problem with this is that if you are using multiple cameras, there is a very good chance that you will end up with movie files with identical names that are in fact different takes from different scenes. If your digital media person on set isn’t careful, this is where you can lose a ton of your footage really quickly, and it will be lost for good.

Some newer Canon cameras allow you to either name the files as you want or default to different default naming conventions. Don’t let a different naming convention throw you off; you can still follow the steps listed here.

There is a workaround for this numbering issue even though it is not perfect. You still may end up with duplicate numbered files, but it will cut down dramatically on potential duplications and help you get organized for post.

Take one of your cameras and a card that you want to use with that camera, and follow these steps to set up a new numbering system for your video files:

1. Insert the card into your camera.

2. Go to Set-Up Menu 1 ⇒ File Numbering ⇒ Auto Reset, as shown in Figure 8-3. (Set-Up Menu 1 is the first yellow square that looks like a wrench with one small square.)

Figure 8-3: Select Auto Reset in the menu.

3. Take a picture or a quick video on the card, as shown in Figure 8-4, and confirm that it is present on the card.

Figure 8-4: Confirm that you have a picture or video on the card.

4. Remove the card, and insert it into your card reader on the computer.

5. Rename the IMG or MVI file to the number you want to start with for that camera (in other words, MVI_3000, as shown in Figure 8-5), and then remove the card. You must adhere to the numbering convention of the DSLR—in this case, a four-digit file number.

Figure 8-5: Rename the photo or video file.

6. Insert the card back into the camera, and take another picture or movie clip.

7. Verify that the new file incremented from the file you had on the card (that is, the new clip would be MVI_3001, as in Figure 8-6).

Figure 8-6: Confirm that the new image or video is incrementing properly.

8. Delete the image or movie test file in the camera or from the computer, and you are set.

The camera will now increment with this number unless you insert another card with a different numbering sequence or you choose to reset the autonumbering from the camera menu.

Go through this for each camera you will use for shooting. If you have numbered each camera, then set the four-digit number to match that camera’s number. For example, camera 1 would read MVI_1000, camera 2 would read MVI_2000, and so on.

Once you have formatted each camera and the card, it will continue to number in sequence unless you reset the numbering in the menu.

Naming Your Files in the Camera

Some cameras such as the Canon 5D Mark III and the Canon 1DC allow you to enter any naming convention for your files. If your camera allows you to add your own prefix or use any naming convention you want, then you can skip the previous steps and just type in your own system.

Nikon

Nikon cameras number their files DSC_XXXX.avi for video and DSC_XXXX.jpg and DSC_XXXX.NEF for images as the default naming/numbering convention. In the menu settings, Nikon allows you to custom set the first three letters of the files but not the numbers. You can choose to set the first three letters either as your scene number or as any letters you assign to your scene. If you choose to set the prefix as your scene number, then you need to make sure to change the numbering to the new scene number when you move to a new scene. The benefits here are that you won’t have duplicate file numbers and that you don’t need to set up the file numbers on your computer.

Normally it doesn’t matter which card goes into which camera, but for this workflow you need to assign cards to a camera. The reason for this is the settings you set up in the camera are written to a card and stored on the card. If you format a card on your camera 1 and insert it into camera 2, the numbering will continue the numbering sequence from camera 1, not from camera 2. Make sure your camera operator and digital media manager on the set know which cards are for which camera so you don’t accidentally move the card to another camera before reformatting the card. If you need or want to use the card in camera B after using it in camera A, it just needs to be reformatted in camera B and set to the next number in your numbering sequence for that camera.

Setting Up Separate Folders in the Camera

Having all your movie files numbered in sequence can make it difficult to separate all the video files for a given scene you are shooting. If you don’t have the time to copy all the files from your compact flash (CF) card before you move on to a new scene, you can set up a new folder on your card to hold the new video files. This way, you can separate the video files from one scene to another in the camera as opposed to at the end of the day or the shoot when you have more files to deal with.

You set up your card number the same way we talked about on each camera and start shooting your first scene. When you move to the next scene, you need to follow the instructions in the next sections, depending on what kind of camera you have.

Canon

To set up separate folders on your Canon camera, follow these steps:

1. Go to Set-Up Menu 1 (the first yellow square), and choose Select Folder. Press the Set button to select (Figure 8-7).

2. Scroll down to Create Folder. Press the Set button to select (Figure 8-8).

Figure 8-7: Choose Select Folder from the menu.

Figure 8-8: Select Create Folder from the menu.

3. The camera will ask you whether you want to create folder 101. It numbers sequentially, so you have no manual control over the number unless you change the number on the card from your computer. Select OK.

4. Select the folder you want to start capturing your videos to (Figure 8-9), and press Set.

Figure 8-9: Select the newly created folder from the menu.

Nikon

To set up separate folders on your Nikon camera, follow these steps:

1. Go to the menu and select Shooting Menu ⇒ Active Folder (Figure 8-10).

Figure 8-10: Select Active Folder from the menu.

2. Select Active Folder ⇒ New Folder Number (Figure 8-11).

Figure 8-11: Select New Folder Number from the menu.

3. Select the folder number you want (Figure 8-12). Press OK.

Figure 8-12: Select the folder number you want to create.

4. To select one of your folders, go to Shooting Menu ⇒ Active Folder ⇒ Select Folder.

5. Scroll up or down, select the folder of your choice (Figure 8-13), and click OK.

Figure 8-13: Choose your newly created active folder from the menu.

Using the Camera’s Clock as a Timestamp

Set the clocks on your camera accurately before you begin shooting. If you make sure to do this, you have another safety net for finding your footage. You can sort for the timestamp on the files and find the footage that was shot from different angles on the same day. You should not do this instead of the numbering convention but as an added search function for post.

Understanding Current Types of Capture Media

As the number of cameras you have available to shoot with grows, so does the universe of capture media you have to choose from. Here are the leading ones you need to be aware of and prepare that you will likely be using at some point in the near future if you do not currently use them in your existing workflow.

Figure 8-14: SanDisk Extreme Pro SD cards

SD Cards

Many leading cameras either exclusively use or have a slot that allows you to use Secure Digital (SD) cards (Figure 8-14). These have been around since almost the beginning of digital still cameras and many home video HD camcorders. They are the most abundant and most readily available capture media in the world.

What you need to be aware of as the landscape continues to evolve is the storage capacity of the card as well as the rates at which it can write data to the card.

CF Cards

Figure 8-15: SanDisk Extreme Pro UDMA 7 CF card

CF cards (Figure 8-15) are another popular capture media for a variety of cameras. Everything from the Canon 5D Mark II and Mark III to the C300 and others has the ability to capture to CF cards (and/or SD cards at the same time).

Note that you have to pay attention to the read/write speeds of the cards. If you have an older camera and have been using CF cards, your current cards may not be fast enough to use in newer cameras with higher bit rates and faster write speeds. Some people make the mistake of saying “it fits” and think they are OK. Check the speed requirements for your camera and make sure the cards you buy (or have on hand) will work. You can always use newer, faster cards in older cameras but it doesn’t work the other way around. If you are shooting stills, the speed of the cards is less important than when you shoot video. It is critical for your cards to have high sustained data-transfer rates when shooting HD video.

SSD Drives

Figure 8-16: The Blackmagic Cinema Camera has an SSD slot for its capture media.

Many filmmakers may not be familiar with this new capture medium. SSD drives are like mini portable hard drives (Figure 8-16). They are solid-state drives, so they don’t have moving parts like a traditional hard drive. What is interesting about the use of an SSD drive is the huge capacity you can achieve. Unlike SD and CF cards, which at the top end usually don’t go above 128 GB, you can get SSD drives starting at 250 GB and going to 1 TB or more. You can think of SSD cards as huge memory cards that have the capacity of a hard drive.

Up until this point all capture media have been more or less internal to the camera you are using. One of the more recent developments has come in the use of external recorders. The Atomos devices are my new favorite production tools (Figure 8-17). They can double as an external monitor as well as capture higher-quality footage than if I recorded directly into the camera.

For instance, any camera with a clean HDMI or SDI output can be captured to an external device. Clean output means that your camera can output the signal from your camera without any of the display overlays, so the image you record in the external device is clean from any words, logos, or overlays. You will need to check your camera of choice because it is a mixed bag as to which cameras and which manufacturers allow clean output signals. When you capture internally into your camera, you are limited by the chip speed and the processing power of your camera. That means higher compression and lower data rates can be captured inside the camera.

Figure 8-17: Atomos Ninja 2 external recorder on a Canon 5D Mark III

When I am shooting with my Canon 5D Mark III, I can use the Atomos Ninja Blade to capture high-quality ProRes clips instead of using my H.264 files internally from the camera. All the files are captured to an SSD drive.

Transferring Files from Capture Media to Hard Drive

For your SD and CF cards you need to find a good card reader you can use to transfer the files to your hard drive. Currently a USB 3.0 card reader is probably your best bet.

In a pinch you can connect your camera directly to your computer (Figure 8-18) and transfer files from the camera to your hard drive. I typically don’t like to tie up the camera for this amount of time, so I recommend where possible to stick to a good card reader. If you choose to connect your camera to your laptop, make sure you have enough battery life in your camera to finish any transfers.

Transferring your SSD drives is not quite as simple. Outside of investing in SSD drives, you will need a dock or device of some sort to transfer the files to your hard drive for editing. So for many people, working with SSD drives means you must have another set of equipment to make everything work together.

Figure 8-18: Canon 5D Mark III connected directly to the computer via a USB cable

Figure 8-19: You can use an adapter cable to directly connect an SSD drive to your computer.

You can use a couple of different methods to transfer files from your SSD drives. First, you can use an adapter cable to connect your SSD drive to the computer (currently USB 3.0), as shown in Figure 8-19. This is a good option because of the small size. It is easy to travel with and have on standby.

Another option is to obtain an external enclosure (Figure 8-20). If you go this route, then you can leave it connected to your computer and just slide the SSD drive into the enclosure when you need it. This is a little cumbersome but it does work.

Figure 8-20: SSD empty enclosure case ready to put your SDD card into

The best method, in my opinion, is to use a dock connector (Figure 8-21). This allows you to connect SSD drives, 3.5″ external drives, and other devices to your computer. This does stay attached to your computer; you just slip in your SSD drive and you are ready to transfer files. One reason I like this is it that it gives you an option to connect more than one drive into the same device, so it multi-purposes your purchase.

Once you have the labeling and workflow set up in the camera, the second part of field management is transferring your footage from your card to a hard drive. The key is to start with a system that makes sense and will allow you to track and find footage long after you leave the set.

How Much Drive Space Will You Need?

Figure 8-21: Newer Technology dock that allows you to insert both SSD drives and 3.5″ hard drives into the same dock

It is hard to estimate exactly how much hard drive space you will need in advance of shooting. The actual file size for each video can vary based on what camera you are using, the resolution you are recording the video at, the frame rate you are using, the picture style settings used, and whether you are shooting for slow motion. Because of the numerous variables and the fact that the camera manufacturers generally don’t publish the data rate their video files are recorded at (or it’s very difficult to find), there is no chart you can use as reference.

Downloading and Viewing Movie Files from a CF or SD Card

In case you’re just beginning to shoot video, here are the steps to download movie files and view them on your computer:

1. Remove the card from the camera and place it into your card reader.

2. Open the card folder on your computer. Navigate to the folder named DCIM and then to the camera folder.

3. Copy this folder to your backup drive.

4. When the card has fully copied to your backup drive, open a few of the video files to make sure they are all working fine.

5. Repeat steps 3 and 4 with a second backup drive.

6. Delete the card.

7. Remove the card and place it in the camera.

With that said, we can help you estimate pretty accurately how much hard drive space you will need. Take the camera you have chosen to shoot your project with and shoot 5 to 10 test shots that are exactly 5 or 10 minutes in length. When you have completed that, copy those files onto your computer and view the file sizes. You now have all the data you need to estimate your hard drive space requirements. Add together the file sizes of your sample clips and divide by the number of clips. This will give you the average file size for the length of time of your test clips.

Now you can figure out the amount of hard drive space needed for your shoot. Determine the average file size per minute from your test clips. Then estimate the number of minutes you will be shooting per day. Multiply that by the number of cameras you are using and by the total number of days you plan to shoot. This will get you a pretty good estimate of how large a hard drive you will need.

If you are shooting in a studio or near civilization, then you can always run out and get more drives. If you are shooting on location or in a remote area, then you need to be much more careful about having enough hard drive space, because you don’t have the option of running down to your local computer store. Buy more drive space than you calculate you need if you will be shooting remotely. If you run out of drive space, it is just like running out of film or tape. Your production will shut down until you get more drive space in set.

How Do You Keep Your Labeling Straight?

You went through the process of labeling your camera and cards and setting up your file numbering. When you copy from your CF card to your hard drive, you want to make sure the labeling carries through to your computer. First, set up folders for the days you will be shooting (for example, YYMMDD) so that you can quickly find footage from any particular day of your shoot.

Using the YYMMDD naming convention guarantees that you will never have a duplicate date and that your files will always be in order from the oldest footage you shot to the newest.

Next, set up a folder on your hard drive for each camera (for example, C1) and, within that folder, a subfolder for each card that was used or will be used by that camera (for example, C1A, C3B). Then when the camera department sends a card to be copied, you can put the labeled card into the labeled folder from the labeled camera. This may sound a bit confusing, but try doing this while you are testing before you go out on the set. In addition, make sure the system is clear to the camera department and the data manager for the shoot.

Understanding Different Capture Formats

Let’s quickly cover the capture formats for the various DSLR cameras currently on the market:

· Nikon’s DSLR cameras use Motion JPEG AVI files as the capture format for video.

· Panasonic DSLR cameras use AVCHD as the capture format for video.

· Canon DSLR cameras, on the other hand, have embraced video recording in H.264 movie files.

· Sony cameras record in XAVC or AVCHD formats.

All the major NLE editors can handle any video files coming from any of the leading camera manufacturers. If you buy a brand-new camera two years from now, you might run into an issue, but with the growing impact of DSLR cameras on the market, it is unlikely that you will see issues with capture formats not working with editing programs in the future.

Delivery Format vs. Capture Format

In general, a capture format is one you should use when you want the highest-resolution video signal (preferably uncompressed). Then you are starting with the best possible image, and you have more latitude in making color corrections and adjustments to the footage. In general, higher bit rates and high-resolution capture formats are much too large to burn onto a DVD or stream on the Web.

A delivery format is a compressed video format that provides for the best preservation of the original footage’s quality at the lowest bit rates and file size. This allows for easier streaming and the ability to be burned to DVDs and other storage devices.

Backing Up Your Footage on Set

As soon as a card or SSD drive is removed from the camera and handed to the person handling the media management, it must be treated like exposed film. If it gets damaged or lost, you are out of luck. The card should be backed up no less than two times prior to being cleared. If you have the footage on only one card or one hard drive, you are at major risk for losing your footage and having to reshoot or not finish your project.

Once the footage has been backed up two or more times, you are free to clear the card and send it back to the camera department. If you are low on time and people, you can skip clearing the card on the computer and have the camera department reformat the card in the camera. The only thing to note is that if there is any video or picture on the card when it is first put in the camera, then the camera will continue numbering from that number even if you format the card before you shoot the next scene. To be safe, clear the card before sending it back to the camera department, or have the camera team clear the card and set up the numbering again to make sure it continues in sequence.

The hard drives you are using are obviously essential to data management and must be protected (Figure 8-22). Most of the time, however, hard drives are thrown about with reckless abandon. Clearly you don’t want your hard drives to fall or suffer any sort of impact or encounter heat or water damage. This means that on set your hard drives should have a weatherproof carrying case and should be stored in a climate-controlled environment.

Figure 8-22: Real-life example of how not to handle the hard drives that carry your footage

Hard drives can be damaged by pulling the plug or turning off the drive incorrectly. Unplugging a hard drive without properly shutting it down or ejecting it can also lead to data loss, so develop a system that includes a shutdown procedure.

Static electricity can also damage your hard drive. As you swap or touch hard drives, make sure you discharge any electric charge prior to touching the hard drive, and if you are in a static-prone environment, use an antistatic mat.

Organizing Data on Set

Here is what you will need to have on set for your data management and storage:

· Multiple CF cards or SSD drives to make sure you can rotate cards/drives into your system and have enough on hand to be backing up and shooting at the same time without waiting

· An SD/CF card reader or SDD dock so you can transfer the footage onto a hard drive in the field

· Tape and a Sharpie pen for labeling cards, hard drives, and many other unforeseen things

· Notepad and pen/pencil for taking notes and keeping track of footage, drives, cards, takes, script notes, and who knows what else

· Laptop for viewing the footage and using it as the hub for all data transfers and logging

· Minimum of two hard drives so you can always have two backups of all your footage before the capture media get cleared and sent back into production

We’ll talk more about backups in Chapter 10.

Managing Files on the Set

Normally when shooting film or tape, you can track and find your footage easily by the roll or tape number, your slate, and your script notes. But DSLR filmmaking is a tapeless workflow with no time code or standardized numbering system.

While we were shooting The Shamus, ideally we would have loved to slate as a reference for every scene and take, not to mention how this would help sync audio in post. But because we were shooting in Italy without permits, we had to keep a low profile. We knew we would be kicked out of many of the areas we were shooting if we tried to slate the takes. For us, the data management of the files was critical because if we messed up, we would more or less have to watch each shot of our thousands of takes to label what scene, take, and location we were at.

We did this by following our system of a folder with the date, a folder with the labeled camera, and a subfolder of the labeled card. This way, we were able to take our shooting schedule and match what footage was in which folder and find it quickly and sync in post without any formal slate or script notes to follow. We are aware that this is not an ideal way to run a shoot, but since many people are using the cameras to shoot in locations they could not before, this is a critical step to master to save you time and stress.

Arranging Your Footage

Now that you have your footage on several drives with dozens or hundreds of folders with thousands or even tens of thousands of clips, you will want to find a way to organize and tag all of the footage.

It’s not a matter of whether someone will have to copy and back up files on the set but a question of how much time they will have to do the copying and backing up. We recommend having a full-time person on set to do your data management. If you don’t have the budget or personnel, then be prepared for some late nights. The better you can organize and label on set, the less work you will have to do prior to getting started editing. Then you will be thankful you were so organized on set or fixed any problems with labeling footage.

The file structure that works really well is setting up a folder for each day of your shoot, as shown in Figure 8-23. Within that folder you can put a folder from each camera you used that day. In each of those folders, you can put folders from each card that each camera used. This way, you have a record of what was shot on what day, and if there is trouble with a camera or a data card, then it is easy to track down what was shot on the troubled camera or the bad card.

Figure 8-23: Example of file structure from a two-camera shoot

Figure 8-24: File structure of the H4n Zoom audio files from The Shamus

You should also have a folder each day for your audio files, as shown in Figure 8-24. This will cut down on searching for audio clips later, and if you have any trouble with automating the syncing of your audio, then you can work day by day and not get lost in what you have done and what still needs to be worked on.

You will be using your script notes, actual shot list, and audio notes to double-check your files, match scene numbers with files, and find any missing or unlabeled files.