Today’s post comes from GEOVIA Technical Architect, Charles Elliott.
For some time now, file servers or web storage systems have been the go-to mechanism for people in mining to share their data. However, there are numerous problems with this approach. More sophisticated tactics are necessary to drive both better collaboration and organization so that users don’t waste time searching for data, or worse, use the wrong data when making key decisions.
Most file servers use the same approach, which is to replicate your PC file and folder structure on a centralized server. This is fine when there are a limited number of files in play, but when there are hundreds or thousands of files shared by multiple people at different versions and from differing locations, things quickly start to fall apart.
There are numerous reasons for this, but here are the Top 10:
1. Inability to search for data effectively
The principal reason for data confusion stems from every user having to follow the folder structure that was set up by the initial user. If, for example, the folder structure is set up by a mine site, but you want to search by author, there is no easy way to allow you to do this effectively. You can create subfolders by author. but you will still have to waste time drilling into each folder until you find what you are looking for. Worse still, there can be tens or hundreds of topics by which your data should be organized, (state, commodity, lease number etc.), and even having a text search on your file system won’t necessarily turn up the right results or may result in numerous false positives.
The key to being able to find data successfully is by using metadata to describe the information in your data store. However, you need to ensure that everyone doesn’t make up their own categories or you may soon find that one person has stored data under “Author” while someone else has stored the same data under “Authors”. Furthermore, a free-for-all tends to lead to the use of nebulous concepts like “Keyword”, which means different things to different people. As a result, it is important to have a controlled structure for metadata, which allows everyone to catalogue and find data in a consistent and ordered way that doesn’t rely on file and folder names.
2. File systems aren’t designed for big mining files
Mining data can consist both of large numbers of small files, for example string files or survey data files as well as multi-gigabyte block models and grid models. Being able to handle big files effectively is a challenge for generic file servers because they need to transmit the entire file over the network each time. In addition, once the update is made, any previous versions (no matter how valuable) are instantly overwritten. That block model that consisted of three months’ intense geological interpretation has just been overwritten by the intern who was “just playing around” and you know that IT doesn’t have a backup, or only one that is less than a month old, anyway.
3. Multiple versions use many times the disk space and network bandwidth
When dealing with large files, there is a tendency to make a copy for each version. Every version takes up as much disk space as the previous one and a bit more, and of course nobody deletes the old ones. And, we all know that disk space is not cheap. The more data that is stored, the more confusing it is to sort out the wheat from the chaff, and the harder it is to determine which data is correct. In addition, all this data has to be backed up and stored somewhere else incurring additional costs. If some of your users are located at remote sites, then all this data needs to be transmitted over the network, taking increasing amounts of time, which easily becomes out of step again. If your network isn’t always up to scratch, then the right data may never get there.
4. Easy to make multiple copies – no single version of the truth
When users want to experiment with a particular scenario, and especially when using valuable data, users tend to make duplicate copies (“just in case”). It is so easy just to click Ctrl C and then Ctrl V from the root level of a folder to make sure that there is a copy of everything. After a few users have done this, who knows which copy is the correct/up-to-date/production copy?
5. Little or no auditing of who did what, when or why
PC file systems aren’t known for their auditing capability. However in mining it is frequently important to know who did that last change. Was it the fully qualified geologist who did that reserves estimate you are submitting to the stock exchange, or was it just something the student was playing around with? We have come across mining companies that have been fined tens of thousands of dollars for this one reason alone.
6. Not possible to back-track to previous versions
If we do find that the correct version of the data has been overwritten, how do we go back to the old/correct version? Maybe someone made a copy? Maybe not? Perhaps IT has a copy somewhere? Maybe someone has a copy on an old USB drive? Wouldn’t it be better if we could simply roll back to the previous version and be sure in the knowledge that all the correct updates are there?
7. Applications need to have a permanent connection to a network drive and will not work off-site
If your mine site is on the west coast of the US, then you are in luck—it probably has a good maintained connection to the internet with good bandwidth and few delays. However, if it is elsewhere, you might find that connecting to a file server at HQ to be challenging. If your file server is at site, are you sure that it is physically secure and that someone is backing it up every day? Most file servers and web storage mechanisms need a permanent connection to the internet to function correctly. If the link goes down part way through a file copy, the whole process will abort or you will end up with some corrupted files.
8. File servers are very slow when used for heavy graphical operations
Anyone who uses 3D design software, mapping products or mine planning tools will know that trying to access the files over the network can be extremely frustrating. The software continually goes back and forth between the server and the PC memory trying to read the bits of data it needs for its computations. When dealing with large block models, for example, it can be very slow visualizing the data that you need frequently, which makes work impossible. The only effective way to work is to copy all the data onto the local machine, but this then leads to multiple versions and clashing updates between users, as discussed in Point 4.
9. Limited or painful security
There are two ways to make people ineffective at work: one is to supply them with no data, and the other is to flood them with data. Not only is effective role-based security vital for ensuring that data doesn’t get into the wrong hands, but also it also ensures that users only see what they need to, not the 10,000 files that were used by finance for the last audit. Controlling security on file servers can be tricky, especially if you don’t have direct access to the servers. Most organizations rely on repeated requests to the IT department, but this often gets tiresome and users find ways around the problem or give up.
Users need to share data effectively with people who share a similar job function. Many systems allow you to “share” a file with another user, but it can be very difficult to collaborate with multiple users and then maintain updates to that data in a controlled and consistent way. Frequently, users are frustrated by having their data overwritten when changes are made by multiple people. This wastes time as they have to do redo their work.
10. No ability to be notified of changes to items of interest
If someone changes data that is vital to your job role, but forgets to tell you, how will you ever know? That vital reserves estimate or the parameters feeding the strategic plan could be changed, but unless some tells you or you pay very careful attention to the date and time stamps on those files, you may never even know of these changes. Even then, if users are in different time zones, you may not notice the change. Ideally, you would want to be notified when an item changes, but at the same time, you don’t want your Inbox spammed with notices every time a file changes on the server…
So what’s the Solution?
You won’t be surprised to know that GEOVIA has a solution for all these problems in the form of its secure data collaboration product GEOVIA Hub™. If you have two or more staff in your company who need to share data then you should give this application this some serious consideration. The combination of central storage combined with auditing, versioning, metadata storage, and remote access capabilities provide a number of key advantages for every mining business:
- Create and maintain a central repository of your project data to reduce errors and improve productivity.
- A centralized secure collaboration infrastructure for mine planning can, depending on the implementation, provide infrastructure support for several regulatory requirements such as JORC, SAMREC, NI-43/101 and Sarbanes-Oxley.
- Simplify the database building process by allowing you to easily combine data from multiple sources.
- Increase productivity by allowing staff members at any location to share data or work with the same data at the same time.
- Reduce rework as a result of lost or overwritten data
- Less time is spent by staff searching for data, and when they find it, they know the data is the correct version
- Eliminate costly mistakes spent in working with incorrect data and uncoordinated projects
- Incorrect reporting and invalid decision making are reduced
For more information on Hub, click here, or contact your local GEOVIA account representative.
Latest posts by GEOVIA (see all)
- Join the GEOVIA User community - July 13, 2021
- GEMS Software Update - June 21, 2021
- Reducing Impacts on Water - May 25, 2021
