Announcement

Collapse
No announcement yet.

Asset Management - The 2018/2019 ACU way

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Asset Management - The 2018/2019 ACU way

    Can we use ACU2018/19 as an asset management solution?

    Consider this asset management scenario:

    - Curating a large set of images, 10's of GB, 100's of thousands of images. In my case they are machine 'training' images
    - Images are kept in hierarchical set of folders
    - Original images are read-only
    - Images are (or become) part of a backup system
    - Images (assets) are catalogged, graded, AC-keyworded, etc, using the AC database (ie., AC managed)
    - New assets arrive regularly, thousands or even 10's of thousands at a time
    - Initial keywording can be often done in batches as the new sets are often pre-classified in sequence or sub-folders
    - Additional keywords and other attributes are added over time
    - Single user

    So ultimately we need an asset management database to select and build sets of images.
    The originals are read-only and we don't want to modify the original metadata, or trigger unneccesary backups.
    We don't want AC attributes to be embedded in originals, but do want to retain the original metadata.

    Professional photographers might see many parallels in their own workflows.

    All good so far. ACU can do most complex searches, except maybe filter for absence of certain keywords.
    A search can take some 10's of seconds or minutes to display results on a fast machine. But that's understandable using a DB.
    At least it works and seems stable and reliable. Many classification options.

    But now we need to physically reorganize the photos.
    Split and move the photos to several other hard drives, while retaining the AC attributes.
    Or maybe sometime in the future rename the asset drive letters or alter pathes without losing attributes.

    Have read through the forums and about 10 years of Google history on ways users have found to manipulate the AC database and photo collections.
    Can move the database itself using the directions I found in the article.
    Can split the database too. All good there.
    https://acdsystems.desk.com/customer...a-new-location

    But for the photos, as I understand it, the 'AC way' to move photos while retaining the AC attributes is to use AC copy/move.

    And this works using ACU2018. But it takes a very long time.
    Even with fast drives and very fast CPU, the fastest AC copy I get is about 1 photo/sec (ie., for <100kB files).
    To move (or copy/delete when done) a large number of photos using AC this way takes days or weeks, tying up the collection.

    Whereas a windows copy is tremedously faster, but of course it doesn't transfer the AC attributes in the DB.
    Using Windows copy, it will copy on the order of 100 files per second. Using ACU copy the best I found was 1 file per second.

    However using Windows copy, as the path name for the files change, the linkage in the DB is lost (as I understand it, >orphans).
    As in many other programs, an option is given to 'repair' the path of moved files:
    https://acdsystems.desk.com/customer...do-i-fix-this-
    However I have not succeeded in being able to re-link moved photos. Either my workflow is not supported or it doesn't work.

    So what's going on and what's the solution? (besides embedding. I understand embedding)

    I note that disk usage is very low, a small fraction of its potential. Memory usage is also quite low.
    Total CPU during ACU copy sits at slightly more than 25%.
    ACU2018 itself sits steady at 25%. This is like its only using 1 core (of the 4+). Of course the GPU doesn't matter for copies.
    The fastest CPU I've used is running ~3.7GHz.
    I've even changed systems and OS (moving the asset drives), antivirus set to ignore the associated folders. Same.
    It doesn't seem to matter how fast the drives are (SSDs or HDDs), only CPU clock speed seems to make a real difference.
    Training images are usually JPGs or PNGs and are mostly <100kB.

    What can I do to dramatically speed this up? The AC way? When embedding isn't an option?
    What can AC be doing to take all this time? That insight might help inspire some solutions.
    Is this something to do with automatic thumbnail creation? If so can I turn that off for the copy?

    If you have been here please share your solutions (besides embedding) and performance figures.
    How long does it take you to copy ~1000 small/medium size jpg's with their AC-keywords etc?

    Thanks. Tom


  • #2
    We have >280,000 raw files in stock. None ever gets moved, copied or altered after import to the master file folder system.
    The import of up to 2,500 raw files a day is done automatically in background: Just insert the sd card in to the reader. This part isn't done using AC.
    AC does write all meta data to xmp side car files for these raw files.
    Assets are created as "Categories" in AC. We don't use the stubbed "Collection" feature. This is part of my daily business with up to 1,000 images.
    These categories are exported to jpg or tiff when finished and moved/send to customers or online services.
    The exported images never get get catalogued in AC and aren't included in the backups. They get deleted shortly after the order is finished.
    .
    Our experience is similar to yours. AC uses just one core and seems to ignore the GPU for the tasks described above.
    I'd never ever use AC to copy/move images, the db is much to slow.
    The searches are dead slow, modern systems are much faster. AC's search feature gives lots of time for writing post in this forum :-)

    [Edit]
    Notice to your problems with embedded meta data: This has already been discussed. Read this: https://discuss.pixls.us/t/linux-app...onvention/2688
    It discusses the way how adobe wants meta data to de stored and the problems this raises. One of the links is broken, but the document can be found here: https://wwwimages2.adobe.com/content...ationPart3.pdf
    As a result of the discussion digikam for linux offers to use external xmp files for all file types, but also uses a different naming scheme making it completely incompatible with the industry standard. However it might offer a solution for you.


    Last edited by Emil; 09-15-2018, 01:59 AM.

    Comment


    • #3
      +1 Emil

      I am at only about 130,000 RAW files... but here is some of my experiences with Ult2018
      (I am hoping the upcoming Ult2019 may change things)

      Because of the DB, I expect ACD to be natively slower than Win10 with either copy/move files, Here is a data point from my local system using "copy".
      ACD; 28 seconds; 772 files; ~2.5M each; 1.8G Total
      win10; 2.5 seconds; 772 files; ~2.5M each; 1.8G Total
      However, 2.5 seconds to 28 seconds is ~10x slower

      Ways to speed things up
      * NVME drives connected directly to the PCIe bus create the numbers above. NVME over SCSI/SATA/SAS or worse yet, standard spinning platters over SCSI/SATA/SAS have a much larger impact on ACD than other applications. SCSI/SATA/SAS can 2x to 4x the listed times.

      * Be pathological with optimizing the database. Optimization every month is a minimum. When very busy, I have been know to kick off the optiimization every evening.


      Here are some ways to slow things down
      * Local files that are synced to the cloud slow things down. ACD is uniquely interupted by cloud syncing (y experience is with, Amazon files, Google Drive, OneDrive or Synology). In my experience, a syncing cloud drive slows down ACD actions by 2x or greater. NOTE: Most operation you will not notice this issue. It is very apparent with moving/copying large quantities of files using the ACD interface.

      * Multicore can help, but use of virtual cores seems to slow ACD batch and file processes. My experience suggests ACD does not play well with Hyperthreading. My system with an actual 4-cores becomes slower when configure as 8-virtual cores. This may or may not be the case with 2-core systems configured to 4-virtual cores. Your Mileage May Vary

      * Unproven: Based on cloud experiences, I would think MobileSync would also interfere with things. I do not use Mobile Sync so I do not know for sure.


      Wish list...
      Move/Copy in the background has been on my wish list for a long time. Why can't I can start working in "folder 2" as move/copy finishes with "folder 1". Of course, you can open a second instance, but that seems to have other issues.

      Optimize database and "close" application without user intervention.... Instead of Optimize database and prompt user for a "restart"



      Comment


      • #4
        Originally posted by GusPanella View Post
        Of course, you can open a second instance, but that seems to have other issues.
        My experience on this is very bad. Imho running several instances of AC (any version sine v2) will most likely crash after some time. For larger export jobs, under pressure, I prefer to use another copy of AC in a virtual machine. This allows to continue filling selections/categories or edits in the first instance. But it also forces to create a copy of the db first.

        Wish list... Move/Copy in the background has been on my wish list for a long time.
        It should be possible to run any task in background. Creation of proxy images in [developed] folder works pretty well.

        Comment


        • #5
          If its true that there's nothing users can do to significantly speed up AC copy/move, a couple of users have suggested putting the copy/move into the background which could allow AC to continue to be usable when lengthy copy/moves are taking place.

          Originally posted by Emil View Post
          .. For larger export jobs, under pressure, I prefer to use another copy of AC in a virtual machine. This allows to continue filling selections/categories or edits in the first instance. But it also forces to create a copy of the db first.
          Emil. Maybe you'll be able to improve my understanding of how this hack works.

          Because the db is not multi-user, a second instance of AC is created in a VM (on the same machine) with a copy of the db and used for restructuring the files, and this runs in the background (possibly for days) until complete. Meanwhile the initial/primary AC instance is continued to be used for new work/classification/selection/edits. So at the end we have two dbs, a db with the file restructure and a db with the latest classifications (keywords, etc) and files.

          Can these two db's be synchronized again and merged back to the primary (initial) db? Presumably the file/photo drives are shared between the two systems? Is auto-indexing of new files turned off in the VM instance?

          Last edited by TomPac; 09-17-2018, 03:05 PM.

          Comment


          • #6
            Originally posted by TomPac View Post
            Emil. Maybe you'll be able to improve my understanding of how this hack works.
            The second copy of the db in the vm just is a slave. It's always copied from the master first and overwrites the old slave. In fact it's a bit more complicated: The master db is quite big (>20gb) and to save some time when coping the master to the slave, the thumbs*.* files are omitted. Instead these few files are taken from a completely empty db located on the vm itself. The vm doesn't do anything else than export categories/selections, no editing of meta data is done in the vm. The images are not even viewed or browsed. Sounds pretty complicated, but its all done using batches and after all these years I don't even think about it; the few clicks go pretty automatic.

            Please keep in mind that all this is done because I usually export several hundred images daily, which takes lots of time and blocks AC and only if I need to do other tasks with AC at the same time.

            [This chapter mainly is addressed to the dev team, if they care:]
            The master - slave db concept has another benefit. When exporting images you may add the exported images to the db (Option "Preserve database information"). If you do, the exported images clutter the db. But it allows to view/organize the assignment of the exported images to the categories/selections which is needed because you can only export a single folder or to the folder of the master image files. There is no way to export images in to sub folders using category or selection names. You have to recreate this assignment by hand using lots of copy/paste! It's a tragedy! Imho exported images should not clutter the db and it should be possible to export images in to a folder using category/selection names if you export several categories/selections in a single batch. I often export dozens of categories in a single batch and then I'm forced to recreate the categories as folders and drag drop the exported images from AC in to these folders.

            Can these two db's be synchronized again and merged back to the primary (initial) db? Presumably the file/photo drives are shared between the two systems?
            All images are stored on a NAS. We don't use any of the proprietary AC meta data tags any more except the categories, because they aren't written(embedded)/read on the fly. Any instance of AC could modify the XMP tags, but because the slave db in the vm doesn't contain any thumbs, browsing would be to slow. So I don't do it at all.



            Comment

            Working...
            X