[TUTORIAL] Texture batch conversion and modification (Albedo-ready)

Discussion in 'Modding' started by FuNK!, Jul 15, 2018.

  1. FuNK!

    FuNK! Registered

    Joined:
    May 26, 2013
    Messages:
    475
    Likes Received:
    276
    Hey there,
    if you have ever tried to convert a track or mod with a massive amount of textures you will surely know that it takes a lot of time to edit the DDS textures one by one to suit the Albedo requirements by rFactor2.

    Tutorial for ImageMagick and DirectXTex (by @FuNK!)
    Note: Any of the commands below can just be copy-pasted. You don't need to re-type them!
    Note: It was reported that on some systems you have to add "magick" before the "mogrify" command.


    1. Install ImageMagick
    Get the latest version here: ImageMagick for Windows

    2. Download the DirectXTex tools
    Get the latest version here: Microsoft DirectXTex releases on GitHub
    You have to download »texconv.exe« and »texdiag.exe«.

    3. (Optional) Download @Lazza's "rF2_TexFix" program
    This tool replaces the manual sorting process as described in section 7, as well as the modification and conversion of the textures as described in section 8.
    Simply place the rF2_TexFix.exe in your root directory (see section 5)
    https://www.dropbox.com/s/qols3xqtnoqn4yd/rF2_TexFix.zip?dl=0

    4. (Optional) Install the Cmder command-line tool
    This step is optional because you can just use the Windows command shell (cmd). Cmder is for the power users (installation is manual, Cmder provides some more/better features).
    Get the latest version here: Cmder releases on GitHub
    Follow the installation instructions here.

    5. Prepare folders and tools
    Choose a directory where you want to do the texture conversions and edits. Create the following (sub-)directories there:
    Code:
    \dds-source
    \dds-source\dxt1
    \dds-source\dxt5
    \png-dxt1
    \png-dxt5
    \output
    Now copy the DirectXTex tools »texconv.exe« and »texdiag.exe« into the root directory.

    (Info) You may chose your own directory names, but then you have to modify the commands below to suit this.

    6. Open a command shell in the root folder
    Open a new Windows command shell (cmd) in the root folder by press and hold shift key and rightclick somewhere in a empty area. From the menu select »Open command window here«.

    (Optional) If you use Cmder and have installed it properly, you can do a rightclick somewhere in the folder and select »Cmder here«.

    7. Copy your source DDS files and sort them according to their format
    (Info) You can skip this step, if you already know the DDS formats. Just move the files in the respective »dxt*« sub-directory of »dds-source«. Sorting is necessary because after conversion the source DDS to PNG files the DDS format information is lost and a conversion back to DDS will automatically convert the PNG to DXT5 DDS files.

    (Info) Please exclude any cubemaps, bumpmaps or specmaps from the source files as the don't need to be modified.


    Copy all your source DDS files in the »source« folder. Run the following command to gather the format information:
    Code:
    texdiag info dds-source\*.dds > output.txt && START output.txt
    This command creates an »output.txt« file in the root directory which includes all basic information of the DDS files in the »source« directory. The »output.txt« will be directly opened with the Editor (it's the Windows default, of course you may use another program). You can avoid this by only using the first part of this command (until »&&«).
    Screen the »output.txt« for BC1 (a.k.a. DXT1) and BC3 (a.k.a. DXT5). Move the files into the respective sub-directory in your source directory (DXT1 to »dxt1« and »DXT5 to »dxt5«).

    (Info) Normally the files in DXT1 format are the minority, so moving these first might be the fastest way to sort the files.

    8. Convert, modify and convert back with ImageMagick and texconv
    (Info) ImageMagick gives the best lossless results for PNG conversion, so this is the preferred method. Texconv can do the conversion too, but the results are compressed too much which will result in even worse DDS later. Therefore I don't include the alternative commands here, but if you are interested have a look into the DirectXTex Wiki.
    Run the following command:
    Code:
    mogrify -format png -define png:compression-level=0 -define png:compression-filter=5 -define png:compression-strategy=2 -depth 8 -channel RGB +level 7.8%,78.4% -path png-dxt1\ dds-source\dxt1\*.dds >> log.txt && mogrify -format png -define png:compression-level=0 -define png:compression-filter=5 -define png:compression-strategy=2 -depth 8 -channel RGB +level 7.8%,78.4% -path png-dxt5\ dds-source\dxt5\*.dds >> log.txt && texconv -nologo -timing -y -f DXT1 -o output png-dxt1\*.png >> log.txt && texconv -timing -nologo -y -f DXT5 -o output png-dxt5\*.png >> log.txt
    This command will convert the DDS files to PNG based on their source folders (i.e. format) and modify the color levels at the same time. The last part of the command then converts the modified PNG back to DDS and save them in the »output« directory. During all steps the console output gets written into a log.txt file (that's much easier than scrolling the console)

    Please note: In case you only have DXT1 or DXT5 textures the above command will fail. You need to use the following separated commands instead:

    For DXT1:
    Code:
    mogrify -format png -define png:compression-level=0 -define png:compression-filter=5 -define png:compression-strategy=2 -depth 8 -channel RGB +level 7.8%,86.3% -path png-dxt1\ dds-source\dxt1\*.dds >> %FileDateTime%_log.txt && texconv -nologo -timing -y -f DXT1 -o output png-dxt1\*.png >> %FileDateTime%_log.txt
    For DXT5:
    Code:
    mogrify -format png -define png:compression-level=0 -define png:compression-filter=5 -define png:compression-strategy=2 -depth 8 -channel RGB +level 7.8%,86.3% -path png-dxt5\ dds-source\dxt5\*.dds >> %FileDateTime%_log.txt && texconv -nologo -timing -y -f DXT5 -o output png-dxt5\*.png >> %FileDateTime%_log.txt
    Now you are done and that just took (without the installation of the tools) just a couple of minutes instead of hours or days! Awesome, right?

    Tutorial for Photoshop (by @KittX)
    (Info) This tutorial uses the macro-capabilities of Photoshop. Follow these simple steps to configure your macro to batch-edit DDS files. You have to record one macro for DXT1 files and one for DXT5. You can follow the steps 4 to 6 from the tutorial above to sort your files.

    1. Open one of the files to edit in Photoshop, go to Window-Actions, and you have actions tab near the history tab.
    2. Press "Create new action" (sheet icon), name it appropriately
    3. Press record. Now every action you make is recorded. But it can be then removed and re-recorded if needed.
    4. Apply color levels , and save it in the needed dds format*. It will record the dds saving settings, so you can make 2 sets of actions - for dxt1 and dxt5 for example.
    5. Once you've saved, press stop and your action is ready to use.
    6. Go to File > Automate > Batch, choose your action, choose folder and it'll convert your files.
    *But make sure you revert the changes to the file you've opened for recording actions, and save it before going batch process, or make sure it is in another folder. Otherwise it will open that file again, and will apply levels again.

    ========================
    Please note: DDS is a compressed format. Open a DDS, edit it and save it again will unfortunately result in a loss of quality. Make sure to not do more than maybe one or too editing steps. Better solution is to go with original files (e.g. with skins or textures in their PSD format).

    Let me know your comments and report any problems/difficulties. And maybe someone is willing to improve this to a full tool (batch file, etc.)!? All programs are open source, so even redistribution in your own program would be allowed ;)

    Credits:
    Thanks @Marcel Offermans for (accidentally) reminding me of ImageMagick's great features and possibilities!
    Thanks to @KittX to add how you could achieve batch-editing with Photoshop!
    Thanks to @Lazza for providing the little program to translate the manual process into a automatic one!!
     
    Last edited: Aug 11, 2018
  2. McFlex

    McFlex Registered

    Joined:
    Feb 23, 2012
    Messages:
    733
    Likes Received:
    173
    OMG you are freaking awesome :cool::cool::cool: Sorting DXT1 and DXT5 textures is such a pain. Thank you very very very much.

    @Christopher Elliott Please make this thread sticky
     
    FuNK! likes this.
  3. FuNK!

    FuNK! Registered

    Joined:
    May 26, 2013
    Messages:
    475
    Likes Received:
    276
    You are very welcome! ;)

    In my opinion tutorials should have their own subforum or tag/label :oops:
     
    Mauro likes this.
  4. lagg

    lagg Registered

    Joined:
    Oct 1, 2012
    Messages:
    1,215
    Likes Received:
    829
    Thank you very much @FuNK!
    This is very useful
    Is it possible that there is an error in this command?
    Code:
    texdiag info source\*.dds > output.txt && START output.txt
    Is it possible that it is
    Code:
    texdiag info dds-source\*.dds > output.txt && START output.txt
     
    Mauro and FuNK! like this.
  5. FuNK!

    FuNK! Registered

    Joined:
    May 26, 2013
    Messages:
    475
    Likes Received:
    276
    Good eyes! Thanks, I fixed it ;)
     
    lagg likes this.
  6. ApainPlost

    ApainPlost Member

    Joined:
    May 6, 2016
    Messages:
    83
    Likes Received:
    121
    Thanks for your work and your tuto @FuNK!

    What about BC2 and B8G8R8A8 Format ?

    When i run the following command "mogrify -format png -define png:compression-level ...." ... the CMD return "unknown command ...."
     
    Last edited: Jul 17, 2018
  7. FuNK!

    FuNK! Registered

    Joined:
    May 26, 2013
    Messages:
    475
    Likes Received:
    276
    I know it's maybe a stupid question but did you install ImageMagick properly?
    If so, please try the first part of the command only (until the &&). If you still get that error please add the command “magick“ before mogrify.
     
    ApainPlost likes this.
  8. ApainPlost

    ApainPlost Member

    Joined:
    May 6, 2016
    Messages:
    83
    Likes Received:
    121
    Ok Thanks ... I have made a mistake. It work now.

    When I run the command ... it was in the wrong folder :p
     
    Last edited: Jul 18, 2018
    FuNK! likes this.
  9. FuNK!

    FuNK! Registered

    Joined:
    May 26, 2013
    Messages:
    475
    Likes Received:
    276
    Cool! What was the mistake?
     
  10. RoWo

    RoWo Registered

    Joined:
    Jul 16, 2014
    Messages:
    283
    Likes Received:
    342
    Where (in what folder) should I run this command?
     
  11. RoWo

    RoWo Registered

    Joined:
    Jul 16, 2014
    Messages:
    283
    Likes Received:
    342
    Ok, I got it.
     
  12. KittX

    KittX Registered

    Joined:
    Jan 11, 2012
    Messages:
    147
    Likes Received:
    60
    There's another way, if you're using Photoshop you can record the actions and then do batch convert within the program, so no installing other tools required.
     
    Alex72 likes this.
  13. FuNK!

    FuNK! Registered

    Joined:
    May 26, 2013
    Messages:
    475
    Likes Received:
    276
    Of course you can, but don't you need Photoshop for that? :D
    If you like, this tut helps those who don't have PS or don't want to spend money on it (like I do). Anyway, if you would like to write a short tut for the PS way, I would still welcome it (more options is allways a good thing) ;)
     
  14. KittX

    KittX Registered

    Joined:
    Jan 11, 2012
    Messages:
    147
    Likes Received:
    60
    Yup, my post was just to make sure Photoshop people are aware that it can be done within the prorgam.
    Here ya go:

    • open one of the files to edit, go to Window-Actions, and you have actions tab near the history tab.
    • Press "Create new action" (sheet icon), name it appropriately, and
    • Press record. Now every action you make is recorded. But it can be then removed and re-recorded if needed.
    • Apply levels, and save it in the needed dds format. It will record the dds saving settings, so you can make 2 sets of actions - for dxt1 and dxt5 for example.
    • Once you've saved, press stop and your action is ready to use.
    • Go to file-automate-batch, choose your action, choose folder and it'll convert your files.

    *But make sure you revert the changes to the file you've opened for recording actions, and save it before going batch process, or make sure it is in another folder. Otherwise it will open that file again, and will apply levels again.
     
    Last edited: Jul 18, 2018
    Alex72, FuNK!, MaD_King and 1 other person like this.
  15. FuNK!

    FuNK! Registered

    Joined:
    May 26, 2013
    Messages:
    475
    Likes Received:
    276
    @KittX I've added your tutorial below mine in the first post. Thank you!
     
    Alex72 likes this.
  16. Lazza

    Lazza Registered

    Joined:
    Oct 5, 2010
    Messages:
    7,006
    Likes Received:
    2,806
    I've done the same with PS (a version of it was free at one point - not sure if it still is) but this external tools method is attractive as it obviously doesn't need PS, plus it's more open to automation. Are there exotic formats to worry about? I've done some texture fixing (albedo) and needed to allow for DXT1, DXT3, and DXT5. Could there also be more?

    It should be possible to automate the separating of the different formats, making the exclusion of spec and bump maps the most manual (annoying) part, but would need to be sure of the various formats first.
     
  17. FuNK!

    FuNK! Registered

    Joined:
    May 26, 2013
    Messages:
    475
    Likes Received:
    276
    You probably mean the CS2. I don't want to go off topic, so I only say that it wasn't and will never be free at all - it's a violation of copyright if you use it without having purchased the license. That's no option for me (and many others). Btw. with GIMP and it's Script-Fu you would be able to write macros too. This again is an argument pro free software again.

    I believe it's possible to sort the files based on their format automatically based on the output of texdiag.exe. At least it generates an output so with some programming skills you should be able to send the textures one by one to the tool, read out if its DXT1, DXT3 or DXT5, and "progress is" (move to a folder or convert/modify it with ImageMagick and convert it back to DDS with Texconv.exe). Unfortunately I cannot do that because my skills aren't sufficient. I tried to write a command script (a.k.a. batch file *.BAT or *.CMD) but failed (e.g. the parameters for mogrify were not handed over properly).

    Exotic formats. Yeah, that DXT3 is almost one. AFAIK it was recommended to use DXT5 instead (you can sort the DXT3 to the DXT5 when processing them). Other formats are Cubemaps and Normalmaps because of their different code. Furthermore there are some DDS with special features which wouldn't load properly in rF2 (e.g. YCoCg DXT5 or RXGB DXT5 if I remember correctly). So in case you see something unusual in the output from Texdiag.exe which is not equal to DXT1 or DXT5, you should have a look what file it is and check if it really needs the fixing or if it needs to be converted to either DXT1 or DXT5.
     
  18. Lazza

    Lazza Registered

    Joined:
    Oct 5, 2010
    Messages:
    7,006
    Likes Received:
    2,806
    So I got around to trying this, it seems I need to run the command as "magick mogrify" rather than just "mogrify" - does this sound reasonable?

    For my use I'm throwing together a quick .exe to automatically generate the texdiag file, sort the source images into their respective folders, and then convert-manipulate-reconvert them all to the output folder. But I'm having to use the "magick mogrify" command so curious if that's just me...

    *Here's my .exe, it shortcuts the above process from Step 4; so just put it in a folder along with texdiag and texconv (and mogrify available), with a source\ folder with your DDS files in it, and it will do the rest. No need for extra folders, it'll create a working folder and an output folder.
    https://www.dropbox.com/s/qols3xqtnoqn4yd/rF2_TexFix.zip?dl=0
     
    Last edited: Jul 29, 2018
    FuNK! and Mauro like this.
  19. FuNK!

    FuNK! Registered

    Joined:
    May 26, 2013
    Messages:
    475
    Likes Received:
    276
    Yeah in some cases it seams you need to run the commands with magick. Normally all commands should be integrated in the system if you have just installed it with their setup files. However, I believe that some Firewall/Antivirus software may influence the normal behavior.

    Anyway, thank you very much for your effort! Would you mind sharing the code too? Maybe that could help others if they decide to build a software for the whole process?

    EDIT: I've added this tool into the first post + tutorial. Thanks!
     
    Last edited: Jul 29, 2018
  20. Lazza

    Lazza Registered

    Joined:
    Oct 5, 2010
    Messages:
    7,006
    Likes Received:
    2,806
    @FuNK! Yeah, antivirus makes sense probably. I wrote my prog to try just 'mogrify' first, then try what I had to use as a backup option.

    This is just a simple/ugly AutoIT script based on your commands.

    Code:
    #include <FileConstants.au3>
    #include <StringConstants.au3>
    #include <Array.au3>
    AutoItSetOption("MustDeclareVars", 1)
    Local $comDiag = "texdiag"
    Local $comConv = "texconv"
    Local $comMogrify = "mogrify"
    SplashTextOn("rF2_TexFix", "Gathering info...", Default, 100)
    Sleep(2000)
    RunWait(@ComSpec & " /c "&$comDiag&" >texinfo.txt")
    If (FileGetSize("texinfo.txt") < 100) then
        SplashOff()
        MsgBox(4096 + 16, "rF2_TexFix", "texdiag not available!")
        Exit
    endif
    RunWait(@ComSpec & " /c "&$comConv&" >texinfo.txt")
    If (FileGetSize("texinfo.txt") < 100) then
        SplashOff()
        MsgBox(4096 + 16, "rF2_TexFix", "texconv not available!")
        Exit
    endif
    RunWait(@ComSpec & " /c "&$comMogrify&" >texinfo.txt")
    If (FileGetSize("texinfo.txt") < 100) then
        $comMogrify = "magick mogrify"
        RunWait(@ComSpec & " /c "&$comMogrify&" >texinfo.txt")
        If (FileGetSize("texinfo.txt") < 100) then
            SplashOff()
            MsgBox(4096 + 16, "rF2_TexFix", "mogrify not available!")
            Exit
        EndIf
    endif
    
    Local $sSourceFolder = "source"
    Local $sWorkingFolder = "working"
    Local $sOutputFolder = "output"
    Local $sDirSep = "\"
    RunWait(@ComSpec & " /c " & $comDiag &" info source\*.dds > texinfo.txt")
    Local $hFileOpen = FileOpen("texinfo.txt", $FO_READ)
    if $hFileOpen = -1 Then
        Exit
    EndIf
    DirCreate($sWorkingFolder)
    DirCreate($sOutputFolder)
    Local $sLine
    Local $sCurFilename, $sCurFormat
    Local $Abort = false
    Local $asFormatList[1] ; array storing names of formats/folders. We're starting with size of 1 and will always have a 'blank' at the end
    While (Not $Abort)
        $sline = FileReadLine($hFileOpen)
        if (@error <> 0) then
            $Abort = true
        Else
            $sline = StringStripWS($sline, $STR_STRIPLEADING + $STR_STRIPTRAILING )
            if (StringLeft($sLine,StringLen($sSourceFolder)+1) = $sSourceFolder & $sDirSep AND StringRight($sLine,4) = ".dds") Then
                $sCurFilename = StringMid($sLine,StringLen($sSourceFolder)+2)
            EndIf
            if (StringLeft($sLine,9) = "format = ") Then
                $sCurFormat = StringMid($sline, 10)
                if (_ArraySearch($asFormatList, $sCurFormat) = -1) Then ; haven't had this format before, let's add it
                    ReDim $asFormatList[UBound($asFormatList) + 1]
                    $asFormatList[UBound($asFormatList)-2] = $sCurFormat
                    DirCreate($sWorkingFolder & $sDirSep & $sCurFormat)
                EndIf
                FileCopy($sSourceFolder & $sDirSep & $sCurFilename, $sWorkingFolder & $sDirSep & $sCurFormat)
            EndIf
        EndIf
    WEnd
    FileClose($hFileOpen)
    ; we now have all the source files in their subfolders in our working folder, and an array with the list of formats - so now we can set up the command to fix and convert them all
    Local $sCommand = @ComSpec & " /c " ; set up the command
    for $i = 0 to (UBound($asFormatList) - 2)
        ; now add the png conversion command for this format
        $sCommand &= $comMogrify&" -format png -define png:compression-level=0 -define png:compression-filter=5 -define png:compression-strategy=2 -depth 8 -channel RGB +level 7.8%,78.4% -path "&$sWorkingFolder&$sDirSep&$asFormatList[$i]&$sDirSep&" "&$sWorkingFolder&$sDirSep&$asFormatList[$i]&$sDirSep&"*.dds >> log.txt && "
        ; note we're adding && at the end of each command here
    Next
    for $i = 0 to (UBound($asFormatList) - 2)
        ; now to convert back to dds files in the output folder
        $sCommand &= $comConv&" -nologo -timing -y -f "&$asFormatList[$i]&" -o "&$sOutputFolder&" "&$sWorkingFolder&$sDirSep&$asFormatList[$i]&$sDirSep&"*.png >> log.txt"
        if $i < (UBound($asFormatList) - 2) Then
            $sCommand &= " && " ; not the final command, so get ready to add another next loop
        EndIf
    Next
    SplashTextOn("rF2_TexFix", "Running commands...", Default, 100)
    ; now we can run the command
    RunWait($sCommand)
    SplashOff()
    Local $res = MsgBox(4096 + 4, Default, "Finished. Would you like to remove all temp files/folders?")
    if $res = 6 Then
        ; user wants to remove all the temp stuff
        DirRemove($sWorkingFolder,1)
    EndIf
    
    
     
    FuNK! likes this.

Share This Page