Restore "assets" table population and documentation tweaks (#38)

The AssetBundleHandler got lost in recent code change, so restore it.

Add a few cross links to the new Addressables Build Layout parsing.

Fix the section "Matching content back to the source asset" because it wasn't explaining that some source asset paths actually are available in the build output (for AssetBundle explicit assets, and the scenes in a Player Build)

New example showing how to find implicitly referenced assets.

Author: SkowronskiAndrew

Analyzer/README.md

@@ -186,9 +186,19 @@ important types like Meshes, Shaders, Texture2D and AnimationClips.  For example
 File, then rows are added to both the `objects` table and the `meshes` table.  The meshes table contains columns that only apply to Mesh objects, for example the number of vertices, indices, bones, and channels.  The `mesh_view` is a view that joins the `objects` table with the `meshes` table, so that you can see all the properties of a Mesh object in one place.
 
 Each supported Unity object type follows the same pattern:
-* A Handler class in the SQLite/Handlers (e.g. [MeshHandler.cs](./SQLite/Handler/MeshHandler.cs).
-* The registration of the handler in the m_Handlers dictionary in [SQLiteWriter.cs](./SQLite/SQLiteWriter.cs).
+* A Handler class in the SQLite/Handlers, e.g. [MeshHandler.cs](./SQLite/Handler/MeshHandler.cs).
+* The registration of the handler in the m_Handlers dictionary in [SerializedFileSQLiteWriter.cs](./SQLite/Writers/SerializedFileSQLiteWriter.cs).
 * SQL statements defining extra tables and views associated with the type, e.g. [Mesh.sql](./SQLite/Resources/Mesh.sql).
-* A Reader class that uses RandomAccessReader to read properties from the serialized object. e.g. [Mesh.cs](./SerializedObjects/Mesh.cs).
+* A Reader class that uses RandomAccessReader to read properties from the serialized object, e.g. [Mesh.cs](./SerializedObjects/Mesh.cs).
 
 It would be possible to extend the Analyze library to add additional columns for the existing types, or by following the same pattern to add additional types.  The [dump](../UnityDataTool/README.md#dump) feature of UnityDataTool is a useful way to see the property names and other details of the serialization for a type.  Based on that information, code in the Reader class can use the RandomAccessReader to retrieve those properties to bring them into the SQLite database.
+
+## Supporting Other File Formats
+
+Another direction of possible extension is to support analyzing additional file formats, beyond Unity SerializedFiles.
+
+This the approach taken to analyze Addressables Build Layout files, which are JSON files using the format defined in [BuildLayout.cs](./SQLite/Parsers/Models/BuildLayout.cs).
+
+Support for another file format could be added by deriving an additional class from SQLiteWriter and implementing a class derived from ISQLiteFileParser.  Then follow the existing code structure convention to add new Commands (derived from AbstractCommand) and Resource .sql files to establish additional tables in the database.
+
+An example of another file format that could be useful to support, as the tool evolves, are the yaml [.manifest files](https://docs.unity3d.com/Manual/assetbundles-file-format.html), generated by BuildPipeline.BuildAssetBundles().

Analyzer/SQLite/Handlers/AssetBundleHandler.cs

@@ -0,0 +1,96 @@
+using System;
+using System.Text.RegularExpressions;
+using Microsoft.Data.Sqlite;
+using UnityDataTools.Analyzer.SerializedObjects;
+using UnityDataTools.FileSystem.TypeTreeReaders;
+
+namespace UnityDataTools.Analyzer.SQLite.Handlers;
+
+public class AssetBundleHandler : ISQLiteHandler
+{
+    SqliteCommand m_InsertCommand;
+    private SqliteCommand m_InsertDepCommand;
+    private Regex m_SceneNameRegex = new Regex(@"([^//]+)\.unity");
+
+    public void Init(SqliteConnection db)
+    {
+        using var command = db.CreateCommand();
+        command.CommandText = Resources.AssetBundle;
+        command.ExecuteNonQuery();
+
+        m_InsertCommand = db.CreateCommand();
+
+        m_InsertCommand.CommandText = "INSERT INTO assets(object, name) VALUES(@object, @name)";
+        m_InsertCommand.Parameters.Add("@object", SqliteType.Integer);
+        m_InsertCommand.Parameters.Add("@name", SqliteType.Text);
+
+        m_InsertDepCommand = db.CreateCommand();
+
+        m_InsertDepCommand.CommandText = "INSERT INTO asset_dependencies(object, dependency) VALUES(@object, @dependency)";
+        m_InsertDepCommand.Parameters.Add("@object", SqliteType.Integer);
+        m_InsertDepCommand.Parameters.Add("@dependency", SqliteType.Integer);
+    }
+
+    public void Process(Context ctx, long objectId, RandomAccessReader reader, out string name, out long streamDataSize)
+    {
+        var assetBundle = AssetBundle.Read(reader);
+
+        foreach (var asset in assetBundle.Assets)
+        {
+            if (!assetBundle.IsSceneAssetBundle)
+            {
+                var fileId = ctx.LocalToDbFileId[asset.PPtr.FileId];
+                var objId = ctx.ObjectIdProvider.GetId((fileId, asset.PPtr.PathId));
+                m_InsertCommand.Transaction = ctx.Transaction;
+                m_InsertCommand.Parameters["@object"].Value = objId;
+                m_InsertCommand.Parameters["@name"].Value = asset.Name;
+                m_InsertCommand.ExecuteNonQuery();
+
+                for (int i = asset.PreloadIndex; i < asset.PreloadIndex + asset.PreloadSize; ++i)
+                {
+                    var dependency = assetBundle.PreloadTable[i];
+                    var depFileId = ctx.LocalToDbFileId[dependency.FileId];
+                    var depId = ctx.ObjectIdProvider.GetId((depFileId, dependency.PathId));
+                    m_InsertDepCommand.Transaction = ctx.Transaction;
+                    m_InsertDepCommand.Parameters["@object"].Value = objId;
+                    m_InsertDepCommand.Parameters["@dependency"].Value = depId;
+                    m_InsertDepCommand.ExecuteNonQuery();
+                }
+            }
+            else
+            {
+                var match = m_SceneNameRegex.Match(asset.Name);
+
+                if (match.Success)
+                {
+                    var sceneName = match.Groups[1].Value;
+                    var objId = ctx.ObjectIdProvider.GetId((ctx.SerializedFileIdProvider.GetId(sceneName), 0));
+                    m_InsertCommand.Transaction = ctx.Transaction;
+                    m_InsertCommand.Parameters["@object"].Value = objId;
+                    m_InsertCommand.Parameters["@name"].Value = asset.Name;
+                    m_InsertCommand.ExecuteNonQuery();
+                }
+            }
+        }
+
+        name = assetBundle.Name;
+        streamDataSize = 0;
+    }
+
+    public void Finalize(SqliteConnection db)
+    {
+        using var command = new SqliteCommand();
+        command.Connection = db;
+        command.CommandText = "CREATE INDEX asset_dependencies_object ON asset_dependencies(object)";
+        command.ExecuteNonQuery();
+
+        command.CommandText = "CREATE INDEX asset_dependencies_dependency ON asset_dependencies(dependency)";
+        command.ExecuteNonQuery();
+    }
+
+    void IDisposable.Dispose()
+    {
+        m_InsertCommand?.Dispose();
+        m_InsertDepCommand?.Dispose();
+    }
+}

Analyzer/SQLite/Parsers/SerializedFileParser.cs

@@ -64,7 +64,7 @@ bool ShouldIgnoreFile(string file)
         private static readonly HashSet<string> IgnoredExtensions = new()
     {
         ".txt", ".resS", ".resource", ".json", ".dll", ".pdb", ".exe", ".manifest", ".entities", ".entityheader",
-        ".ini", ".config"
+        ".ini", ".config", ".hash"
     };
 
         bool ProcessFile(string file, string rootDirectory)

Analyzer/SQLite/Writers/SerializedFileSQLiteWriter.cs

@@ -35,6 +35,7 @@ public class SerializedFileSQLiteWriter : IDisposable
         { "Shader", new ShaderHandler() },
         { "AudioClip", new AudioClipHandler() },
         { "AnimationClip", new AnimationClipHandler() },
+        { "AssetBundle", new AssetBundleHandler() },
         { "PreloadData", new PreloadDataHandler() },
     };
 

Documentation/addressables-build-reports.md

@@ -114,6 +114,19 @@ You can analyze a directory with both asset bundles (*.bundle) and json files (*
 
 Once the data is in the database, you can run queries to analyze your Addressables build:
 
+
+#### Find all implicit assets for an explicit asset
+```sql
+-- Find implicitly included assets for a given explicit asset id
+SELECT a.explicit_asset_id, b.id, b.asset_path, b.asset_path 
+  FROM addressables_build_explicit_asset_internal_referenced_other_assets a, 
+       addressables_build_data_from_other_assets b
+ WHERE a.internal_referenced_other_asset_rid = b.id 
+   AND a.build_id = b.build_id;
+   AND a.explicit_asset_id = 5092 
+   AND a.build_id = 3;
+```
+
 #### Find the cache name for an addressables bundle
 Addressables renames bundles to make it possible to do content updates. Internally bundles are still named by their internal hash and are cached based upon this name. If you want to lookup how a remote bundle will be cached in Unity's [cache](https://docs.unity3d.com/ScriptReference/Caching.html) you can use the addressables_build_cached_bundles view.
 ```sql
@@ -165,4 +178,4 @@ group by a.id;
 SELECT name, start_time, duration, error 
 FROM addressables_builds 
 WHERE coalesce(error, '') != '';
-```
+```

Documentation/analyze-examples.md

@@ -220,9 +220,13 @@ This is a large subject, see [Comparing Builds](comparing-builds.md).
 
 ## Example: Matching content back to the source asset
 
-UnityDataTool works on the output of a Unity build, which, by its very nature, only contains the crucial data needed to efficiently load built content in the Player.  So it does not include any information about the assets and scenes in the project that was used to create that build.  However you may want to match content back to the original source asset or scene.  For example if the size of an AssetBundle has unexpectedly changed between builds then you may want to track down which source assets could be responsible for that change.  Or you may want to confirm that some particular image has been included in the build.
+UnityDataTool works on the output of a Unity build, which, by its very nature, only contains the crucial data needed to efficiently load built content in the Player.  It does not include complete information about the assets and scenes in the project that was used to create that build.  You may want to match content back to the original source asset or scene.  For example if the size of an AssetBundle has unexpectedly changed between builds then you may want to track down which source assets could be responsible for that change.  Or you may want to confirm that some particular image has been included in the build.
 
-In many cases the source asset can be inferred based on your specific knowledge of your project, and how the build was configured.  For example the level files in a Player build match the Scenes in the Build Profile Scene list.  And the content of AssetBundles is driven from the assignment of specific assets to those AssetBundles (or Addressable groups).
+For AssetBundles partial asset information can be found in the m_Containers list, inside the AssetBundle object.  This records assets that were explicitly added to AssetBundles.  In the database this can be found in the `assets` table.  However, assets that are included in the build implicitly (because they are referenced from the explicitly added assets) will not be recorded anywhere in the AssetBundle content. 
+
+Similarly for a player build the only paths populated in the `assets` table are the scenes from the Build Profile Scene List.  The paths of the assets in the sharedAsset files is not recorded anywhere in the build output.
+
+In many cases the source asset can be inferred based on your specific knowledge of your project, and how the build was configured.  For example the level files in a Player build match the Scenes in the Build Profile Scene list.  And the content of AssetBundles is driven from the assignment of specific assets to those AssetBundles (or Addressable groups), along with assets they depend on.
 
 Also, in many cases the name of objects matches the file name of the asset.  For example the Texture2D "red" object probably comes from a file named red.png somewhere in the project.  
 
@@ -235,5 +239,4 @@ Examples of alternative sources of build information:
 * The [BuildReport](https://docs.unity3d.com/ScriptReference/Build.Reporting.BuildReport.html) has detailed source information in the PackedAssets section.  The [BuildReportInspector](https://github.com/Unity-Technologies/BuildReportInspector) is a useful way to view data from the BuildReport.
 * The Editor log reports a lot of information during a build. 
 * Regular AssetBundle builds create [.manifest files](https://docs.unity3d.com/Manual/assetbundles-file-format.html), which contain information about the source assets and types.
-* Addressable builds do not produce BuildReport files, nor .manifest files. But there is similar reporting, for example the [Build Layout Report](https://docs.unity3d.com/Packages/[email protected]/manual/BuildLayoutReport.html).
-
+* Addressable builds do not produce BuildReport files, nor .manifest files. But UnityDataTools supports analyzing the [Addressables Build Reports](addressables-build-reports.md) and will populate the `addressables_build_explicit_assets` and `addressables_build_data_from_other_assets` tables.