Merge the collectible work into the triumph branch

Brings collectible pickups together with the triumph and lore book work they
depend on. A pickup identifies its activity, resolves the book that activity
feeds, and marks the next chapter the account does not hold; the triumph side
supplies the claim store, the score totalling and the progress bars that make
that visible.

Conflicts were resolved by taking whichever side was a superset, which for the
shared domains was the collectible branch: records and claims were copied from
here earlier today and extended with the lore row join and the persistence.
Three files needed merging by hand. The account encoder keeps both sides: the
table publishing a warm start needs, then claims, book gates, Triumph Score and
node progress in that order, since the overlays depend on the tables being
published first. The character encoder keeps the triumph version, which already
had the publish retry the collectible one lacked. The project file was rebuilt
from the tree rather than resolved line by line.

The merge duplicated 26 table constants and two storage members, since those had
been copied across by hand before the branches met. Duplicates are removed
rather than left to shadow each other.

.gitignore

@@ -13,3 +13,4 @@
 
 # CMake build files
 /build/
+/out/

README.md

@@ -19,7 +19,7 @@ Destiny 2 Offline Exploration Mod
 
 ## WIP
 
-This mod is work in progress. Things might break or work in unexpected ways. There is also currently
+This mod is a work in progress. Things might break or work in unexpected ways. There is also currently
 a lack of documentation. This will improve over the coming weeks.
 
 ## Support Me
@@ -89,14 +89,14 @@ Pull Requests are welcome. Please follow these rules:
 
 - **No Copyrighted Data** - All game data should be extracted at runtime.
 - **Code Formatting** - Stick to the provided clang-format and clang-tidy configs.
-- **Clean Code** - Try to post readable high quality code, follow the projects existing style of
+- **Clean Code** - Try to post readable high-quality code, follow the project's existing style of
   comment and add docs.
 - **Provide Documentation** - Please explain what you changed, why you changed it and the effects it
   has in detail, it saves me a lot of work.
 - **Follow Up** - If something with the PR is not right, I will reply and ask you to fix it.
 - **One Feature** - Do not put multiple features into one PR.
-- **Complete Implementations** - Do not PR features that are not completed and/or have non functional parts.
-- **Server Focus** - For features that are intended to be part of the server, don't abuse client patches. Sometimes its needed but mostly everything should go through the right requests and pushes.
+- **Complete Implementations** - Do not PR features that are not completed and/or have non-functional parts.
+- **Server Focus** - For features that are intended to be part of the server, don't abuse client patches. Sometimes it's needed but mostly everything should go through the right requests and pushes.
 
 ## Credits
 

Sunrise/Sunrise.vcxproj

@@ -831,6 +831,10 @@
     <ClCompile Include="src\state\build_data\records\record_build_data_runtime.cpp" />
     <ClCompile Include="src\state\build_data\records\record_catalog.cpp" />
     <ClCompile Include="src\state\record_claims\record_claims.cpp" />
+    <ClCompile Include="src\state\activity\current_activity.cpp" />
+    <ClCompile Include="src\state\build_data\records\record_persistence.cpp" />
+    <ClCompile Include="src\state\build_data\sobjects\sobject_catalog.cpp" />
+    <ClCompile Include="src\state\lore\lore_grant.cpp" />
   </ItemGroup>
   <ItemGroup Condition="'$(SunriseRunClangTidy)'=='true'">
     <ClCompile Remove="vendor\detours\detours.cpp" />
@@ -1443,6 +1447,10 @@
     <ClInclude Include="src\state\build_data\records\definition.h" />
     <ClInclude Include="src\state\build_data\records\record_catalog.h" />
     <ClInclude Include="src\state\record_claims\record_claims.h" />
+    <ClInclude Include="src\state\activity\current_activity.h" />
+    <ClInclude Include="src\state\build_data\records\record_persistence.h" />
+    <ClInclude Include="src\state\build_data\sobjects\sobject_catalog.h" />
+    <ClInclude Include="src\state\lore\lore_grant.h" />
   </ItemGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
 </Project>

Sunrise/src/client/content/items/packages/internal.h

@@ -55,6 +55,14 @@ inline constexpr std::size_t kContainerCandidates = 16;
 /** Lock-owned storage kept off the caller stack, shared by every stage of the pass. */
 struct Storage {
     reader::Scratch scratch{};
+    /** Node rows held until the value slot and owned records are resolved. */
+    std::array<state::build_data::nodes::Definition,
+               state::build_data::nodes::kDefinitionCapacity>
+        nodeRows{};
+    /** Record rows held until the completion flag mapping is resolved. */
+    std::array<state::build_data::records::Definition,
+               state::build_data::records::kDefinitionCapacity>
+        recordRows{};
     std::vector<std::byte> container{};
     std::vector<std::byte> child{};
     std::vector<std::byte> root{};
@@ -84,16 +92,7 @@ struct Storage {
         entryBucketRows{};
     std::array<state::build_data::progressions::Definition,
                state::build_data::progressions::kDefinitionCapacity>
-        progressionRows{};
-    /** Record rows held until the completion flag mapping is resolved. */
-    std::array<state::build_data::records::Definition,
-               state::build_data::records::kDefinitionCapacity>
-        recordRows{};
-    /** Node rows held until the value slot and owned records are resolved. */
-    std::array<state::build_data::nodes::Definition,
-               state::build_data::nodes::kDefinitionCapacity>
-        nodeRows{};
-    std::array<state::build_data::collectibles::Definition,
+        progressionRows{};    std::array<state::build_data::collectibles::Definition,
                state::build_data::collectibles::kDefinitionCapacity>
         collectibleRows{};
     std::array<state::build_data::material_requirements::Definition,
@@ -395,6 +394,16 @@ void report(std::size_t published, const char* reason) noexcept;
  * @param itemDefinitionCount Number of rows in the installed item index table.
  * @return True when every tag, class, bound, and item link validates and publishes.
  */
+/**
+ * Reads the records table and resolves each record's completion flag to a bank index.
+ * @param source Package source.
+ * @param scratch Reader scratch.
+ * @param root Investment root bytes.
+ * @param blob Scratch storage for the tables.
+ * @param output Row storage in native record order.
+ * @param count Receives the number of rows read.
+ * @return True when both tables read and every row fits.
+ */
 [[nodiscard]] bool build_collectibles(const reader::Source& source,
                                       Storage& storage,
                                       std::span<const std::byte> root,

Sunrise/src/client/content/items/packages/package_collectible_build.cpp

@@ -1,3 +1,7 @@
+#include <array>
+#include <cstdio>
+#include "../../../../core/logging/log.h"
+#include "../../../../state/build_data/sobjects/sobject_catalog.h"
 #include <cstring>
 #include <limits>
 
@@ -12,6 +16,42 @@ bool build_collectibles(const reader::Source& source,
                         std::span<const std::byte> root,
                         std::uint64_t itemDefinitionCount) noexcept {
     namespace domain = state::build_data::collectibles;
+
+    // The definition table an incident target names. Read here because this pass already holds an
+    // open source, and because a collectible picked up in the world arrives as an incident: without
+    // this table its target is a bare number.
+    if (state::build_data::sobjects::count() == 0) {
+        namespace sobjects = state::build_data::sobjects;
+        // Geometry is documented and asserted rather than divided out: count at +112, rows at +128,
+        // forty bytes each, name hash at +0, selector group at +32, ordinal at +34, type at +36.
+        constexpr std::size_t kCountOffset = 112;
+        constexpr std::size_t kRowBase = 128;
+        constexpr std::size_t kRowStride = 40;
+        for (const std::uint32_t tag : {0x81327CD4U, 0x80B9E5BFU}) {
+            std::vector<std::byte> blob{};
+            if (!reader::read_tag(source, storage.scratch, tag, blob)
+                || blob.size() < kRowBase) {
+                continue;
+            }
+            std::uint64_t rowCount = 0;
+            std::memcpy(&rowCount, blob.data() + kCountOffset, sizeof rowCount);
+            if (rowCount == 0 || rowCount > sobjects::kDefinitionCapacity
+                || kRowBase + static_cast<std::size_t>(rowCount) * kRowStride > blob.size()) {
+                continue;
+            }
+            std::vector<sobjects::Definition> rows(static_cast<std::size_t>(rowCount));
+            for (std::size_t row = 0; row < rows.size(); ++row) {
+                const std::size_t at = kRowBase + row * kRowStride;
+                std::memcpy(&rows[row].nameHash, blob.data() + at, sizeof(std::uint32_t));
+                std::memcpy(&rows[row].selectorGroup, blob.data() + at + 32, sizeof(std::uint16_t));
+                std::memcpy(&rows[row].nodeOrdinal, blob.data() + at + 34, sizeof(std::uint16_t));
+                std::memcpy(&rows[row].typeCode, blob.data() + at + 36, sizeof(std::int32_t));
+            }
+            (void)sobjects::replace(std::span<const sobjects::Definition>{rows});
+            break;
+        }
+    }
+
     if (state::build_data::collectible_definitions_ready()) {
         return true;
     }
@@ -87,6 +127,11 @@ bool build_collectibles(const reader::Source& source,
         output.collectibleHash = collectibleHash;
         output.collectibleIndex = static_cast<std::uint16_t>(row);
         output.itemDefinitionIndex = itemDefinitionIndex;
+        // The lore row this collectible unlocks. The record displaying the same row is the chapter
+        // it completes, so the two tables join on the row rather than on any hash. Read after the
+        // reset above, or it would be cleared.
+        std::memcpy(&output.loreRow, table.data() + at + tables::kLoreRowOffset,
+                    sizeof output.loreRow);
         if (requirementSetIndex == domain::kUnavailableMaterialRequirementSetIndex) {
             continue;
         }

Sunrise/src/client/content/items/packages/package_item_build.cpp

@@ -1,3 +1,4 @@
+#include "../../../../state/build_data/records/record_persistence.h"
 #include "../../../../state/build_data/nodes/node_persistence.h"
 #include <Windows.h>
 
@@ -166,7 +167,9 @@ bool build() noexcept {
                                 nodeCount)) {
                     if (state::build_data::publish_node_definitions(
                             std::span(storage.nodeRows).first(nodeCount))) {
-                        // Kept in its own file, so the next start does not need this pass at all.
+                        // Kept in its own file: the build data cache does not carry this domain, so
+                        // without it a warm start runs with no node table and nothing can be
+                        // granted.
                         (void)state::build_data::nodes::store(
                             std::span(storage.nodeRows).first(nodeCount));
                     }
@@ -180,8 +183,14 @@ bool build() noexcept {
                                   storage.child,
                                   storage.recordRows,
                                   recordCount)) {
-                    (void)state::build_data::publish_record_definitions(
-                        std::span(storage.recordRows).first(recordCount));
+                    if (state::build_data::publish_record_definitions(
+                            std::span(storage.recordRows).first(recordCount))) {
+                        // Kept beside the node table and for the same reason: the build data cache
+                        // does not carry this domain, so a warm start would have no records and
+                        // nothing could resolve a chapter.
+                        (void)state::build_data::records::store(
+                            std::span(storage.recordRows).first(recordCount));
+                    }
                 }
             }
             if (!state::build_data::investment_constants_ready()) {

Sunrise/src/client/content/items/packages/package_record_build.cpp

@@ -145,6 +145,9 @@ bool build_records(const reader::Source& source,
         domain::Definition& definition = output[static_cast<std::size_t>(row)];
         definition = {};
         definition.definitionIndex = static_cast<std::uint16_t>(row);
+        // The lore row this record displays, or 0xFFFF for a book's parent triumph.
+        std::memcpy(&definition.loreRow, blob.data() + at + tables::kLoreRowOffset,
+                    sizeof definition.loreRow);
         // The shipped table tops out at 500, so anything wider is not a score and is dropped.
         definition.scoreValue = score <= 0xFFFFU ? static_cast<std::uint16_t>(score) : 0U;
         std::int16_t categorySlot = 0;
@@ -162,6 +165,22 @@ bool build_records(const reader::Source& source,
         }
         ++count;
     }
+    // TEMPORARY: what score a lore record actually carries in this build. The published manifest
+    // shows ScoreValue 0 for Confessions Entry I, which does not match the game, so read the rows
+    // rather than trust either. 1707 is the book's parent triumph and 1708 to 1716 its chapters.
+    for (std::uint64_t row = 1707; row <= 1716 && row < count; ++row) {
+        std::array<char, 180> line{};
+        const int told = std::snprintf(
+            line.data(), line.size(), "ev=lorescore record=%llu score=%u lore_row=%u flag=%u",
+            static_cast<unsigned long long>(row), static_cast<unsigned>(output[row].scoreValue),
+            static_cast<unsigned>(output[row].loreRow),
+            static_cast<unsigned>(output[row].completionFlagIndex));
+        if (told > 0) {
+            core::log::write(core::log::Channel::client, core::log::Level::info,
+                             {line.data(), static_cast<std::size_t>(told)});
+        }
+    }
+
     report("ok", static_cast<unsigned long long>(count));
     return count != 0;
 }

Sunrise/src/middleware/content/packages/tables/definition_index_table.h

@@ -31,6 +31,78 @@ inline constexpr std::uint16_t kAbsentPackageId = 0xFFFFU;
 /** Element class of the item index table inside the investment container. */
 inline constexpr std::uint32_t kItemIndexTableClass = 0x80807BE8U;
 /** The investment root holds the installed collectible definition table at this slot. */
+/** Investment root slot of the unlock flag mapping tables. */
+/** The investment root holds the progression definition table at this slot. */
+/** Investment root slot of the records and lore table. */
+inline constexpr std::size_t kRecordTableSlot = 72;
+/** One record row, wider than any field this pass reads. */
+inline constexpr std::size_t kRecordRowStride = 216;
+/** Unlock slot of the record's completion flag, or a non-positive value when it has none. */
+inline constexpr std::size_t kRecordCompletionFlagOffset = 100;
+/**
+ * Lore row a record names, or 0xFFFF when it names none.
+ *
+ * A collectible row carries the same field at the same offset, which is what joins the two: a
+ * collectible and the record it unlocks name one lore row. A record with no lore is a book's parent
+ * triumph rather than one of its chapters.
+ */
+inline constexpr std::size_t kLoreRowOffset = 0x2C;
+/** Investment root slot of the lore table: 1425 rows of sixteen bytes. */
+inline constexpr std::size_t kLoreTableSlot = 52;
+/** One lore row, and the definition hash inside it. */
+inline constexpr std::size_t kLoreRowStride = 16;
+inline constexpr std::size_t kLoreHashOffset = 8;
+/** Points the record is worth. Zero for lore and for the interval records that score per step. */
+inline constexpr std::size_t kRecordScoreOffset = 92;
+/** A record names its category's value slot here. The record's own bar reads the next slot up. */
+inline constexpr std::size_t kRecordCategoryExpressionField = 120;
+/** Investment root slot of the four unlock value mapping tables. */
+inline constexpr std::size_t kUnlockValueMapTableSlot = 113;
+/** Array descriptor of the account object's value mapping table. */
+inline constexpr std::size_t kAccountValueMapDescriptor = 8;
+/** One unlock expression instruction: an opcode then its operand. */
+inline constexpr std::size_t kUnlockInstructionStride = 8;
+/** Opcodes run to fifteen; anything wider means the field is not an expression. */
+inline constexpr std::uint32_t kUnlockOpcodeCeiling = 20;
+/** The opcode that reads a value slot. */
+inline constexpr std::uint32_t kUnlockReadValueOpcode = 10;
+/** The opcode that tests a flag. */
+inline constexpr std::uint32_t kUnlockReadFlagOpcode = 1;
+/** Array payloads begin after a sixteen byte header. */
+inline constexpr std::size_t kHeaderSkip = 16;
+/**
+ * Upper bound on how many instructions a node expression may hold.
+ *
+ * This was thirty-two, which was a guess rather than a measurement, and it silently hid a real gate:
+ * one lore book carries fifty-nine instructions, so its expression was rejected before it was read
+ * and the category looked as though nothing gated it at all. The bound exists only to stop a wild
+ * count being walked as if it were an expression, so it is set well above anything observed.
+ */
+inline constexpr std::int64_t kNodeExpressionCapacity = 128;
+/** Investment root slot of the presentation node table. */
+inline constexpr std::size_t kPresentationNodeTableSlot = 63;
+/** One node row. Measured from the spacing of four known node hashes, not divided out of the blob. */
+inline constexpr std::size_t kNodeRowStride = 168;
+/** A node's expression sits at one of these two fields, never both. */
+inline constexpr std::size_t kNodeExpressionFieldPrimary = 64;
+inline constexpr std::size_t kNodeExpressionFieldAlternate = 48;
+/** Records a node owns, four bytes each as a row then a gate. */
+inline constexpr std::size_t kNodeChildRecordField = 136;
+inline constexpr std::size_t kNodeChildRecordStride = 4;
+/** A category's parent record reads the slot immediately above the category's own. */
+inline constexpr std::int32_t kNodeParentSlotStep = 1;
+/** Array descriptor of the character object's flag mapping table, sized to that bank. */
+inline constexpr std::size_t kCharacterFlagMapDescriptor = 40;
+/** Array descriptor of the character object's value mapping table, sized to that bank. */
+inline constexpr std::size_t kCharacterValueMapDescriptor = 24;
+inline constexpr std::size_t kUnlockFlagMapTableSlot = 111;
+/** Array descriptor of the account object's flag mapping table. */
+inline constexpr std::size_t kAccountFlagMapDescriptor = 8;
+/** One mapping row: a source and the slot it feeds. */
+inline constexpr std::size_t kUnlockMapRowStride = 8;
+/** The destination slot within one mapping row. */
+inline constexpr std::size_t kUnlockMapDestinationSlotOffset = 4;
+
 inline constexpr std::size_t kCollectibleTableSlot = 19;
 /** Definition class recorded for the installed investment root tag. */
 inline constexpr std::uint32_t kInvestmentRootClass = 0x80807D84U;
@@ -87,68 +159,6 @@ inline constexpr std::uint32_t kSandboxPerkClass = 0x808077BCU;
 inline constexpr std::size_t kStatBlockOffset = 112;
 /** The investment root holds the constants blob at this slot. */
 inline constexpr std::size_t kInvestmentConstantsSlot = 11;
-/** The investment root holds the progression definition table at this slot. */
-/** Investment root slot of the records and lore table. */
-inline constexpr std::size_t kRecordTableSlot = 72;
-/** One record row, wider than any field this pass reads. */
-inline constexpr std::size_t kRecordRowStride = 216;
-/** Unlock slot of the record's completion flag, or a non-positive value when it has none. */
-inline constexpr std::size_t kRecordCompletionFlagOffset = 100;
-/** Points the record is worth. Zero for lore and for the interval records that score per step. */
-inline constexpr std::size_t kRecordScoreOffset = 92;
-/** A record names its category's value slot here. The record's own bar reads the next slot up. */
-inline constexpr std::size_t kRecordCategoryExpressionField = 120;
-/** A category's parent record reads the slot immediately above the category's own. */
-inline constexpr std::int32_t kNodeParentSlotStep = 1;
-
-/** Investment root slot of the presentation node table. */
-inline constexpr std::size_t kPresentationNodeTableSlot = 63;
-/** One node row. Measured from the spacing of four known node hashes, not divided out of the blob. */
-inline constexpr std::size_t kNodeRowStride = 168;
-/** A node's expression sits at one of these two fields, never both. */
-inline constexpr std::size_t kNodeExpressionFieldPrimary = 64;
-inline constexpr std::size_t kNodeExpressionFieldAlternate = 48;
-/** Records a node owns, four bytes each as a row then a gate. */
-inline constexpr std::size_t kNodeChildRecordField = 136;
-inline constexpr std::size_t kNodeChildRecordStride = 4;
-/**
- * Upper bound on how many instructions a node expression may hold.
- *
- * This was thirty-two, which was a guess rather than a measurement, and it silently hid a real gate:
- * one lore book carries fifty-nine instructions, so its expression was rejected before it was read
- * and the category looked as though nothing gated it at all. The bound exists only to stop a wild
- * count being walked as if it were an expression, so it is set well above anything observed.
- */
-inline constexpr std::int64_t kNodeExpressionCapacity = 128;
-
-/** One unlock expression instruction: an opcode then its operand. */
-inline constexpr std::size_t kUnlockInstructionStride = 8;
-/** Opcodes run to fifteen; anything wider means the field is not an expression. */
-inline constexpr std::uint32_t kUnlockOpcodeCeiling = 20;
-/** The opcode that reads a value slot. */
-inline constexpr std::uint32_t kUnlockReadValueOpcode = 10;
-/** The opcode that tests a flag. */
-inline constexpr std::uint32_t kUnlockReadFlagOpcode = 1;
-/** Array payloads begin after a sixteen byte header. */
-inline constexpr std::size_t kHeaderSkip = 16;
-
-/** Investment root slot of the four unlock value mapping tables. */
-inline constexpr std::size_t kUnlockValueMapTableSlot = 113;
-/** Array descriptor of the account object's value mapping table. */
-inline constexpr std::size_t kAccountValueMapDescriptor = 8;
-/** Array descriptor of the character object's value mapping table, sized to that bank. */
-inline constexpr std::size_t kCharacterValueMapDescriptor = 24;
-
-/** Investment root slot of the five unlock flag mapping tables. */
-inline constexpr std::size_t kUnlockFlagMapTableSlot = 111;
-/** Array descriptor of the account object's flag mapping table, the bank flag runs author. */
-inline constexpr std::size_t kAccountFlagMapDescriptor = 8;
-/** Array descriptor of the character object's flag mapping table, sized to that bank. */
-inline constexpr std::size_t kCharacterFlagMapDescriptor = 40;
-/** One mapping row: the unlock hash, then the destination slot its object byte feeds. */
-inline constexpr std::size_t kUnlockMapRowStride = 8;
-/** Destination slot offset inside one mapping row. */
-inline constexpr std::size_t kUnlockMapDestinationSlotOffset = 4;
 
 inline constexpr std::size_t kProgressionTableSlot = 68;
 /** Element class of the progression definition table. */

Sunrise/src/middleware/datagen/family4/account/account_encoder.cpp

@@ -1,6 +1,15 @@
+#include "../../../../state/build_data/records/record_persistence.h"
+#include "../../../../state/build_data/records/record_catalog.h"
+#include "../../../../state/build_data/nodes/node_persistence.h"
 #include <atomic>
-
+#include "../../../../core/logging/log.h"
+#include "../../../../state/build_data/records/definition.h"
+#include <cstdio>
+#include <array>
+#include <span>
+#include <vector>
 #include "../../../../state/build_data/nodes/node_catalog.h"
+#include "../../../../state/record_claims/record_claims.h"
 #include "account_encoder.h"
 
 #include <algorithm>
@@ -8,8 +17,6 @@
 #include <limits>
 
 #include "../../../../state/build_data/runtime.h"
-#include "../../../../state/build_data/records/definition.h"
-#include "../../../../state/record_claims/record_claims.h"
 #include "../../../../state/unlocks/unlocks_runtime.h"
 #include "../progression/progression_bank_keys.h"
 #include "layout.h"
@@ -98,14 +105,32 @@ bool encode(const state::AccountState& state, std::span<std::byte> output) noexc
     // Acquired flags and objective progress are authored policy, published once per process.
     const state::unlocks::Table& unlocks = state::unlocks::get();
     object.acquiredFlags = unlocks.accountFlags;
-    // Claims arrive after boot, so they cannot be in the authored policy. Lay them over the
-    // bank here, which is the one place every Family-4 account image passes through.
-    (void)state::record_claims::apply(object.acquiredFlags);
-    // Lore book categories are gated on a flag they cannot set by being played: with no title shown
-    // there is nothing inside to claim, and nothing to claim leaves the gate shut.
-    (void)state::build_data::nodes::apply_visibility(object.acquiredFlags);
     object.profileUnlockFlags = unlocks.profileFlags;
     object.objectiveValues = unlocks.objectiveValues;
+    // The node and record tables are not in the build data cache, so on a warm start the package
+    // pass is skipped and both are empty. The account image needs them: without nodes no book gate
+    // is satisfied and every book reads as unnamed. Publishing here, latched on success, is what
+    // makes a warm start look like a cold one.
+    {
+        static std::atomic<bool> published{false};
+        if (!published.load(std::memory_order_relaxed)) {
+            const bool haveNodes = state::build_data::nodes::count() != 0
+                                   || state::build_data::nodes::load_and_publish();
+            const bool haveRecords = state::build_data::records::count() != 0
+                                     || state::build_data::records::load_and_publish();
+            if (haveNodes && haveRecords) {
+                published.store(true, std::memory_order_relaxed);
+            }
+        }
+    }
+
+    // Claims are laid over the authored bank on the way out, so a claimed record reads Acquired on
+    // the next image. The authored policy itself is immutable and is never edited.
+    (void)state::record_claims::apply(object.acquiredFlags);
+    // A lore book's category is gated: some read a flag, some test their own progress value. A book
+    // gated on progress cannot open by being played, since with no title shown there is nothing
+    // inside to collect. Satisfying the gate is what makes the book readable at all.
+    (void)state::build_data::nodes::apply_visibility(object.acquiredFlags);
     // Triumph Score is a plain replicated value the client never derives, so total the claims made
     // this session over whatever the policy authored and publish the sum.
     if (state::build_data::records::kTriumphScoreValueIndex < object.objectiveValues.size()) {
@@ -115,6 +140,7 @@ bool encode(const state::AccountState& state, std::span<std::byte> output) noexc
     // A node's progress bar reads a value slot and shows whatever it holds, so the claimed children
     // have to be counted into it here or the bar never moves.
     (void)state::record_claims::apply_node_progress(object.objectiveValues);
+
     for (layout::CharacterUnlockBlock& block : object.characterUnlocks) {
         block.flags = unlocks.characterFlags;
     }

Sunrise/src/server/bap/bap_route.cpp

@@ -184,6 +184,32 @@ void clear_session(Session& session) noexcept {
 
 } // namespace
 
+/** Arms every active peer to re-read the account, including the origin. */
+void arm_account_resync_everywhere() noexcept {
+    g_accountGeneration = g_accountGeneration == (std::numeric_limits<std::uint64_t>::max)()
+                              ? 1
+                              : g_accountGeneration + 1;
+    std::size_t armed = 0;
+    for (auto& peer : g_sessions) {
+        if (peer.id == 0 || !peer.authenticated || !peer.queuez.family4Active) {
+            continue;
+        }
+        peer.accountResyncGeneration = g_accountGeneration;
+        peer.accountResyncArmed = true;
+        ++armed;
+    }
+    std::array<char, core::log::kLineCapacity> line{};
+    const int count =
+        std::snprintf(line.data(), line.size(),
+                      "ev=queuez stage=resync_arm_all result=ok generation=%llu peers=%zu",
+                      static_cast<unsigned long long>(g_accountGeneration), armed);
+    if (count > 0) {
+        core::log::write(core::log::Channel::server, core::log::Level::debug,
+                         {line.data(), static_cast<std::size_t>(count)});
+    }
+}
+
+
 /** Applies one serialized BAP connection lifecycle event. */
 bool consume(const client::network::BapRequest& request,
              client::network::BapResponse& response) noexcept {

Sunrise/src/server/bap/encrypted/activity_host_manager/activity_host_manager_route.cpp

@@ -1,3 +1,4 @@
+#include "../../../../state/activity/current_activity.h"
 #include "activity_host_manager_route.h"
 
 #include <array>
@@ -59,6 +60,10 @@ choose_copy(const request_selection::ActivityManagerSelectionResult& parsed) noe
  * @param source Parsed selection carrying a package name.
  */
 void report_selection(const request_selection::ActivityManagerSelection& source) noexcept {
+    // Keep the bubble: a pickup later needs to know which activity it happened in, and the incident
+    // that reports it carries the bubble at a different place per incident type.
+    state::activity::set_current_bubble(source.hasArrivalBubbleHash ? source.arrivalBubbleHash
+                                                                    : state::activity::kNoBubble);
     std::array<char, core::log::kLineCapacity> line{};
     const int written =
         std::snprintf(line.data(),

Sunrise/src/server/bap/encrypted/activity_message/receipts/activity_message_receipts.cpp

@@ -4,6 +4,13 @@
  * was read so the caller can record one arrival receipt. None of them acts on what it read.
  */
 
+#include "../../../../../state/activity/current_activity.h"
+#include "../../../../../state/lore/lore_grant.h"
+#include <vector>
+#include <algorithm>
+#include "../../../../../state/build_data/collectibles/collectible_catalog.h"
+#include "../../../../../state/build_data/sobjects/sobject_catalog.h"
+#include "../../../../bap/internal.h"
 #include "activity_message_receipts.h"
 
 #include <array>
@@ -294,6 +301,106 @@ Framed frame_incident(const message::Request& request) noexcept {
            parsed.selectorLength,
            static_cast<unsigned>(parsed.hasOptionalBlock),
            parsed.payloadLength);
+    // Resolve the target against the definition table it indexes, so an incident says what it is
+    // rather than carrying a bare number. Nothing acts on it yet; this is what acting on it needs.
+    if (accepted) {
+        state::build_data::sobjects::Definition definition{};
+        if (state::build_data::sobjects::find(
+                static_cast<std::uint16_t>(parsed.primaryTarget), definition)) {
+            report(core::log::Level::info,
+                   "ev=activity stage=sobject target=%u hash=0x%08X type=%d group=%u ordinal=%u",
+                   parsed.primaryTarget,
+                   definition.nameHash,
+                   definition.typeCode,
+                   static_cast<unsigned>(definition.selectorGroup),
+                   static_cast<unsigned>(definition.nodeOrdinal));
+        } else {
+            report(core::log::Level::warn,
+                   "ev=activity stage=sobject target=%u result=unresolved rows=%zu",
+                   parsed.primaryTarget,
+                   state::build_data::sobjects::count());
+        }
+
+        // The type payload, which is what the schema describes. The parser separates it from
+        // the envelope; reading the whole message body instead puts every field at an offset the
+        // schema does not predict.
+        const std::span<const std::byte> body{parsed.payload.data(), parsed.payloadLength};
+
+        // The common header every type payload starts with: definition 0x80809512. A u32 sequence,
+        // then a three bit discriminator carrying a bias of one, then a 64 bit identity read only
+        // when that discriminator selects the player branch. Decoding it is the cheapest check that
+        // the payload is being read from the right place: the identity should be a SOID.
+        if (body.size() >= 13) {
+            std::uint64_t cursor = 0;
+            const auto take = [&body, &cursor](std::size_t width) noexcept -> std::uint64_t {
+                std::uint64_t value = 0;
+                for (std::size_t step = 0; step < width; ++step) {
+                    const std::size_t at = cursor + step;
+                    const auto byte = static_cast<std::uint8_t>(body[at / 8]);
+                    value = (value << 1U) | ((byte >> (7 - (at % 8))) & 1U);
+                }
+                cursor += width;
+                return value;
+            };
+            const std::uint64_t sequence = take(32);
+            const std::uint64_t kindRaw = take(3);
+            const std::uint64_t identity = take(64);
+            // Which book. The payload does not name the chapter it granted -- its only
+            // per-object content is a position -- so the reward is chosen here instead. The bubble
+            // does say which activity the pickup happened in, and an activity's pickups feed one
+            // book, which is a far smaller association than one entry per object in the world.
+            //
+            // The bubble sits at bit 126, the first of three consecutive u32 fields of the nested
+            // block. That position is measured: the same hash appears at 126, 158 and 190, and the
+            // character SOID lands at bit 35 exactly where the header schema puts it.
+            //
+            // Only type 2 carries this layout. Another type read against it yields a plausible
+            // number that means nothing, which is how a counter was once mistaken for an identity.
+            // A vase reports type 2 and a dead ghost type 10. Each type has its own schema, so
+            // the bubble sits somewhere different in each payload, and more types will turn up.
+            // The activity selection already named the bubble once, so it is read from there and
+            // the payload is not decoded at all.
+            constexpr std::array<std::int32_t, 2> kPickupTypeCodes{2, 10};
+            state::build_data::sobjects::Definition row{};
+            bool pickupType = false;
+            if (state::build_data::sobjects::find(static_cast<std::uint16_t>(parsed.primaryTarget),
+                                                  row)) {
+                for (const std::int32_t code : kPickupTypeCodes) {
+                    pickupType = pickupType || row.typeCode == code;
+                }
+            }
+            if (pickupType) {
+                const std::uint32_t bubble = state::activity::current_bubble();
+                const std::uint16_t node = state::lore::book_for_bubble(bubble);
+                const state::lore::GrantOutcome outcome = state::lore::grant_next_chapter(node);
+                if (outcome == state::lore::GrantOutcome::granted) {
+                    // Nothing else will stage an account image for this peer: a pickup is not a web
+                    // service transaction and has no response to carry the change back. Arm every
+                    // peer, the origin included, so the chapter appears without a relaunch and a
+                    // second pickup in the same run sees the first one already held.
+                    bap::arm_account_resync_everywhere();
+                }
+                report(outcome == state::lore::GrantOutcome::granted ? core::log::Level::info
+                                                                     : core::log::Level::warn,
+                       "ev=activity stage=lore bubble=0x%08X node=%u result=%s record=%u item=%d",
+                       bubble, static_cast<unsigned>(node),
+                       state::lore::grant_outcome_name(outcome),
+                       static_cast<unsigned>(outcome == state::lore::GrantOutcome::granted
+                                                 ? state::lore::last_granted_record()
+                                                 : 0),
+                       state::lore::last_item_granted() ? 1 : 0);
+            }
+
+            report(core::log::Level::info,
+                   "ev=activity stage=header target=%u sequence=%llu kind_raw=%llu "
+                   "identity=0x%016llX",
+                   parsed.primaryTarget,
+                   static_cast<unsigned long long>(sequence),
+                   static_cast<unsigned long long>(kindRaw),
+                   static_cast<unsigned long long>(identity));
+        }
+    }
+
     if (!accepted) {
         // A refused target index would index the consumer's table unbounded, so the body is kept
         // and never relayed.

Sunrise/src/server/bap/internal.h

@@ -182,6 +182,16 @@ struct Session {
     bool abilityRefreshArmed{};
 };
 
+/**
+ * Arms every active peer to re-read the account, including the one that caused the change.
+ *
+ * `publish_account_mutation` deliberately skips the origin, because a web service transaction
+ * carries the new account back in its own response. A change made outside such a transaction has no
+ * response to carry, so the peer that caused it would otherwise keep showing stale state until some
+ * unrelated action happened to stage an image. Picking up a collectible is such a change.
+ */
+void arm_account_resync_everywhere() noexcept;
+
 namespace plaintext {
 
 /**

Sunrise/src/state/activity/current_activity.cpp

@@ -0,0 +1,22 @@
+#include "current_activity.h"
+
+#include <atomic>
+
+namespace sunrise::state::activity {
+namespace {
+
+std::atomic<std::uint32_t> g_bubble{kNoBubble};
+
+} // namespace
+
+/** Records the arrival bubble of the activity just selected. */
+void set_current_bubble(std::uint32_t bubble) noexcept {
+    g_bubble.store(bubble, std::memory_order_relaxed);
+}
+
+/** @return The arrival bubble of the current activity. */
+std::uint32_t current_bubble() noexcept {
+    return g_bubble.load(std::memory_order_relaxed);
+}
+
+} // namespace sunrise::state::activity

Sunrise/src/state/activity/current_activity.h

@@ -0,0 +1,29 @@
+#pragma once
+
+#include <cstdint>
+
+namespace sunrise::state::activity {
+
+/**
+ * The activity the account is currently in.
+ *
+ * A collectible pickup arrives as an incident, and which book it should fill depends on where the
+ * player is. The incident does carry the bubble, but only in a position that differs per incident
+ * type, and there are several: a vase reports type 2 and a dead ghost type 10, each with its own
+ * schema. The selection that put the player there already names the bubble once, so keeping it is
+ * cheaper than decoding every type's payload and works for types not yet seen.
+ */
+
+/** No activity has been selected yet. */
+inline constexpr std::uint32_t kNoBubble = 0;
+
+/**
+ * Records the arrival bubble of the activity just selected.
+ * @param bubble Arrival bubble hash, or kNoBubble when the selection carried none.
+ */
+void set_current_bubble(std::uint32_t bubble) noexcept;
+
+/** @return The arrival bubble of the current activity, or kNoBubble. */
+[[nodiscard]] std::uint32_t current_bubble() noexcept;
+
+} // namespace sunrise::state::activity

Sunrise/src/state/build_data/cache/read/cache_file_reader.cpp

@@ -76,13 +76,16 @@ namespace {
 }
 
 /**
- * Every format below the current one is out of date, so a version bump needs no edit here.
- * Listing them one by one left a bumped version unknown, and a valid old cache read as corrupt.
+ * A cache written by any other format is out of date, so a version bump needs no edit here.
+ * Listing them one by one left a bumped version unknown, and a valid cache read as corrupt.
+ * A newer file is another build's cache rather than a damaged one, so it rebuilds the same way.
+ * Reading it as corrupt instead failed the whole boot until the file was deleted by hand, which
+ * is what downgrading the module did.
  * @param version Cache prefix version.
- * @return True when the cache is older than the current format.
+ * @return True when the cache was not written by the current format.
  */
 [[nodiscard]] bool stale_format(std::uint32_t version) noexcept {
-    return version < records::kCacheFormatVersion;
+    return version != records::kCacheFormatVersion;
 }
 
 /** @return The pending status, or invalid when the file fails to close. */
@@ -151,9 +154,6 @@ LoadStatus load(const wchar_t* path,
     if (stale_format(prefix.version)) {
         return close_with(file, LoadStatus::stale);
     }
-    if (prefix.version != records::kCacheFormatVersion) {
-        return close_with(file, LoadStatus::invalid);
-    }
 
     LARGE_INTEGER beginning{};
     records::Header header{};

Sunrise/src/state/build_data/collectibles/collectible_catalog.cpp

@@ -110,6 +110,22 @@ bool snapshot(std::span<Definition> output, std::size_t& count) noexcept {
     return g_definitions.snapshot(output, count);
 }
 
+/** Finds the collectible that unlocks one lore row. */
+bool find_by_lore_row(std::uint16_t loreRow, Definition& definition) noexcept {
+    if (loreRow == kUnavailableLoreRow) {
+        return false;
+    }
+    const Lock::Shared guard(g_lock);
+    for (const Definition& row : g_definitions.rows()) {
+        if (row.loreRow != loreRow) {
+            continue;
+        }
+        definition = row;
+        return true;
+    }
+    return false;
+}
+
 /** @return Number of published rows, read under the catalog lock. */
 std::size_t count() noexcept {
     const Lock::Shared guard(g_lock);

Sunrise/src/state/build_data/collectibles/collectible_catalog.h

@@ -13,6 +13,9 @@ inline constexpr std::size_t kDefinitionCapacity = 1U << 15U;
 inline constexpr std::uint16_t kUnavailableItemDefinitionIndex = 0xFFFFU;
 /** A collectible with no acquisition charge carries this native requirement-set sentinel. */
 inline constexpr std::uint16_t kUnavailableMaterialRequirementSetIndex = 0xFFFFU;
+
+/** A collectible that unlocks no lore carries this instead of a row. */
+inline constexpr std::uint16_t kUnavailableLoreRow = 0xFFFFU;
 /** Installed requirement sets contain at most six material rows. */
 inline constexpr std::size_t kMaterialRequirementCapacity = 6;
 
@@ -31,6 +34,13 @@ struct Definition {
     std::uint16_t collectibleIndex{};
     std::uint16_t itemDefinitionIndex{kUnavailableItemDefinitionIndex};
     std::uint16_t materialRequirementSetIndex{kUnavailableMaterialRequirementSetIndex};
+    /**
+     * Lore row this collectible unlocks, or kUnavailableLoreRow when it unlocks none.
+     *
+     * The record that displays the same row is the chapter this collectible completes, so the pair
+     * joins on the row rather than on any hash.
+     */
+    std::uint16_t loreRow{kUnavailableLoreRow};
     std::uint8_t materialRequirementCount{};
     std::array<MaterialRequirement, kMaterialRequirementCapacity> materialRequirements{};
 };
@@ -60,4 +70,16 @@ void clear() noexcept;
 /** @return Number of installed-build collectible mappings. */
 [[nodiscard]] std::size_t count() noexcept;
 
+/**
+ * Finds the collectible that unlocks one lore row.
+ *
+ * The reverse of the record join: a chapter record and the collectible that grants it name the same
+ * lore row, so this reaches the collectible whose item the account has to own for the lore itself to
+ * be readable rather than obscured.
+ * @param loreRow Lore table row.
+ * @param definition Receives the collectible.
+ * @return True when a collectible names that row.
+ */
+[[nodiscard]] bool find_by_lore_row(std::uint16_t loreRow, Definition& definition) noexcept;
+
 } // namespace sunrise::state::build_data::collectibles

Sunrise/src/state/build_data/records/definition.h

@@ -20,6 +20,9 @@ inline constexpr std::uint16_t kTriumphScoreValueIndex = 2115U;
 /** A record whose completion flag no mapping table addresses carries this instead of an index. */
 inline constexpr std::uint16_t kUnavailableFlagIndex = 0xFFFFU;
 
+/** A record that displays no lore carries this instead of a row. */
+inline constexpr std::uint16_t kUnavailableLoreRow = 0xFFFFU;
+
 /** A record naming no category value slot carries this instead of an index. */
 inline constexpr std::uint16_t kUnavailableValueIndex = 0xFFFFU;
 
@@ -35,6 +38,13 @@ struct Definition {
     std::uint16_t definitionIndex{};
     /** Account flag bank mapping row, or kUnavailableFlagIndex when the slot is unaddressable. */
     std::uint16_t completionFlagIndex{kUnavailableFlagIndex};
+    /**
+     * Lore row this record displays, or kUnavailableLoreRow when it displays none.
+     *
+     * A chapter of a lore book names one. A book's parent triumph names none, which is a reliable
+     * way to tell a parent from a chapter without relying on its position in the child list.
+     */
+    std::uint16_t loreRow{kUnavailableLoreRow};
     /** Points this record is worth, which the shipped table keeps at 500 or below. */
     std::uint16_t scoreValue{};
     /**

Sunrise/src/state/build_data/records/record_catalog.cpp

@@ -55,6 +55,22 @@ bool snapshot(std::span<Definition> output, std::size_t& count) noexcept {
     return g_definitions.snapshot(output, count);
 }
 
+/** Finds the record that displays one lore row. */
+bool find_by_lore_row(std::uint16_t loreRow, Definition& definition) noexcept {
+    if (loreRow == kUnavailableLoreRow) {
+        return false;
+    }
+    const Lock::Shared guard(g_lock);
+    for (const Definition& row : g_definitions.rows()) {
+        if (row.loreRow != loreRow) {
+            continue;
+        }
+        definition = row;
+        return true;
+    }
+    return false;
+}
+
 /** @return Number of generated record definitions, read under the lock. */
 std::size_t count() noexcept {
     const Lock::Shared guard(g_lock);

Sunrise/src/state/build_data/records/record_catalog.h

@@ -43,4 +43,15 @@ void clear() noexcept;
 /** @return Number of generated record definitions, read under the lock. */
 [[nodiscard]] std::size_t count() noexcept;
 
+/**
+ * Finds the record that displays one lore row.
+ *
+ * A collectible and the chapter record it completes name the same lore row, so this is the join
+ * between them. A book's parent triumph names no row and is never returned.
+ * @param loreRow Lore table row, from a collectible or from elsewhere.
+ * @param definition Receives the record.
+ * @return True when exactly one record displays that row.
+ */
+[[nodiscard]] bool find_by_lore_row(std::uint16_t loreRow, Definition& definition) noexcept;
+
 } // namespace sunrise::state::build_data::records

Sunrise/src/state/build_data/records/record_persistence.cpp

@@ -0,0 +1,150 @@
+#include "record_persistence.h"
+
+#include <array>
+#include <cstdio>
+#include <cstring>
+#include <vector>
+
+#include <windows.h>
+
+#include "../../../core/logging/log.h"
+#include "../../../core/filesystem/path.h"
+#include "../runtime.h"
+#include "record_catalog.h"
+
+namespace sunrise::state::build_data::records {
+namespace {
+
+/** The record file lives beside the claim file, in the directory the build data cache already owns. */
+constexpr std::wstring_view kRecordFileSuffix = L"\\cache\\record_definitions.bin";
+/** Identifies the file on sight, so an unrelated file of the right length cannot be read as one. */
+constexpr std::array<char, 8> kMagic{'S', 'N', 'R', 'S', 'R', 'E', 'C', '1'};
+
+core::path::Buffer g_path{};
+bool g_pathReady{};
+
+void report(const char* stage, const char* result, std::size_t detail) noexcept {
+    std::array<char, 128> line{};
+    const int written = std::snprintf(
+        line.data(), line.size(), "ev=recstore stage=%s result=%s rows=%zu", stage, result, detail);
+    if (written > 0) {
+        core::log::write(core::log::Channel::state,
+                         std::string_view{result} == "ok" ? core::log::Level::info
+                                                          : core::log::Level::warn,
+                         {line.data(), static_cast<std::size_t>(written)});
+    }
+}
+
+/**
+ * Header carried ahead of the rows.
+ *
+ * The row width is written out and checked on the way back in. The rows are stored as they sit in
+ * memory, so a build whose Definition has changed shape must not read an older file as if it had
+ * not: the width mismatch rejects it, extraction runs, and the file is replaced.
+ */
+struct Header {
+    std::array<char, 8> magic{};
+    std::uint32_t rows{};
+    std::uint32_t rowWidth{};
+};
+
+} // namespace
+
+/** Derives the record file path and publishes any table already held. */
+bool initialize(void* module) noexcept {
+    g_pathReady = false;
+    if (!core::path::artifact_directory(module, g_path)
+        || !core::path::append(g_path, kRecordFileSuffix)) {
+        report("initialize", "path_fail", 0);
+        return false;
+    }
+    g_pathReady = true;
+    return true;
+}
+
+/** Reads the record table and publishes it. Separate from initialize because a publish is only
+ *  accepted once the build data runtime is up, which is later than path setup. */
+bool load_and_publish() noexcept {
+    if (!g_pathReady) {
+        return false;
+    }
+    const HANDLE file = CreateFileW(
+        g_path.chars.data(), GENERIC_READ, FILE_SHARE_READ, nullptr, OPEN_EXISTING, 0, nullptr);
+    if (file == INVALID_HANDLE_VALUE) {
+        // A first run, not a fault. Extraction will build the table and write it.
+        report("load", "absent", 0);
+        return false;
+    }
+
+    Header header{};
+    DWORD read = 0;
+    const bool headerRead =
+        ReadFile(file, &header, sizeof header, &read, nullptr) != FALSE && read == sizeof header;
+    if (!headerRead || std::memcmp(header.magic.data(), kMagic.data(), kMagic.size()) != 0
+        || header.rowWidth != sizeof(Definition) || header.rows == 0
+        || header.rows > kDefinitionCapacity) {
+        CloseHandle(file);
+        report("load", headerRead ? "rejected" : "header_fail", header.rows);
+        return false;
+    }
+
+    std::vector<Definition> rows(header.rows);
+    const auto expected = static_cast<DWORD>(rows.size() * sizeof(Definition));
+    const bool rowsRead =
+        ReadFile(file, rows.data(), expected, &read, nullptr) != FALSE && read == expected;
+    CloseHandle(file);
+    if (!rowsRead) {
+        report("load", "read_fail", rows.size());
+        return false;
+    }
+    // Replaces the catalog directly rather than going through publish_node_definitions. That path
+    // opens a publication transaction, and a saved cache freezes every domain, so on the warm start
+    // this exists for it is refused before validation is even reached. The freeze is there to keep
+    // published domains agreeing with the cache; this domain is not in the cache, and the rows being
+    // restored are the ones extraction published and wrote here, so the same checks are run and the
+    // agreement the freeze protects is not touched.
+    if (!valid(std::span<const Definition>{rows})
+        || !replace(std::span<const Definition>{rows})) {
+        report("load", "publish_fail", rows.size());
+        return false;
+    }
+    report("load", "ok", rows.size());
+    return true;
+}
+
+/** Writes the record table so the next start does not need the package pass to rebuild it. */
+bool store(std::span<const Definition> definitions) noexcept {
+    if (!g_pathReady || definitions.empty() || definitions.size() > kDefinitionCapacity) {
+        return false;
+    }
+    Header header{};
+    header.magic = kMagic;
+    header.rows = static_cast<std::uint32_t>(definitions.size());
+    header.rowWidth = static_cast<std::uint32_t>(sizeof(Definition));
+
+    const HANDLE file = CreateFileW(g_path.chars.data(),
+                                    GENERIC_WRITE,
+                                    0,
+                                    nullptr,
+                                    CREATE_ALWAYS,
+                                    FILE_ATTRIBUTE_NORMAL,
+                                    nullptr);
+    if (file == INVALID_HANDLE_VALUE) {
+        report("store", "open_fail", definitions.size());
+        return false;
+    }
+    DWORD written = 0;
+    bool complete =
+        WriteFile(file, &header, sizeof header, &written, nullptr) != FALSE
+        && written == sizeof header;
+    if (complete) {
+        const auto size = static_cast<DWORD>(definitions.size() * sizeof(Definition));
+        complete = WriteFile(file, definitions.data(), size, &written, nullptr) != FALSE
+                   && written == size;
+    }
+    complete = CloseHandle(file) != FALSE && complete;
+    report("store", complete ? "ok" : "write_fail", definitions.size());
+    return complete;
+}
+
+} // namespace sunrise::state::build_data::records

Sunrise/src/state/build_data/records/record_persistence.h

@@ -0,0 +1,44 @@
+#pragma once
+
+#include <cstddef>
+#include <span>
+
+#include "definition.h"
+
+namespace sunrise::state::build_data::records {
+
+/**
+ * Keeps the record table across restarts, in a file of its own.
+ *
+ * The build data cache does not carry this domain, and the extraction pass only runs when the
+ * investment refresh gate finds something missing. Nodes are not in that gate, so a warm start ran
+ * with an empty record table: no category was counted and no progress bar could move, whatever had
+ * been claimed. Adding records to either gate is worse than the bug -- one is inert because it only
+ * governs cache writes, and the other retries the package pass forever because records do not publish
+ * on that path -- so the domain is persisted beside the claim file instead. Neither gate changes,
+ * and a warm start simply finds the table already published.
+ */
+
+/**
+ * Derives the record file path and publishes any table already held.
+ * A missing file is not a failure: it is a first run, and extraction will write one.
+ * @param module Loaded DLL, used to find the artifact directory.
+ * @return True when the path resolves. Loading is best effort and reported separately.
+ */
+[[nodiscard]] bool initialize(void* module) noexcept;
+
+/**
+ * Reads the held table and publishes it, if there is one.
+ * Kept apart from initialize because a publish is only accepted once the build data runtime is up.
+ * @return True when a table was published.
+ */
+[[nodiscard]] bool load_and_publish() noexcept;
+
+/**
+ * Writes the record table so the next start does not need the package pass to rebuild it.
+ * @param definitions Complete rows in native node order, as published.
+ * @return True when the file is written whole.
+ */
+[[nodiscard]] bool store(std::span<const Definition> definitions) noexcept;
+
+} // namespace sunrise::state::build_data::records

Sunrise/src/state/build_data/runtime.h

@@ -13,6 +13,8 @@
 #include "hash_names/definition.h"
 #include "inventory/buckets/definition.h"
 #include "items/details/definition.h"
+#include "nodes/definition.h"
+#include "records/definition.h"
 #include "items/item_catalog.h"
 #include "items/socket_plugs/definition.h"
 #include "material_requirements/material_requirement_catalog.h"
@@ -97,6 +99,40 @@ publish_item_definitions(std::span<const items::Definition> definitions) noexcep
                                               items::Definition& definition) noexcept;
 
 /** @return True when the whole dense collectible definition table is in State. */
+
+
+/** @return True when the whole presentation node table is in State. */
+[[nodiscard]] bool node_definitions_ready() noexcept;
+
+/**
+ * Publishes the whole presentation node table in one step.
+ * @param definitions Complete dense rows in native node order.
+ * @return True when the rows pass the domain checks and fit fixed State storage.
+ */
+[[nodiscard]] bool
+publish_node_definitions(std::span<const nodes::Definition> definitions) noexcept;
+
+
+/** @return True when the whole record definition table is in State. */
+[[nodiscard]] bool record_definitions_ready() noexcept;
+
+/**
+ * Publishes the whole record definition table in one step.
+ * @param definitions Complete dense rows in native record order.
+ * @return True when the rows pass the domain checks and fit fixed State storage.
+ */
+[[nodiscard]] bool
+publish_record_definitions(std::span<const records::Definition> definitions) noexcept;
+
+/**
+ * Resolves the native record row an opcode-1801 claim names.
+ * @param definitionIndex Native record row carried by the claim.
+ * @param definition Receives the row, including its completion flag index, only on success.
+ * @return True when the table is complete and the row exists.
+ */
+[[nodiscard]] bool find_record_definition(std::uint16_t definitionIndex,
+                                          records::Definition& definition) noexcept;
+
 [[nodiscard]] bool collectible_definitions_ready() noexcept;
 
 /**

Sunrise/src/state/build_data/sobjects/sobject_catalog.cpp

@@ -0,0 +1,56 @@
+#include "sobject_catalog.h"
+
+#include "../table.h"
+
+namespace sunrise::state::build_data::sobjects {
+namespace {
+
+Lock g_lock;
+Table<Definition, kDefinitionCapacity> g_definitions;
+
+} // namespace
+
+/** Clears the table while no reader can observe a partial replacement. */
+void clear() noexcept {
+    const Lock::Exclusive guard(g_lock);
+    g_definitions.clear();
+}
+
+/**
+ * Checks the rows are dense and fit.
+ *
+ * The table has no index column: a row's position is its identity, which is what makes the wire's
+ * target index meaningful. So there is nothing to cross-check a row against, and the only thing
+ * worth asserting is the shape.
+ */
+bool valid(std::span<const Definition> definitions) noexcept {
+    return !definitions.empty() && definitions.size() <= kDefinitionCapacity;
+}
+
+/** Replaces the whole table in one step. */
+bool replace(std::span<const Definition> definitions) noexcept {
+    if (!valid(definitions)) {
+        return false;
+    }
+    const Lock::Exclusive guard(g_lock);
+    return g_definitions.replace(definitions);
+}
+
+/** Finds one row by the target index an incident carries. */
+bool find(std::uint16_t targetIndex, Definition& definition) noexcept {
+    const Lock::Shared guard(g_lock);
+    const std::span<const Definition> rows = g_definitions.rows();
+    if (targetIndex >= rows.size()) {
+        return false;
+    }
+    definition = rows[targetIndex];
+    return true;
+}
+
+/** @return Number of installed rows, read under the lock. */
+std::size_t count() noexcept {
+    const Lock::Shared guard(g_lock);
+    return g_definitions.count();
+}
+
+} // namespace sunrise::state::build_data::sobjects

Sunrise/src/state/build_data/sobjects/sobject_catalog.h

@@ -0,0 +1,58 @@
+#pragma once
+
+#include <cstddef>
+#include <cstdint>
+#include <span>
+
+namespace sunrise::state::build_data::sobjects {
+
+/**
+ * The definition table an incident target names.
+ *
+ * An incident is the client's one general typed gameplay event, and its target is a 13-bit index
+ * into this table. Nothing acts on an incident today, so a collectible picked up in the world tells
+ * the server nothing. Resolving the target is the first step of acting on one.
+ *
+ * The table is shipped package data. Its class is `0x80807C9B` and two installed tags carry it with
+ * byte-identical hash and type-code columns, so a row index means the same thing against either.
+ */
+
+/** The wire target is thirteen bits, and 7763 rows install. */
+inline constexpr std::size_t kDefinitionCapacity = 8192;
+
+/** A row carrying this type code has no handler and cannot produce a schema. */
+inline constexpr std::int32_t kAbsentTypeCode = -1;
+
+/** One row of the table, reduced to what resolving a target needs. */
+struct Definition {
+    /** FNV-1 of the definition name. */
+    std::uint32_t nameHash{};
+    /** Selects the payload shape. Runs -1 to 48. */
+    std::int32_t typeCode{kAbsentTypeCode};
+    /** Compressed-selector group, or 0xFFFF for none. */
+    std::uint16_t selectorGroup{};
+    /** Position in that group's node table. */
+    std::uint16_t nodeOrdinal{};
+};
+
+/** Clears every generated row. */
+void clear() noexcept;
+
+/** @return True when the rows are dense and within capacity. */
+[[nodiscard]] bool valid(std::span<const Definition> definitions) noexcept;
+
+/** Replaces the whole table in one step. */
+[[nodiscard]] bool replace(std::span<const Definition> definitions) noexcept;
+
+/**
+ * Finds one row by the target index an incident carries.
+ * @param targetIndex Index from the wire, already range-checked by the incident validator.
+ * @param definition Receives the row.
+ * @return True when the index is in range.
+ */
+[[nodiscard]] bool find(std::uint16_t targetIndex, Definition& definition) noexcept;
+
+/** @return Number of installed rows. */
+[[nodiscard]] std::size_t count() noexcept;
+
+} // namespace sunrise::state::build_data::sobjects

Sunrise/src/state/lore/lore_grant.cpp

@@ -0,0 +1,175 @@
+#include "lore_grant.h"
+
+#include <array>
+#include <atomic>
+#include <cstddef>
+#include <span>
+#include <vector>
+
+#include "../build_data/nodes/definition.h"
+#include "../build_data/nodes/node_catalog.h"
+#include "../build_data/nodes/node_persistence.h"
+#include "../build_data/records/definition.h"
+#include "../build_data/records/record_catalog.h"
+#include "../build_data/records/record_persistence.h"
+#include "../build_data/collectibles/collectible_catalog.h"
+#include "../build_data/inventory/buckets/definition.h"
+#include "../build_data/items/details/definition.h"
+#include "../runtime/runtime.h"
+#include "../build_data/runtime.h"
+#include "../record_claims/record_claims.h"
+
+namespace sunrise::state::lore {
+namespace {
+
+/**
+ * Which book an activity's pickups feed.
+ *
+ * One entry per activity, not one per object. This is the whole of the authored data the chosen
+ * design needs, and it is the part that would be replaced if Bungie's own object-to-reward mapping
+ * were ever reproduced from activity and spawn data.
+ *
+ * Only the Menagerie is known so far, measured from its pickups: the bubble its incidents carry
+ * against the book those pickups fill.
+ */
+struct BubbleBook {
+    std::uint32_t bubble;
+    std::uint16_t node;
+};
+
+constexpr std::array<BubbleBook, 2> kBubbleBooks{{
+    // caluseum_experience, whose vases fill Confessions. Node 838 confirmed against the published
+    // manifest: nine chapters, Entry I on record 1708 carrying lore hash 0x58C9C088.
+    {0x811C9DC5U, 838U},
+    // The destination whose dead ghosts fill Ghost Stories. Node 817 identified by child count --
+    // the manifest lists 23 records and only two books here have 23 chapters -- and by section,
+    // since Ghost Stories sits under The Light and the other candidate, node 847, does not.
+    {0x5FE28198U, 817U},
+}};
+
+std::atomic<std::uint16_t> g_lastGranted{0};
+std::atomic<bool> g_lastItemGranted{false};
+
+} // namespace
+
+/** @return A short name for the outcome, for logs. */
+const char* grant_outcome_name(GrantOutcome outcome) noexcept {
+    switch (outcome) {
+    case GrantOutcome::granted:
+        return "granted";
+    case GrantOutcome::unknownBook:
+        return "unknown_book";
+    case GrantOutcome::noNodeTable:
+        return "no_node_table";
+    case GrantOutcome::bookNotFound:
+        return "book_not_found";
+    case GrantOutcome::noChildren:
+        return "no_children";
+    case GrantOutcome::noRecords:
+        return "no_records";
+    case GrantOutcome::noChapters:
+        return "no_chapters";
+    case GrantOutcome::bookComplete:
+        return "book_complete";
+    case GrantOutcome::refused:
+        return "refused";
+    }
+    return "unknown";
+}
+
+/** @return The presentation node of the book that activity's pickups feed. */
+std::uint16_t book_for_bubble(std::uint32_t bubble) noexcept {
+    for (const BubbleBook& entry : kBubbleBooks) {
+        if (entry.bubble == bubble) {
+            return entry.node;
+        }
+    }
+    return kNoBook;
+}
+
+/** Grants the next chapter of one book that the account does not already hold. */
+GrantOutcome grant_next_chapter(std::uint16_t node) noexcept {
+    if (node == kNoBook) {
+        return GrantOutcome::unknownBook;
+    }
+
+    namespace nodes = build_data::nodes;
+    namespace records = build_data::records;
+    // On a warm start the package pass is skipped, so the node table is never published and every
+    // book looks empty. The table is kept in its own file for exactly this; publishing it here
+    // costs nothing once it has succeeded.
+    static std::atomic<bool> published{false};
+    if (!published.load(std::memory_order_relaxed)) {
+        const bool haveNodes = nodes::count() != 0 || nodes::load_and_publish();
+        const bool haveRecords = records::count() != 0 || records::load_and_publish();
+        if (haveNodes && haveRecords) {
+            published.store(true, std::memory_order_relaxed);
+        }
+    }
+    std::vector<nodes::Definition> rows(nodes::kDefinitionCapacity);
+    std::size_t count = 0;
+    if (!nodes::snapshot(std::span<nodes::Definition>{rows}, count) || count == 0) {
+        return GrantOutcome::noNodeTable;
+    }
+
+    const nodes::Definition* book = nullptr;
+    for (std::size_t row = 0; row < count; ++row) {
+        if (rows[row].definitionIndex == node) {
+            book = &rows[row];
+            break;
+        }
+    }
+    if (book == nullptr) {
+        return GrantOutcome::bookNotFound;
+    }
+    if (book->childCount == 0) {
+        return GrantOutcome::noChildren;
+    }
+
+    bool sawRecord = false;
+    bool sawChapter = false;
+    for (std::size_t child = 0; child < book->childCount; ++child) {
+        records::Definition record{};
+        if (!build_data::find_record_definition(book->children[child], record)) {
+            continue;
+        }
+        sawRecord = true;
+        if (record.completionFlagIndex == records::kUnavailableFlagIndex) {
+            continue;
+        }
+        // A child naming no lore row is the book's parent triumph, not a chapter. Granting it would
+        // mark the book complete without giving any of its contents.
+        if (record.loreRow == records::kUnavailableLoreRow) {
+            continue;
+        }
+        sawChapter = true;
+        // Finding lore completes a chapter; claiming it is the player's act, not this one. So the
+        // record is left claimable rather than claimed, and a chapter already in either state is
+        // passed over.
+        if (record_claims::claimed(record.completionFlagIndex)
+            || record_claims::claimable(record.completionFlagIndex)) {
+            continue;
+        }
+        if (!record_claims::mark_claimable(record.completionFlagIndex)) {
+            return GrantOutcome::refused;
+        }
+        g_lastGranted.store(record.definitionIndex, std::memory_order_relaxed);
+        return GrantOutcome::granted;
+    }
+    if (sawChapter) {
+        return GrantOutcome::bookComplete;
+    }
+    return sawRecord ? GrantOutcome::noChapters : GrantOutcome::noRecords;
+}
+
+/** @return True when the last grant also gave the collectible's item. */
+bool last_item_granted() noexcept {
+    return g_lastItemGranted.load(std::memory_order_relaxed);
+}
+
+/** @return The record row the last successful grant claimed. */
+std::uint16_t last_granted_record() noexcept {
+    return g_lastGranted.load(std::memory_order_relaxed);
+}
+
+} // namespace sunrise::state::lore

Sunrise/src/state/lore/lore_grant.h

@@ -0,0 +1,68 @@
+#pragma once
+
+#include <cstdint>
+
+namespace sunrise::state::lore {
+
+/**
+ * Granting a lore chapter when a collectible is picked up.
+ *
+ * The incident a pickup emits does not name what it granted. Its only per-object content is a
+ * position: the same object picked twice produces a payload differing by one bit, the sequence,
+ * while a different object differs only across the position vector. Reproducing Bungie's choice of
+ * chapter would mean mapping every object in the world to a reward, from activity and spawn data.
+ *
+ * So the chapter is chosen here instead. A pickup grants the next chapter of its book that the
+ * account does not already hold, which is what a player observes anyway: collect, and the book
+ * fills in. The association needed is one book per activity rather than one reward per object.
+ */
+
+/** A node index that names no book. */
+inline constexpr std::uint16_t kNoBook = 0xFFFFU;
+
+/** Why one pickup granted nothing. */
+enum class GrantOutcome : std::uint8_t {
+    granted,
+    /** The bubble is not associated with a book. */
+    unknownBook,
+    /** The node table is not published, so no book can be read. */
+    noNodeTable,
+    /** The node table is published but does not contain this book. */
+    bookNotFound,
+    /** The book is present but owns no children. */
+    noChildren,
+    /** The children are present but no record resolves, so the record table is missing. */
+    noRecords,
+    /** Records resolve but none of them is a chapter. */
+    noChapters,
+    /** Every chapter of the book is already held. */
+    bookComplete,
+    /** The claim store refused the write. */
+    refused,
+};
+
+/** @return A short name for the outcome, for logs. */
+[[nodiscard]] const char* grant_outcome_name(GrantOutcome outcome) noexcept;
+
+/**
+ * @param bubble Bubble hash the incident carried.
+ * @return The presentation node of the book that activity's pickups feed, or kNoBook.
+ */
+[[nodiscard]] std::uint16_t book_for_bubble(std::uint32_t bubble) noexcept;
+
+/**
+ * Grants the next chapter of one book that the account does not already hold.
+ * Chapters are the node's children that name a lore row; the child naming none is the book's parent
+ * triumph and is never granted.
+ * @param node Presentation node of the book.
+ * @return What happened, so a refusal reads differently from a completed book.
+ */
+[[nodiscard]] GrantOutcome grant_next_chapter(std::uint16_t node) noexcept;
+
+/** @return The record row the last successful grant claimed. Only meaningful after `granted`. */
+[[nodiscard]] std::uint16_t last_granted_record() noexcept;
+
+/** @return True when the last grant also gave the collectible's item, which the lore is gated on. */
+[[nodiscard]] bool last_item_granted() noexcept;
+
+} // namespace sunrise::state::lore

Sunrise/src/state/record_claims/record_claims.cpp

@@ -1,6 +1,5 @@
 #include <atomic>
 
-#include "../build_data/nodes/node_persistence.h"
 #include "record_claims.h"
 
 #include <array>
@@ -38,6 +37,8 @@ constexpr std::uint32_t kMaximumEntries = 8192;
 
 std::mutex g_lock;
 std::array<std::uint64_t, kWordCount> g_claimed{};
+/** Records complete but not yet claimed. A claim supersedes this, never the other way round. */
+std::array<std::uint64_t, kWordCount> g_claimable{};
 std::array<std::uint16_t, kIndexCapacity> g_scoreByIndex{};
 std::size_t g_count{};
 std::uint32_t g_score{};
@@ -190,6 +191,7 @@ bool initialize(void* module) noexcept {
 void clear() noexcept {
     const std::lock_guard<std::mutex> guard(g_lock);
     g_claimed.fill(0);
+    g_claimable.fill(0);
     g_scoreByIndex.fill(0);
     g_count = 0;
     g_score = 0;
@@ -236,6 +238,24 @@ std::size_t apply(std::span<std::uint8_t> accountFlags) noexcept {
             }
         }
     }
+
+    // Claimable records fill in behind the claims: a record that is both stays claimed, since a
+    // claim is the later state and overwriting it would undo what the player did.
+    for (std::size_t word = 0; word < g_claimable.size(); ++word) {
+        std::uint64_t bits = g_claimable[word] & ~g_claimed[word];
+        while (bits != 0) {
+            const auto offset = static_cast<std::size_t>(std::countr_zero(bits));
+            bits &= bits - 1;
+            const std::size_t index = word * kWordBits + offset;
+            if (index >= accountFlags.size()) {
+                continue;
+            }
+            if (accountFlags[index] != unlocks::kFlagAvailable) {
+                accountFlags[index] = unlocks::kFlagAvailable;
+                ++changed;
+            }
+        }
+    }
     return changed;
 }
 
@@ -302,14 +322,6 @@ struct NodeProgress {
 
 /** Writes each node's claimed-child count into the value slot its bar reads. */
 std::size_t apply_node_progress(std::span<std::int32_t> objectiveValues) noexcept {
-    // The account image is the latest point anything asks for the node table, and on a warm start it
-    // may still be unpublished: the content pass is skipped and an earlier publish can be refused.
-    // Latches on success, so a run that has its table pays nothing.
-    static std::atomic<bool> nodesPublished{false};
-    if (!nodesPublished.load(std::memory_order_relaxed)
-        && build_data::nodes::load_and_publish()) {
-        nodesPublished.store(true, std::memory_order_relaxed);
-    }
     // Every category's own index, so the walk can tell a free slot above a category from the next
     // category along. Without it, writing the slot above drives whichever book owns that slot.
     std::vector<char> categoryFlags(objectiveValues.size(), 0);
@@ -379,6 +391,34 @@ std::size_t apply_character_node_progress(std::span<std::int32_t> characterValue
     return progress.written;
 }
 
+/** @return True when this index is already held. */
+bool claimed(std::uint16_t flagIndex) noexcept {
+    const std::lock_guard<std::mutex> guard(g_lock);
+    return claimed_locked(flagIndex);
+}
+
+/** Marks one record complete but unclaimed. */
+bool mark_claimable(std::uint16_t flagIndex) noexcept {
+    if (static_cast<std::size_t>(flagIndex) >= kIndexCapacity) {
+        return false;
+    }
+    const std::lock_guard<std::mutex> guard(g_lock);
+    g_claimable[static_cast<std::size_t>(flagIndex) / kWordBits] |=
+        1ULL << (static_cast<std::size_t>(flagIndex) % kWordBits);
+    return true;
+}
+
+/** @return True when this index is marked claimable. */
+bool claimable(std::uint16_t flagIndex) noexcept {
+    if (static_cast<std::size_t>(flagIndex) >= kIndexCapacity) {
+        return false;
+    }
+    const std::lock_guard<std::mutex> guard(g_lock);
+    return (g_claimable[static_cast<std::size_t>(flagIndex) / kWordBits]
+            & (1ULL << (static_cast<std::size_t>(flagIndex) % kWordBits)))
+           != 0;
+}
+
 /** @return Total score of every held claim. */
 std::uint32_t total_score() noexcept {
     const std::lock_guard<std::mutex> guard(g_lock);

Sunrise/src/state/record_claims/record_claims.h

@@ -37,6 +37,19 @@ void clear() noexcept;
  */
 [[nodiscard]] bool claim(std::uint16_t flagIndex, std::uint16_t scoreValue) noexcept;
 
+/**
+ * Marks one record complete but unclaimed, so the client offers it as claimable.
+ *
+ * Finding lore completes a triumph; claiming it is a separate act by the player. A record marked
+ * this way and later claimed carries the claim instead, so the two never conflict.
+ * @param flagIndex Mapping-table row whose object byte feeds the record's completion flag.
+ * @return True when the index is in range and the record is now marked.
+ */
+[[nodiscard]] bool mark_claimable(std::uint16_t flagIndex) noexcept;
+
+/** @return True when this index is marked claimable, whether or not it is also claimed. */
+[[nodiscard]] bool claimable(std::uint16_t flagIndex) noexcept;
+
 /**
  * Lays every held claim over one account flag bank.
  * @param accountFlags Bank already filled from the authored policy.

Sunrise/src/state/runtime/state_runtime.cpp

@@ -1,4 +1,6 @@
+#include "../build_data/records/record_persistence.h"
 #include "../build_data/nodes/node_persistence.h"
+#include "../record_claims/record_claims.h"
 #include <Windows.h>
 
 #include <algorithm>
@@ -14,7 +16,6 @@
 #include "../../core/settings/settings.h"
 #include "../activity/defaults/activity_defaults_validation.h"
 #include "../build_data/runtime.h"
-#include "../record_claims/record_claims.h"
 #include "equipment/configured_equipment_identity.h"
 #include "runtime.h"
 #include "state.h"
@@ -207,12 +208,11 @@ bool initialize(void* module,
     if (!build_data::initialize(module, runtime::equipment::configured_hash(runtimeAccount))) {
         return false;
     }
-    // Claims are held beside the build data cache, so a restart keeps what the client already
-    // shows as Acquired. A missing file is a first run, not a failure.
-    (void)record_claims::initialize(module);
-    // Publishes the node table if a previous run wrote one, so a warm start counts
-    // categories without needing the package pass to run again.
+    // Claims are held beside the build data cache, so a restart keeps what the client already shows
+    // as Acquired. A missing file is a first run, not a failure.
     (void)build_data::nodes::initialize(module);
+    (void)build_data::records::initialize(module);
+    (void)record_claims::initialize(module);
     // A cache hit already has the complete plug relation, so publish canonical profile identities
     // in the first State image.  On a first cache build, snapshot preparation repeats this step
     // after package extraction has published the relation.

Sunrise/src/state/unlocks/definition.h

@@ -32,6 +32,17 @@ using ProgressionBank = std::array<ProgressionLanes, build_data::progressions::k
 
 /** A set acquired flag is stored as its biased 2-bit true value. */
 inline constexpr std::uint8_t kFlagSet = 2;
+
+/**
+ * A record that is complete but not yet claimed, which the client offers as claimable.
+ *
+ * The flag reads as a small enum rather than a boolean: zero and two are the values the authored
+ * policy uses, and one is unused by it. Finding lore should leave a triumph waiting to be claimed
+ * rather than claiming it, which is what this value is for. That reading is inferred from the gap
+ * in the enum, not measured, so a record that stays invisible when set to it is the sign it is
+ * wrong.
+ */
+inline constexpr std::uint8_t kFlagAvailable = 1;
 /** A clear acquired flag is stored as zero. */
 inline constexpr std::uint8_t kFlagClear = 0;