Bug 1517924 - Stackmap creation for wasm-via-Ion. r=lhansen.
authorJulian Seward <jseward@acm.org>
Tue, 05 Mar 2019 16:03:32 +0100
changeset 520393 01c2d53e22c42738e5cc7ea6bda4cdfe14270226
parent 520278 cb76b90f829c7687c981c4dcb1fde868167ce765
child 520394 3682b31958e1b0c3604eddea18c712bb29ed35dc
push id10862
push userffxbld-merge
push dateMon, 11 Mar 2019 13:01:11 +0000
treeherdermozilla-beta@a2e7f5c935da [default view] [failures only]
perfherder[talos] [build metrics] [platform microbench] (compared to previous push)
reviewerslhansen
bugs1517924
milestone67.0a1
first release with
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
last release without
nightly linux32
nightly linux64
nightly mac
nightly win32
nightly win64
Bug 1517924 - Stackmap creation for wasm-via-Ion. r=lhansen. This patch creates stackmaps when compiling wasm via Ion. The resulting maps (wasm::StackMap) have the same type as those created by the baseline compiler, and the user thereof (Instance::traceFrame) is unchanged -- it doesn't know or care which compiler produced a map. Maps are created for calls and for resumable traps -- that is, the stack overflow checks at function entry and at loop heads. As with the baseline compiler, for non-debug builds, stackmaps are omitted when they would cover only non-ref words. For debug builds, they are never omitted, as that makes GC-time assertions on map-boundary-correctness more effective. Summary of implementation ~~~~~~~~~~~~~~~~~~~~~~~~~ (1) The front end (WasmIonCompile.cpp) is assumed to generate MIR nodes with type MIRType::RefOrNull to indicate ref-ness as needed. (2) When lowering MIR to LIR, nodes requiring a stackmap are marked by calling assignWasmSafepoint [LIRGenerator::{lowerWasmCall, visitWasmInterruptCheck}]. (3) When lowering LIR to machine insns, for calls and traps, the LSafepoint created by (2) is associated with a specific assembler offset (machine insn) and is tagged with the relevant lower-limit masm.framePushed value [CodeGenerator::{emitWasmCallBase, visitWasmInterruptCheck}]. (4) After code generation (including regalloc) has finished, all the LSafepoints created by (2)/(3) are visited, and from them wasm::StackMaps are created [CodeGenerator::generateWasm, CreateStackMapFromLSafepoint]. The StackMaps are added to our running collection thereof. CodeGenerator::generateWasm also creates the function entry stack map [CreateStackMapForFunctionEntryTrap]. Changes to existing structure ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When generating stackmaps for call instructions, it is crucial to correctly establish the boundary between the caller and callee's stackmaps. The boundary is defined thusly: the upper end of the callee's stackmap corresponds with the the highest-addressed stack word that carries an argument value, and the lower end of the caller's stackmap is the next word above that, and includes any and all padding pushed before the arguments proper. Hence the following change: all MWasmCallNodes must now carry the value StackArgAreaSizeUnaligned(outgoing arg tys), and so that has been added as an extra field, stackArgAreaSizeUnaligned_. [FunctionCompiler::{callDirect, callIndirect, callImport, builtinCall, builtinInstanceMethodCall]. This applies to MWasmCallNodes created by callDirect, callIndirect, callImport, builtinCall and builtinInstanceMethodCall. For the latter two, the outgoing argument size is derived from type information in the callee's SymbolicAddressSignature. Other details of implementation not mentioned above, in no particular order ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * Some JitSpewery has been added. * There has been some renaming of variables in BaseCompiler::beginFunction so as to emphasise the commonality with new function CreateStackMapForFunctionEntryTrap, since they perform essentially the same job. * LSafepoint has two new wasm-only fields isWasmTrap_ and framePushedAtStackMapBase_, needed to determine the map base point when converting to a wasm::StackMap. * (debug only) MDefinition::printOpcode now prints the type of each value, so it's possible to look at MIR dumps and see if the types are right. * MacroAssembler::wasmReserveStackChecked has been rewritten, although it generates the same code: - it returns both: the assembler offset of the trapping insn, so we can key the associated StackMap to it, and - the number of bytes of stack allocated before the trap, so that CreateStackMapForFunctionEntryTrap can take those into account - The somewhat krunky control flow if (amount > MAX_UNCHECKED_LEAF_FRAME_SIZE), and the opposite test later, has been merged into a single test. I find this safer and easier to reason about. * GenerateStackmapEntriesForTrapExit is used by both compilers and so has been moved to the common area WasmGC.{cpp,h}. * |struct StackMap| and |class StackMaps| have been moved from WasmTypes.h to WasmGC.h.
js/src/jit-test/tests/wasm/gc/anyref-global-postbarrier.js
js/src/jit-test/tests/wasm/gc/anyref-global-prebarrier.js
js/src/jit/CodeGenerator.cpp
js/src/jit/CodeGenerator.h
js/src/jit/LIR.cpp
js/src/jit/LIR.h
js/src/jit/Lowering.cpp
js/src/jit/MIR.cpp
js/src/jit/MIR.h
js/src/jit/MacroAssembler.cpp
js/src/jit/MacroAssembler.h
js/src/jit/shared/Lowering-shared.cpp
js/src/jit/shared/Lowering-shared.h
js/src/wasm/WasmBaselineCompile.cpp
js/src/wasm/WasmCode.h
js/src/wasm/WasmGC.cpp
js/src/wasm/WasmGC.h
js/src/wasm/WasmInstance.cpp
js/src/wasm/WasmIonCompile.cpp
js/src/wasm/WasmTypes.h
--- a/js/src/jit-test/tests/wasm/gc/anyref-global-postbarrier.js
+++ b/js/src/jit-test/tests/wasm/gc/anyref-global-postbarrier.js
@@ -60,17 +60,25 @@ test(exportsObj);
 // Test stacks reported in profiling mode in a separate way, to not perturb
 // the behavior of the tested functions.
 if (!isSingleStepProfilingEnabled)
     quit(0);
 
 enableGeckoProfiling();
 
 const EXPECTED_STACKS = [
-    ['', '!>', '0,!>', '<,0,!>', 'GC postbarrier,0,!>', '<,0,!>', '0,!>', '!>', ''],
+    // Expected output for (simulator+baseline).
+    ['', '!>', '0,!>', '<,0,!>', 'GC postbarrier,0,!>',
+     '<,0,!>', '0,!>', '!>', ''],
+
+    // Expected output for (simulator+via-Ion).
+    ['', '!>', '0,!>', '<,0,!>', 'filtering GC postbarrier,0,!>',
+     '<,0,!>', '0,!>', '!>', ''],
+
+    // Expected output for other configurations.
     ['', '!>', '0,!>', '!>', ''],
 ];
 
 function testStacks(exports) {
     // Test post-write barrier in wasm code.
     {
         let nomnom = new Baguette(15);
         startProfiling();
--- a/js/src/jit-test/tests/wasm/gc/anyref-global-prebarrier.js
+++ b/js/src/jit-test/tests/wasm/gc/anyref-global-prebarrier.js
@@ -29,17 +29,40 @@ if (!isSingleStepProfilingEnabled) {
     quit(0);
 }
 
 enableGeckoProfiling();
 startProfiling();
 gczeal(4, 1);
 e.set(obj);
 gczeal(0);
-assertEqPreciseStacks(endProfiling(), [['', '!>', '0,!>', '!>', '']]);
+assertEqPreciseStacks(
+    endProfiling(),
+    [
+        // Expected output for (simulator+via-Ion).
+        ['', '!>', '0,!>', '<,0,!>', 'filtering GC postbarrier,0,!>',
+         '<,0,!>', '0,!>', '!>', ''],
+
+        // Expected output for (simulator+baseline).
+        ['', '!>', '0,!>', '<,0,!>', 'GC postbarrier,0,!>',
+         '<,0,!>', '0,!>', '!>', ''],
+
+        // Expected output for other configurations.
+        ['', '!>', '0,!>', '!>', ''],
+    ]);
 
 startProfiling();
 gczeal(4, 1);
 e.set(null);
 gczeal(0);
 
 // We're losing stack info in the prebarrier code.
-assertEqPreciseStacks(endProfiling(), [['', '!>', '0,!>', '', '0,!>', '!>', '']]);
+assertEqPreciseStacks(
+    endProfiling(),
+    [
+        // Expected output for (simulator+via-Ion).
+        ['', '!>', '0,!>', '', '0,!>', '<,0,!>', 'filtering GC postbarrier,0,!>',
+         '<,0,!>', '0,!>', '!>', ''],
+
+        // Expected output for other configurations.
+        ['', '!>', '0,!>', '', '0,!>', '!>', ''],
+    ]);
+
--- a/js/src/jit/CodeGenerator.cpp
+++ b/js/src/jit/CodeGenerator.cpp
@@ -5912,16 +5912,17 @@ static void DumpTrackedOptimizations(Tra
   if (!JitSpewEnabled(JitSpew_OptimizationTracking)) {
     return;
   }
 
   optimizations->spew(JitSpew_OptimizationTracking);
 }
 
 bool CodeGenerator::generateBody() {
+  JitSpew(JitSpew_Codegen, "==== BEGIN CodeGenerator::generateBody ====\n");
   IonScriptCounts* counts = maybeCreateScriptCounts();
 
 #if defined(JS_ION_PERF)
   PerfSpewer* perfSpewer = &perfSpewer_;
   if (gen->compilingWasm()) {
     perfSpewer = &gen->perfSpewer();
   }
 #endif
@@ -6055,16 +6056,17 @@ bool CodeGenerator::generateBody() {
       return false;
     }
 
 #if defined(JS_ION_PERF)
     perfSpewer->endBasicBlock(masm);
 #endif
   }
 
+  JitSpew(JitSpew_Codegen, "==== END CodeGenerator::generateBody ====\n");
   return true;
 }
 
 // Out-of-line object allocation for LNewArray.
 class OutOfLineNewArray : public OutOfLineCodeBase<CodeGenerator> {
   LNewArray* lir_;
 
  public:
@@ -7245,17 +7247,21 @@ void CodeGenerator::visitGetNextEntryFor
   if (lir->mir()->mode() == MGetNextEntryForIterator::Map) {
     emitGetNextEntryForIterator<MapIteratorObject, ValueMap>(lir);
   } else {
     MOZ_ASSERT(lir->mir()->mode() == MGetNextEntryForIterator::Set);
     emitGetNextEntryForIterator<SetIteratorObject, ValueSet>(lir);
   }
 }
 
-void CodeGenerator::emitWasmCallBase(MWasmCall* mir, bool needsBoundsCheck) {
+template <size_t Defs>
+void CodeGenerator::emitWasmCallBase(LWasmCallBase<Defs>* lir) {
+  MWasmCall* mir = lir->mir();
+  bool needsBoundsCheck = lir->needsBoundsCheck();
+
   MOZ_ASSERT((sizeof(wasm::Frame) + masm.framePushed()) % WasmStackAlignment ==
              0);
   static_assert(
       WasmStackAlignment >= ABIStackAlignment &&
           WasmStackAlignment % ABIStackAlignment == 0,
       "The wasm stack alignment should subsume the ABI-required alignment");
 
 #ifdef DEBUG
@@ -7294,37 +7300,46 @@ void CodeGenerator::emitWasmCallBase(MWa
       break;
     case wasm::CalleeDesc::BuiltinInstanceMethod:
       masm.wasmCallBuiltinInstanceMethod(desc, mir->instanceArg(),
                                          callee.builtin());
       switchRealm = false;
       break;
   }
 
+  // Note the assembler offset for the associated LSafePoint.
+  markSafepointAt(masm.currentOffset(), lir);
+
+  // Now that all the outbound in-memory args are on the stack, note the
+  // required lower boundary point of the associated StackMap.
+  lir->safepoint()->setFramePushedAtStackMapBase(
+      masm.framePushed() - mir->stackArgAreaSizeUnaligned());
+  MOZ_ASSERT(!lir->safepoint()->isWasmTrap());
+
   if (reloadRegs) {
     masm.loadWasmTlsRegFromFrame();
     masm.loadWasmPinnedRegsFromTls();
     if (switchRealm) {
       masm.switchToWasmTlsRealm(ABINonArgReturnReg0, ABINonArgReturnReg1);
     }
   } else {
     MOZ_ASSERT(!switchRealm);
   }
 }
 
 void CodeGenerator::visitWasmCall(LWasmCall* ins) {
-  emitWasmCallBase(ins->mir(), ins->needsBoundsCheck());
+  emitWasmCallBase(ins);
 }
 
 void CodeGenerator::visitWasmCallVoid(LWasmCallVoid* ins) {
-  emitWasmCallBase(ins->mir(), ins->needsBoundsCheck());
+  emitWasmCallBase(ins);
 }
 
 void CodeGenerator::visitWasmCallI64(LWasmCallI64* ins) {
-  emitWasmCallBase(ins->mir(), ins->needsBoundsCheck());
+  emitWasmCallBase(ins);
 }
 
 void CodeGenerator::visitWasmLoadSlot(LWasmLoadSlot* ins) {
   MIRType type = ins->type();
   Register container = ToRegister(ins->containerRef());
   Address addr(container, ins->offset());
   AnyRegister dst = ToAnyRegister(ins->output());
 
@@ -10198,27 +10213,351 @@ void CodeGenerator::visitRest(LRest* lir
     masm.movePtr(ImmPtr(nullptr), temp2);
   }
   masm.bind(&joinAlloc);
 
   emitRest(lir, temp2, numActuals, temp0, temp1, numFormals, templateObject,
            false, ToRegister(lir->output()));
 }
 
+// A stackmap creation helper.  Create a stackmap from a vector of booleans.
+// The caller owns the resulting stackmap.
+
+typedef Vector<bool, 128, SystemAllocPolicy> StackMapBoolVector;
+
+static wasm::StackMap* ConvertStackMapBoolVectorToStackMap(
+                          const StackMapBoolVector& vec, bool hasRefs) {
+  wasm::StackMap* stackMap = wasm::StackMap::create(vec.length());
+  if (!stackMap) {
+    return nullptr;
+  }
+
+  bool hasRefsObserved = false;
+  size_t i = 0;
+  for (bool b : vec) {
+    if (b) {
+      stackMap->setBit(i);
+      hasRefsObserved = true;
+    }
+    i++;
+  }
+  MOZ_RELEASE_ASSERT(hasRefs == hasRefsObserved);
+
+  return stackMap;
+}
+
+// Create a stackmap from the given safepoint, with the structure:
+//
+//   <reg dump area, if trap>
+//   |       ++ <body (general spill)>
+//   |               ++ <space for Frame>
+//   |                       ++ <inbound args>
+//   |                                       |
+//   Lowest Addr                             Highest Addr
+//
+// The caller owns the resulting stackmap.  This assumes a grow-down stack.
+//
+// For non-debug builds, if the stackmap would contain no pointers, no
+// stackmap is created, and nullptr is returned.  For a debug build, a
+// stackmap is always created and returned.
+static bool CreateStackMapFromLSafepoint(LSafepoint& safepoint,
+                                         const MachineState& trapExitLayout,
+                                         size_t trapExitLayoutNumWords,
+                                         size_t nInboundStackArgBytes,
+                                         wasm::StackMap** result) {
+  // Ensure this is defined on all return paths.
+  *result = nullptr;
+
+  // The size of the wasm::Frame itself.
+  const size_t nFrameBytes = sizeof(wasm::Frame);
+
+  // This is the number of bytes in the general spill area, below the Frame.
+  const size_t nBodyBytes = safepoint.framePushedAtStackMapBase();
+
+  // This is the number of bytes in the general spill area, the Frame, and the
+  // incoming args, but not including any trap (register dump) area.
+  const size_t nNonTrapBytes = nBodyBytes + nFrameBytes + nInboundStackArgBytes;
+  MOZ_ASSERT(nNonTrapBytes % sizeof(void*) == 0);
+
+  // This is the total number of bytes covered by the map.
+  const DebugOnly<size_t> nTotalBytes = nNonTrapBytes +
+      (safepoint.isWasmTrap() ? (trapExitLayoutNumWords * sizeof(void*)) : 0);
+
+  // Create the stackmap initially in this vector.  Since most frames will
+  // contain 128 or fewer words, heap allocation is avoided in the majority of
+  // cases.  vec[0] is for the lowest address in the map, vec[N-1] is for the
+  // highest address in the map.
+  StackMapBoolVector vec;
+
+  // Keep track of whether we've actually seen any refs.
+  bool hasRefs = false;
+
+  // REG DUMP AREA, if any.
+  const LiveGeneralRegisterSet gcRegs = safepoint.gcRegs();
+  GeneralRegisterForwardIterator gcRegsIter(gcRegs);
+  if (safepoint.isWasmTrap()) {
+    // Deal with roots in registers.  This can only happen for safepoints
+    // associated with a trap.  For safepoints associated with a call, we
+    // don't expect to have any live values in registers, hence no roots in
+    // registers.
+    if (!vec.appendN(false, trapExitLayoutNumWords)) {
+      return false;
+    }
+    for (; gcRegsIter.more(); ++gcRegsIter) {
+      Register reg = *gcRegsIter;
+      size_t offsetFromTop =
+        reinterpret_cast<size_t>(trapExitLayout.address(reg));
+
+      // If this doesn't hold, the associated register wasn't saved by
+      // the trap exit stub.  Better to crash now than much later, in
+      // some obscure place, and possibly with security consequences.
+      MOZ_RELEASE_ASSERT(offsetFromTop < trapExitLayoutNumWords);
+
+      // offsetFromTop is an offset in words down from the highest
+      // address in the exit stub save area.  Switch it around to be an
+      // offset up from the bottom of the (integer register) save area.
+      size_t offsetFromBottom = trapExitLayoutNumWords - 1 - offsetFromTop;
+
+      vec[offsetFromBottom] = true;
+      hasRefs = true;
+    }
+  } else {
+    // This map is associated with a call instruction.  We expect there to be
+    // no live ref-carrying registers, and if there are we're in deep trouble.
+    MOZ_RELEASE_ASSERT(!gcRegsIter.more());
+  }
+
+  // BODY (GENERAL SPILL) AREA and FRAME and INCOMING ARGS
+  // Deal with roots on the stack.
+  size_t wordsSoFar = vec.length();
+  if (!vec.appendN(false, nNonTrapBytes / sizeof(void*))) {
+    return false;
+  }
+  const LSafepoint::SlotList& gcSlots = safepoint.gcSlots();
+  for (SafepointSlotEntry gcSlot : gcSlots) {
+    // The following needs to correspond with JitFrameLayout::slotRef
+    // gcSlot.stack == 0 means the slot is in the args area
+    if (gcSlot.stack) {
+      // It's a slot in the body allocation, so .slot is interpreted
+      // as an index downwards from the Frame*
+      MOZ_ASSERT(gcSlot.slot <= nBodyBytes);
+      uint32_t offsetInBytes = nBodyBytes - gcSlot.slot;
+      MOZ_ASSERT(offsetInBytes % sizeof(void*) == 0);
+      vec[wordsSoFar + offsetInBytes / sizeof(void*)] = true;
+    } else {
+      // It's an argument slot
+      MOZ_ASSERT(gcSlot.slot < nInboundStackArgBytes);
+      uint32_t offsetInBytes = nBodyBytes + nFrameBytes + gcSlot.slot;
+      MOZ_ASSERT(offsetInBytes % sizeof(void*) == 0);
+      vec[wordsSoFar + offsetInBytes / sizeof(void*)] = true;
+    }
+    hasRefs = true;
+  }
+
+#ifndef DEBUG
+  // We saw no references, and this is a non-debug build, so don't bother
+  // building the stackmap.
+  if (!hasRefs) {
+    return true;
+  }
+#endif
+
+  // Convert vec into a wasm::StackMap.
+  MOZ_ASSERT(vec.length() * sizeof(void*) == nTotalBytes);
+  wasm::StackMap* stackMap = ConvertStackMapBoolVectorToStackMap(vec, hasRefs);
+  if (!stackMap) {
+    return false;
+  }
+  if (safepoint.isWasmTrap()) {
+    stackMap->setExitStubWords(trapExitLayoutNumWords);
+  }
+
+  // Record in the map, how far down from the highest address the Frame* is.
+  // Take the opportunity to check that we haven't marked any part of the
+  // Frame itself as a pointer.
+  stackMap->setFrameOffsetFromTop((nInboundStackArgBytes + nFrameBytes)
+                                  / sizeof(void*));
+#ifdef DEBUG
+  for (uint32_t i = 0; i < nFrameBytes / sizeof(void*); i++) {
+    MOZ_ASSERT(stackMap->getBit(stackMap->numMappedWords -
+                                stackMap->frameOffsetFromTop + i) == 0);
+  }
+#endif
+
+  *result = stackMap;
+  return true;
+}
+
+// Generate a stackmap for a function's stack-overflow-at-entry trap, with
+// the structure:
+//
+//    <reg dump area>
+//    |       ++ <space reserved before trap, if any>
+//    |               ++ <space for Frame>
+//    |                       ++ <inbound arg area>
+//    |                                           |
+//    Lowest Addr                                 Highest Addr
+//
+// The caller owns the resulting stackmap.  This assumes a grow-down stack.
+//
+// For non-debug builds, if the stackmap would contain no pointers, no
+// stackmap is created, and nullptr is returned.  For a debug build, a
+// stackmap is always created and returned.
+//
+// The "space reserved before trap" is the space reserved by
+// MacroAssembler::wasmReserveStackChecked, in the case where the frame is
+// "small", as determined by that function.
+static bool CreateStackMapForFunctionEntryTrap(
+                const wasm::ValTypeVector& argTypes,
+                const MachineState& trapExitLayout,
+                size_t trapExitLayoutWords,
+                size_t nBytesReservedBeforeTrap,
+                size_t nInboundStackArgBytes,
+                wasm::StackMap** result) {
+  // Ensure this is defined on all return paths.
+  *result = nullptr;
+
+  // The size of the wasm::Frame itself.
+  const size_t nFrameBytes = sizeof(wasm::Frame);
+
+  // The size of the register dump (trap) area.
+  const size_t trapExitLayoutBytes = trapExitLayoutWords * sizeof(void*);
+
+  // This is the total number of bytes covered by the map.
+  const DebugOnly<size_t> nTotalBytes =
+      trapExitLayoutBytes + nBytesReservedBeforeTrap + nFrameBytes +
+      nInboundStackArgBytes;
+
+  // Create the stackmap initially in this vector.  Since most frames will
+  // contain 128 or fewer words, heap allocation is avoided in the majority of
+  // cases.  vec[0] is for the lowest address in the map, vec[N-1] is for the
+  // highest address in the map.
+  StackMapBoolVector vec;
+
+  // Keep track of whether we've actually seen any refs.
+  bool hasRefs = false;
+
+  // REG DUMP AREA
+  wasm::ExitStubMapVector trapExitExtras;
+  if (!GenerateStackmapEntriesForTrapExit(argTypes, trapExitLayout,
+                                          trapExitLayoutWords,
+                                          &trapExitExtras)) {
+    return false;
+  }
+  MOZ_ASSERT(trapExitExtras.length() == trapExitLayoutWords);
+
+  if (!vec.appendN(false, trapExitLayoutWords)) {
+    return false;
+  }
+  for (size_t i = 0; i < trapExitLayoutWords; i++) {
+    vec[i] = trapExitExtras[i];
+    hasRefs |= vec[i];
+  }
+
+  // SPACE RESERVED BEFORE TRAP
+  MOZ_ASSERT(nBytesReservedBeforeTrap % sizeof(void*) == 0);
+  if (!vec.appendN(false, nBytesReservedBeforeTrap / sizeof(void*))) {
+    return false;
+  }
+
+  // SPACE FOR FRAME
+  if (!vec.appendN(false, nFrameBytes / sizeof(void*))) {
+    return false;
+  }
+
+  // INBOUND ARG AREA
+  MOZ_ASSERT(nInboundStackArgBytes % sizeof(void*) == 0);
+  const size_t numStackArgWords = nInboundStackArgBytes / sizeof(void*);
+
+  const size_t wordsSoFar = vec.length();
+  if (!vec.appendN(false, numStackArgWords)) {
+    return false;
+  }
+
+  for (ABIArgIter<const wasm::ValTypeVector> i(argTypes); !i.done(); i++) {
+    ABIArg argLoc = *i;
+    const wasm::ValType& ty = argTypes[i.index()];
+    MOZ_ASSERT(ToMIRType(ty) != MIRType::Pointer);
+    if (argLoc.kind() != ABIArg::Stack || !ty.isReference()) {
+      continue;
+    }
+    uint32_t offset = argLoc.offsetFromArgBase();
+    MOZ_ASSERT(offset < nInboundStackArgBytes);
+    MOZ_ASSERT(offset % sizeof(void*) == 0);
+    vec[wordsSoFar + offset / sizeof(void*)] = true;
+    hasRefs = true;
+  }
+
+#ifndef DEBUG
+  // We saw no references, and this is a non-debug build, so don't bother
+  // building the stackmap.
+  if (!hasRefs) {
+    return true;
+  }
+#endif
+
+  // Convert vec into a wasm::StackMap.
+  MOZ_ASSERT(vec.length() * sizeof(void*) == nTotalBytes);
+  wasm::StackMap* stackMap = ConvertStackMapBoolVectorToStackMap(vec, hasRefs);
+  if (!stackMap) {
+    return false;
+  }
+  stackMap->setExitStubWords(trapExitLayoutWords);
+
+  stackMap->setFrameOffsetFromTop(nFrameBytes / sizeof(void*)
+                                  + numStackArgWords);
+#ifdef DEBUG
+  for (uint32_t i = 0; i < nFrameBytes / sizeof(void*); i++) {
+    MOZ_ASSERT(stackMap->getBit(stackMap->numMappedWords -
+                                stackMap->frameOffsetFromTop + i) == 0);
+  }
+#endif
+
+  *result = stackMap;
+  return true;
+}
+
 bool CodeGenerator::generateWasm(wasm::FuncTypeIdDesc funcTypeId,
                                  wasm::BytecodeOffset trapOffset,
-                                 wasm::FuncOffsets* offsets) {
+                                 const wasm::ValTypeVector& argTypes,
+                                 const MachineState& trapExitLayout,
+                                 size_t trapExitLayoutNumWords,
+                                 wasm::FuncOffsets* offsets,
+                                 wasm::StackMaps* stackMaps) {
   JitSpew(JitSpew_Codegen, "# Emitting wasm code");
 
+  size_t nInboundStackArgBytes = StackArgAreaSizeUnaligned(argTypes);
+
   wasm::GenerateFunctionPrologue(masm, funcTypeId, mozilla::Nothing(), offsets);
 
+  MOZ_ASSERT(masm.framePushed() == 0);
+
   if (omitOverRecursedCheck()) {
     masm.reserveStack(frameSize());
   } else {
-    masm.wasmReserveStackChecked(frameSize(), trapOffset);
+    std::pair<CodeOffset, uint32_t> pair =
+        masm.wasmReserveStackChecked(frameSize(), trapOffset);
+    CodeOffset trapInsnOffset = pair.first;
+    size_t nBytesReservedBeforeTrap = pair.second;
+
+    wasm::StackMap* functionEntryStackMap = nullptr;
+    if (!CreateStackMapForFunctionEntryTrap(argTypes, trapExitLayout,
+            trapExitLayoutNumWords, nBytesReservedBeforeTrap,
+            nInboundStackArgBytes, &functionEntryStackMap)) {
+      return false;
+    }
+    // In debug builds, we'll always have a stack map, even if there are no
+    // refs to track.
+    MOZ_ALWAYS_TRUE(functionEntryStackMap);
+    if (functionEntryStackMap &&
+        !stackMaps->add((uint8_t*)(uintptr_t)trapInsnOffset.offset(),
+                        functionEntryStackMap)) {
+      functionEntryStackMap->destroy();
+      return false;
+    }
   }
 
   MOZ_ASSERT(masm.framePushed() == frameSize());
 
   if (!generateBody()) {
     return false;
   }
 
@@ -10242,21 +10581,40 @@ bool CodeGenerator::generateWasm(wasm::F
   offsets->end = masm.currentOffset();
 
   MOZ_ASSERT(!masm.failureLabel()->used());
   MOZ_ASSERT(snapshots_.listSize() == 0);
   MOZ_ASSERT(snapshots_.RVATableSize() == 0);
   MOZ_ASSERT(recovers_.size() == 0);
   MOZ_ASSERT(bailouts_.empty());
   MOZ_ASSERT(graph.numConstants() == 0);
-  MOZ_ASSERT(safepointIndices_.empty());
   MOZ_ASSERT(osiIndices_.empty());
   MOZ_ASSERT(icList_.empty());
   MOZ_ASSERT(safepoints_.size() == 0);
   MOZ_ASSERT(!scriptCounts_);
+
+  // Convert the safepoints to stackmaps and add them to our running
+  // collection thereof.
+  for (SafepointIndex& index : safepointIndices_) {
+    wasm::StackMap* stackMap = nullptr;
+    if (!CreateStackMapFromLSafepoint(*index.safepoint(), trapExitLayout,
+            trapExitLayoutNumWords, nInboundStackArgBytes, &stackMap)) {
+      return false;
+    }
+    // In debug builds, we'll always have a stack map.
+    MOZ_ALWAYS_TRUE(stackMap);
+    if (!stackMap) {
+      continue;
+    }
+    if (!stackMaps->add((uint8_t*)(uintptr_t)index.displacement(), stackMap)) {
+      stackMap->destroy();
+      return false;
+    }
+  }
+
   return true;
 }
 
 bool CodeGenerator::generate() {
   JitSpew(JitSpew_Codegen, "# Emitting code for script %s:%u:%u",
           gen->info().script()->filename(), gen->info().script()->lineno(),
           gen->info().script()->column());
 
@@ -12989,16 +13347,24 @@ void CodeGenerator::visitInterruptCheck(
   masm.bind(ool->rejoin());
 }
 
 void CodeGenerator::visitWasmInterruptCheck(LWasmInterruptCheck* lir) {
   MOZ_ASSERT(gen->compilingWasm());
 
   masm.wasmInterruptCheck(ToRegister(lir->tlsPtr()),
                           lir->mir()->bytecodeOffset());
+
+  markSafepointAt(masm.currentOffset(), lir);
+
+  // Note that masm.framePushed() doesn't include the register dump area.
+  // That will be taken into account when the StackMap is created from the
+  // LSafepoint.
+  lir->safepoint()->setFramePushedAtStackMapBase(masm.framePushed());
+  lir->safepoint()->setIsWasmTrap();
 }
 
 void CodeGenerator::visitWasmTrap(LWasmTrap* lir) {
   MOZ_ASSERT(gen->compilingWasm());
   const MWasmTrap* mir = lir->mir();
 
   masm.wasmTrap(mir->trap(), mir->bytecodeOffset());
 }
--- a/js/src/jit/CodeGenerator.h
+++ b/js/src/jit/CodeGenerator.h
@@ -25,16 +25,18 @@
 #elif defined(JS_CODEGEN_MIPS64)
 #  include "jit/mips64/CodeGenerator-mips64.h"
 #elif defined(JS_CODEGEN_NONE)
 #  include "jit/none/CodeGenerator-none.h"
 #else
 #  error "Unknown architecture!"
 #endif
 
+#include "wasm/WasmGC.h"
+
 namespace js {
 namespace jit {
 
 enum class SwitchTableType { Inline, OutOfLine };
 
 template <SwitchTableType tableType>
 class OutOfLineSwitch;
 class OutOfLineTestObject;
@@ -68,17 +70,21 @@ class CodeGenerator final : public CodeG
  public:
   CodeGenerator(MIRGenerator* gen, LIRGraph* graph,
                 MacroAssembler* masm = nullptr);
   ~CodeGenerator();
 
   MOZ_MUST_USE bool generate();
   MOZ_MUST_USE bool generateWasm(wasm::FuncTypeIdDesc funcTypeId,
                                  wasm::BytecodeOffset trapOffset,
-                                 wasm::FuncOffsets* offsets);
+                                 const wasm::ValTypeVector& argTys,
+                                 const MachineState& trapExitLayout,
+                                 size_t trapExitLayoutNumWords,
+                                 wasm::FuncOffsets* offsets,
+                                 wasm::StackMaps* stackMaps);
 
   MOZ_MUST_USE bool link(JSContext* cx, CompilerConstraintList* constraints);
 
   void emitOOLTestObject(Register objreg, Label* ifTruthy, Label* ifFalsy,
                          Register scratch);
   void emitIntToString(Register input, Register output, Label* ool);
 
   void visitOutOfLineRegExpMatcher(OutOfLineRegExpMatcher* ool);
@@ -230,17 +236,18 @@ class CodeGenerator final : public CodeG
                                  Register temp1, Register temp2);
 
   template <class IteratorObject, class OrderedHashTable>
   void emitGetNextEntryForIterator(LGetNextEntryForIterator* lir);
 
   template <class OrderedHashTable>
   void emitLoadIteratorValues(Register result, Register temp, Register front);
 
-  void emitWasmCallBase(MWasmCall* mir, bool needsBoundsCheck);
+  template <size_t Defs>
+  void emitWasmCallBase(LWasmCallBase<Defs>* lir);
 
   template <size_t NumDefs>
   void emitIonToWasmCallBase(LIonToWasmCallBase<NumDefs>* lir);
 
   IonScriptCounts* maybeCreateScriptCounts();
 
   // This function behaves like testValueTruthy with the exception that it can
   // choose to let control flow fall through when the object is truthy, as
--- a/js/src/jit/LIR.cpp
+++ b/js/src/jit/LIR.cpp
@@ -161,16 +161,19 @@ LMoveGroup* LBlock::getExitMoveGroup(Tem
 void LBlock::dump(GenericPrinter& out) {
   out.printf("block%u:\n", mir()->id());
   for (size_t i = 0; i < numPhis(); ++i) {
     getPhi(i)->dump(out);
     out.printf("\n");
   }
   for (LInstructionIterator iter = begin(); iter != end(); iter++) {
     iter->dump(out);
+    if (iter->safepoint()) {
+      out.printf(" SAFEPOINT(0x%p) ", iter->safepoint());
+    }
     out.printf("\n");
   }
 }
 
 void LBlock::dump() {
   Fprinter out(stderr);
   dump(out);
   out.finish();
--- a/js/src/jit/LIR.h
+++ b/js/src/jit/LIR.h
@@ -1347,16 +1347,35 @@ class LSafepoint : public TempObject {
 #endif
 
   // The subset of liveRegs which contains pointers to slots/elements.
   LiveGeneralRegisterSet slotsOrElementsRegs_;
 
   // List of slots which have slots/elements pointers.
   SlotList slotsOrElementsSlots_;
 
+  // Wasm only: with what kind of instruction is this LSafepoint associated?
+  // true => wasm trap, false => wasm call.
+  bool isWasmTrap_;
+
+  // Wasm only: what is the value of masm.framePushed() that corresponds to
+  // the lowest-addressed word covered by the StackMap that we will generate
+  // from this LSafepoint?  This depends on the instruction:
+  //
+  // if isWasmTrap_ == true:
+  //    masm.framePushed() unmodified.  Note that when constructing the
+  //    StackMap we will add entries below this point to take account of
+  //    registers dumped on the stack as a result of the trap.
+  //
+  // if isWasmTrap_ == false:
+  //    masm.framePushed() - StackArgAreaSizeUnaligned(arg types for the call),
+  //    because the map does not include the outgoing args themselves, but
+  //    it does cover any and all alignment space above them.
+  uint32_t framePushedAtStackMapBase_;
+
  public:
   void assertInvariants() {
     // Every register in valueRegs and gcRegs should also be in liveRegs.
 #ifndef JS_NUNBOX32
     MOZ_ASSERT((valueRegs().bits() & ~liveRegs().gprs().bits()) == 0);
 #endif
     MOZ_ASSERT((gcRegs().bits() & ~liveRegs().gprs().bits()) == 0);
   }
@@ -1366,17 +1385,19 @@ class LSafepoint : public TempObject {
         osiCallPointOffset_(0),
         gcSlots_(alloc),
         valueSlots_(alloc)
 #ifdef JS_NUNBOX32
         ,
         nunboxParts_(alloc)
 #endif
         ,
-        slotsOrElementsSlots_(alloc) {
+        slotsOrElementsSlots_(alloc),
+        isWasmTrap_(false),
+        framePushedAtStackMapBase_(0) {
     assertInvariants();
   }
   void addLiveRegister(AnyRegister reg) {
     liveRegs_.addUnchecked(reg);
     assertInvariants();
   }
   const LiveRegisterSet& liveRegs() const { return liveRegs_; }
 #ifdef CHECK_OSIPOINT_REGISTERS
@@ -1609,16 +1630,27 @@ class LSafepoint : public TempObject {
     // would be wrong.
     return osiCallPointOffset_ + Assembler::PatchWrite_NearCallSize();
   }
   uint32_t osiCallPointOffset() const { return osiCallPointOffset_; }
   void setOsiCallPointOffset(uint32_t osiCallPointOffset) {
     MOZ_ASSERT(!osiCallPointOffset_);
     osiCallPointOffset_ = osiCallPointOffset;
   }
+
+  bool isWasmTrap() const { return isWasmTrap_; }
+  void setIsWasmTrap() { isWasmTrap_ = true; }
+
+  uint32_t framePushedAtStackMapBase() const {
+    return framePushedAtStackMapBase_;
+  }
+  void setFramePushedAtStackMapBase(uint32_t n) {
+    MOZ_ASSERT(framePushedAtStackMapBase_ == 0);
+    framePushedAtStackMapBase_ = n;
+  }
 };
 
 class LInstruction::InputIterator {
  private:
   LInstruction& ins_;
   size_t idx_;
   bool snapshot_;
 
--- a/js/src/jit/Lowering.cpp
+++ b/js/src/jit/Lowering.cpp
@@ -2550,16 +2550,18 @@ void LIRGenerator::visitInterruptCheck(M
   add(lir, ins);
   assignSafepoint(lir, ins);
 }
 
 void LIRGenerator::visitWasmInterruptCheck(MWasmInterruptCheck* ins) {
   auto* lir =
       new (alloc()) LWasmInterruptCheck(useRegisterAtStart(ins->tlsPtr()));
   add(lir, ins);
+
+  assignWasmSafepoint(lir, ins);
 }
 
 void LIRGenerator::visitWasmTrap(MWasmTrap* ins) {
   add(new (alloc()) LWasmTrap, ins);
 }
 
 void LIRGenerator::visitWasmReinterpret(MWasmReinterpret* ins) {
   if (ins->type() == MIRType::Int64) {
@@ -4508,16 +4510,18 @@ void LIRGenerator::visitWasmCall(MWasmCa
     return;
   }
 
   if (ins->type() == MIRType::None) {
     add(lir, ins);
   } else {
     defineReturn(lir, ins);
   }
+
+  assignWasmSafepoint(lir, ins);
 }
 
 void LIRGenerator::visitSetDOMProperty(MSetDOMProperty* ins) {
   MDefinition* val = ins->value();
 
   Register cxReg, objReg, privReg, valueReg;
   GetTempRegForIntArg(0, 0, &cxReg);
   GetTempRegForIntArg(1, 0, &objReg);
--- a/js/src/jit/MIR.cpp
+++ b/js/src/jit/MIR.cpp
@@ -603,24 +603,26 @@ void MTest::filtersUndefinedOrNull(bool 
 
 #ifdef JS_JITSPEW
 void MDefinition::printOpcode(GenericPrinter& out) const {
   PrintOpcodeName(out, op());
   for (size_t j = 0, e = numOperands(); j < e; j++) {
     out.printf(" ");
     if (getUseFor(j)->hasProducer()) {
       getOperand(j)->printName(out);
+      out.printf(":%s", StringFromMIRType(getOperand(j)->type()));
     } else {
       out.printf("(null)");
     }
   }
 }
 
 void MDefinition::dump(GenericPrinter& out) const {
   printName(out);
+  out.printf(":%s", StringFromMIRType(type()));
   out.printf(" = ");
   printOpcode(out);
   out.printf("\n");
 
   if (isInstruction()) {
     if (MResumePoint* resume = toInstruction()->resumePoint()) {
       resume->dump(out);
     }
@@ -1241,17 +1243,17 @@ HashNumber MWasmFloatConstant::valueHash
 }
 
 bool MWasmFloatConstant::congruentTo(const MDefinition* ins) const {
   return ins->isWasmFloatConstant() && type() == ins->type() &&
          u.bits_ == ins->toWasmFloatConstant()->u.bits_;
 }
 
 HashNumber MWasmNullConstant::valueHash() const {
-  return ConstantValueHash(MIRType::Pointer, 0);
+  return ConstantValueHash(MIRType::RefOrNull, 0);
 }
 
 #ifdef JS_JITSPEW
 void MControlInstruction::printOpcode(GenericPrinter& out) const {
   MDefinition::printOpcode(out);
   for (size_t j = 0; j < numSuccessors(); j++) {
     if (getSuccessor(j)) {
       out.printf(" block%u", getSuccessor(j)->id());
@@ -5487,18 +5489,21 @@ MDefinition* MWasmUnsignedToFloat32::fol
     }
   }
 
   return this;
 }
 
 MWasmCall* MWasmCall::New(TempAllocator& alloc, const wasm::CallSiteDesc& desc,
                           const wasm::CalleeDesc& callee, const Args& args,
-                          MIRType resultType, MDefinition* tableIndex) {
-  MWasmCall* call = new (alloc) MWasmCall(desc, callee);
+                          MIRType resultType,
+                          uint32_t stackArgAreaSizeUnaligned,
+                          MDefinition* tableIndex) {
+  MWasmCall* call = new (alloc) MWasmCall(desc, callee,
+                                          stackArgAreaSizeUnaligned);
   call->setResultType(resultType);
 
   if (!call->argRegs_.init(alloc, args.length())) {
     return nullptr;
   }
   for (size_t i = 0; i < call->argRegs_.length(); i++) {
     call->argRegs_[i] = args[i].reg;
   }
@@ -5516,20 +5521,21 @@ MWasmCall* MWasmCall::New(TempAllocator&
   }
 
   return call;
 }
 
 MWasmCall* MWasmCall::NewBuiltinInstanceMethodCall(
     TempAllocator& alloc, const wasm::CallSiteDesc& desc,
     const wasm::SymbolicAddress builtin, const ABIArg& instanceArg,
-    const Args& args, MIRType resultType) {
+    const Args& args, MIRType resultType, uint32_t stackArgAreaSizeUnaligned) {
   auto callee = wasm::CalleeDesc::builtinInstanceMethod(builtin);
   MWasmCall* call =
-      MWasmCall::New(alloc, desc, callee, args, resultType, nullptr);
+      MWasmCall::New(alloc, desc, callee, args, resultType,
+                     stackArgAreaSizeUnaligned, nullptr);
   if (!call) {
     return nullptr;
   }
 
   MOZ_ASSERT(instanceArg != ABIArg());
   call->instanceArg_ = instanceArg;
   return call;
 }
--- a/js/src/jit/MIR.h
+++ b/js/src/jit/MIR.h
@@ -12023,47 +12023,54 @@ class MWasmStackArg : public MUnaryInstr
   uint32_t spOffset() const { return spOffset_; }
   void incrementOffset(uint32_t inc) { spOffset_ += inc; }
 };
 
 class MWasmCall final : public MVariadicInstruction, public NoTypePolicy::Data {
   wasm::CallSiteDesc desc_;
   wasm::CalleeDesc callee_;
   FixedList<AnyRegister> argRegs_;
+  uint32_t stackArgAreaSizeUnaligned_;
   ABIArg instanceArg_;
 
-  MWasmCall(const wasm::CallSiteDesc& desc, const wasm::CalleeDesc& callee)
-      : MVariadicInstruction(classOpcode), desc_(desc), callee_(callee) {}
+  MWasmCall(const wasm::CallSiteDesc& desc, const wasm::CalleeDesc& callee,
+            uint32_t stackArgAreaSizeUnaligned)
+      : MVariadicInstruction(classOpcode), desc_(desc), callee_(callee),
+        stackArgAreaSizeUnaligned_(stackArgAreaSizeUnaligned) {}
 
  public:
   INSTRUCTION_HEADER(WasmCall)
 
   struct Arg {
     AnyRegister reg;
     MDefinition* def;
     Arg(AnyRegister reg, MDefinition* def) : reg(reg), def(def) {}
   };
   typedef Vector<Arg, 8, SystemAllocPolicy> Args;
 
   static MWasmCall* New(TempAllocator& alloc, const wasm::CallSiteDesc& desc,
                         const wasm::CalleeDesc& callee, const Args& args,
-                        MIRType resultType, MDefinition* tableIndex = nullptr);
+                        MIRType resultType, uint32_t stackArgAreaSizeUnaligned,
+                        MDefinition* tableIndex = nullptr);
 
   static MWasmCall* NewBuiltinInstanceMethodCall(
       TempAllocator& alloc, const wasm::CallSiteDesc& desc,
       const wasm::SymbolicAddress builtin, const ABIArg& instanceArg,
-      const Args& args, MIRType resultType);
+      const Args& args, MIRType resultType, uint32_t stackArgAreaSizeUnaligned);
 
   size_t numArgs() const { return argRegs_.length(); }
   AnyRegister registerForArg(size_t index) const {
     MOZ_ASSERT(index < numArgs());
     return argRegs_[index];
   }
   const wasm::CallSiteDesc& desc() const { return desc_; }
   const wasm::CalleeDesc& callee() const { return callee_; }
+  uint32_t stackArgAreaSizeUnaligned() const {
+    return stackArgAreaSizeUnaligned_;
+  }
 
   bool possiblyCalls() const override { return true; }
 
   const ABIArg& instanceArg() const { return instanceArg_; }
 };
 
 class MWasmSelect : public MTernaryInstruction, public NoTypePolicy::Data {
   MWasmSelect(MDefinition* trueExpr, MDefinition* falseExpr,
--- a/js/src/jit/MacroAssembler.cpp
+++ b/js/src/jit/MacroAssembler.cpp
@@ -3422,41 +3422,43 @@ void MacroAssembler::wasmInterruptCheck(
                                         wasm::BytecodeOffset bytecodeOffset) {
   Label ok;
   branch32(Assembler::Equal, Address(tls, offsetof(wasm::TlsData, interrupt)),
            Imm32(0), &ok);
   wasmTrap(wasm::Trap::CheckInterrupt, bytecodeOffset);
   bind(&ok);
 }
 
-void MacroAssembler::wasmReserveStackChecked(uint32_t amount,
-                                             wasm::BytecodeOffset trapOffset) {
-  // If the frame is large, don't bump sp until after the stack limit check so
-  // that the trap handler isn't called with a wild sp.
-
+std::pair<CodeOffset, uint32_t>
+MacroAssembler::wasmReserveStackChecked(uint32_t amount,
+                                        wasm::BytecodeOffset trapOffset) {
   if (amount > MAX_UNCHECKED_LEAF_FRAME_SIZE) {
+    // The frame is large.  Don't bump sp until after the stack limit check so
+    // that the trap handler isn't called with a wild sp.
     Label ok;
     Register scratch = ABINonArgReg0;
     moveStackPtrTo(scratch);
     subPtr(Address(WasmTlsReg, offsetof(wasm::TlsData, stackLimit)), scratch);
     branchPtr(Assembler::GreaterThan, scratch, Imm32(amount), &ok);
     wasmTrap(wasm::Trap::StackOverflow, trapOffset);
+    CodeOffset trapInsnOffset = CodeOffset(currentOffset());
     bind(&ok);
+    reserveStack(amount);
+    return std::pair<CodeOffset, uint32_t>(trapInsnOffset, 0);
   }
 
   reserveStack(amount);
-
-  if (amount <= MAX_UNCHECKED_LEAF_FRAME_SIZE) {
-    Label ok;
-    branchStackPtrRhs(Assembler::Below,
-                      Address(WasmTlsReg, offsetof(wasm::TlsData, stackLimit)),
-                      &ok);
-    wasmTrap(wasm::Trap::StackOverflow, trapOffset);
-    bind(&ok);
-  }
+  Label ok;
+  branchStackPtrRhs(Assembler::Below,
+                    Address(WasmTlsReg, offsetof(wasm::TlsData, stackLimit)),
+                    &ok);
+  wasmTrap(wasm::Trap::StackOverflow, trapOffset);
+  CodeOffset trapInsnOffset = CodeOffset(currentOffset());
+  bind(&ok);
+  return std::pair<CodeOffset,uint32_t>(trapInsnOffset, amount);
 }
 
 CodeOffset MacroAssembler::wasmCallImport(const wasm::CallSiteDesc& desc,
                                           const wasm::CalleeDesc& callee) {
   // Load the callee, before the caller's registers are clobbered.
   uint32_t globalDataOffset = callee.importGlobalDataOffset();
   loadWasmGlobalPtr(globalDataOffset + offsetof(wasm::FuncImportTls, code),
                     ABINonArgReg0);
--- a/js/src/jit/MacroAssembler.h
+++ b/js/src/jit/MacroAssembler.h
@@ -1753,18 +1753,22 @@ class MacroAssembler : public MacroAssem
  public:
   // ========================================================================
   // wasm support
 
   CodeOffset wasmTrapInstruction() PER_SHARED_ARCH;
 
   void wasmTrap(wasm::Trap trap, wasm::BytecodeOffset bytecodeOffset);
   void wasmInterruptCheck(Register tls, wasm::BytecodeOffset bytecodeOffset);
-  void wasmReserveStackChecked(uint32_t amount,
-                               wasm::BytecodeOffset trapOffset);
+
+  // Returns a pair: the offset of the undefined (trapping) instruction, and
+  // the number of extra bytes of stack allocated prior to the trap
+  // instruction proper.
+  std::pair<CodeOffset, uint32_t>
+  wasmReserveStackChecked(uint32_t amount, wasm::BytecodeOffset trapOffset);
 
   // Emit a bounds check against the wasm heap limit, jumping to 'label' if
   // 'cond' holds. If JitOptions.spectreMaskIndex is true, in speculative
   // executions 'index' is saturated in-place to 'boundsCheckLimit'.
   void wasmBoundsCheck(Condition cond, Register index,
                        Register boundsCheckLimit, Label* label)
       DEFINED_ON(arm, arm64, mips32, mips64, x86_shared);
 
--- a/js/src/jit/shared/Lowering-shared.cpp
+++ b/js/src/jit/shared/Lowering-shared.cpp
@@ -282,8 +282,21 @@ void LIRGeneratorShared::assignSafepoint
 
   osiPoint_ = new (alloc()) LOsiPoint(ins->safepoint(), postSnapshot);
 
   if (!lirGraph_.noteNeedsSafepoint(ins)) {
     abort(AbortReason::Alloc, "noteNeedsSafepoint failed");
     return;
   }
 }
+
+void LIRGeneratorShared::assignWasmSafepoint(LInstruction* ins,
+                                             MInstruction* mir) {
+  MOZ_ASSERT(!osiPoint_);
+  MOZ_ASSERT(!ins->safepoint());
+
+  ins->initSafepoint(alloc());
+
+  if (!lirGraph_.noteNeedsSafepoint(ins)) {
+    abort(AbortReason::Alloc, "noteNeedsSafepoint failed");
+    return;
+  }
+}
--- a/js/src/jit/shared/Lowering-shared.h
+++ b/js/src/jit/shared/Lowering-shared.h
@@ -318,16 +318,19 @@ class LIRGeneratorShared {
   void assignSnapshot(LInstruction* ins, BailoutKind kind);
 
   // Marks this instruction as needing to call into either the VM or GC. This
   // function may build a snapshot that captures the result of its own
   // instruction, and as such, should generally be called after define*().
   void assignSafepoint(LInstruction* ins, MInstruction* mir,
                        BailoutKind kind = Bailout_DuringVMCall);
 
+  // Marks this instruction as needing a wasm safepoint.
+  void assignWasmSafepoint(LInstruction* ins, MInstruction* mir);
+
   void lowerConstantDouble(double d, MInstruction* mir) {
     define(new (alloc()) LDouble(d), mir);
   }
   void lowerConstantFloat32(float f, MInstruction* mir) {
     define(new (alloc()) LFloat32(f), mir);
   }
 
  public:
--- a/js/src/wasm/WasmBaselineCompile.cpp
+++ b/js/src/wasm/WasmBaselineCompile.cpp
@@ -2150,49 +2150,19 @@ struct StackMapGenerator {
   // stub, as generated by GenerateTrapExit().
   //
   // The resulting map must correspond precisely with the stack layout
   // created for the integer registers as saved by (code generated by)
   // GenerateTrapExit().  To do that we use trapExitLayout_ and
   // trapExitLayoutNumWords_, which together comprise a description of the
   // layout and are created by GenerateTrapExitMachineState().
   MOZ_MUST_USE bool generateStackmapEntriesForTrapExit(
-      const ValTypeVector& args, ExitStubMapVector& extras) {
-    MOZ_ASSERT(extras.empty());
-
-    // If this doesn't hold, we can't distinguish saved and not-saved
-    // registers in the MachineState.  See MachineState::MachineState().
-    MOZ_ASSERT(trapExitLayoutNumWords_ < 0x100);
-
-    if (!extras.appendN(false, trapExitLayoutNumWords_)) {
-      return false;
-    }
-
-    for (ABIArgIter<const ValTypeVector> i(args); !i.done(); i++) {
-      if (!i->argInRegister() || i.mirType() != MIRType::RefOrNull) {
-        continue;
-      }
-
-      size_t offsetFromTop =
-          reinterpret_cast<size_t>(trapExitLayout_.address(i->gpr()));
-
-      // If this doesn't hold, the associated register wasn't saved by
-      // the trap exit stub.  Better to crash now than much later, in
-      // some obscure place, and possibly with security consequences.
-      MOZ_RELEASE_ASSERT(offsetFromTop < trapExitLayoutNumWords_);
-
-      // offsetFromTop is an offset in words down from the highest
-      // address in the exit stub save area.  Switch it around to be an
-      // offset up from the bottom of the (integer register) save area.
-      size_t offsetFromBottom = trapExitLayoutNumWords_ - 1 - offsetFromTop;
-
-      extras[offsetFromBottom] = true;
-    }
-
-    return true;
+      const ValTypeVector& args, ExitStubMapVector* extras) {
+    return GenerateStackmapEntriesForTrapExit(args, trapExitLayout_,
+                                              trapExitLayoutNumWords_, extras);
   }
 
   // Creates a stackmap associated with the instruction denoted by
   // |assemblerOffset|, incorporating pointers from the current operand
   // stack |stk|, incorporating possible extra pointers in |extra| at the
   // lower addressed end, and possibly with the associated frame having a
   // ref-typed DebugFrame as indicated by |refDebugFrame|.
   MOZ_MUST_USE bool createStackMap(const char* who,
@@ -4140,17 +4110,17 @@ class BaseCompiler final : public BaseCo
     }
 
     // Generate a stack-overflow check and its associated stack map.
 
     fr.checkStack(ABINonArgReg0, BytecodeOffset(func_.lineOrBytecode));
 
     const ValTypeVector& args = funcType().args();
     ExitStubMapVector extras;
-    if (!stackMapGenerator_.generateStackmapEntriesForTrapExit(args, extras)) {
+    if (!stackMapGenerator_.generateStackmapEntriesForTrapExit(args, &extras)) {
       return false;
     }
     if (!createStackMap("stack check", extras, masm.currentOffset(),
                         HasRefTypedDebugFrame::No)) {
       return false;
     }
 
     size_t reservedBytes = fr.fixedAllocSize() - masm.framePushed();
--- a/js/src/wasm/WasmCode.h
+++ b/js/src/wasm/WasmCode.h
@@ -18,16 +18,17 @@
 
 #ifndef wasm_code_h
 #define wasm_code_h
 
 #include "jit/shared/Assembler-shared.h"
 #include "js/HashTable.h"
 #include "threading/ExclusiveData.h"
 #include "vm/MutexIDs.h"
+#include "wasm/WasmGC.h"
 #include "wasm/WasmTypes.h"
 
 namespace js {
 
 struct AsmJSMetadata;
 
 namespace wasm {
 
--- a/js/src/wasm/WasmGC.cpp
+++ b/js/src/wasm/WasmGC.cpp
@@ -18,16 +18,54 @@
 
 #include "wasm/WasmGC.h"
 #include "wasm/WasmInstance.h"
 #include "jit/MacroAssembler-inl.h"
 
 namespace js {
 namespace wasm {
 
+bool GenerateStackmapEntriesForTrapExit(
+    const ValTypeVector& args, const MachineState& trapExitLayout,
+    const size_t trapExitLayoutNumWords,  ExitStubMapVector* extras) {
+  MOZ_ASSERT(extras->empty());
+
+  // If this doesn't hold, we can't distinguish saved and not-saved
+  // registers in the MachineState.  See MachineState::MachineState().
+  MOZ_ASSERT(trapExitLayoutNumWords < 0x100);
+
+  if (!extras->appendN(false, trapExitLayoutNumWords)) {
+    return false;
+  }
+
+  for (ABIArgIter<const ValTypeVector> i(args); !i.done(); i++) {
+    MOZ_ASSERT(i.mirType() != MIRType::Pointer);
+    if (!i->argInRegister() || i.mirType() != MIRType::RefOrNull) {
+      continue;
+    }
+
+    size_t offsetFromTop =
+       reinterpret_cast<size_t>(trapExitLayout.address(i->gpr()));
+
+    // If this doesn't hold, the associated register wasn't saved by
+    // the trap exit stub.  Better to crash now than much later, in
+    // some obscure place, and possibly with security consequences.
+    MOZ_RELEASE_ASSERT(offsetFromTop < trapExitLayoutNumWords);
+
+    // offsetFromTop is an offset in words down from the highest
+    // address in the exit stub save area.  Switch it around to be an
+    // offset up from the bottom of the (integer register) save area.
+    size_t offsetFromBottom = trapExitLayoutNumWords - 1 - offsetFromTop;
+
+    (*extras)[offsetFromBottom] = true;
+  }
+
+  return true;
+}
+
 void EmitWasmPreBarrierGuard(MacroAssembler& masm, Register tls,
                              Register scratch, Register valueAddr,
                              Label* skipBarrier) {
   // If no incremental GC has started, we don't need the barrier.
   masm.loadPtr(
     Address(tls, offsetof(TlsData, addressOfNeedsIncrementalBarrier)),
     scratch);
   masm.branchTest32(Assembler::Zero, Address(scratch, 0), Imm32(0x1),
--- a/js/src/wasm/WasmGC.h
+++ b/js/src/wasm/WasmGC.h
@@ -21,16 +21,226 @@
 
 #include "jit/MacroAssembler.h"
 
 namespace js {
 namespace wasm {
 
 using namespace js::jit;
 
+// Definitions for stack maps.
+
+typedef Vector<bool, 32, SystemAllocPolicy> ExitStubMapVector;
+
+struct StackMap final {
+  // A StackMap is a bit-array containing numMappedWords bits, one bit per
+  // word of stack.  Bit index zero is for the lowest addressed word in the
+  // range.
+  //
+  // This is a variable-length structure whose size must be known at creation
+  // time.
+  //
+  // Users of the map will know the address of the wasm::Frame that is covered
+  // by this map.  In order that they can calculate the exact address range
+  // covered by the map, the map also stores the offset, from the highest
+  // addressed word of the map, of the embedded wasm::Frame.  This is an
+  // offset down from the highest address, rather than up from the lowest, so
+  // as to limit its range to 11 bits, where
+  // 11 == ceil(log2(MaxParams * sizeof-biggest-param-type-in-words))
+  //
+  // The map may also cover a ref-typed DebugFrame.  If so that can be noted,
+  // since users of the map need to trace pointers in such a DebugFrame.
+  //
+  // Finally, for sanity checking only, for stack maps associated with a wasm
+  // trap exit stub, the number of words used by the trap exit stub save area
+  // is also noted.  This is used in Instance::traceFrame to check that the
+  // TrapExitDummyValue is in the expected place in the frame.
+
+  // The total number of stack words covered by the map ..
+  uint32_t numMappedWords : 30;
+
+  // .. of which this many are "exit stub" extras
+  uint32_t numExitStubWords : 6;
+
+  // Where is Frame* relative to the top?  This is an offset in words.
+  uint32_t frameOffsetFromTop : 11;
+
+  // Notes the presence of a ref-typed DebugFrame.
+  uint32_t hasRefTypedDebugFrame : 1;
+
+ private:
+  static constexpr uint32_t maxMappedWords = (1 << 30) - 1;
+  static constexpr uint32_t maxExitStubWords = (1 << 6) - 1;
+  static constexpr uint32_t maxFrameOffsetFromTop = (1 << 11) - 1;
+
+  uint32_t bitmap[1];
+
+  explicit StackMap(uint32_t numMappedWords)
+      : numMappedWords(numMappedWords),
+        numExitStubWords(0),
+        frameOffsetFromTop(0),
+        hasRefTypedDebugFrame(0) {
+    const uint32_t nBitmap = calcNBitmap(numMappedWords);
+    memset(bitmap, 0, nBitmap * sizeof(bitmap[0]));
+  }
+
+ public:
+  static StackMap* create(uint32_t numMappedWords) {
+    uint32_t nBitmap = calcNBitmap(numMappedWords);
+    char* buf =
+        (char*)js_malloc(sizeof(StackMap) + (nBitmap - 1) * sizeof(bitmap[0]));
+    if (!buf) {
+      return nullptr;
+    }
+    return ::new (buf) StackMap(numMappedWords);
+  }
+
+  void destroy() { js_free((char*)this); }
+
+  // Record the number of words in the map used as a wasm trap exit stub
+  // save area.  See comment above.
+  void setExitStubWords(uint32_t nWords) {
+    MOZ_ASSERT(numExitStubWords == 0);
+    MOZ_RELEASE_ASSERT(nWords <= maxExitStubWords);
+    MOZ_ASSERT(nWords <= numMappedWords);
+    numExitStubWords = nWords;
+  }
+
+  // Record the offset from the highest-addressed word of the map, that the
+  // wasm::Frame lives at.  See comment above.
+  void setFrameOffsetFromTop(uint32_t nWords) {
+    MOZ_ASSERT(frameOffsetFromTop == 0);
+    MOZ_RELEASE_ASSERT(nWords <= maxFrameOffsetFromTop);
+    MOZ_ASSERT(frameOffsetFromTop < numMappedWords);
+    frameOffsetFromTop = nWords;
+  }
+
+  // If the frame described by this StackMap includes a DebugFrame for a
+  // ref-typed return value, call here to record that fact.
+  void setHasRefTypedDebugFrame() {
+    MOZ_ASSERT(hasRefTypedDebugFrame == 0);
+    hasRefTypedDebugFrame = 1;
+  }
+
+  inline void setBit(uint32_t bitIndex) {
+    MOZ_ASSERT(bitIndex < numMappedWords);
+    uint32_t wordIndex = bitIndex / wordsPerBitmapElem;
+    uint32_t wordOffset = bitIndex % wordsPerBitmapElem;
+    bitmap[wordIndex] |= (1 << wordOffset);
+  }
+
+  inline uint32_t getBit(uint32_t bitIndex) const {
+    MOZ_ASSERT(bitIndex < numMappedWords);
+    uint32_t wordIndex = bitIndex / wordsPerBitmapElem;
+    uint32_t wordOffset = bitIndex % wordsPerBitmapElem;
+    return (bitmap[wordIndex] >> wordOffset) & 1;
+  }
+
+ private:
+  static constexpr uint32_t wordsPerBitmapElem = sizeof(bitmap[0]) * 8;
+
+  static uint32_t calcNBitmap(uint32_t numMappedWords) {
+    MOZ_RELEASE_ASSERT(numMappedWords <= maxMappedWords);
+    uint32_t nBitmap =
+        (numMappedWords + wordsPerBitmapElem - 1) / wordsPerBitmapElem;
+    return nBitmap == 0 ? 1 : nBitmap;
+  }
+};
+
+// This is the expected size for a map that covers 32 or fewer words.
+static_assert(sizeof(StackMap) == 12, "wasm::StackMap has unexpected size");
+
+class StackMaps {
+ public:
+  // A Maplet holds a single code-address-to-map binding.  Note that the
+  // code address is the lowest address of the instruction immediately
+  // following the instruction of interest, not of the instruction of
+  // interest itself.  In practice (at least for the Wasm Baseline compiler)
+  // this means that |nextInsnAddr| points either immediately after a call
+  // instruction, after a trap instruction or after a no-op.
+  struct Maplet {
+    uint8_t* nextInsnAddr;
+    StackMap* map;
+    Maplet(uint8_t* nextInsnAddr, StackMap* map)
+        : nextInsnAddr(nextInsnAddr), map(map) {}
+    void offsetBy(uintptr_t delta) { nextInsnAddr += delta; }
+    bool operator<(const Maplet& other) const {
+      return uintptr_t(nextInsnAddr) < uintptr_t(other.nextInsnAddr);
+    }
+  };
+
+ private:
+  bool sorted_;
+  Vector<Maplet, 0, SystemAllocPolicy> mapping_;
+
+ public:
+  StackMaps() : sorted_(false) {}
+  ~StackMaps() {
+    for (size_t i = 0; i < mapping_.length(); i++) {
+      mapping_[i].map->destroy();
+      mapping_[i].map = nullptr;
+    }
+  }
+  MOZ_MUST_USE bool add(uint8_t* nextInsnAddr, StackMap* map) {
+    MOZ_ASSERT(!sorted_);
+    return mapping_.append(Maplet(nextInsnAddr, map));
+  }
+  MOZ_MUST_USE bool add(const Maplet& maplet) {
+    return add(maplet.nextInsnAddr, maplet.map);
+  }
+  void clear() {
+    for (size_t i = 0; i < mapping_.length(); i++) {
+      mapping_[i].nextInsnAddr = nullptr;
+      mapping_[i].map = nullptr;
+    }
+    mapping_.clear();
+  }
+  bool empty() const { return mapping_.empty(); }
+  size_t length() const { return mapping_.length(); }
+  Maplet get(size_t i) const { return mapping_[i]; }
+  Maplet move(size_t i) {
+    Maplet m = mapping_[i];
+    mapping_[i].map = nullptr;
+    return m;
+  }
+  void offsetBy(uintptr_t delta) {
+    for (size_t i = 0; i < mapping_.length(); i++) mapping_[i].offsetBy(delta);
+  }
+  void sort() {
+    MOZ_ASSERT(!sorted_);
+    std::sort(mapping_.begin(), mapping_.end());
+    sorted_ = true;
+  }
+  const StackMap* findMap(uint8_t* nextInsnAddr) const {
+    struct Comparator {
+      int operator()(Maplet aVal) const {
+        if (uintptr_t(mTarget) < uintptr_t(aVal.nextInsnAddr)) {
+          return -1;
+        }
+        if (uintptr_t(mTarget) > uintptr_t(aVal.nextInsnAddr)) {
+          return 1;
+        }
+        return 0;
+      }
+      explicit Comparator(uint8_t* aTarget) : mTarget(aTarget) {}
+      const uint8_t* mTarget;
+    };
+
+    size_t result;
+    if (BinarySearchIf(mapping_, 0, mapping_.length(), Comparator(nextInsnAddr),
+                       &result)) {
+      return mapping_[result].map;
+    }
+
+    return nullptr;
+  }
+};
+
+// Supporting code for creation of stackmaps.
+
 // StackArgAreaSizeUnaligned returns the size, in bytes, of the stack arg area
 // size needed to pass |argTypes|, excluding any alignment padding beyond the
 // size of the area as a whole.  The size is as determined by the platforms
 // native ABI.
 //
 // StackArgAreaSizeAligned returns the same, but rounded up to the nearest 16
 // byte boundary.
 //
@@ -82,16 +292,25 @@ static inline size_t AlignStackArgAreaSi
   return AlignBytes(unalignedSize, 16u);
 }
 
 template <class T>
 static inline size_t StackArgAreaSizeAligned(const T& argTypes) {
   return AlignStackArgAreaSize(StackArgAreaSizeUnaligned(argTypes));
 }
 
+// At a resumable wasm trap, the machine's registers are saved on the stack by
+// (code generated by) GenerateTrapExit().  This function writes into |args| a
+// vector of booleans describing the ref-ness of the saved integer registers.
+// |args[0]| corresponds to the low addressed end of the described section of
+// the save area.
+MOZ_MUST_USE bool GenerateStackmapEntriesForTrapExit(
+    const ValTypeVector& args, const MachineState& trapExitLayout,
+    const size_t trapExitLayoutNumWords,  ExitStubMapVector* extras);
+
 // Shared write barrier code.
 //
 // A barriered store looks like this:
 //
 //   Label skipPreBarrier;
 //   EmitWasmPreBarrierGuard(..., &skipPreBarrier);
 //   <COMPILER-SPECIFIC ACTIONS HERE>
 //   EmitWasmPreBarrierCall(...);
--- a/js/src/wasm/WasmInstance.cpp
+++ b/js/src/wasm/WasmInstance.cpp
@@ -1393,16 +1393,22 @@ uintptr_t Instance::traceFrame(JSTracer*
   const uintptr_t scanStart = uintptr_t(frame) +
                               (map->frameOffsetFromTop * sizeof(void*)) -
                               numMappedBytes;
   MOZ_ASSERT(0 == scanStart % sizeof(void*));
 
   // Do what we can to assert that, for consecutive wasm frames, their stack
   // maps also abut exactly.  This is a useful sanity check on the sizing of
   // stack maps.
+  //
+  // In debug builds, the stackmap construction machinery goes to considerable
+  // efforts to ensure that the stackmaps for consecutive frames abut exactly.
+  // This is so as to ensure there are no areas of stack inadvertently ignored
+  // by a stackmap, nor covered by two stackmaps.  Hence any failure of this
+  // assertion is serious and should be investigated.
   MOZ_ASSERT_IF(highestByteVisitedInPrevFrame != 0,
                 highestByteVisitedInPrevFrame + 1 == scanStart);
 
   uintptr_t* stackWords = (uintptr_t*)scanStart;
 
   // If we have some exit stub words, this means the map also covers an area
   // created by a exit stub, and so the highest word of that should be a
   // constant created by (code created by) GenerateTrapExit.
--- a/js/src/wasm/WasmIonCompile.cpp
+++ b/js/src/wasm/WasmIonCompile.cpp
@@ -19,19 +19,21 @@
 #include "wasm/WasmIonCompile.h"
 
 #include "mozilla/MathAlgorithms.h"
 
 #include "jit/CodeGenerator.h"
 
 #include "wasm/WasmBaselineCompile.h"
 #include "wasm/WasmBuiltins.h"
+#include "wasm/WasmGC.h"
 #include "wasm/WasmGenerator.h"
 #include "wasm/WasmOpIter.h"
 #include "wasm/WasmSignalHandlers.h"
+#include "wasm/WasmStubs.h"
 #include "wasm/WasmValidate.h"
 
 using namespace js;
 using namespace js::jit;
 using namespace js::wasm;
 
 using mozilla::IsPowerOfTwo;
 using mozilla::Maybe;
@@ -1058,17 +1060,18 @@ class FunctionCompiler {
     if (inDeadCode()) {
       *def = nullptr;
       return true;
     }
 
     CallSiteDesc desc(lineOrBytecode, CallSiteDesc::Func);
     MIRType ret = ToMIRType(funcType.ret());
     auto callee = CalleeDesc::function(funcIndex);
-    auto* ins = MWasmCall::New(alloc(), desc, callee, call.regArgs_, ret);
+    auto* ins = MWasmCall::New(alloc(), desc, callee, call.regArgs_, ret,
+                               StackArgAreaSizeUnaligned(funcType.args()));
     if (!ins) {
       return false;
     }
 
     curBlock_->add(ins);
     *def = ins;
     return true;
   }
@@ -1102,17 +1105,19 @@ class FunctionCompiler {
     } else {
       MOZ_ASSERT(funcType.id.kind() != FuncTypeIdDescKind::None);
       const TableDesc& table = env_.tables[tableIndex];
       callee = CalleeDesc::wasmTable(table, funcType.id);
     }
 
     CallSiteDesc desc(lineOrBytecode, CallSiteDesc::Dynamic);
     auto* ins = MWasmCall::New(alloc(), desc, callee, call.regArgs_,
-                               ToMIRType(funcType.ret()), index);
+                               ToMIRType(funcType.ret()),
+                               StackArgAreaSizeUnaligned(funcType.args()),
+                               index);
     if (!ins) {
       return false;
     }
 
     curBlock_->add(ins);
     *def = ins;
     return true;
   }
@@ -1123,17 +1128,18 @@ class FunctionCompiler {
     if (inDeadCode()) {
       *def = nullptr;
       return true;
     }
 
     CallSiteDesc desc(lineOrBytecode, CallSiteDesc::Dynamic);
     auto callee = CalleeDesc::import(globalDataOffset);
     auto* ins = MWasmCall::New(alloc(), desc, callee, call.regArgs_,
-                               ToMIRType(funcType.ret()));
+                               ToMIRType(funcType.ret()),
+                               StackArgAreaSizeUnaligned(funcType.args()));
     if (!ins) {
       return false;
     }
 
     curBlock_->add(ins);
     *def = ins;
     return true;
   }
@@ -1144,17 +1150,18 @@ class FunctionCompiler {
     if (inDeadCode()) {
       *def = nullptr;
       return true;
     }
 
     CallSiteDesc desc(lineOrBytecode, CallSiteDesc::Symbolic);
     auto callee = CalleeDesc::builtin(builtin.identity);
     auto* ins =
-        MWasmCall::New(alloc(), desc, callee, call.regArgs_, builtin.retType);
+        MWasmCall::New(alloc(), desc, callee, call.regArgs_, builtin.retType,
+                       StackArgAreaSizeUnaligned(builtin));
     if (!ins) {
       return false;
     }
 
     curBlock_->add(ins);
     *def = ins;
     return true;
   }
@@ -1166,17 +1173,17 @@ class FunctionCompiler {
     if (inDeadCode()) {
       *def = nullptr;
       return true;
     }
 
     CallSiteDesc desc(lineOrBytecode, CallSiteDesc::Symbolic);
     auto* ins = MWasmCall::NewBuiltinInstanceMethodCall(
         alloc(), desc, builtin.identity, call.instanceArg_, call.regArgs_,
-        builtin.retType);
+        builtin.retType, StackArgAreaSizeUnaligned(builtin));
     if (!ins) {
       return false;
     }
 
     curBlock_->add(ins);
     *def = ins;
     return true;
   }
@@ -4152,23 +4159,35 @@ bool wasm::IonCompileFunctions(const Mod
   WasmMacroAssembler masm(alloc);
 
   // Swap in already-allocated empty vectors to avoid malloc/free.
   MOZ_ASSERT(code->empty());
   if (!code->swap(masm)) {
     return false;
   }
 
+  // Create a description of the stack layout created by GenerateTrapExit().
+  MachineState trapExitLayout;
+  size_t trapExitLayoutNumWords;
+  GenerateTrapExitMachineState(&trapExitLayout, &trapExitLayoutNumWords);
+
   for (const FuncCompileInput& func : inputs) {
+    JitSpew(JitSpew_Codegen, "# ========================================");
+    JitSpew(JitSpew_Codegen, "# ==");
+    JitSpew(JitSpew_Codegen,
+            "# wasm::IonCompileFunctions: starting on function index %d",
+            (int)func.index);
+
     Decoder d(func.begin, func.end, func.lineOrBytecode, error);
 
     // Build the local types vector.
 
+    const ValTypeVector& argTys = env.funcTypes[func.index]->args();
     ValTypeVector locals;
-    if (!locals.appendAll(env.funcTypes[func.index]->args())) {
+    if (!locals.appendAll(argTys)) {
       return false;
     }
     if (!DecodeLocalEntries(d, env.types, env.gcTypesEnabled(), &locals)) {
       return false;
     }
 
     // Set up for Ion compilation.
 
@@ -4212,25 +4231,33 @@ bool wasm::IonCompileFunctions(const Mod
       }
 
       FuncTypeIdDesc funcTypeId = env.funcTypes[func.index]->id;
 
       CodeGenerator codegen(&mir, lir, &masm);
 
       BytecodeOffset prologueTrapOffset(func.lineOrBytecode);
       FuncOffsets offsets;
-      if (!codegen.generateWasm(funcTypeId, prologueTrapOffset, &offsets)) {
+      if (!codegen.generateWasm(funcTypeId, prologueTrapOffset, argTys,
+                                trapExitLayout, trapExitLayoutNumWords,
+                                &offsets, &code->stackMaps)) {
         return false;
       }
 
       if (!code->codeRanges.emplaceBack(func.index, func.lineOrBytecode,
                                         offsets)) {
         return false;
       }
     }
+
+    JitSpew(JitSpew_Codegen,
+            "# wasm::IonCompileFunctions: completed function index %d",
+            (int)func.index);
+    JitSpew(JitSpew_Codegen, "# ==");
+    JitSpew(JitSpew_Codegen, "# ========================================");
   }
 
   masm.finish();
   if (masm.oom()) {
     return false;
   }
 
   return code->swap(masm);
--- a/js/src/wasm/WasmTypes.h
+++ b/js/src/wasm/WasmTypes.h
@@ -1801,222 +1801,16 @@ class CallSiteTarget {
     MOZ_ASSERT(kind_ == TrapExit);
     MOZ_ASSERT(packed_ < uint32_t(Trap::Limit));
     return Trap(packed_);
   }
 };
 
 typedef Vector<CallSiteTarget, 0, SystemAllocPolicy> CallSiteTargetVector;
 
-typedef Vector<bool, 32, SystemAllocPolicy> ExitStubMapVector;
-
-struct StackMap final {
-  // A StackMap is a bit-array containing numMappedWords bits, one bit per
-  // word of stack.  Bit index zero is for the lowest addressed word in the
-  // range.
-  //
-  // This is a variable-length structure whose size must be known at creation
-  // time.
-  //
-  // Users of the map will know the address of the wasm::Frame that is covered
-  // by this map.  In order that they can calculate the exact address range
-  // covered by the map, the map also stores the offset, from the highest
-  // addressed word of the map, of the embedded wasm::Frame.  This is an
-  // offset down from the highest address, rather than up from the lowest, so
-  // as to limit its range to 11 bits, where
-  // 11 == ceil(log2(MaxParams * sizeof-biggest-param-type-in-words))
-  //
-  // The map may also cover a ref-typed DebugFrame.  If so that can be noted,
-  // since users of the map need to trace pointers in such a DebugFrame.
-  //
-  // Finally, for sanity checking only, for stack maps associated with a wasm
-  // trap exit stub, the number of words used by the trap exit stub save area
-  // is also noted.  This is used in Instance::traceFrame to check that the
-  // TrapExitDummyValue is in the expected place in the frame.
-
-  // The total number of stack words covered by the map ..
-  uint32_t numMappedWords : 30;
-
-  // .. of which this many are "exit stub" extras
-  uint32_t numExitStubWords : 6;
-
-  // Where is Frame* relative to the top?  This is an offset in words.
-  uint32_t frameOffsetFromTop : 11;
-
-  // Notes the presence of a ref-typed DebugFrame.
-  uint32_t hasRefTypedDebugFrame : 1;
-
- private:
-  static constexpr uint32_t maxMappedWords = (1 << 30) - 1;
-  static constexpr uint32_t maxExitStubWords = (1 << 6) - 1;
-  static constexpr uint32_t maxFrameOffsetFromTop = (1 << 11) - 1;
-
-  uint32_t bitmap[1];
-
-  explicit StackMap(uint32_t numMappedWords)
-      : numMappedWords(numMappedWords),
-        numExitStubWords(0),
-        frameOffsetFromTop(0),
-        hasRefTypedDebugFrame(0) {
-    const uint32_t nBitmap = calcNBitmap(numMappedWords);
-    memset(bitmap, 0, nBitmap * sizeof(bitmap[0]));
-  }
-
- public:
-  static StackMap* create(uint32_t numMappedWords) {
-    uint32_t nBitmap = calcNBitmap(numMappedWords);
-    char* buf =
-        (char*)js_malloc(sizeof(StackMap) + (nBitmap - 1) * sizeof(bitmap[0]));
-    if (!buf) {
-      return nullptr;
-    }
-    return ::new (buf) StackMap(numMappedWords);
-  }
-
-  void destroy() { js_free((char*)this); }
-
-  // Record the number of words in the map used as a wasm trap exit stub
-  // save area.  See comment above.
-  void setExitStubWords(uint32_t nWords) {
-    MOZ_ASSERT(numExitStubWords == 0);
-    MOZ_RELEASE_ASSERT(nWords <= maxExitStubWords);
-    MOZ_ASSERT(nWords <= numMappedWords);
-    numExitStubWords = nWords;
-  }
-
-  // Record the offset from the highest-addressed word of the map, that the
-  // wasm::Frame lives at.  See comment above.
-  void setFrameOffsetFromTop(uint32_t nWords) {
-    MOZ_ASSERT(frameOffsetFromTop == 0);
-    MOZ_RELEASE_ASSERT(nWords <= maxFrameOffsetFromTop);
-    MOZ_ASSERT(frameOffsetFromTop < numMappedWords);
-    frameOffsetFromTop = nWords;
-  }
-
-  // If the frame described by this StackMap includes a DebugFrame for a
-  // ref-typed return value, call here to record that fact.
-  void setHasRefTypedDebugFrame() {
-    MOZ_ASSERT(hasRefTypedDebugFrame == 0);
-    hasRefTypedDebugFrame = 1;
-  }
-
-  inline void setBit(uint32_t bitIndex) {
-    MOZ_ASSERT(bitIndex < numMappedWords);
-    uint32_t wordIndex = bitIndex / wordsPerBitmapElem;
-    uint32_t wordOffset = bitIndex % wordsPerBitmapElem;
-    bitmap[wordIndex] |= (1 << wordOffset);
-  }
-
-  inline uint32_t getBit(uint32_t bitIndex) const {
-    MOZ_ASSERT(bitIndex < numMappedWords);
-    uint32_t wordIndex = bitIndex / wordsPerBitmapElem;
-    uint32_t wordOffset = bitIndex % wordsPerBitmapElem;
-    return (bitmap[wordIndex] >> wordOffset) & 1;
-  }
-
- private:
-  static constexpr uint32_t wordsPerBitmapElem = sizeof(bitmap[0]) * 8;
-
-  static uint32_t calcNBitmap(uint32_t numMappedWords) {
-    MOZ_RELEASE_ASSERT(numMappedWords <= maxMappedWords);
-    uint32_t nBitmap =
-        (numMappedWords + wordsPerBitmapElem - 1) / wordsPerBitmapElem;
-    return nBitmap == 0 ? 1 : nBitmap;
-  }
-};
-
-// This is the expected size for a map that covers 32 or fewer words.
-static_assert(sizeof(StackMap) == 12, "wasm::StackMap has unexpected size");
-
-class StackMaps {
- public:
-  // A Maplet holds a single code-address-to-map binding.  Note that the
-  // code address is the lowest address of the instruction immediately
-  // following the instruction of interest, not of the instruction of
-  // interest itself.  In practice (at least for the Wasm Baseline compiler)
-  // this means that |nextInsnAddr| points either immediately after a call
-  // instruction, after a trap instruction or after a no-op.
-  struct Maplet {
-    uint8_t* nextInsnAddr;
-    StackMap* map;
-    Maplet(uint8_t* nextInsnAddr, StackMap* map)
-        : nextInsnAddr(nextInsnAddr), map(map) {}
-    void offsetBy(uintptr_t delta) { nextInsnAddr += delta; }
-    bool operator<(const Maplet& other) const {
-      return uintptr_t(nextInsnAddr) < uintptr_t(other.nextInsnAddr);
-    }
-  };
-
- private:
-  bool sorted_;
-  Vector<Maplet, 0, SystemAllocPolicy> mapping_;
-
- public:
-  StackMaps() : sorted_(false) {}
-  ~StackMaps() {
-    for (size_t i = 0; i < mapping_.length(); i++) {
-      mapping_[i].map->destroy();
-      mapping_[i].map = nullptr;
-    }
-  }
-  MOZ_MUST_USE bool add(uint8_t* nextInsnAddr, StackMap* map) {
-    MOZ_ASSERT(!sorted_);
-    return mapping_.append(Maplet(nextInsnAddr, map));
-  }
-  MOZ_MUST_USE bool add(const Maplet& maplet) {
-    return add(maplet.nextInsnAddr, maplet.map);
-  }
-  void clear() {
-    for (size_t i = 0; i < mapping_.length(); i++) {
-      mapping_[i].nextInsnAddr = nullptr;
-      mapping_[i].map = nullptr;
-    }
-    mapping_.clear();
-  }
-  bool empty() const { return mapping_.empty(); }
-  size_t length() const { return mapping_.length(); }
-  Maplet get(size_t i) const { return mapping_[i]; }
-  Maplet move(size_t i) {
-    Maplet m = mapping_[i];
-    mapping_[i].map = nullptr;
-    return m;
-  }
-  void offsetBy(uintptr_t delta) {
-    for (size_t i = 0; i < mapping_.length(); i++) mapping_[i].offsetBy(delta);
-  }
-  void sort() {
-    MOZ_ASSERT(!sorted_);
-    std::sort(mapping_.begin(), mapping_.end());
-    sorted_ = true;
-  }
-  const StackMap* findMap(uint8_t* nextInsnAddr) const {
-    struct Comparator {
-      int operator()(Maplet aVal) const {
-        if (uintptr_t(mTarget) < uintptr_t(aVal.nextInsnAddr)) {
-          return -1;
-        }
-        if (uintptr_t(mTarget) > uintptr_t(aVal.nextInsnAddr)) {
-          return 1;
-        }
-        return 0;
-      }
-      explicit Comparator(uint8_t* aTarget) : mTarget(aTarget) {}
-      const uint8_t* mTarget;
-    };
-
-    size_t result;
-    if (BinarySearchIf(mapping_, 0, mapping_.length(), Comparator(nextInsnAddr),
-                       &result)) {
-      return mapping_[result].map;
-    }
-
-    return nullptr;
-  }
-};
-
 // A wasm::SymbolicAddress represents a pointer to a well-known function that is
 // embedded in wasm code. Since wasm code is serialized and later deserialized
 // into a different address space, symbolic addresses must be used for *all*
 // pointers into the address space. The MacroAssembler records a list of all
 // SymbolicAddresses and the offsets of their use in the code for later patching
 // during static linking.
 
 enum class SymbolicAddress {