Skip to main content

core/intrinsics/
mod.rs

1//! Compiler intrinsics.
2//!
3//! The functions in this module are implementation details of `core` and should
4//! not be used outside of the standard library. We generally provide access to
5//! intrinsics via stable wrapper functions. Use these instead.
6//!
7//! These are the imports making intrinsics available to Rust code. The actual implementations live in the compiler.
8//! Some of these intrinsics are lowered to MIR in <https://github.com/rust-lang/rust/blob/HEAD/compiler/rustc_mir_transform/src/lower_intrinsics.rs>.
9//! The remaining intrinsics are implemented for the LLVM backend in <https://github.com/rust-lang/rust/blob/HEAD/compiler/rustc_codegen_ssa/src/mir/intrinsic.rs>
10//! and <https://github.com/rust-lang/rust/blob/HEAD/compiler/rustc_codegen_llvm/src/intrinsic.rs>,
11//! and for const evaluation in <https://github.com/rust-lang/rust/blob/HEAD/compiler/rustc_const_eval/src/interpret/intrinsics.rs>.
12//!
13//! Intrinsics don't need a body. However, they optionally can have a body, which we call the
14//! "fallback body". This will be used by codegen backends that do not have a dedicated
15//! implementation of the intrinsic, making it easier to add new intrinsics for specific operations
16//! without having to implement them in each codegen backend. The fallback body obviously has to be
17//! a valid implementation of the documented specification of the intrinsic. In some cases, the
18//! fallback body will be *equivalent* to the specification. Note that this is a strong requirement:
19//! if the spec says "UB if input `x` is even", then a valid implementation can just ignore this and
20//! do whatever it wants in that case; an *equivalent* implementation needs to actually check this
21//! condition and trigger UB in that case (e.g. by using `hint::assert_unchecked()`). Similar, if
22//! the spec says "returns `x` or `y` non-deterministically", then an *equivalent* implementation
23//! must actually do non-deterministic choice and return either value (e.g. by invoking some other
24//! language operation that has the same non-determinism). Intrinsics with such a fallback body that
25//! is equivalent to the spec may be marked with `#[miri::intrinsic_fallback_is_spec]`; the fallback
26//! body will then also be used by Miri for UB checking. When in doubt, do not use this attribute or
27//! ask the Miri maintainers for advice.
28//!
29//! Intrinsics are, in general, language extensions. Therefore, t-lang should be involved whenever a
30//! new intrinsic is exposed to stable code. However, if an intrinsic is marked
31//! `#[miri::intrinsic_fallback_is_spec]` with a fallback body that only uses stable features (or if
32//! such a fallback body could be written, but for one reason or another the actual fallback body is
33//! different), and if it also does not make other promises that go beyond observable program
34//! behavior (such as steering the optimizer in a particular direction), then an intrinsic may be
35//! used without t-lang involvement.
36//!
37//! # Const intrinsics
38//!
39//! In order to make an intrinsic unstable usable at compile-time, copy the implementation from
40//! <https://github.com/rust-lang/miri/blob/master/src/intrinsics> to
41//! <https://github.com/rust-lang/rust/blob/HEAD/compiler/rustc_const_eval/src/interpret/intrinsics.rs>
42//! and make the intrinsic declaration below a `const fn`. This should be done in coordination with
43//! wg-const-eval.
44//!
45//! If an intrinsic is supposed to be used from a `const fn` with a `rustc_const_stable` attribute,
46//! `#[rustc_intrinsic_const_stable_indirect]` needs to be added to the intrinsic. Such a change
47//! requires T-lang approval, because it may bake a feature into the language that cannot be
48//! replicated in user code without compiler support. The same exception as above applies for
49//! `#[miri::intrinsic_fallback_is_spec]` intrinsics.
50//!
51//! # Volatiles
52//!
53//! The volatile intrinsics provide operations intended to act on I/O
54//! memory, which are guaranteed to not be reordered by the compiler
55//! across other volatile intrinsics. See [`read_volatile`][ptr::read_volatile]
56//! and [`write_volatile`][ptr::write_volatile].
57//!
58//! # Atomics
59//!
60//! The atomic intrinsics provide common atomic operations on machine
61//! words, with multiple possible memory orderings. See the
62//! [atomic types][atomic] docs for details.
63//!
64//! # Unwinding
65//!
66//! Rust intrinsics may, in general, unwind. If an intrinsic can never unwind, add the
67//! `#[rustc_nounwind]` attribute so that the compiler can make use of this fact.
68//!
69//! However, even for intrinsics that may unwind, rustc assumes that a Rust intrinsics will never
70//! initiate a foreign (non-Rust) unwind, and thus for panic=abort we can always assume that these
71//! intrinsics cannot unwind.
72
73#![unstable(
74    feature = "core_intrinsics",
75    reason = "intrinsics are unlikely to ever be stabilized, instead \
76                      they should be used through stabilized interfaces \
77                      in the rest of the standard library",
78    issue = "none"
79)]
80
81use crate::ffi::{VaArgSafe, VaList};
82use crate::marker::{ConstParamTy, DiscriminantKind, PointeeSized, Tuple};
83use crate::num::imp::libm;
84use crate::{mem, ptr};
85
86mod bounds;
87pub mod fallback;
88pub mod gpu;
89mod macros;
90pub mod mir;
91pub mod reflection;
92pub mod simd;
93
94use macros::intrinsic_dispatch_on_type;
95
96// These imports are used for simplifying intra-doc links
97#[allow(unused_imports)]
98#[cfg(all(target_has_atomic = "8", target_has_atomic = "32", target_has_atomic = "ptr"))]
99use crate::sync::atomic::{
100    self, AtomicBool, AtomicI32, AtomicIsize, AtomicPtr, AtomicU32, Ordering,
101};
102
103/// A type for atomic ordering parameters for intrinsics. This is a separate type from
104/// `atomic::Ordering` so that we can make it `ConstParamTy` and fix the values used here without a
105/// risk of leaking that to stable code.
106#[allow(missing_docs)]
107#[derive(Debug, ConstParamTy, PartialEq, Eq)]
108pub enum AtomicOrdering {
109    // These values must match the compiler's `AtomicOrdering` defined in
110    // `rustc_middle/src/ty/consts/int.rs`!
111    Relaxed = 0,
112    Release = 1,
113    Acquire = 2,
114    AcqRel = 3,
115    SeqCst = 4,
116}
117
118// N.B., these intrinsics take raw pointers because they mutate aliased
119// memory, which is not valid for either `&` or `&mut`.
120
121/// Stores a value if the current value is the same as the `old` value,
122/// with the returned `T` being the previous value and the returned `bool` being
123/// whether the exchange was successful.
124/// `T` must be an integer or pointer type.
125///
126/// # Safety
127///
128/// * If `T` is an integer type, this is equivalent to [`Atomic<T>::from_ptr`] followed by [`Atomic<T>::compare_exchange`].
129///   Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
130///
131/// * If `T` is a pointer type, this is equivalent to [`AtomicPtr<P>::from_ptr`] followed by [`AtomicPtr<P>::compare_exchange`],
132///   where `P` is the pointee type of the pointer.
133///   Refer to the documentation of [`AtomicPtr<P>::from_ptr`] for safety requirements.
134///
135/// The stabilized version of this intrinsic is available on the
136/// [`atomic`] types via the `compare_exchange` method.
137/// For example, [`AtomicBool::compare_exchange`].
138///
139/// [`Atomic<T>::from_ptr`]: AtomicI32::from_ptr
140/// [`Atomic<T>::compare_exchange`]: AtomicI32::compare_exchange
141/// [`AtomicPtr<P>::from_ptr`]: AtomicPtr::from_ptr
142/// [`AtomicPtr<P>::compare_exchange`]: AtomicPtr::compare_exchange
143#[rustc_intrinsic]
144#[rustc_nounwind]
145pub const unsafe fn atomic_cxchg<
146    T: Copy,
147    const ORD_SUCC: AtomicOrdering,
148    const ORD_FAIL: AtomicOrdering,
149>(
150    dst: *mut T,
151    old: T,
152    src: T,
153) -> (T, bool);
154
155/// Stores a value if the current value is the same as the `old` value,
156/// with the returned `T` being the previous value and the returned `bool` being
157/// whether the exchange was successful.
158/// `T` must be an integer or pointer type. The comparison may spuriously fail.
159///
160/// # Safety
161///
162/// * If `T` is an integer type, this is equivalent to [`Atomic<T>::from_ptr`] followed by [`Atomic<T>::compare_exchange_weak`].
163///   Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
164///
165/// * If `T` is a pointer type, this is equivalent to [`AtomicPtr<P>::from_ptr`] followed by [`AtomicPtr<P>::compare_exchange_weak`],
166///   where `P` is the pointee type of the pointer.
167///   Refer to the documentation of [`AtomicPtr<P>::from_ptr`] for safety requirements.
168///
169/// The stabilized version of this intrinsic is available on the
170/// [`atomic`] types via the `compare_exchange_weak` method.
171/// For example, [`AtomicBool::compare_exchange_weak`].
172///
173/// [`Atomic<T>::from_ptr`]: AtomicI32::from_ptr
174/// [`Atomic<T>::compare_exchange_weak`]: AtomicI32::compare_exchange_weak
175/// [`AtomicPtr<P>::from_ptr`]: AtomicPtr::from_ptr
176/// [`AtomicPtr<P>::compare_exchange_weak`]: AtomicPtr::compare_exchange_weak
177#[rustc_intrinsic]
178#[rustc_nounwind]
179pub const unsafe fn atomic_cxchgweak<
180    T: Copy,
181    const ORD_SUCC: AtomicOrdering,
182    const ORD_FAIL: AtomicOrdering,
183>(
184    _dst: *mut T,
185    _old: T,
186    _src: T,
187) -> (T, bool);
188
189/// Loads the current value of the pointer.
190/// `T` must be an integer or pointer type.
191///
192/// # Safety
193///
194/// * If `VOLATILE` is `true`, this is equivalent to [`Atomic<T>::load_volatile`].
195///   Refer to the documentation of that method for safety requirements.
196///
197/// * If `VOLATILE` is `false`, this is equivalent to [`Atomic<T>::from_ptr`] followed
198///   by [`Atomic<T>::load`]. Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
199///
200/// The stabilized version of this intrinsic is available on the
201/// [`atomic`] types via the `load` method. For example, [`AtomicBool::load`].
202///
203/// [`Atomic<T>::load_volatile`]: AtomicI32::load_volatile
204/// [`Atomic<T>::from_ptr`]: AtomicI32::from_ptr
205/// [`Atomic<T>::load`]: AtomicI32::load
206#[rustc_intrinsic]
207#[rustc_nounwind]
208pub const unsafe fn atomic_load<T: Copy, const ORD: AtomicOrdering, const VOLATILE: bool>(
209    src: *const T,
210) -> T;
211
212/// Stores the value at the specified memory location.
213/// `T` must be an integer or pointer type.
214///
215/// # Safety
216///
217/// * If `VOLATILE` is `true`, this is equivalent to [`Atomic<T>::store_volatile`].
218///   Refer to the documentation of that method for safety requirements.
219///
220/// * If `VOLATILE` is `false`, this is equivalent to [`Atomic<T>::from_ptr`] followed
221///   by [`Atomic<T>::store`]. Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
222///
223/// The stabilized version of this intrinsic is available on the
224/// [`atomic`] types via the `store` method. For example, [`AtomicBool::store`].
225///
226/// [`Atomic<T>::store_volatile`]: AtomicI32::store_volatile
227/// [`Atomic<T>::from_ptr`]: AtomicI32::from_ptr
228/// [`Atomic<T>::store`]: AtomicI32::store
229#[rustc_intrinsic]
230#[rustc_nounwind]
231pub const unsafe fn atomic_store<T: Copy, const ORD: AtomicOrdering, const VOLATILE: bool>(
232    dst: *mut T,
233    val: T,
234);
235
236/// Stores the value at the specified memory location, returning the old value.
237/// `T` must be an integer or pointer type.
238///
239/// # Safety
240///
241/// * If `T` is an integer type, this is equivalent to [`Atomic<T>::from_ptr`] followed by [`Atomic<T>::swap`].
242///   Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
243///
244/// * If `T` is a pointer type, this is equivalent to [`AtomicPtr<P>::from_ptr`] followed by [`AtomicPtr<P>::swap`],
245///   where `P` is the pointee type of the pointer.
246///   Refer to the documentation of [`AtomicPtr<P>::from_ptr`] for safety requirements.
247///
248/// The stabilized version of this intrinsic is available on the
249/// [`atomic`] types via the `swap` method. For example, [`AtomicBool::swap`].
250///
251/// [`Atomic<T>::from_ptr`]: AtomicI32::from_ptr
252/// [`Atomic<T>::swap`]: AtomicI32::swap
253/// [`AtomicPtr<P>::from_ptr`]: AtomicPtr::from_ptr
254/// [`AtomicPtr<P>::swap`]: AtomicPtr::swap
255#[rustc_intrinsic]
256#[rustc_nounwind]
257pub const unsafe fn atomic_xchg<T: Copy, const ORD: AtomicOrdering>(dst: *mut T, src: T) -> T;
258
259/// Adds to the current value, returning the previous value.
260/// `T` must be an integer or pointer type.
261/// `U` must be the same as `T` if that is an integer type, or `usize` if `T` is a pointer type.
262///
263/// # Safety
264///
265/// * If `T` is an integer type, this is equivalent to [`Atomic<T>::from_ptr`] followed by [`Atomic<T>::fetch_add`].
266///   Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
267///
268/// * If `T` is a pointer type, this is equivalent to [`AtomicPtr<P>::from_ptr`] followed by [`AtomicPtr<P>::fetch_byte_add`],
269///   where `P` is the pointee type of the pointer.
270///   Refer to the documentation of [`AtomicPtr<P>::from_ptr`] for safety requirements.
271///
272/// The stabilized version of this intrinsic is available on the
273/// [`atomic`] types via the `fetch_add` method. For example, [`AtomicIsize::fetch_add`].
274///
275/// [`Atomic<T>::from_ptr`]: AtomicI32::from_ptr
276/// [`Atomic<T>::fetch_add`]: AtomicI32::fetch_add
277/// [`AtomicPtr<P>::from_ptr`]: AtomicPtr::from_ptr
278/// [`AtomicPtr<P>::fetch_byte_add`]: AtomicPtr::fetch_byte_add
279#[rustc_intrinsic]
280#[rustc_nounwind]
281pub const unsafe fn atomic_xadd<T: Copy, U: Copy, const ORD: AtomicOrdering>(
282    dst: *mut T,
283    src: U,
284) -> T;
285
286/// Subtract from the current value, returning the previous value.
287/// `T` must be an integer or pointer type.
288/// `U` must be the same as `T` if that is an integer type, or `usize` if `T` is a pointer type.
289///
290/// # Safety
291///
292/// * If `T` is an integer type, this is equivalent to [`Atomic<T>::from_ptr`] followed by [`Atomic<T>::fetch_sub`].
293///   Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
294///
295/// * If `T` is a pointer type, this is equivalent to [`AtomicPtr<P>::from_ptr`] followed by [`AtomicPtr<P>::fetch_byte_sub`],
296///   where `P` is the pointee type of the pointer.
297///   Refer to the documentation of [`AtomicPtr<P>::from_ptr`] for safety requirements.
298///
299/// The stabilized version of this intrinsic is available on the
300/// [`atomic`] types via the `fetch_sub` method. For example, [`AtomicIsize::fetch_sub`].
301///
302/// [`Atomic<T>::from_ptr`]: AtomicI32::from_ptr
303/// [`Atomic<T>::fetch_sub`]: AtomicI32::fetch_sub
304/// [`AtomicPtr<P>::from_ptr`]: AtomicPtr::from_ptr
305/// [`AtomicPtr<P>::fetch_byte_sub`]: AtomicPtr::fetch_byte_sub
306#[rustc_intrinsic]
307#[rustc_nounwind]
308pub const unsafe fn atomic_xsub<T: Copy, U: Copy, const ORD: AtomicOrdering>(
309    dst: *mut T,
310    src: U,
311) -> T;
312
313/// Bitwise and with the current value, returning the previous value.
314/// `T` must be an integer or pointer type.
315/// `U` must be the same as `T` if that is an integer type, or `usize` if `T` is a pointer type.
316///
317/// # Safety
318///
319/// * If `T` is an integer type, this is equivalent to [`Atomic<T>::from_ptr`] followed by [`Atomic<T>::fetch_and`].
320///   Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
321///
322/// * If `T` is a pointer type, this is equivalent to [`AtomicPtr<P>::from_ptr`] followed by [`AtomicPtr<P>::fetch_and`],
323///   where `P` is the pointee type of the pointer.
324///   Refer to the documentation of [`AtomicPtr<P>::from_ptr`] for safety requirements.
325///
326/// The stabilized version of this intrinsic is available on the
327/// [`atomic`] types via the `fetch_and` method. For example, [`AtomicBool::fetch_and`].
328///
329/// [`Atomic<T>::from_ptr`]: AtomicI32::from_ptr
330/// [`Atomic<T>::fetch_and`]: AtomicI32::fetch_and
331/// [`AtomicPtr<P>::from_ptr`]: AtomicPtr::from_ptr
332/// [`AtomicPtr<P>::fetch_and`]: AtomicPtr::fetch_and
333#[rustc_intrinsic]
334#[rustc_nounwind]
335pub const unsafe fn atomic_and<T: Copy, U: Copy, const ORD: AtomicOrdering>(
336    dst: *mut T,
337    src: U,
338) -> T;
339
340/// Bitwise nand with the current value, returning the previous value.
341/// `T` must be an integer or pointer type.
342/// `U` must be the same as `T` if that is an integer type, or `usize` if `T` is a pointer type.
343///
344/// # Safety
345///
346/// This is equivalent to [`Atomic<T>::from_ptr`] followed by [`Atomic<T>::fetch_nand`].
347/// Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
348///
349/// The stabilized version of this intrinsic is available on the
350/// [`atomic`] types via the `fetch_nand` method. For example, [`AtomicBool::fetch_nand`].
351///
352/// [`Atomic<T>::from_ptr`]: AtomicI32::from_ptr
353/// [`Atomic<T>::fetch_nand`]: AtomicI32::fetch_nand
354#[rustc_intrinsic]
355#[rustc_nounwind]
356pub const unsafe fn atomic_nand<T: Copy, U: Copy, const ORD: AtomicOrdering>(
357    dst: *mut T,
358    src: U,
359) -> T;
360
361/// Bitwise or with the current value, returning the previous value.
362/// `T` must be an integer or pointer type.
363/// `U` must be the same as `T` if that is an integer type, or `usize` if `T` is a pointer type.
364///
365/// # Safety
366///
367/// * If `T` is an integer type, this is equivalent to [`Atomic<T>::from_ptr`] followed by [`Atomic<T>::fetch_or`].
368///   Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
369///
370/// * If `T` is a pointer type, this is equivalent to [`AtomicPtr<P>::from_ptr`] followed by [`AtomicPtr<P>::fetch_or`],
371///   where `P` is the pointee type of the pointer.
372///   Refer to the documentation of [`AtomicPtr<P>::from_ptr`] for safety requirements.
373///
374/// The stabilized version of this intrinsic is available on the
375/// [`atomic`] types via the `fetch_or` method. For example, [`AtomicBool::fetch_or`].
376///
377/// [`Atomic<T>::from_ptr`]: AtomicI32::from_ptr
378/// [`Atomic<T>::fetch_or`]: AtomicI32::fetch_or
379/// [`AtomicPtr<P>::from_ptr`]: AtomicPtr::from_ptr
380/// [`AtomicPtr<P>::fetch_or`]: AtomicPtr::fetch_or
381#[rustc_intrinsic]
382#[rustc_nounwind]
383pub const unsafe fn atomic_or<T: Copy, U: Copy, const ORD: AtomicOrdering>(
384    dst: *mut T,
385    src: U,
386) -> T;
387
388/// Bitwise xor with the current value, returning the previous value.
389/// `T` must be an integer or pointer type.
390/// `U` must be the same as `T` if that is an integer type, or `usize` if `T` is a pointer type.
391///
392/// # Safety
393///
394/// * If `T` is an integer type, this is equivalent to [`Atomic<T>::from_ptr`] followed by [`Atomic<T>::fetch_xor`].
395///   Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
396///
397/// * If `T` is a pointer type, this is equivalent to [`AtomicPtr<P>::from_ptr`] followed by [`AtomicPtr<P>::fetch_xor`],
398///   where `P` is the pointee type of the pointer.
399///   Refer to the documentation of [`AtomicPtr<P>::from_ptr`] for safety requirements.
400///
401/// The stabilized version of this intrinsic is available on the
402/// [`atomic`] types via the `fetch_xor` method. For example, [`AtomicBool::fetch_xor`].
403///
404/// [`Atomic<T>::from_ptr`]: AtomicI32::from_ptr
405/// [`Atomic<T>::fetch_xor`]: AtomicI32::fetch_xor
406/// [`AtomicPtr<P>::from_ptr`]: AtomicPtr::from_ptr
407/// [`AtomicPtr<P>::fetch_xor`]: AtomicPtr::fetch_xor
408#[rustc_intrinsic]
409#[rustc_nounwind]
410pub const unsafe fn atomic_xor<T: Copy, U: Copy, const ORD: AtomicOrdering>(
411    dst: *mut T,
412    src: U,
413) -> T;
414
415/// Maximum with the current value using a signed comparison.
416/// `T` must be a signed integer type.
417///
418/// # Safety
419///
420/// This is equivalent to [`Atomic<T>::from_ptr`] followed by [`Atomic<T>::fetch_max`].
421/// Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
422///
423/// The stabilized version of this intrinsic is available on the
424/// [`atomic`] signed integer types via the `fetch_max` method. For example, [`AtomicI32::fetch_max`].
425///
426/// [`Atomic<T>::from_ptr`]: AtomicI32::from_ptr
427/// [`Atomic<T>::fetch_max`]: AtomicI32::fetch_max
428#[rustc_intrinsic]
429#[rustc_nounwind]
430pub const unsafe fn atomic_max<T: Copy, const ORD: AtomicOrdering>(dst: *mut T, src: T) -> T;
431
432/// Minimum with the current value using a signed comparison.
433/// `T` must be a signed integer type.
434///
435/// # Safety
436///
437/// This is equivalent to [`Atomic<T>::from_ptr`] followed by [`Atomic<T>::fetch_min`].
438/// Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
439///
440/// The stabilized version of this intrinsic is available on the
441/// [`atomic`] signed integer types via the `fetch_min` method. For example, [`AtomicI32::fetch_min`].
442///
443/// [`Atomic<T>::from_ptr`]: AtomicI32::from_ptr
444/// [`Atomic<T>::fetch_min`]: AtomicI32::fetch_min
445#[rustc_intrinsic]
446#[rustc_nounwind]
447pub const unsafe fn atomic_min<T: Copy, const ORD: AtomicOrdering>(dst: *mut T, src: T) -> T;
448
449/// Minimum with the current value using an unsigned comparison.
450/// `T` must be an unsigned integer type.
451///
452/// # Safety
453///
454/// This is equivalent to [`Atomic<T>::from_ptr`] followed by [`Atomic<T>::fetch_min`].
455/// Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
456///
457/// The stabilized version of this intrinsic is available on the
458/// [`atomic`] unsigned integer types via the `fetch_min` method. For example, [`AtomicU32::fetch_min`].
459///
460/// [`Atomic<T>::from_ptr`]: AtomicU32::from_ptr
461/// [`Atomic<T>::fetch_min`]: AtomicU32::fetch_min
462#[rustc_intrinsic]
463#[rustc_nounwind]
464pub const unsafe fn atomic_umin<T: Copy, const ORD: AtomicOrdering>(dst: *mut T, src: T) -> T;
465
466/// Maximum with the current value using an unsigned comparison.
467/// `T` must be an unsigned integer type.
468///
469/// # Safety
470///
471/// This is equivalent to [`Atomic<T>::from_ptr`] followed by [`Atomic<T>::fetch_max`].
472/// Refer to the documentation of [`Atomic<T>::from_ptr`] for safety requirements.
473///
474/// The stabilized version of this intrinsic is available on the
475/// [`atomic`] unsigned integer types via the `fetch_max` method. For example, [`AtomicU32::fetch_max`].
476///
477/// [`Atomic<T>::from_ptr`]: AtomicU32::from_ptr
478/// [`Atomic<T>::fetch_max`]: AtomicU32::fetch_max
479#[rustc_intrinsic]
480#[rustc_nounwind]
481pub const unsafe fn atomic_umax<T: Copy, const ORD: AtomicOrdering>(dst: *mut T, src: T) -> T;
482
483/// An atomic fence.
484///
485/// The stabilized version of this intrinsic is available in
486/// [`atomic::fence`].
487#[rustc_intrinsic]
488#[rustc_nounwind]
489pub const unsafe fn atomic_fence<const ORD: AtomicOrdering>();
490
491/// An atomic fence for synchronization within a single thread.
492///
493/// The stabilized version of this intrinsic is available in
494/// [`atomic::compiler_fence`].
495#[rustc_intrinsic]
496#[rustc_nounwind]
497pub const unsafe fn atomic_singlethreadfence<const ORD: AtomicOrdering>();
498
499/// The `prefetch` intrinsic is a hint to the code generator to insert a prefetch instruction
500/// for the given address if supported; otherwise, it is a no-op.
501/// Prefetches have no effect on the behavior of the program but can change its performance
502/// characteristics.
503///
504/// The `LOCALITY` argument is a temporal locality specifier ranging from (0) - no locality,
505/// to (3) - extremely local keep in cache.
506///
507/// This intrinsic does not have a stable counterpart.
508#[rustc_intrinsic]
509#[rustc_nounwind]
510#[miri::intrinsic_fallback_is_spec]
511pub const fn prefetch_read_data<T, const LOCALITY: i32>(data: *const T) {
512    // This operation is a no-op, unless it is overridden by the backend.
513    let _ = data;
514}
515
516/// The `prefetch` intrinsic is a hint to the code generator to insert a prefetch instruction
517/// for the given address if supported; otherwise, it is a no-op.
518/// Prefetches have no effect on the behavior of the program but can change its performance
519/// characteristics.
520///
521/// The `LOCALITY` argument is a temporal locality specifier ranging from (0) - no locality,
522/// to (3) - extremely local keep in cache.
523///
524/// This intrinsic does not have a stable counterpart.
525#[rustc_intrinsic]
526#[rustc_nounwind]
527#[miri::intrinsic_fallback_is_spec]
528pub const fn prefetch_write_data<T, const LOCALITY: i32>(data: *const T) {
529    // This operation is a no-op, unless it is overridden by the backend.
530    let _ = data;
531}
532
533/// The `prefetch` intrinsic is a hint to the code generator to insert a prefetch instruction
534/// for the given address if supported; otherwise, it is a no-op.
535/// Prefetches have no effect on the behavior of the program but can change its performance
536/// characteristics.
537///
538/// The `LOCALITY` argument is a temporal locality specifier ranging from (0) - no locality,
539/// to (3) - extremely local keep in cache.
540///
541/// This intrinsic does not have a stable counterpart.
542#[rustc_intrinsic]
543#[rustc_nounwind]
544#[miri::intrinsic_fallback_is_spec]
545pub const fn prefetch_read_instruction<T, const LOCALITY: i32>(data: *const T) {
546    // This operation is a no-op, unless it is overridden by the backend.
547    let _ = data;
548}
549
550/// The `prefetch` intrinsic is a hint to the code generator to insert a prefetch instruction
551/// for the given address if supported; otherwise, it is a no-op.
552/// Prefetches have no effect on the behavior of the program but can change its performance
553/// characteristics.
554///
555/// The `LOCALITY` argument is a temporal locality specifier ranging from (0) - no locality,
556/// to (3) - extremely local keep in cache.
557///
558/// This intrinsic does not have a stable counterpart.
559#[rustc_intrinsic]
560#[rustc_nounwind]
561#[miri::intrinsic_fallback_is_spec]
562pub const fn prefetch_write_instruction<T, const LOCALITY: i32>(data: *const T) {
563    // This operation is a no-op, unless it is overridden by the backend.
564    let _ = data;
565}
566
567/// Executes a breakpoint trap, for inspection by a debugger.
568///
569/// The public version of this intrinsic is [`core::arch::breakpoint`].
570#[rustc_intrinsic]
571#[rustc_nounwind]
572pub fn breakpoint();
573
574/// Executes a breakpoint trap, for inspection by a debugger. This is allowed to resume.
575///
576/// The public version of this intrinsic is [`core::process::resumable_breakpoint`].
577#[rustc_intrinsic]
578#[rustc_nounwind]
579#[miri::intrinsic_fallback_is_spec]
580pub const fn resumable_breakpoint() {}
581
582/// A container for the string arguments passed to [`codeview_annotation`].
583#[unstable(feature = "codeview_annotation", issue = "163964")]
584pub trait CodeViewAnnotationArgs {
585    /// The string arguments to `codeview_annotation`.
586    const ARGS: &[&str];
587}
588
589/// Emits a call to [`llvm.codeview.annotation`](https://llvm.org/docs/LangRef.html#llvm-codeview-annotation-intrinsic)
590/// which results in [`CodeViewAnnotationArgs::ARGS`] being written to the
591/// PDB as an `S_ANNOTATION` record.
592///
593/// Works only with targets that use PDB debuginfo and with the LLVM backend.
594/// Is a no-op on other targets and backends.
595#[rustc_intrinsic]
596#[rustc_nounwind]
597#[miri::intrinsic_fallback_is_spec]
598pub fn codeview_annotation<T: CodeViewAnnotationArgs>() {
599    // const block to catch any errors in `T::ARGS`
600    const {
601        let _ignore = T::ARGS;
602    }
603}
604
605/// Magic intrinsic that derives its meaning from attributes
606/// attached to the function.
607///
608/// For example, dataflow uses this to inject static assertions so
609/// that `rustc_peek(potentially_uninitialized)` would actually
610/// double-check that dataflow did indeed compute that it is
611/// uninitialized at that point in the control flow.
612///
613/// This intrinsic should not be used outside of the compiler.
614#[rustc_nounwind]
615#[rustc_intrinsic]
616pub fn rustc_peek<T>(_: T) -> T;
617
618/// Ungracefully aborts the execution of the process.
619///
620/// This is the intrinsic for directly implementing [`core::process::crash`] which, on most
621/// platforms, will invoke an invalid instruction. On Unix, the process will probably terminate with
622/// a signal like `SIGABRT`, `SIGILL`, `SIGTRAP`, `SIGSEGV` or `SIGBUS`. The precise behavior is not
623/// guaranteed and not stable.
624///
625/// [`std::process::abort`](../../std/process/fn.abort.html) is to be preferred if possible,
626/// as its behavior is more user-friendly and more stable.
627///
628/// Note that, unlike most intrinsics, this is safe to call; it does not require an `unsafe` block.
629/// Therefore, implementations must not require the user to uphold any safety invariants.
630///
631/// The stabilized version of this intrinsic is [`core::process::crash`].
632#[rustc_nounwind]
633#[rustc_intrinsic]
634pub const fn abort_immediate() -> !;
635
636/// Informs the optimizer that this point in the code is not reachable,
637/// enabling further optimizations.
638///
639/// N.B., this is very different from the `unreachable!()` macro: Unlike the
640/// macro, which panics when it is executed, it is *undefined behavior* to
641/// reach code marked with this function.
642///
643/// The stabilized version of this intrinsic is [`core::hint::unreachable_unchecked`].
644#[rustc_intrinsic_const_stable_indirect]
645#[rustc_nounwind]
646#[rustc_intrinsic]
647pub const unsafe fn unreachable() -> !;
648
649/// Informs the optimizer that a condition is always true.
650/// If the condition is false, the behavior is undefined.
651///
652/// No code is generated for this intrinsic, but the optimizer will try
653/// to preserve it (and its condition) between passes, which may interfere
654/// with optimization of surrounding code and reduce performance. It should
655/// not be used if the invariant can be discovered by the optimizer on its
656/// own, or if it does not enable any significant optimizations.
657///
658/// The stabilized version of this intrinsic is [`core::hint::assert_unchecked`].
659#[rustc_nounwind]
660#[rustc_intrinsic]
661pub const unsafe fn assume(b: bool) {
662    if !b {
663        // SAFETY: the caller must guarantee the argument is never `false`
664        unsafe { unreachable() }
665    }
666}
667
668/// Hints to the compiler that current code path is cold.
669///
670/// Note that, unlike most intrinsics, this is safe to call;
671/// it does not require an `unsafe` block.
672/// Therefore, implementations must not require the user to uphold
673/// any safety invariants.
674///
675/// The stabilized version of this intrinsic is [`core::hint::cold_path`].
676#[rustc_intrinsic]
677#[rustc_nounwind]
678#[miri::intrinsic_fallback_is_spec]
679#[cold]
680pub const fn cold_path() {}
681
682/// Hints to the compiler that branch condition is likely to be true.
683/// Returns the value passed to it.
684///
685/// Any use other than with `if` statements will probably not have an effect.
686///
687/// Note that, unlike most intrinsics, this is safe to call;
688/// it does not require an `unsafe` block.
689/// Therefore, implementations must not require the user to uphold
690/// any safety invariants.
691///
692/// This intrinsic does not have a stable counterpart.
693#[rustc_nounwind]
694#[inline(always)]
695pub const fn likely(b: bool) -> bool {
696    if b {
697        true
698    } else {
699        cold_path();
700        false
701    }
702}
703
704/// Hints to the compiler that branch condition is likely to be false.
705/// Returns the value passed to it.
706///
707/// Any use other than with `if` statements will probably not have an effect.
708///
709/// Note that, unlike most intrinsics, this is safe to call;
710/// it does not require an `unsafe` block.
711/// Therefore, implementations must not require the user to uphold
712/// any safety invariants.
713///
714/// This intrinsic does not have a stable counterpart.
715#[rustc_nounwind]
716#[inline(always)]
717pub const fn unlikely(b: bool) -> bool {
718    if b {
719        cold_path();
720        true
721    } else {
722        false
723    }
724}
725
726/// Returns either `true_val` or `false_val` depending on condition `b` with a
727/// hint to the compiler that this condition is unlikely to be correctly
728/// predicted by a CPU's branch predictor (e.g. a binary search).
729///
730/// This is otherwise functionally equivalent to `if b { true_val } else { false_val }`.
731///
732/// Note that, unlike most intrinsics, this is safe to call;
733/// it does not require an `unsafe` block.
734/// Therefore, implementations must not require the user to uphold
735/// any safety invariants.
736///
737/// The public form of this intrinsic is [`core::hint::select_unpredictable`].
738/// However unlike the public form, the intrinsic will not drop the value that
739/// is not selected.
740#[rustc_const_unstable(feature = "core_intrinsics", issue = "none")]
741#[rustc_intrinsic]
742#[rustc_nounwind]
743#[miri::intrinsic_fallback_is_spec]
744#[inline]
745pub const fn select_unpredictable<T>(b: bool, true_val: T, false_val: T) -> T {
746    if b {
747        forget(false_val);
748        true_val
749    } else {
750        forget(true_val);
751        false_val
752    }
753}
754
755/// A guard for unsafe functions that cannot ever be executed if `T` is uninhabited:
756/// This will statically either panic, or do nothing. It does not *guarantee* to ever panic,
757/// and should only be called if an assertion failure will imply language UB in the following code.
758///
759/// This intrinsic does not have a stable counterpart.
760#[rustc_intrinsic_const_stable_indirect]
761#[rustc_nounwind]
762#[rustc_intrinsic]
763pub const fn assert_inhabited<T>();
764
765/// A guard for unsafe functions that cannot ever be executed if `T` does not permit
766/// zero-initialization: This will statically either panic, or do nothing. It does not *guarantee*
767/// to ever panic, and should only be called if an assertion failure will imply language UB in the
768/// following code.
769///
770/// This intrinsic does not have a stable counterpart.
771#[rustc_intrinsic_const_stable_indirect]
772#[rustc_nounwind]
773#[rustc_intrinsic]
774pub const fn assert_zero_valid<T>();
775
776/// A guard for `std::mem::uninitialized`. This will statically either panic, or do nothing. It does
777/// not *guarantee* to ever panic, and should only be called if an assertion failure will imply
778/// language UB in the following code.
779///
780/// This intrinsic does not have a stable counterpart.
781#[rustc_intrinsic_const_stable_indirect]
782#[rustc_nounwind]
783#[rustc_intrinsic]
784pub const fn assert_mem_uninitialized_valid<T>();
785
786/// Gets a reference to a static `Location` indicating where it was called.
787///
788/// Note that, unlike most intrinsics, this is safe to call;
789/// it does not require an `unsafe` block.
790/// Therefore, implementations must not require the user to uphold
791/// any safety invariants.
792///
793/// Consider using [`core::panic::Location::caller`] instead.
794#[rustc_intrinsic_const_stable_indirect]
795#[rustc_nounwind]
796#[rustc_intrinsic]
797pub const fn caller_location() -> &'static crate::panic::Location<'static>;
798
799/// Moves a value out of scope without running drop glue.
800///
801/// This exists solely for [`crate::mem::forget_unsized`]; normal `forget` uses
802/// `ManuallyDrop` instead.
803///
804/// Note that, unlike most intrinsics, this is safe to call;
805/// it does not require an `unsafe` block.
806/// Therefore, implementations must not require the user to uphold
807/// any safety invariants.
808#[rustc_intrinsic_const_stable_indirect]
809#[rustc_nounwind]
810#[rustc_intrinsic]
811pub const fn forget<T: ?Sized>(_: T);
812
813/// Reinterprets the bits of a value of one type as another type.
814///
815/// Both types must have the same size. Compilation will fail if this is not guaranteed.
816///
817/// `transmute` is semantically equivalent to a bitwise move of one type
818/// into another. It copies the bits from the source value into the
819/// destination value, then forgets the original. Note that source and destination
820/// are passed by-value, which means if `Src` or `Dst` contain padding, that padding
821/// is *not* guaranteed to be preserved by `transmute`.
822///
823/// Both the argument and the result must be [valid](../../nomicon/what-unsafe-does.html) at
824/// their given type. Violating this condition leads to [undefined behavior][ub]. The compiler
825/// will generate code *assuming that you, the programmer, ensure that there will never be
826/// undefined behavior*. It is therefore your responsibility to guarantee that every value
827/// passed to `transmute` is valid at both types `Src` and `Dst`. Failing to uphold this condition
828/// may lead to unexpected and unstable compilation results. This makes `transmute` **incredibly
829/// unsafe**. `transmute` should be the absolute last resort.
830///
831/// Because `transmute` is a by-value operation, alignment of the *transmuted values
832/// themselves* is not a concern. As with any other function, the compiler already ensures
833/// both `Src` and `Dst` are properly aligned. However, when transmuting values that *point
834/// elsewhere* (such as pointers, references, boxes…), the caller has to ensure proper
835/// alignment of the pointed-to values.
836///
837/// The [nomicon](../../nomicon/transmutes.html) has additional documentation.
838///
839/// [ub]: ../../reference/behavior-considered-undefined.html
840///
841/// # Transmutation between pointers and integers
842///
843/// Special care has to be taken when transmuting between pointers and integers, e.g.
844/// transmuting between `*const ()` and `usize`.
845///
846/// Transmuting *pointers to integers* in a `const` context is [undefined behavior][ub], unless
847/// the pointer was originally created *from* an integer. (That includes this function
848/// specifically, integer-to-pointer casts, and helpers like [`dangling`][crate::ptr::dangling],
849/// but also semantically-equivalent conversions such as punning through `repr(C)` union
850/// fields.) Any attempt to use the resulting value for integer operations will abort
851/// const-evaluation. (And even outside `const`, such transmutation is touching on many
852/// unspecified aspects of the Rust memory model and should be avoided. See below for
853/// alternatives.)
854///
855/// Transmuting *integers to pointers* is a largely unspecified operation. It is likely *not*
856/// equivalent to an `as` cast. Doing non-zero-sized memory accesses with a pointer constructed
857/// this way is currently considered undefined behavior.
858///
859/// All this also applies when the integer is nested inside an array, tuple, struct, or enum.
860/// However, `MaybeUninit<usize>` is not considered an integer type for the purpose of this
861/// section. Transmuting `*const ()` to `MaybeUninit<usize>` is fine---but then calling
862/// `assume_init()` on that result is considered as completing the pointer-to-integer transmute
863/// and thus runs into the issues discussed above.
864///
865/// In particular, doing a pointer-to-integer-to-pointer roundtrip via `transmute` is *not* a
866/// lossless process. If you want to round-trip a pointer through an integer in a way that you
867/// can get back the original pointer, you need to use `as` casts, or replace the integer type
868/// by `MaybeUninit<$int>` (and never call `assume_init()`). If you are looking for a way to
869/// store data of arbitrary type, also use `MaybeUninit<T>` (that will also handle uninitialized
870/// memory due to padding). If you specifically need to store something that is "either an
871/// integer or a pointer", use `*mut ()`: integers can be converted to pointers and back without
872/// any loss (via `as` casts or via `transmute`).
873///
874/// # Examples
875///
876/// There are a few things that `transmute` is really useful for.
877///
878/// Turning a pointer into a function pointer. This is *not* portable to
879/// machines where function pointers and data pointers have different sizes.
880///
881/// ```
882/// fn foo() -> i32 {
883///     0
884/// }
885/// // Crucially, we `as`-cast to a raw pointer before `transmute`ing to a function pointer.
886/// // This avoids an integer-to-pointer `transmute`, which can be problematic.
887/// // Transmuting between raw pointers and function pointers (i.e., two pointer types) is fine.
888/// let pointer = foo as fn() -> i32 as *const ();
889/// let function = unsafe {
890///     std::mem::transmute::<*const (), fn() -> i32>(pointer)
891/// };
892/// assert_eq!(function(), 0);
893/// ```
894///
895/// Extending a lifetime, or shortening an invariant lifetime. This is
896/// advanced, very unsafe Rust!
897///
898/// ```
899/// struct R<'a>(&'a i32);
900/// unsafe fn extend_lifetime<'b>(r: R<'b>) -> R<'static> {
901///     unsafe { std::mem::transmute::<R<'b>, R<'static>>(r) }
902/// }
903///
904/// unsafe fn shorten_invariant_lifetime<'b, 'c>(r: &'b mut R<'static>)
905///                                              -> &'b mut R<'c> {
906///     unsafe { std::mem::transmute::<&'b mut R<'static>, &'b mut R<'c>>(r) }
907/// }
908/// ```
909///
910/// # Alternatives
911///
912/// Don't despair: many uses of `transmute` can be achieved through other means.
913/// Below are common applications of `transmute` which can be replaced with safer
914/// constructs.
915///
916/// Turning raw bytes (`[u8; SZ]`) into `u32`, `f64`, etc.:
917///
918/// ```
919/// # #![allow(unnecessary_transmutes)]
920/// let raw_bytes = [0x78, 0x56, 0x34, 0x12];
921///
922/// let num = unsafe {
923///     std::mem::transmute::<[u8; 4], u32>(raw_bytes)
924/// };
925///
926/// // use `u32::from_ne_bytes` instead
927/// let num = u32::from_ne_bytes(raw_bytes);
928/// // or use `u32::from_le_bytes` or `u32::from_be_bytes` to specify the endianness
929/// let num = u32::from_le_bytes(raw_bytes);
930/// assert_eq!(num, 0x12345678);
931/// let num = u32::from_be_bytes(raw_bytes);
932/// assert_eq!(num, 0x78563412);
933/// ```
934///
935/// Turning a pointer into a `usize`:
936///
937/// ```no_run
938/// let ptr = &0;
939/// let ptr_num_transmute = unsafe {
940///     std::mem::transmute::<&i32, usize>(ptr)
941/// };
942///
943/// // Use an `as` cast instead
944/// let ptr_num_cast = ptr as *const i32 as usize;
945/// ```
946///
947/// Note that using `transmute` to turn a pointer to a `usize` is (as noted above) [undefined
948/// behavior][ub] in `const` contexts. Also outside of consts, this operation might not behave
949/// as expected -- this is touching on many unspecified aspects of the Rust memory model.
950/// Depending on what the code is doing, the following alternatives are preferable to
951/// pointer-to-integer transmutation:
952/// - If the code just wants to store data of arbitrary type in some buffer and needs to pick a
953///   type for that buffer, it can use [`MaybeUninit`][crate::mem::MaybeUninit].
954/// - If the code actually wants to work on the address the pointer points to, it can use `as`
955///   casts or [`ptr.addr()`][pointer::addr].
956///
957/// Turning a `*mut T` into a `&mut T`:
958///
959/// ```
960/// let ptr: *mut i32 = &mut 0;
961/// let ref_transmuted = unsafe {
962///     std::mem::transmute::<*mut i32, &mut i32>(ptr)
963/// };
964///
965/// // Use a reborrow instead
966/// let ref_casted = unsafe { &mut *ptr };
967/// ```
968///
969/// Turning a `&mut T` into a `&mut U`:
970///
971/// ```
972/// let ptr = &mut 0;
973/// let val_transmuted = unsafe {
974///     std::mem::transmute::<&mut i32, &mut u32>(ptr)
975/// };
976///
977/// // Now, put together `as` and reborrowing - note the chaining of `as`
978/// // `as` is not transitive
979/// let val_casts = unsafe { &mut *(ptr as *mut i32 as *mut u32) };
980/// ```
981///
982/// Turning a `&str` into a `&[u8]`:
983///
984/// ```
985/// // this is not a good way to do this.
986/// let slice = unsafe { std::mem::transmute::<&str, &[u8]>("Rust") };
987/// assert_eq!(slice, &[82, 117, 115, 116]);
988///
989/// // You could use `str::as_bytes`
990/// let slice = "Rust".as_bytes();
991/// assert_eq!(slice, &[82, 117, 115, 116]);
992///
993/// // Or, just use a byte string, if you have control over the string
994/// // literal
995/// assert_eq!(b"Rust", &[82, 117, 115, 116]);
996/// ```
997///
998/// Turning a `Vec<&T>` into a `Vec<Option<&T>>`.
999///
1000/// To transmute the inner type of the contents of a container, you must make sure to not
1001/// violate any of the container's invariants. For `Vec`, this means that both the size
1002/// *and alignment* of the inner types have to match. Other containers might rely on the
1003/// size of the type, alignment, or even the `TypeId`, in which case transmuting wouldn't
1004/// be possible at all without violating the container invariants.
1005///
1006/// ```
1007/// let store = [0, 1, 2, 3];
1008/// let v_orig = store.iter().collect::<Vec<&i32>>();
1009///
1010/// // clone the vector as we will reuse them later
1011/// let v_clone = v_orig.clone();
1012///
1013/// // Using transmute: this relies on the unspecified data layout of `Vec`, which is a
1014/// // bad idea and could cause Undefined Behavior.
1015/// // However, it is no-copy.
1016/// let v_transmuted = unsafe {
1017///     std::mem::transmute::<Vec<&i32>, Vec<Option<&i32>>>(v_clone)
1018/// };
1019///
1020/// let v_clone = v_orig.clone();
1021///
1022/// // This is the suggested, safe way.
1023/// // It may copy the entire vector into a new one though, but also may not.
1024/// let v_collected = v_clone.into_iter()
1025///                          .map(Some)
1026///                          .collect::<Vec<Option<&i32>>>();
1027///
1028/// let v_clone = v_orig.clone();
1029///
1030/// // This is the proper no-copy, unsafe way of "transmuting" a `Vec`, without relying on the
1031/// // data layout. Instead of literally calling `transmute`, we perform a pointer cast, but
1032/// // in terms of converting the original inner type (`&i32`) to the new one (`Option<&i32>`),
1033/// // this has all the same caveats. Besides the information provided above, also consult the
1034/// // [`from_raw_parts`] documentation.
1035/// let (ptr, len, capacity) = v_clone.into_raw_parts();
1036/// let v_from_raw = unsafe {
1037///     Vec::from_raw_parts(ptr.cast::<*mut Option<&i32>>(), len, capacity)
1038/// };
1039/// ```
1040///
1041/// [`from_raw_parts`]: ../../std/vec/struct.Vec.html#method.from_raw_parts
1042///
1043/// Implementing `split_at_mut`:
1044///
1045/// ```
1046/// use std::{slice, mem};
1047///
1048/// // There are multiple ways to do this, and there are multiple problems
1049/// // with the following (transmute) way.
1050/// fn split_at_mut_transmute<T>(slice: &mut [T], mid: usize)
1051///                              -> (&mut [T], &mut [T]) {
1052///     let len = slice.len();
1053///     assert!(mid <= len);
1054///     unsafe {
1055///         let slice2 = mem::transmute::<&mut [T], &mut [T]>(slice);
1056///         // first: transmute is not type safe; all it checks is that T and
1057///         // U are of the same size. Second, right here, you have two
1058///         // mutable references pointing to the same memory.
1059///         (&mut slice[0..mid], &mut slice2[mid..len])
1060///     }
1061/// }
1062///
1063/// // This gets rid of the type safety problems; `&mut *` will *only* give
1064/// // you a `&mut T` from a `&mut T` or `*mut T`.
1065/// fn split_at_mut_casts<T>(slice: &mut [T], mid: usize)
1066///                          -> (&mut [T], &mut [T]) {
1067///     let len = slice.len();
1068///     assert!(mid <= len);
1069///     unsafe {
1070///         let slice2 = &mut *(slice as *mut [T]);
1071///         // however, you still have two mutable references pointing to
1072///         // the same memory.
1073///         (&mut slice[0..mid], &mut slice2[mid..len])
1074///     }
1075/// }
1076///
1077/// // This is how the standard library does it. This is the best method, if
1078/// // you need to do something like this
1079/// fn split_at_stdlib<T>(to_split: &mut [T], mid: usize)
1080///                       -> (&mut [T], &mut [T]) {
1081///     let len = to_split.len();
1082///     assert!(mid <= len);
1083///     unsafe {
1084///         let ptr = to_split.as_mut_ptr();
1085///         let fst = slice::from_raw_parts_mut(ptr, mid);
1086///         let snd = slice::from_raw_parts_mut(ptr.add(mid), len - mid);
1087///         // The function now has three mutable references to overlapping memory:
1088///         // `to_split`, `fst`, and `snd`.
1089///         // `to_split` is never used after `let ptr = ...` so it can be treated as "dead".
1090///         // This leaves two "live" mutable slice references, `fst` and `snd`, with no overlap.
1091///         (fst, snd)
1092///     }
1093/// }
1094/// ```
1095#[stable(feature = "rust1", since = "1.0.0")]
1096#[rustc_allowed_through_unstable_modules(
1097    message = "import this function via the `mem` module instead",
1098    module = "mem"
1099)]
1100#[rustc_const_stable(feature = "const_transmute", since = "1.56.0")]
1101#[rustc_diagnostic_item = "transmute"]
1102#[rustc_nounwind]
1103#[rustc_intrinsic]
1104pub const unsafe fn transmute<Src, Dst>(src: Src) -> Dst;
1105
1106/// Like [`transmute`], but even less checked at compile-time: rather than
1107/// giving an error for `size_of::<Src>() != size_of::<Dst>()`, it's
1108/// **Undefined Behavior** at runtime.
1109///
1110/// Prefer normal `transmute` where possible, for the extra checking, since
1111/// both do exactly the same thing at runtime, if they both compile.
1112///
1113/// This is not expected to ever be exposed directly to users, rather it
1114/// may eventually be exposed through some more-constrained API.
1115#[rustc_intrinsic_const_stable_indirect]
1116#[rustc_nounwind]
1117#[rustc_intrinsic]
1118pub const unsafe fn transmute_unchecked<Src, Dst>(src: Src) -> Dst;
1119
1120/// Returns `true` if the actual type given as `T` requires drop
1121/// glue; returns `false` if the actual type provided for `T`
1122/// implements `Copy`.
1123///
1124/// If the actual type neither requires drop glue nor implements
1125/// `Copy`, then the return value of this function is unspecified.
1126///
1127/// Note that, unlike most intrinsics, this can only be called at compile-time
1128/// as backends do not have an implementation for it. The only caller (its
1129/// stable counterpart) wraps this intrinsic call in a `const` block so that
1130/// backends only see an evaluated constant.
1131///
1132/// The stabilized version of this intrinsic is [`mem::needs_drop`](crate::mem::needs_drop).
1133#[rustc_intrinsic_const_stable_indirect]
1134#[rustc_nounwind]
1135#[rustc_intrinsic]
1136#[rustc_comptime]
1137pub fn needs_drop<T: ?Sized>() -> bool;
1138
1139/// Calculates the offset from a pointer.
1140///
1141/// This is implemented as an intrinsic to avoid converting to and from an
1142/// integer, since the conversion would throw away aliasing information.
1143///
1144/// This can only be used with `Ptr` as a raw pointer type (`*mut` or `*const`)
1145/// to a `Sized` pointee and with `Delta` as `usize` or `isize`.  Any other
1146/// instantiations may arbitrarily misbehave, and that's *not* a compiler bug.
1147///
1148/// # Safety
1149///
1150/// If the computed offset is non-zero, then both the starting and resulting pointer must be
1151/// either in bounds or at the end of an allocation. If either pointer is out
1152/// of bounds or arithmetic overflow occurs then this operation is undefined behavior.
1153///
1154/// The stabilized version of this intrinsic is [`pointer::offset`].
1155#[must_use = "returns a new pointer rather than modifying its argument"]
1156#[rustc_intrinsic_const_stable_indirect]
1157#[rustc_nounwind]
1158#[rustc_intrinsic]
1159pub const unsafe fn offset<Ptr: bounds::BuiltinDeref, Delta>(dst: Ptr, offset: Delta) -> Ptr;
1160
1161/// Calculates the offset from a pointer, potentially wrapping.
1162///
1163/// This is implemented as an intrinsic to avoid converting to and from an
1164/// integer, since the conversion inhibits certain optimizations.
1165///
1166/// # Safety
1167///
1168/// Unlike the `offset` intrinsic, this intrinsic does not restrict the
1169/// resulting pointer to point into or at the end of an allocated
1170/// object, and it wraps with two's complement arithmetic. The resulting
1171/// value is not necessarily valid to be used to actually access memory.
1172///
1173/// The stabilized version of this intrinsic is [`pointer::wrapping_offset`].
1174#[must_use = "returns a new pointer rather than modifying its argument"]
1175#[rustc_intrinsic_const_stable_indirect]
1176#[rustc_nounwind]
1177#[rustc_intrinsic]
1178pub const unsafe fn arith_offset<T>(dst: *const T, offset: isize) -> *const T;
1179
1180/// Projects to the `index`-th element of `slice_ptr`, as the same kind of pointer
1181/// as the slice was provided -- so `&mut [T] → &mut T`, `&[T] → &T`,
1182/// `*mut [T] → *mut T`, or `*const [T] → *const T` -- without a bounds check.
1183///
1184/// This is exposed via `<usize as SliceIndex>::get(_unchecked)(_mut)`,
1185/// and isn't intended to be used elsewhere.
1186///
1187/// Expands in MIR to `{&, &mut, &raw const, &raw mut} (*slice_ptr)[index]`,
1188/// depending on the types involved, so no backend support is needed.
1189///
1190/// # Safety
1191///
1192/// - `index < PtrMetadata(slice_ptr)`, so the indexing is in-bounds for the slice
1193/// - the resulting offsetting is in-bounds of the allocation, which is
1194///   always the case for references, but needs to be upheld manually for pointers
1195#[rustc_nounwind]
1196#[rustc_intrinsic]
1197pub const unsafe fn slice_get_unchecked<
1198    ItemPtr: bounds::ChangePointee<[T], Pointee = T, Output = SlicePtr>,
1199    SlicePtr,
1200    T,
1201>(
1202    slice_ptr: SlicePtr,
1203    index: usize,
1204) -> ItemPtr;
1205
1206/// Masks out bits of the pointer according to a mask.
1207///
1208/// Note that, unlike most intrinsics, this is safe to call;
1209/// it does not require an `unsafe` block.
1210/// Therefore, implementations must not require the user to uphold
1211/// any safety invariants.
1212///
1213/// Consider using [`pointer::mask`] instead.
1214#[rustc_nounwind]
1215#[rustc_intrinsic]
1216pub fn ptr_mask<T>(ptr: *const T, mask: usize) -> *const T;
1217
1218/// Equivalent to the appropriate `llvm.memcpy.p0i8.0i8.*` intrinsic, with
1219/// a size of `count` * `size_of::<T>()` and an alignment of `align_of::<T>()`.
1220///
1221/// This intrinsic does not have a stable counterpart.
1222/// # Safety
1223///
1224/// The safety requirements are consistent with [`copy_nonoverlapping`]
1225/// while the read and write behaviors are volatile,
1226/// which means it will not be optimized out unless `_count` or `size_of::<T>()` is equal to zero.
1227///
1228/// [`copy_nonoverlapping`]: ptr::copy_nonoverlapping
1229#[rustc_intrinsic]
1230#[rustc_nounwind]
1231pub unsafe fn volatile_copy_nonoverlapping_memory<T>(dst: *mut T, src: *const T, count: usize);
1232/// Equivalent to the appropriate `llvm.memmove.p0i8.0i8.*` intrinsic, with
1233/// a size of `count * size_of::<T>()` and an alignment of `align_of::<T>()`.
1234///
1235/// The volatile parameter is set to `true`, so it will not be optimized out
1236/// unless size is equal to zero.
1237///
1238/// This intrinsic does not have a stable counterpart.
1239#[rustc_intrinsic]
1240#[rustc_nounwind]
1241pub unsafe fn volatile_copy_memory<T>(dst: *mut T, src: *const T, count: usize);
1242/// Equivalent to the appropriate `llvm.memset.p0i8.*` intrinsic, with a
1243/// size of `count * size_of::<T>()` and an alignment of `align_of::<T>()`.
1244///
1245/// This intrinsic does not have a stable counterpart.
1246/// # Safety
1247///
1248/// The safety requirements are consistent with [`write_bytes`] while the write behavior is volatile,
1249/// which means it will not be optimized out unless `_count` or `size_of::<T>()` is equal to zero.
1250///
1251/// [`write_bytes`]: ptr::write_bytes
1252#[rustc_intrinsic]
1253#[rustc_nounwind]
1254pub const unsafe fn volatile_set_memory<T>(dst: *mut T, val: u8, count: usize);
1255
1256/// Performs a volatile load from the `src` pointer.
1257///
1258/// The stabilized version of this intrinsic is [`core::ptr::read_volatile`].
1259#[rustc_intrinsic]
1260#[rustc_nounwind]
1261pub const unsafe fn volatile_load<T>(src: *const T) -> T;
1262/// Performs a volatile store to the `dst` pointer.
1263///
1264/// The stabilized version of this intrinsic is [`core::ptr::write_volatile`].
1265#[rustc_intrinsic]
1266#[rustc_nounwind]
1267pub const unsafe fn volatile_store<T>(dst: *mut T, val: T);
1268
1269/// Performs a volatile load from the `src` pointer
1270/// The pointer is not required to be aligned.
1271///
1272/// This intrinsic does not have a stable counterpart.
1273#[rustc_intrinsic]
1274#[rustc_nounwind]
1275#[rustc_diagnostic_item = "intrinsics_unaligned_volatile_load"]
1276pub unsafe fn unaligned_volatile_load<T>(src: *const T) -> T;
1277/// Performs a volatile store to the `dst` pointer.
1278/// The pointer is not required to be aligned.
1279///
1280/// This intrinsic does not have a stable counterpart.
1281#[rustc_intrinsic]
1282#[rustc_nounwind]
1283#[rustc_diagnostic_item = "intrinsics_unaligned_volatile_store"]
1284pub unsafe fn unaligned_volatile_store<T>(dst: *mut T, val: T);
1285
1286intrinsic_dispatch_on_type! {
1287    /// Returns the square root of a floating-point value.
1288    ///
1289    /// The stabilized versions of this intrinsic are available on the float primitives via the
1290    /// `sqrt` method. For example, [`f32::sqrt`](../../std/primitive.f32.html#method.sqrt).
1291    #[rustc_nounwind]
1292    #[rustc_intrinsic]
1293    pub fn sqrt<T: bounds::FloatPrimitive>(x: T) -> T;
1294
1295    f16 => { libm::sqrtf16(x) }
1296    f32 => { libm::sqrtf(x) }
1297    f64 => { libm::sqrt(x) }
1298    f128 => { libm::sqrtf128(x) }
1299}
1300
1301intrinsic_dispatch_on_type! {
1302    /// Raises a floating-point value to an integer power.
1303    ///
1304    /// The stabilized versions of this intrinsic are available on the float primitives via the
1305    /// `powi` method. For example, [`f32::powi`](../../std/primitive.f32.html#method.powi).
1306    #[rustc_nounwind]
1307    #[inline]
1308    #[rustc_intrinsic]
1309    pub fn powi<T: bounds::FloatPrimitive>(a: T, x: i32) -> T;
1310
1311    f16 => { powi(a as f32, x) as f16 }
1312    f32 => { libm::__powisf2(a, x) }
1313    f64 => { libm::__powidf2(a, x) }
1314    f128 => { libm::__powitf2(a, x) }
1315}
1316
1317intrinsic_dispatch_on_type! {
1318    /// Returns the sine of a floating-point value.
1319    ///
1320    /// The stabilized versions of this intrinsic are available on the float primitives via the
1321    /// `sin` method. For example, [`f32::sin`](../../std/primitive.f32.html#method.sin).
1322    #[rustc_nounwind]
1323    #[inline]
1324    #[rustc_intrinsic]
1325    pub fn sin<T: bounds::FloatPrimitive>(x: T) -> T;
1326
1327    f16 => { sin(x as f32) as f16 }
1328    f32 => {
1329        cfg_select! {
1330            all(target_env = "msvc", target_arch = "x86") => sin(x as f64) as f32,
1331            _ => libm::likely_available::sinf(x),
1332        }
1333    }
1334    f64 => { libm::likely_available::sin(x) }
1335    f128 => { libm::maybe_available::sinf128(x) }
1336}
1337
1338intrinsic_dispatch_on_type! {
1339    /// Returns the cosine of a floating-point value.
1340    ///
1341    /// The stabilized versions of this intrinsic are available on the float primitives via the
1342    /// `cos` method. For example, [`f32::cos`](../../std/primitive.f32.html#method.cos).
1343    #[rustc_nounwind]
1344    #[inline]
1345    #[rustc_intrinsic]
1346    pub fn cos<T: bounds::FloatPrimitive>(x: T) -> T;
1347
1348    f16 => { cos(x as f32) as f16 }
1349    f32 => {
1350        cfg_select! {
1351            all(target_env = "msvc", target_arch = "x86") => cos(x as f64) as f32,
1352            _ => libm::likely_available::cosf(x),
1353        }
1354    }
1355    f64 => { libm::likely_available::cos(x) }
1356    f128 => { libm::maybe_available::cosf128(x) }
1357}
1358
1359intrinsic_dispatch_on_type! {
1360    /// Raises a floating-point value to a power of the same type.
1361    ///
1362    /// The stabilized versions of this intrinsic are available on the float primitives via the
1363    /// `powf` method. For example, [`f32::powf`](../../std/primitive.f32.html#method.powf).
1364    #[rustc_nounwind]
1365    #[inline]
1366    #[rustc_intrinsic]
1367    pub fn powf<T: bounds::FloatPrimitive>(a: T, x: T) -> T;
1368
1369    f16 => { powf(a as f32, x as f32) as f16 }
1370    f32 => {
1371        cfg_select! {
1372            all(target_env = "msvc", target_arch = "x86") => powf(a as f64, x as f64) as f32,
1373            _ => libm::likely_available::powf(a, x),
1374        }
1375    }
1376    f64 => { libm::likely_available::pow(a, x) }
1377    f128 => { libm::maybe_available::powf128(a, x) }
1378}
1379
1380intrinsic_dispatch_on_type! {
1381    /// Returns the exponential of a floating-point value.
1382    ///
1383    /// The stabilized versions of this intrinsic are available on the float primitives via the
1384    /// `exp` method. For example, [`f32::exp`](../../std/primitive.f32.html#method.exp).
1385    #[rustc_nounwind]
1386    #[inline]
1387    #[rustc_intrinsic]
1388    pub fn exp<T: bounds::FloatPrimitive>(x: T) -> T;
1389
1390    f16 => { exp(x as f32) as f16 }
1391    f32 => {
1392        cfg_select! {
1393            all(target_env = "msvc", target_arch = "x86") => exp(x as f64) as f32,
1394            _ => libm::likely_available::expf(x),
1395        }
1396    }
1397    f64 => { libm::likely_available::exp(x) }
1398    f128 => { libm::maybe_available::expf128(x) }
1399}
1400
1401intrinsic_dispatch_on_type! {
1402    /// Returns 2 raised to the power of a floating-point value.
1403    ///
1404    /// The stabilized versions of this intrinsic are available on the float primitives via the
1405    /// `exp2` method. For example, [`f32::exp2`](../../std/primitive.f32.html#method.exp2).
1406    #[rustc_nounwind]
1407    #[inline]
1408    #[rustc_intrinsic]
1409    pub fn exp2<T: bounds::FloatPrimitive>(x: T) -> T;
1410
1411    f16 => { exp2(x as f32) as f16 }
1412    f32 => {
1413        cfg_select! {
1414            all(target_env = "msvc", target_arch = "x86") => exp2(x as f64) as f32,
1415            _ => libm::likely_available::exp2f(x),
1416        }
1417    }
1418    f64 => { libm::likely_available::exp2(x) }
1419    f128 => { libm::maybe_available::exp2f128(x) }
1420}
1421
1422intrinsic_dispatch_on_type! {
1423    /// Returns the natural logarithm of a floating-point value.
1424    ///
1425    /// The stabilized versions of this intrinsic are available on the float primitives via the
1426    /// `ln` method. For example, [`f32::ln`](../../std/primitive.f32.html#method.ln).
1427    #[rustc_nounwind]
1428    #[inline]
1429    #[rustc_intrinsic]
1430    pub fn log<T: bounds::FloatPrimitive>(x: T) -> T;
1431
1432    f16 => { log(x as f32) as f16 }
1433    f32 => {
1434        cfg_select! {
1435            all(target_env = "msvc", target_arch = "x86") => log(x as f64) as f32,
1436            _ => libm::likely_available::logf(x),
1437        }
1438    }
1439    f64 => { libm::likely_available::log(x) }
1440    f128 => { libm::maybe_available::logf128(x) }
1441}
1442
1443intrinsic_dispatch_on_type! {
1444    /// Returns the base 10 logarithm of a floating-point value.
1445    ///
1446    /// The stabilized versions of this intrinsic are available on the float primitives via the
1447    /// `log10` method. For example, [`f32::log10`](../../std/primitive.f32.html#method.log10).
1448    #[rustc_nounwind]
1449    #[inline]
1450    #[rustc_intrinsic]
1451    pub fn log10<T: bounds::FloatPrimitive>(x: T) -> T;
1452
1453    f16 => { log10(x as f32) as f16 }
1454    f32 => {
1455        cfg_select! {
1456            all(target_env = "msvc", target_arch = "x86") => log10(x as f64) as f32,
1457            _ => libm::likely_available::log10f(x),
1458        }
1459    }
1460    f64 => { libm::likely_available::log10(x) }
1461    f128 => { libm::maybe_available::log10f128(x) }
1462}
1463
1464intrinsic_dispatch_on_type! {
1465    /// Returns the base 2 logarithm of a floating-point value.
1466    ///
1467    /// The stabilized versions of this intrinsic are available on the float primitives via the
1468    /// `log2` method. For example, [`f32::log2`](../../std/primitive.f32.html#method.log2).
1469    #[rustc_nounwind]
1470    #[inline]
1471    #[rustc_intrinsic]
1472    pub fn log2<T: bounds::FloatPrimitive>(x: T) -> T;
1473
1474    f16 => { log2(x as f32) as f16 }
1475    f32 => {
1476        cfg_select! {
1477            all(target_env = "msvc", target_arch = "x86") => log2(x as f64) as f32,
1478            _ => libm::likely_available::log2f(x),
1479        }
1480    }
1481    f64 => { libm::likely_available::log2(x) }
1482    f128 => { libm::maybe_available::log2f128(x) }
1483}
1484
1485intrinsic_dispatch_on_type! {
1486    /// Returns `a * b + c` without rounding the intermediate result.
1487    ///
1488    /// The stabilized versions of this intrinsic are available on the float primitives via the
1489    /// `mul_add` method. For example,
1490    /// [`f32::mul_add`](../../std/primitive.f32.html#method.mul_add).
1491    #[rustc_nounwind]
1492    #[inline]
1493    #[rustc_intrinsic_const_stable_indirect]
1494    #[rustc_intrinsic]
1495    // The fallback bodies call into `libm`, which is not available at compile time. This is fine
1496    // because const-eval implements this intrinsic itself and never runs the fallback body.
1497    #[rustc_do_not_const_check]
1498    pub const fn fma<T: bounds::FloatPrimitive>(a: T, b: T, c: T) -> T;
1499
1500    f16 => { libm::fmaf16(a, b, c) }
1501    f32 => { libm::fmaf(a, b, c) }
1502    f64 => { libm::fma(a, b, c) }
1503    f128 => { libm::fmaf128(a, b, c) }
1504}
1505
1506/// Returns `a * b + c`, non-deterministically executing either a fused multiply-add or two
1507/// operations with rounding of the intermediate result.
1508///
1509/// The operation is fused if the code generator determines that target
1510/// instruction set has support for a fused operation, and that the fused
1511/// operation is more efficient than the equivalent, separate pair of mul
1512/// and add instructions. It is unspecified whether or not a fused operation
1513/// is selected, and that may depend on optimization level and context, for
1514/// example.
1515///
1516/// The stabilized versions of this intrinsic are available on the float primitives via the
1517/// `mul_add_relaxed` method. For example,
1518/// [`f32::mul_add_relaxed`](../../std/primitive.f32.html#method.mul_add_relaxed).
1519#[inline]
1520#[rustc_intrinsic_const_stable_indirect]
1521#[rustc_intrinsic]
1522#[rustc_nounwind]
1523#[rustc_do_not_const_check]
1524pub const fn fmuladd<T: bounds::FloatPrimitive>(a: T, b: T, c: T) -> T {
1525    a * b + c
1526}
1527
1528intrinsic_dispatch_on_type! {
1529    /// Returns the largest integer less than or equal to a floating-point value.
1530    ///
1531    /// The stabilized versions of this intrinsic are available on the float primitives via the
1532    /// `floor` method. For example, [`f32::floor`](../../std/primitive.f32.html#method.floor).
1533    #[rustc_nounwind]
1534    #[inline]
1535    #[rustc_intrinsic_const_stable_indirect]
1536    #[rustc_intrinsic]
1537    // The fallback bodies call into `libm`, which is not available at compile time. This is fine
1538    // because const-eval implements this intrinsic itself and never runs the fallback body.
1539    #[rustc_do_not_const_check]
1540    pub const fn floor<T: bounds::FloatPrimitive>(x: T) -> T;
1541
1542    f16 => { libm::floorf16(x) }
1543    f32 => { libm::floorf(x) }
1544    f64 => { libm::floor(x) }
1545    f128 => { libm::floorf128(x) }
1546}
1547
1548intrinsic_dispatch_on_type! {
1549    /// Returns the smallest integer greater than or equal to a floating-point value.
1550    ///
1551    /// The stabilized versions of this intrinsic are available on the float primitives via the
1552    /// `ceil` method. For example, [`f32::ceil`](../../std/primitive.f32.html#method.ceil).
1553    #[rustc_nounwind]
1554    #[inline]
1555    #[rustc_intrinsic_const_stable_indirect]
1556    #[rustc_intrinsic]
1557    // The fallback bodies call into `libm`, which is not available at compile time. This is fine
1558    // because const-eval implements this intrinsic itself and never runs the fallback body.
1559    #[rustc_do_not_const_check]
1560    pub const fn ceil<T: bounds::FloatPrimitive>(x: T) -> T;
1561
1562    f16 => { libm::ceilf16(x) }
1563    f32 => { libm::ceilf(x) }
1564    f64 => { libm::ceil(x) }
1565    f128 => { libm::ceilf128(x) }
1566}
1567
1568intrinsic_dispatch_on_type! {
1569    /// Returns the integer part of a floating-point value.
1570    ///
1571    /// The stabilized versions of this intrinsic are available on the float primitives via the
1572    /// `trunc` method. For example, [`f32::trunc`](../../std/primitive.f32.html#method.trunc).
1573    #[rustc_nounwind]
1574    #[inline]
1575    #[rustc_intrinsic_const_stable_indirect]
1576    #[rustc_intrinsic]
1577    // The fallback bodies call into `libm`, which is not available at compile time. This is fine
1578    // because const-eval implements this intrinsic itself and never runs the fallback body.
1579    #[rustc_do_not_const_check]
1580    pub const fn trunc<T: bounds::FloatPrimitive>(x: T) -> T;
1581
1582    f16 => { libm::truncf16(x) }
1583    f32 => { libm::truncf(x) }
1584    f64 => { libm::trunc(x) }
1585    f128 => { libm::truncf128(x) }
1586}
1587
1588intrinsic_dispatch_on_type! {
1589    /// Returns the nearest integer to a floating-point value. Rounds half-way cases to the number
1590    /// with an even least significant digit.
1591    ///
1592    /// The stabilized versions of this intrinsic are available on the float primitives via the
1593    /// `round_ties_even` method. For example,
1594    /// [`f32::round_ties_even`](../../std/primitive.f32.html#method.round_ties_even).
1595    #[rustc_nounwind]
1596    #[inline]
1597    #[rustc_intrinsic_const_stable_indirect]
1598    #[rustc_intrinsic]
1599    // The fallback bodies call into `libm`, which is not available at compile time. This is fine
1600    // because const-eval implements this intrinsic itself and never runs the fallback body.
1601    #[rustc_do_not_const_check]
1602    pub const fn round_ties_even<T: bounds::FloatPrimitive>(x: T) -> T;
1603
1604    f16 => { libm::roundevenf16(x) }
1605    f32 => { libm::roundevenf(x) }
1606    f64 => { libm::roundeven(x) }
1607    f128 => { libm::roundevenf128(x) }
1608}
1609
1610intrinsic_dispatch_on_type! {
1611    /// Returns the nearest integer to a floating-point value. Rounds half-way cases away from
1612    /// zero.
1613    ///
1614    /// The stabilized versions of this intrinsic are available on the float primitives via the
1615    /// `round` method. For example, [`f32::round`](../../std/primitive.f32.html#method.round).
1616    #[rustc_nounwind]
1617    #[inline]
1618    #[rustc_intrinsic_const_stable_indirect]
1619    #[rustc_intrinsic]
1620    // The fallback bodies call into `libm`, which is not available at compile time. This is fine
1621    // because const-eval implements this intrinsic itself and never runs the fallback body.
1622    #[rustc_do_not_const_check]
1623    pub const fn round<T: bounds::FloatPrimitive>(x: T) -> T;
1624
1625    f16 => { libm::roundf16(x) }
1626    f32 => { libm::roundf(x) }
1627    f64 => { libm::round(x) }
1628    f128 => { libm::roundf128(x) }
1629}
1630
1631/// Float addition that allows optimizations based on algebraic rules.
1632/// Requires that inputs and output of the operation are finite, causing UB otherwise.
1633///
1634/// This intrinsic does not have a stable counterpart.
1635#[rustc_intrinsic]
1636#[rustc_nounwind]
1637pub unsafe fn fadd_fast<T: bounds::FloatPrimitive>(a: T, b: T) -> T;
1638
1639/// Float subtraction that allows optimizations based on algebraic rules.
1640/// Requires that inputs and output of the operation are finite, causing UB otherwise.
1641///
1642/// This intrinsic does not have a stable counterpart.
1643#[rustc_intrinsic]
1644#[rustc_nounwind]
1645pub unsafe fn fsub_fast<T: bounds::FloatPrimitive>(a: T, b: T) -> T;
1646
1647/// Float multiplication that allows optimizations based on algebraic rules.
1648/// Requires that inputs and output of the operation are finite, causing UB otherwise.
1649///
1650/// This intrinsic does not have a stable counterpart.
1651#[rustc_intrinsic]
1652#[rustc_nounwind]
1653pub unsafe fn fmul_fast<T: bounds::FloatPrimitive>(a: T, b: T) -> T;
1654
1655/// Float division that allows optimizations based on algebraic rules.
1656/// Requires that inputs and output of the operation are finite, causing UB otherwise.
1657///
1658/// This intrinsic does not have a stable counterpart.
1659#[rustc_intrinsic]
1660#[rustc_nounwind]
1661pub unsafe fn fdiv_fast<T: bounds::FloatPrimitive>(a: T, b: T) -> T;
1662
1663/// Float remainder that allows optimizations based on algebraic rules.
1664/// Requires that inputs and output of the operation are finite, causing UB otherwise.
1665///
1666/// This intrinsic does not have a stable counterpart.
1667#[rustc_intrinsic]
1668#[rustc_nounwind]
1669pub unsafe fn frem_fast<T: bounds::FloatPrimitive>(a: T, b: T) -> T;
1670
1671/// Converts with LLVM’s fptoui/fptosi, which may return undef for values out of range
1672/// (<https://github.com/rust-lang/rust/issues/10184>)
1673///
1674/// Stabilized as [`f32::to_int_unchecked`] and [`f64::to_int_unchecked`].
1675#[rustc_intrinsic]
1676#[rustc_nounwind]
1677pub unsafe fn float_to_int_unchecked<Float: bounds::FloatPrimitive, Int: Copy>(value: Float)
1678-> Int;
1679
1680/// Float addition that allows optimizations based on algebraic rules.
1681///
1682/// Stabilized as [`f16::algebraic_add`], [`f32::algebraic_add`], [`f64::algebraic_add`] and [`f128::algebraic_add`].
1683#[rustc_intrinsic_const_stable_indirect]
1684#[rustc_nounwind]
1685#[rustc_intrinsic]
1686pub const fn fadd_algebraic<T: bounds::FloatPrimitive>(a: T, b: T) -> T;
1687
1688/// Float subtraction that allows optimizations based on algebraic rules.
1689///
1690/// Stabilized as [`f16::algebraic_sub`], [`f32::algebraic_sub`], [`f64::algebraic_sub`] and [`f128::algebraic_sub`].
1691#[rustc_intrinsic_const_stable_indirect]
1692#[rustc_nounwind]
1693#[rustc_intrinsic]
1694pub const fn fsub_algebraic<T: bounds::FloatPrimitive>(a: T, b: T) -> T;
1695
1696/// Float multiplication that allows optimizations based on algebraic rules.
1697///
1698/// Stabilized as [`f16::algebraic_mul`], [`f32::algebraic_mul`], [`f64::algebraic_mul`] and [`f128::algebraic_mul`].
1699#[rustc_intrinsic_const_stable_indirect]
1700#[rustc_nounwind]
1701#[rustc_intrinsic]
1702pub const fn fmul_algebraic<T: bounds::FloatPrimitive>(a: T, b: T) -> T;
1703
1704/// Float division that allows optimizations based on algebraic rules.
1705///
1706/// Stabilized as [`f16::algebraic_div`], [`f32::algebraic_div`], [`f64::algebraic_div`] and [`f128::algebraic_div`].
1707#[rustc_intrinsic_const_stable_indirect]
1708#[rustc_nounwind]
1709#[rustc_intrinsic]
1710pub const fn fdiv_algebraic<T: bounds::FloatPrimitive>(a: T, b: T) -> T;
1711
1712/// Float remainder that allows optimizations based on algebraic rules.
1713///
1714/// Stabilized as [`f16::algebraic_rem`], [`f32::algebraic_rem`], [`f64::algebraic_rem`] and [`f128::algebraic_rem`].
1715#[rustc_intrinsic_const_stable_indirect]
1716#[rustc_nounwind]
1717#[rustc_intrinsic]
1718pub const fn frem_algebraic<T: bounds::FloatPrimitive>(a: T, b: T) -> T;
1719
1720/// Integer `min`imum, signed or unsigned depending on `T`.
1721///
1722/// Allowed only on `uN`, `iN`, `usize`, and `isize`.
1723/// (Not on `bool` nor on `char`.)
1724///
1725/// Stabilized as [`u16::min`] and [`i64::min`] and similar.
1726#[rustc_const_unstable(feature = "const_cmp", issue = "143800")]
1727#[rustc_nounwind]
1728#[rustc_intrinsic]
1729#[miri::intrinsic_fallback_is_spec]
1730pub const fn integer_min<T: [const] bounds::IntegerPrimitive>(a: T, b: T) -> T {
1731    if a < b { a } else { b }
1732}
1733
1734/// Integer `max`imum, signed or unsigned depending on `T`.
1735///
1736/// Allowed only on `uN`, `iN`, `usize`, and `isize`.
1737/// (Not on `bool` nor on `char`.)
1738///
1739/// Stabilized as [`u16::max`] and [`i64::max`] and similar.
1740#[rustc_const_unstable(feature = "const_cmp", issue = "143800")]
1741#[rustc_nounwind]
1742#[rustc_intrinsic]
1743#[miri::intrinsic_fallback_is_spec]
1744pub const fn integer_max<T: [const] bounds::IntegerPrimitive>(a: T, b: T) -> T {
1745    if a < b { b } else { a }
1746}
1747
1748/// Returns the number of bits set in an integer type `T`
1749///
1750/// Note that, unlike most intrinsics, this is safe to call;
1751/// it does not require an `unsafe` block.
1752/// Therefore, implementations must not require the user to uphold
1753/// any safety invariants.
1754///
1755/// The stabilized versions of this intrinsic are available on the integer
1756/// primitives via the `count_ones` method. For example,
1757/// [`u32::count_ones`]
1758#[rustc_intrinsic_const_stable_indirect]
1759#[rustc_nounwind]
1760#[rustc_intrinsic]
1761pub const fn ctpop<T: Copy>(x: T) -> u32;
1762
1763/// Returns the number of leading unset bits (zeroes) in an integer type `T`.
1764///
1765/// Note that, unlike most intrinsics, this is safe to call;
1766/// it does not require an `unsafe` block.
1767/// Therefore, implementations must not require the user to uphold
1768/// any safety invariants.
1769///
1770/// The stabilized versions of this intrinsic are available on the integer
1771/// primitives via the `leading_zeros` method. For example,
1772/// [`u32::leading_zeros`]
1773///
1774/// # Examples
1775///
1776/// ```
1777/// #![feature(core_intrinsics)]
1778/// # #![allow(internal_features)]
1779///
1780/// use std::intrinsics::ctlz;
1781///
1782/// let x = 0b0001_1100_u8;
1783/// let num_leading = ctlz(x);
1784/// assert_eq!(num_leading, 3);
1785/// ```
1786///
1787/// An `x` with value `0` will return the bit width of `T`.
1788///
1789/// ```
1790/// #![feature(core_intrinsics)]
1791/// # #![allow(internal_features)]
1792///
1793/// use std::intrinsics::ctlz;
1794///
1795/// let x = 0u16;
1796/// let num_leading = ctlz(x);
1797/// assert_eq!(num_leading, 16);
1798/// ```
1799#[rustc_intrinsic_const_stable_indirect]
1800#[rustc_nounwind]
1801#[rustc_intrinsic]
1802pub const fn ctlz<T: Copy>(x: T) -> u32;
1803
1804/// Like `ctlz`, but extra-unsafe as it returns `undef` when
1805/// given an `x` with value `0`.
1806///
1807/// This intrinsic does not have a stable counterpart.
1808///
1809/// # Examples
1810///
1811/// ```
1812/// #![feature(core_intrinsics)]
1813/// # #![allow(internal_features)]
1814///
1815/// use std::intrinsics::ctlz_nonzero;
1816///
1817/// let x = 0b0001_1100_u8;
1818/// let num_leading = unsafe { ctlz_nonzero(x) };
1819/// assert_eq!(num_leading, 3);
1820/// ```
1821#[rustc_intrinsic_const_stable_indirect]
1822#[rustc_nounwind]
1823#[rustc_intrinsic]
1824pub const unsafe fn ctlz_nonzero<T: Copy>(x: T) -> u32;
1825
1826/// Returns the number of trailing unset bits (zeroes) in an integer type `T`.
1827///
1828/// Note that, unlike most intrinsics, this is safe to call;
1829/// it does not require an `unsafe` block.
1830/// Therefore, implementations must not require the user to uphold
1831/// any safety invariants.
1832///
1833/// The stabilized versions of this intrinsic are available on the integer
1834/// primitives via the `trailing_zeros` method. For example,
1835/// [`u32::trailing_zeros`]
1836///
1837/// # Examples
1838///
1839/// ```
1840/// #![feature(core_intrinsics)]
1841/// # #![allow(internal_features)]
1842///
1843/// use std::intrinsics::cttz;
1844///
1845/// let x = 0b0011_1000_u8;
1846/// let num_trailing = cttz(x);
1847/// assert_eq!(num_trailing, 3);
1848/// ```
1849///
1850/// An `x` with value `0` will return the bit width of `T`:
1851///
1852/// ```
1853/// #![feature(core_intrinsics)]
1854/// # #![allow(internal_features)]
1855///
1856/// use std::intrinsics::cttz;
1857///
1858/// let x = 0u16;
1859/// let num_trailing = cttz(x);
1860/// assert_eq!(num_trailing, 16);
1861/// ```
1862#[rustc_intrinsic_const_stable_indirect]
1863#[rustc_nounwind]
1864#[rustc_intrinsic]
1865pub const fn cttz<T: Copy>(x: T) -> u32;
1866
1867/// Like `cttz`, but extra-unsafe as it returns `undef` when
1868/// given an `x` with value `0`.
1869///
1870/// This intrinsic does not have a stable counterpart.
1871///
1872/// # Examples
1873///
1874/// ```
1875/// #![feature(core_intrinsics)]
1876/// # #![allow(internal_features)]
1877///
1878/// use std::intrinsics::cttz_nonzero;
1879///
1880/// let x = 0b0011_1000_u8;
1881/// let num_trailing = unsafe { cttz_nonzero(x) };
1882/// assert_eq!(num_trailing, 3);
1883/// ```
1884#[rustc_intrinsic_const_stable_indirect]
1885#[rustc_nounwind]
1886#[rustc_intrinsic]
1887pub const unsafe fn cttz_nonzero<T: Copy>(x: T) -> u32;
1888
1889/// Reverses the bytes in an integer type `T`.
1890///
1891/// Note that, unlike most intrinsics, this is safe to call;
1892/// it does not require an `unsafe` block.
1893/// Therefore, implementations must not require the user to uphold
1894/// any safety invariants.
1895///
1896/// The stabilized versions of this intrinsic are available on the integer
1897/// primitives via the `swap_bytes` method. For example,
1898/// [`u32::swap_bytes`]
1899#[rustc_intrinsic_const_stable_indirect]
1900#[rustc_nounwind]
1901#[rustc_intrinsic]
1902pub const fn bswap<T: Copy>(x: T) -> T;
1903
1904/// Reverses the bits in an integer type `T`.
1905///
1906/// Note that, unlike most intrinsics, this is safe to call;
1907/// it does not require an `unsafe` block.
1908/// Therefore, implementations must not require the user to uphold
1909/// any safety invariants.
1910///
1911/// The stabilized versions of this intrinsic are available on the integer
1912/// primitives via the `reverse_bits` method. For example,
1913/// [`u32::reverse_bits`]
1914#[rustc_intrinsic_const_stable_indirect]
1915#[rustc_nounwind]
1916#[rustc_intrinsic]
1917pub const fn bitreverse<T: Copy>(x: T) -> T;
1918
1919/// Does a three-way comparison between the two arguments,
1920/// which must be of character or integer (signed or unsigned) type.
1921///
1922/// This was originally added because it greatly simplified the MIR in `cmp`
1923/// implementations, and then LLVM 20 added a backend intrinsic for it too.
1924///
1925/// The stabilized version of this intrinsic is [`Ord::cmp`].
1926#[rustc_intrinsic_const_stable_indirect]
1927#[rustc_nounwind]
1928#[rustc_intrinsic]
1929pub const fn three_way_compare<T: Copy>(lhs: T, rhss: T) -> crate::cmp::Ordering;
1930
1931/// Combine two values which have no bits in common.
1932///
1933/// This allows the backend to implement it as `a + b` *or* `a | b`,
1934/// depending which is easier to implement on a specific target.
1935///
1936/// # Safety
1937///
1938/// Requires that `(a & b) == 0`, or equivalently that `(a | b) == (a + b)`.
1939///
1940/// Otherwise it's immediate UB.
1941#[rustc_const_unstable(feature = "disjoint_bitor", issue = "135758")]
1942#[rustc_nounwind]
1943#[rustc_intrinsic]
1944#[track_caller]
1945#[miri::intrinsic_fallback_is_spec] // the fallbacks all `assume` to tell Miri
1946pub const unsafe fn disjoint_bitor<T: [const] fallback::DisjointBitOr>(a: T, b: T) -> T {
1947    // SAFETY: same preconditions as this function.
1948    unsafe { fallback::DisjointBitOr::disjoint_bitor(a, b) }
1949}
1950
1951/// Performs checked integer addition.
1952///
1953/// Note that, unlike most intrinsics, this is safe to call;
1954/// it does not require an `unsafe` block.
1955/// Therefore, implementations must not require the user to uphold
1956/// any safety invariants.
1957///
1958/// The stabilized versions of this intrinsic are available on the integer
1959/// primitives via the `overflowing_add` method. For example,
1960/// [`u32::overflowing_add`]
1961#[rustc_intrinsic_const_stable_indirect]
1962#[rustc_nounwind]
1963#[rustc_intrinsic]
1964pub const fn add_with_overflow<T: Copy>(x: T, y: T) -> (T, bool);
1965
1966/// Performs checked integer subtraction
1967///
1968/// Note that, unlike most intrinsics, this is safe to call;
1969/// it does not require an `unsafe` block.
1970/// Therefore, implementations must not require the user to uphold
1971/// any safety invariants.
1972///
1973/// The stabilized versions of this intrinsic are available on the integer
1974/// primitives via the `overflowing_sub` method. For example,
1975/// [`u32::overflowing_sub`]
1976#[rustc_intrinsic_const_stable_indirect]
1977#[rustc_nounwind]
1978#[rustc_intrinsic]
1979pub const fn sub_with_overflow<T: Copy>(x: T, y: T) -> (T, bool);
1980
1981/// Performs checked integer multiplication
1982///
1983/// Note that, unlike most intrinsics, this is safe to call;
1984/// it does not require an `unsafe` block.
1985/// Therefore, implementations must not require the user to uphold
1986/// any safety invariants.
1987///
1988/// The stabilized versions of this intrinsic are available on the integer
1989/// primitives via the `overflowing_mul` method. For example,
1990/// [`u32::overflowing_mul`]
1991#[rustc_intrinsic_const_stable_indirect]
1992#[rustc_nounwind]
1993#[rustc_intrinsic]
1994pub const fn mul_with_overflow<T: Copy>(x: T, y: T) -> (T, bool);
1995
1996/// Performs full-width multiplication and addition with a carry:
1997/// `multiplier * multiplicand + addend + carry`.
1998///
1999/// This is possible without any overflow.  For `uN`:
2000///    MAX * MAX + MAX + MAX
2001/// => (2ⁿ-1) × (2ⁿ-1) + (2ⁿ-1) + (2ⁿ-1)
2002/// => (2²ⁿ - 2ⁿ⁺¹ + 1) + (2ⁿ⁺¹ - 2)
2003/// => 2²ⁿ - 1
2004///
2005/// For `iN`, the upper bound is MIN * MIN + MAX + MAX => 2²ⁿ⁻² + 2ⁿ - 2,
2006/// and the lower bound is MAX * MIN + MIN + MIN => -2²ⁿ⁻² - 2ⁿ + 2ⁿ⁺¹.
2007///
2008/// This currently supports unsigned integers *only*, no signed ones.
2009/// The stabilized versions of this intrinsic are available on integers.
2010#[rustc_const_unstable(feature = "core_intrinsics", issue = "none")]
2011#[rustc_nounwind]
2012#[rustc_intrinsic]
2013#[miri::intrinsic_fallback_is_spec]
2014pub const fn carrying_mul_add<T: [const] fallback::CarryingMulAdd<Unsigned = U>, U>(
2015    multiplier: T,
2016    multiplicand: T,
2017    addend: T,
2018    carry: T,
2019) -> (U, T) {
2020    multiplier.carrying_mul_add(multiplicand, addend, carry)
2021}
2022
2023/// Performs an exact division, resulting in undefined behavior where
2024/// `x % y != 0` or `y == 0` or `x == T::MIN && y == -1`
2025///
2026/// This intrinsic does not have a stable counterpart.
2027#[rustc_intrinsic_const_stable_indirect]
2028#[rustc_nounwind]
2029#[rustc_intrinsic]
2030pub const unsafe fn exact_div<T: Copy>(x: T, y: T) -> T;
2031
2032/// Performs an unchecked division, resulting in undefined behavior
2033/// where `y == 0` or `x == T::MIN && y == -1`
2034///
2035/// Safe wrappers for this intrinsic are available on the integer
2036/// primitives via the `checked_div` method. For example,
2037/// [`u32::checked_div`]
2038#[rustc_intrinsic_const_stable_indirect]
2039#[rustc_nounwind]
2040#[rustc_intrinsic]
2041pub const unsafe fn unchecked_div<T: Copy>(x: T, y: T) -> T;
2042/// Returns the remainder of an unchecked division, resulting in
2043/// undefined behavior when `y == 0` or `x == T::MIN && y == -1`
2044///
2045/// Safe wrappers for this intrinsic are available on the integer
2046/// primitives via the `checked_rem` method. For example,
2047/// [`u32::checked_rem`]
2048#[rustc_intrinsic_const_stable_indirect]
2049#[rustc_nounwind]
2050#[rustc_intrinsic]
2051pub const unsafe fn unchecked_rem<T: Copy>(x: T, y: T) -> T;
2052
2053/// Performs an unchecked left shift, resulting in undefined behavior when
2054/// `y < 0` or `y >= N`, where N is the width of T in bits.
2055///
2056/// Safe wrappers for this intrinsic are available on the integer
2057/// primitives via the `checked_shl` method. For example,
2058/// [`u32::checked_shl`]
2059#[rustc_intrinsic_const_stable_indirect]
2060#[rustc_nounwind]
2061#[rustc_intrinsic]
2062pub const unsafe fn unchecked_shl<T: Copy, U: Copy>(x: T, y: U) -> T;
2063/// Performs an unchecked right shift, resulting in undefined behavior when
2064/// `y < 0` or `y >= N`, where N is the width of T in bits.
2065///
2066/// Safe wrappers for this intrinsic are available on the integer
2067/// primitives via the `checked_shr` method. For example,
2068/// [`u32::checked_shr`]
2069#[rustc_intrinsic_const_stable_indirect]
2070#[rustc_nounwind]
2071#[rustc_intrinsic]
2072pub const unsafe fn unchecked_shr<T: Copy, U: Copy>(x: T, y: U) -> T;
2073
2074/// Returns the result of an unchecked addition, resulting in
2075/// undefined behavior when `x + y > T::MAX` or `x + y < T::MIN`.
2076///
2077/// The stable counterpart of this intrinsic is `unchecked_add` on the various
2078/// integer types, such as [`u16::unchecked_add`] and [`i64::unchecked_add`].
2079#[rustc_intrinsic_const_stable_indirect]
2080#[rustc_nounwind]
2081#[rustc_intrinsic]
2082pub const unsafe fn unchecked_add<T: Copy>(x: T, y: T) -> T;
2083
2084/// Returns the result of an unchecked subtraction, resulting in
2085/// undefined behavior when `x - y > T::MAX` or `x - y < T::MIN`.
2086///
2087/// The stable counterpart of this intrinsic is `unchecked_sub` on the various
2088/// integer types, such as [`u16::unchecked_sub`] and [`i64::unchecked_sub`].
2089#[rustc_intrinsic_const_stable_indirect]
2090#[rustc_nounwind]
2091#[rustc_intrinsic]
2092pub const unsafe fn unchecked_sub<T: Copy>(x: T, y: T) -> T;
2093
2094/// Returns the result of an unchecked multiplication, resulting in
2095/// undefined behavior when `x * y > T::MAX` or `x * y < T::MIN`.
2096///
2097/// The stable counterpart of this intrinsic is `unchecked_mul` on the various
2098/// integer types, such as [`u16::unchecked_mul`] and [`i64::unchecked_mul`].
2099#[rustc_intrinsic_const_stable_indirect]
2100#[rustc_nounwind]
2101#[rustc_intrinsic]
2102pub const unsafe fn unchecked_mul<T: Copy>(x: T, y: T) -> T;
2103
2104/// Performs rotate left.
2105///
2106/// Note that, unlike most intrinsics, this is safe to call;
2107/// it does not require an `unsafe` block.
2108/// Therefore, implementations must not require the user to uphold
2109/// any safety invariants.
2110///
2111/// The stabilized versions of this intrinsic are available on the integer
2112/// primitives via the `rotate_left` method. For example,
2113/// [`u32::rotate_left`]
2114#[rustc_nounwind]
2115#[rustc_intrinsic]
2116#[rustc_allow_const_fn_unstable(const_trait_impl)]
2117#[cfg_attr(miri, track_caller)]
2118#[miri::intrinsic_fallback_is_spec]
2119pub const fn rotate_left<T: fallback::FunnelShift>(x: T, shift: u32) -> T {
2120    // Make sure to call the intrinsic for `funnel_shl`, not the fallback impl.
2121    // SAFETY: we modulo `shift` so that the result is definitely less than the size of
2122    // `T` in bits.
2123    unsafe { unchecked_funnel_shl(x, x, shift % (mem::size_of::<T>() as u32 * 8)) }
2124}
2125
2126/// Performs rotate right.
2127///
2128/// Note that, unlike most intrinsics, this is safe to call;
2129/// it does not require an `unsafe` block.
2130/// Therefore, implementations must not require the user to uphold
2131/// any safety invariants.
2132///
2133/// The stabilized versions of this intrinsic are available on the integer
2134/// primitives via the `rotate_right` method. For example,
2135/// [`u32::rotate_right`]
2136#[rustc_nounwind]
2137#[rustc_intrinsic]
2138#[rustc_allow_const_fn_unstable(const_trait_impl)]
2139#[cfg_attr(miri, track_caller)]
2140#[miri::intrinsic_fallback_is_spec]
2141pub const fn rotate_right<T: fallback::FunnelShift>(x: T, shift: u32) -> T {
2142    // Make sure to call the intrinsic for `funnel_shr`, not the fallback impl.
2143    // SAFETY: we modulo `shift` so that the result is definitely less than the size of
2144    // `T` in bits.
2145    unsafe { unchecked_funnel_shr(x, x, shift % (mem::size_of::<T>() as u32 * 8)) }
2146}
2147
2148/// Wrapping (modular) addition. Computes `a + b`,
2149/// wrapping around at the boundary of the type.
2150///
2151/// Note that, unlike most intrinsics, this is safe to call;
2152/// it does not require an `unsafe` block.
2153/// Therefore, implementations must not require the user to uphold
2154/// any safety invariants.
2155///
2156/// The stabilized versions of this intrinsic are available on the integer
2157/// primitives via the `wrapping_add` method. For example,
2158/// [`u32::wrapping_add`]
2159#[rustc_intrinsic_const_stable_indirect]
2160#[rustc_nounwind]
2161#[rustc_intrinsic]
2162pub const fn wrapping_add<T: Copy>(a: T, b: T) -> T;
2163/// Wrapping (modular) subtraction. Computes `a - b`,
2164/// wrapping around at the boundary of the type.
2165///
2166/// Note that, unlike most intrinsics, this is safe to call;
2167/// it does not require an `unsafe` block.
2168/// Therefore, implementations must not require the user to uphold
2169/// any safety invariants.
2170///
2171/// The stabilized versions of this intrinsic are available on the integer
2172/// primitives via the `wrapping_sub` method. For example,
2173/// [`u32::wrapping_sub`]
2174#[rustc_intrinsic_const_stable_indirect]
2175#[rustc_nounwind]
2176#[rustc_intrinsic]
2177pub const fn wrapping_sub<T: Copy>(a: T, b: T) -> T;
2178/// Wrapping (modular) multiplication. Computes `a *
2179/// b`, wrapping around at the boundary of the type.
2180///
2181/// Note that, unlike most intrinsics, this is safe to call;
2182/// it does not require an `unsafe` block.
2183/// Therefore, implementations must not require the user to uphold
2184/// any safety invariants.
2185///
2186/// The stabilized versions of this intrinsic are available on the integer
2187/// primitives via the `wrapping_mul` method. For example,
2188/// [`u32::wrapping_mul`]
2189#[rustc_intrinsic_const_stable_indirect]
2190#[rustc_nounwind]
2191#[rustc_intrinsic]
2192pub const fn wrapping_mul<T: Copy>(a: T, b: T) -> T;
2193
2194/// Computes `a + b`, saturating at numeric bounds.
2195///
2196/// Note that, unlike most intrinsics, this is safe to call;
2197/// it does not require an `unsafe` block.
2198/// Therefore, implementations must not require the user to uphold
2199/// any safety invariants.
2200///
2201/// The stabilized versions of this intrinsic are available on the integer
2202/// primitives via the `saturating_add` method. For example,
2203/// [`u32::saturating_add`]
2204#[rustc_intrinsic_const_stable_indirect]
2205#[rustc_nounwind]
2206#[rustc_intrinsic]
2207pub const fn saturating_add<T: Copy>(a: T, b: T) -> T;
2208/// Computes `a - b`, saturating at numeric bounds.
2209///
2210/// Note that, unlike most intrinsics, this is safe to call;
2211/// it does not require an `unsafe` block.
2212/// Therefore, implementations must not require the user to uphold
2213/// any safety invariants.
2214///
2215/// The stabilized versions of this intrinsic are available on the integer
2216/// primitives via the `saturating_sub` method. For example,
2217/// [`u32::saturating_sub`]
2218#[rustc_intrinsic_const_stable_indirect]
2219#[rustc_nounwind]
2220#[rustc_intrinsic]
2221pub const fn saturating_sub<T: Copy>(a: T, b: T) -> T;
2222
2223/// Funnel Shift left.
2224///
2225/// Concatenates `a` and `b` (with `a` in the most significant half),
2226/// creating an integer twice as wide. Then shift this integer left
2227/// by `shift`), and extract the most significant half. If `a` and `b`
2228/// are the same, this is equivalent to a rotate left operation.
2229///
2230/// It is undefined behavior if `shift` is greater than or equal to the
2231/// bit size of `T`.
2232///
2233/// Safe versions of this intrinsic are available on the integer primitives
2234/// via the `funnel_shl` method. For example, [`u32::funnel_shl`].
2235#[rustc_intrinsic]
2236#[rustc_nounwind]
2237#[rustc_intrinsic_const_stable_indirect]
2238#[rustc_do_not_const_check] // Use built-in impl to avoid const-checks in the fallback body.
2239pub const unsafe fn unchecked_funnel_shl<T: fallback::FunnelShift>(a: T, b: T, shift: u32) -> T {
2240    // SAFETY: caller ensures that `shift` is in-range
2241    unsafe { a.unchecked_funnel_shl(b, shift) }
2242}
2243
2244/// Funnel Shift right.
2245///
2246/// Concatenates `a` and `b` (with `a` in the most significant half),
2247/// creating an integer twice as wide. Then shift this integer right
2248/// by `shift` (taken modulo the bit size of `T`), and extract the
2249/// least significant half. If `a` and `b` are the same, this is equivalent
2250/// to a rotate right operation.
2251///
2252/// It is undefined behavior if `shift` is greater than or equal to the
2253/// bit size of `T`.
2254///
2255/// Safer versions of this intrinsic are available on the integer primitives
2256/// via the `funnel_shr` method. For example, [`u32::funnel_shr`]
2257#[rustc_intrinsic]
2258#[rustc_nounwind]
2259#[rustc_intrinsic_const_stable_indirect]
2260#[rustc_do_not_const_check] // Ise built-in impl to avoid const-checks in the fallback body.
2261pub const unsafe fn unchecked_funnel_shr<T: fallback::FunnelShift>(a: T, b: T, shift: u32) -> T {
2262    // SAFETY: caller ensures that `shift` is in-range
2263    unsafe { a.unchecked_funnel_shr(b, shift) }
2264}
2265
2266/// Carryless multiply.
2267///
2268/// Safe versions of this intrinsic are available on the integer primitives
2269/// via the `carryless_mul` method. For example, [`u32::carryless_mul`].
2270#[rustc_intrinsic]
2271#[rustc_nounwind]
2272#[rustc_const_unstable(feature = "core_intrinsics", issue = "none")]
2273#[miri::intrinsic_fallback_is_spec]
2274pub const fn carryless_mul<T: [const] fallback::CarrylessMul>(a: T, b: T) -> T {
2275    a.carryless_mul(b)
2276}
2277
2278/// This is an implementation detail of [`crate::ptr::read`] and should
2279/// not be used anywhere else.  See its comments for why this exists.
2280///
2281/// This intrinsic can *only* be called where the pointer is a local without
2282/// projections (`read_via_copy(ptr)`, not `read_via_copy(*ptr)`) so that it
2283/// trivially obeys runtime-MIR rules about derefs in operands.
2284#[rustc_intrinsic_const_stable_indirect]
2285#[rustc_nounwind]
2286#[rustc_intrinsic]
2287pub const unsafe fn read_via_copy<T>(ptr: *const T) -> T;
2288
2289/// This is an implementation detail of [`crate::ptr::write`] and should
2290/// not be used anywhere else.  See its comments for why this exists.
2291///
2292/// This intrinsic can *only* be called where the pointer is a local without
2293/// projections (`write_via_move(ptr, x)`, not `write_via_move(*ptr, x)`) so
2294/// that it trivially obeys runtime-MIR rules about derefs in operands.
2295#[rustc_intrinsic_const_stable_indirect]
2296#[rustc_nounwind]
2297#[rustc_intrinsic]
2298pub const unsafe fn write_via_move<T>(ptr: *mut T, value: T);
2299
2300/// Returns the value of the discriminant for the variant in 'v';
2301/// if `T` has no discriminant, returns `0`.
2302///
2303/// Note that, unlike most intrinsics, this is safe to call;
2304/// it does not require an `unsafe` block.
2305/// Therefore, implementations must not require the user to uphold
2306/// any safety invariants.
2307///
2308/// The stabilized version of this intrinsic is [`core::mem::discriminant`].
2309#[rustc_intrinsic_const_stable_indirect]
2310#[rustc_nounwind]
2311#[rustc_intrinsic]
2312pub const fn discriminant_value<T>(v: &T) -> <T as DiscriminantKind>::Discriminant;
2313
2314/// Rust's "try catch" construct for unwinding. Invokes the function pointer `try_fn` with the
2315/// data pointer `data`, and calls `catch_fn` if unwinding occurs while `try_fn` runs.
2316/// Returns `true` if unwinding occurred and `catch_fn` was called; returns `false` otherwise.
2317///
2318/// `catch_fn` must not unwind.
2319///
2320/// The third argument is a function called if an unwind occurs (both Rust `panic` and foreign
2321/// unwinds). This function takes the data pointer and a pointer to the target- and
2322/// runtime-specific exception object that was caught.
2323///
2324/// Note that in the case of a foreign unwinding operation, the exception object data may not be
2325/// safely usable from Rust, and should not be directly exposed via the standard library. To
2326/// prevent unsafe access, the library implementation may either abort the process or present an
2327/// opaque error type to the user.
2328///
2329/// For more information, see the compiler's source, as well as the documentation for the stable
2330/// version of this intrinsic, `std::panic::catch_unwind`.
2331#[rustc_intrinsic]
2332#[rustc_nounwind]
2333pub unsafe fn catch_unwind<Data: ptr::Thin>(
2334    _try_fn: unsafe fn(*mut Data),
2335    _data: *mut Data,
2336    _catch_fn: unsafe fn(*mut Data, *mut u8),
2337) -> bool;
2338
2339/// Emits a `nontemporal` store, which gives a hint to the CPU that the data should not be held
2340/// in cache. Except for performance, this is fully equivalent to `ptr.write(val)`.
2341///
2342/// Not all architectures provide such an operation. For instance, x86 does not: while `MOVNT`
2343/// exists, that operation is *not* equivalent to `ptr.write(val)` (`MOVNT` writes can be reordered
2344/// in ways that are not allowed for regular writes).
2345#[rustc_intrinsic]
2346#[rustc_nounwind]
2347pub unsafe fn nontemporal_store<T>(ptr: *mut T, val: T);
2348
2349/// See documentation of `<*const T>::offset_from` for details.
2350#[rustc_intrinsic_const_stable_indirect]
2351#[rustc_nounwind]
2352#[rustc_intrinsic]
2353pub const unsafe fn ptr_offset_from<T>(ptr: *const T, base: *const T) -> isize;
2354
2355/// See documentation of `<*const T>::offset_from_unsigned` for details.
2356#[rustc_nounwind]
2357#[rustc_intrinsic]
2358#[rustc_intrinsic_const_stable_indirect]
2359pub const unsafe fn ptr_offset_from_unsigned<T>(ptr: *const T, base: *const T) -> usize;
2360
2361/// See documentation of `<*const T>::guaranteed_eq` for details.
2362/// Returns `2` if the result is unknown.
2363/// Returns `1` if the pointers are guaranteed equal.
2364/// Returns `0` if the pointers are guaranteed inequal.
2365#[rustc_intrinsic]
2366#[rustc_nounwind]
2367#[rustc_do_not_const_check]
2368#[inline]
2369#[miri::intrinsic_fallback_is_spec]
2370pub const fn ptr_guaranteed_cmp<T>(ptr: *const T, other: *const T) -> u8 {
2371    (ptr == other) as u8
2372}
2373
2374/// Determines whether the raw bytes of the two values are equal.
2375///
2376/// This is particularly handy for arrays, since it allows things like just
2377/// comparing `i96`s instead of forcing `alloca`s for `[6 x i16]`.
2378///
2379/// Above some backend-decided threshold this will emit calls to `memcmp`,
2380/// like slice equality does, instead of causing massive code size.
2381///
2382/// Since this works by comparing the underlying bytes, the actual `T` is
2383/// not particularly important.  It will be used for its size and alignment,
2384/// but any validity restrictions will be ignored, not enforced.
2385///
2386/// # Safety
2387///
2388/// It's UB to call this if any of the *bytes* in `*a` or `*b` are uninitialized.
2389/// Note that this is a stricter criterion than just the *values* being
2390/// fully-initialized: if `T` has padding, it's UB to call this intrinsic.
2391///
2392/// At compile-time, it is furthermore UB to call this if any of the bytes
2393/// in `*a` or `*b` have provenance.
2394///
2395/// (The implementation is allowed to branch on the results of comparisons,
2396/// which is UB if any of their inputs are `undef`.)
2397#[rustc_nounwind]
2398#[rustc_intrinsic]
2399pub const unsafe fn raw_eq<T>(a: &T, b: &T) -> bool;
2400
2401/// Lexicographically compare `[left, left + bytes)` and `[right, right + bytes)`
2402/// as unsigned bytes, returning negative if `left` is less, zero if all the
2403/// bytes match, or positive if `left` is greater.
2404///
2405/// This underlies things like `<[u8]>::cmp`, and will usually lower to `memcmp`.
2406///
2407/// # Safety
2408///
2409/// `left` and `right` must each be [valid] for reads of `bytes` bytes.
2410///
2411/// Note that this applies to the whole range, not just until the first byte
2412/// that differs.  That allows optimizations that can read in large chunks.
2413///
2414/// [valid]: crate::ptr#safety
2415#[rustc_nounwind]
2416#[rustc_intrinsic]
2417#[rustc_const_unstable(feature = "const_cmp", issue = "143800")]
2418pub const unsafe fn compare_bytes(left: *const u8, right: *const u8, bytes: usize) -> i32;
2419
2420/// See documentation of [`std::hint::black_box`] for details.
2421///
2422/// [`std::hint::black_box`]: crate::hint::black_box
2423#[rustc_nounwind]
2424#[rustc_intrinsic]
2425#[rustc_intrinsic_const_stable_indirect]
2426pub const fn black_box<T>(dummy: T) -> T;
2427
2428/// Selects which function to call depending on the context.
2429///
2430/// If this function is evaluated at compile-time, then a call to this
2431/// intrinsic will be replaced with a call to `called_in_const`. It gets
2432/// replaced with a call to `called_at_rt` otherwise.
2433///
2434/// This function is safe to call, but note the stability concerns below.
2435///
2436/// # Type Requirements
2437///
2438/// The two functions must be both function items. They cannot be function
2439/// pointers or closures. The first function must be a `const fn`.
2440///
2441/// `arg` will be the tupled arguments that will be passed to either one of
2442/// the two functions, therefore, both functions must accept the same type of
2443/// arguments. Both functions must return RET.
2444///
2445/// # Stability concerns
2446///
2447/// Rust has not yet decided that `const fn` are allowed to tell whether
2448/// they run at compile-time or at runtime. Therefore, when using this
2449/// intrinsic anywhere that can be reached from stable, it is crucial that
2450/// the end-to-end behavior of the stable `const fn` is the same for both
2451/// modes of execution. (Here, Undefined Behavior is considered "the same"
2452/// as any other behavior, so if the function exhibits UB at runtime then
2453/// it may do whatever it wants at compile-time.)
2454///
2455/// Here is an example of how this could cause a problem:
2456/// ```no_run
2457/// #![feature(const_eval_select)]
2458/// #![feature(core_intrinsics)]
2459/// # #![allow(internal_features)]
2460/// use std::intrinsics::const_eval_select;
2461///
2462/// // Standard library
2463/// pub const fn inconsistent() -> i32 {
2464///     fn runtime() -> i32 { 1 }
2465///     const fn compiletime() -> i32 { 2 }
2466///
2467///     // ⚠ This code violates the required equivalence of `compiletime`
2468///     // and `runtime`.
2469///     const_eval_select((), compiletime, runtime)
2470/// }
2471///
2472/// // User Crate
2473/// const X: i32 = inconsistent();
2474/// let x = inconsistent();
2475/// assert_eq!(x, X);
2476/// ```
2477///
2478/// Currently such an assertion would always succeed; until Rust decides
2479/// otherwise, that principle should not be violated.
2480#[rustc_const_unstable(feature = "const_eval_select", issue = "124625")]
2481#[rustc_intrinsic]
2482pub const fn const_eval_select<ARG: Tuple, F, G, RET>(
2483    _arg: ARG,
2484    _called_in_const: F,
2485    _called_at_rt: G,
2486) -> RET
2487where
2488    G: FnOnce<ARG, Output = RET>,
2489    F: const FnOnce<ARG, Output = RET>;
2490
2491/// A macro to make it easier to invoke const_eval_select. Use as follows:
2492/// ```rust,ignore (just a macro example)
2493/// const_eval_select!(
2494///     @capture { arg1: i32 = some_expr, arg2: T = other_expr } -> U:
2495///     if const #[attributes_for_const_arm] {
2496///         // Compile-time code goes here.
2497///     } else #[attributes_for_runtime_arm] {
2498///         // Run-time code goes here.
2499///     }
2500/// )
2501/// ```
2502/// The `@capture` block declares which surrounding variables / expressions can be
2503/// used inside the `if const`.
2504/// Note that the two arms of this `if` really each become their own function, which is why the
2505/// macro supports setting attributes for those functions. Both functions are marked as `#[inline]`.
2506///
2507/// See [`const_eval_select()`] for the rules and requirements around that intrinsic.
2508pub(crate) macro const_eval_select {
2509    (
2510        @capture$([$($binders:tt)*])? { $($arg:ident : $ty:ty = $val:expr),* $(,)? } $( -> $ret:ty )? :
2511        if const
2512            $(#[$compiletime_attr:meta])* $compiletime:block
2513        else
2514            $(#[$runtime_attr:meta])* $runtime:block
2515    ) => {{
2516        #[inline]
2517        $(#[$runtime_attr])*
2518        fn runtime$(<$($binders)*>)?($($arg: $ty),*) $( -> $ret )? {
2519            $runtime
2520        }
2521
2522        #[inline]
2523        $(#[$compiletime_attr])*
2524        const fn compiletime$(<$($binders)*>)?($($arg: $ty),*) $( -> $ret )? {
2525            // Don't warn if one of the arguments is unused.
2526            $(let _ = $arg;)*
2527
2528            $compiletime
2529        }
2530
2531        const_eval_select(($($val,)*), compiletime, runtime)
2532    }},
2533    // We support leaving away the `val` expressions for *all* arguments
2534    // (but not for *some* arguments, that's too tricky).
2535    (
2536        @capture$([$($binders:tt)*])? { $($arg:ident : $ty:ty),* $(,)? } $( -> $ret:ty )? :
2537        if const
2538            $(#[$compiletime_attr:meta])* $compiletime:block
2539        else
2540            $(#[$runtime_attr:meta])* $runtime:block
2541    ) => {
2542        $crate::intrinsics::const_eval_select!(
2543            @capture$([$($binders)*])? { $($arg : $ty = $arg),* } $(-> $ret)? :
2544            if const
2545                $(#[$compiletime_attr])* $compiletime
2546            else
2547                $(#[$runtime_attr])* $runtime
2548        )
2549    },
2550}
2551
2552/// Returns whether the argument's value is statically known at
2553/// compile-time.
2554///
2555/// This is useful when there is a way of writing the code that will
2556/// be *faster* when some variables have known values, but *slower*
2557/// in the general case: an `if is_val_statically_known(var)` can be used
2558/// to select between these two variants. The `if` will be optimized away
2559/// and only the desired branch remains.
2560///
2561/// Formally speaking, this function non-deterministically returns `true`
2562/// or `false`, and the caller has to ensure sound behavior for both cases.
2563/// In other words, the following code has *Undefined Behavior*:
2564///
2565/// ```no_run
2566/// #![feature(core_intrinsics)]
2567/// # #![allow(internal_features)]
2568/// use std::hint::unreachable_unchecked;
2569/// use std::intrinsics::is_val_statically_known;
2570///
2571/// if !is_val_statically_known(0) { unsafe { unreachable_unchecked(); } }
2572/// ```
2573///
2574/// This also means that the following code's behavior is unspecified; it
2575/// may panic, or it may not:
2576///
2577/// ```no_run
2578/// #![feature(core_intrinsics)]
2579/// # #![allow(internal_features)]
2580/// use std::intrinsics::is_val_statically_known;
2581///
2582/// assert_eq!(is_val_statically_known(0), is_val_statically_known(0));
2583/// ```
2584///
2585/// Unsafe code may not rely on `is_val_statically_known` returning any
2586/// particular value, ever. However, the compiler will generally make it
2587/// return `true` only if the value of the argument is actually known.
2588///
2589/// # Type Requirements
2590///
2591/// `T` must be either a `bool`, a `char`, a primitive numeric type (e.g. `f32`,
2592/// but not `NonZeroISize`), or any thin pointer (e.g. `*mut String`).
2593/// Any other argument types *may* cause a compiler error.
2594///
2595/// ## Pointers
2596///
2597/// When the input is a pointer, only the pointer itself is
2598/// ever considered. The pointee has no effect. Currently, these functions
2599/// behave identically:
2600///
2601/// ```
2602/// #![feature(core_intrinsics)]
2603/// # #![allow(internal_features)]
2604/// use std::intrinsics::is_val_statically_known;
2605///
2606/// fn foo(x: &i32) -> bool {
2607///     is_val_statically_known(x)
2608/// }
2609///
2610/// fn bar(x: &i32) -> bool {
2611///     is_val_statically_known(
2612///         (x as *const i32).addr()
2613///     )
2614/// }
2615/// # _ = foo(&5_i32);
2616/// # _ = bar(&5_i32);
2617/// ```
2618#[rustc_const_stable_indirect]
2619#[rustc_nounwind]
2620#[rustc_intrinsic]
2621pub const fn is_val_statically_known<T: Copy>(_arg: T) -> bool {
2622    false
2623}
2624
2625/// Non-overlapping *typed* swap of a single value.
2626///
2627/// The codegen backends will replace this with a better implementation when
2628/// `T` is a simple type that can be loaded and stored as an immediate.
2629///
2630/// The stabilized form of this intrinsic is [`crate::mem::swap`].
2631///
2632/// # Safety
2633/// Behavior is undefined if any of the following conditions are violated:
2634///
2635/// * Both `x` and `y` must be [valid] for both reads and writes.
2636///
2637/// * Both `x` and `y` must be properly aligned.
2638///
2639/// * The region of memory beginning at `x` must *not* overlap with the region of memory
2640///   beginning at `y`.
2641///
2642/// * The memory pointed by `x` and `y` must both contain values of type `T`.
2643///
2644/// [valid]: crate::ptr#safety
2645#[rustc_nounwind]
2646#[inline]
2647#[rustc_intrinsic]
2648pub const unsafe fn typed_swap_nonoverlapping<T>(x: *mut T, y: *mut T) {
2649    // SAFETY: The caller provided single non-overlapping items behind
2650    // pointers, so swapping them with `count: 1` is fine.
2651    unsafe { ptr::swap_nonoverlapping(x, y, 1) };
2652}
2653
2654/// Returns whether we should perform some UB-checking at runtime. This eventually evaluates to
2655/// `cfg!(ub_checks)`, but behaves different from `cfg!` when mixing crates built with different
2656/// flags: if the crate has UB checks enabled or carries the `#[rustc_preserve_ub_checks]`
2657/// attribute, evaluation is delayed until monomorphization (or until the call gets inlined into
2658/// a crate that does not delay evaluation further); otherwise it can happen any time.
2659///
2660/// The common case here is a user program built with ub_checks linked against the distributed
2661/// sysroot which is built without ub_checks but with `#[rustc_preserve_ub_checks]`.
2662/// For code that gets monomorphized in the user crate (i.e., generic functions and functions with
2663/// `#[inline]`), gating assertions on `ub_checks()` rather than `cfg!(ub_checks)` means that
2664/// assertions are enabled whenever the *user crate* has UB checks enabled. However, if the
2665/// user has UB checks disabled, the checks will still get optimized out. This intrinsic is
2666/// primarily used by [`crate::ub_checks::assert_unsafe_precondition`].
2667///
2668/// # Consteval
2669///
2670/// In consteval, this function currently returns `true`. This is because the value of the `ub_checks`
2671/// configuration can differ across crates, but we need this function to always return the same
2672/// value in consteval in order to avoid unsoundness.
2673#[inline(always)]
2674#[rustc_intrinsic]
2675pub const fn ub_checks() -> bool {
2676    cfg!(ub_checks)
2677}
2678
2679/// Returns whether we should perform some overflow-checking at runtime. This eventually evaluates to
2680/// `cfg!(overflow_checks)`, but behaves different from `cfg!` when mixing crates built with different
2681/// flags: if the crate has overflow checks enabled or carries the `#[rustc_inherit_overflow_checks]`
2682/// attribute, evaluation is delayed until monomorphization (or until the call gets inlined into
2683/// a crate that does not delay evaluation further); otherwise it can happen any time.
2684///
2685/// The common case here is a user program built with overflow_checks linked against the distributed
2686/// sysroot which is built without overflow_checks but with `#[rustc_inherit_overflow_checks]`.
2687/// For code that gets monomorphized in the user crate (i.e., generic functions and functions with
2688/// `#[inline]`), gating assertions on `overflow_checks()` rather than `cfg!(overflow_checks)` means that
2689/// assertions are enabled whenever the *user crate* has overflow checks enabled. However if the
2690/// user has overflow checks disabled, the checks will still get optimized out.
2691///
2692/// # Consteval
2693///
2694/// In consteval, this function currently returns `true`. This is because the value of the `overflow_checks`
2695/// configuration can differ across crates, but we need this function to always return the same
2696/// value in consteval in order to avoid unsoundness.
2697#[inline(always)]
2698#[rustc_intrinsic]
2699pub const fn overflow_checks() -> bool {
2700    cfg!(debug_assertions)
2701}
2702
2703/// Allocates a block of memory at compile time.
2704/// At runtime, just returns a null pointer.
2705///
2706/// # Safety
2707///
2708/// - The `align` argument must be a power of two.
2709///    - At compile time, a compile error occurs if this constraint is violated.
2710///    - At runtime, it is not checked.
2711#[rustc_const_unstable(feature = "const_heap", issue = "79597")]
2712#[rustc_nounwind]
2713#[rustc_intrinsic]
2714#[miri::intrinsic_fallback_is_spec]
2715pub const unsafe fn const_allocate(_size: usize, _align: usize) -> *mut u8 {
2716    // const eval overrides this function, but runtime code for now just returns null pointers.
2717    // See <https://github.com/rust-lang/rust/issues/93935>.
2718    crate::ptr::null_mut()
2719}
2720
2721/// Deallocates a memory which allocated by `intrinsics::const_allocate` at compile time.
2722/// At runtime, it does nothing.
2723///
2724/// # Safety
2725///
2726/// - The `align` argument must be a power of two.
2727///    - At compile time, a compile error occurs if this constraint is violated.
2728///    - At runtime, it is not checked.
2729/// - If the `ptr` is created in an another const, this intrinsic doesn't deallocate it.
2730/// - If the `ptr` is pointing to a local variable, this intrinsic doesn't deallocate it.
2731#[rustc_const_unstable(feature = "core_intrinsics", issue = "none")]
2732#[rustc_nounwind]
2733#[rustc_intrinsic]
2734#[miri::intrinsic_fallback_is_spec]
2735pub const unsafe fn const_deallocate(_ptr: *mut u8, _size: usize, _align: usize) {
2736    // Runtime NOP
2737}
2738
2739/// Convert the allocation this pointer points to into immutable global memory.
2740/// The pointer must point to the beginning of a heap allocation.
2741/// This operation only makes sense during compile time. At runtime, it does nothing.
2742#[rustc_const_unstable(feature = "const_heap", issue = "79597")]
2743#[rustc_nounwind]
2744#[rustc_intrinsic]
2745#[miri::intrinsic_fallback_is_spec]
2746pub const unsafe fn const_make_global(ptr: *mut u8) -> *const u8 {
2747    // const eval overrides this function; at runtime, it is a NOP.
2748    ptr
2749}
2750
2751/// Check if the pre-condition `cond` has been met.
2752///
2753/// By default, if `contract_checks` is enabled, this will panic with no unwind if the condition
2754/// returns false.
2755///
2756/// Note that this function is a no-op during constant evaluation.
2757#[unstable(feature = "contracts_internals", issue = "128044")]
2758// Calls to this function get inserted by an AST expansion pass, which uses the equivalent of
2759// `#[allow_internal_unstable]` to allow using `contracts_internals` functions. Const-checking
2760// doesn't honor `#[allow_internal_unstable]`, so for the const feature gate we use the user-facing
2761// `contracts` feature rather than the perma-unstable `contracts_internals`
2762#[rustc_const_unstable(feature = "contracts", issue = "128044")]
2763#[lang = "contract_check_requires"]
2764#[rustc_intrinsic]
2765pub const fn contract_check_requires<C: Fn() -> bool + Copy>(cond: C) {
2766    const_eval_select!(
2767        @capture[C: Fn() -> bool + Copy] { cond: C } :
2768        if const {
2769                // Do nothing
2770        } else {
2771            if !cond() {
2772                // Emit no unwind panic in case this was a safety requirement.
2773                crate::panicking::panic_nounwind("failed requires check");
2774            }
2775        }
2776    )
2777}
2778
2779/// Check if the post-condition `cond` has been met.
2780///
2781/// By default, if `contract_checks` is enabled, this will panic with no unwind if the condition
2782/// returns false.
2783///
2784/// If `cond` is `None`, then no postcondition checking is performed.
2785///
2786/// Note that this function is a no-op during constant evaluation.
2787#[unstable(feature = "contracts_internals", issue = "128044")]
2788// Similar to `contract_check_requires`, we need to use the user-facing
2789// `contracts` feature rather than the perma-unstable `contracts_internals`.
2790// Const-checking doesn't honor allow_internal_unstable logic used by contract expansion.
2791#[rustc_const_unstable(feature = "contracts", issue = "128044")]
2792#[lang = "contract_check_ensures"]
2793#[rustc_intrinsic]
2794pub const fn contract_check_ensures<C: Fn(&Ret) -> bool + Copy, Ret>(
2795    cond: Option<C>,
2796    ret: Ret,
2797) -> Ret {
2798    const_eval_select!(
2799        @capture[C: Fn(&Ret) -> bool + Copy, Ret] { cond: Option<C>, ret: Ret } -> Ret :
2800        if const {
2801            // Do nothing
2802            ret
2803        } else {
2804            if let crate::option::Option::Some(cond) = cond && !cond(&ret) {
2805                // Emit no unwind panic in case this was a safety requirement.
2806                crate::panicking::panic_nounwind("failed ensures check");
2807            }
2808            ret
2809        }
2810    )
2811}
2812
2813/// The intrinsic will return the size stored in that vtable.
2814///
2815/// # Safety
2816///
2817/// `ptr` must point to a vtable.
2818#[rustc_nounwind]
2819#[rustc_intrinsic]
2820pub unsafe fn vtable_size(ptr: *const ()) -> usize;
2821
2822/// The intrinsic will return the alignment stored in that vtable.
2823///
2824/// # Safety
2825///
2826/// `ptr` must point to a vtable.
2827#[rustc_nounwind]
2828#[rustc_intrinsic]
2829pub unsafe fn vtable_align(ptr: *const ()) -> usize;
2830
2831/// The size of a type in bytes.
2832///
2833/// Note that, unlike most intrinsics, this is safe to call;
2834/// it does not require an `unsafe` block.
2835/// Therefore, implementations must not require the user to uphold
2836/// any safety invariants.
2837///
2838/// More specifically, this is the offset in bytes between successive
2839/// items of the same type, including alignment padding.
2840///
2841/// Note that, unlike most intrinsics, this can only be called at compile-time
2842/// as backends do not have an implementation for it. The only caller (its
2843/// stable counterpart) wraps this intrinsic call in a `const` block so that
2844/// backends only see an evaluated constant.
2845///
2846/// The stabilized version of this intrinsic is [`core::mem::size_of`].
2847#[rustc_nounwind]
2848#[rustc_intrinsic_const_stable_indirect]
2849#[rustc_intrinsic]
2850#[rustc_comptime]
2851pub fn size_of<T>() -> usize;
2852
2853/// The minimum alignment of a type.
2854///
2855/// Note that, unlike most intrinsics, this is safe to call;
2856/// it does not require an `unsafe` block.
2857/// Therefore, implementations must not require the user to uphold
2858/// any safety invariants.
2859///
2860/// Note that, unlike most intrinsics, this can only be called at compile-time
2861/// as backends do not have an implementation for it. The only caller (its
2862/// stable counterpart) wraps this intrinsic call in a `const` block so that
2863/// backends only see an evaluated constant.
2864///
2865/// The stabilized version of this intrinsic is [`core::mem::align_of`].
2866#[rustc_nounwind]
2867#[rustc_intrinsic_const_stable_indirect]
2868#[rustc_intrinsic]
2869#[rustc_comptime]
2870pub fn align_of<T>() -> usize;
2871
2872/// The offset of a field inside a type.
2873///
2874/// Note that, unlike most intrinsics, this is safe to call;
2875/// it does not require an `unsafe` block.
2876/// Therefore, implementations must not require the user to uphold
2877/// any safety invariants.
2878///
2879/// This intrinsic can only be evaluated at compile-time, and should only appear in
2880/// constants or inline const blocks.
2881///
2882/// The stabilized version of this intrinsic is [`core::mem::offset_of`].
2883/// This intrinsic is also a lang item so `offset_of!` can desugar to calls to it.
2884#[rustc_nounwind]
2885#[rustc_intrinsic_const_stable_indirect]
2886#[rustc_intrinsic]
2887#[lang = "offset_of"]
2888#[rustc_comptime]
2889pub fn offset_of<T: PointeeSized>(variant: u32, field: u32) -> usize;
2890
2891/// The offset of a field queried by its field representing type.
2892///
2893/// Returns the offset of the field represented by `F`. This function essentially does the same as
2894/// the [`offset_of`] intrinsic, but expects the field to be represented by a generic rather than
2895/// the variant and field indices. This also is a safe intrinsic and can only be evaluated at
2896/// compile-time, so it should only appear in constants or inline const blocks.
2897///
2898/// There should be no need to call this intrinsic manually, as its value is used to define
2899/// [`Field::OFFSET`](crate::field::Field::OFFSET), which is publicly accessible.
2900#[rustc_intrinsic]
2901#[rustc_comptime]
2902pub fn field_offset<F: crate::field::Field>() -> usize;
2903
2904/// Returns the number of variants of the type `T` cast to a `usize`;
2905/// if `T` has no variants, returns `0`. Uninhabited variants will be counted.
2906///
2907/// Note that, unlike most intrinsics, this can only be called at compile-time
2908/// as backends do not have an implementation for it. The only caller (its
2909/// stable counterpart) wraps this intrinsic call in a `const` block so that
2910/// backends only see an evaluated constant.
2911///
2912/// The to-be-stabilized version of this intrinsic is [`crate::mem::variant_count`].
2913#[rustc_nounwind]
2914#[rustc_intrinsic]
2915#[rustc_comptime]
2916pub fn variant_count<T>() -> usize;
2917
2918/// The size of the referenced value in bytes.
2919///
2920/// The stabilized version of this intrinsic is [`core::mem::size_of_val`].
2921///
2922/// # Safety
2923///
2924/// See [`crate::mem::size_of_val_raw`] for safety conditions.
2925#[rustc_nounwind]
2926#[rustc_intrinsic]
2927#[rustc_intrinsic_const_stable_indirect]
2928pub const unsafe fn size_of_val<T: ?Sized>(ptr: *const T) -> usize;
2929
2930/// The required alignment of the referenced value.
2931///
2932/// The stabilized version of this intrinsic is [`core::mem::align_of_val`].
2933///
2934/// # Safety
2935///
2936/// See [`crate::mem::align_of_val_raw`] for safety conditions.
2937#[rustc_nounwind]
2938#[rustc_intrinsic]
2939#[rustc_intrinsic_const_stable_indirect]
2940pub const unsafe fn align_of_val<T: ?Sized>(ptr: *const T) -> usize;
2941
2942/// Gets a static string slice containing the name of a type.
2943///
2944/// Note that, unlike most intrinsics, this can only be called at compile-time
2945/// as backends do not have an implementation for it. The only caller (its
2946/// stable counterpart) wraps this intrinsic call in a `const` block so that
2947/// backends only see an evaluated constant.
2948///
2949/// The stabilized version of this intrinsic is [`core::any::type_name`].
2950#[rustc_nounwind]
2951#[rustc_intrinsic]
2952#[rustc_comptime]
2953pub fn type_name<T: ?Sized>() -> &'static str;
2954
2955/// Gets the actual field `TypeId` of the [`FieldRepresentingType`]'s `TypeId`.
2956///
2957/// The more user-friendly version of this intrinsic is [`core::mem::type_info::FieldId::type_id`].
2958///
2959/// [`FieldRepresentingType`]: crate::field::FieldRepresentingType
2960#[rustc_intrinsic]
2961#[rustc_comptime]
2962pub fn field_representing_type_actual_type_id(
2963    _frt_type_id: crate::any::TypeId,
2964) -> crate::any::TypeId;
2965
2966/// Gets the name of the field represented by the [`FieldRepresentingType`]'s `TypeId`.
2967///
2968/// The more user-friendly version of this intrinsic is [`core::mem::type_info::FieldId::name`].
2969///
2970/// [`FieldRepresentingType`]: crate::field::FieldRepresentingType
2971#[rustc_intrinsic]
2972#[rustc_comptime]
2973pub fn field_representing_type_name(_frt_type_id: crate::any::TypeId) -> &'static str;
2974
2975/// Gets the name of the field represented by the [`FieldRepresentingType`]'s `TypeId`.
2976///
2977/// The more user-friendly version of this intrinsic is [`core::mem::type_info::FieldId::name`].
2978///
2979/// [`FieldRepresentingType`]: crate::field::FieldRepresentingType
2980#[rustc_intrinsic]
2981#[rustc_comptime]
2982pub fn field_representing_type_offset(_frt_type_id: crate::any::TypeId) -> usize;
2983
2984/// Lowers in MIR to `Rvalue::Aggregate` with `AggregateKind::RawPtr`.
2985///
2986/// This is used to implement functions like `slice::from_raw_parts_mut` and
2987/// `ptr::from_raw_parts` in a way compatible with the compiler being able to
2988/// change the possible layouts of pointers.
2989#[rustc_nounwind]
2990#[rustc_intrinsic_const_stable_indirect]
2991#[rustc_intrinsic]
2992pub const fn aggregate_raw_ptr<P: bounds::BuiltinDeref, D, M>(data: D, meta: M) -> P
2993where
2994    <P as bounds::BuiltinDeref>::Pointee: ptr::Pointee<Metadata = M>;
2995
2996/// Lowers in MIR to `Rvalue::UnaryOp` with `UnOp::PtrMetadata`.
2997///
2998/// This is used to implement functions like `ptr::metadata`.
2999#[rustc_nounwind]
3000#[rustc_intrinsic_const_stable_indirect]
3001#[rustc_intrinsic]
3002pub const fn ptr_metadata<P: ptr::Pointee<Metadata = M> + PointeeSized, M>(ptr: *const P) -> M;
3003
3004/// This is an accidentally-stable alias to [`ptr::copy_nonoverlapping`]; use that instead.
3005// Note (intentionally not in the doc comment): `ptr::copy_nonoverlapping` adds some extra
3006// debug assertions; if you are writing compiler tests or code inside the standard library
3007// that wants to avoid those debug assertions, directly call this intrinsic instead.
3008#[stable(feature = "rust1", since = "1.0.0")]
3009#[rustc_allowed_through_unstable_modules(
3010    message = "import this function via the `ptr` module instead",
3011    module = "ptr"
3012)]
3013#[rustc_const_stable(feature = "const_intrinsic_copy", since = "1.83.0")]
3014#[rustc_nounwind]
3015#[rustc_intrinsic]
3016pub const unsafe fn copy_nonoverlapping<T>(src: *const T, dst: *mut T, count: usize);
3017
3018/// This is an accidentally-stable alias to [`ptr::copy`]; use that instead.
3019// Note (intentionally not in the doc comment): `ptr::copy` adds some extra
3020// debug assertions; if you are writing compiler tests or code inside the standard library
3021// that wants to avoid those debug assertions, directly call this intrinsic instead.
3022#[stable(feature = "rust1", since = "1.0.0")]
3023#[rustc_allowed_through_unstable_modules(
3024    message = "import this function via the `ptr` module instead",
3025    module = "ptr"
3026)]
3027#[rustc_const_stable(feature = "const_intrinsic_copy", since = "1.83.0")]
3028#[rustc_nounwind]
3029#[rustc_intrinsic]
3030pub const unsafe fn copy<T>(src: *const T, dst: *mut T, count: usize);
3031
3032/// This is an accidentally-stable alias to [`ptr::write_bytes`]; use that instead.
3033// Note (intentionally not in the doc comment): `ptr::write_bytes` adds some extra
3034// debug assertions; if you are writing compiler tests or code inside the standard library
3035// that wants to avoid those debug assertions, directly call this intrinsic instead.
3036#[stable(feature = "rust1", since = "1.0.0")]
3037#[rustc_allowed_through_unstable_modules(
3038    message = "import this function via the `ptr` module instead",
3039    module = "ptr"
3040)]
3041#[rustc_const_stable(feature = "const_intrinsic_copy", since = "1.83.0")]
3042#[rustc_nounwind]
3043#[rustc_intrinsic]
3044pub const unsafe fn write_bytes<T>(dst: *mut T, val: u8, count: usize);
3045
3046/// Returns the minimum of two floating-point values, ignoring NaN.
3047///
3048/// This behaves like IEEE 754-2019 minimumNumber, *except* that it does not order signed
3049/// zeros deterministically. In particular:
3050/// If one of the arguments is NaN (quiet or signaling), then the other argument is returned. If
3051/// both arguments are NaN, returns NaN. If the inputs compare equal (such as for the case of `+0.0`
3052/// and `-0.0`), either input may be returned non-deterministically.
3053///
3054/// Note that, unlike most intrinsics, this is safe to call;
3055/// it does not require an `unsafe` block.
3056/// Therefore, implementations must not require the user to uphold
3057/// any safety invariants.
3058///
3059/// The stabilized versions of this intrinsic are available on the float primitives via the
3060/// `min` method. For example, [`f32::min`].
3061#[rustc_nounwind]
3062#[rustc_intrinsic_const_stable_indirect]
3063#[rustc_intrinsic]
3064#[rustc_do_not_const_check]
3065pub const fn minimum_number_nsz<T: bounds::FloatPrimitive>(x: T, y: T) -> T {
3066    if x.is_nan() || y <= x {
3067        y
3068    } else {
3069        // Either y > x or y is a NaN.
3070        x
3071    }
3072}
3073
3074/// Returns the maximum of two floating-point values, ignoring NaN.
3075///
3076/// This behaves like IEEE 754-2019 maximumNumber, *except* that it does not order signed
3077/// zeros deterministically. In particular:
3078/// If one of the arguments is NaN (quiet or signaling), then the other argument is returned. If
3079/// both arguments are NaN, returns NaN. If the inputs compare equal (such as for the case of `+0.0`
3080/// and `-0.0`), either input may be returned non-deterministically.
3081///
3082/// Note that, unlike most intrinsics, this is safe to call;
3083/// it does not require an `unsafe` block.
3084/// Therefore, implementations must not require the user to uphold
3085/// any safety invariants.
3086///
3087/// The stabilized versions of this intrinsic are available on the float primitives via the
3088/// `max` method. For example, [`f32::max`].
3089#[rustc_nounwind]
3090#[rustc_intrinsic_const_stable_indirect]
3091#[rustc_intrinsic]
3092#[rustc_do_not_const_check]
3093pub const fn maximum_number_nsz<T: bounds::FloatPrimitive>(x: T, y: T) -> T {
3094    if x.is_nan() || y >= x {
3095        y
3096    } else {
3097        // Either y < x or y is a NaN.
3098        x
3099    }
3100}
3101
3102/// Returns the minimum of two floating-point values, propagating NaN.
3103///
3104/// This behaves like IEEE 754-2019 minimum. In particular:
3105/// If one of the arguments is NaN, then a NaN is returned using the usual NaN propagation rules.
3106/// For this operation, -0.0 is considered to be strictly less than +0.0.
3107///
3108/// Note that, unlike most intrinsics, this is safe to call;
3109/// it does not require an `unsafe` block.
3110/// Therefore, implementations must not require the user to uphold
3111/// any safety invariants.
3112#[rustc_nounwind]
3113#[rustc_const_unstable(feature = "core_intrinsics", issue = "none")]
3114#[rustc_intrinsic]
3115#[rustc_do_not_const_check]
3116pub const fn minimum<T: bounds::FloatPrimitive>(x: T, y: T) -> T {
3117    if x < y {
3118        x
3119    } else if y < x {
3120        y
3121    } else if x == y {
3122        if x.is_sign_negative() && y.is_sign_positive() { x } else { y }
3123    } else {
3124        // At least one input is NaN. Use `+` to perform NaN propagation and quieting.
3125        x + y
3126    }
3127}
3128
3129/// Returns the maximum of two floating-point values, propagating NaN.
3130///
3131/// This behaves like IEEE 754-2019 maximum. In particular:
3132/// If one of the arguments is NaN, then a NaN is returned using the usual NaN propagation rules.
3133/// For this operation, -0.0 is considered to be strictly less than +0.0.
3134///
3135/// Note that, unlike most intrinsics, this is safe to call;
3136/// it does not require an `unsafe` block.
3137/// Therefore, implementations must not require the user to uphold
3138/// any safety invariants.
3139#[rustc_nounwind]
3140#[rustc_const_unstable(feature = "core_intrinsics", issue = "none")]
3141#[rustc_intrinsic]
3142#[rustc_do_not_const_check]
3143pub const fn maximum<T: bounds::FloatPrimitive>(x: T, y: T) -> T {
3144    if x > y {
3145        x
3146    } else if y > x {
3147        y
3148    } else if x == y {
3149        if x.is_sign_positive() && y.is_sign_negative() { x } else { y }
3150    } else {
3151        // At least one input is NaN. Use `+` to perform NaN propagation and quieting.
3152        x + y
3153    }
3154}
3155
3156/// Returns the absolute value of a floating-point value.
3157///
3158/// The stabilized versions of this intrinsic are available on the float
3159/// primitives via the `abs` method. For example, [`f32::abs`].
3160#[rustc_nounwind]
3161#[rustc_intrinsic]
3162#[rustc_intrinsic_const_stable_indirect]
3163#[rustc_do_not_const_check] // Use built-in impl to avoid const-checks in the fallback body.
3164#[miri::intrinsic_fallback_is_spec]
3165pub const fn fabs<T: bounds::FloatPrimitive>(x: T) -> T {
3166    T::from_bits(x.to_bits() & !T::SIGN_MASK)
3167}
3168
3169/// Copies the sign from `y` to `x` for floating-point values.
3170///
3171/// The stabilized versions of this intrinsic are available on the float
3172/// primitives via the `copysign` method. For example, [`f32::copysign`].
3173#[inline]
3174#[rustc_nounwind]
3175#[rustc_intrinsic_const_stable_indirect]
3176#[rustc_intrinsic]
3177#[rustc_do_not_const_check]
3178#[miri::intrinsic_fallback_is_spec]
3179pub const fn copysign<T: bounds::FloatPrimitive>(x: T, y: T) -> T {
3180    T::from_bits((x.to_bits() & !T::SIGN_MASK) | (y.to_bits() & T::SIGN_MASK))
3181}
3182
3183/// Generates the LLVM body for the automatic differentiation of `f` using Enzyme,
3184/// with `df` as the derivative function and `args` as its arguments.
3185///
3186/// Used internally as the body of `df` when expanding the `#[autodiff_forward]`
3187/// and `#[autodiff_reverse]` attribute macros.
3188///
3189/// Type Parameters:
3190/// - `F`: The original function to differentiate. Must be a function item.
3191/// - `G`: The derivative function. Must be a function item.
3192/// - `T`: A tuple of arguments passed to `df`.
3193/// - `R`: The return type of the derivative function.
3194///
3195/// This shows where the `autodiff` intrinsic is used during macro expansion:
3196///
3197/// ```rust,ignore (macro example)
3198/// #[autodiff_forward(df1, Dual, Const, Dual)]
3199/// pub fn f1(x: &[f64], y: f64) -> f64 {
3200///     unimplemented!()
3201/// }
3202/// ```
3203///
3204/// expands to:
3205///
3206/// ```rust,ignore (macro example)
3207/// #[rustc_autodiff]
3208/// #[inline(never)]
3209/// pub fn f1(x: &[f64], y: f64) -> f64 {
3210///     ::core::panicking::panic("not implemented")
3211/// }
3212/// #[rustc_autodiff(Forward, 1, Dual, Const, Dual)]
3213/// pub fn df1(x: &[f64], bx_0: &[f64], y: f64) -> (f64, f64) {
3214///     ::core::intrinsics::autodiff(f1::<>, df1::<>, (x, bx_0, y))
3215/// }
3216/// ```
3217#[rustc_nounwind]
3218#[rustc_intrinsic]
3219pub const fn autodiff<F, G, T: crate::marker::Tuple, R>(f: F, df: G, args: T) -> R;
3220
3221/// Generates the LLVM body of a wrapper function to offload a kernel `f`.
3222///
3223/// Type Parameters:
3224/// - `F`: The kernel to offload. Must be a function item.
3225/// - `T`: A tuple of arguments passed to `f`.
3226/// - `R`: The return type of the kernel.
3227///
3228/// Arguments:
3229/// - `f`: The kernel function to offload.
3230/// - `workgroup_dim`: A 3D size specifying the number of workgroups to launch.
3231/// - `thread_dim`: A 3D size specifying the number of threads per workgroup.
3232/// - `dyn_cache`: The amount of dynamic shared memory to request for the kernel.
3233/// - `device_id`: The device to offload to. Use `-1` to select the default device.
3234/// - `args`: A tuple of arguments forwarded to `f`.
3235///
3236/// Example usage (pseudocode):
3237///
3238/// ```rust,ignore (pseudocode)
3239/// fn kernel(x: *mut [f64; 128]) {
3240///     core::intrinsics::offload(kernel_1, [256, 1, 1], [32, 1, 1], 0, -1, (x,))
3241/// }
3242///
3243/// #[cfg(target_os = "linux")]
3244/// extern "C" {
3245///     pub fn kernel_1(array_b: *mut [f64; 128]);
3246/// }
3247///
3248/// #[cfg(not(target_os = "linux"))]
3249/// #[rustc_offload_kernel]
3250/// extern "gpu-kernel" fn kernel_1(x: *mut [f64; 128]) {
3251///     unsafe { (*x)[0] = 21.0 };
3252/// }
3253/// ```
3254///
3255/// For reference, see the Clang documentation on offloading:
3256/// <https://clang.llvm.org/docs/OffloadingDesign.html>.
3257#[rustc_nounwind]
3258#[rustc_intrinsic]
3259pub const fn offload<F, T: crate::marker::Tuple, R>(
3260    f: F,
3261    workgroup_dim: [u32; 3],
3262    thread_dim: [u32; 3],
3263    dyn_cache: u32,
3264    device_id: i32,
3265    args: T,
3266) -> R;
3267
3268/// Returns the number of offload devices available on the system.
3269///
3270/// Use this to discover which `device_id` values are valid to pass to
3271/// [`offload`]. Devices are numbered from `0` to the returned value minus one.
3272///
3273/// Returns `0` if no offloading devices are present.
3274#[rustc_nounwind]
3275#[rustc_intrinsic]
3276pub const fn offload_get_num_devices() -> i32;
3277
3278/// Inform Miri that a given pointer definitely has a certain alignment.
3279#[cfg(miri)]
3280#[rustc_allow_const_fn_unstable(const_eval_select)]
3281pub(crate) const fn miri_promise_symbolic_alignment(ptr: *const (), align: usize) {
3282    unsafe extern "Rust" {
3283        /// Miri-provided extern function to promise that a given pointer is properly aligned for
3284        /// "symbolic" alignment checks. Will fail if the pointer is not actually aligned or `align` is
3285        /// not a power of two. Has no effect when alignment checks are concrete (which is the default).
3286        fn miri_promise_symbolic_alignment(ptr: *const (), align: usize);
3287    }
3288
3289    const_eval_select!(
3290        @capture { ptr: *const (), align: usize}:
3291        if const {
3292            // Do nothing.
3293        } else {
3294            // SAFETY: this call is always safe.
3295            unsafe {
3296                miri_promise_symbolic_alignment(ptr, align);
3297            }
3298        }
3299    )
3300}
3301
3302/// Loads an argument of type `T` from the `va_list` `ap` and increment the
3303/// argument `ap` points to.
3304///
3305/// # Safety
3306///
3307/// This function is only sound to call when:
3308///
3309/// - there is a next variable argument available.
3310/// - the next argument's type must be ABI-compatible with the type `T`.
3311/// - the next argument must have a properly initialized value of type `T`.
3312///
3313/// Calling this function with an incompatible type, an invalid value, or when there
3314/// are no more variable arguments, is unsound.
3315///
3316#[rustc_intrinsic]
3317#[rustc_nounwind]
3318pub const unsafe fn va_arg<T: VaArgSafe>(ap: &mut VaList<'_>) -> T;
3319
3320/// Duplicates a variable argument list. The returned list is initially at the same position as
3321/// the one in `src`, but can be advanced independently.
3322///
3323/// Codegen backends should not have custom behavior for this intrinsic, they should always use
3324/// this fallback implementation. This intrinsic *does not* map to the LLVM `va_copy` intrinsic.
3325///
3326/// This intrinsic exists only as a hook for Miri and constant evaluation, and is used to detect UB
3327/// when a variable argument list is used incorrectly.
3328#[rustc_intrinsic]
3329#[rustc_nounwind]
3330pub const fn va_copy<'f>(src: &VaList<'f>) -> VaList<'f> {
3331    // This fallback body exploits the fact that our codegen backends all just use
3332    // a plain memcpy to duplicate VaList. This assumption is wrong for Miri.
3333    assert!(!cfg!(miri), "fallback body is incorrect under Miri");
3334
3335    src.duplicate()
3336}
3337
3338/// Destroy the variable argument list `ap` after initialization with `va_start` (part of the
3339/// desugaring of `...`) or `va_copy`.
3340///
3341/// Code generation backends should not provide a custom implementation for this intrinsic. This
3342/// intrinsic *does not* map to the LLVM `va_end` intrinsic.
3343///
3344/// This function is a no-op on all current targets, but used as a hook for const evaluation to
3345/// detect UB when a variable argument list is used incorrectly.
3346///
3347/// # Safety
3348///
3349/// `ap` must not be used to access variable arguments after this call.
3350///
3351#[rustc_intrinsic]
3352#[rustc_nounwind]
3353pub const unsafe fn va_end(ap: &mut VaList<'_>) {
3354    /* deliberately does nothing */
3355}
3356
3357/// Returns the return address of the caller function (after inlining) in a best-effort manner or a null pointer if it is not supported on the current backend.
3358/// Returning an accurate value is a quality-of-implementation concern, but no hard guarantees are
3359/// made about the return value: formally, the intrinsic non-deterministically returns
3360/// an arbitrary pointer without provenance.
3361///
3362/// Note that unlike most intrinsics, this is safe to call. This is because it only finds the return address of the immediate caller, which is guaranteed to be possible.
3363/// Other forms of the corresponding gcc or llvm intrinsic (which can have wildly unpredictable results or even crash at runtime) are not exposed.
3364#[rustc_intrinsic]
3365#[rustc_nounwind]
3366pub fn return_address() -> *const () {
3367    core::ptr::null()
3368}